Trigger.dev Content Pipeline
95 durable background tasks spanning article generation, social discovery, media production, newsletter automation, and content distribution, with full Langfuse observability, 33 platform integrations, and 12 automated cron schedules.
Highlights
- 95 durable task files across 11 operational domains
- Full Langfuse tracing for every LLM call with cost tracking and quality scoring
- 33 platform integrations (Gemini, Replicate, ElevenLabs, HeyGen, Cloudinary, Resend, Tavily, and more)
- 12 automated cron schedules for daily operations
- Media production: audio TTS, video generation, carousel rendering, image pipelines
- Multi-channel publishing: CMS, newsletters, social distribution
The Engine
This is the production backbone of Trending Society: a durable task pipeline that handles the entire content lifecycle from ingestion through publishing. It coordinates AI-powered article generation, editorial media production, newsletter assembly, and multi-channel distribution across a unified execution layer built on Trigger.dev.
Architecture: 11 Domains
The pipeline is organized into 11 task domains, each responsible for a distinct operational concern:
| Domain | Tasks | Purpose |
|---|---|---|
| Article | 16 | Core article pipeline: classify, research, rewrite, image generation, publishing |
| Media | 12 | Audio synthesis (ElevenLabs), video generation, carousel rendering (Remotion), image pipelines |
| Cron | 12 | Automated schedules: daily digest, RSS polling, newsletter assembly, health checks |
| AI | 7 | Vision analysis (Gemini), financial data enrichment, carousel data extraction |
| Content | 6 | URL ingestion, metadata enrichment, newsletter link processing |
| Sync | 6 | Cross-platform delivery: email (Resend), social publishing, Google Drive |
| Social | 5 | Social discovery and distribution |
| Editorial | 1 | Long-form editorial generation |
| Newsletter | 1 | Deep-dive newsletter content generation |
| Video | 1 | AI video generation with multi-model routing |
| Remix | 1 | Cross-format content adaptation orchestrator |
33 Platform Integrations
Every external service is abstracted into a shared integration layer with typed clients and centralized error handling:
- AI Models: Gemini (text + vision), Replicate (image + video models), OpenAI, Anthropic
- Media: Cloudinary (asset management), ElevenLabs (speech synthesis), HeyGen (avatar video), Remotion (motion graphics)
- Data: Supabase (primary database), Airtable, Google Sheets/Drive
- Distribution: Resend (transactional + marketing email), Slack, Telegram
- Ingestion: Apify (structured data extraction), Inoreader (RSS feeds)
- Observability: Langfuse (LLM tracing + cost tracking), Linear (project management), GitHub (CI/CD)
- Commerce: Stripe (billing), Shopify (storefront)
Observability
Every LLM call is traced through Langfuse with full production telemetry:
- Prompt versioning via a centralized prompt registry
- Per-generation cost tracking (input/output tokens × model pricing)
- Deterministic quality scoring with automated pass/fail gates
- Full trace trees showing end-to-end pipeline execution
- Thousands of generations tracked with complete audit trails
Durability
Tasks are built on Trigger.dev's durable execution model. Every step is individually retryable, and every failure is captured with structured error context. Critical steps halt the pipeline and trigger alert hooks. Non-critical enrichment steps degrade gracefully and allow the pipeline to continue. The system is designed for reliability at scale with no silent failures.
