The Engine
This is the production backbone of Trending Society: a durable task pipeline that handles the entire content lifecycle from ingestion through publishing. It coordinates AI-powered article generation, editorial media production, newsletter assembly, and multi-channel distribution across a unified execution layer built on Trigger.dev.
Architecture: 12 Domains
The pipeline is organized into 12 task domains, each responsible for a distinct operational concern:
| Domain | Tasks | Purpose |
|---|---|---|
| Article | 16 | Core article pipeline: classify, research, rewrite, image generation, publishing |
| Media | 16 | Audio synthesis (ElevenLabs), video generation, carousel rendering (Remotion), image pipelines |
| Cron | 16 | Automated schedules: daily digest, RSS polling, newsletter assembly, health checks |
| AI | 8 | Vision analysis (Gemini), financial data enrichment, carousel data extraction |
| Content | 11 | URL ingestion, metadata enrichment, newsletter link processing |
| Sync | 7 | Cross-platform delivery: email (Resend), social publishing, Google Drive |
| Social | 15 | Social ingestion, enrichment, and scheduled posting |
| Editorial | 1 | Long-form editorial generation |
| Newsletter | 1 | Deep-dive newsletter content generation |
| Video | 1 | AI video generation with multi-model routing |
| Repurpose | 1 | Cross-format content adaptation orchestrator |
| Release | 2 | Release detection and release-to-post drafting |
35 Platform Integrations
Every external service is abstracted into a shared integration layer with typed clients and centralized error handling:
- AI Models: Gemini (text + vision), Anthropic, Replicate (image + video models)
- Media: Cloudinary (asset management), ElevenLabs (speech synthesis), HeyGen (avatar video), Remotion (motion graphics)
- Data: Supabase (primary database), Google Sheets/Drive
- Distribution: Resend (transactional + marketing email), Slack, Telegram
- Ingestion: Apify (structured data extraction), Inoreader (RSS feeds)
- Observability: Langfuse (LLM tracing + cost tracking), Linear (project management), GitHub (CI/CD)
- Commerce: Stripe (billing), Shopify (storefront)
Observability
LLM calls are traced through Langfuse with full production telemetry:
- Prompts versioned in code and resolved through Trigger.dev prompts
- Per-generation cost tracking (input/output tokens × model pricing)
- A quality score written to each generation and sent to Langfuse
- Full trace trees showing end-to-end pipeline execution
- 31,871 of 33,850 LLM calls carry a trace
Durability
Tasks are built on Trigger.dev's durable execution model. Every step is individually retryable, and every failure is captured with structured error context. Critical steps halt the pipeline and trigger alert hooks. Non-critical enrichment steps degrade gracefully and allow the pipeline to continue. Retries run up to 3 attempts, and a task can run for up to an hour.
