Real-Time Data Pipeline Builder

Build a real-time data pipeline for [USE CASE: live dashboard / event tracking / notifications / analytics].

Stack: [LANGUAGE/FRAMEWORK]
Data sources: [DESCRIBE]

Latency requirement: [SECONDS]

Provide:
1. Architecture: Event source → Message queue → Processor → Storage → Consumer
2. Message Queue: Kafka vs Redis Streams vs RabbitMQ — selection and setup
3. Event Schema: Define event types with TypeScript/JSON Schema
4. Producer: Code for publishing events
5. Consumer: Code for processing events (transformation, enrichment)
6. Storage: Time-series DB or analytics DB setup
7. Real-Time API: WebSocket or SSE endpoint for frontend
8. Error handling: Dead letter queue, retry logic, alerting
9. Scaling: How to handle 10x, 100x current volume
10. Monitoring: Pipeline health dashboard and lag alerting

Build a real-time data pipeline for [USE CASE: live dashboard / event tracking / notifications / analytics].

Stack: [LANGUAGE/FRAMEWORK]
Data sources: [DESCRIBE]
Latency requirement: [SECONDS]

Provide:
1. Architecture: Event source → Message queue → Processor → Storage → Consumer
2. Message Queue: Kafka vs Redis Streams vs RabbitMQ — selection and setup
3. Event Schema: Define event types with TypeScript/JSON Schema
4. Producer: Code for publishing events
5. Consumer: Code for processing events (transformation, enrichment)
6. Storage: Time-series DB or analytics DB setup
7. Real-Time API: WebSocket or SSE endpoint for frontend
8. Error handling: Dead letter queue, retry logic, alerting
9. Scaling: How to handle 10x, 100x current volume
10. Monitoring: Pipeline health dashboard and lag alerting