ETL Pipeline Designer

Design an ETL (Extract, Transform, Load) pipeline for [DATA USE CASE].

Sources: [LIST DATA SOURCES]
Destination: [DATA WAREHOUSE / LAKE / DATABASE]

Volume: [DATA SIZE AND FREQUENCY]

Provide:
1. **Architecture Diagram**: Source → Extract → Transform → Load → Serve
2. **Extract**: Connection method for each source (API, database, file, stream)
3. **Transform**: Business logic, data cleaning rules, calculations, joins
4. **Load**: Insert strategy (full refresh, incremental, upsert, SCD)
5. **Scheduling**: Orchestration tool and DAG design (Airflow, dbt, Prefect)
6. **Data Quality**: Validation checks at each stage
7. **Error Handling**: What happens when a step fails (retry, alert, quarantine)
8. **Monitoring**: Pipeline health dashboard and alerting
9. **Documentation**: Data dictionary and lineage for all transformed fields
10. **Code**: Complete implementation in [PYTHON / SQL / DBT]

Design an ETL (Extract, Transform, Load) pipeline for [DATA USE CASE].

Sources: [LIST DATA SOURCES]
Destination: [DATA WAREHOUSE / LAKE / DATABASE]
Volume: [DATA SIZE AND FREQUENCY]

Provide:
1. **Architecture Diagram**: Source → Extract → Transform → Load → Serve
2. **Extract**: Connection method for each source (API, database, file, stream)
3. **Transform**: Business logic, data cleaning rules, calculations, joins
4. **Load**: Insert strategy (full refresh, incremental, upsert, SCD)
5. **Scheduling**: Orchestration tool and DAG design (Airflow, dbt, Prefect)
6. **Data Quality**: Validation checks at each stage
7. **Error Handling**: What happens when a step fails (retry, alert, quarantine)
8. **Monitoring**: Pipeline health dashboard and alerting
9. **Documentation**: Data dictionary and lineage for all transformed fields
10. **Code**: Complete implementation in [PYTHON / SQL / DBT]