Data Pipeline & ETL Design

Help me design a data pipeline for [MY USE CASE].

Data source: [WHERE DATA COMES FROM]
Destination: [WHERE IT NEEDS TO GO]

Volume: [DAILY RECORDS]
Frequency: [REAL-TIME / HOURLY / DAILY / WEEKLY]
Tech preference: [PYTHON / SQL / AIRFLOW / CLOUD / OPEN TO SUGGESTIONS]

Provide:
1. **Pipeline Architecture**: Source → transform → load design
2. **Data Extraction**: How to pull data from the source reliably
3. **Transformation Logic**: Cleaning, mapping, and enriching data
4. **Loading Strategy**: Incremental vs full load approach
5. **Scheduling**: How to run the pipeline on a schedule
6. **Error Handling**: What to do when data is missing, malformed, or late
7. **Monitoring**: How to know the pipeline is working (and when it isn't)
8. **Data Quality**: Validation checks to ensure data integrity
9. **Scaling**: How the pipeline handles growing data volumes
10. **Tool Selection**: Best tools for my scale and technical level

Help me design a data pipeline for [MY USE CASE].

Data source: [WHERE DATA COMES FROM]
Destination: [WHERE IT NEEDS TO GO]
Volume: [DAILY RECORDS]
Frequency: [REAL-TIME / HOURLY / DAILY / WEEKLY]
Tech preference: [PYTHON / SQL / AIRFLOW / CLOUD / OPEN TO SUGGESTIONS]

Provide:
1. **Pipeline Architecture**: Source → transform → load design
2. **Data Extraction**: How to pull data from the source reliably
3. **Transformation Logic**: Cleaning, mapping, and enriching data
4. **Loading Strategy**: Incremental vs full load approach
5. **Scheduling**: How to run the pipeline on a schedule
6. **Error Handling**: What to do when data is missing, malformed, or late
7. **Monitoring**: How to know the pipeline is working (and when it isn't)
8. **Data Quality**: Validation checks to ensure data integrity
9. **Scaling**: How the pipeline handles growing data volumes
10. **Tool Selection**: Best tools for my scale and technical level