Profession 29 · Free professional tutorial
Data Engineer & AI Data Pipeline Specialist: Complete Prompt Library
120 ready-to-customize prompts across SQL, Python, ETL/ELT, Airflow, dbt, Spark, data warehouses, cloud streaming, reliability and stakeholder delivery. Learn the workflow, copy a prompt, replace the bracketed fields, and independently validate every output.
120 prompts12 modulesSearch + filterCopy-ready templates
How to use this library
Step 1. Choose your deliverable or search by task/shortcode. Step 2. Use Copy Prompt and replace the seven bracketed project inputs with approved, non-sensitive details. Step 3. Ask your AI assistant to generate a proposed artifact. Step 4. Verify schema, data quality, security, runbooks and estimates in a safe test environment before production.
The IDs below are navigation references, not executable WordPress shortcodes. AI-generated SQL, configuration and pipeline code must be reviewed and tested. Never paste credentials or private production data into public AI tools.
Find a prompt
120 of 120 prompts
No matches. Try a shorter term or reset filters.
120 copy-ready prompts
Module 01: Discovery & Architecture
Ten practical templates for discovery & architecture.
Source Inventory
Use this template to create or review a source inventory deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Source Inventory. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on source inventory; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for source inventory; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for source inventory, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Source Inventory: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Contract Discovery
Use this template to create or review a data contract discovery deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Contract Discovery. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data contract discovery; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data contract discovery; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data contract discovery, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Contract Discovery: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Business Requirements Translation
Use this template to create or review a business requirements translation deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Business Requirements Translation. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on business requirements translation; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for business requirements translation; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for business requirements translation, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Business Requirements Translation: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Source-To-Target Mapping
Use this template to create or review a source-to-target mapping deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Source-To-Target Mapping. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on source-to-target mapping; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for source-to-target mapping; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for source-to-target mapping, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Source-To-Target Mapping: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Batch Versus Streaming Decision
Use this template to create or review a batch versus streaming decision deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Batch Versus Streaming Decision. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on batch versus streaming decision; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for batch versus streaming decision; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for batch versus streaming decision, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Batch Versus Streaming Decision: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Architecture Options Matrix
Use this template to create or review a architecture options matrix deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Architecture Options Matrix. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on architecture options matrix; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for architecture options matrix; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for architecture options matrix, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Architecture Options Matrix: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Flow Diagram
Use this template to create or review a data flow diagram deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Flow Diagram. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data flow diagram; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data flow diagram; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data flow diagram, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Flow Diagram: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Ownership Raci
Use this template to create or review a data ownership RACI deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Ownership Raci. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data ownership RACI; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data ownership RACI; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data ownership RACI, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Ownership Raci: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Nonfunctional Requirements
Use this template to create or review a nonfunctional requirements deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Nonfunctional Requirements. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on nonfunctional requirements; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for nonfunctional requirements; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for nonfunctional requirements, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Nonfunctional Requirements: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Delivery Roadmap
Use this template to create or review a delivery roadmap deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Delivery Roadmap. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on delivery roadmap; Map business outcomes to data contracts, system boundaries, dependencies and measurable acceptance criteria. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for delivery roadmap; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for delivery roadmap, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Delivery Roadmap: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 02: SQL & Data Modeling
Ten practical templates for sql & data modeling.
Relational Schema Assessment
Use this template to create or review a relational schema assessment deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Relational Schema Assessment. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on relational schema assessment; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for relational schema assessment; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for relational schema assessment, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Relational Schema Assessment: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Dimensional Model Blueprint
Use this template to create or review a dimensional model blueprint deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Dimensional Model Blueprint. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on dimensional model blueprint; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for dimensional model blueprint; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for dimensional model blueprint, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Dimensional Model Blueprint: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Star Schema Design
Use this template to create or review a star schema design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Star Schema Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on star schema design; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for star schema design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for star schema design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Star Schema Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Slowly Changing Dimensions
Use this template to create or review a slowly changing dimensions deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Slowly Changing Dimensions. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on slowly changing dimensions; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for slowly changing dimensions; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for slowly changing dimensions, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Slowly Changing Dimensions: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Sql Transformation Review
Use this template to create or review a SQL transformation review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Sql Transformation Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on SQL transformation review; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for SQL transformation review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for SQL transformation review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Sql Transformation Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Window Function Design
Use this template to create or review a window function design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Window Function Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on window function design; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for window function design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for window function design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Window Function Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Incremental Merge Logic
Use this template to create or review a incremental merge logic deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Incremental Merge Logic. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on incremental merge logic; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for incremental merge logic; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for incremental merge logic, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Incremental Merge Logic: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Partition And Clustering Plan
Use this template to create or review a partition and clustering plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Partition And Clustering Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on partition and clustering plan; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for partition and clustering plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for partition and clustering plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Partition And Clustering Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Query Optimization Review
Use this template to create or review a query optimization review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Query Optimization Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on query optimization review; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for query optimization review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for query optimization review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Query Optimization Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Model Documentation
Use this template to create or review a data model documentation deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Model Documentation. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data model documentation; Specify table grains, keys, SQL behavior, sample rows and validation queries. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data model documentation; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data model documentation, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Model Documentation: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 03: Python & Pipeline Coding
Ten practical templates for python & pipeline coding.
Python Ingestion Scaffold
Use this template to create or review a Python ingestion scaffold deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Python Ingestion Scaffold. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Python ingestion scaffold; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Python ingestion scaffold; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Python ingestion scaffold, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Python Ingestion Scaffold: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Pandas Transformation Plan
Use this template to create or review a Pandas transformation plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Pandas Transformation Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Pandas transformation plan; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Pandas transformation plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Pandas transformation plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Pandas Transformation Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Api Pagination Handler
Use this template to create or review a API pagination handler deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Api Pagination Handler. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on API pagination handler; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for API pagination handler; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for API pagination handler, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Api Pagination Handler: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Retry And Backoff Design
Use this template to create or review a retry and backoff design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Retry And Backoff Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on retry and backoff design; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for retry and backoff design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for retry and backoff design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Retry And Backoff Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Configuration Management
Use this template to create or review a configuration management deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Configuration Management. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on configuration management; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for configuration management; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for configuration management, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Configuration Management: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Type Validation Strategy
Use this template to create or review a type validation strategy deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Type Validation Strategy. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on type validation strategy; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for type validation strategy; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for type validation strategy, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Type Validation Strategy: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Memory Efficient File Processing
Use this template to create or review a memory efficient file processing deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Memory Efficient File Processing. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on memory efficient file processing; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for memory efficient file processing; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for memory efficient file processing, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Memory Efficient File Processing: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Unit-Test Fixtures
Use this template to create or review a unit-test fixtures deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Unit-Test Fixtures. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on unit-test fixtures; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for unit-test fixtures; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for unit-test fixtures, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Unit-Test Fixtures: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Pipeline Package Structure
Use this template to create or review a pipeline package structure deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Pipeline Package Structure. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on pipeline package structure; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for pipeline package structure; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for pipeline package structure, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Pipeline Package Structure: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Code Review And Refactoring
Use this template to create or review a code review and refactoring deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Code Review And Refactoring. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on code review and refactoring; Produce readable Python design or pseudocode, dependencies, error handling and test cases. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for code review and refactoring; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for code review and refactoring, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Code Review And Refactoring: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 04: ETL, ELT & Orchestration
Ten practical templates for etl, elt & orchestration.
Etl Versus Elt Comparison
Use this template to create or review a ETL versus ELT comparison deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Etl Versus Elt Comparison. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on ETL versus ELT comparison; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for ETL versus ELT comparison; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for ETL versus ELT comparison, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Etl Versus Elt Comparison: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Extract Step Specification
Use this template to create or review a extract step specification deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Extract Step Specification. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on extract step specification; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for extract step specification; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for extract step specification, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Extract Step Specification: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Transformation Dependency Graph
Use this template to create or review a transformation dependency graph deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Transformation Dependency Graph. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on transformation dependency graph; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for transformation dependency graph; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for transformation dependency graph, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Transformation Dependency Graph: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Idempotent Loading
Use this template to create or review a idempotent loading deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Idempotent Loading. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on idempotent loading; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for idempotent loading; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for idempotent loading, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Idempotent Loading: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Late-Arriving Data Handling
Use this template to create or review a late-arriving data handling deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Late-Arriving Data Handling. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on late-arriving data handling; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for late-arriving data handling; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for late-arriving data handling, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Late-Arriving Data Handling: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Backfill Runbook
Use this template to create or review a backfill runbook deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Backfill Runbook. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on backfill runbook; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for backfill runbook; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for backfill runbook, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Backfill Runbook: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Airflow Dag Outline
Use this template to create or review a Airflow DAG outline deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Airflow Dag Outline. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Airflow DAG outline; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Airflow DAG outline; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Airflow DAG outline, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Airflow Dag Outline: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Schedule And Sla Design
Use this template to create or review a schedule and SLA design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Schedule And Sla Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on schedule and SLA design; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for schedule and SLA design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for schedule and SLA design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Schedule And Sla Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Job Dependency Failure Handling
Use this template to create or review a job dependency failure handling deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Job Dependency Failure Handling. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on job dependency failure handling; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for job dependency failure handling; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for job dependency failure handling, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Job Dependency Failure Handling: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Orchestration Migration Plan
Use this template to create or review a orchestration migration plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Orchestration Migration Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on orchestration migration plan; Define load cadence, idempotency, retry, backfill and orchestrator state transitions. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for orchestration migration plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for orchestration migration plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Orchestration Migration Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 05: dbt & Analytics Engineering
Ten practical templates for dbt & analytics engineering.
Dbt Project Structure
Use this template to create or review a dbt project structure deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Dbt Project Structure. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on dbt project structure; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for dbt project structure; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for dbt project structure, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Dbt Project Structure: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Staging Model Specification
Use this template to create or review a staging model specification deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Staging Model Specification. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on staging model specification; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for staging model specification; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for staging model specification, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Staging Model Specification: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Intermediate Model Design
Use this template to create or review a intermediate model design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Intermediate Model Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on intermediate model design; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for intermediate model design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for intermediate model design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Intermediate Model Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Mart Model Definition
Use this template to create or review a mart model definition deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Mart Model Definition. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on mart model definition; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for mart model definition; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for mart model definition, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Mart Model Definition: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Dbt Incremental Strategy
Use this template to create or review a dbt incremental strategy deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Dbt Incremental Strategy. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on dbt incremental strategy; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for dbt incremental strategy; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for dbt incremental strategy, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Dbt Incremental Strategy: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Schema Tests Design
Use this template to create or review a schema tests design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Schema Tests Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on schema tests design; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for schema tests design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for schema tests design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Schema Tests Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Custom Business Test
Use this template to create or review a custom business test deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Custom Business Test. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on custom business test; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for custom business test; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for custom business test, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Custom Business Test: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Source Freshness Rules
Use this template to create or review a source freshness rules deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Source Freshness Rules. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on source freshness rules; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for source freshness rules; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for source freshness rules, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Source Freshness Rules: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Model Lineage Documentation
Use this template to create or review a model lineage documentation deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Model Lineage Documentation. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on model lineage documentation; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for model lineage documentation; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for model lineage documentation, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Model Lineage Documentation: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Dbt Pull-Request Review
Use this template to create or review a dbt pull-request review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Dbt Pull-Request Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on dbt pull-request review; Use explicit dbt refs, sources, materialization choices, lineage and actionable tests. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for dbt pull-request review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for dbt pull-request review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Dbt Pull-Request Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 06: Spark & Distributed Processing
Ten practical templates for spark & distributed processing.
Spark Use-Case Assessment
Use this template to create or review a Spark use-case assessment deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Spark Use-Case Assessment. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Spark use-case assessment; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Spark use-case assessment; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Spark use-case assessment, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Spark Use-Case Assessment: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Dataframe Transformation Plan
Use this template to create or review a DataFrame transformation plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Dataframe Transformation Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on DataFrame transformation plan; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for DataFrame transformation plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for DataFrame transformation plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Dataframe Transformation Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Join Strategy Review
Use this template to create or review a join strategy review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Join Strategy Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on join strategy review; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for join strategy review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for join strategy review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Join Strategy Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Skew Diagnosis
Use this template to create or review a data skew diagnosis deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Skew Diagnosis. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data skew diagnosis; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data skew diagnosis; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data skew diagnosis, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Skew Diagnosis: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Shuffle Reduction Proposal
Use this template to create or review a shuffle reduction proposal deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Shuffle Reduction Proposal. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on shuffle reduction proposal; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for shuffle reduction proposal; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for shuffle reduction proposal, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Shuffle Reduction Proposal: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Spark Partition Planning
Use this template to create or review a Spark partition planning deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Spark Partition Planning. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Spark partition planning; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Spark partition planning; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Spark partition planning, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Spark Partition Planning: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Schema Evolution In Parquet
Use this template to create or review a schema evolution in Parquet deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Schema Evolution In Parquet. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on schema evolution in Parquet; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for schema evolution in Parquet; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for schema evolution in Parquet, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Schema Evolution In Parquet: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Distributed Job Testing
Use this template to create or review a distributed job testing deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Distributed Job Testing. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on distributed job testing; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for distributed job testing; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for distributed job testing, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Distributed Job Testing: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Spark Performance Investigation
Use this template to create or review a Spark performance investigation deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Spark Performance Investigation. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Spark performance investigation; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Spark performance investigation; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Spark performance investigation, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Spark Performance Investigation: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Batch-To-Stream Migration
Use this template to create or review a batch-to-stream migration deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Batch-To-Stream Migration. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on batch-to-stream migration; Address partitioning, shuffle, skew, executor resources and measurable performance evidence. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for batch-to-stream migration; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for batch-to-stream migration, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Batch-To-Stream Migration: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 07: Warehouses & Lakehouses
Ten practical templates for warehouses & lakehouses.
Warehouse Platform Comparison
Use this template to create or review a warehouse platform comparison deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Warehouse Platform Comparison. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on warehouse platform comparison; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for warehouse platform comparison; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for warehouse platform comparison, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Warehouse Platform Comparison: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Lake Zone Architecture
Use this template to create or review a data lake zone architecture deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Lake Zone Architecture. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data lake zone architecture; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data lake zone architecture; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data lake zone architecture, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Lake Zone Architecture: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Lakehouse Table Layout
Use this template to create or review a lakehouse table layout deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Lakehouse Table Layout. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on lakehouse table layout; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for lakehouse table layout; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for lakehouse table layout, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Lakehouse Table Layout: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Snowflake Loading Design
Use this template to create or review a Snowflake loading design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Snowflake Loading Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Snowflake loading design; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Snowflake loading design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Snowflake loading design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Snowflake Loading Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Bigquery Cost-Aware Query Plan
Use this template to create or review a BigQuery cost-aware query plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Bigquery Cost-Aware Query Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on BigQuery cost-aware query plan; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for BigQuery cost-aware query plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for BigQuery cost-aware query plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Bigquery Cost-Aware Query Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Redshift Table Review
Use this template to create or review a Redshift table review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Redshift Table Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Redshift table review; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Redshift table review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Redshift table review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Redshift Table Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Delta Or Iceberg Adoption Plan
Use this template to create or review a Delta or Iceberg adoption plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Delta Or Iceberg Adoption Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Delta or Iceberg adoption plan; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Delta or Iceberg adoption plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Delta or Iceberg adoption plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Delta Or Iceberg Adoption Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Retention And Archival Policy
Use this template to create or review a retention and archival policy deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Retention And Archival Policy. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on retention and archival policy; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for retention and archival policy; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for retention and archival policy, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Retention And Archival Policy: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Cross-System Reconciliation
Use this template to create or review a cross-system reconciliation deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Cross-System Reconciliation. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on cross-system reconciliation; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for cross-system reconciliation; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for cross-system reconciliation, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Cross-System Reconciliation: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Warehouse Migration Playbook
Use this template to create or review a warehouse migration playbook deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Warehouse Migration Playbook. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on warehouse migration playbook; Consider storage formats, compute separation, governance and migration safety. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for warehouse migration playbook; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for warehouse migration playbook, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Warehouse Migration Playbook: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 08: Cloud & Streaming Pipelines
Ten practical templates for cloud & streaming pipelines.
Cloud Ingestion Architecture
Use this template to create or review a cloud ingestion architecture deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Cloud Ingestion Architecture. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on cloud ingestion architecture; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for cloud ingestion architecture; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for cloud ingestion architecture, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Cloud Ingestion Architecture: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Object Storage Landing Zone
Use this template to create or review a object storage landing zone deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Object Storage Landing Zone. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on object storage landing zone; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for object storage landing zone; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for object storage landing zone, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Object Storage Landing Zone: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Managed Scheduler Selection
Use this template to create or review a managed scheduler selection deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Managed Scheduler Selection. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on managed scheduler selection; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for managed scheduler selection; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for managed scheduler selection, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Managed Scheduler Selection: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Kafka Topic And Key Design
Use this template to create or review a Kafka topic and key design deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Kafka Topic And Key Design. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on Kafka topic and key design; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for Kafka topic and key design; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for Kafka topic and key design, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Kafka Topic And Key Design: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Event Schema Validation
Use this template to create or review a event schema validation deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Event Schema Validation. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on event schema validation; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for event schema validation; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for event schema validation, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Event Schema Validation: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Event-Time Watermark Strategy
Use this template to create or review a event-time watermark strategy deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Event-Time Watermark Strategy. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on event-time watermark strategy; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for event-time watermark strategy; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for event-time watermark strategy, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Event-Time Watermark Strategy: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Exactly-Once Claim Assessment
Use this template to create or review a exactly-once claim assessment deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Exactly-Once Claim Assessment. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on exactly-once claim assessment; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for exactly-once claim assessment; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for exactly-once claim assessment, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Exactly-Once Claim Assessment: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Stream Replay Procedure
Use this template to create or review a stream replay procedure deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Stream Replay Procedure. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on stream replay procedure; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for stream replay procedure; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for stream replay procedure, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Stream Replay Procedure: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Cdc Architecture
Use this template to create or review a CDC architecture deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Cdc Architecture. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on CDC architecture; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for CDC architecture; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for CDC architecture, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Cdc Architecture: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Cloud Pipeline Disaster Recovery
Use this template to create or review a cloud pipeline disaster recovery deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Cloud Pipeline Disaster Recovery. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on cloud pipeline disaster recovery; Define delivery guarantees, event time, replay, source offsets and cloud failure modes. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for cloud pipeline disaster recovery; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for cloud pipeline disaster recovery, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Cloud Pipeline Disaster Recovery: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 09: Data Quality & Governance
Ten practical templates for data quality & governance.
Data Profiling Checklist
Use this template to create or review a data profiling checklist deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Profiling Checklist. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data profiling checklist; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data profiling checklist; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data profiling checklist, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Profiling Checklist: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Completeness Rules
Use this template to create or review a completeness rules deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Completeness Rules. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on completeness rules; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for completeness rules; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for completeness rules, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Completeness Rules: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Uniqueness And Deduplication
Use this template to create or review a uniqueness and deduplication deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Uniqueness And Deduplication. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on uniqueness and deduplication; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for uniqueness and deduplication; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for uniqueness and deduplication, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Uniqueness And Deduplication: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Referential Integrity Checks
Use this template to create or review a referential integrity checks deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Referential Integrity Checks. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on referential integrity checks; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for referential integrity checks; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for referential integrity checks, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Referential Integrity Checks: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Pii Classification And Minimization
Use this template to create or review a PII classification and minimization deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Pii Classification And Minimization. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on PII classification and minimization; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for PII classification and minimization; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for PII classification and minimization, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Pii Classification And Minimization: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Schema Drift Response
Use this template to create or review a schema drift response deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Schema Drift Response. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on schema drift response; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for schema drift response; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for schema drift response, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Schema Drift Response: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Quality Scorecard
Use this template to create or review a data quality scorecard deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Quality Scorecard. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data quality scorecard; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data quality scorecard; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data quality scorecard, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Quality Scorecard: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Catalog Metadata Template
Use this template to create or review a catalog metadata template deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Catalog Metadata Template. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on catalog metadata template; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for catalog metadata template; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for catalog metadata template, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Catalog Metadata Template: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Access Control Review
Use this template to create or review a access control review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Access Control Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on access control review; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for access control review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for access control review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Access Control Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Retention And Deletion Workflow
Use this template to create or review a retention and deletion workflow deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Retention And Deletion Workflow. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on retention and deletion workflow; Define quality dimensions, ownership, auditability and least-privilege controls. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for retention and deletion workflow; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for retention and deletion workflow, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Retention And Deletion Workflow: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 10: Testing, Monitoring & Reliability
Ten practical templates for testing, monitoring & reliability.
Pipeline Test Pyramid
Use this template to create or review a pipeline test pyramid deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Pipeline Test Pyramid. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on pipeline test pyramid; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for pipeline test pyramid; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for pipeline test pyramid, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Pipeline Test Pyramid: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Integration Test Harness
Use this template to create or review a integration test harness deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Integration Test Harness. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on integration test harness; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for integration test harness; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for integration test harness, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Integration Test Harness: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Synthetic Dataset Generator
Use this template to create or review a synthetic dataset generator deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Synthetic Dataset Generator. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on synthetic dataset generator; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for synthetic dataset generator; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for synthetic dataset generator, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Synthetic Dataset Generator: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
End-To-End Reconciliation
Use this template to create or review a end-to-end reconciliation deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: End-To-End Reconciliation. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on end-to-end reconciliation; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for end-to-end reconciliation; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for end-to-end reconciliation, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. End-To-End Reconciliation: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Observability Metric Catalog
Use this template to create or review a observability metric catalog deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Observability Metric Catalog. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on observability metric catalog; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for observability metric catalog; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for observability metric catalog, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Observability Metric Catalog: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Alert Threshold Review
Use this template to create or review a alert threshold review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Alert Threshold Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on alert threshold review; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for alert threshold review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for alert threshold review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Alert Threshold Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Incident Triage Runbook
Use this template to create or review a incident triage runbook deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Incident Triage Runbook. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on incident triage runbook; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for incident triage runbook; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for incident triage runbook, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Incident Triage Runbook: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Root Cause Analysis Template
Use this template to create or review a root cause analysis template deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Root Cause Analysis Template. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on root cause analysis template; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for root cause analysis template; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for root cause analysis template, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Root Cause Analysis Template: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Sla/Slo Proposal
Use this template to create or review a SLA/SLO proposal deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Sla/Slo Proposal. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on SLA/SLO proposal; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for SLA/SLO proposal; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for SLA/SLO proposal, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Sla/Slo Proposal: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Production Release Checklist
Use this template to create or review a production release checklist deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Production Release Checklist. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on production release checklist; Distinguish expected behavior from observed results; include telemetry and incident recovery. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for production release checklist; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for production release checklist, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Production Release Checklist: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 11: Performance, Cost & Security
Ten practical templates for performance, cost & security.
Workload Cost Baseline
Use this template to create or review a workload cost baseline deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Workload Cost Baseline. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on workload cost baseline; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for workload cost baseline; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for workload cost baseline, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Workload Cost Baseline: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Query Cost Reduction Plan
Use this template to create or review a query cost reduction plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Query Cost Reduction Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on query cost reduction plan; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for query cost reduction plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for query cost reduction plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Query Cost Reduction Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Storage Lifecycle Optimization
Use this template to create or review a storage lifecycle optimization deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Storage Lifecycle Optimization. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on storage lifecycle optimization; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for storage lifecycle optimization; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for storage lifecycle optimization, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Storage Lifecycle Optimization: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Compute Right-Sizing
Use this template to create or review a compute right-sizing deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Compute Right-Sizing. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on compute right-sizing; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for compute right-sizing; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for compute right-sizing, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Compute Right-Sizing: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Secrets Management Checklist
Use this template to create or review a secrets management checklist deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Secrets Management Checklist. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on secrets management checklist; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for secrets management checklist; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for secrets management checklist, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Secrets Management Checklist: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Least Privilege Access Review
Use this template to create or review a least privilege access review deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Least Privilege Access Review. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on least privilege access review; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for least privilege access review; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for least privilege access review, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Least Privilege Access Review: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Encryption And Key Inventory
Use this template to create or review a encryption and key inventory deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Encryption And Key Inventory. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on encryption and key inventory; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for encryption and key inventory; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for encryption and key inventory, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Encryption And Key Inventory: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Supply Chain Dependency Audit
Use this template to create or review a supply chain dependency audit deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Supply Chain Dependency Audit. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on supply chain dependency audit; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for supply chain dependency audit; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for supply chain dependency audit, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Supply Chain Dependency Audit: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Capacity Forecast
Use this template to create or review a capacity forecast deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Capacity Forecast. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on capacity forecast; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for capacity forecast; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for capacity forecast, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Capacity Forecast: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Technical Debt Prioritization
Use this template to create or review a technical debt prioritization deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Technical Debt Prioritization. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on technical debt prioritization; Estimate costs with stated assumptions and evaluate security controls and trade-offs. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for technical debt prioritization; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for technical debt prioritization, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Technical Debt Prioritization: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Module 12: Stakeholder Delivery & Remote Career
Ten practical templates for stakeholder delivery & remote career.
Stakeholder Status Update
Use this template to create or review a stakeholder status update deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Stakeholder Status Update. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on stakeholder status update; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for stakeholder status update; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for stakeholder status update, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Stakeholder Status Update: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Data Product Acceptance Criteria
Use this template to create or review a data product acceptance criteria deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Data Product Acceptance Criteria. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on data product acceptance criteria; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for data product acceptance criteria; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for data product acceptance criteria, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Data Product Acceptance Criteria: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Architecture Decision Record
Use this template to create or review a architecture decision record deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Architecture Decision Record. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on architecture decision record; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for architecture decision record; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for architecture decision record, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Architecture Decision Record: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Handover And Sop Pack
Use this template to create or review a handover and SOP pack deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Handover And Sop Pack. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on handover and SOP pack; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for handover and SOP pack; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for handover and SOP pack, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Handover And Sop Pack: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Technical Demo Outline
Use this template to create or review a technical demo outline deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Technical Demo Outline. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on technical demo outline; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for technical demo outline; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for technical demo outline, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Technical Demo Outline: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Portfolio Project Brief
Use this template to create or review a portfolio project brief deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Portfolio Project Brief. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on portfolio project brief; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for portfolio project brief; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for portfolio project brief, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Portfolio Project Brief: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Github Readme Draft
Use this template to create or review a GitHub README draft deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Github Readme Draft. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on GitHub README draft; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for GitHub README draft; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for GitHub README draft, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Github Readme Draft: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Remote Job Skill Gap Plan
Use this template to create or review a remote job skill gap plan deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Remote Job Skill Gap Plan. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on remote job skill gap plan; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for remote job skill gap plan; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for remote job skill gap plan, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Remote Job Skill Gap Plan: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Interview Scenario Rehearsal
Use this template to create or review a interview scenario rehearsal deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Interview Scenario Rehearsal. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on interview scenario rehearsal; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for interview scenario rehearsal; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for interview scenario rehearsal, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Interview Scenario Rehearsal: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Client Proposal And Scope
Use this template to create or review a client proposal and scope deliverable for a real data engineering project.
You are a senior data engineer and AI-assisted data pipeline specialist. Create a practical deliverable for: Client Proposal And Scope. PROJECT INPUTS (replace every bracket): Business objective: [BUSINESS_OBJECTIVE] Source systems and data samples: [SOURCE_SYSTEMS_AND_SAMPLE_SCHEMA] Destination and platform: [DESTINATION_AND_PLATFORM] Stack (SQL/Python/dbt/Airflow/Spark/cloud): [TECH_STACK_AND_VERSIONS] Data volume, cadence and latency target: [VOLUME_CADENCE_SLA] Data classification, access and compliance: [PRIVACY_AND_ACCESS_REQUIREMENTS] Known constraints and budget: [CONSTRAINTS_AND_BUDGET] TASK 1. Focus specifically on client proposal and scope; Create a stakeholder-ready artifact with honest scope, evidence, handoff and portfolio relevance. 2. Explain prerequisites, assumptions, unanswered questions, and the decision criteria for this scenario. Do not fabricate metrics, executed commands, sources, or existing system access. 3. Produce an executable specification: inputs, schema contracts, transformation or configuration steps, outputs, failure handling, ownership and rollback/recovery. Include illustrative SQL, Python, YAML or pseudocode only when suitable for client proposal and scope; label it as example code and adapt it to [TECH_STACK_AND_VERSIONS]. 4. Provide at least three relevant edge cases for client proposal and scope, including nulls/duplicates, schema changes, partial failure, unexpected volumes or late data as applicable. State test data, expected outcomes and monitoring signals. 5. Compare at least two sensible implementation approaches, listing operational complexity, correctness, cost and security trade-offs. Do not claim exact performance without measurements. 6. End with a verification checklist, evidence to collect, measurable acceptance criteria, risks, and a concise stakeholder summary. Keep secrets and personal data out of example output. OUTPUT FORMAT A. Goal and assumptions B. Client Proposal And Scope: recommended deliverable C. Step-by-step implementation / example artifact D. Tests and expected results E. Risks, security, recovery and cost F. Acceptance criteria and next actions. Write in clear professional English; distinguish a proposed solution from a validated production result.
Hands-on portfolio project: Reliable retail analytics pipeline
Use fictional sales, customers and order-line CSV data to document a source contract; write staging SQL and Python validation; design incremental loads; orchestrate a daily run; create dbt tests; add quality alerts; propose a warehouse mart; document an incident simulation and delivery checklist. Present a GitHub README, architecture diagram, sample input/output, reproducible tests and a clear statement that the work is a demonstration project.
Professional checklist
Confirm business grain and schema; document ownership and SLAs; do not expose secrets or real personal information; make loads idempotent; check partial failures, late data and schema drift; validate reconciliation; monitor freshness, cost and error rates; ensure rollback and human approval for production changes.
Frequently asked questions
What is an AI data pipeline prompt?
It is a structured instruction for planning or reviewing a data engineering deliverable such as ingestion, transformation, tests or operational documentation.
Do these prompts replace SQL and engineering skills?
No. They speed up drafting and review, but engineering judgment, tool knowledge, testing and security review remain necessary.
Can I use this library in remote job portfolios?
Yes. Demonstrate a reproducible project with synthetic data and clearly label proposed versus executed results.
Are these WordPress shortcodes?
No. DE-001 through DE-120 are searchable prompt identifiers for convenient reference.
