Fabric Pipelines, Copy Jobs and Analytics Orchestration

Instructor: Alaa Hady · Language: English or Arabic · Suggested duration: 2 days · Level: Intermediate to Advanced (Level 200–300)

Objectives

• Design metadata-driven, parameter-controlled orchestration pipelines in Fabric Data Factory.
• Ingest data from on-premises and cloud sources using Copy Activity, Fast Copy, and Copy Jobs (batch & incremental).
• Coordinate multi-step workflows across Dataflows, Notebooks, Stored Procedures, and Semantic Model refreshes.
• Implement production-grade error handling, retry policies, alerting, and CI/CD deployment.

Outline and contents

1. Fabric Data Factory overview: Data Pipelines, Copy Job, Apache Airflow jobs, and On-Premises/VNet Data Gateways
2. Building Metadata-Driven Pipelines: dynamic parameters, variables, expressions, Lookup, ForEach loops, If Condition, and Switch activities
3. High-throughput ingestion patterns: Copy Activity optimization, Incremental Copy Jobs, and CDC (Change Data Capture) patterns
4. Orchestrating heterogeneous workloads: chaining Dataflows Gen2, PySpark Notebooks, Warehouse T-SQL Stored Procedures, and Script activities
5. Triggering Power BI / Fabric Semantic Model refreshes via the native Semantic Model Refresh activity (eliminating custom REST API hacks)
6. Defensive pipeline engineering: activity dependency conditions (On Success, On Failure, On Completion, On Skip), retry logic, timeouts, and failure logging
7. Automated notifications: integrating Office 365 Outlook and Microsoft Teams webhook alerts on pipeline failure or SLA breach
8. Scheduling, storage event triggers, and Monitoring Hub diagnostics
9. CI/CD portability: parameterizing workspace/item IDs and managing connections across Dev, Test, and Prod
10. Hands-on Lab: Building a metadata-driven master orchestration pipeline that ingests 10 tables dynamically, runs transformations, refreshes the semantic model, and alerts Teams

Target audience

Data engineers, analytics engineers, ADF/SSIS developers migrating to Fabric, and BI platform teams.

Expected outcomes

Participants can replace fragile, hardcoded schedules with resilient, metadata-driven orchestration pipelines that manage the entire end-to-end data refresh lifecycle.