Fabric Notebooks, PySpark & Semantic Link (sempy)

Instructor: Alaa Hady · Language: English or Arabic · Suggested duration: 2–3 days · Level: Intermediate to Advanced (Level 200–300)

Objectives

• Use Fabric Notebooks, Data Wrangler, and PySpark to process, cleanse, and transform large-scale datasets.
• Master DataFrame API and Spark SQL patterns for Bronze-to-Silver-to-Gold engineering.
• Leverage Semantic Link (sempy) to bridge Python notebooks and Power BI semantic models (reading DAX measures, validating functional dependencies, and automating BPA/memory audits).
• Structure modular, reusable, and performant Spark workloads using Environments and NotebookUtils.

Outline and contents

1. Fabric Spark architecture: Starter Pools (instant startup), custom Spark Pools, Runtime versions, High Concurrency mode, and Environments
2. Interactive authoring: Notebook UI, Data Wrangler (low-code pandas/Spark code generation), and Copilot in Notebooks
3. PySpark DataFrame & Spark SQL fundamentals: reading CSV/JSON/Parquet, explicit schemas, filtering, joins, aggregations, and window functions
4. Data cleansing & validation at scale: handling corrupt records, deduplication, null imputation, and schema drift
5. Writing to Delta Lake: mode("overwrite"), mode("append"), dynamic partition overwrite, and DeltaTable.forName() Upsert (MERGE) operations
6. Orchestrating notebooks: notebookutils (mssparkutils) for file system operations, secrets (Azure Key Vault), and runMultiple DAG execution
7. Semantic Link (sempy) & Semantic Link Labs:
   • Querying Power BI semantic models and DAX measures directly into Spark/pandas DataFrames
   • Validating data quality against semantic relationships and functional dependencies
   • Automating VertiPaq Analyzer, Best Practice Analyzer, and Direct Lake framing via Python
8. Spark performance tuning: partitioning strategies, broadcast joins, avoiding Python UDF bottlenecks, and Native Execution Engine (Photon-style vectorized engine)
9. Hands-on Capstone Lab: Building an end-to-end PySpark Medallion pipeline + automated Semantic Model health audit notebook

Target audience

Data engineers, technical analysts, analytics engineers, and data scientists with basic SQL or Python familiarity.

Expected outcomes

Participants can engineer fast, code-first PySpark data transformations in Fabric and use sempy to automate advanced Power BI administration and validation workflows.