Course Information
Registration period
Year-round Recruitment
Course Level
Study Mode
Duration
5 Hour(s) 1 Minute(s)
Language
English
Taught by
Shanmukh Sattiraju
Rating
1 views
Course Overview
Complete end to end Project on OTT platform using Azure Data Factory and Azure Synapse Analytics from Scratch with Labs
Let me introduce you to Azure Data Factory + Synapse Analytics – End-to-End ETL Project
This course is designed to help you build a complete, real-world ETL solution using Azure Data Factory and Azure Synapse Analytics, exactly the way it is done in enterprise data engineering projects.
This is not a theory-only course.
You will design, build, automate, secure, and deploy a production-style ETL pipeline, starting from data ingestion all the way to reporting in Power BI.
Inside this end-to-end Azure ETL program, you will learn:
1. AZURE DATA FACTORY FOUNDATIONS
Understand ADF architecture and create pipelines, linked services, datasets, triggers, and control flows.
2. SECURE DATA INGESTION FROM ON-PREMISE & CLOUD
Use Self-Hosted Integration Runtime to ingest data from on-premise sources into Azure Data Lake.
3. INCREMENTAL INGESTION STRATEGIES
Implement real-world incremental load patterns using file names, last modified dates, and daily ingestion logic.
4. DATA TRANSFORMATION USING AZURE SYNAPSE
Transform raw data using PySpark notebooks in Azure Synapse Analytics.
5. ORCHESTRATION BETWEEN ADF & SYNAPSE
Trigger Synapse notebooks from ADF and orchestrate end-to-end workflows seamlessly.
6. DATA LOADING & SERVING LAYER
Load transformed data into Azure SQL Database and prepare it for analytics consumption.
7. REPORTING WITH POWER BI
Create Power BI reports on top of the transformed and loaded datasets.
8. ERROR HANDLING & MONITORING
Implement alerts, failure handling, and monitoring for enterprise-grade pipelines.
9. CI/CD FOR AZURE DATA FACTORY
Set up CI/CD pipelines from scratch to deploy ADF solutions across environments.
10. COMPLETE HANDS-ON PROJECT
Apply everything you learn in a single end-to-end Azure Data Engineering project.
Course Content
- 10 section(s)
- 88 lecture(s)
- Section 1 Introduction
- Section 2 Environment Setup
- Section 3 Data Ingestion
- Section 4 Transformation
- Section 5 Data Loading
- Section 6 Enchancements
- Section 7 Orchestration
- Section 8 Reporting
- Section 9 Additional Section - Configuring Continuous Integration Continuous deployment
- Section 10 Conclusion
What You’ll Learn
- BUILD AN END-TO-END AZURE ETL PIPELINE – Design and implement a complete ETL solution using Azure Data Factory and Azure Synapse Analytics., AZURE DATA FACTORY FROM SCRATCH – Create and configure Azure Data Factory, linked services, datasets, pipelines, and triggers., INCREMENTAL DATA INGESTION PATTERNS – Build incremental pipelines using last modified date, file name patterns, and daily file ingestion logic., ON-PREMISE TO CLOUD INGESTION – Ingest data securely from on-premise systems to Azure Data Lake using Self-Hosted Integration Runtime., AZURE DATA LAKE IMPLEMENTATION – Create and configure Azure Data Lake and design folder structures for real ETL projects., SECURE SECRETS WITH AZURE KEY VAULT – Use Azure Key Vault to manage secrets and integrate them with ADF linked services., ADF CONTROL FLOW ACTIVITIES – Understand and implement GetMetadata, ForEach, If Condition, and dynamic pipeline logic., AUTOMATED PIPELINES IN ADF – Build fully automated pipelines with triggers, error handling, and monitoring., ORCHESTRATION USING AZURE DATA FACTORY – Orchestrate multiple pipelines and dependencies in an enterprise-ready way., AZURE SYNAPSE ANALYTICS SETUP – Create and configure Azure Synapse Analytics workspace and Spark pools., DATA TRANSFORMATION WITH SYNAPSE – Transform data using PySpark in Synapse notebooks with real business logic., PYSPARK TRANSFORMATION LOGIC – Write PySpark code for cleansing, aggregations, filtering, and transformations., INCREMENTAL TRANSFORMATION LOGIC – Transform only today’s or changed data efficiently in Synapse., CALL SYNAPSE NOTEBOOKS FROM ADF – Trigger and control Synapse notebooks directly from Azure Data Factory pipelines., LOAD DATA INTO AZURE SQL DATABASE – Load curated data into Azure SQL Database as part of the ETL process., POWER BI REPORTING – Build Power BI reports on top of transformed and loaded data., ERROR HANDLING & ALERTING – Send automatic alert emails when pipelines fail and handle failure scenarios., CI/CD FOR ADF FROM SCRATCH – Configure Continuous Integration and Continuous Deployment for Azure Data Factory., PROJECT-ORIENTED LEARNING – Work on a complete, hands-on ETL project using Azure Data Engineering services.
Skills covered in this course
Reviews
-
CCarlos Linares
Very fun course. I'm using all this useful knowledge at my job
-
SSavera Mishra
thank you
-
SSomisetty Pavan Kalyan
Very clear explanation. Really appreciate your efforts. Thank you.
-
VVamsi Ponnam
Excellent training!
