Course Information
Registration period
Year-round Recruitment
Course Level
Study Mode
Duration
22 Hour(s) 23 Minute(s)
Language
English
Taught by
Prashant Kumar Pandey, Learning Journal
Rating
1 views
Course Overview
Master Spark Structured Streaming using Python (PySpark) on Azure Databricks Cloud with a end-to-end Project
About the Course
I am creating Apache Spark and Databricks - Stream Processing in Lakehouse using the Python Language and PySpark API. This course will help you understand Real-time Stream processing using Apache Spark and Databricks Cloud and apply that knowledge to build real-time stream processing solutions. This course is example-driven and follows a working session-like approach. We will take a live coding approach and explain all the needed concepts.
Capstone Project
This course also includes an End-To-End Capstone project. The project will help you understand the real-life project design, coding, implementation, testing, and CI/CD approach.
Who should take this Course?
I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with those implementing Apache Spark at the ground level.
Spark Version used in the Course.
This Course is using the Apache Spark 3.5. I have tested all the source code and examples used in this Course on Azure Databricks Cloud using Databricks Runtime 14.1.
Course Content
- 9 section(s)
- 108 lecture(s)
- Section 1 Before you start
- Section 2 Setup your environment
- Section 3 Getting Started with Spark Streaming
- Section 4 Kafka for Data Engineers
- Section 5 Streaming Aggregates and State Management
- Section 6 Working with Databricks Platform
- Section 7 Capstone Project - Implementing Real-time Project in Lakehouse
- Section 8 Final Word
- Section 9 Archived - Old Course Content
What You’ll Learn
- Real-time Stream Processing Concepts
- Spark Structured Streaming APIs and Architecture
- Working with Streaming Sources and Sinks
- Kafka for Data Engineers
- Working With Kafka Source and Integrating Spark with Kafka
- State-less and State-full Streaming Transformations
- Windowing Aggregates using Spark Stream
- Watermarking and State Cleanup
- Streaming Joins and Aggregation
- Handling Memory Problems with Streaming Joins
- Working with Azure Databricks
- Capstone Project - Streaming application in Lakehouse
Reviews
-
PPrajwal C
It helped me exceptionally in understanding the spark streaming. Thank you Sir for this great course.
-
SSri Pal
I really enjoyed all your courses and you are one of the best instructors on Udemy. I registered for one course, realized the depth of your knowledge and exteperise, and your way of structuring the lectures to break complex topics - and became a fan. I registered for all your courses! thanks Mr Pandey!
-
DDeepu M
The course covers all the essential topics and provides a good foundation for beginners. However, the project explanations could be more detailed to help learners better connect the concepts. Also, the codebase could use an update to align with the latest Databricks Free Tier offerings. Overall, it’s a valuable course with room for improvement in clarity and relevance.
-
RRam Vriksh
awesome course for learning the spark streaming and how kafka work,,
