Udemy

Master Data Engineering using GCP Data Analytics

立即報名
  • 8,277 名學生
  • 更新於 11/2023
  • 可獲發證書
4.5
(839 個評分)
  • 提示:
  • 透過本站連結購買Udemy課程,本站將獲得推廣佣金,有助未來提供更多實用進修課程資訊給讀者。

課程資料

報名日期*
全年招生
課程級別
學習模式
修業期
19 小時 36 分鐘
教學語言
英語
授課導師
Durga Viswanatha Raju Gadiraju, Pratik Kumar, Madhuri Gadiraju, Phani Bhushan Bozzam
證書
  • 可獲發
  • (證書的發放與分配,依課程提供者的政策及安排而定。)
評分
4.5
(839 個評分)
課程連結
*提示: 以上時間表僅供參考,以機構網站為準。

課程簡介

Master Data Engineering using GCP Data Analytics

Learn GCS for Data Lake, BigQuery for Data Warehouse, GCP Dataproc and Databricks for Big Data Pipelines

Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using GCP Data Analytics Stack. It includes services such as Google Cloud Storage, Google BigQuery, GCP Dataproc, Databricks on GCP, and many more.

  • As part of this course, first you will go ahead and setup environment to learn using VS Code on Windows and Mac.

  • Once the environment is ready, you need to sign up for Google Cloud Account. We will provide all the instructions to sign up for Google Cloud Account including reviewing billing as well as getting USD 300 Credit.

  • We typically use Cloud Object Storage as Data Lake. As part of this course, you will learn how to use Google Cloud Storage as Data Lake along with how to manage the files in Google Cloud Storage both by using commands as well as Python. It also covers, integration of Pandas with files in Google Cloud Storage.

  • GCP provides RDBMS as service via Cloud SQL. You will learn how to setup Postgresql Database Server using Cloud SQL. Once the Database Server is setup, you will also take care of setting up required application database and user. You will also understand how to develop Python based applications by integrating with GCP Secretmanager to retrieve the credentials.

  • One of the key usage of Data is nothing but building reports and dashboards. Typically reports and dashboards are built using reporting tools pointing to Data Warehouse. As part of Google Data Analytics Services, BigQuery can be used as Data Warehouse. You will learn the features of BigQuery as a Data Warehouse along with key integrations using Python and Pandas.

  • At times, we need to process heavy volumes of data which also known as Big Data Processing. GCP Dataproc is a fully manage Big Data Service with Hadoop, Spark, Kafka, etc. You will not only learn how to setup the GCP Dataproc cluster, but also you will learn how to use single node Dataproc cluster for the development. You will setup development environment using VS Code with remote connection to the Dataproc Cluster.

  • Once you understand how to get started with Big Data Processing using Dataproc, you will take care of building end to end ELT Data Pipelines using Dataproc Workflow Templates. You will learn all key commands to submit Dataproc Jobs as well as Workflows. You will end up building ELT Pipelines using Spark SQL.

  • While Dataproc is GCP Native Big Data Service, Databricks is another prominent Big Data Service available in GCP. You will also understand how to get started with Databricks on GCP.

  • Once you go through the details about how to get started with Databricks on GCP, you will take care of building end to end ELT Datapipelins using Databricks Jobs and Workflows.

  • Towards the end of the course you should be fairly comfortable with BigQuery for Data Warehouse and GCP Dataproc for Data Processing, you will learn how to integrate these two key services by building end to end ELT Data Pipeline using Dataproc Workflow. You will also understand how to include Pyspark based application with Spark BigQuery connector as part of the Pipeline.

  • In the process of building Data Pipelines, you will also revise application development life cycle of Spark, troubleshooting issues related to the spark using relevant web interfaces such as YARN Timeline Server, Spark UI, etc.

課程章節

  • 13 個章節
  • 283 堂課
  • 第 1 章 Introduction to Data Engineering using GCP Data Analytics
  • 第 2 章 Setup Environment for Data Engineering using GCP Data Analytics
  • 第 3 章 Getting Started with GCP for Data Engineering using GCP Data Analytics
  • 第 4 章 Setting up Data Lake using Google Cloud Storage
  • 第 5 章 Setup Postgres Database using Cloud SQL
  • 第 6 章 Build Data Warehouse using Google Big Query
  • 第 7 章 Big Data Processing using Google Dataproc
  • 第 8 章 ELT Data Pipelines using Dataproc on GCP
  • 第 9 章 Big Data Processing using Databricks on GCP
  • 第 10 章 ELT Data Pipelines using Databricks on GCP
  • 第 11 章 Integration of Spark on Google Dataproc and BigQuery
  • 第 12 章 Data Pipeline Orchestration using Google Cloud Composer
  • 第 13 章 Data Pipelines using DBT, Airflow and Google BigQuery

課程內容

  • Data Engineering leveraging Services under GCP Data Analytics
  • Setup Development Environment using Visual Studio Code on Windows
  • Building Data Lake using GCS
  • Process Data in the Data Lake using Python and Pandas
  • Build Data Warehouse using Google BigQuery
  • Loading Data into Google BigQuery tables using Python and Pandas
  • Setup Development Environment using Visual Studio Code on Google Dataproc with Remote Connection
  • Big Data Processing or Data Engineering using Google Dataproc
  • Run Spark SQL based applications as Dataproc Jobs using Commands
  • Build Spark SQL based ELT Data Pipelines using Google Dataproc Workflow Templates
  • Run or Instantiate ELT Data Pipelines or Dataproc Workflow Template using gcloud dataproc commands
  • Big Data Processing or Data Engineering using Databricks on GCP
  • Integration of GCS and Databricks on GCP
  • Build and Run Spark based ELT Data Pipelines using Databricks Workflows on GCP
  • Integration of Spark on Dataproc with Google BigQuery
  • Build and Run Spark based ELT Pipeline using Google Dataproc Workflow Template with BigQuery Integration


評價

  • P
    Pritam Khatke
    5.0

    good

  • R
    Ramanagoud Parakanalli
    5.0

    The explaination is excellent and very easily understandable.

  • I
    Ishan Sharma
    5.0

    Good and informative

  • C
    Chayne Azriel
    2.5

    good explanation of the different tools, not very good order and structure

立即關注瀏覽更多

本網站使用Cookies來改善您的瀏覽體驗,請確定您同意及接受我們的私隱政策與使用條款才繼續瀏覽。

我已閱讀及同意