Course Information
Registration period
Year-round Recruitment
Course Level
Study Mode
Duration
2 Hour(s) 38 Minute(s)
Language
English
Taught by
Stefan Hyltoft
Rating
4 views
Course Overview
Learn how to create a web crawler using various methods on popular sites like H&M, Amazon, LinkedIn, AliExpress!
Do you want to build a webcrawler in Nodejs?
In this course you will learn how to build a webcrawler using the newest JavaScript syntax with popular sites like H&M, Amazon, LinkedIn and AliExpress!
You'll learn how to find hidden API's on sites like H&M and AliExpress and see how you can even avoid building a web crawler in the first place, you can save a lot of time this way!
Then I show how to build a web crawler for Amazon the test-driven way, by building out tests for the various product page layouts there is on Amazon.
After that we'll take a look at how to automate login and scraping profiles from LinkedIn using Puppeteer, the automated Chromium browser!
Course Content
- 5 section(s)
- 38 lecture(s)
- Section 1 Intro to web crawling and web scraping
- Section 2 Getting all products from H&M and saving it to mongodb (Method 1)
- Section 3 AliExpress - getting lots of products and prices using Method 1 (hidden API)
- Section 4 Building a Amazon webcrawler in Nodejs (Method 2, HTTP Requests)
- Section 5 Puppeteer web crawling on LinkedIn
What You’ll Learn
- Differences between web crawling and web scraping in Nodejs
- The 3 main methods to use in web crawling, and when to use what method!
- How to get data from sites like H&M and AliExpress easily and fast using their hidden API's
- How to build a web crawler for server rendered sites like Amazon to crawl all their products
- How to build a Puppeteer based web crawler for a site that requires JavaScript like Linkedin
Reviews
-
BBoots Edman
Was a good introduction to webcrawling. I had a personal project that would require it and it helped me get it sorted.
-
AA. Liam Khuen
I liked the course and thought it was extremely insightful. I work on a team that aggregates data from various sites from different countries using different standards. Our tools are already extremely built out and sometimes that can make it hard for people to explain how things work. This course has given me an idea of the basics of our own tools. With the above being said - the course needs a refresh. I would not recommend this to anyone that is looking to learn beyond the basics/concepts of scraping/crawling. The sites used as examples have changed since the release of this content (not unusual). This has resulted in some examples being out of date. If you are not adept enough, you may not be able to find alternatives - and some things are just not reproducible in the way that the instructor actions them.
-
SSammy Butts
seems to be very out of date.
-
JJorge Ayuso
Very usefull and insighfull course. Thanks
