Skip to main content

Scraping Complex Websites Made Easy: A Step-by-Step Guide - Part 2

 

Demystifying Web Scraping: How to Extract Data from Complex Websites


In the previous part, of this tutorial, we did some base work like setting up the spider, collecting products, finding price API, and some other useful lessons. I will advise you to check that out as well.

In part 2 of Scraping Complex Websites, we will be seeing how to extract the extra products that are loaded as we scroll down.
Some things we will do involve
  • Finding the right request from the network tab
  • copying and formatting the curl request
  • Mimicking the browser request in scrapy
  • Making a post request in scrapy
  • Merging the code
This video part has a lot of technical details that i decided to make in form of a video. feel free to check the source code on GitHub














Labels:

Comments

Post a Comment

Popular posts from this blog

I Created a Trello Clone (Karaban Style Board) - Frontend and Backend

Hey there, fellow tech enthusiasts! Have you ever had one of those moments when you're itching to learn something new and embark on a project that not only challenges your skills but also proves your knowledge? Well, that was me during a six-month internship, and that's how this journey began. What Sparked the Idea? So, what inspired me to dive headfirst into a Trello clone project? It was partly out of curiosity, partly a desire to create something substantial, and partly because I needed to build a project for my internship. And let's face it, Trello, with all its boards, lists, and cards, seemed like a complex system with a lot going on – the perfect challenge to take on. Unpacking Trello Clones for Newbies Before we dive too deep, let me clarify what a Trello clone is for those who may not be familiar with it. Trello is like a Swiss army knife for teams – it's a work management system that helps you keep track of tasks. You've got boards that represent teams or ...
Post a Comment
Read more

Introducing the GCE Results Web App: Check Your Results with Ease

Introducing the GCE Results Web App for Cameroonian Students Hey there, fellow students! Today, I'm thrilled to share a personal project of mine that I've been working on—a web app designed to make checking GCE results in Cameroon a breeze [] https://gceresults.jeangineer.com/ . As a student myself, I understand the anxiety and anticipation that comes with waiting for exam results. That's why I created this user-friendly web app to help students across the country access their GCE results quickly and conveniently. Let's dive into the features and benefits of this app and how it can transform the way you check your results. Benefits of the GCE Results Checker Website Streamlined Result Checking Made Simple: Say goodbye to long queues and scanning through large pdfs to find your results. With the GCE Results Web App, you can now check your results from the comfort of your own home or anywhere with an internet connection. No more unnecessary stress or wasted time—it's ...
Post a Comment
Read more