Scraping public data from the web, transforming it, and using it for a new product can become a very successful business.
What kind of web scraping projects have you worked on and which tools did you use?
122 views
Replies
Best
I had a website that scraped automotive listings and looked at the year, model, mileage, options and price to determine if it was a good deal (this was before everyone was doing it)
I found the whole process of scraping messy and a bit shady (listing sites really wanted to protect their data) so I eventually abandoned it. Data ownership is a very messy subject which I decided to avoid completely.
Decided to build a CMS instead - no reliance on external data :)
It is currently in private release and I think it offers quite a few competitive features that separate it from the competition.
Report
@stefan_morris Yes it can be messy. Especially the data ownership. But it's not illegal in general. It really depends on the use-case.
With which tech stack are you building the CMS?
Report
@david_gregorian I agree, it's not necessarily illegal but depending on the site, it can break their Terms of Use agreement, which is where it can get messy.
My CMS is a SaaS platform built with Vue/Nuxt and MongoDB. I'm still ramping up but there's a bit of information on my website (check out the docs) at https://shustudios.com
I'm currently looking for a few beta testers.
Report
@stefan_morris Is your CMS completely headless? For example like Contentful?
Report
@david_gregorian Yes, it is! It uses a REST API, but you can define the endpoints yourself in the CMS, as well as what data it should return. This gives you the best of both worlds between a REST API and a GraphQL API in my opinion.
Report
I would like to scrap LinkedIn comments from a post. How can I do this?
Report
@scott_k_wilder if you are skilled with javascript, try out puppeteer. It is a package which you can use with NodeJS. There are plenty tutorials for it :)
Report
Thank you. Appreciate it. Will have to find someone to help with JS.
Report
I've used fetch()
(:
Report
@itspablo so you fetched the raw HTML, right? Did you transform it somehow afterwards to be able to traverse the DOM?
Report
I scrape local web sites from various countries, I use Python as the programming language and Bsoup library which is really easy and https://scrape.do as the proxy gateway (Couldn’t get the job done without them since local web sites I scrape usually requires local residential IPs.
Report
@metehan_cetinkaya Thanks for the hint with scrape.do! I'll definitely check it out for proxy rotations next time :)
Report
I've been working in web scraping for almost 10 years. The most demand we've seen is from the e-commerce industry in terms of the volume of the data scraped. The common use cases are price monitoring, competitive intelligence, reputation monitoring, etc. Another hot use case is extracting data from Linkedin.
If I have to list the number of use-cases our data scraping supported - it will be more than 100 very different use cases across 20+ industries. Initially, we started with Python frameworks like scrapy and then built our own tools internally.
I'm the founder of Datahut(https://datahut.co/), a data ( web scraped ) as a service provider.
Report
@tonypaul_hb Sounds interesting. Are you still using Python or did you switch to another tech tack meanwhile?
Report
Web scraping, a technique used to extract data from websites, has become an integral part of the data collection strategy for many companies. With the vast amount of information available online, companies are using web scraping to gather valuable insights, monitor competitors, and automate various processes. I was able to find ZenRows, it is a powerful tool that you can learn more info about and realize that it is an effective way to scale web scraping. It allows developers and companies to extract data from websites quickly and efficiently. With its intuitive interface and extensive features, ZenRows simplifies the web scraping process, allowing users to focus on utilizing the extracted data rather than worrying about the technical complexities of scraping.
Replies