Web Scraping 🔍🔥

David Gregorian
43 replies
Scraping public data from the web, transforming it, and using it for a new product can become a very successful business. What kind of web scraping projects have you worked on and which tools did you use?

Replies

Bertha Kgokong
Software Developer | Entrepreneur
(1) Scrapping job listing websites and creating your own product, mailing list etc for job hunters tools - python, selenium, Beautiful Soup
Share
David Gregorian
Co-Founder of Aplano
@berthakgokong Sounds interesting!
Nik Hazell
Product Led Growth, SaaS, Environment
I never finished it - but I started a Strava scraping project. I think there's a ton of suuuuper interesting data in there, although I did it for interests sake, rather than to monetise it. And yep, like @berthakgokong says - Python, Beautiful Soup, etc.
Share
David Gregorian
Co-Founder of Aplano
@berthakgokong @nik_hazell Also pretty cool. I think collecting data for a while and then figuring out what do to with it later is also not a bad idea. The value of data in general will be rising in the future. Have you tried puppeteer?
Share
Nik Hazell
Product Led Growth, SaaS, Environment
@berthakgokong @david_gregorian I haven't - would you recommend?
David Gregorian
Co-Founder of Aplano
@berthakgokong @nik_hazell You should check it out. The usability is pretty good, especially if you use it with Typescript. It is based on Chromium. All in all it has some quirks when controlling a headless browser engine, but I think that's not the fault of Puppeteer itself.
Share
Fabian Maume
Founder of Tetriz.io
QApop is build using NodeJS Puppetter and AWS lambda. I also have some side income from consulting around Phantombuster
David Gregorian
Co-Founder of Aplano
@fabian_maume QApop looks really good! Thanks for sharing :) Did you already think about applying the same to other (famous) forums?
Stefan Morris
I fight for the users
I had a website that scraped automotive listings and looked at the year, model, mileage, options and price to determine if it was a good deal (this was before everyone was doing it) I found the whole process of scraping messy and a bit shady (listing sites really wanted to protect their data) so I eventually abandoned it. Data ownership is a very messy subject which I decided to avoid completely. Decided to build a CMS instead - no reliance on external data :) It is currently in private release and I think it offers quite a few competitive features that separate it from the competition.
David Gregorian
Co-Founder of Aplano
@stefan_morris Yes it can be messy. Especially the data ownership. But it's not illegal in general. It really depends on the use-case. With which tech stack are you building the CMS?
Stefan Morris
I fight for the users
@david_gregorian I agree, it's not necessarily illegal but depending on the site, it can break their Terms of Use agreement, which is where it can get messy. My CMS is a SaaS platform built with Vue/Nuxt and MongoDB. I'm still ramping up but there's a bit of information on my website (check out the docs) at https://shustudios.com I'm currently looking for a few beta testers.
David Gregorian
Co-Founder of Aplano
@stefan_morris Is your CMS completely headless? For example like Contentful?
Stefan Morris
I fight for the users
@david_gregorian Yes, it is! It uses a REST API, but you can define the endpoints yourself in the CMS, as well as what data it should return. This gives you the best of both worlds between a REST API and a GraphQL API in my opinion.
Amirali Nurmagomedov
Co-founder @ AnnounceKit
I remember my rookie days at coding. I was usually doing a lot of parsing, mostly bots fetching videos from various web sources. Everything done with preg_match function in PHP 🥲
Share
David Gregorian
Co-Founder of Aplano
@amirali_nurmagomedov Damn that's old school :P How long ago was that?
Amirali Nurmagomedov
Co-founder @ AnnounceKit
@david_gregorian it was 2006-2007, damn 16 years ago :(
Share
Victor G. Björklund
Building Remote Teams @ Jawdropping.io
Job websites, company databases, google serp, booking sites, etc. Mostly using google scrapy.
David Gregorian
Co-Founder of Aplano
@victorbjorklund What do you mean by google scrapy?
Share
Renat Gabitov
Funny thing, I scraped the "Top Most Upvoted Products" using Bardeen.ai (our tool). It worked really nicely. BUT I wanted to figure out which month is the best to launch, and turns out they haven't updated that page, so now I gotta scrape the all products. https://www.producthunt.com/e/50... Let's see where this takes me.
Share
David Gregorian
Co-Founder of Aplano
@renat_gabitov Haha I also thought about it once. Can't you use the graphql api of producthunt? I think it is not public...
Michael Silber
Product Lead at Product Hunt
@renat_gabitov @david_gregorian You can for sure use our public API for projects https://api.producthunt.com/v2/docs
David Gregorian
Co-Founder of Aplano
@renat_gabitov @product_at_producthunt ah nice, thanks for the hint Michael :)
Jared Wright
Developer
https://Metaheads.xyz - search engine for fb comments. nodejs + selenium :)
Share
David Gregorian
Co-Founder of Aplano
@jawerty Looks awesome! Does it store all the scraped data on a custom db? Or is there something happening on the fly, when doing a search?
Brandon
Some Projects – LinkedIn, Szalesforce (AppExchange), GitHub, Amazon, Food Inspection Scores (Texas), Google, Government Data Sets, CraigsList, Library, lots of sites... Tools (that I like) – Scrapestorm, Import.io, ParseHub, OctoParse, Scrapy, RPA Tools (UIPath, Automation Anywhere, etc), Selenium, CLI (wget, curl, shell scripts)... Tools vary depending upon task - haven't found one tool that I can consistently use for everything ..
Share
David Gregorian
Co-Founder of Aplano
@brandon_dfw Alright, cool. Thanks for sharing! I will check out some of those tools, as I haven't heard of them yet.
Scott K Wilder
Customer Engagement, Community & Growth
I would like to scrap LinkedIn comments from a post. How can I do this?
David Gregorian
Co-Founder of Aplano
@scott_k_wilder if you are skilled with javascript, try out puppeteer. It is a package which you can use with NodeJS. There are plenty tutorials for it :)
Scott K Wilder
Customer Engagement, Community & Growth
Thank you. Appreciate it. Will have to find someone to help with JS.
Naimur Rahman
Backend Engineer
I worked with Nodejs and puppeteer to scrape many complex sites for clients but now want to make software/tools as a side business. Any idea for me guys?
Share
David Gregorian
Co-Founder of Aplano
@naimur103 If you are so experienced with scraping stuff, maybe you could develop a no-code tool for creating custom scrapers :) Through a SaaS
stretch07
average teenage web/node.js dev
I've used fetch() (:
David Gregorian
Co-Founder of Aplano
@itspablo so you fetched the raw HTML, right? Did you transform it somehow afterwards to be able to traverse the DOM?
james smith
Working in ejobsitesoftware.com
We at ejobsitesoftware used to receive many queries for the jobs database. So we have built a custom job scrapper in Laravel using Goutte. Check screenshot - http://cricketu.com/web-scrap/
Share
David Gregorian
Co-Founder of Aplano
@jobboardsoftware That looks pretty cool James! Did you think about publishing it? (Paid or open source)
james smith
Working in ejobsitesoftware.com
@david_gregorian We plan to use it along with Job Board Software - https://www.ejobsitesoftware.com and provide job database to job board owners
Metehan Çetinkaya
Golang , Python Developer @Zeo
I scrape local web sites from various countries, I use Python as the programming language and Bsoup library which is really easy and https://scrape.do as the proxy gateway (Couldn’t get the job done without them since local web sites I scrape usually requires local residential IPs.
David Gregorian
Co-Founder of Aplano
@metehan_cetinkaya Thanks for the hint with scrape.do! I'll definitely check it out for proxy rotations next time :)
Tony paul
Head of Sales and Marketing at Datahut
I've been working in web scraping for almost 10 years. The most demand we've seen is from the e-commerce industry in terms of the volume of the data scraped. The common use cases are price monitoring, competitive intelligence, reputation monitoring, etc. Another hot use case is extracting data from Linkedin. If I have to list the number of use-cases our data scraping supported - it will be more than 100 very different use cases across 20+ industries. Initially, we started with Python frameworks like scrapy and then built our own tools internally. I'm the founder of Datahut(https://datahut.co/), a data ( web scraped ) as a service provider.
David Gregorian
Co-Founder of Aplano
@tonypaul_hb Sounds interesting. Are you still using Python or did you switch to another tech tack meanwhile?
Balázsi Róbert
Founder of DataGrab.io, NomadCoder.work
I'm building a no-code web scraping tool called https://datagrab.io.
David Gregorian
Co-Founder of Aplano
@balazsi_robert Looks pretty dope! Did you create a chrome add-on?
Share
Balázsi Róbert
Founder of DataGrab.io, NomadCoder.work
@david_gregorian Thanks, David! Yes, I did! :)