General
Share and discuss tech, products, business, startups, or product recommendations
Start new thread

Web Scraping 🔍🔥

by•
Scraping public data from the web, transforming it, and using it for a new product can become a very successful business. What kind of web scraping projects have you worked on and which tools did you use?
186 views

Add a comment

Replies

Best
(1) Scrapping job listing websites and creating your own product, mailing list etc for job hunters tools - python, selenium, Beautiful Soup
Sounds interesting!
I never finished it - but I started a Strava scraping project. I think there's a ton of suuuuper interesting data in there, although I did it for interests sake, rather than to monetise it. And yep, like says - Python, Beautiful Soup, etc.
Also pretty cool. I think collecting data for a while and then figuring out what do to with it later is also not a bad idea. The value of data in general will be rising in the future. Have you tried puppeteer?
I haven't - would you recommend?
You should check it out. The usability is pretty good, especially if you use it with Typescript. It is based on Chromium. All in all it has some quirks when controlling a headless browser engine, but I think that's not the fault of Puppeteer itself.
Funny thing, I scraped the "Top Most Upvoted Products" using Bardeen.ai (our tool). It worked really nicely. BUT I wanted to figure out which month is the best to launch, and turns out they haven't updated that page, so now I gotta scrape the all products. Let's see where this takes me.
Haha I also thought about it once. Can't you use the graphql api of producthunt? I think it is not public...
You can for sure use our public API for projects
ah nice, thanks for the hint Michael :)
I remember my rookie days at coding. I was usually doing a lot of parsing, mostly bots fetching videos from various web sources. Everything done with preg_match function in PHP 🥲
Damn that's old school :P How long ago was that?
it was 2006-2007, damn 16 years ago :(
Job websites, company databases, google serp, booking sites, etc. Mostly using google scrapy.
What do you mean by google scrapy?
sounds awesome! Just wondering if any legal risks
I'm building a no-code web scraping tool called .
Looks pretty dope! Did you create a chrome add-on?
Thanks, David! Yes, I did! :)
- search engine for fb comments. nodejs + selenium :)
Looks awesome! Does it store all the scraped data on a custom db? Or is there something happening on the fly, when doing a search?
I worked with Nodejs and puppeteer to scrape many complex sites for clients but now want to make software/tools as a side business. Any idea for me guys?
If you are so experienced with scraping stuff, maybe you could develop a no-code tool for creating custom scrapers :) Through a SaaS
We at ejobsitesoftware used to receive many queries for the jobs database. So we have built a custom job scrapper in Laravel using Goutte. Check screenshot -
That looks pretty cool James! Did you think about publishing it? (Paid or open source)
We plan to use it along with Job Board Software - and provide job database to job board owners
is build using NodeJS Puppetter and AWS lambda. I also have some side income from consulting around
QApop looks really good! Thanks for sharing :) Did you already think about applying the same to other (famous) forums?
12
Next
Last