Web Scraping 🔍🔥

(1) Scrapping job listing websites and creating your own product, mailing list etc for job hunters tools - python, selenium, Beautiful Soup

I never finished it - but I started a Strava scraping project. I think there's a ton of suuuuper interesting data in there, although I did it for interests sake, rather than to monetise it. And yep, like @berthakgokong says - Python, Beautiful Soup, etc.

QApop is build using NodeJS Puppetter and AWS lambda. I also have some side income from consulting around Phantombuster

I had a website that scraped automotive listings and looked at the year, model, mileage, options and price to determine if it was a good deal (this was before everyone was doing it) I found the whole process of scraping messy and a bit shady (listing sites really wanted to protect their data) so I eventually abandoned it. Data ownership is a very messy subject which I decided to avoid completely. Decided to build a CMS instead - no reliance on external data :) It is currently in private release and I think it offers quite a few competitive features that separate it from the competition.

I remember my rookie days at coding. I was usually doing a lot of parsing, mostly bots fetching videos from various web sources. Everything done with preg_match function in PHP 🥲

Job websites, company databases, google serp, booking sites, etc. Mostly using google scrapy.

Funny thing, I scraped the "Top Most Upvoted Products" using Bardeen.ai (our tool). It worked really nicely. BUT I wanted to figure out which month is the best to launch, and turns out they haven't updated that page, so now I gotta scrape the all products. https://www.producthunt.com/e/50... Let's see where this takes me.

https://Metaheads.xyz - search engine for fb comments. nodejs + selenium :)

Some Projects – LinkedIn, Szalesforce (AppExchange), GitHub, Amazon, Food Inspection Scores (Texas), Google, Government Data Sets, CraigsList, Library, lots of sites... Tools (that I like) – Scrapestorm, Import.io, ParseHub, OctoParse, Scrapy, RPA Tools (UIPath, Automation Anywhere, etc), Selenium, CLI (wget, curl, shell scripts)... Tools vary depending upon task - haven't found one tool that I can consistently use for everything ..

I would like to scrap LinkedIn comments from a post. How can I do this?

I worked with Nodejs and puppeteer to scrape many complex sites for clients but now want to make software/tools as a side business. Any idea for me guys?

I've used fetch() (:

We at ejobsitesoftware used to receive many queries for the jobs database. So we have built a custom job scrapper in Laravel using Goutte. Check screenshot - http://cricketu.com/web-scrap/

I scrape local web sites from various countries, I use Python as the programming language and Bsoup library which is really easy and https://scrape.do as the proxy gateway (Couldn’t get the job done without them since local web sites I scrape usually requires local residential IPs.

I've been working in web scraping for almost 10 years. The most demand we've seen is from the e-commerce industry in terms of the volume of the data scraped. The common use cases are price monitoring, competitive intelligence, reputation monitoring, etc. Another hot use case is extracting data from Linkedin. If I have to list the number of use-cases our data scraping supported - it will be more than 100 very different use cases across 20+ industries. Initially, we started with Python frameworks like scrapy and then built our own tools internally. I'm the founder of Datahut(https://datahut.co/), a data ( web scraped ) as a service provider.

I'm building a no-code web scraping tool called https://datagrab.io.