The web scraping and automation platform

Apify (formerly Apifier) extracts data from websites, crawls lists of URLs and automates workflows on the web. Turn any website into an API in a few minutes!




You need to become a Contributor to join the discussion - Find out how.
Jan ČurnMaker@jancurn · Co-founder & CEO of Apify
@passingnotes Hey David, thanks! We're considering using a similar model as GitHub - all free accounts have to make their crawlers public and only paid accounts can have them private. Do you think people would be okay with that?
David Carpe@passingnotes · Thinker & Layabout
@jancurn I think people creating certain crawlers will be fine sharing, but may want anonymity for community sharing
David Carpe@passingnotes · Thinker & Layabout
nice! would love to see a community driven collection of custom crawlers (to simply recycle or emulate)
Kat Manalac@katmanalac · Partner, Y Combinator
Apifier is a web scraper that extracts structured data from any website using a few simple lines of JavaScript. For example, imagine you found a website selling shoes and want to get a spreadsheet with all the shoe sizes, colors, prices etc. You could create such a spreadsheet manually using copy and paste, but that would take you a lot of time and frustration. Or you could setup Apifier to do this for you in a few seconds. Apifier is a startup to launch from the YC Fellowship.
Jan ČurnMaker@jancurn · Co-founder & CEO of Apify
Hi Hunters, we’re Jan and Jakub, the makers of Apifier. About a year ago we looked for a web scrapper for one of our consulting projects but none of the existing ones actually worked for websites that we needed. So we decided to build a new one. Unlike the point-and-click web scrapers, Apifier doesn't run into troubles with complicated or dynamic websites and you don't need to work around a new user interface when you start using it - you define your scraper using the same JavaScript code that you already use for your front-end web development. We’re looking forward to hear what you think! We’ll be around to answer any questions.
Greg Gerber@gerbz · Making my dent
@jancurn hey guys! I love scrapers =} but the bigger issues of scraping "the websites we need" are around ips/proxies what are you doing on that front?
Jan ČurnMaker@jancurn · Co-founder & CEO of Apify
@gerbz, yeah absolutely, we're actually rotating a number of proxies. We can also arrange for people to use their own list of proxies if they want.