Crawly

Never write another web scraper

#2 Product of the WeekMarch 18, 2016
+1

Turn websites into data in seconds. Crawly spiders and extracts complete structured data from an entire website.

Discussion
Would you recommend this product?
1 Review5.0/5
@datarade god mode scraping collection.
Upvote (12)Share
@datarade This list should be a collection!
Upvote (11)Share
@datarade Great list! Have any of these had @Kimonify-like capabilities to generate api's? @skrypt
MakerHiring
@nwkwan diffbot does =)
MakerHiring
Hey ProductHunters! Crawly is a free tool I built that uses Diffbot's automatic article extraction api to turn web content into structured data. I've used it for creating a centralized database of all of our content, but you could also use it to do content audits / migrations or analyze your competitor's content. It's currently limited to 200 pages and only articles at the moment, but I plan to add support for scraping products in the future. Any other features you'd like to see added?
Hiring
@druwynings images?
@druwynings You should really update that page to clarify the "articles only" caveat. Especially when the tagline on your home page is "No rules required"
MakerHiring
@callmeed Will do! Like I mentioned, support for products, discussions, images, and videos is in the works.
Awesome!!! Looking for a replacement for Kimono (https://www.kimonolabs.com/) since they got acquired by Palantir. Need something to power my Slack menu bot ;)
I would love to be able to scrape sports stats easily!
Looks great. Congrats on the launch! I can see some nice applications for this. One suggestion - I understand that it might take a while to scrape the data, but an instant email to say it will be X minutes, or just a notification after email input would be good, to manage expectations. I used it 10 mins ago, and am on the verge of tears that I still have no email... ;-)
Cannot GET /results/56ebdf254b0bfe03003ef0d8 :-(
MakerHiring
@neilcocker Hey Neil, things should be back to normal. Servers were crumbling under the PH load :)
MakerHiring
@neilcocker I didn't want to inundate people with unnecessary emails, but I don't want people crying either...
@druwynings Tears are over. All working now. Very impressive. Good work - this will definitely be very useful.