Scrape websites visually

#2 Product of the DayFebruary 22, 2016



You need to become a Contributor to join the discussion - Find out how.
Sam Doshi@samir_doshi · Co Founder @ Relayo.com
@datarade Thanks for posting - educational !
Kumar Thangudu@datarade · Technologist
@samir_doshi ;)
Albi Wiedersberg@wiedersberg
@datarade Kimono is closing down
Deandre Durr☀️@dredurr · Growth Hacker
@datarade sounds like you should make a collection of Scrapers.
Enoch Tang@enoch4gor · Cofounder of Zlement.com
@datarade Wow this is an amazing list! Thanks!
Stuti@stuticlicks · Fake Nomad. Biz Dev @Mobiefit. India
@datarade This is really helpful. Thanks for sharing!
Nate Ritter@nateritter · CEO, Perfect Space
@datarade kimonolabs.com was bought and is now defunct
Micah Carroll@childishgester · UI at Ring Video Doorbell
@datarade You're the man 😀
Joe Minock@joeminock · AthleteProgress.com / Rails & JS Dev
@datarade Really appreciate this list! Especially bummed that Kimono is gone, though.
Sahil Chaturvedi@sahilc0 · Co-founder, Ader (eSports sponsorships)
@datarade Jeez that's a lot! Are they all similar, or just different use cases?
Gianni D'Alerta@giannidalerta · Director of Marketing @Decentral.ca
@datarade love this!
Kumar Thangudu@datarade · Technologist
@giannidalerta follow me and check out my blog. My plan is to continue these type of comments. ;)
Nick Kwan@nwkwan · Head of Growth, Pakible (YC W15)
@datarade I've seen a lot of your epic scraper comments on PH, so looks like you are the scraper king haha. What are your go-to scrapers nowadays for social networks like Linkedin + Twitter? Looking for something web based / usable on Mac. Data-miner.io is good, but a bit complex for us non-technicals. Many sites like Portia, Import, etc don't work on these sites.
Kumar Thangudu@datarade · Technologist
@nwkwan Scraping king? No. Have I used and/or looked at more enterprise and consumer apps than anyone on the planet? potentially. I would reachout to jakub@apifier and talk shop with him about this. He's your best bet. Knows the most about this space.
Nick Kwan@nwkwan · Head of Growth, Pakible (YC W15)
@jakubbalada thoughts? Checking out @apifier now.. thx @datarade
Jakub Balada@jakubbalada · Co-founder, Apifier
@nwkwan @apifier @datarade You can use Apifier for scraping Linkedin, but you would need a lot of proxies for that site. Let me know if you need some help with crawler configuration...
GuessBox@guessboxapp · Official GuessBox ProductHunt Profile
@datarade also http://guessbox.io :)
Gabriel Puliatti@gpuliatti · Founder, Emptor
I've worked for Scrapinghub for the past two years, happy to answer any questions about Portia, its big brother Scrapy, or any part of our platform... or even web scraping in general! Us and our users are currently crawling 3.5 pages billion per month, or around 80,000 pages per minute. So we know a little bit about scraping. :)
Jake Miller@jpmillions · hyperLincoln
@gpuliatti I'm curious to hear your perspective to) Pala tired acquisition and sudden shutdo not of kimono labs and how they handled that?
Gabriel Puliatti@gpuliatti · Founder, Emptor
@jpmillions We're open source guys (and gals)… so we're definitely saddened to see users of a closed platform treated this way. OTOH, we've seen a lot of customers coming to try Portia from Kimono. :) We're actively working on ways to help people port their Kimono crawlers, so keen on hearing anyone in this boat! Email me directly (gabriel@scrapinghub.com) or sign up to the mailing list at the bottom of this post (https://goo.gl/CGxsFl). Both Portia and Scrapy are fully open source, and any crawlers (created or running) in our platform are fully exportable and interoperable with open technologies. While we are focused on the long-term and so doubt our platform will be shut down any time soon, if that ever happened, all of our users would be able to export their crawlers and use them on their own infrastructure. We've done this ourselves for some of our Professional Services clients who want us to build scrapers but also run things on their own infrastructure.
Evan Lodge@evanlodge · Co-Founder, HigherMe
@gpuliatti I tried scraping a list of 27,000 urls... the browser crashed. Is there any easier way of adding URLs to the scrape?
ScrapingHub crew have been doing this a long time, and deliver good service. Since the untimely demise/acquihire of KimonoLabs, I'll be giving this a try.
Elia Morling@tribaling · IdeaHunt.io
Looks cool. I am curious to learn what the top uses case for Portia are? I understand that people scrape, but what interesting things do they do with the data?
Gabriel Puliatti@gpuliatti · Founder, Emptor
@tribaling A few use-cases from our past client projects: - Scrape eCommerce sites that sell your products, to check for price violations and review data. - Build a broad crawler covering thousands of sites to automatically discover contact and profiles information for a specific industry. - Parse all shop locations for a number of big brands to provide a locator for users looking for a specific type of shop. - Build a database of interesting candidates to hire, by matching various sources of internet profiles with a series of filters which you or the HR team are interested in. I know people building boutique businesses on basic web scraping… like someone who uses our platform to offer a service that allows people to monitor Amazon Kindle Books pricing, and get alerted when the price drops or the book goes on sale. In effect, bringing Amazon's data "back to the people" to allow them to make better choices. But of course, most of the $$$ value comes from being a Fortune 500 company and being able to understand a lot more about the world, your industry and your competition. We help both large and small increase their reach and get access to the best technology. :)
Christien@clouvi · Managing Partner of SellPersonal.com
Cool and freaky logo!