Marvin.Z

The idea to create this product came from my full-time work. Our company had a web crawling system to scrape news articles from hundreds of websites. However, it is written in .Net Core and very hard to use, given we had a mix of Selenium, general Python scripts, RPAs. We had to deal with compatibility issues with each type of spiders. That's a huge problem. So I came up with this idea to...

Hi everyone, I am the author of Crawlab, an open source distributed web crawler admin platform. The coolest part is that it can run spider programs written in any languages, regardless of Python, Node.js, Java, etc. It also supports node monitoring, task management, cron jobs, code editing, Git integration and more awesome features, not to mention its beautiful frontend UI. Our goal is to make...