Puppeteer API

Convert your website into an API

#5 Product of the DayMay 23, 2020
Introducing our new integration of Puppeteer with scraping agent to allow users to write custom puppeteer scripts in Node.js backed by Agenty cloud for performance and scaling.
See examples on sidebar or on Github - https://github.com/agenty/agenty-chrome-api
Discussion
Would you recommend this product?
No reviews yet
Hi Product Hunters! I will be live today for discussions, please ask question or share your feedback. I will try to respond all questions and plan the improvements in next version, if you have any feedback.
@vikashrathee wait, so how does this turn a website into an API?
@chrismessina Just write a puppeteer script and use that to send POST request to https://chrome.agenty.com/function API to turn the website into API. For example, here is an example code to navigate to site > extract 4 fields > return JSON const fetch = require('node-fetch'); const jsonBody = {"code" : "// Read the `url` from request, goto the page, extract products\n// capture screenshot and return the results\n\nmodule.exports = async ({ page, request }) => {\n const response = await page.goto(request.url);\n console.log(response.status());\n const result = await page.evaluate(() => {\n const data = [];\n var products = document.querySelectorAll('.product_pod');\n for (var product of products) {\n data.push({\n product_name: product.querySelector('h3').textContent,\n product_price: product.querySelector('.price_color').textContent,\n product_availability: product.querySelector('.availability').textContent,\n product_image: \"http://books.toscrape.com\" + product.querySelector('.thumbnail').getAttribute(\"src\"),\n product_link: \"http://books.toscrape.com\" + product.querySelector('h3 > a').getAttribute(\"href\")\n });\n }\n return data;\n });\n\n await page.screenshot({ path: 'books-toscrape.png', fullPage: true });\n\n return {\n data: result,\n type: 'application/json'\n };\n};" , "request" : {"url":"http://books.toscrape.com/"} }; fetch('https://chrome.agenty.com/functi...', { method: 'post', body: jsonBody, headers: { 'Content-Type': 'application/json' }, }) .then(res => res.json()) .then(json => console.log(json)); See the postman collection here - https://github.com/Agenty/agenty... or example scripts in sidebar.
@chrismessina See this GIF using Postman -
Not too sure to understand exactly how easy it works. I will try to give it a try this afternoon. Anyway, like the idea πŸ˜‰
@phenrysay It's scripting if you are a developer. But if not, no worries. We have point-and-click extension as well to setup scraping agents https://chrome.google.com/websto...