Data Scraping Studio

@vikashrathee wait, so how does this turn a website into an API?

@chrismessina Just write a puppeteer script and use that to send POST request to https://chrome.agenty.com/function API to turn the website into API. For example, here is an example code to navigate to site > extract 4 fields > return JSON const fetch = require('node-fetch'); const jsonBody = {"code" : "// Read the `url` from request, goto the page, extract products\n// capture screenshot and return the results\n\nmodule.exports = async ({ page, request }) => {\n const response = await page.goto(request.url);\n console.log(response.status());\n const result = await page.evaluate(() => {\n const data = [];\n var products = document.querySelectorAll('.product_pod');\n for (var product of products) {\n data.push({\n product_name: product.querySelector('h3').textContent,\n product_price: product.querySelector('.price_color').textContent,\n product_availability: product.querySelector('.availability').textContent,\n product_image: \"http://books.toscrape.com\" + product.querySelector('.thumbnail').getAttribute(\"src\"),\n product_link: \"http://books.toscrape.com\" + product.querySelector('h3 > a').getAttribute(\"href\")\n });\n }\n return data;\n });\n\n await page.screenshot({ path: 'books-toscrape.png', fullPage: true });\n\n return {\n data: result,\n type: 'application/json'\n };\n};" , "request" : {"url":"http://books.toscrape.com/"} }; fetch('https://chrome.agenty.com/functi...', { method: 'post', body: jsonBody, headers: { 'Content-Type': 'application/json' }, }) .then(res => res.json()) .then(json => console.log(json)); See the postman collection here - https://github.com/Agenty/agenty... or example scripts in sidebar.

@chrismessina See this GIF using Postman -

@phenrysay It's scripting if you are a developer. But if not, no worries. We have point-and-click extension as well to setup scraping agents https://chrome.google.com/websto...