ScrapingDuck

Update: Simplified Endpoints for RAG & LLM Pipelines 🦆

We’ve streamlined ScrapingDuck to solve the biggest friction point in data ingestion: maintaining headless infrastructure just to get clean text.

We have introduced three focused endpoints to adhere to the KISS principle:

• /source: Direct raw HTML (for standard scraping).

• /result: Full metadata + content (for auditing).

• /article: (New) Extracts main content and strips the noise (nav, ads, footers) -> Perfect for your LLM pipeline.

The /article endpoint is specifically designed to reduce token usage and noise for RAG applications. You no longer need to parse the DOM manually; we handle the JS rendering and extraction server-side.

Example Usage (Python):

import requests

# concise configuration
API_KEY = "YOUR_API_KEY"
TARGET_URL = "https://example.com/blog-post"

# Use the 'article' endpoint to get cleaner data for LLMs
response = requests.get(
    "https://api.scrapingduck.com/v1/...",
    params={
        "api_key": API_KEY,
        "url": TARGET_URL
    }
)

# Robust error handling is recommended for production
if response.status_code == 200:
    data = response.json()
    # 'content' contains the parsed article text, ready for embedding
    print(f"Extracted Article: {data.get('content')[:100]}...")
else:
    print(f"Error scraping: {response.text}")

Check out the full docs at scrapingduck.com/docs/ or get started for free at app.scrapingduck.com. Feedback is welcome!