web_harvester
Scrapers stop after one grab. This one keeps watching.
5 followers
Scrapers stop after one grab. This one keeps watching.
5 followers
Most scrapers extract once and quit. web_harvester is a no-code Chrome extension + Python CLI that keeps running on your schedule and tells you what changed. Pick a page, tick the fields (no code, it finds them), leave it on. It pings you only when something moves: price drop, back in stock, new listing, rank or review shift, via Slack, Discord, or email. Data lands in Sheets/CSV/Excel/SQL, stacked by date. Public data, your own session, robots respected, no CAPTCHA or proxy evasion.
How does it actually handle sites that change their layout or DOM structure between scrapes - does it break silently or try to adapt?
@ymet_k37866 Great question — that's the failure mode I worry about most, so let me be specific rather than hand-wave.
It does not try to auto-heal. What it does instead is bind to the most stable thing available. The recon engine proposes an extraction strategy in a fixed order: JSON-LD first, then embedded JSON (Next.js data blobs, window vars, af_data), and CSS selectors only as the last resort. That ordering is the real answer to layout churn — a visual redesign usually leaves the JSON-LD or the embedded JSON payload intact, so the recipe never notices the site got restyled. In those structured modes a field can also carry a fallback chain (price|offers.price|offers.0.price), tried in order until one resolves. Selector mode is the brittle one, and it is deliberately last.
When something does break, it does not pretend it worked: the run is flagged empty instead of ok, and the extension tells you it returned 0 rows plus the usual causes (blocked, login-gated, dynamic render, field mismatch).
Two honest gaps, since you asked directly:
A break does not reach your webhook. Alerts fire on price drops and new items, so a broken recipe currently looks like a quiet day if Slack is all you watch. A "returned 0, previously N" alert is the obvious fix.
The empty run still becomes the new baseline, so the next successful run can surface as a burst of "new items" rather than a clean recovery.
Both are going on the list — thanks for pushing on exactly the right spot.
@ymet_k37866 Following up, since I said both gaps were going on the list.
Both are fixed in the code now. They are not on the Chrome Web Store yet - they ship in the next update, so what is live today still has the old behaviour. Saying it plainly so nobody assumes the build they have already does this.
(1) A break now reaches your webhook. A run that returns 0 rows when the previous run returned N is treated as a signal in its own right, instead of the alert count looking only at price drops and new items. The webhook fires, the message and the subject line lead with "0 rows - previously N" plus the likely causes, and the toolbar badge turns red. First-ever runs and recipes that were already empty are excluded, so it does not fire on day one or repeat the same alert every day.
(2) An empty run no longer becomes the baseline. The 0-row snapshot is not stored at all, so the last good snapshot stays the comparison point and the recovery run diffs against that - one genuinely new item instead of a burst of fake ones. There is a test that walks exactly that break-then-recover path.
12 tests around those two behaviours, extension suite at 150 passing.
One caveat I should keep honest: alerts and webhooks are a paid-tier feature, so on the free tier a broken recipe shows up in the extension popup (0 rows plus the likely causes) rather than being pushed to you.
Thanks again - your question is what surfaced both of these.