Viktor.comAn AI coworker that actually does the work
Radar – Open Source Media Intelligence
Open-source intelligence for news & social monitoring
26 followers
Open-source intelligence for news & social monitoring
26 followers
Open-source media intelligence & social listening — a self-hostable alternative to Talkwalker/Brandwatch, built on free data sources and Claude AI. - Scognamiglio1969/radar-intelligence
self-hosting an OSINT tool free of platform lock-in is a real gap for smaller teams. the part I'd worry about for a project like this, more than the AI summarization layer, is the scraping itself - Reddit and Twitter/X have both tightened ToS enforcement and rate limits against unofficial scrapers over the past couple years. is Radar pulling from official APIs where they exist (even rate-limited free tiers) or scraping HTML/RSS directly, and has anyone actually gotten rate-limited or blocked running this for real?
@galdayan Great question, and honestly the right thing to worry about, the collection layer is where a project like this lives or dies, not the LLM part. Here's exactly how Radar handles it, connector by connector, because the answer is different for each source.
Short version: Radar uses official APIs and official feeds wherever they exist. There is no HTML scraping of Reddit or Twitter/X, no headless browser, no undocumented private endpoints. Every connector is one small file in lib/connectors/, so you can audit each one yourself.
The tiers:
Reddit → official OAuth API (oauth.reddit.com) when you provide a free script-app client ID/secret; falls back to the public .json endpoints with a descriptive User-Agent if you don't. Both are Reddit's own documented surfaces — not scraping. This is the connector most exposed to their ToS tightening, which is exactly why it's classified in the UI as "free · requires a free key": you authenticate as yourself, under your own app registration and your own rate budget, so you're a first-class API client rather than an anonymous scraper they'd want to block.
Twitter/X → official API v2 only (api.x.com/2/tweets/search/recent), Bearer token, public_metrics expansions. It's gated behind the paid Basic tier ($200/mo) and the connector is disabled by default with that reason shown in the UI. There is deliberately no unofficial X path, post-2023 that's a fast way to get an IP/account nuked, and I wasn't going to ship it.
GDELT (api.gdeltproject.org/api/v2/doc), Hacker News (Algolia's official hn.algolia.com/api/v1), Bluesky (official AT Protocol xrpc/app.bsky.feed.searchPosts), Mastodon (per-instance /api/v1/timelines/tag), YouTube (Data API v3 with your free key) → all first-party JSON APIs, no scraping.
Google News and generic RSS → these consume the publisher-provided RSS/Atom feeds (Google News' own /rss/search output, parsed with fast-xml-parser). That's syndication designed to be machine-read, not HTML scraping of a rendered page.
Telegram → reads the public t.me/s/<channel> preview pages. This is the one connector that parses HTML, and only for channels the operator explicitly opts into.
On rate limiting / getting blocked in practice: yes, it's a real constraint and the architecture assumes it. A few concrete points:
Every request carries a 15s timeout, cache: no-store, and an honest descriptive User-Agent (no spoofing a browser). No hammering, collection runs on a scheduled ingest cycle, not a tight loop.
Connectors run through Promise.allSettled and a collect() helper that swallows individual failures — if Reddit throws an HTTP 429 or a source times out, that connector returns empty for that cycle and the rest of the run is unaffected. You get degraded coverage, not a crashed job. So a rate-limit event is a non-fatal, self-healing condition by design.
Because it's BYOK and self-hosted, the rate budget is yours, per deployment, you're not sharing a pool with thousands of other users of a SaaS, which is the usual reason people hit hard walls fast.
Full transparency on the honest limitation you're circling: the two connectors most likely to see enforcement pressure over time are Reddit's public .json fallback (mitigated by supplying your own app credentials) and Telegram's t.me/s/ HTML. Neither X nor any major network is scraped. If a provider changes terms, the connector is ~40 lines you can disable or fix in isolation, which is precisely the argument for it being open-source and self-hosted rather than a black-box SaaS that silently breaks.
I'd genuinely welcome a PR or an issue if you stress-test the Reddit path at volume, that's the one where real-world data from more operators would be most useful. Repo's AGPL, connectors are in lib/connectors/.
@scognamiglio1969 this is a genuinely thorough answer, connector-by-connector is exactly what I was hoping for instead of a "trust me" reply. the Telegram t.me/s/ HTML fallback is the one that'd worry me most long-term since that's the only unofficial surface in the list - is that opt-in per channel something the operator configures, or does it kick in automatically when a channel has no other way in?
@galdayan Great question, and you've zeroed in on exactly the right one. Yes: the Telegram t.me/s/ path is opt-in per channel, configured by the operator, and it never kicks in automatically.
Concretely: Telegram is off by default and has no keyword-discovery mode at all. In a project's Settings you paste an explicit list of public channel usernames (e.g. @durov, @some_news_channel) into the "Telegram channels to watch" field. The connector only ever fetches t.me/s/<that-exact-username> , the public web preview of channels you named.
There's no crawler that "finds a way in" when a channel has no API surface; if you don't list a channel, Radar never touches Telegram. So the only unofficial surface in the whole stack is both (a) limited to public preview pages Telegram itself serves to logged-out browsers, and (b) gated behind a deliberate per-channel opt-in by the operator.
That was a conscious design choice precisely because it's the one non-official surface: I didn't want it to be something that silently expands its footprint. If Telegram ever changes those preview pages, it degrades to "that one connector returns nothing", the Promise.allSettled architecture means the rest of the run is unaffected — and it's ~40 lines you can disable in one line.
If you end up stress-testing it, I'd genuinely love a GitHub issue or PR, the Telegram and Reddit-public paths are the two I most want real-world data on. And if the project's useful to you, a ⭐ on the repo (github.com/Scognamiglio1969/radar-intelligence) and a signal-boost would mean a lot, it's a solo, self-funded OSS effort and word of mouth is the whole distribution strategy right now.
Thanks for engaging this carefully rather than just upvoting and moving on. 🙏
@scognamiglio1969 that's a genuinely satisfying answer - not "we don't scrape" but "here's the one gray-area path, here's exactly how narrow it is, and here's the one-line kill switch." I won't be stress-testing it myself but appreciate you laying it out this precisely instead of hand-waving past the question.
self-hosted social listening sounds great, especially the Claude integration for sentiment analysis. curious how it handles the noise from free data sources though.
@meralyarba57925 Thanks! That’s exactly what I’m hoping for. Radar is still evolving, and my biggest hope is that people will actually use it, contribute to it, and help it grow into a great open-source project. Feedback on the quality of the free data sources is especially valuable.
Finally tried Radar over the weekend and the self-host setup was smoother than I expected, only hit one dependency snag. The Claude-powered sentiment summaries on Reddit threads actually felt useful, not just generic.
@berkelzfx Thanks for trying it and for the detailed feedback! I’m glad the setup went well overall and that you found the Claude summaries genuinely useful. I hope you’ll stick around, open issues or PRs if you find anything, and help Radar grow into a solid open-source project.
@berkelzfx are you a real person or a bot? Just to be sure...
How does it compare to something like Brandwatch in terms of source coverage, especially for niche or non-English markets?
@cemalfzvr BOT!
Finally gave Radar a spin on my own server and the setup was surprisingly painless. Really liked how it pulls from free sources without locking the good stuff behind a paywall.
@rfan6fvt Thanks, I really appreciate it! Making it easy to self-host and keeping the core features free was one of my main goals. I hope you’ll keep using it, share ideas, and help the project grow as an open-source community effort.
@rfan6fvt BOT!!
Finally gave Radar a spin and was surprised how clean the self-host setup was, took maybe ten minutes before I was pulling mentions. The Claude-powered sentiment summaries actually feel useful instead of generic.
@serpil896502 BOT!!