Launched this week

Filament by Galaxy
Move data using your terminal or agent. Fast & open source.
449 followers
Move data using your terminal or agent. Fast & open source.
449 followers
Filament is a data replication engine written in Go. It supports full loads, incremental replication, and change data capture (CDC). As data moves through the engine, Filament verifies Arrow batches at the sink handoff and, where the sink supports it, checks the exact serialized payload. You can run Filament as a standalone binary or embed it in a Go application.






?makers, if you rank #1 on @Product Hunt today, will you take on the Subway Challenge and navigate the entire NYC Subway system in the shortest time possible? @mitchbregs @leonkozlowski
@leonkozlowski @fmerian yes, without a doubt. I ran a half marathon once because I lost fantasy football... game for this challenge.
PS - a sneak peak of our new website (not released yet!) for the PH community :)
@leonkozlowski @fmerian I still gotta clean up the copy tho, lol
@leonkozlowski @mitchbregs ship it!
@mitchbregs omg! the current world record is 22 hours, 14 minutes, and 10 seconds. game on
@mitchbregs @leonkozlowski @fmerian let's go!
@mitchbregs @leonkozlowski @german_merlo1 hell yes! thanks for your continuous support
@mitchbregs @leonkozlowski @fmerian guys please do it even if you rank #2, would be great to have a stream as well!
@leonkozlowski @fmerian @kate_ramakaieva 😅
@mitchbregs @leonkozlowski @kate_ramakaieva focusing on #1 for now haha
@fmerianthis looks like a challenge worth taking on the next time I'm in New York.
@nathan_tarbert legend! there are 472 NYC subway stations according to @Grok. game on!
Coming from PHP world, for a moment I was like "wow, that's a drastic pivot for Laravel Filament" :-) Congrats on the launch folks!
If we wanted to contribute Omnigraph connector, should we add it directly to https://github.com/galaxy-io/filament/tree/main/connectors ? Or do you have a separate mechanism for community connectors?
@pronskiy haha, we have gotten a lot of "there is a product named this" 🤷
feel free to contribute directly or submit an issue and can chat through any thoughts/comments/concerns in issues! your support and contribution means the world :)
Hi all - I'm Leon, one of the co-founders of @Filament by Galaxy. We are super excited about what filament can bring to your data stack, we've seen it reduce latency by up to 15x against some other tools in production.
After working in the data space for nearly the last decade we've seen countless tools (and pricing models... ha) for moving your own enterprise data into warehouses/lakes. We wanted to bring a fresh take on how to do this in the AI era with speed and trust (filament also converted me to a gopher, ✌️ python lol).
Major shoutout to @atterpac on the brunt of the ideation, design, and implementation of filament. If you struggle with cost, scalability, or disaster recovery in your current data replication tool we would love to help you test this out!
Source code: https://github.com/galaxy-io/filament
Docs: https://filament.getgalaxy.io/pages/guides/get-started/introduction
Benchmarks: https://github.com/galaxy-io/benchmarks/tree/main/data-movement
As always we would love feedback, stars, and for you to genuinely battle test filament on your own workloads and we're here to help!
@leonkozlowski doing this with you has already been the journey of a lifetime and i can't imagine doing it with anyone else.. just getting started brother
@leonkozlowski @mitchbregs keep up the great work, looking forward to your future launches
Whoa this is cool! Congrats! 👏
One question: how does Filament handle schema changes mid-replication, like a column added or a type changed in the source? Or is the source locked-down so those changes can't happen?
@cheevahagadog thank you!! 🙏 and nope - the source isnt locked.
Filament checks the schema every run. New nullable columns can flow through, but riskier stuff like a type change makes it stop and ask instead of guessing. If the schema changes mid-run it pauses without losing its place, so you look it over and hit rerun!
I understand the concept, but I want o learn more: Who uses this? Whats the primary uses cases for this?
@alchemist_studio hey Andre! primary use case is for teams/data engineers who need to move large volumes of data from their app databases or SaaS products into a data warehouse/lake to support analytics use cases!
this idea is so sweet! How do you verify the data arrived correctly?
Congratulations on the launch!
@mufassir_kazi thank you!! to answer:
every batch gets a checksum before it goes to the sink and the sink recomputes it right before writing so we catch anything that changed in between. some sinks also checksum the exact bytes they send over the wire. Checkpoints only move forward after the destination commits - so nothing gets marked done until it actually lands!
@mitchbregs so cool! will be definitely checking this out.
@mitchbregs @mufassir_kazi lfg! view source code here: github.com/galaxy-io/filament
what happens to the route's slot when the primary fails over, does it get created with failover=true on pg17+
@bbueno no failover flag on the slot today - great point tho and something we can consider for sure
fair enough, thanks Mitchell