Launched this week

Filament by Galaxy
Move data using your terminal or agent. Fast & open source.
449 followers
Move data using your terminal or agent. Fast & open source.
449 followers
Filament is a data replication engine written in Go. It supports full loads, incremental replication, and change data capture (CDC). As data moves through the engine, Filament verifies Arrow batches at the sink handoff and, where the sink supports it, checks the exact serialized payload. You can run Filament as a standalone binary or embed it in a Go application.






Appreciate the distinction between checking an Arrow batch at the sink handoff and checking the serialized payload where the sink supports it. Showing that boundary in the run result would help someone understand what was actually verified.
@oleksii_sekundant 🙌
Checksumming the Arrow batch and recomputing it at the sink is a good answer to the quiet failure in pipelines, where rows arrive but not quite as they left. Advancing checkpoints only after the destination commits makes replays safer too. How do you handle a schema change mid CDC stream, like a column type change on the source, without breaking verification or forcing a full reload?
The combination of CDC, incremental replication, and batch verification is interesting. How does Filament handle recovery after a destination outage, especially when the source continues receiving writes during downtime?