All activity
OpenAutoLoader is a high-performance, incremental ingestion engine built on Polars. It tracks processed files, handles schema drift with Rescue Mode, and sinks data to Delta Lake with full ACID guarantees. No Spark, no JVM, just fast Python.
OpenAutoLoaderDatabricks Auto Loader, but open-source and Spark-free.
Nitish Katkadeleft a comment
Hey PH! 👋 I'm Nitish, a Data Engineer from India and I built OpenAutoLoader as a hobby project scratching my own itch as a Data Engineer tired of the Spark tax. Every time I needed a simple incremental pipeline, the answer was always "just use Spark." But Spark comes with a massive operational overhead for jobs that really don't need it. I wanted the elegance of Databricks Auto Loader schema...
OpenAutoLoaderDatabricks Auto Loader, but open-source and Spark-free.