Hey PH! 👋 I'm Nitish, a Data Engineer from India and I built OpenAutoLoader as a hobby project scratching my own itch as a Data Engineer tired of the Spark tax. Every time I needed a simple incremental pipeline, the answer was always "just use Spark." But Spark comes with a massive operational overhead for jobs that really don't need it. I wanted the elegance of Databricks Auto Loader schema inference, incremental checkpointing, new-file detection without spinning up a cluster just to move some CSVs. So I built it on Polars. Fast, lightweight, and a joy to work with. It's MIT licensed, a passion project, and built for anyone who just wants clean incremental pipelines without the overhead. Would love your honest feedback, what's missing, what's confusing, what would make you actually switch to it? 🙏

OpenAutoLoader

Databricks Auto Loader, but open-source and Spark-free.

Databricks Auto Loader, but open-source and Spark-free.