Dataform

Hi PH! I'm one of the makers of Dataform, and we're excited to show you what we've built. With Dataform you can: 🛠 Build complex SQL workflows to transform raw data to prepared datasets 💻 Ensure data quality with tests and follow software engineering best practices for data management 🔔 Receive notifications and alerts if things go wrong 🔎 Document your data and discover datasets in a data catalog Dataform is really two products in one: First, it's an open source SDK that sits on top SQL that make it easy to define new data tables, re-use code across scripts, test data quality and document your data. On top of that SDK, we also built an online platform to collaborate as team, develop quickly and not have to maintain infrastructure. In 5 min you can write a query, test it, add it to a schedule, push the changes to GitHub (or submit a PR) and get alerted if one of your jobs fail. We would love to hear your feedback and answer any questions you might have! Thanks!

Dataform has changed the way that we process and transform data stored in BigQuery and Redshift warehouses - can't recommend this enough for data teams building out complex SQL-based workflows. I like to think of Dataform as an "IDE for the data warehouse", where teams can leverage best practices from software development (CI/CD, Q/A checks and assertions), ensuring that downstream stakeholders are always consuming data of high quality.

I couldn't recommend Dataform more highly to any data team. Dataform has enabled us to fundamentally change the way we work, giving us the ability to easily schedule and perform quality checks on our data. We can then be sure that our data consumers throughout the business are using the single source of truth we have defined. It is an indispensable part of our stack, sitting between our data warehouse and our self service data tool (we use Looker).

Dataform has enabled our team of 4 product analysts to create, schedule, maintain and continuously improve a DAG with 322 nodes (and counting!). The maintenance of the whole warehouse has been dramatically reduced, and now takes up less than 5% of the teams time. Couldn't recommend it enough, difficult to imagine working without it now!

Dataform has become a crucial part of our BI/analytics stack. Effective analytics teams require a robust modeling layer for consistency, repeatability and efficiency, and Dataform is the best tool to build it. Analysts can easily create and test new models, with the safeguards of version control, branches, testing etc. Plus, centralizing our analysis in Dataform makes it easy to preserve and share the assumptions/queries behind key metrics. Overall, a super useful, high-ROI product.