Why computing DORA metrics is hard 🤯

Thomas Gerber
5 replies
The DORA metrics 📊 are a set of metrics that measure the quality and velocity of software delivery of an engineering organization. By measuring and continuously iterating on these metrics, engineering teams can deliver better software to their customers faster, and achieve significantly better business outcomes 📈. So why aren’t those metrics widely available in your engineering ⚙️ organization? Because they are difficult to compute. Here are a few of the hurdles we faced when we were building Salesforce Einstein 🧠, and the reason we started faros.ai (https://faros.ai). Keep these (and us!) in mind when you start this journey. Let’s start with the time to restore service metric. ✅ Good news: you can probably get this metric from your incident management system. Assuming you are OK with not having a single pane of glass for your metrics. But don’t get ahead of yourself! Now let’s look at change failure rate. This is a metric that combines your incident and your deployment data. This means you need to extract and load 💾 data from both sources somewhere to compute the metric. You cannot just get it from one of your systems. That’s just the first difficulty. Next you need to correlate your incidents with your deployments. Then, let’s take a look at weekly deployments. It is common for Engineering organizations to have several CI/CD systems. The data from the different systems will need to be modeled and normalized. More fun! Lead time for changes ups the ante: not only do you need data from multiple systems (commits, pull requests, build, artifacts, deployments), that data needs to link properly. Your system will need to be robust enough to gracefully deal with missing data and out-of-order ingestion. Most likely you will also need to impute changesets for your deployments. Finally, if you overcome those difficulties, you will want to roll up those metrics along organization lines, slice and dice by application, by stage, and over time. All of a sudden, you have to maintain a service catalog, deal with hierarchical roll ups and drill downs, … 😓 📣 TLDR: DORA metrics are difficult to compute; don’t build them from scratch. Use tools and products that deal with these difficulties to get to value faster. 🎯 Check out Faros Community Edition: https://github.com/faros-ai/faro...


Vitaly Gordon
Thank you for the post, Thomas! Which source systems do you support out of the box? Can I tweak the DORA metrics the system ships with?
Thomas Gerber
@v10n For source systems: GitHub, GitLab, Bitbucket. Full list here: https://community.faros.ai/docs/... All metrics can be inspected and tweaked (thank you Metabase!): https://community.faros.ai/docs/...
Nicole Ogloza
Great article, thank you!
Demand Guru
Great post @thomas_gerber1 - It is truly amazing how 4 metrics: deployment frequency (DF), lead time for changes (LT), mean time to recovery (MTTR), and change failure rate (CFR) can tell the difference between low performance teams and elite engineering teams!
Geoffrey Graveaud
That depend of the context of an organisation, a department or also context's team. Faros is ONE solution between multiples. Start from scratch allows to discover a great response/solution for a team/department/organization #dynamic system complex