Cloud World Model
Simulate AWS, GCP & DigitalOcean without paying the bill
Start new thread

Why we want to build a company-owned cloud benchmarks

by
•

If companies are going to use Cloud World Model to test infrastructure decisions before spending real money, they should know how closely its simulations match the real cloud.

Documentation from cloud providers is useful, but it cannot replace testing that we run, measure, and publish ourselves.

For our latest benchmark, we deployed an application stack on AWS using an Application Load Balancer, two m5.large instances, and a db.r5.large MySQL database. We sent real traffic through it and monitored the environment with Amazon CloudWatch.

The owned run produced approximately 0% CRUD errors under our held-out traffic spike. Our overall accuracy score is now approximately 97.4%.

Over time, we want to build more company-owned benchmarks across different cloud services, architectures, workloads, regions, and providers.

The goal is not to produce one impressive number. It is to create a growing body of reproducible evidence showing where Cloud World Model is accurate, where it needs improvement, and which results still rely on documented behavior.

That is how we believe a cloud simulator earns trust.

You can read about the benchmark here:

The benchmark repository is also public:

What cloud architecture or workload would you want us to benchmark next?

38 views

Add a comment

Replies

Best

Failure scenarios would be interesting next. Seeing what happens when the DB is overloaded or an instance drops would tell you a lot.

 

Agreed. Cloud World Model can already simulate scenarios such as database overload and instance loss, but we have not yet built company-owned cloud benchmarks for those failure modes.

Comparing the simulated errors and recovery behavior with what we observe on AWS would be a valuable next step. Great suggestion.

For me, the useful part would be knowing when I can actually trust the simulation. A few clear examples of “works well here, be careful here” would go a long way.

 

Yeah, I agree with this. Right now, Cloud World Model tells you what’s directly supported, what’s inferred, what isn’t covered, and the assumptions behind the result.

But putting a label on something isn’t enough. People should be able to see: we tested this, you can trust it here. We haven’t fully tested that yet, so be careful there.

That’s a big reason I want us to keep building benchmarks we run ourselves.

Agree with all of it. The sentence I would underline is the one about results that still rely on documented behaviour. I ran one of my own checks five times in production and published run ids, credits and wall time. The paragraph people read was the limitation: two claims with the same missing evidence got different negative labels. Right outcome, inconsistent labels. Five fixtures are evidence, not an accuracy rate, and saying so made the number believable. Mine still has no baseline against a person doing it by hand.

 

That’s a really good distinction. Five repeatable runs show consistency, but without a baseline you still can’t call that an accuracy rate.

The labeling point matters too. If two claims have the same evidence gap, they should not get different labels. That just creates confusion.

That’s what we’re trying to be careful about with the 97.4% number. The number matters, but showing exactly what was measured, what was documentation-backed, and what is still missing is what makes it believable. Thanks for sharing this.