FlakyTest.dev

How Python teams fix flaky tests

77 followers

How Python teams fix flaky tests

77 followers

Visit website

Testing and QA software

The single place to manage, monitor, and report on your test suite performance.

Free Options

Launch tags:Software Engineering•Developer Tools

Launch Team

Maker

📌

Hello Product Hunt! 👋 I've been working on a product that helps teams manage flaky tests. If you or your team ever feel like you need to spend more time re-running tests on pull requests, this product was made for you! The app's main feature is the 'mute' feature. It allows you to silence a flaky test without any code changes. A muted test continues to run on your CI, collecting errors, but it won't fail your entire build until it's unmuted (which happens automatically after a week by default). This way, your team can maintain control without being overwhelmed by forgotten skipped tests. Besides the mute feature, the product has all the other bells and whistles you'd expect: statistics, graphs, breakdown per test, error logs, etc. I've also tried to make the setup as simple as possible, so all you need to do is install a package and add an environment variable in your CI. I focused on Python 🐍 for the initial launch, but I'll add support for other languages and frameworks based on feedback. This is my first product launch on Product Hunt, so I appreciate any feedback, thank you 🙏

Report

2yr ago

ReactSymbols UI Kit

for minute i thought “is it cypress flakiness detection, how they do it?”… unfortunately, it is not cypress tool :(

Report

2yr ago

Maker

@jsifalda no, flakytest does not yet support cypress, but I might add an integration in the future.

Report

2yr ago

Congrats on the launch! FlakyTest.dev seems like a smart solution to a common problem. How does the 'mute' feature ensure that flaky tests are addressed in a timely manner without being overlooked or forgotten?

Report

2yr ago

Maker

@vasily_kolosov1 Thank you for the kind comment! 💚 The mute feature has a timeout, so if you don't fix the test in that timeout it is going to start failing again. My plan is to also implement reminder emails before the timeout runs out so that the person who muted the test and/or the author of the test are aware.

Report

2yr ago

Opengeni

Does it work as apart of a GitHub actions pipeline?

Report

2yr ago

Maker

@sentry_co Hey, yes, it does It should run in any CI just as long as you have the flakytest plugin pip package installed and the FLAKYTEST_SECRET_TOKEN env variable set. I haven't gotten around to writing instructions for other CI systems yet, but if you let me know what you are using I'll make sure to prioritize writing docs for that!

Report

2yr ago

Opengeni

@anze3db I think the most common use case would be Github actions. I use that for my remote test pipeline. Amongst others

Report

2yr ago

Maker

@sentry_co I think I misread your original comment. I thought you were asking if it works anywhere else besides GitHub Actions. I do have a doc explaining how to set up GitHub Actions: https://flakytest.dev/ci/ since yes, I think it's the most common option these days!

Report

2yr ago

Brizy

Great product, looking straight!

Report

2yr ago

Maker

@dimitrie_baitanciuc Thank you!

Report

2yr ago

Retail Connect

Congrats on launching flakytest.dev, Anže! The 'mute' feature is super clever; it lets teams ignore flaky tests without forgetting about them, which sounds like a lifesaver. Can't wait to see it grow and include more languages. Great job!

Report

2yr ago

Maker

@shoppietra Thank you Pan! I wanted to validate the idea first with Python since that's the language I'm most familiar with, but if people really like the idea I'll be adding more languages soon!

Report

2yr ago

Congratulations on the launch of your product ! The 'mute' feature sounds like a game-changer for streamlining the testing process and maintaining control without unnecessary interruptions. It's great to hear about the simplicity of setup and the inclusion of essential features like statistics and error logs. As you mentioned focusing on Python initially, I'm curious about your plans for expanding support to other languages and frameworks. How do you prioritize which languages and frameworks to add next based on user feedback and demand?

Report

2yr ago

Maker

@yasec_kowalski Thank you for the kind comment 🙏 Other languages and frameworks will be based on user feedback so do please let me know what combination you'd like to see added first! I'm thinking that JavaScript/Playwright might be a good next step to add since it's fairly popular combination and browser tests tend to be flakier than unit tests in general, but the final decision will be based on user demand!

Report

2yr ago

1 2

Reviews