bottest.ai - No-code automated chatbot testing

@noah_moscovici Hey Noah, cool idea. I'm currently building a chatbot product. We have some evals already setup. Is there an easy way for us to import this into your tool? Check out our launch and lmk if we're actually a good fit for what you're building I signed up :)

@shashank_sanjay Thank you! We just launched our beta program which has full access and is completely free for the next 6 months. I'm curious to learn more about your current setup. We don't have a direct to customer way to import evals yet, but depending on what type of data you have and how you are storing it there may be some easy ways we can import it for you!

@dash4u Good question! I'll answer your question in two ways, since it seems to hit on a couple of good questions: 1. How do we evaluate whether something is "correct"? We have an AI-powered evaluation engine that breaks down the conversation between you and the chatbot, and compares it to the baseline from when you originally recorded the test. Our AI evaluation engine can accurately pick out the key differences in the Tests, and any inconsistencies or deviations from the baseline. The engine can then determine whether these differences are major enough to fail the test, or pass the test if they are just minor changes with no semantic difference. 2. What if my definition of "correct" is unique to my product? You can fully customize what constitutes a "pass" in bottest.ai. This is called "Success Criteria" and can be customized on a Test level, or on an entire Suite level. You can fully define exactly what types of difference should pass and which should fail. So if you don't care about tone or intent differences and only factual information, you can specify that in your configuration! Or maybe you're expecting the chatbot to respond with a lot of variance, and you just want to make sure the general tone/intent is the same, you can fully specify that as well. There's unlimited freedom when it comes to customizing this aspect of the bottest.ai evaluation process

@michaelgreen Thank you! We really appreciate the support!

@star_boat Thank you for your thoughts! You asked a couple good questions so hopefully I can answer them: 1. What types of evaluations does your AI engine perform that traditional methods miss? Traditional testing methods fail when testing chatbots in 3 main ways: a) Language based responses are subjective and non-deterministic. Unlike traditional software testing, where you have an input with a determined expected output, responses from a chatbot require a nuanced evaluation based on semantic meaning. b) Upgrades or improvements in the AI can cause unexpected issues elsewhere. Each change to the underlying LLM or AI model may improve the quality of answers on some questions but cause quality degradation on other prompts. Developing a high-quality AI chatbot without extensive regression testing is practically impossible. c) The subject matter experts often aren't the ones maintaining the code. The people in your team who can best determine how the chatbot should perform are rarely the same engineers working on maintaining the tests. This makes it very difficult to build comprehensive automated test coverage for a chatbot. Our AI-evaluation engine solves the first point, allowing for language responses to be evaluated in a multi step process that can pick apart key differences you care about and skip the differences like rephrasing/synonyms or other noise. However, using an AI to test the quality of another AI isn't anything new, The true power of bottest.ai comes from the other 2 points, allowing automated regression tests across all questions whenever changes in your chatbot happens, and the ability for product owners to directly manage the tests and quality (no more passing prompts and expected responses to developers, who then have to judge whether a response is "good" when they aren't experts in the niche of your chatbot!) --- 2. How do you ensure different language support? Our AI-powered evaluation engine is built language agnostic. Your failure reasons/details or any information about the tests will be in English, but our engine can handle any language that the conversation may be happening in. For example, if your chatbot serves an international customer base, you can record and test in that conversation, but all of the information that you need (such as why tests failed) will stay in English. --- Hopefully that answers your questions! Thank you for your support, and let me know if I can answer any other questions :)

@noah_moscovici Thanks for your reply, that's help a lot. :)

@elke_qin Thanks for the kind words! You asked some amazing questions: 1. How does the AI-powered evaluation engine actually work? We have a multi step process in our evaluation engine, including 1) picking out any differences between the current conversation and the baseline one. 2) determining the severity of each difference based on our "Success Criteria" 3) overall determining a pass or fail based on these evaluated differences. 2, Will it adapt as the chatbot's conversation style evolves? Will users need to manually tweak anything? I hope I understand this question correctly in that you're asking if our evaluation engine will have a dynamic understanding of a chatbot's style of speaking and can evolve to better understand what a "pass" or "fail" means to you. Here's the great thing: you don't have to use our "Success Criteria" and you have full customization powers to define exactly what a "pass" and "fail" should be for your product and your situation! You can define this Success Criteria on an individual Test level, or altogether on a Suite level. This Success Criteria will be used in our evaluation engine to determine whether the test passes or fails, so you can really customize and fine tune the process to match up with exactly how a pass and fail should be for your use-case. 3. Are there any analytics or reports? Yes! This is a huge part of the bottest.ai platform, and it's something that is of upmost importance when testing chatbots (but is almost always forgotten when using in-house testing or custom build solutions due to budget/time constraints). After every single Suite Run (this is when you run all the Tests in your Suite at once), an automatic report will be generated that compares the Run against a previous one and highlights key differences/improvements/degradations so you have full visibility on how your Tests are performing. Additionally, there is a dedicated Analytics page where you can track your trending Success data, Performance data (how long responses took for your chatbot), and Usage data. Analytics and reporting is something we spent a lot of time on at bottest.ai to make sure this feature is useful for our users, and we encourage you to check it out! The whole platform is completely free to use during our beta program for the next 6 months, so I invite you to try it out and let us know your thoughts!

@christopherdavidanderson Thanks for the feedback! We appreciate it a lot!

@big_tree Thanks for the feedback and comment! We appreciate your support! Definitely reach out if you have any issues or questions, we want to help support you!

@dance17219 Thank you! We really appreciate your support and your feedback!

@arthur_millerb Thanks for the feedback! Let us know if you have any questions or need help!

@nitsovacoo Thanks for the comment and feedback! That's actually a great concern that we've thought about a lot at bottest.ai. The main issue with using the tools you mentioned (Botium and TestMyBot) is that they are heavily reliant on developers being involved with setting up, maintaining, and overseeing the testing process. Most of the times, the subject matters in the product (for example let's say it's a customer service chatbot) are the not the developers. Instead, the product experts would be the product managers or other non-technical folks who have the insights to know whether a response is acceptable. bottest.ai is focused on building a completely no-code solution where the product owners themselves can have full control over the testing process. It frees up your developer's time from having to maintain a testing framework, and it let's you have confidence in your tests, since the product expert is the one able to easily quality test the responses. Our AI evaluation engine has a mutli step process to catch those tricky conversational quirks like you mentioned, but of course since we in beta there is always room for improvement :D Hope you will try out the product, and let us know your thoughts! We want this product to be as useful to users as possible!

@danielwchen Right now we are focusing efforts on making the product as no-code friendly while adding some really powerful features (like running tests fully on our backend servers). However, providing an API is definitely something we will add in the short future. Thanks for the feedback Daniel!