We've been overwhelmed by the outpouring of positive feedback and the many great moments enabled by Bob's Buddy in just a few days.

We also understand that many users would like to know more about Bob's Buddy's accuracy, current issues and what's next for it, so we wanted to go over some of this in more detail. For more details on how Bob's Buddy works, please read our original announcement article.

How do we measure the accuracy of Bob's Buddy?

We decided early on in the development of Bob's Buddy that we wanted to have a very robust system to monitor its performance and accuracy, so we built various metrics right into Bob's Buddy from the get-go. These include hourly and daily accuracy reports, automated terminal failure monitoring, among others. Let's go over some of those things.

Terminal Failures

The worst case scenario for a prediction by Bob's Buddy is what we call terminal failures, which happens when a combat round has an outcome that is incompatible with a prediction (for example, when you lose a combat in which Bob's Buddy said you had a 0% chance to lose). These only happen in very specific circumstances, like in the presence of the few known issues we're listing below.

Terminal failures are pretty bad for the user's experience, and so we monitor them very closely. The good news is that they are already very rare. At the time of writing, Bob's Buddy current terminal failure rate is 0.005%, or 1 instance for every 20,000 turns tracked.

We've already tracked and provided combat predictions for roughly seven million turns, so we have a wealth of data at our disposal to continue to improve. This is a good starting point and we're confident this number will drop significantly as we fix some known issues. Speaking of which..

Other Factors

One of the constraints of the method we use is the amount of Monte Carlo simulations we can realistically run on a user's machine vs the real number of possible outcomes. Even somewhat "simple" combats with 6-7 minions on each side of the board already present billions of different possible outcomes, which is an unfeasible amount of simulations to run. With hundreds of thousands of users each playing dozens of turns, there are bound to be a few turns where players get so lucky or unlucky that Bob's Buddy misses it entirely.

The way we reconcile that is by striking a balance between accuracy and agility. After running Bob's Buddy over tens of thousands of turns in our database, we found that having 10,000 simulations strikes a great balance between running quickly and producing very accurate results.

On most machines that can run Hearthstone we're able to run those 10,000 simulations in an instant, but on some older machines we might reduce that number so we can show you a result in time. Rest assured though that even on the oldest machines we'll only show you a result if we're able to complete a significant number of simulations.

What's Next?

Our priority right now is to fix all the known issues and include support to the opponent's secrets. We're committed to continue to support Bob's Buddy while we evaluate more meaningful updates.

We hope this article helped shed some light on Bob's Buddy. If you have any questions or feedback, don't hesitate to reach out to us on Twitter, Facebook, Instagram, Discord or via email.

You can download the Hearthstone Deck Tracker with Bob's Buddy here.