Courtesy of u/CriticKitten via r/hereoesofthestorm subreddit

The claim has often been made in the past that Hotslogs isn't a reliable source of information for various reasons, mostly having to do with the lower sample size from people leaving the site or various other things. So when the developers posted their in-house statistics for all of the game's healers, I thought this would be a perfect opportunity to put this claim to the test.

First, here's a link to the developer post from the AMA, so you can verify their figures.

I proceeded to create a modified version of my usual tracking sheet to compare these figures with Hotslogs's current figures, using standard error rates as a basis for tracking the margin of error. I filtered Hotslogs's results using Diamond/Master games only, though I could not replicate the Lvl 10+ filter that the devs typically use.

The results I found were....quite surprising, and since my Twitter network is somewhat limited, I thought I should share them with the community.

Here's an album which shows the results I found.

You are also welcome to view the spreadsheet I used to come up with these tables.

Regions that have green text only fall within the error rate, meaning that Hotslogs's figures are reasonably accurate for those heroes. Regions that are shaded green with white text fall within the middle 50% of the error range, meaning they are very accurate. And finally, regions that are in red text fall outside of the error range, meaning that Hotslogs is inaccurate on those particular win rates.

THE CONCLUSION: Hotslogs is surprisingly on-point with its figures. Despite the sample size, the figures on Hotslogs are reasonably accurate for almost every single healer, with the sole exception of Deckard Cain. Considering just how many differences there are between the way Hotslogs does its filtering and how the devs do theirs, as well as the fact that I couldn't do reliable level-filtering like the devs do, that's some pretty respectable results overall.

This is not to say, of course, that there isn't some room to improve. I think in particular, the level filter needs to be fixed on the Hotslogs site to allow for levels above 20, perhaps allowing users to specify a certain range of levels, so that its figures can more accurately match up with how the devs filter their own data. And while these figures were fairly accurate, this doesn't mean that we should ignore the variety of things that can potentially throw off the results, such as biases in the sampling or the greater level of sampling inaccuracy that can come with niche heroes that don't see as much use. However, I think it's safe to say that the claim "Hotslogs isn't accurate" is an unfair one. Hotslogs isn't 100% right, but this (admittedly anecdotal) instance shows that their figures are reasonable enough to get a good picture of what things look like, at least until we have a full fledged Blizzard API.