UPDATE: Roll20 now uses QuantumRoll to generate "truly random" dice rolls. Read more on the wiki: https://wiki.roll20.net/QuantumRollSince day 1 we've had threads come up about the randomness of the dice. We finally decided to put the RNG to the test. With some nodejs magic I hooked up the RNG that Roll20 uses to feed data into dieharder, a suite of tests to measure the strength of a RNG algorithm. Over a long weekend I left the Roll20 RNG and a benchmark RNG (AES_OFB) run through the complete suite of tests. After generating over 64.7 TRILLION random numbers from the two algorithms the results are in!But wait ... you don't think you can simply look at a big statistical report without any prior knowledge do you? Of course not! you need to read P-Values and the Null Hypothesis before delving into the arcane mysteries of the dieharder reports.Ok, that was an invigorating read wasn't it?! Now on to the results:
AES_OFB
- The benchmark / gold-standard RNG built into dieharder
Roll20 RNG
- The cryptographically secure pure JS RNG used in Roll20
https://gist.github.com/edalquist/4757300
WAIT!!! That report says the Roll20 RNG is WEAK!!!!! It must be generating terrible numbers and breaking all my games! (you didn't read that page turner about P-Values and the Null Hypothesis did you). Ok, fine the TL;DR; of that big wall of text is this:
We can't prove that a RNG is good
We can prove than a RNG is bad
There is a catch, since a good RNG is RANDOM sometimes it might appear bad but actually be good
So we run each test with a lot of data a lot of times and measure the statistical distribution of the result using some math that is over my head and end up with the P-Value
P-Values very close either bound (1 or 0) are suspect and flagged as WEAK in the test suite
So yes Roll20's RNG has a few WEAK tests, AES_OFB does as well and after running through the same set of tests multiple times for both RNGs we get 1-3 WEAK tests for each on each run but the tests that are flagged as weak vary. Using the result of this test suite we are even more confident in the Roll20 RNG than we were to start with, all of the analysis we have done shows the RNG to generate evenly distributed numbers and the dieharder tests reinforce these results. I personally GM games on Roll20 regularly and while we have seen streaks of numbers or things that seem crazy we really have to remember that we saw that stuff with physical dice too. For any of you that are still lucky enough to play in person I'd like to request that you write down all of your rolls for the night. Every time any die is rolled all night write down the result on one long piece of paper and see if any streaks/patterns show up :)