Roll20 uses cookies to improve your experience on our site. Cookies enable you to enjoy certain features, social sharing functionality, and tailor message and display ads to your interests on our site and others. They also help us understand how our site is being used. By continuing to use our site, you consent to our use of cookies. Update your cookie preferences .
×
Create a free account

Roll20 RNG Testing

Hello there!

I started a campaign with my friends a couple of weeks ago and everything is perfect, except for this one thing:

Theres too many repeated dice rolls result.

First, the three players rolling consecutively a skill check got 1 on 1d20 in sequence. Then, on another game session, one player got 18 on 1d20 in three consecutive attack rolls. When preparing games and rolling treasure tables, I got consecutives results as well. Yesterday, same thing happened to one of my players. Almost every session has related issues.

Being a developer that studied a lot random number generator to solve problems at my work job, I know that random number, especially at distributed systems is very hard to acomplish reliably. And reading the documentations and topics from previous similar questions in the forum, I understand that Roll20 does a good job at this and uses good practices. But, this is a problem and it's happening.

Anyone else is having this issue?

April 16 (5 years ago)
GiGs
Pro
Sheet Author
API Scripter

Thats just probability for you and the law of large numbers. You're looking at a handful of game nights, when there are millions of people making rolls in a similar time frame - in some game, somewhere, people are going to see strings of repeated numbers. But it doesnt say anything at all about the accuracy of roll20's RNG. 

You also now primed to pay more attention to your rolls, so whenever you see repetition you'll notice it, even if its different numbers repeating.

If you cant let it go, you need to record every dice roll result made in every session for a few sessions - not just the repetitions, every single roll - and store them in excel or something, and analyse them over time, to see if its having any notable effect.


April 16 (5 years ago)
keithcurtis
Forum Champion
Marketplace Creator
API Scripter

To add on, you will notice and confirm duplicate rolls: 2,2,2, but likely never notice a repeated: 15,6,13 sequence. At a table, I have seen 5 20s in a row rolled on physical dice, a 1:20,000 chance. But I'm sure everyone on this board who has been playing for any length of time has similar stories.

If you would like to run a statistical analysis generate a good sized sample set, At least 1000d20, and analyze the results.

Thanks Gigs!

Makes sense. But the problem is that, in that kind of problem, large number of results don't shows the problem. It hides it, actually.

But I'm ok with it and there's ways of remedy the problem during my games. I just think is important to feedback Roll20 and comunicate other users as well.

Thanks again!

Thanks keithcurtis!

Of course. I've seen some rolls like that also. I'm writing a post about it because this is happening too many times in my games. Not at a normal rate. Every game session it happens multiple times.

Maybe theres no probelm with Roll20 and the problem is with my browser or else. But I'm pretty sure it's happening. I'm familiar with problems like that. I studied this for long time and I understand that this is something VERY difficult to detect.

Anyways. Report sent!

Thanks for the reply!

April 16 (5 years ago)

Edited April 17 (5 years ago)
Andreas J.
Forum Champion
Sheet Author
Translator

No, the problem is how (badly) human perceives randomness. Keith and GiGs brought up a few points on the matter, but didn't share any links, so it's time I take out the big guns.

I wrote a long reply to a previous instance to this topic that should highlight more aspects and resources to understand what's going on in general:

No, the dice rolls are not broken

Human intuition on randomness is flawed, and here is an attempt to explain why

Roll20 has a page to show how random the rolls are. The thing about randomness that people don't often get, is that streaks" of "good or "bad" luck is to be expected. Truly random doesn't feel random.

Steve(Original poster) says:
But there is something bugged about the way the rolls are distributed over a short period, like 200 rolls

That is unlikely, but expected to happen to someone, at times, given the large number of rolls made on Roll20. You know how there exist people who have got hit by lightning more than once, or have won the lottery more than once? It's unlikely to happen to any given person, but the chance of it happening to someone, given large enough pool of samples, is extremely large.
The chance of something happening even at small scale is something we don't have an accurate intuition about, for example the Birthday paradox, which is the statistical oddity that with a random group of 23 people, it's a 50% chance at last two people share a birthday.

Math & Science behind randomness

Then there is also the fact that when the results are given instantly, it feels less random than if we see numbers flutter a bit before they appear. Slot machines and gambling games likely uses this trick of slowly showing the result to feel more random and increase the anticipation for the result.

Psychology behind randomness (and why we're bad at perceiving it)

If you watch the "Randomness is Random" video, it demonstrate quite well how we think streaks of "good" or "bad" luck are longer and more prevalent than if we'd try to make a random sequence of numbers ourselves. That's why some random shuffle features on music playlists aren't always completely random, as they might prevent longer streaks of songs from appearing in their original order, or bias the system to try not to have too many songs from the same artist/album to play back to back.

Previous Forum Threads on the topic

This come up every now and then, and is repeatedly explained every single time. Here is a fairly long comprehensive list of previous discussions on the topic:

TL;DR: Humans have a bad intuition on what is or isn't random, and this have been debunked every time it's been discussed here.

Thank, Andreas J.!

Alright, alright! Don't feel offended. I like Roll20 a lot. Hahahah!

But look, I was not saying that is broken. Just that, in specifc cases, may not work as the designed way.

Be in mind that machines can't generate "random" numbers. Algorithms just make it seems like its random based in certain variables, that if it were the same, or at least very similar, may generate the same result. And, being a web system, with the algorithm, server cache, client cache, Internet volatility, etc... maybe it could have some issues in some very very specifc cases. In those cases, big chunk of data doesn't helps and doesn't shows that it work fine for EVERY case, because it is not the rule.

Well, it surely happened with me, it was not my human perception and I'm sure of it, but for sure it's not broken. Works very well.

I could also put lots of links about random number generation development problems, but it's not my intetion. Just reporting a very specifc issue as a form of feedback to Roll20 crew to help the application evolve. We all gain with it, doesn't cause any problems or offends anybody, so it's done!

Well, I will let you guys know if it happens again!

Thanks a lot for the attention!

April 16 (5 years ago)
GiGs
Pro
Sheet Author
API Scripter


INGO DJAN said:

But look, I was not saying that is broken. Just that, in specifc cases, may not work as the designed way.

Be in mind that machines can't generate "random" numbers. 

As those threads posted above show, you are certainly not the first to point this out. All of us commenting understand that, but we are confident that roll20 has done as good a job as is possible. Their RNG is way more robust and random than that built into most programming languages, and they do have a couple of articles about it somewhere that are not doubt linked in some of the posts above about Quantum Roll.


April 16 (5 years ago)
Kraynic
Pro
Sheet Author

I think it is working as intended, the "problem" is this:  it isn't randomizing for an individual player's rolls, but for all rolls made on the platform.  How many 2's I roll in a row don't really matter (it was 3 of those in a row in a recent game session as a player), because there may have been several thousand d20's rolled on Roll20 in the time it took for my skill checks to come up.  The recent peak (from the page in the second link of Andreas' post) at the time of this post was over 425,000 rolls in an hour.  That translates to 7,083.33 rolls per minute and 118.06 rolls per second.  I just happened to hit it at times that 2 came up again.

I admit that knowing that intellectually doesn't actually make me feel any better about rolling a bunch of 2's.


GiGs said:


INGO DJAN said:

But look, I was not saying that is broken. Just that, in specifc cases, may not work as the designed way.

Be in mind that machines can't generate "random" numbers. 

...we are confident that roll20 has done as good a job as is possible. Their RNG is way more robust and random than that built into most programming languages...


Exactly! I read the articles before posting and that is why I don't think it is broken. Just some specific issues that may even not be a issue "de facto" with Roll20. Heheh! Again! I having a great experience with Roll20. And I think reports like this from their users helps them a lot.

April 16 (5 years ago)
The Aaron
Roll20 Production Team
API Scripter

A random sample from a random source will produce a random result, regardless of the period of sampling. Random means the value is not predictable based on past events.  People conflate Random with evenly distributed.  Drawing all the cards from a deck is evenly distributed, you'll never draw the ace of spades twice until you shuffle.  (But you could draw 3 aces, 1 in 5525 odds.)

Roll20's RNG is based on the fluctuation of photons, and is actually far more random than physical dice.  The ease of generating random numbers on Roll20 leads to many discussions about "patterns" in the rolls.  Rolls you make at the gaming table are transient, you are not keeping an easy to refer to journal of each roll result and so do not see the patterns in your physical experience.  The odds of rolling the same number consecutively three times is only 1 in 400.  People spend thousands of dollars on much worse odds every day.  Even rolling 3 20s in a row (a specific number, rather than any number) is only 1 in 8000.  Still very practical odds.

April 16 (5 years ago)
Andreas J.
Forum Champion
Sheet Author
Translator

In short, "true" randomness doesn't "feel" random, hence why some places that want to produce a random "feeling" actually manipulates the results so it conforts better to what feels random for us.


The Aaron said:

A random sample from a random source will produce a random result, regardless of the period of sampling. Random means the value is not predictable based on past events.  People conflate Random with evenly distributed.  Drawing all the cards from a deck is evenly distributed, you'll never draw the ace of spades twice until you shuffle.  (But you could draw 3 aces, 1 in 5525 odds.)

Roll20's RNG is based on the fluctuation of photons, and is actually far more random than physical dice.  The ease of generating random numbers on Roll20 leads to many discussions about "patterns" in the rolls.  Rolls you make at the gaming table are transient, you are not keeping an easy to refer to journal of each roll result and so do not see the patterns in your physical experience.  The odds of rolling the same number consecutively three times is only 1 in 400.  People spend thousands of dollars on much worse odds every day.  Even rolling 3 20s in a row (a specific number, rather than any number) is only 1 in 8000.  Still very practical odds.

Thanks, The Aaron!

Understood! It is still a algorithm, based on variables, server and client sided. It may works perfectly and still someone may experience a "issue".


Andreas J. said:

In short, "true" randomness doesn't "feel" random, hence why some places that want to produce a random "feeling" actually manipulates the results so it conforts better to what feels random for us.


There a tons of ways.

April 16 (5 years ago)
keithcurtis
Forum Champion
Marketplace Creator
API Scripter

What specifically do you see as a problem, (i.e. that is not mathematically supportable) in Roll20's method of producing random die results? Anecdotal reports are not mathematically supportable.

But the problem is that, in that kind of problem, large number of results don't shows the problem. It hides it, actually.

Could you walk me through that? The smaller the sample size the larger the variance. This is basic to probability and statistics. If you feel that Roll20 produces too many duplicates in a row, then a large sample size absolutely will reveal that, given the proper analysis. Look for patterns.

Track every die roll in your game for a couple sessions, then you should be able to see if there is an issue or not. You can't look at a 3-roll run, or even 50 rolls. You need to look over the long haul.

keithcurtis said:

What specifically do you see as a problem, (i.e. that is not mathematically supportable) in Roll20's method of producing random die results? Anecdotal reports are not mathematically supportable.

A hypothetical case, not specific mine:

Let's say that you played 10 game sessions, with like 20 die rolls per session. At every session, at least one time you get 3 consecutives same results on die rolls, and that this only happens when the rolls are made short time after the one another. If you wait some seconds to roll again, the problem is not shown. Do you think this is normal? Do you believe that this might happen so often? Is this human perception? Would you try to get information from the developers?

Could you walk me through that? The smaller the sample size the larger the variance. This is basic to probability and statistics. If you feel that Roll20 produces too many duplicates in a row, then a large sample size absolutely will reveal that, given the proper analysis. Look for patterns.

I will try! Imagine another hypothetical case:

You are trying to generate groups of 10000 random numbers for a program in a universe of 1 to 100. As you studied the final result set statistically, everything seemed great. Than a user informs you that they a getting consecutive result, like 5 to 8 results constantly, everytime they executes the program.

And then, you don't get offended, and looks part of the result set, noting multiple times, from time to time, consecutive rolls like this: (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), ( 2, 2, 2, 2, 2, 2, 2, 2 ,2 ,2 ,2), (3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3)... (100, 100, 100, 100, 100, 100, 100). So you realize that this is not OK, because this is a "random" roll and is statistically and probability WRONG. Although your program is not wrong, the result is. You would never perceive the problem analyzing big chunks of data, because with (BASIC, as you mentioned) statistics and probability this (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), ( 2, 2, 2, 2, 2, 2, 2, 2 ,2 ,2 ,2), (3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3), (100, 100, 100, 100, 100, 100, 100) is similar to (1,1, 2, 2, 3, 3...100, 100). Finally, you analize your code and sees that, with your algorithm, when user does not adjust his computer time/clock, the problem happens. You curse the user and all his past, but at least learnt something new about Stats and Probs.

Be well, friend!


Rabulias said:

Track every die roll in your game for a couple sessions, then you should be able to see if there is an issue or not. You can't look at a 3-roll run, or even 50 rolls. You need to look over the long haul.


That'ss is the problem. Only occurs on specifcs cases, not all the time.

April 17 (5 years ago)
keithcurtis
Forum Champion
Marketplace Creator
API Scripter

If you don't check for repeating pattern strings in a statistical analysis, the analysis is flawed. If you are concerned about strings of consecutive multiples appearing in smaller queries, you can also build your poll like [[10d100]][[10d100]]...[[10d100]] for 100 repetitions. But this is sophistry. Since there is no evidence that this is happening, nor even any reason to suspect it might be happening until you try. Just noticing a few 20-20-20's in a few month's play means nothing. Even noting a string of 6 20s in a row means nothing until you do an analysis to ensure you are not merely at one end of a probability curve.


When it comes down to it, you have made a statement in the top post:

Theres too many repeated dice rolls result.

This statement requires statistical proof or it is merely a subjective opinion. Believe me, as Andreas posted, many people have accused Roll20 dice of any number of biases, but in all of those threads, there's not one shred of evidence. That being said, there are mountains of evidence of bias in physical dice. Until such evidence is forthcoming, I'm absolutely more than fine with the apparent* randomness of the d20 roller.



*In any conversation about randomness, at some point the discussion approaches the philosophy of determination, and I refuse to discuss philosophy. ;)

Thanks again for your time, keithcurtis.

keithcurtis said:

If you don't check for repeating pattern strings in a statistical analysis, the analysis is flawed. 

Right?!

keithcurtis said:

When it comes down to it, you have made a statement in the top post:

Theres too many repeated dice rolls result.

This statement requires statistical proof or it is merely a subjective opinion. Believe me, as Andreas posted, many people have accused Roll20 dice of any number of biases, but in all of those threads, there's not one shred of evidence. That being said, there are mountains of evidence of bias in physical dice. Until such evidence is forthcoming, I'm absolutely more than fine with the apparent* randomness of the d20 roller.



*In any conversation about randomness, at some point the discussion approaches the philosophy of determination, and I refuse to discuss philosophy. ;)

It is in fact subjective. I understand that it's probably not broken and that Roll20 does a great job, as I stated many times, but it was a odd occurrence that happened with me and not meant to acuse Roll20 die rolls of being biased or present any proof of it. Just report my experience and bring attention to it. As for the tests, I believe that is not my job to do it, but Roll20's. I'm just a user and a stakeholder and I believe that experience feedbacks from users are very important to them and transparency even more.

Please undestand that it was MY experience and I wanted to know if anybody was having the same problem, that, as I wrote in previous post, might no even be a problem.

I have had strings of the same number multiple times in a row on a physical die, and in Roll20. That's randomness for you. It seems weird to us because we humans are not random. Your experience in this case is normal and not noteworthy, though, so Roll20 does not need to address it or test anything at this point. Not to be rude or dismissive of you, but to me it's like posting "hey the sun came up today!" that you got a few runs of the same number on a die. :-)

April 17 (5 years ago)

Edited April 17 (5 years ago)
keithcurtis
Forum Champion
Marketplace Creator
API Scripter

And INGO DJAN, I hope it doesn't feel like we're picking on you. But this thread subject comes up a lot, particularly from new users. It's like being at a party, and every third guest who shows up at the door wants to talk about the Witcher, but everyone at the party talked about the Witcher three hours (and three parties) ago and are tired of talking about the Witcher. ;)



This post probably already feels dated and will only become more so

Rabulias said:

...Your experience in this case is normal and not noteworthy, though, so Roll20 does not need to address it or test anything at this point. Not to be rude or dismissive of you, but to me it's like posting "hey the sun came up today!" that you got a few runs of the same number on a die. :-)

Well, I post about a problem I experienced, so...


keithcurtis said:

And INGO DJAN, I hope it doesn't feel like we're picking on you. But this thread subject comes up a lot, particularly from new users. It's like being at a party, and every third guest who shows up at the door wants to talk about the Witcher, but everyone at the party talked about the Witcher three hours (and three parties) ago and are tired of talking about the Witcher. ;)



This post probably already feels dated and will only become more so

Of course, no problem! I really should have choose my words more carefully and looked for an appropriate place to write this.

April 17 (5 years ago)
Andreas J.
Forum Champion
Sheet Author
Translator

INGO DJAN said:

Of course, no problem! I really should have choose my words more carefully and looked for an appropriate place to write this.

No, it was the right place to post. But it's surprising that you seem to completely disregard the rebuttals by other's & the infodump posted.


Anyway, I'll stop responding to this thread as well. At least I've improved my post somewhat, for the next time this comes up again.

April 17 (5 years ago)

Definately nice to learn more about the intricacies of the rng, so thanks for going over this.


Andreas J. said:

INGO DJAN said:

Of course, no problem! I really should have choose my words more carefully and looked for an appropriate place to write this.

No, it was the right place to post. But it's surprising that you seem to completely disregard the rebuttals by other's & the infodump posted.


Anyway, I'll stop responding to this thread as well. At least I've improved my post somewhat, for the next time this comes up again.

Sorry if it looked like this, it was not my intention. I understood what you posted and what people said, but I expressed my experience only.

If I buy a new physical dice and from time to time it gets repeated 20 results, I will feels odd. Doesn't mean there's somethings wrong with the dice, but doesn't mean that isn't strange. And I though there was no problem and post about it at a forum.

But as keithcurtis said, must be very tiresome for you guys to talk about it again and again, and for that I'm sorry. And that's why I don't think it is the right place.

I glad that the post helps.

Thanks for your time.

April 17 (5 years ago)
Kraynic
Pro
Sheet Author


INGO DJAN said:


If I buy a new physical dice and from time to time it gets repeated 20 results, I will feels odd. Doesn't mean there's somethings wrong with the dice, but doesn't mean that isn't strange. And I though there was no problem and post about it at a forum.


That is what I tried to address in my post.  It isn't repeated results.  It looks like repeated results to you, but on the roll20 side, there may have been a thousand (or more) rolls between your 2 dice rolls that happened to turn up the same number result.


April 17 (5 years ago)
Andreas J.
Forum Champion
Sheet Author
Translator

INGO DJAN said:

If I buy a new physical dice and from time to time it gets repeated 20 results, I will feels odd. Doesn't mean there's somethings wrong with the dice, but doesn't mean that isn't strange.

Your hypothetical is nowhere near the same ballpark as your described experience, so I don't see the point. It was same with your other hypothetical. If anything like that would happen, sure, you'd have a point, but it's nowhere near the same thing as rolling three 1 in a row with a d20 every now and then.

Watch the Randomness is Random-video, it really should show the flaws in how we thing randomness works and how we expect things to behave.