Jump to content

RNG Experiment: Attack Coinflips


  • Please log in to reply

03 March 2017 - 12:56 AM

#1

Sakura150612

    Elite Trainer

  • Sakura150612

Introduction:

For a very long time, many users have questioned the integrity of the game's RNG, claiming that it is broken in one way or another. These complaints range from broken coin flips, to excessive mulligans, to bad shuffling. These claims have for the most part have had little to no data, which means that the only forms of proof we had of the RNG working correctly were the words of the Staff and common sense. Of course, some people simply refuse to believe in the authorities for some reason, so I've decided to gather a significant amount of data myself to shed some actual light on this perceived issue.

 

I'm going to be frank, I pretend to burry the issue of the RNG once and for all eventually. For now I will begin with a small experiment on the simplest for of RNG (coin flips) to get this rolling, but I intend to tackle the other forms of RNG when I have time, which might not be anytime soon.

 

The reasons why mulligans, deck shuffling and other types of random events are more complex to investigate are mainly that 1) Many time doing so requires knowledge of what ended up in your deck and what ended up in your prizes. It's possible, not too hard to do, but time consuming to do repeatedly, as I do need significant data pools to make a valid study, and 2) Perhaps more importantly than the first reason, is that the math behind calculating the probabilities of some of these is an absolute nightmare. I do not have the time to spend on doing these calculations at the current time.

 

This experiment will cover 2 parts mainly:

  1. A comparison of 100 coin flips in each AI games and PvP games in order to show there is no significant difference between them. Most importantly, to show anyone who might be skeptical of data collected in AI games (which to be honest is a necessity, given that collecting hundreds or thousands of coin flips in PvP in a short time span is tough and very time consuming) that you do not get worse coin flips in PvP than you would in AI.
  2. A collection of 500 coin flips in AI games, including the initial 100. This is to show that the spread of data is in fact more than less even at higher amounts of data.

Some Concepts I'll Use:

Before starting I'd like to point out some things that are necessary to understand my mini study.

 

Sampling Error and Acceptable Range​

While the values should tend to an average in the long run, there DOES exist a way of telling if small data pools correspond to a working RNG. As you'd expect, the more data the closer we get to a 50-50 split and the less data the opposite is true, and there is a mathematical way of determining if the deviation from a 50-50 split is within the expectations of a normal, functioning RNG, which is to say the Error of a sample. The Error gives us a range of values where the distribution of heads and tails is considered to be acceptable (meaning it indicates an RNG that works correctly).

 

To determine the error of a sample where the outcome is binary (yes or no, heads or tails, etc.), we must divide 1 by the square root of the sample size, giving us a percent of acceptable error.

 

Error = 1 / (S Size)1/2

 

The range of acceptable heads/tails distributions is the expected average (meaning 50%, which I will be referring to in its decimal for, 0.50) ± the error.

 

Range: Lower End = 0.50 - Error; Upper End = 0.50 + Error

 

For the most part I took samples in the AI games in groups of 10 or similar, and in the PvP games in groups of 5 or similar. To give everyone an idea, the ranges from sample sizes between 4 and 12 are the following:

  • 4: ±0.50 [0.00-1.00]
  • 5: ±0.45 [0.05-0.95]
  • 6: ±0.41 [0.09-0.91]
  • 7: ±0.38 [0.12-0.88]
  • 8: ±0.35 [0.15-0.85]
  • 9: ±0.33 [0.17-0.83]
  • 10: ±0.32 [0.18-0.82]
  • 11: ±0.30 [0.20-0.80]
  • 12: ±0.29 [0.21-0.79]

As you can see, this means that any group of 4 (or less) consecutive coin flips can have absolutely any result at all and it would still be within expectations. For this reason, it's extremely hard to spot any anomalies in the short run with any sampling size that is of 4 or lower.

 

While a 5% difference might not seem like a lot, it really does make a visible difference. For any sample size that has a range other than 0% to 100% you can tell that the vast majority of the samples will not exceed this mathematically determined range.

 

For the 2 overall sample sizes, 100 and 500, the ranges are as follows:

  • 100: ±0.10 [0.40-0.60]
  • 500: ±0.04 [0.46-0.54]

 

Aberrants

What I will determine as an aberrant is any sample that does not fall within the range that I established using the previous formulas. I  want to note that all of my samples are of 5 consecutive coin flips or higher, meaning that I will always have a small probability of obtaining an aberrant.

 

Basically, I have 2 checks for telling if the RNG is working properly or not:

 

> Long Run: The overall sample must be close to a 50-50 spread and within the expected range.

 

> Short Run: The vast majority of the small samples must be within expectations. If the aberrants are very few and far between, chances are that the RNG is perfectly normal. Sadly I do not have a formula to determine how many aberrants would constitute a "broken RNG" and how many would constitute "exceptionally good/bad luck" which could be written off by a small sample size, so this is slightly subjective. However, if the total number of aberrants is blatantly abysmal, then we can all conclude that the RNG is fine.

 

The Decklist and the Collection Method:

I determined that the easiest method for collecting coin flips was using AOR 20/98 Gyarados. For this purposed, I assembled a clone of my Archie's Blastoise deck with some modifications, the idea being to charge Gyarados up with Blastoise. I want to note that this decklist is not optimized for either data collection or actually playing (trying to win), and it has remnants of the original deck which do not serve any purpose here.

 

****** Pokémon Trading Card Game Deck List ******
 
##Pokémon - 16
* 2 Shaymin-EX
* 2 Victini
* 2 Exeggcute
* 1 Jirachi-EX
* 3 Magikarp
* 3 Gyarados
* 1 Manaphy-EX
* 2 Blastoise
 
##Trainer Cards - 32
* 1 Professor Sycamore
* 1 Computer Search
* 1 Rough Seas
* 4 Trainers' Mail
* 3 Acro Bike
* 1 Startling Megaphone
* 4 Puzzle of Time
* 4 Battle Compressor Team Flare Gear
* 4 Ultra Ball
* 2 Archie's Ace in the Hole
* 4 VS Seeker
* 3 Superior Energy Retrieval
 
##Energy - 12
* 12 Water Energy  3
 
Total Cards - 60
 
****** Deck List Generated by the Pokémon TCG Online www.pokemon.com/TCGO  Sakura******
 
This is the list that I used for AI games. The PvP version had some changes to make it more consistent, as most players won't just sit there as I charge 10+ energies on Gyarados (for example, I only used 9 energies in the PvP deck). This deck was (surprisingly enough) not that terrible in PvP, even netting me some wins despite not being what I was aiming for, but that's not what I'm here to discuss so lets move on.
 
I would also like to quickly note that the idea behind Fliptini is not so much to get a better damage output, but to get more flips out of my attacks for the sake of collecting data. In PvP I even decided to reflip in some cases where I would have gotten a KO with the initial flip, simply because I was having a hard time finding people who would stay for a long game with this deck (most people either conceded immediately after I had my combo, or destroyed my poor magikarps before I ever got to do anything. Also decks with bench damage like bats were a no-go in this experiment because it was impossible to save the magikarps before they got to evolve).

 

Part 1: AI RNG V.S. PvP RNG

I will be listing the samples (numbered in the order they were registered), the spread of each one and the total at the end. If there are any aberrants I will explicitly mark them so in a bright red text note. I want to note as well that for PvP I couldn't get a perfect 100 samples, because it's harder to manipulate the number of energies I can attach and how many times I can attack when I'm playing against actual people, assuming that they are trying to win (most of them are). For this reason I ended up with 104 flips in PvP instead of 100, but it doesn't make too much of a difference.

 

AI Games Data:

1) ​H:6; T:4

2) H:4; T:6

3) H:6; T:4

4) H:5; T:5

5) H:5; T:5

6) H:2; T:8

7) H:4; T:6

8) H:3; T:7

9) H:5; T:5

10) H:6; T:4

 

Totals Flips: 100 

Total Heads and Tails: H:46; T:54

Number of Aberrants: 0

Range: 0.40-0.60

Total Within Expectation: Yes

 

PvP Games Data:

1) H:3; T:2

2) H:2; T:3

3) H:2; T:5

4) H:6; T:1

5) H:4; T:3

6) H:4; T:3

7) H:6 T:2

8) H:6; T:2

9) H:4; T:1

10) H:3; T:2

11) H:2; T:4

12) H:2; T:4

13) H:4; T:3

14) H:3; T:2

15) H:3; T:2

16) H:3; T:4

17) H:6; T:1

18) H:0; T:7 ABERRANT

 

 

Totals Flips: 104 

Total Heads and Tails: H:63; T:41

Number of Aberrants: 1

Range: 0.40-0.60

Total Within Expectation: Yes

 

As you can see, in both scenarios the total is within expectations (AI was in the lower quarter while PVP in the upper end), and there was exactly 1 aberrant sample in all of them, although there were a couple more that cut it close (6 and 1 is close to the limit but still within expectations).

 

From these samples I can conclude two things:

  1. RNG in AI and PvP should be about the same. In fact, my luck in PvP was better than in AI, which is not what I am trying to prove but it does serve to show that people aren't getting screwed over in their games at least. While some may still claim they do, I have mathematically proven that any streaks of good and bad luck I may have gotten (with exactly 1 exception) are within expectations, and both averages are also within expectations. This means that any data I collect in AI will be considered as a valid representation of what happens in PvP.
  2. Initially, it would seem that the RNG is completely fine. To further prove this I proceded to collect 400 more samples in AI to gather a total of 500 coin flips.

Part 2: Collection of 500 Coin Flips

For the second part, I took the initial 100 flips and added 400 more. This took me a total of 50 different samples of varying sizes. The following is the list of the samples (note, once more, that the first 10 samples are the same 10 I postes in the first part):

 

1) ​H:6; T:4

2) H:4; T:6

3) H:6; T:4

4) H:5; T:5

5) H:5; T:5

6) H:2; T:8

7) H:4; T:6

8) H:3; T:7

9) H:5; T:5

10) H:6; T:4

11) H:7; T:3

12) H:6; T:4

13) H:2; T:8

14) H:4; T:6

15) H:6; T:4

16) H:6; T:4

17) H:5; T:5

18) H:7; T:3

19) H:6; T:4

20) H:4; T:6

21) H:4; T:4

22) H:6; T:2

23) H:5; T:4

24) H:2; T:7

25) H:5; T:4

26) H:3; T:6

27) H:7; T:4

28) H:5; T:6

29) H:7; T:4

30) H:8; T:3

31) H:6; T:5

32) H:7; T:4

33) H:6; T:3

34) H:3; T:6

35) H:5; T:5

36) H:6; T:4

37) H:5; T:5

38) H:4; T:6

39) H:6; T:4

40) H:4; T:6

41) H:5; T:5

42) H:4; T:6

43) H:5; T:5

44) H:4; T:6

45) H:3; T:7

46) H:5; T:5

47) H:2; T:9 ABERRANT

48) H:8; T:3

49) H:6; T:5

50) H:5; T:6

 

Totals Flips: 500 

Total Heads and Tails: H:250; T:250

Number of Aberrants: 1

Range: 0.46-0.54

Total Within Expectation: Yes

 

Much to my surprise, the final result was an exact 50-50 split. Anything between 230 heads and 270 heads would have been within expectations, but this time the total was dead in the middle of the range. I think it's also important to note that there was exactly 1 aberrant in 50 samples, which I think could be considered blatantly abysmal compared to the total.

 

Conclusions:

With this, I believed I have proven once and for all that coin flips are nowhere near to be as completely broken as some people claim. I would like to explore some more cards that require coin flips and the start of the game flip as well, but I'd like to point out that many people have claimed (incorrectly) that their coin flips for attacks and similar things are way off, which this small study completely disproves.

 

While we may never know the RNG algorithm (and would probably not be able to interpret it even if we did), I have now empirically proven that coin flips have shown absolutely no indication at all of being broken.

 

Thus, the ultimate conclusion of my experiment is that the RNG is working as expected.


  • 27

03 March 2017 - 01:09 AM

#2

Mod_Elderberry

    Moderator

  • Mod_Elderberry

Thoughtful, thorough, and filled with data? I like this post sooooo much!


Moderator Elderberry
Pokémon TCG Online Moderator

Need help from the support team? Visit the support portal and submit a ticket!


03 March 2017 - 01:10 AM

#3

Sakura150612

    Elite Trainer

  • Sakura150612

Thoughtful, thorough, and filled with data? I like this post sooooo much!

 

Thank you very much :D I really appreciate the comment. I just honestly hope this will help clear out some of the common questions and doubts regarding the RNG.


  • 1

03 March 2017 - 01:14 AM

#4

SuperStone

    Elite Trainer

  • SuperStone

Excellent. :) I think I'll record my next ten matches with Kingdra just for fun; it looks like you've been pretty thorough.


The rest of the world must act as they see fit.  If TPCi insists on keeping the Chat canned, that is their decision.

 

But that does not stop me from opposing it.

And I will oppose it forever.

  • 0

03 March 2017 - 01:23 AM

#5

Evmaster37201

    Junior Trainer

  • Evmaster37201

Where is my clapping emote? This post is full of win.


♥~ Treat others how you want to be treated ~

  • 1

03 March 2017 - 01:39 AM

#6

Felidae_

    Elite Trainer

  • Felidae_

Using science and data, to determine the RNG, rather than pure speculation, small sample sizes and a “gut feeling” ? What heresy is this ?!


The shadows of the abyss are like the petals of a monstrous flower that shall blossom within the skull and expand the mind beyond what any man can bear, but whether it decays under the earth or above on green fields, or out to sea or in the very air, all shall come to revelation, and to revel, in the knowledge of the strangling fruit - and the hand of the sinner shall rejoice, for there is no sin in shadow or in light that the seeds of the dead cannot forgive...

 

  • 3

03 March 2017 - 01:39 AM

#7

Rainbow-XN

    Expert Trainer

  • Rainbow-XN

Academic. Excellent analysis derived from thorough and thoughtful effort.

 

After reading it I'm positive that the RNG is broken :D


All time favorite:

Steelix, Haunter, Scizor, Beedrill, Crobat, Gyarados

 

 

  • 6

03 March 2017 - 02:02 AM

#8

EpicShonenGuy

    Senior Trainer

  • EpicShonenGuy

Can you explore the coins flips for Torkoal from S&M when you have time?

 

Early this day i had a match where that Torkoal from his second attack that does 50 damage and puts the opponents active pokemon paralyzed, was hitting heads every turn and killed my active darkrai-ex that way and when i placed my other prepared darkrai-ex, it still kept hitting heads and i was forced to concede, because it looked like i was bound too lose that way. 


  • 0

03 March 2017 - 02:42 AM

#9

Sakura150612

    Elite Trainer

  • Sakura150612

This was pinned!? Omg I'm litterally crying, it has been years since the last time a player made post was pinned, and mine was the first one after the long break. I'm am extremely honored that my contribution is held in such high esteem, and I honestly hope this post can be of help to people in understanding the RNG. Thank you VERY much for pinning my post :D


  • 6

03 March 2017 - 02:53 AM

#10

samuraison

    Veteran Trainer

  • samuraison

This was pinned!? Omg I'm litterally crying, it has been years since the last time a player made post was pinned, and mine was the first one after the long break. I'm am extremely honored that my contribution is held in such high esteem, and I honestly hope this post can be of help to people in understanding the RNG. Thank you VERY much for pinning my post :D


Congrats. Couldn't have happened to a better person. You always go the extra mile with your posts so I figured it would happen eventually.

samuraison AKA Son Samurai

Team Skull Moderator

Alola Region Office

"I'll take you on in Pokémon, yo! Been playing this game since '98, bro!"

  • 3

03 March 2017 - 03:29 PM

#11

theotherguytm

    Expert Trainer

  • theotherguytm

This was pinned!? Omg I'm litterally crying, it has been years since the last time a player made post was pinned, and mine was the first one after the long break. I'm am extremely honored that my contribution is held in such high esteem, and I honestly hope this post can be of help to people in understanding the RNG. Thank you VERY much for pinning my post :D

Mathematic and logic triumph yet again. This is the power of numbers, showing nothing but truth. No bias can be found in raw data. (Data after a well-trained statistician has got hold of it is a completely different matter, and can be considered an outlier.)


This is my OTHER signature.

 

http://forums.pokemontcg.com/topic/42500-the-mindless-nonsense-begins-my-ideas/

  • 0

03 March 2017 - 04:22 PM

#12

samuraison

    Veteran Trainer

  • samuraison

Can you explore the coins flips for Torkoal from S&M when you have time?
 
Early this day i had a match where that Torkoal from his second attack that does 50 damage and puts the opponents active pokemon paralyzed, was hitting heads every turn and killed my active darkrai-ex that way and when i placed my other prepared darkrai-ex, it still kept hitting heads and i was forced to concede, because it looked like i was bound too lose that way.


Ask yourself why the developers would program one card to have terrible flips. Especially a card that has no significance in tournament play.

TL;DR - it's easier to code every card to work the same. You had bad luck.

samuraison AKA Son Samurai

Team Skull Moderator

Alola Region Office

"I'll take you on in Pokémon, yo! Been playing this game since '98, bro!"

  • 1

03 March 2017 - 05:54 PM

#13

PokeeZepp

    Veteran Trainer

  • PokeeZepp

People do forget quite quickly the times they win, and remember deeply the times they lose.  This post hopefully can help them understand that life happens both directions, and no one is 'never lucky'.


Anything above this line is just my opinion.  Anything below the line is, too.

  • 0

03 March 2017 - 07:15 PM

#14

Sakura150612

    Elite Trainer

  • Sakura150612

Congrats. Couldn't have happened to a better person. You always go the extra mile with your posts so I figured it would happen eventually.

Thank you :D

 

Can you explore the coins flips for Torkoal from S&M when you have time?

While I do intend to explore further the different forms of RNG in the game, like I said in the original post that might not be happening anytime soon. Right now I had the free time to do this much, but if I start taking specific requests there will be no end to this. There are a plethora of different attacks which require coin flips, so even if I did this people could suspect that one of the untested attacks could be malfunctioning. Again, I will do some more in the future to show that what's true for one coin flip attack is true for all of them, but right now I will leave it at this.


  • 0

03 March 2017 - 07:25 PM

#15

EpicShonenGuy

    Senior Trainer

  • EpicShonenGuy

Ask yourself why the developers would program one card to have terrible flips. Especially a card that has no significance in tournament play.

TL;DR - it's easier to code every card to work the same. You had bad luck.

That's most likely the case, but hitting heads for 8 or 9 (no breaks in between) turns it's kinda weird i've had really bad luck before but nothing compared to that, maybe it was just a fluke, but one thing i can tell it was definitely weird


  • 0

03 March 2017 - 07:28 PM

#16

EpicShonenGuy

    Senior Trainer

  • EpicShonenGuy

 

 

While I do intend to explore further the different forms of RNG in the game, like I said in the original post that might not be happening anytime soon. Right now I had the free time to do this much, but if I start taking specific requests there will be no end to this. There are a plethora of different attacks which require coin flips, so even if I did this people could suspect that one of the untested attacks could be malfunctioning. Again, I will do some more in the future to show that what's true for one coin flip attack is true for all of them, but right now I will leave it at this.

Sure, no problem, don't push yourself to hard, great post btw.


Edited by EpicShonenGuy, 03 March 2017 - 07:28 PM.

  • 0

03 March 2017 - 07:32 PM

#17

Sakura150612

    Elite Trainer

  • Sakura150612

That's most likely the case, but hitting heads for 8 or 9 (no breaks in between) turns it's kinda weird i've had really bad luck before but nothing compared to that, maybe it was just a fluke, but one thing i can tell it was definitely weird

 

This is what I described in my study as an aberrant. As you can see, it's definitely possible (PvP data, sample #18). But, as you can also appreciate, it was one set of coin flips among 18 samples. Or if you want to consider the overall result, it was one in 68 groups of coin flips (there was a second aberrant too, but nothing as egregious as 0/7, which is similar to what you describe).

 

Simply put, if it happens only once in a long while, it's nothing that would indicate the RNG is off somewhere. Due to the way probabilities work, in theory any combination of heads an tails can happen even in a healthy RNG. The only difference between a good RNG and a broken one is that said aberrants should happen only very sporadically in a good one, while a broken one should show them very often.

 

Sure, no problem, don't push yourself to hard, great post btw.

 

Thank you :)


Edited by Sakura150612, 03 March 2017 - 07:33 PM.

  • 0

04 March 2017 - 09:53 AM

#18

Mekkah

    Trainer

  • Mekkah

no look you don't understand it only happens to meeee the game is rigged against meee 

 

/s

 

Nice to have some actual facts instead of the alternative anecdotal ones.


  • 0

04 March 2017 - 02:33 PM

#19

SuperStone

    Elite Trainer

  • SuperStone

Well, I got my ten games in. :) As you can see, there were a fair number that ended in concessions. :P

       Heads, Tails

Game 1-  4, 4
Game 2- 2, 1
Game 3-2, 2
Game 4- 9, 9
Game 5- 2, 1
Game 6- 2, 2
Game 7- 2, 0
Game 8- 1, 1
Game 9- 10, 14
Game 10- 10, 14

 

Total- 43, 48


Edited by SuperStone, 04 March 2017 - 02:34 PM.

The rest of the world must act as they see fit.  If TPCi insists on keeping the Chat canned, that is their decision.

 

But that does not stop me from opposing it.

And I will oppose it forever.

  • 0

04 March 2017 - 04:37 PM

#20

Mod_Elderberry

    Moderator

  • Mod_Elderberry

That's most likely the case, but hitting heads for 8 or 9 (no breaks in between) turns it's kinda weird i've had really bad luck before but nothing compared to that, maybe it was just a fluke, but one thing i can tell it was definitely weird

Why is getting heads 8 or 9 times in a row evidence of something being weird with rng?

For instance, the chance of getting 8 heads in a row is 0.5or 1 in 256 coin flips. That seems like a very real possible outcome to me. That it happens is not very spectacular to me, what would be really impressive is calling it before it happens ;P

Another way to put it is, it is perfectly possible to roll 20 heads in a row with a fair coin, the chances of doing that are 1 in 1,048,576 (or, about one in a million). Does it happen every time you play? Not likely. ---  but, can it happen? Assuredly; Why not? --- especially if you consider how many people play PTCGO and how many times constantly the coin is flipped... it's bound to happen to at least someone once in a while.

 

Well, that's my opinion on that subject at least.


Moderator Elderberry
Pokémon TCG Online Moderator

Need help from the support team? Visit the support portal and submit a ticket!