Analysis of NAF statistics

Got a great idea and/or proposal for BloodBowl?

Moderator: TFF Mods

AndeeT
Rookie
Rookie
Posts: 28
Joined: Thu Jul 28, 2016 7:49 pm

Analysis of NAF statistics

Post by AndeeT »

Hi,

My first time posting here, so "Hi!" everyone.

I thought I would share my small analysis of NAF statistics (thank you doubleskulls for making this freely available).

I started by looking if there was a correlation between the number of matches played per race and the win percentage per race. I used the total data for all NAF competitions (http://naf.talkfantasyfootball.org/tota ... tions.html); 161,080 matches in total. Plotting the number of matches per race (X axis) against the win percentage for that race (Y axis) gives the resulting scatter graph;
MATCHES_VS_WINS.jpg
Sorry I can't add the race name as a data label! That would make it easier to interpret. If anyone has a more up-to-date version of Excel than me maybe you could post it with data labels. For now you can look at the table on the hyperlink above to give an idea of where each race sits on the graph.

So, to my eyes it looks like there is at least some kind of linear correlation between the number of matches played and the win percentage of a race. If we perform a Pearson's Correlation Coefficient on this data, we get an answer of 0.59. This test statistic can range from -1 (perfect negative linear correlation) to +1 (perfect positive linear correlation). 0.59 suggests that there is a medium positive linear correlation between the number of matches played per race and the win percentage for that race.

But what does this mean? Well, one way to interpret the correlation is that the those races with more matches under their belt are more likely to have a greater win rate. i.e. Win rate is not necessarily determined by which tier the race is in; if every team had the same number of accumulated matches we might see a lot less disparity between the win rates of the races...

...On the flip side; if we assume that some races are intrinsically 'better', i.e. assume that the tier system is correct, then the above correlation would simply imply that coaches tend to opt for the race they know to be intrinsically 'better'. This popularity leads to the high game count of those races which are known to have a greater win percentage.

To take the analysis further, if we remove those three data points that lie in the bottom left of the scatter graph (left to right; Ogre, Halfling, Goblin) on the assumption that they don't really fit this trend, the Pearson Correlation Coefficient goes up to 0.69. This is a slightly stronger correlation. These three races also happen to be the races that make up "Tier 3" in the tier system that Tom Anders has described in the past, so the data in the scatter graph here certainly support that tier system. Looking at this scatter graph alone you could even define "Two" tiers; Tier 2 being Ogres, Halfling and Goblin and Tier 1 being the other races.


Please let me know your thoughts and comments! I love data and graphs and Blood Bowl, so I jumped at the chance to try this out. A caveat of scatter graphs and correlations in general is that on their own, they CAN NOT imply causality. To really tell if the number of matches determines win percentage, we would need some kind of experiment or simulation, in which we had all races having played the same amount of games. The scatter graph above would suggest that if this were the case, we might see very similar win rates between races.

Doubleskulls; would such a simulation be possible on your database by pulling a specified number of random match results for each race? The results would be very interesting!

Best Wishes,

AndeeT
You do not have the required permissions to view the files attached to this post.

Reason: ''
CyberedElf
Veteran
Veteran
Posts: 257
Joined: Fri May 31, 2013 12:52 am

Re: Analysis of NAF statistics

Post by CyberedElf »

Thanx for the work. The one thing I found odd is that most people look at win ratio instead of % win. I don't know that this would change anything significantly, but some teams have gotten more draws than others. If you are using one axis to measure "has performed better," I would think (wins + draws/2)/total would be more useful than wins/total.

Reason: ''
Image
Itchen Masack
Ex-Mega Star, now just a Super Star
Ex-Mega Star, now just a Super Star
Posts: 1390
Joined: Thu Feb 09, 2012 3:12 pm

Re: Analysis of NAF statistics

Post by Itchen Masack »

Quick Itchen, read this thread before 'the usuals' turn up and spam the feck out of it. Someone has used a graph in their first ever post. Turfwar!

Obviously i dont understand any of the graph, but it looks pretty. Well done :D

Reason: ''
AndeeT
Rookie
Rookie
Posts: 28
Joined: Thu Jul 28, 2016 7:49 pm

Re: Analysis of NAF statistics

Post by AndeeT »

Thanks Itchen...

The graph is showing that as the number of matches a race has played increases, the percentage of games they have won also increases. There is a linear relationship/correlation.

As to why this might be happening, well I take a few guesses in the original post. Another situation that might account for this correlation is that those races with high numbers of games played are popular due to the same coaches bringing the same race back to tournaments. This would account for the higher win percentage in those races as we know that if you keep playing with the same race, you understand better how the team plays and are therefore more likely to win the newer matches. Of course, it's impossible to say this is the case without seeing the numbers of unique coaches per race, and I am not sure that this info is readily available.

CyberedElf - thank you. I will re-do the graph using wins+draws as you have suggested. Indeed it will be interesting to see the results!

Best Wishes

AndeeT

Reason: ''
User avatar
Darkson
Da Spammer
Posts: 24047
Joined: Mon Aug 12, 2002 9:04 pm
Location: The frozen ruins of Felstad
Contact:

Re: Analysis of NAF statistics

Post by Darkson »

Not sure if it makes any difference to the graph, or your point, but feel I should point out that not all teams have existed for the same amount of time. For example, Spanner, Underworld and Pact have only been NAF ranked since Aug 2008 (my tournament was the first to allow them) and, iirc, when the NAF first started in 2002 there were only 16 races.

Reason: ''
Currently an ex-Blood Bowl coach, most likely to be found dying to Armoured Skeletons in the frozen ruins of Felstad, or bleeding into the arena sands of Rome or burning rubber for Mars' entertainment.
Itchen Masack
Ex-Mega Star, now just a Super Star
Ex-Mega Star, now just a Super Star
Posts: 1390
Joined: Thu Feb 09, 2012 3:12 pm

Re: Analysis of NAF statistics

Post by Itchen Masack »

Darkson wrote:For example, Spanner, Underworld and Pact have only been NAF ranked since Aug 2008
Bit rude! Also Slann too.

Reason: ''
AndeeT
Rookie
Rookie
Posts: 28
Joined: Thu Jul 28, 2016 7:49 pm

Re: Analysis of NAF statistics

Post by AndeeT »

Hi Darkson,

I think it makes sense that those teams that are newer to Blood Bowl and to NAF tournaments have less accumulated matches. The point regarding the correlation still stands. For example, Chaos Pact come in at the bottom left of the line of best fit on the scatter graph (5413 matches played vs. 0.36 win percentage), which is to be expected, as you rightly pointed out, they came into the NAF much later than the bulk of other teams.

I will try to get a graph with the Race names on the graph; this would make it easier to interpret.

Reason: ''
AndeeT
Rookie
Rookie
Posts: 28
Joined: Thu Jul 28, 2016 7:49 pm

Re: Analysis of NAF statistics

Post by AndeeT »

Hi,

I have some updated graphs to show.

The first one is just a repeat of the original graph but with data labels showing the race names. Some are hard to read as the graph is a bit sorry but I don't know a better way of showing this! If anyone has an idea please let me know.
WIN_PERCENT_LABEL.jpg
You do not have the required permissions to view the files attached to this post.

Reason: ''
AndeeT
Rookie
Rookie
Posts: 28
Joined: Thu Jul 28, 2016 7:49 pm

Re: Analysis of NAF statistics

Post by AndeeT »

The second graph is what CyberedElf requested; the same scatter graph with number of matches on the X axis, but the Y axis shows the Win % and Draw % added together (i.e. if you interpret 'success' as draws and wins, rather than just wins).
WIN_DRAW_PERCENT_LABEL.jpg
I thought by counting success in this way (Win + draw) it might change the relationship somewhat, but interestingly, it is almost exactly the same, indicating that Draws vs no of matches correlates in a similar way to Wins vs no of matches.

Repeating the Pearson Correlation Coefficient gives me the same answer for all races (0.59), but just ever so slightly higher (0.71 rather than 0.69) when excluding those Races that are outliers (Ogre, Goblin, Halfling).

Happy to discuss!
You do not have the required permissions to view the files attached to this post.

Reason: ''
CyberedElf
Veteran
Veteran
Posts: 257
Joined: Fri May 31, 2013 12:52 am

Re: Analysis of NAF statistics

Post by CyberedElf »

I know I'm being a stickler, but I mentioned using win + draw/2, not just win + draw. But, if win + draw didn't change things much, I highly doubt the other will either.
More technical squabbling:
win + draw = 1 - loss
So your second calculations is inverse of matches played versus % loss.

Reason: ''
Image
dode74
Ex-Cyanide/Focus toadie
Posts: 2565
Joined: Fri Jul 24, 2009 4:55 pm
Location: Near Reading, UK

Re: Analysis of NAF statistics

Post by dode74 »

Nice idea for a question.
the Y axis shows the Win % and Draw % added together
That's not what he asked for. It's wins + draws/2, which actually equates directly to the last column in the NAF data you linked: Rec (record, I presume). That gives this chart:
Image
No massive changes, but some relative positions are clearly different (e.g. Wood Elves).

Couple of things worth mentioning: the data is descriptive rather than inferential. Drawing conclusions from this graph alone will be prone to errors. Some of the biases are obvious, such as low TV bias and lack of progression/attrition. Others are perhaps less so: ruleset changes and tiering within tournaments (some tourneys give "low power" races bonuses), for example. I'm not sure removing "outliers" as a means to identify correlation is particularly wise: Orcs are an outlier, for example. If you want to ask about T1 teams, though, you can reasonably eliminate all the non-T1 teams from your analysis - Pact, Slann, Vamps, Underworld and the T3 teams. If you do that r=0.60 for the T1 teams.
One thing perhaps worth mentioning is the strong negative correlation (r= -0.79) for the non-T1 teams. If your hypothesis that teams are played more because they win more is true then the non-T1 teams don't hold to that at all. That wouldn't be surprising: few people play those teams expecting to win as they are known to be below par.

Might be worthwhile looking at FUMBBL and Cyanide data too...

Reason: ''
dode74
Ex-Cyanide/Focus toadie
Posts: 2565
Joined: Fri Jul 24, 2009 4:55 pm
Location: Near Reading, UK

Re: Analysis of NAF statistics

Post by dode74 »

So, FUMBBL data:
Image

Overall, r=0.47. For the non-T1 teams r=0.69 (strong correlation and the opposite of the NAF trend) and for the T1 teams r=-0.27(!), so a fairly weak negative correlation and again the opposite of the NAF trend.

Edit for source: http://fumbbldata.azurewebsites.net/stats2.html

Reason: ''
AndeeT
Rookie
Rookie
Posts: 28
Joined: Thu Jul 28, 2016 7:49 pm

Re: Analysis of NAF statistics

Post by AndeeT »

Awesome! Thank you for your lovely graphs dode!

(And sorry I got the win/draw thing wrong; did this in a rush today on my lunch break)

A few questions for you dode;
What program do you use to create your graphs?
What does the dotted line represent?


I totally agree that scatter plots alone aren't enough reason to validate or form a theory but they certainly beg the question as to why there is a pattern at all. Is the pattern to be expected and I am reading too far into it?

Nice idea to isolate non-tier one teams and very interesting that it shows opposite trends in FUMBBL. Even greater interest that there is very poor correlation in FUMBBL for tier 1 teams.
I know other analyses show differences between FUMBBL and NAF for other statistics; does this come down to differences between a closed tournament vs perpetual league ? Wonder if it's a similar thing here. I don't know too much around FUMBBL; is it only perpetual league matches?

And thank you for the link to he FUMBBL data - hadn't found that yet

Best Wishes

AndeeT

Reason: ''
AndeeT
Rookie
Rookie
Posts: 28
Joined: Thu Jul 28, 2016 7:49 pm

Re: Analysis of NAF statistics

Post by AndeeT »

The point about tiering in tournaments and rule set changes didn't even cross my mind; great point to consider. Not sure if we can drill into the data at that level? Rule set changes might be doable by selecting matches between certain years.

Assuming there is some validity in looking into this pattern further; In terms of doing some inferential stats, my idea for an experiment to see if there is an effect of number of games played is to randomly pull match results from the database until all races have an equal amount of matches (~1000 per race?). The hypothesis for this simulation would be that all races should have the same win (or win+draw/2) percentage. If the hypothesis is true we could conclude there is at least some validity there. Of course, as you have shown, this would have to be specific to dataset/match type.

My first problem is that I probably don't have access rights to do that and secondly, I don't know enough about databases and random sampling to do that! Is there anyone around that would be able to discuss the feasibility of this? Is anyone that bothered? Should I just go down the pub and play some Blood Bowl?! Hehe.

Then there is the question of which test statistic to analyse the data with. Would it be fair to run an ANOVA between all races? We could certainly make sample sizes large enough but I don't know if BB match results are normally distributed? Can we assume that the mean for each race will be normally distributed given that we have ~1000 samples per race and the Central Limit Theorum.

Would love to discuss some more!

Best Wishes,

AndeeT

Reason: ''
dode74
Ex-Cyanide/Focus toadie
Posts: 2565
Joined: Fri Jul 24, 2009 4:55 pm
Location: Near Reading, UK

Re: Analysis of NAF statistics

Post by dode74 »

Hey AndeeT

Mine are made in Excel 2016.
The dotted lines are the tier 1 bounds, there purely for clear reference points on the y-axis.
Is the pattern to be expected and I am reading too far into it?
I think what you have is a hypothesis: people tend to play teams more which win more. Whether that hypothesis is true or not is a matter where you look. I've not looked at the Cyanide data yet (it'd be old BB1 data to get the races, tbh, and there are other errors) but I expect FOL (Cyanide matchmaking league) data to be similar to FUMBBL as it is a similar environment. It'd be interesting to look at league data as a 3rd environment as well. I have league data for OCC (Cyanide scheduled leagu) but won't use that as there is a known bias in games played because some teams were released well before others.

The FUMBBL data I used was Blackbox (use the dropdown in the link to get that) which is a perpetual matchmaking league, and it is possible the trends are environment driven: some things such as team play rate seem to differ considerably with environment. Ranked (perpetual league where you and your opponent both agree to the match) is another FUMBBL environment worth looking at (see below).

I don't think there is data at NAF level to enable us to look at tiering at each tournament and any specific house rules, but you can split it down by ruleset here: http://naf.talkfantasyfootball.org/ You don't see much difference between the overall data and LRB6 only data.
The hypothesis for this simulation would be that all races should have the same win (or win+draw/2) percentage.
There has already been considerable analysis done into this. One set of graphs can be found here: http://forum.bloodbowl-game.com/viewtop ... 988#p58988

FUMBBL Ranked:
Image
All teams r=0.39, T1 only r=-0.15, non-T1 r=0.1

Reason: ''
Post Reply