Pantheon+ has been engaged for over a year now in playing and grading MMOs from mostly fantasy titles. Their group comprises gamers from a variety of backgrounds and ages. The one thing they have in common is their love of MMOs and support for Pantheon: Rise of the Fallen.
Having a varied group of MMO gamers produce subjective grades on new and old titles predominantly helped provide community sentiment over different mechanics, visuals, Quality of Life (QOL) implements, and ultimately they hoped their lessons learned would educate the developers of Pantheon.
Fast forward to today, thre are 12 MMOs that have enough feedback to allow for analysis. The idea that the streams and podcasts could help illuminate what players want was short-sighted. In the details of the subjective grades, from A+ to the dreaded FFF- (Triple F Minus created by NathanNapalm), there lies the real findings of the study. Biases from a range of gamers are clearly seen, trends across diverse gamers emerge, and real lessons can be learned.
First, the structure should be defined. Each game had a set of criteria to grade, which changed slightly over time. Each criteria was graded by each individual from A+ to FFF-. Each possible grade in between was given a value, starting from FFF- at 0.1, F- at 1, F at 2, F+ at 3… all the way to A+ at a value of 15. Then, like criteria were lumped together. Not every game has the same criteria that was graded so some data are incomplete, but if it was close to the same sentiment, the criteria were consolidated. Having changed the grades to values, statistical analysis can now be done on the findings. Here is a look at the resulting table of data.
The players and games are visible on the left two columns. It’s important to note the far right column of overall is not an average but a subjective aggregate of the overall experience.
First, let’s consider the individuals for some fun and to prove out the methods.
One player, who for the purposes of this experiment we will call Draq, has graded nearly all of the games. We have a decent set of data. For instance, under Final Fantasy 14, Draq gave the questing mechanic in the game a 0.1, or FFF-. In Embers Adrift, Draq scored their grouping mechanics at a 14, a solid A. To give idea of some of these other numbers, just think that a C is 8 and you can pretty much get the idea from there with each grade having a + and a -.
It looks like Draq wasn’t very happy with most of the game’s questing mechanics. Lost Ark, Fractured, and Ember’s Adrift all received low scores for the questing mechanics in those titles. So, we can look at the average questing grades across all titles and get a sense of how Draq views questing across the MMO genre currently. We can do the same across all the criteria.
The average only tells us a limited story. With enough great games for questing, his average would balance out and it would not be noticeable. So we add another metric, an average deviation. This metric let’s us know how far from average Draq views these specific criteria. The larger the absolute value, the more he has strong feelings one way or another. With a high deviation, even in an average, we know he had large swings away from that C grade.
So, now we know that Draq has strong feelings about the Classes and the Lore in the titles he played, with hard swings in favor of the way classes were handled and hating the lore and the questing. Hey, Draq does hate lore. We must be on the right track.
Averages from the full data table were taken across each criteria. The difference under the average is how far off Draq’s averages were from the rest of the players. Although Draq hated the questing in these titles, apparently so did most others, with him being within one grade move from the average, so if the group thought C-, Draq thought D to D+. But he has much stronger disagreements between the classes, having really liked playing his wizard most of the time, and hating lore almost across the board.
Now, back to deviation for a second. Think of deviation as an emotional response. It is a strong feeling one way or the other. In this case, a deviation number covers both ways from a grade. So, if the average thought about combat was a C, a Deviation rating of 2 is actually 1 point on either side of C, or from a C- to a C+. A 4 would swing both sides two grade steps. Think of transmogging. There are very few in our community that don’t have a instant reaction to that word, either for it or against it. It means a lot to both sides, so both sides go to extremes to make their points. That is deviation. Not a lot of meh reactions to transmogging. Deviation was also captured across the full data table for each criteria. So, now we have average emotional responses across the criteria. If we compare the group to an individual, say Draq, we can see his emotional response on classes was much more favorable than the others in the group, enough to bring a C+ up to a solid B. And that’s basically his response to the grading, he liked the class play so much, it drove him to give much higher grades than most of the others. He cared more, you could say. If you know Draq, you can attest to how much being a wizard means to him in his games, almost the one thing that makes a game for him, if his particular class is dropping DPS bombs. Other criteria did not illicit stronger responses than the group as a whole.
So, that works for one person. Let’s try one more.
Here is another player we will, for the purposes of this research, refer to as NathanNapalm. Nate’s average up top seem to show fairly solid C’s all the way across, middle of the road, easy going. But wait, his differences from the group’s averages are telling a different story. The number all the way to the right, -6.5 is the difference in his averages and the group’s across all criteria. Hey, Nat is a very Negative Nellie. And he does not like PvP, apparently.
Now, look at his Deviation scores, or his mental stability metrics, one could say. Most of Draq’s ranged in the 1s to 3s. Nate is living La Vida Loca with wild swings from the 4s to the 6s. That’s showing that he doesn’t think anything is average; he hates it fervently or he loves it fervently. Luckily for those in his vicinity, most of his wild-eyed reactions were positive, coming in positive numbers, so good thoughts outweighing bad thoughts. And, looking at his Deviation difference from the group, he’s pretty solid in there with everyone else’s feelings on AVERAGE. But, it all goes dark looking at his Fun/Addictability and Overall scores. Nate was a whopping four grade points aka over a full letter grade LOWER on average than what the group thought. Basically, what we see here is a man who will think every game is a flaming bag of dog terds thrown into a dumpster fire dropped into a volcano of garbage until Pantheon comes out. Yep, that seems to fit.
Ok, so the metrics work on personal scales. What does this tell us on the whole?
Here is the roll-ups from the full data table, the average scores and the average deviations, emotional responses. Down below is the ratio of the emotional responses to the average scores. This basically tells us the lower the percentage, the more people enjoy something across the board without being fanatical about it. The lower the number, the more it incited people to violence with each other, or in very high numbers, with the industry as a whole. Here’s the main conclusions from Pantheon+ escapades into today’s MMOs:
I am not going to bother posting the Mark Twain quote about statistics, but it was still an interesting dive across people and the games they suffer, or rather, play through.
I would also rank ESO lore up there, so that is legit /wink
I noticed that "One player, who for the purposes of this experiment we will call Benonai." was missing from the list... a statistics voyeur.
I read this three times. It's not research study in the typical sense. I don't want to be critical of it so here are the abridged bullet points:
I'm going to comment on something that is completely irrelevant to extracting useful info from these tables.
It's really disappointing to not see a lore score for AoC. It is one of my favorite MMORPGs of all time and the lore is just classic.
That said, if I am going to have any meaningful thoughts about this it is going to take time to digest. Thanks Benonai for your obsession and sharing it with us.
Tigersin, research study is a little tongue-in-cheek bud. Not publishing this to a journal or getting it peer reviewed lol. Thanks for putting the abridged version out.
Ruinar, the 101 class has gone through a few different question overhauls and the lore question wasn't being used yet when they did AOC. Would have liked to see that too.
"You can't Monday Morning Quarterback from the field-"
got a good chuckle from this
