Elenbogen-inspired post about playtesting
A friend of mine (Andrew Elenbogen) tweeted this somewhat recently:
"Competitive Magic is the art of correctly generalizing from sample sizes too small to draw real conclusions."
It got me to thinking about a few things that bother me that people do a lot. What's the point of playtesting with so-called small sample sizes?
In my view, there's a few reasons to playtest, and almost none of them involve deriving percentages or some metric, which might sound odd coming from a statistician.
Here's some good reasons to playtest:
a) Learning how your deck works. Often times, I'll look at a deck on paper and know _most_ of its tricks, but not all of them until you actually play.
b) Learning what matters in matchups. Quick example: Playing a few games of UR Phoenix versus like Dredge can inform you of how to use your Thing in the Ice correctly (namely usually save it so you can KO them into 1-2 hits). Playing a few games to learn how games 'usually' play out or what plans might succeed is good.
c) Trying different sideboard configurations (related to point b) to see if a slightly different plan might work better. This is somewhat related to a) below, but just being careful is good enough to avoid this.
Bad reasons to playtest:
a) Trying to assess statistical significance of a card change. the number of games required to assess something like that isn't really attainable, but then by that point the other decks have already shifted.
b) Trying to measure a direct win %. Again, similarly, the shifting landscape of the opposition decks makes this a bad idea.
c) Trying to get an accurate measure of a metagame using a small n from (for example: MTGO Leagues). Assuming you have an enormous sample size in a short time frame, this is more doable, but it's not really a good idea for (n<100, this is a judgment call) for a short time frame.
The last related point that I need to get off my chest is that intellectual honesty is required if you want to succeed in Magic, and is indeed a difficult thing to achieve.
You need to be able to let go of a deck, or understand why it's bad in order to move on and try new things.
"Competitive Magic is the art of correctly generalizing from sample sizes too small to draw real conclusions."
It got me to thinking about a few things that bother me that people do a lot. What's the point of playtesting with so-called small sample sizes?
In my view, there's a few reasons to playtest, and almost none of them involve deriving percentages or some metric, which might sound odd coming from a statistician.
Here's some good reasons to playtest:
a) Learning how your deck works. Often times, I'll look at a deck on paper and know _most_ of its tricks, but not all of them until you actually play.
b) Learning what matters in matchups. Quick example: Playing a few games of UR Phoenix versus like Dredge can inform you of how to use your Thing in the Ice correctly (namely usually save it so you can KO them into 1-2 hits). Playing a few games to learn how games 'usually' play out or what plans might succeed is good.
c) Trying different sideboard configurations (related to point b) to see if a slightly different plan might work better. This is somewhat related to a) below, but just being careful is good enough to avoid this.
Bad reasons to playtest:
a) Trying to assess statistical significance of a card change. the number of games required to assess something like that isn't really attainable, but then by that point the other decks have already shifted.
b) Trying to measure a direct win %. Again, similarly, the shifting landscape of the opposition decks makes this a bad idea.
c) Trying to get an accurate measure of a metagame using a small n from (for example: MTGO Leagues). Assuming you have an enormous sample size in a short time frame, this is more doable, but it's not really a good idea for (n<100, this is a judgment call) for a short time frame.
The last related point that I need to get off my chest is that intellectual honesty is required if you want to succeed in Magic, and is indeed a difficult thing to achieve.
You need to be able to let go of a deck, or understand why it's bad in order to move on and try new things.
posted by Jarvis Yu at 4:37 AM
0 Comments:
Post a Comment
<< Home