Someone else’s data says nothing about your game

Data is evidence, not an answer. That line is not mine, but it is one of the ones I use most in this industry, from Slay the Spire designer Anthony Giovannetti’s GDC 2019 talk. The talk is called Metrics Driven Design and Balance, so the man who based his game’s design and balance on metrics from end to end stands in the middle of his own presentation and tells you not to trust metrics too much.

Look at Polden, the studio everyone points to right now. Cartel Pilots Wanted collected 400k+ and Totally Secure Airport 450k+ wishlists, and they got them off a trailer alone, before the games went into full production. People see that, land on “so we need to shoot vertical video”, tell it as if they had reinvented the wheel, and are now trying to apply it to every game.

Polden’s plan is to ship 20 games and get one hit out of them. Their budget per game is in the 50-150k dollar band and they mostly take in formulaic, action heavy arcade friendslop from outside.

So what is in front of you is not a tactic you can copy, it is an operation. That model only works if you first find 20 games and then have the budget to absorb 19 failures. It is basically the brute force strategy Voodoo and Rollic ran on mobile. This model is not sustainable or applicable with 5-6 games a year, or with a prototype.

Polden shares its launches that did not work too, openly. But being open does not make that data useful to you. When a publisher publishes its data there is always a reason, and here it is to do the filtering up front. Once you write down plainly what failed, the developer with a similar game does not apply, and even if they do, the publisher has the data to say “this did not work before”. The numbers they share are there to pull developers into their own portfolio, a kind of head hunting.

On top of that, the number itself is weak data. Wishlists collected off a trailer, with nothing playable anywhere, show that the video worked rather than the signals that make a game a game. Cuffbust went through this recently, it launched at 20 dollars with 100 thousand wishlists, its reviews turned Mostly Negative, and it had to cut the price to 10 dollars in the first hours and then to 5.79. A simple but arrogant pricing mistake killed all the momentum and trust around the game, then the problems inside the game killed the game. Polden’s games carry the same risk, but so far they are doing very well and moving correctly. Keep in mind as well that all the data they share is only about friendslop games.

For me it went the other way. I launched Metal Thunder with 3k WL, it sold 100 copies a day at launch and dropped to 10-15 within six weeks, and by the end of January it was down to 5. Up to that point this is normal for 3k wishlists, and someone with no experience would decide the game was over and go next. But that was exactly the stretch where I believed in the game and shipped update after update through January and February. It worked, and daily sales hit 315, three and a half months after launch. That momentum got me a Daily Deal, and on the day of the Daily Deal I got into the top 3 of the sales charts in a lot of countries.

People with no real experience believe anyone who shares data, blindly, and switch into loud lecture mode: “data does not lie, man, trust the data, not experience, look, it is right there”. Data does not lie, true, but data has no voice either. Someone decided what to measure, over what window to measure it, and what to share and what to leave out. All of those decisions come from experience and you only see the aftermath. Even when you listen to Zukowski you should listen only as a benchmark, and he says so himself anyway.

The industry is also full of half baked “publishing managers” who have never been in the kitchen, unfortunately. They have a parasocial relationship with the people who share data, they think they know them well and that they understand every dynamic. Then they hand out context free advice to anyone who walks past.

Reading charts about games correctly takes experience. What those numbers serve, what they were calculated on, which condition you do not have: to see that, you need to have made those decisions yourself, knowing the whole design context. One of the people who leans on metrics most cares about this enough to put it on a GDC slide, while the guy who writes “here is the proof” underneath it, who has only seen the marketing side of 3-5 games in his life, does not even know what he is missing.

So what do you do? Take a lesson from every metric you see, not a conclusion, and treat it as a benchmark. Then read those benchmarks against your own game. Not “this many wishlists came in”, but “in this genre, with this budget, with this reach, this many can come in”. Then ask yourself what would have to be true for you to get the same. The moment you write that list you can already see whether the number fits you.

You know your game better than anyone. Nobody knows which decision you made and why, or what led to what, better than you do. That is why I looked at that flat line in January and did not quit. Someone else’s chart does not replace that knowledge, it is only a tool that helps you read it.

The same goes for the numbers I share in this series. They are evidence too, not an answer.