Some bugs hit you in the face. You launch the game, run around a bit, the entire UI stops working or your character can't move anymore. These are the ones most of us are probably used to. But sometimes bugs lurk for weeks or months on end without ever surfacing. Do these ones really matter? You could argue a bug no one experiences isn't really a bug at all. I'm not writing this post because of how serious this particular issue was. Instead, it was just a really rewarding moment when I accidentally discovered it.
Analytics
As a small indie developer, my ability to schedule and execute playtests can only take me so far. At some point the game needs data to act on. About two weeks ago I added analytics into Atma which is something I've been excited about for a long time.
I've worked with analytics in the past. In fact, one of my first ever assignments as a professional developer was integrating one of Activision's proprietary analytics libraries into Unreal Engine for the Tony Hawk's Pro Skater 1 + 2 remaster. Implementing is one thing, analyzing is another. I've taken a quick stab at querying some information and making a dashboard when I was with Disruptive, but never anything significant. I certainly never developed any real confidence in how to efficiently use analytics features.
For now, that didn't matter. I just needed to start tracking information. Learning to use it is something that can come with time. As a way to test my initial implementation, I started asking Claude to look at my analytics data and create little test reports for me. And it did a pretty good job! Well, a lot of its conclusions were...misleading to say the least. It doesn't really understand the game or what was being tested. But the data was real, the charts were fun to look at, and one report had a particularly interesting piece of information hidden in it.
The Bug
With this report I was really curious to check out some newly implemented combat stats. The bar graph tracks three player combats and shows some damage dealt and some taken. My initial impression was that it looked great. The line items underneath them seemed odd though. There were five. Maybe this is just Claude messing up with its report, but another interesting piece is that the two extra combats showed zero damage dealt and zero damage taken. Not much of a combat really. So we investigated a bit more.
Reviewing
I'll be honest after all these years of programming I'm a bit embarrassed by the exact bugs here. But this blog is about transparency in the development process, not covering these things up to make me look good.
It turns out a dying ship has about a 1.2 second delay before it is actually destroyed. This was a hack implemented by Claude to address an edge case where killshot hitsplats wouldn't have time to render on the actor, and I simply missed it by reviewing too carelessly. There's no reason to keep the ship alive after death on the server. Clients will have to find another way to properly handle their cosmetic behaviors. This alone didn't cause the zero damage combats though.
The other issue was the machinery behind the cannon component. I try to funnel most unit actions through my BaistaScript system for consistency. When units die, it cancels all existing BaistaScripts. I didn't hold to that for ship firing. Instead, the ship's controller managed its own firing logic. The UAtmaScript_FireCannon was bypassed entirely for AI ships. In effect, this issue meant an extra FireCannon call could succeed during our 1.2 second death window from the previous issue. So the simple fix here is to stay consistent and make sure AI are using the FireCannon script as well, which means it gets cleaned up when a unit dies and cancels the running scripts. Another angle to tackle this from is performing a simple IsAlive() check in the cannon firing function.
Well, normally that's exactly what would happen. Except my ships have another glaring issue. They never fired their death scripts at all. They fired a death event and triggered the death virtual function but never the death script. At the moment the death flow is split between a virtual function on the actor and a bespoke UAtmaScript_Death, and I find that confusing. I remember initially setting this up and not being able to decide on how I wanted it to behave long term. Well, we are finally at the point where that split is costing us. So death in general needs to be reviewed and hardened.
Going Forward
It's likely that all of these bugs would have been caught in one way or another eventually, but I smiled because they were caught by my new analytics. Going forward it's pretty clear that analytics has the potential to be a great bug catching tool. A player logs out with 50 extra wood and didn't pick anything up or harvest anything? Could be something to look into. Someone killed a tier 8 battleship with their tier 4 battleship? Maybe an exploit, maybe a bug, maybe the tier 8 simply didn't have any ammo and just sat there for 45 minutes. Did someone delete a character when they weren't online? These are the types of questions that can be really difficult to encounter during normal playtesting but trivially caught with proper analytics in place as Atma scales up.
The hardest part now is figuring out what questions to ask.