First Closed Alpha Complete: Stats and Lessons Learned

by Director

The first Closed Alpha test was held this last weekend, and I think it went very well! There server was incredibly stable; we only had a couple of "crashes" over the course of the 48 hour period, and both of those were bugs that were easily fixed. I think at this point we could leave the server up for long periods of time with very minimal maintenance which is a great step toward the release of the game.

Here are a few fun stats that we pulled after the Alpha test ended:

  • Zardulu was the player with the most time played; they played an incredible 22.5 hours during the 48 hour testing period. (Did you even sleep?!)
  • 189 people participated in the test, playing a total of 708 hours. That means the average time played per person was 3.5 hours. That's actually a lot higher than I expected!
  • We have a total Alpha testing "pool" of around 250 people right now, which means that about 75% of everyone who could have played did so. I think that's probably a little lower than I expected, but I assume that no matter when we do the test there will be people with prior commitments.
  • 46 people played more than 6 hours during the test, which I would consider "a lot."
  • There were 8 people who reached the max level (level eight) as a Warrior, 15 max-level Rangers, 16 max-level Musketeers, and 17 max-level Runemages. I'm not sure if that's more an indicator of popularity or an indicator of how hard it is to level up a Warrior currently :-)

Lessons Learned

There are a few things that we (as the Dev Team) learned from the test that I think are worth noting.

No matter how long the test is, the peak is right after it starts.

In our previous 10-hour stress tests, we always saw a peak of players in the few hours after the test started. With a 48-hour test, I assumed we would see a peak when the test started, then peaks during the evening hours in the US. I expected that we would have about as many people play on Friday as we did on Thursday, and maybe even the most people on Friday night since it was before the weekend.

In reality, we had by far the highest peak concurrent count right after the test started, with around 50 people online. After that it dropped off, and even during the "peak" evening hours we never got back above 25 people online at once. One of the players, Damage, actually put together a handy spreadsheet showing the concurrent count by hour that you can see here.

I think this is worth noting because there is a "critical point" at which I think the game starts to really feel empty, and that's when there's less than around 15 people online. On the first day we had no problem having double that number online pretty much the whole time, while on the second day we hovered right around that point, even dipping below it sometimes. To me, that says we should continue to do targeted tests for the Alpha, and probably that 24 hour tests are better than 48. Not only does that cut less into actual development time (since pretty much no work gets done on the game while there is a test going on), but it will concentrate the small player base together more so that everyone has a better experience where they can actually find people to group with. Lots of people noted that the game is much better in a group than solo, and honestly that's kind of the point of an MMO, so while we will do some more tuning to make sure that as a solo player you can level up more easily when you're Level 1, at the same time I think it's smart for us to do as much testing as possible at concurrent numbers that are reflective of what the game will have most of the time once it's live.

So as of right now I'm planning on the next Alpha test being May 5th, 2017 and being 24 hours long instead of 48.

Don't split the player base (yet)

Another note along the same lines, we probably should have put the PvP Arena out right away, rather than on the second day. Our original thought process was that by the second day there would be lots of people who had played to max level on the classes and would be wanting something to do. However, in reality on the second day the player count was so much lower it made getting queues to pop for the PvP Arena a little difficult, and introducing that also took people out of the shared world which made the game feel even emptier.

So while the PvP Arena was a lot of fun, I think in the future we'll either enable it right from the beginning, or not enable it at all. Right now my plan is to probably alternate it so that the next test it's not enabled, then the one after that it is, etc.

Patch sizes must come down

We introduced a new launcher and patching system for this test, and while it worked great unfortunately we didn't realize that our builds created by the game engine (Unity) were not going to work as well as we thought. The patcher is smart enough to only download files that have changed since the last release, but Unity does not create the same files if you do two builds even if nothing has changed in one key area. There is a feature called "static batching" this is very helpful for getting higher performance out of objects that don't move, and Unity does this by pre-generating "combined" versions of the objects when you build the game. However, this build generates a different file every time you do a build, even if the assets haven't been altered or moved in the level. About 1 GB of the 2GB total filesize of the game is due to these level files, and so every time we do a new build there's 1GB of stuff that's "changed" even when nothing has really been changed by us.

In the short-term, we are simply going to turn off the Static Batching feature. That will result in the total download size of the game dropped dramatically (from 2GB to about 1.1GB), and it should make it so that patches will only be < 100MB in most cases. We'll see what type of performance hit that causes, but in my initial testing the game is still running fine.

This community is awesome

The last takeaway for us as Dev Team is simply that the community that is building up around the game so far has been awesome. You all have done a great job of reporting issues, working with us to track down bugs, and offering feedback and suggestions that will help make the game better. At the same time, when we've had to say "We can't do this" or "that doesn't work with our vision from the game," you've taken that in stride. Finally, you've been amazingly helpful to each other in-game, teaching people spells, grouping together to take down monsters and bosses, and just generally making it a super fun place to hang out and have a good time. I really hope that "vibe" continues forward into the full game, because if so this is going to be a place that everyone with a VR headset wants to come and spend time.

What's Next?

So as I already mentioned, the next test will be May 5th, 2017 and last 24 hours. We will start it at 12 PM (Noon) US Central Time again.

Although I'm not going to give too many specifics right now since what gets done in the next two weeks may change, our focus is on:

  • Fixing game-breaking bugs from the last test, (e.g. Warrior sword throwing issues, menu issues, etc.)
  • Re-doing how XP awarding works so that we can better balance it between solo players and those in groups
  • Re-doing the Runemage runes and introducing new abilities to the Runemage class
  • Introducing new death mechanics including repair costs for gear and the "tombstone" as well as the ability to resurrect other players

We'll release more information about those things here on the blog as we get started on them. Until next time!