A Day in the Life of Rhialto

Fantastic insight into running, developing, and managing a huge game like Wyvern from Rhialto himself:

I thought I’d share what it’s like to be in charge of Wyvern, to give you a sense of the scope of what I’m doing every day.

I’m responsible for literally everything, but most of you probably don’t realize how much work it is. So here is a rundown of some of the things I need to spend my time on.

Game Server Code

I wrote almost all of, and maintain all of, Wyvern’s large code base. It is about half a million lines of code: Mostly Java (75%), maybe 12% Python, and the rest misc (kotlin, perl, ruby). I have a long-running project to convert all the python to kotlin, and spend a few hours a month on it. Kotlin is faster, safer, and easier to debug than Python, so it is worth spending the effort.

Half a million lines of code is a lot for a single person. The code base is large enough to require dedicated time for maintenance and cleanup, even though all that is (hopefully) invisible to the players.

Game Content

Wyvern has a lot of content. We have about 20,000 unique, hand-authored objects (monsters, structures, spells, etc.) and about 10,000 game maps. Only about 4000 of the game maps are reachable, as far as I can tell. But it is tricky to know for sure, because the wizards did a ton of copy and paste, both in general, and also specifically for randomizing quests. It is a horrible mess and I will have to spend years cleaning it up. But I suspect we actually have around 3200 “real” maps (maps that are not slightly-different copies of other maps).

Wyvern also has about 30,000 image files. I have actually been working very hard to reduce the amount of content, particularly unused content from wizards whose areas are no longer linked into the game. The less content I have to deal with, the better, because it makes it easier to make big changes and fixes. It sucks when I have to fix a broken teleporter, and I discover the teleporter was copied into 10 different copies of the same map, so I have to fix it 10 times. Really sucks. Copy and paste sucks. I’ll leave it at that.

So I spend a lot of time maintaining and curating the content — most of it trying to undo all the copy and paste without breaking anything. But stuff does sometimes break.

The massive pile of game content is also part of why I prefer redoing existing areas instead of adding new ones. Adding new areas just piles stuff on, creating more of a maintenance headache. Whereas redoing areas allows me to dig in and clean them up while I’m at it. For instance, I un-randomized the Mist Temple when I turned it into a Mission. That caused some bugs that have since been cleaned up, but it was worth it, because it eliminated something like 8 virtually identical copies of the mazes beneath the temple.

I would love to add back in player-contributed content, but I think by the time you’ve finished reading this, you’ll realize that it’s going to create even more work for me — I just don’t have time to deal with it right now, even if all the tools existed (which they don’t, yet.)

Game Clients

The game has 4 clients right now, though two of them are for Rhialto’s private use only. The clients are ways to connect to (and play) the game. There is the old Java client, which has not been kept up to date with all the new features (e.g., missions). So it’s sort of broken, but I do use it when I can.

And there’s also my trusty old telnet client, which lets me connect in text-only mode and chat. I might open a text-only version of Wyvern up to people in the future.

The other two clients are iOS and Android, both of which were a ton of work. And I’ll be adding a PC client, starting work on it later this year. I do not know yet what language the PC client will be written in. The choice is a trade-off between lots of things: learning curve, cross-platform capability (does it only run on Windows, or can it also run on Mac and Linux?), performance, how fast I can get it finished, and so on.

Maintaining all the clients myself is a lot of work, especially since they are all written in different languages. (The Android one is in Kotlin, and the telnet one is in python right now, so they are all in different languages.)

When you have a lot of clients, then every time you add a new client-side feature, you have to add it to every client. The work is multiplied by the number of clients, at least if you want to keep them all at feature parity. Right now the iOS client is pretty far behind the Android client in terms of features, so it needs work just to catch up.

The iOS and Android clients also have the extra complexity of having to deal with digital signatures and certificates for the Apple Store and Google Play Store, as well as all the other cruft required (like the Apple review process) in order to get anything pushed out.

The clients take up a lot of my time.

Storage

The game has to remember all the changes to your players and your homes. Wyvern has a MySQL database that requires constant maintenance — index updates, new tables, backups, quotas, etc.

The game also logs a great many actions to permanent storage. I have not had much time to do anything with the logs, but it’s a rich data set which I can someday use for trophies/achievements, game balance, etc.

Storage is an area that is just scraping by right now, but it’s going to become a bigger problem as the player base grows.

Scaling

The game is designed to scale up when we have lots of players. It runs on a single server today because we can, but when it gets more players, there will only be about 100 per server.

I wrote a lot of game code dedicated to the glue that allows Wyvern to run in the cloud and scale up and down. Unfortunately it is nowhere near enough. The approaches that I took were easy, but expensive. I get a big fat bill for cloud computing costs each month. The game is just barely covering its expenses right now, since we also have contract artists cranking out new artwork to support new features and area remodels. But at least it is not losing money any more.

It will take me a lot of work to cut the costs down. It’s possible, by reworking most of my cloud infrastructure. But it’s going to be a lot of work. And it will be easier when there are fewer players, so I should probably do some of that work this year. It’s a bit stressful, thinking about all that work to do, since none of it really makes the game advance in any way. It’s just about survival.

Customer Service

It might sound silly, but I spend a lot of my time on customer service for the players. Some of the services I provide today include:

• I still help people look up their old characters, and migrate them to their new email addresses.
• I add people to the Android alpha, and will keep doing that until it launches.
• I make gallery portraits for demigods, by hand, using GIMP. I am clearly no artist, but I do my best.
• I receive and process HoF images from new HoFers. This is a lot of manual work that I should automate someday.
• I handle failed crown transactions, when they happen, and do refunds, etc.
• Almost every day I help new players who seem like they could use a hand.
• I spend a ton of time answering questions on reddit. I don’t get them all, but I try.
• I manage the bug list and process bug reports from players in-game, in email, and on reddit.
• I contribute to the Wiki, when I can.
• I send game artwork to various people who want it, although this is starting to be too time-consuming.

Customer service takes up a lot of my time — time that would probably be better spent gearing up the game for 10x or 100x as many players. But I care about the player base, so I do my best, as long as people don’t abuse it.

I think I’ve done a pretty good job of not taking the easy way out and just banishing people who are abusive, entitled, or toxic. I can appreciate why the Wizards did it before, since it’s tempting. But I view you as customers, so I only banish people who have bad names or have been seriouslyabusive (getting around chat filters, etc.)

Tooling

You can’t do customer service without having the tools for it.

It’s all behind the scenes right now, but Wyvern has tons of tools. I have the Map Editor, of course, which needs a lot more work. But I also have tons of scripts and programs that I use for everything from fixing broken financial transactions, to tweaking player properties for every single player, to doing email migrations, reporting and metrics, and much more.

Wyvern probably has 100 or so active tools that I use every day, and I’m adding new ones constantly. It’s sort of a pain, but it’s also a necessary part of running a real production system.

Billing

NagaLinh and I spend a lot of money on the game, and we have to track the finances carefully. We record everything. It’s all done under an official company (Ghost Track Inc), and we have banks and stuff that are all being very nice even though we are not making a significant amount of money. But we have separate tax filings, official paperwork, corporate notes and other stuff that takes up time — time that I’d rather spend working on the game, but it’s unavoidable.

Legal

I spent most of the last 3 years working on legal issues. It has cost us at least $150k, maybe more (I’d have to look), and we’re always spending more: copyrights, trademark and patent filings, contracts, agreements, all sorts of stuff. It’s very expensive and has historically been very time-consuming, though the situation is improving somewhat.

On the bright side, the game is very well protected now.

Updates/Patches

I have a constant trickle of new features and bug fixes rolling out. They fall into several categories:

• Major new additions, such as the Android client and the Missions system.
• Content remodels. Many of our old areas were pretty ugly, but we know we can do better. So I’ve remodeled 6 cities so far — Vesimas, Stensele, Minath, New Verden, Fae Wyston and Alaria. It’s surface-only right now (not the building interiors), but once the cities are done I’ll start on the internal areas.
• Bug fixes. I have a bug list a mile long, and it’s pretty much always growing. But I fix as many as I can while I’m working, and occasionally take days off to just fix bugs.
• Quality of Life (QoL) fixes. Lots of bugs and some feature requests. I prioritize these roughly by how often people complain about them. I try to do a couple of QoL pushes a month.
• New content, such as adding new monsters to the game (silver dragons, onryos, etc.), or adding new RDs. Over the past few months I’ve added a lot — scroll tubes, for instance: things you don’t think about, but stuff’s getting added all the time.
• Medium new features, such as new spells (e.g. the Detect Monsters spell), or the ability to configure button-bar buttons with a single press. I actually do a lot of new features, a few every week, though I’ve not been good about patch notes lately.
• Game Balance — I frequently tweak guilds, races, and items to try to nudge everything to be a little closer. We’ve made a ton of progress on this, but there’s lots more to do.

The Major Additions are the most important, prioritization-wise. The game won’t grow unless I can get these things launched. Other upcoming Major Additions include international translations (Korean, Spanish, Japanese, etc.) and the PC Client.

However, all the others are important just to keep the current players coming back. So I have to strike a balance, as best I can.

Game Presence

I try to log into the game several times a day, and spend as much time in-game as I possibly can.

This is in part to show everyone that it’s not abandoned. I work very hard on it, but if I’m not around, then it can feel like there’s nothing going on.

It’s also because I like hanging out with a lot of you, and the shout conversations are often hilarious.

And it’s of course also because people often have glitches or issues with their characters that I need to fix by hand.

Putting It All Together

I spend a lot of time on Wyvern, and it’s not even my day job, so basically I’m working almost all the time. I have to be careful to have a little bit of work-life balance, so NagaLinh and I will watch a movie or tv show every now and then. She works really hard too, doing tons of playtesting and also taking care of Rhialto so I can focus.

I would love nothing more than to be able to do this full-time. I honestly don’t know how long it’s going to take before the game gets there — it might be years. I learned while working for Jeff Bezos that if you know you have a good idea, and it’s not super successful at first, then you keep hammering at it and changing it until it takes off. I have a lot of tricks up my sleeve to get Wyvern to take off, but it will take a couple of years to implement them all.

In the meantime, I want to thank you all for playing the game and being so patient as I work through all the stuff we want done. I get stressed out sometimes, since many of you tend to forget (or never knew to begin with) how much work this is for me. But I try to be patient, because I know you love the game like I do.

Anyway, I gotta get back to work, but this was fun. I hope you have a better picture of how the Wyvern sausage is made, so to speak.