$99 Mineserver: The devil is in the details

You may recall my three sons ran a successful Kickstarter campaign last fall for their $99 Mineserver, a multiuser Minecraft server the size of a pack (not a carton) of cigarettes. On the eve of their product finally shipping here’s an update with some lessons for any complex technical project.

At the time we shot the Kickstarter video my kids already had in hand a functional prototype. Everything seen in the video was real and the boys felt that only producing custom cases really stood in the way of shipping. How wrong they were!

First we needed a mobile app to administer the server so we hired an experienced mobile developer through guru.com. His credentials were great but maybe it should have been a tip-off when, right after we made the first payment, the developer moved from Europe to India. The United States Postal Service guaranteed us it would take no more than five days for our development hardware to reach Mumbai. It took three weeks. And our USPS refund for busting the delivery guarantee has yet to appear.

We were naive. The original development estimate was exceeded in the first week and we were up to more than 8X by the time we pulled the plug. Still the developer kept trying to charge us, eventually sending the project to arbitration, which we won.

Our saving grace was we found a commercial app we could license for less than a dollar per unit. Why hadn’t we done this earlier? Well it wasn’t available for our ARM platform and didn’t work with our preferred Minecraft server, called Cuberite (formerly MC Server).

Enter, stage right, the lawyers of Minecraft developer Mojang in Sweden.

Mojang is a peculiar outfit. They are nominally owned by Microsoft yet Redmond is only very slowly starting to exert control. I’m not sure the boys and girls in Sweden even know that. Minecraft server software is free, Mojang makes its money (lots of money) from client licenses, so they and Microsoft ought to want third-party hardware like ours to be successful. But no, that’s not the case.

I’ve known people at Microsoft since the company was two years old so it was easy to reach out for support. Maybe we could do some co-marketing?

Nope, that’s against Mojang rules, Redmond told us, their eyes rolling at the same time.

We urged them to consider a hardware certification program. We’d gladly pay a small royalty to be deemed Ready-for-Minecraft.

Nope, Mojang refuses to have anything to do with hardware developers. Oh, and both our logo and font were in violation of Mojang copyrights, so change those right away, please.

Swedes are very polite but firm, much like Volvos.

Minecraft, which is written in Java, is nominally Open Source, but there are some peculiar restrictions on distributing the code. The server software can’t be distributed pre-compiled. For that matter it also can’t be distributed even as source code if the delivery vehicle is a piece of operational hardware like our Mineserver. Our box would have to ship empty then download the source and compile it for our ARM platform before the first use, making everything a lot more difficult.

Now let’s be clear, this particular restriction technically only applies to one version of the Minecraft server, usually called Vanilla -- the multiuser server distributed for free directly by Mojang. There are other Minecraft servers that, in theory, we ought to be able to ship with our little boxes except Mojang has all the developers so freaked that nobody does it. Besides, Vanilla is the official Minecraft server and some people won’t accept anything else.

But our experience shows Vanilla Minecraft isn’t very good at all. In fact it is our least favorite server, primarily because it supports only a single core on our four-core and eight-core boxes. As such Vanilla supports the lowest number of concurrent Minecraft players. A better server like Spigot can support 2-3 times as many users as Vanilla.

The best Minecraft server of all in our opinion is Cuberite, which is also the only one written in C++ instead of Java. Cuberite extracts far more performance from our hardware than any other server, which is why we chose to make it our de facto installation. We’ll also support Vanilla, Spigot and Tekkit Lite (you can switch between them), but Cuberite will be the first server to compile on the machine.

The only problem with Cuberite is that the off-the-shelf admin application we discovered doesn’t support it. Or didn’t. The very cooperative admin developer in the UK is extending his product to support Cuberite. This should be done soon and waiting for Cuberite is a major reason why we haven’t shipped. We’re hoping to have it in a few more days.

But waiting for Cuberite wasn’t our only problem. We had to develop a dynamic DNS system, Wi-Fi support, and make sure the units were totally reliable.

Oh, and our laser cutter burst into flames.

Understand that for a Mineserver or Mineserver Pro, the sysadmin also typically goes by another title -- Mom. Our administration tool allows her to control the server from any Internet-connected computer including Android and iOS mobile phones. She can bump or ban players from the frozen food aisle, monitor in-game text chat, reboot the server -- anything. It’s a very powerful and easy-to-use tool.

While we were waiting for Cuberite support we added something else for Mom to worry about, a Mumble server. Mumble is open source voice chat with very low latency. We were able to add Mumble to Mineserver because the CPU load is very low with all encoding and decoding done in the client and the server acting mainly as a VoIP switch. If she wants to, Mama can listen to the Mumble feed  and step-in if little Johnny drops an F-bomb.

Every Mineserver has its own individual name chosen by the customer. This server name, rather than an IP address, is how whitelisted players find the game. Consulting with dynamic DNS experts prior to the Kickstarter campaign this sounded easy to do with a combination of A records and SRV records. But it’s not so easy because Mom doesn’t want to have to do port forwarding, so that meant adding other techniques like UPnP, which is tough to do if it’s not turned-on in your router. We eventually developed what the boys believe is a 95 percent solution. In 95 percent of cases it should work right out of the box with the remaining five percent falling on the slim shoulders of some Cringely kid.

Every Mineserver is assembled by a specific child who is also responsible for product support. His e-mail address is right on the case and if something doesn’t work he can ssh, telnet, or VNC into the box to fix it.

Somewhere in this mix of challenges we lost our primary Linux consultant. We still don’t know what happened to him, he just stopped responding to e-mails. The next consultant really didn’t have enough time for us, but finally we found a guy with the help of our admin developer who has been doing a great job. He helped us switch our Linux server distribution with several positive results and helped come up with the custom distro we use today.

But still there were problems, specifically Wi-Fi.

Wi-Fi was something we’d rather not do at all, but it has become the new Ethernet (even Ethernet inventor Bob Metcalfe pretends Wi-Fi is Ethernet, which it isn’t but we still love Bob). Many home networks are entirely Wi-Fi. We feel the best way to use a Mineserver even in Wi-Fi-only homes is by plugging the included CAT6 cable into a router or access point port and using the router’s Wi-Fi capability. But some customers don’t want to plug anything into anything, so we’ve included native Wi-Fi support in some Mineservers and all Mineserver Pros. That sounds easier to do than it actually was.

Mineservers are headless so how do you set an SSID or password the first time? Good question, but one we finally solved. Mineservers can be configured and administered entirely without wires if needed. In most situations the customer will plug-in their Mineserver to power and it will just work. If it doesn’t, then an 11 year-old will fix it. That first power-up will involve downloading and compiling the selected server software, which can be changed at any time. It’s a process that takes 5-10 minutes and then you are up and running.

What we hope is our final technical problem has been particularly vexing. We now have three Mineservers and three Mineserver Pros running at the sonic.net data center here in Santa Rosa. All six servers plus a power strip and a gig-Ethernet switch fit on a one foot square piece of plywood. The truly great folks at Sonic gave us half a rack and we fill perhaps one percent of that, meaning you could probably put 1200 Mineservers in a full rack -- enough to support up to 60,000 players. But operating in this highly-secure facility with its ultra-clean power and unlimited bandwidth we began to notice during testing that sometimes the servers would just disappear from the net. One minute the IP would be there and the next minute it would be gone.

We’re still waiting for Cuberite support of course, but we even if we had that today we still can’t ship a product that disappears from the net. We’ve tried swapping-out boards but the problem still occurs. Maybe it was the gig-Ethernet switch, so we got a new one, then a bigger one, then an even bigger managed switch. We changed cables. We started fiddling with the software. Each Mineserver board has a serial port so we converted an old Mac Mini to Linux, added a powered USB hub and six UART-to-USB adapters so now our consultant in Texas can use six virtual serial terminals to monitor the test Mineservers 24/7 without having to rely on their Ethernet connections. Everything is being logged so the next time one goes down we’ll know exactly what’s happening.

We’re also in touch with other users of the same board like Lockheed Martin and Lawrence Livermore Lab where they have a cluster of 160. But that’s nothing compared to three kids in Santa Rosa who are right now burning-in 500 boards.

It’s the final bug, we’re approaching it with planning, gusto, and plenty of Captain Crunch, and fully expect to solve this last issue and start shipping next week when the kids are off school for Spring Break.

Mineservers as a business so far aren’t quite as good as the boys had hoped. The Kickstarter units are losing an average of $15 each (so far). But $7500 in the hole is not much cost to start a technology business. And with their marketing strategy (called "F-ing brilliant" by a VC friend) the boys are hoping to sell 100+ post-Kickstarter units per month to eventually pay for college.