Earlier in January, MMORPG
As a fringe benefit there’s also a boost to performance, but more on that later.
The outage happened because the server that MMORPG.Social runs on ran out of disk space. Over the course of one month, we’d managed to accumulate nearly 40GB of images – including 25GB from other instances in the federated network. That’s not just user uploads but also avatars, profile header images, and other graphics that feature in the timelines.
Without monitoring in place to alert me when space runs out, it’d just fill up until the whole thing stops working. Which is what happened.
My immediate fix was simple – purge any images from other servers that were older than seven days. Our own toots on MMORPG
But it’s only a temporary fix. As MMORPG.Social gains more members, who share more of their amazing artwork, screenshots, and in-game photography, I needed a permanent solution that could provide storage that would scale as we grow. This is where Amazon’s AWS S3(Simple Storage Service) comes in.
As a result, the MMORPG.Social ecosystem has grown:
- MMORPG.Social – the core platform based on Mastodon (now running v2.7)
- lite.mmorpg.social – the simplified single-column interface based on Halcyon
- blog.mmorpg.social – this news/status blog, based on WordPress
- images.mmorpg.social – the image repository on AWS S3 & Cloudfront
- mail.mmorpg.social – email delivery performed by Mailgun
(The Cloudfront component acts as a
I’ve also updated the underlying software that powers MMORPG
With all these changes now in place, we should have enough capacity to scale up from the hundreds of members we have now to potentially thousands. I also have monitoring in place to avoid preventable incidents in the future. While I can’t rule out further outages, they should now be a much rarer thing.