Fifthdread had an Oopsie!

My Server PC had a C Drive which was only 180GB in size… It was full. That’s when I decided to upgrade to a 2TB drive. It’d be a simple procedure. Clone the old drive onto the new drive, GG.

Nope. I made a small but HUGE mistake.

The TLDR of it all is… I accidentally cloned my old C drive onto the WRONG Samsung 2TB SSD. That’s right. I copied my C drive onto an existing 2TB Samsung SSD instead of the NEW 2TB Samsung SSD. Simple mistake to make really, but a preventable mistake. They weren’t the exact same type of drive, so I should have seen it. Never the less, I wasn’t thinking, and the result? Data Loss.

How much data loss? Pretty much all my HyperV VMs GONE! That means Element and a bunch of other servers.

Fifthdread Services – The Perfect Utopia?

Fifthdread Services may seem perfect, but you’re only seeing the surface layer. A perfected exterior. Services work, things are happening. Nothing is wrong, and uptime is 99% or higher. It’s a beautiful world, so it’s safe to assume that underneath is just as beautiful, right?

Not so fast.

Fact is, Fifthdread Services has been sort of a cobbled together mess. It’s been pieced together piece by piece over many years, and it wasn’t really designed to scale. I’ve run into many issues along the way, and have band-aid fixes all over. I have to renew my SSL certificates manually every 3 months, so sometimes we get surprise outages because they expire without me catching it. Most things ran on a single Windows 10 box, which for production servers is a big no-no especially because Windows 10 needs to be rebooted to update.

Fifthdread Services is more like Factorio Spaghetti

The reality is, Fifthdread Services has become more like a Factorio first-play-through. You start off small, learning as you go, and eventually grow to the point where you have a big mess. You probably should redo everything, but you already put in so much work that it’s just a lot of effort. “I’ll do it later”, you tell yourself. Maybe you finish your play-though and start again. In my case, my server died. So now, it’s time for the second Factorio play-through.

A second play-though of Factorio looks a lot cleaner. You probably looked online and learned the optimal way to organize things. You picked up a few techniques from those who came before. You learned lessons from last time and know what to avoid this time. It’s just going to be that much better.

That’s what is happening to Fifthdread Services right now.

Truly, I am sorry for the inconvenience. It sucks having to lose data, and re-create accounts. I’m going to do things much better now, given the opportunity. First off, I will have a proper Disaster Recovery Plan, and a Continuity of Operations Plan. These will outline how I will keep our services running, along with how I will recover from various disaster scenarios, much like the one that happened this weekend. With this in mind, I should be able to recover without data loss if this happens again.

However, this was a blessing in disguise. Now, I can just get it over with. Redo EVERYTHING from scratch, clean, and better than ever. Using all the lessons learned from the past, and with the help of my buddy Allyn. lol

The infrastructure on the back-end looks much different now, and is super interesting.

Hello World – Back Online!

Now that everything was lost, it was time to do it over. I’ve already done most of the work and WOW- it’s SO much cleaner. Things like SSL certificates are automated now, updating things will be easier, and overall management of 5DS will be faster and easier than ever. It’s been a long time coming, but I’m finally at the point where I can be proud of this setup. We are finally away from the Factorio Spaghetti mess of the past, and into the organized and clean future. It won’t ever be a perfect utopia, but I think it’ll be somewhere nice in between.

There’s a lot more work to be done, but I thought I would take a break and write a blog post. Hopefully this will be a hard lesson learned, and I can do things a little smarter this time.