It’s been a fantastic week or two, for my server. A series of issues that have been stressful and absolutely no fun to deal with.
The first issue it that one of my 8TB drives is dying. It has a (now) 70k Reallocated sector count in the SMART data, and no signs of slowing down. It had some other errors that generally proceed an outright failure, but a full format has helped with those. However, it has dropped repeatedly from the system, making it unusable for anything, really. It’s also well outside of it’s warranty period.
From there, the system disk got corrupted. The disk itself is fine, works on a new installation, etc. However, prior to wiping the drive, it was getting into Windows, and then freezing. Every time. Safe mode definitely did not help. Restoring from backups did not help, though I only had a weeks worth of backups. So, a reinstall was necessary.
The only problem with reinstalling is that the system in question was a domain controller, and most of my systems were connected to it. It’s something I wasn’t looking forward to having to deal with, because it was either going to be okay (I did have a secondary domain controller) or catastrophic.
One guess which it turned out to be.
So, I took the opportunity to upgrade to Windows Server 2022, since I was on Windows Server 2012 R2, and whose supported lifecycle ends at the end of 2023. A great opportunity to upgrade to much newer technology.
I could have went with some flavor of Linux, but to be blunt, I’m not familiar enough with Linux, and most of my drives are 90+% full. Transitioning to Linux would be painful, at best. Since I wanted to get things up and running quickly, and not have to fight tooth and nail, I went with the tried and true option: the one I’m familiar with.
At this point, you’re probably expecting this to be too easy, just given the issues listed so far … and the general tone here. And you’d be right. At this point, that’s when I found out that the ReFS formatted drives that comprise my storage pool had issues with the newer version of Windows. Namely, Microsoft had some updates (Jan 2022, in fact) that seriously broke backwards compatibility with older versions ReFS. That means that every single one of my drives was showing up as “RAW”. Eventually, I did isolate this to the fact that my controller was presenting these drives as removable drives, which is not supported (ReFS on removable drives).
I could stay on the ReFS drives, at this point, but given that ReFS is ~10 years old at this point, and still having issues, I feel much safer running NTFS. At the very least, my other systems will be able to read them, and recovery would be much easier. The only problem is that I have less than 5TB of free space, and will need to juggle around the drives to get this done.
However, I do have drive(s) on the way to allow me to shuffle around the data and get all the drives reformatted to NTFS. Additionally, I have my server running Windows Server 2022, and with the Essentials Experience installed. And aside from the file systems used by the drives, it’s pretty much all completely back up and running.
The only thing that I really lost was the database for mysql, and the wordpress content. There are a few posts that are too important not to have, so you may have seen a few rapidly released, so that I have them “on file”. (Thank goodness for the Wayback Machine).