[adelie-devel] Re: x86_64 builder - catastrophic disk failure

Hi, The x86_64 builder (ciall.foxkit.us) has had a catastrophic disk failure at around 23:44 CST 18 February 2018. During attempted recovery, there was a power failure in the Tulsa metro at 04:14:47 CST 19 February 2018 that lasted over two hours; regrettably, much longer than the 45 minute backup power runtime. Due to these events occuring so close together, the entire /srv (package staging area), /usr/src (aports git trees that had a few uncommitted patches), and buildroot / (where abuild configuration lived) trees are completely unrecoverable. The /opt/virtual tree, where the Adélie Portability Virtual Machines lived (including NetBSD-based bryan.foxkit.us, OpenBSD-based georgie.foxkit.us, and Arch-based fran.foxkit.us), may be recoverable - it is not yet known at this time. Anyone who had any data on bryan, georgie, or fran should consider that data gone barring an exceptional recovery effort. As their primary purpose was for portability testing only, no backups were ever taken of their virtual disks. The /home tree on ciall was in the process of its monthly back up at the time of the disk failure and it appears only a few files (from my own ~) were lost. No other user data was lost. As x86_64 is a tier 1 architecture, and the Platform Team is not currently considering deprecating it, this unexpected, catastrophic, poorly-timed sequence of failures may cause the alpha5 release date to slip again. This is regretful and unfortunate. Moving forward, we will need to discuss if architecture buildroots and configurations should be stored in Git; if a Git repository isn't deemed a proper solution, they will at least be added to the monthly back up jobs.