Flav wrote on Oct 8
th, 2013 at 1:43pm:
Been, there done that, still got the nightmares of having to rebuild a database from an obsolete backup after a server crash.
( FC-AL controller went wonky and wrote fucking trash on the hard disks... Customer had no relevant backup, we restarted from a 1 year and a half old backup we made on the sly : took us 3 days non stop in 8 hours shifts to recover the server to a point where we could actually start to consider trying to recover the data from the telecom switches, then it took us 2 month to recover and reenter all the missing data... and we still missed some... Trust me I'm quite glad I work in telecom and only have to deal with phone numbers, having to do the thing with another kind of data would have been... a Challenge ).
Friends in telcom software, which from the sounds of the data would make me quit software rather than subject myself to learning comm protocols. I stay near the top of the software moutain, not in the weeds, by choice.

About a decade or so ago we had an IBM upgrade expose a software flaw that started a frenetic spawn of extraneous financial transactions in our systems. It took us three days determine how to fix the issue without making it worse, and was my first deep dive into the arcane gray areas around isolation locking. We didn't even have the option to rollback because of the mix of valid and invalid actions taken during the 3 days, and it took us 9 months to clean the corruption out of our financial system with the appropriate paper trail, which gets audited heavily.
In a true rollback scenario I had a boss use an old system backdoor incorrectly to clear some data, and we found out the fewer parameters supplied, er none, the backdoor just wiped an entire table clear of data. A panicked phone ensued, we shutdown the system and the systems group went to grab the backup and found out the last one was like two weeks old.

At least no one died or was injured, and that can be a real possibility in my line of work, but many "hot seat" meetings ensued.
ETA: The second scenario gave rise to the only true statement you'll ever give a customer when they ask when the system will be fixed. "We're closer than we've ever been!"