FDCPS Severe Outrages thread

[AMDave]

There will be a planned outage for several hours tomorrow while electrical work is carried out.

[AMDave]

Planned outage concluded without incident.
Updates and patches applied before restart.
All good.

[AMDave]

There will be a planned FDCPS outage for further mains electrical work to be carried out.
Planned start is 12 hours from now.
Planned end is 17 hours from now.

[AMDave]

FDCPS planned outage completed on schedule.

[AMDave]

Full DR plan execution and tests for FDCPS completed successfully at 2012-05-20.
Because DR procedures are not 'good' unless you keep verifying that they work.

The FDCPS server PSU fan bearings made some bad noise while at max RPMs during a heat spike about a week ago for a few hours, so it was time to check on a few things.
I made a few updates to the rebuild document during the DR test, but there is one more item outstanding, so I'll commit that back into the project tomorrow.
The project server runs well in a KVM cluster so I'm planning to migrate it to a KVM guest back on the same hardware soon-ish to get all the benefits that brings.
I also verified that the implementation works with Nginx . (very snappy btw)
I'll add that config to the doco for the commit tomorrow.

It went so well I am still looking for the "Gotcha!"

[AMDave]

The FDCPS site and ports may be unavailable for a few minutes in 3 hours time [16:30pm AEST and 06:30 UTC] while I migrate it to another machine to allow the server maintenance and OS upgrade.

[AMDave]

...It went so well I am still looking for the "Gotcha!"

FOUND IT!

That went badly.
The network bridge to the VM version of the server failed.
The version to version upgrade of the OS on the server box failed because the server network configuration failed between 10.10 and 11.10 so it lost all network connectivity and failed to boot, meaning that the next OS version upgrade was a dead-stick.

I am now doing a clean install on the server box: ubuntu-12.04-alternate-amd64

Having tested the DR plans I have the tried and tested recovery path so it's just a matter of time.
Once the OS is running I'll have to rebuild the databases, web and mail servers etc.
Not confident of getting it all done tonight, but it's on the way.

In spite of copious testing, sometimes things that can go wrong still do go wrong.

Damn you to the infernal depths of hell, Murphy!