Currently the SOTAwatch spots (@SOTAwatch) Twitter feed does not appear to be working.
The last 10 spots on The Sotawatch2 web site have not appeared via the twitter feed.
Dave
(M6RUG)
As of 09:56 spots are beginning to appear again!
Dave
(M6RUG)
1 Like
Sorry about the outage. I host this facility at work. It runs on (at least) two servers, which keep an eye on each other so that a standby can take over if the main one fails. The servers run on virtual machines on a pool of real machines, all of which have dual network paths to a high availability storage server. It’s designed to survive failure of any component (not entirely for the benefit of @SOTAwatch, you understand).
This morning I was doing some planned maintenance to upgrade the firmware on a couple of the network switches. This is all supposed to work with little or no disruption by upgrading the two redundant halves in turn, leaving the other carrying the load during the reboots. It didn’t work. I ended up causing some network outages; not in itself a problem because I had advertised an “at risk” period.
Unfortunately our virtual machine infrastructure really hates network outages. If the hosting machines lose connectivity with their backing store, bad things happen. We’re aware of one particular bad thing and I had somebody on standby to deal with that, but for some reason not yet understood it was all much worse this time. I won’t go into details but it took our guru several hours to piece things back together. At least @SOTAwatch started working of its own accord once the VM world was back in business.
What’s worse is that I only actually succeeded in upgrading one of the two switches I needed to do.
I think I’m getting too old for this lark. I can really sympathise when the “big boys” suffer outages. Their systems are even more complicated than mine, and however much redundancy you design in there’s always something that can go horribly wrong.
Martyn M1MAJ
1 Like
No problem Martyn
I use both the web site and the twitter feed so no hassle at ths end but I did see that they were out of synch.
Firmware updates on switches! - been there done that, had the network outages and the complaining customers!.
Thank god I don’t do it any more 
73
Dave
(M6RUG)