Non-motoring > Server maintenance today from midday	Miscellaneous
Thread Author: car4play	Replies: 25

Wed 16 Nov 11 10:49   Server maintenance today from midday - car4play

Today at around just after midday we are planning on moving our remaining servers from London to Sheffield. If all goes well this will involve downtime of around 5 to 10 minutes when we switch over to mirror servers already at that location. Last edited by: car4play on Wed 16 Nov 11 at 10:50 Reply to this message | Report message

Wed 16 Nov 11 10:53   Server maintenance today from midday - Zero

Ok, here is my fiver, I say it will be a two hour outage Last edited by: Zero on Wed 16 Nov 11 at 10:53 Reply to this message | Report message

Wed 16 Nov 11 10:54   Server maintenance today from midday - car4play

"always look on the bright side of life" - can you hear the tune? ;-) Reply to this message | Report message

Wed 16 Nov 11 11:09   Server maintenance today from midday - Zero

Indeed, I always can, just based on past experience tho...;) Reply to this message | Report message

Wed 16 Nov 11 11:10   Server maintenance today from midday - Iffy

How exciting, I shall delay my midday rehab hobble to monitor the situation. Reply to this message | Report message

Wed 16 Nov 11 11:15   Server maintenance today from midday - teabelly

If they're based up north will whippets become essential? I'm with zero. I suspect it will be longer than 5 to 10 minutes... 42 is my bet :) Reply to this message | Report message

Wed 16 Nov 11 11:18   Server maintenance today from midday - Robin O'Reliant

>> 42 is my bet :) >> Isn't that the answer to life and all that? Reply to this message | Report message

Wed 16 Nov 11 13:32   Server maintenance today from midday - Zero

And 42 it was! Reply to this message | Report message

Wed 16 Nov 11 13:45   Server maintenance today from midday - car4play

Yeah - you were right - always takes longer than one imagines. During shutdown of the Virtual machines (VMs) in London, the host computer froze. So we carriend on moving the IP block to Sheffield, at which point the London machine helpfully unfroze. The problem then is that we then had to move the IP block back to London, shutdown the remaining VMs there, reboot both servers to get them to sync correctly with all VMs powered off, then move the IP block back to Sheffield and start the VMs up there. Trying to shortcut seemed too dangerous as we would have had an out-of-sync VM. Sure it would have started ok, but I didn't want it to have potentially more issues than we needed. .. and all that shutting down, editing and restarting took (according to the logs) from 13:10 to 13:28 (18 mins) - give or take a few mins for services to power up. Reply to this message | Report message

Wed 16 Nov 11 14:10   Server maintenance today from midday - Zero

>> Yeah - you were right - always takes longer than one imagines. You are IT guys (and this is no insult - I am an IT guy) and IT guys always stick to the technical facts. And there is always a technical factual reason why stuff does not take the time we thought, and we always wonder why people cant understand the facts and appreciate it. Heres what I learned, the under promise and over deliver technique. Example - Change Meeting Wednesday 16th November 2011, Khoo Labs Inc. Tech guy, (knows it will take 10 minutes - adds on 100% contingency and 10 mins buffer) "well we need to do Blaaaah, and we need to invoke Blahhhhh 2, and Blahhhhhhh 3 needs to be synced, so its 30 minutes outage" Service Manager EXPLODES "30 Minutes!! you cant have 30 minutes, the users need the system, its costs 17 gazzilions an hour for an outage like this, cant be done" Tech guy" well If we dont, all user data will be lost in 24 hours (knows it wont) " "But for you we can cut that down to 20 minutes." Service Manager "SIGH - Ok I will tell the users" Service manager to Users "There will be an outage of 40 minutes", (he has dealt with the tech department before!!!) 18 minutes later System back Users delighted, Service manager delighted, IT guys loaded with kudos., Last edited by: Zero on Wed 16 Nov 11 at 14:12 Reply to this message | Report message

Wed 16 Nov 11 16:07   Server maintenance today from midday - ....

>> Example - Change Meeting Wednesday 16th November 2011, Khoo Labs Inc. >> >> Tech guy, (knows it will take 10 minutes - adds on 100% contingency and 10 >> mins buffer) "well we need to do Blaaaah, and we need to invoke Blahhhhh 2, >> and Blahhhhhhh 3 needs to be synced, so its 30 minutes outage" >> ... >> 18 minutes later System back >> >> Users delighted, Service manager delighted, IT guys loaded with kudos., >> Depends on the industry these days. Most will now moan why you over estimated in the first place. If they wanted rough guestimates they would have outsourced. Reply to this message | Report message

Wed 16 Nov 11 16:15   Server maintenance today from midday - Zero

I was speaking as an outsourcer! Reply to this message | Report message

Wed 16 Nov 11 11:19   Server maintenance today from midday - car4play

Now you're all making me nervous. Reply to this message | Report message

Wed 16 Nov 11 11:21   Server maintenance today from midday - Zero

My day has not been wasted then. Reply to this message | Report message

Wed 16 Nov 11 11:53   Server maintenance today from midday - Ted

Yeah, dream on, London to Sheffield in ' 5 to 10 minutes '......not even the RAF can do that ! Just get a slower train and chill out :-) Ted Reply to this message | Report message       2

Wed 16 Nov 11 12:18   Server maintenance today from midday - car4play

The cluster is split between London and Sheffield already, so moving means powering down the virtual machines in London and starting them up on the existing cluster in Sheffield and then moving the IP block from London to Sheffield so that it is listening and responding there instead of London. We then will lose all contact with London. Someone there will physically power them down and move them in his car to Sheffield. Oh the joys of virtualisation. Reply to this message | Report message

Wed 16 Nov 11 12:00   Server maintenance today from midday - Iffy

...If they're based up north will whippets become essential?... I saw a traffic report on BBC News this morning - jams at 6.30am. To think people pay a premium to live like that. Madness. Reply to this message | Report message       1

Wed 16 Nov 11 12:09   Server maintenance today from midday - rtj70

So has it moved yet? Still working but that might be because nothing has happened yet. Last edited by: rtj70 on Wed 16 Nov 11 at 12:09 Reply to this message | Report message

Wed 16 Nov 11 12:19   Server maintenance today from midday - car4play

The guys in London are running late. Traffic and all that. Reply to this message | Report message

Wed 16 Nov 11 12:43   Server maintenance today from midday - car4play

Ok - going down now. Fingers crossed Reply to this message | Report message

Wed 16 Nov 11 13:41   Server maintenance today from midday - VxFan

Explains why I couldn't get on earlier. Not much help if one didn't see the message prior to the site being temporarily switched off - lol. Reply to this message | Report message

Wed 16 Nov 11 13:49   Server maintenance today from midday - Roger.

There's nowt wrang wi Sheffield! (Except Meadowhall Shopping Centre) ***Shudder!*** Reply to this message | Report message

Wed 16 Nov 11 14:02   Server maintenance today from midday - madf

>> There's nowt wrang wi Sheffield! That a small tactical nuclear weapon could not cure. Reply to this message | Report message

Wed 16 Nov 11 16:11   Server maintenance today from midday - R.P.

That a small tactical nuclear weapon could not cure. Did you see Threads ? Now available on DVD ! Reply to this message | Report message

Wed 16 Nov 11 13:50   Server maintenance today from midday - Pat

Neither could I and neither did I! Pat Reply to this message | Report message

Wed 16 Nov 11 14:58   Server maintenance today from midday - Armel Coussine

Ah, so that's all it was. I am always nervous when I haven't had my coffee yet. For a moment there I fancied civilization had collapsed. I can put the weapons away then. What a nuisance. Reply to this message | Report message