DEFCON 4 (it can be sleep tiems now plz?)

[denny]

We hit DEFCON 1 at work at 4pm this afternoon, when our previously well-behaved (well, okay, but it mostly works) web application suddenly ate all the RAM and swap space on one server, locking the entire machine solid. A few emergency reboots proved that this wasn't an isolated incident, but got us no closer to finding the cause.

At 7pm one of my team headed home, leaving two of us still trying to figure out wtf was going on. At 9pm I rang the MD and told him that we couldn't find the problem, and we were going home. He pointed out rather firmly that we (or at least I) wasn't allowed to leave all of our websites offline for the entire weekend. Fair point, although it took me a while to accept it, given that I had commitments for this weekend that I really didn't want to back out of.

The MD wanted me to go home, rest, and come back in the morning, but we all know when coders are at their best... so while the other dev headed home, I settled in for a long night. At 10pm, the MD turned up at the office to keep me company, which was nice of him. He did offer to make the tea, but I got him to pay for my pizza instead :)

I just got home a few minutes ago, having finally isolated the one missing semi-colon that was causing the whole thing at around half past midnight, and restoring it to its rightful place with suitable before and after testing.

Not the most promising start to the weekend. Tomorrow I get to go to MK to try to figure out what the hell I'm doing with the huge drumkit and various other crap that's clagging up my mate's garage... something I'm really not looking forward to, but then that's exactly why I didn't want to back out of it.

G'night.

Comments

[robinbloke.livejournal.com]

Well; the MD should be happy that you've fixed it! Have a good weekend!

[adamw]

Jesus. That sounds like a bad night. (And it's always something almost insignificant that causes it, too!)

Good to meet you earlier in the week, btw.

[kelemvor.livejournal.com]

Ouch.
I've hit something similar (in the same what that the grass eaten by a sheep is similar to bamboo) - a superfluous comma breaking an SQL statement. Eight hours? *Offers hugs and chocolate*

[sevenhelz.livejournal.com]

mm pizza :)
sell the drumkit. sell it. sell it to me, if you can do so cheaply enough that i can buy it, and if you can keep storing it till we have a house and actual storage space. buahahaha.
xx

[dennyd.livejournal.com]

I want (need!) to sell it, and I'd be very happy to sell it to you (cheap). I don't have anywhere to store it though - which is why I need it gone ASAP.