12497

Damn, but sometimes I really can’t stand people. I’m at work now, by the way, at 7:10AM and have been since 8:00AM yesterday. We had a little emergency here at the office starting Sunday night that should finally be resolved by about 9 or 10 this morning. One of our mailservers that hosts about 30,000 mailboxes started acting up Sunday night. (Geek aside: The server was accepting mail via SMTP properly but was not performing local or offnet delivery of the mail so it was just building up in the queue.) This problem was barely investigated by the person responsible for the server and treated – by people who do not interact directly with customers – as a minor issue.

E-mail issues are NEVER minor issues to the people affects. Many ISP’s have a history of handling these things in a very cavalier manner. AOL, Earthlink, Juno, and AT&T have all been taken to task for the same problem we experienced. However, unlike us, they just deleted the affected mail and apologized to their customers. I did not find that acceptable for my users, and thankfully, neither did my boss. We worked our asses off yesterday to not only restore real-time service, but find a way to deliver the 145,000 pieces of mail that we received to their proper recipients. I spent all last night breaking that mass of mail into smaller chunks that the server could process in addition to its normal load. That involved sitting at my machine – grepping and building batch files – for hours. By the way, this is the ONLY time anything like this has EVER happened here and we have been providing net access and email since 1996.

Technology is not perfect and people have become accustomed to the occasional blow-up. There are people who have been on the phone to tech support crying when their favorite websites were down – and I mean that literally – CRYING. Ebay going down for 10 minutes, The Zone, Yahoo mail (like it did yesterday for an hour) being unavailable can be upsetting, but in the grand scheme of things it is nothing to through a fit about. We have one memorable woman using our service who calls in screaming and crying when a singles/dating site she likes is down. We have taught her how to do some diagnostic work herself – pings and traceroutes – so she KNOWS it’s the dating site that’s down and there is nothing we can do about it, but still she calls and complains.

Every since this emergency began, we have told users who write into tech support that no mail is being lost and that we are working on delivering the mail. It doesn’t matter. The fact that the Internet works at all is a miracle to me and I work in the industry. The fact that a piece of mail can make it from Japan to my mail client in seconds astounds me. The fact that a user can dial up with a 28k modem and establish a – metaphorically speaking – hair thin connection to this global network and access naked pictures from a server outside of Amsterdam is beyond astounding. Yet users, yes – a four letter word sometimes, cannot sit back for a second and just wait for a problem to be fixed or at least

Ahhh screw it. I’m actually too pissed right now to go on.