Marcelo Calbucci

[This is mostly a geek post, so if you are not one, you can just skip it]


I have an error logging system on every project I do that sends me an email every time it logs a system error or an exception on the code. Email is perfect because it’s easy to query – at least on Gmail – and you can feel the pulse of the quality of the code. If I wake up and my “Errors” folder has dozens of errors, it was a good day. If it has hundreds, not so good. If it has thousands, it means I really need to fix something urgently.

For the most part, this system works really well, except when… Twitter has a problem. For example, on Seattle 2.0, we update the Twitter Directory every day grabbing the latest profile information and the social graph of each user. That means thousands of calls per day to Twitter. If Twitter is down for 24 hours, I would wake up to thousands of error messages on my email. Not a big deal, actually.

Yet, on Tuesday I “fixed” a few problems with the code to update the Twitter directory. One of the fixes revolved around a change on the API Twitter did many months ago, but I never got to the point of implementing the new version. That’s the social graph API. Before, you’d just do one call and Twitter would return all the friends or followers of a user, even if she had 100,000 followers. They changed it to use paging, so that you’d get 5,000 at a time.

Long story short, the code worked so I deployed it on Tuesday night, except there was a tiny case that would surface if Twitter was down. And that night Twitter was down for several hours. And that tiny case meant that we would keep calling Twitter on an infinite loop until it responded without an error. In a period of just a couple of hours, our server called Twitter about 200,000 times, failing every call. Logging every call. Sending me 200,000+ emails to tell me it failed.

Yep. I clogged the GMail pipes. When I woke up in the morning, there were still about 80,000 emails on the SMTP queue on the server which I was able to clean up, but more than 120,000 were already out of my server and being delivered to my GMail account. After 48 hours I’m still getting about 2,000 emails per hour. That would not be a problem, except that I’m not getting regular emails.

If you sent me an email on the last 48 hours, I probably didn’t get it yet. I just hope I get it at some point today.