I woke up this morning to find that CoasterBuzz had crashed in a non-trivial way. The whole site was returning 503 errors (service unavailable), which is pretty strange. I restarted the site, and found that the original problem was database timeouts caused by connection pool leaking. Then, my bit of logging code, derived from other logging code but made for CB, was crashing in a bigger way because it couldn't serialize the error to write to the event log. Stupid mistake I made nearly three years ago because I never tested it.
I think it took so long for the problem to manifest because it was a weird edge case, and also because the newer version of the Web server, which the site has only been on since the middle of last year, by default shuts down after x failures in y minutes instead of recycling. Whoops. All kinds of stupid there on my part.
I hate when I encounter things like that. Today it would be unlikely that I'd write code to let something like that happen. Meh. I guess stuff happens. It is an example of why I strongly dislike this sysadmin stuff, and hope that the days of affordable cloud hosting aren't too far off.
No comments yet.