Yeah, I’m pissed. Really pissed. When does comment spam become a DDOS attack? There’s a fine line, and some a**hole crossed it this morning.
I knew there was a problem when I sat down to eat my cereal and read the news this morning and the Windows XP Home login screen told me I had 1,134 unread emails. Thunderbird confirmed it: we were under seige some some dumba** pushing “texas holdem poker.”
Gadgetopia essentially came down. He was firing spams at the site from a pool of IP addresses (probably a zombie network). The Movable Type admin interface wouldn’t respond (“ObjectDriver Error: too many connections” — MySQL was evidently in great pain). I tried to get through to phpMyAdmin to delete them through SQL, but that wouldn’t come up either.
When I got into the office, Joe and I got a command line on the box via SSH. It was ve-e-e-e-ry slow. “top” told us that it was running at about 50 times the normal load, and the process list was filled with “mt-comments.cgi” and “mysqld.”
We changed the permissions on mt-comments.cgi to 644, and killed all the existing processes. The load on the box slowly returned to normal.
The final damage was 1,710 comment spams from 173 different IP addresses. MT-Blacklist stopped virtually all of them from appearing on the site. It also prevented a rebuild of the pages, which helped a little too.
(The ironic thing was that he put the hyperlink in an A tag and I strip HTML, so even if the comments had appeared, the hyperlink wouldn’t have shown up.)
The final mess to clean up is my email account. I now have 1,700+
messages on an IMAP server which is none too pleased about it. Deleting 1,700 messages at a time isn’t a quick operation.
Comments are still down. We’re watching the logs and he’s still trying — we have an attempt as recently as 20 minutes ago. We’ll try to get them back up later today.