Battlefront Chronicles: #1 Mail bomb attack
"Battlefront Chronicles" series shares stories of production mishaps, how they were handled, and the lessons learned along the way.
Please keep in mind that the stories in the Battlefront Chronicles may not all be my own. Some of them come from other engineers working in the field, and their contributors may choose to remain anonymous.
Webhosting
It was way back in the day when I was only 16 years old and was running a small server to host websites on it. To mention, there wasn’t many webhosting companies at the time. The business grew, so I expanded business and actually created a webhosting company under my family business name. I started to accept more and more customers. Some of them migrated from other web hosting providers, because they got hacked there, or were using too many resources. But that was totally fine in my eyes. I always valued this in myself, that helping those in need is honorable. And if they are desperate, I am here to accept them and help them in any way I can.
Webserver
As I am coming from a bit of a black hat-ish background, I knew how to prevent majority of web attacks, even if code was very insecure. Apache’s module mod_security helped me tremendously. I was digging deep into kernel hardening with custom kernel grsec patches and similar stuff. Majority of the webpages were running PHP so I was running suPHP that actually allowed me to limit resources that each customer was using. I never was into cPanel and such proprietary solutions. No. I rather wrote my own control panel. I integrated all required services and tied them together. From ProFTPd, MySQL, Apache web server and mail server that was running Postfix, to disk quotas, cpu and memory limits with cgroups. It was perfect.
Fear of attack
Only thing I was worried about was being DDOS-ed. (Back in the time, internet was very chaotic environment.) So I decided to take it up a notch and bought co-location for my server with, at the time, enormous network pipe of 1Gbit! I was safe. There wasn’t many DDOS attacks that this kind of pipe couldn’t handle and even so, my ISP was very helpful, if I needed to block some traffic before it actually hit my server. So, that part was neatly covered.
The Nightmare
As engineers, we may think we got things covered, feel comfortable that we really nailed this one and nothing can go wrong… there is always another thing that we didn’t think about.
One day, my customers started calling on my mother’s company phone, that their websites that are being hosted on my server are not available. Of course, I got chills in my neck. What could it be, I felt I need to act immediately. When I reflect on it now, I feel I was being a bit silly. Even if the website dedicated to some pet dog wasn’t working for 5mins, its not the end of the world. But at the time, it did feel like that to me. I thought I failed miserably and need to fix this right a way.
I barely could log in to the server through SSH. I noticed it was getting hammered with traffic. CPU was going wild and Load was terrible. But what kind of traffic is this that is causing this? Immediately I started iptraf to pinpoint where the traffic is coming from. It was thousands of different IPs, connecting to… SMTP server! What the hell? Is someone sending a lot of emails to one of my customers? Is this legit traffic or not? I started to dig into it and could pinpoint that ther was only one affected email address that was getting mail bombed. I contacted customer and asked if I can review some of these emails coming in, because I felt something is off and I can’t pinpoint without looking at the content, to say if its legit or not. Customer agreed and I reviewed his incoming mail. It was all rubbish text. A lot of it. Randomly generated. Not a single thing, besides “To:” field that had email of my client was same in any of them. I thought, why is this even hitting my mail server, I have SpamAssassin and I do check in Postfix if there is valid Reverse DNS on IP, before even accepting any mail. How is this coming through? I reviewed some of the offeding IPs and checked Reverse DNSes, MX records and SPF records. They were all fine! They were totally legit emails that are sending enormous amounts of rubbish spam and totally overwhelming my server.
The situation seemed miserable. How can I stop this. I can either just block all SMTP traffic, but then every customer will get affected. I don’t want that. I also don’t know when this attack will stop and they cannot be without emails for all that time. There was no light at the end of the tunnel. But then I said, okay, lets chill out and think. What can we do? And then it hit me. Mail servers won’t quit sending email if service is unavailable, so blocking was out of question. They will just delay it for a while and retry after retry period expires. What will happen if I change the MX record for the domain name of this affected email address…. to 127.0.0.1 or better said: localhost? That will actually make mail servers to resolve domain name to their own mail server and… start spamming themselves. This will also utilize their own spam prevention systems, that if email account doesnt exist on their system, they will discard it. And so decision was made.
Fight back
I called affected customer and explained the situation. I had to break it to him that his email is screwed anyway, but in order to get the attackers to stop, is to let these mailservers DDOS themselves to oblivion and they will hopefully stop after some time. He agreed and I swapped MX record of that domain to localhost. Quite soon, my server’s load went back to normal. Attack was stopping, as TTLs for this MX record was expiring all over the world and my server was back online.
Victory is mine!
Feeling of victory overwhelmed me. I got them good! I managed to throw their own rubbish back at them. Customers started to report everything is awesome and I could finally take a breather. I decided to leave this domain MX pointed to localhost for 24 hours, before I return it back. I was a bit worried, attack would start to happen again, after I reset it to correct IP, but no. Attackers learned their lesson and just stopped with the attack. Luckily, they never tried something like this again, but then again, they probably lost a lot of their zombie nodes, since server admins figured out something fishy is going on, with their mail servers DDOSing themselves and started to remove malicious code doing it.
Interesting read :)