For a variety of reasons, I prefer to run my websites, and those of a few friends, email and various other services on my own server. it is sitting here beside me, occasionally winking its disk access light at me.

Doing this is apparently an open invitation to half of the population of China, 30% of Russia and practically the entire populations of Poland, Vietnam and a host of other countries to attempt to break into the machine, either for purely vandalistic reasons, or to try to use the system to launch further attacks or to deliver SPAM.

Last Christmas I treated myself to the bits to build a more compact, but much faster and more capable server. I, of course, loaded the latest and greatest versions of all the software and frequently load all the latest patches. So far, there have been no direct successful break-ins.

However, there are a continuous stream of people “rattling the doorknob”.
I noticed that some of the log files were getting big. Fast. It was hundreds of attempts per minute to remotely log into the system, trying random user names, and long lists of passwords.
One or two of these, I would just ignore, because they are not going to get in with just a username and password anyway. But the huge volume was just annoying.

I have a firewall between the Internet and my internal network, so I set up a set of rules to allow people three attempts to successfully login (after all, I want to be able to do it myself), and on the third failure, block that IP for 24 hours.

After 24 hours, the list was hundreds of entries long. But there were now many fewer people rattling the doorknob.

Next, I woke up one morning to email on my cell phone with several hundred bounced email messages.
I don’t allow people to route email via my system unless they can authenticate, but someone by continuous trial and error had somehow guessed my wife’s password and was using that to send SPAM – by the thousand. A quick change of password fixed that, but I wasn’t happy looking at the continuous stream of attempts to guess passwords on the email server.

Back to the firewall, and a new set of rules to do basically the same thing that I did for remote login. As soon as it had blocked a few hundred IP addresses, things quietened down.

Looking at the IP addresses in the block lists, most are from China, followed by Russia, Poland and a variety of smaller countries.

A while later, there was another deluge of bounced email messages. A bit of digging around revealed that these were coming via the web server.

I run a few websites. For simplicity, I have tended to set up WordPress for most of these. Its relatively easy for people to use (I don’t have to get too involved) and can be made to look not too bad relatively easily. The problem with WordPress is that it is a pile of s**t. Full of holes for hackers to exploit. I had already somewhat locked down user registration, requiring anyone wanting to register on one of these sites to respond to an email to activate the account. Since 99.99% of the “new users” come from bots, they tend not to reply to email. But that doesn’t stop them trying dictionary attacks, guessing usernames and passwords.

So, you say, what is the worst they can do? Post a few spammy messages?, Well, no. There is a utility in WordPress intended for things like “trackbacks” (which I consider useless anyway), but which also allows basically execution of arbitrary commands — once you have authenticated. That had been used to add extra code to the website to go fetch SPAM and a list of addresses, and to blast it out to those addresses.

A quick delete of the entire website, and a restore from a backup removed the problem. But I had to stop it happening again. First thing I did was to change the permissions on that evil little tool (xmlrpc) to completely disallow access.

My logs now began to fill with error messages caused by attempts to access this file. There were also various other things they were trying to access, including the login process.

Unfortunately, since this was a website, there was no easy way to detect/block these at the external firewall. There is a tool (fail2ban) that can be used to scan log files for certain patterns, and block IPs locally on the machine when it finds those patterns. I took a look, and decided that it was overly complex, so let it drop, and decided to just ignore the garbage in the logs.

Then, someone found a new bug in WordPress, and within a day hacked every WordPress site I had to convert them into sites via which to download all sorts of undesirable content (mostly stolen/hacked software, documentation etc). A new WordPress release was rushed out to fix this hole, but everything was already hacked. So, one by one, delete the website, restore from backup, upgrade WordPress. Of course, there were still thousands of hits on the old download URLs (which no longer worked), as well as attempts to restore the hack.

I gave in. I installed fail2ban.
I set it up such that anyone just touching the xmlrpc file was immediately blocked.
Anyone attempting to access the old hacked URL was immediately blocked.
Anyone trying to authenticate more than N times within M seconds was blocked.

There was still a lot of traffic trying to authenticate. It was coming from random IPs. These people have vast numbers of hacked machines that they use, so you tend not to see the same IP too regularly.

But, this website is a little different. Its mine. Only I log on to it. So I set this one up so that any authentication attempt not coming from my computer gets blocked immediately. That locks out that IP from all the other websites too.

The lockout is about two weeks.

looking at the list of IPs blocked just by fail2ban, there are currently 1,936 of them, and growing steadily.

I do wish these people would go find another job. I find doing this stuff annoying.