I thought I was being DDOSed. Turns out I'm not that important... Stuck at home with man-flu (well, a grotty man-cold) I noticed a couple of days ago that I could not send outgoing mail because my mail server was refusing connections. Even from me. On my small (SheevaPlug) system I set it up so that the machine never need swap or page and stays responsive, and it just rejects extra work if …
You're not being targetting.
Every address on the internet receives scans from all sorts of places looking for things like open SMTP relays. Some of these are malicious (those trying to send SPAM) some not (hopefully your ISP performs such scans so that they can warn their customers when they're stupid) and some academic etc. Get used to them, they're not going away.
Such scans should be pretty quick (a few seconds each), but it sounds like yours are not - this will be down to some misconfiguration of your mailserver - either you're waiting for DNS to return RDNS results, or you're doing something stupid like accepting mail to other domains (whether you go onto deliver it or not).
I don't understand the downvotes Chris has got here. A couple of things immediately jump out from this:
1) Running services on an internet accessible device and then advertising that service on that device to the internet will result in people attempting to access that service. You are advertising that mail for your domain should be delivered to that device, it is entirely crass to then complain that people are sending email to your domain.
2) People can choose to send email to any postbox they so desire. You, as the server admin, can choose to reject any emails that you choose. Again, not malicious, not DDoS, not 'hack attempts' - this is just simply the consequence of running a publicly advertised service.
3) A large part of this was probably self inflicted. Your mail server is configured to only handle very low volumes of mail, and when you get large volumes of mail you immediately refuse to even examine it.
What happens then is that the sending MTA will queue it to resend, and it will keep resending it until your server accepts it - eventually it will give up. Because your server just instantly refused the connection, the number of outstanding emails doesn't go down much, and you stay "DoSed" until eventually the backlog is resolved.
4) Reporting this to the police is the cyber equivalent of calling the fire brigade to get a cat out of a tree.
I don't want to be too harsh to Damon (we're tenuously related by marriage!), but this read like the worst of Rory Cellan-Jones
@Chris W (1) the author is referring to spam email which is different from speculative network probes. (2) He has set up his own mail server, which makes him a hardcore expert on internet-facing security, and likely to know about "probes" (3) I can't see it has anything to do with DNS.
2) People can choose to send email to any postbox they so desire. You, as the server admin, can choose to reject any emails that you choose. Again, not malicious, not DDoS, not 'hack attempts' - this is just simply the consequence of running a publicly advertised service.
Assume you missed the reference to backscatter then?
The original mail isn't being sent to Dom@example.com, it's being sent as though it were _FROM_ Dom@example.com. Receiving MTA is bouncing it, and it's coming back to his server (the mailserver for example.com).
Not DDoS or a hack attempt, but I can't think of a non malicious reason (assuming you consider spam malicious) to forge your email address...
Pretty much anything giving backscatter now is poorly configured. Your average spam zombie machine doesn't bother trying to send rejected mail to the fake From: address, it just drops it on the floor[*] because it knows it's probably fake and not worth wasting bandwidth. The backscatter is from systems that accept the initial mail and then fake a bounce based on subsequent processing, which is not a particularly net-friendly behaviour. Either reject it immediately or just eat it, preferably the former. The only exception to this is a secondary MX, which might not have access to the list of valid users at a site and so will accept it to forward ot the primary MX - it's possible that a spammer has worked out that he can unload a lot more messages that way.
I remember once adding a tweak to my spam filter software to pick out the backscatter that helpfully told me I had a virus and auto-forwarding it to postmaster@site pointing out that I didn't and that they really ought to fix their software to stop spamming me because I never sent the original.
[*] Admittedly this is based on observing stuff supposedly from me to me being given a 5xx by my mail server, and then never seeing an attempt to return to the sender.
No, I didn't miss the reference to backscatter - did you miss the reference to him advertising that this sheevaplug was where all mail for that domain should go?
Point is, if you set up a publicly accessible mail server, and then configure it to be the primary MX record for a domain, it must be capable of dealing with the traffic that it gets.
On the internet, you will get backscatter. You should configure your mail server so that it can handle it.
If you configure it so it cannot handle moderate load, so that it drops all emails instead of rejecting invalid emails (which causes the sender to retry to deliver the email, once an hour for 7 days until it is either accepted or rejected) then you have contrived to DoS yourself. Call the police and tell them that.
It is not relevant that you consider this "unwanted" traffic - it is in fact entirely valid traffic, sent to the primary MX indicated for that domain. The sender is trying in good faith to deliver a non malicious message to the advertised MX for that domain.
This post has been deleted by its author
How does having skills in one field, setting up a private email server, extend to "expert" in another, that is internet security?
Task B requires prior knowledge of field A. Setting up am email server is the hardest job in IT IMO, demanding knowledge in many areas. respect to anyone who has done it.
Have you considered something like the following:
iptables -t filter -A INPUT -d $myip -i eth0 -p tcp -m tcp --dport 25 -m state --state NEW -m recent --set --name filter_$myip_25 --rsource
iptables -t filter -A INPUT -d $myip -i eth0 -p tcp -m tcp --dport 25 -m state --state NEW -m recent --update --seconds 60 --hitcount 3 --rttl --name filter_$myip_25 --rsource -j DROP
iptables -t filter -A INPUT -d $myip -i eth0 -p tcp -m tcp --dport 25 -j ACCEPT
What this will do is limit your traffic to port 25 to 3 new connections per minute per source IP given fixed TTL (to prevent source IP spoof based DoS-ing).
On the odd occassion that regular email gets caught in this, the legitimate server will simply retry a few minutes later. But it should stop endless spam spewing that is DoS-ing your server with a negligible impact on legitimate mail.
If you want to be nastier and you have the TARPIT rule compiled into the kernel, you can use -j TARPIT instead of -j DROP, but only do this if you are very confident that legitimate mail servers will foul of this very, very infrequently.
" What this will do is limit your traffic to port 25 to 3 new connections per minute per source IP given fixed TTL (to prevent source IP spoof based DoS-ing)"
If they spoof the source IP, to try to DoS his box, how do they even complete the 3 way handshake? They might open a load of embryonic connections, but not get to a point where they send actual SMTP
The ps he posted shows SMTP app layer data, like "RCPT to", so clearly the handshake has completed.
What's your iptable rules got to do with fixing his actual issue?
The TTL consideration is to stop a malicious party from performing the DoS where they try to fool the throttling into thinking that an IP has exceeded it's new connections limit. In classing terminology terms:
Alice periodically sends email to Bob. Bob throttles new connections from everyone to 3 per minute. Trudy spoofs Alice's source address and starts issuing new connections to Bob. Trudy doesn't care that the 3-way handshake never completes, beause all she is trying to do is fool Bob's throttling system into thinking Alice has opened too many connections in the given time period. Thus, if Alice then tries to send email to Bob, Bob's throttling system will deny Alice's legitimate connection. If we add the TTL check, and if Trudy's discance from Bob is different from Alice's (pretty decent chance), the Bob's server will not treat this as the same connection (it sees a connection source as (IP,TTL) tuple, rather than just IP), and this will allow Alice's connection despite Trudy's DoS attempt.
Of course Trudy can try to figure out what Alice's TTL to Bob is and try to attack it harder, but that's an extra hurdle to cross and we are not really looking at fighting an escalating arms race, just provide a good enough first-pass solution that covers the basic defence.
The iptables throttling fixes the issue by making no server able to open more than 3 (or some arbitrary number, up to the OP to decide what they want to set it to, but 3 is a probably a good start for a personal mail server where very high bandwidth miling lists aren't used) TCP connections to port 25. If it wants to send more than that, it'll have to time out, wait and retry later. Since MTA won't see any subsequent connection attempts from the source server in that minute, it won't be chewing through all the resources it is chewing through at the moment (because it doesn't have to accept the connection or receive the payload to analyze it). Therefore the little SheevaPlug won't be DoSed and will continue working as it's owner requires. The only caveat is that the occassional email from a relatively high bandwidth mailing list might get delayed once in a while.
That's all great, but it's not the problem he's got.
"Trudy spoofs Alice's source address and starts issuing new connections to Bob. Trudy doesn't care that the 3-way handshake never completes, beause all she is trying to do is fool Bob's throttling system into thinking Alice has opened too many connections in the given time period. "
But his output shows SMTP payload traffic. He's got a problem once you get past this issue you're defeating, of Trudy spoofing Alice's source IP, to open multiple embryonic connections, and therefore deny service to Alice.
Does this actually help mitigate his issue, or is it just something else someone could use to DoS his box that you're mitigating? That isn't actually happening? I can't see how it helps his reported problem
You're missing the point - if the SMTP traffic is coming from one server (or a small number of servers), that server's connectivity would get limited at TCP level. Connections over the 3 new connections/minute limit would get dropped at TCP level, which means they would never deliver SMTP payload. Processing SMTP payload is expensive, while dropping TCP connections is nearly free. So if he were to drop most of the offending traffic before it ever got past TCP layer to SMTP layer, the server wouldn't get overloaded.
"You're missing the point - if the SMTP traffic is coming from one server (or a small number of servers), that server's connectivity would get limited at TCP level. "
No, I'm absolutely not missing the point, you just changed the point.
Up till now you were describing defeating someone who spoofed an ip address.
If you spoof an ip address, you can't complete the TCP handshake.
You can also spoof every ip address in the world, assuming you can route them to the destination (okay, potentially you can spoof the ips on your own ISP's range, or other ranges you can get access to)
While your iptables trick stops someone spoofing THE SAME IP ADDRESS to open embryonic connections more than 3 times in a row, and fixes the "Trudy spoofing Alice's ip" issue you decribed that HE HASN'T GOT, it doesn't fix a problem of many many ip addresses attempting to sending SMTP mail to his domain.
I'm not dismissing your iptable config, it's nice. It's just not the problem he described. You're fixing some other problem.
You even said so yourself, with all the stuff about "Spoofing sender ips"
I didn't change the point at all - if you read the iptables rules I proposed you will see that it was specifically about throttling connections to port 25 to 3/minute. The elaboration on the TTL part for mitigating source IP spoofing DoS vector was there to clarify why that part is included at all, but it was never the primary point of the exercise. If you'd read and understood the iptables rules that would have been clear to you.
If the OP's problem is that he is getting spam floods from a small number of IPs, then the iptables rules listed will alleviate the resource pressure on the MTA by dropping the TCP connections to port 25 from any source IP that has exceeded 3 new connection attempts in any 60 second period. The OP's problem is that his mail server is running out of resources due to excessive SMTP spam traffic. Dropping a substantial chunk of that traffic via iptables at TCP before the packet ever gets as far as the SMTP layer in the stack will reduce pressure on the MTA and help the server not run out of resources.
A similar technique can also be used to mitigate brute force dictionary attacks on any service (simply tune the port number, connection attempts, and time period variables to suit), within reason (obviously it won't help as much against a distributed attack via a botnet with lots of IPs). The point here is that the OP didn't specify whether this was a heavily distributed flood or a flood from a small number of IPs. Without any indication to the contrary, I assumed (correctly or otherwise) that this was caused by a relatively small number of IPs. Still, even if it is a hevily distributed attack, you can whittle away a lot of traffic by using a method like this - potentially enough to make a difference between a usable system or a complete DoS.
I guess I don't understand why you kept describing this as a technique to mitigate attackers spoofing IP addresses, across multiple posts, if that wasn't the primary point of the excercise, and there was no mention of the attackers spoofing IP addresses.
Maybe that's just me not understanding the iptables rules, and not you misrepresenting what you're trying to achieve ;)
Some people in IT run their own infrastructure as a way of learning, particularly learning thing which aren't core to their current day-to-day work. The people I know who have been successful in their careers are the people who've investigated and learned in their own time.
I know Linux because several years ago I saw that Linux was being taken seriously. At the time I was a Windows server/infrastructure specialist, so I devised a "learn Linux" project, to that end I made a MythTV system. This greatly contributed to my knowledge of Linux and how it works and enabled me to move into a job where I used both Linux, UNIX and Windows. Likewise, I run an Exchange server at home so I know how it works, even though it's not currently part of my job.
I'd agree with you to a point (although back in the day, Demon actually required people to run their own SMTP servers and supplied a dos app to do it for them). However, I think that now, most ISPs shut down anything that looks like an open mail relay, and SPAMHaus blocks pretty much the rest of them (including whole blocks of dynamic IP addresses just in case).
I would guess that nowadays most spam either comes from cooperative ISPs, or trojen installed applets that either send the mail directly, send it via your email provider, or log in to someone else's webmail and sends it from there.
Comparatively little. Open relays were a problem in the 90s, but now the main issue is hordes of zombie PCs. Looking at the article attachment and own my mail and ssh logs confirms this: spam and probes come from home user PCs. It's fairly easy to confirm with whois lookups and hostnames which contain clues like "dialup", "ras", "dynamic" and such.
This is also relevant to using iptables: it is helpful for ssh probes but not so much spam.
Fail2Ban is great for blocking relay attempts which are pretty much always spam. It can even ban based on things like SASL authentication failures, incorrect HELO handshaking, or exceeding the maximum number of parallel connections. I've been running it for months and my Iptables list is full of Chinese IP addreses - and a few from some western hosting companies who seem to turn a blind eye to spammers on their infrastructure.
If you need SSH open but want to protect it then DenyHosts is a wonderful system which shares "rogue" IP addresses across its installed base.
I run my own mailserver as well. I first did it many years ago (when there weren't hoards of suitable services on the net) and I was completely peed off with my ISPs borked servers. It also means I can choose my own policies on spam management - including what is (to me) a priority constraint of "if you accept it for delivery then you deliver it", I absolutely do not accept any excuse for accepting a message and then not delivering it*.
It was also a useful learning exercise. It also means that when I use (for example) the multi-function machine to scan stuff and email it to one of us, it doesn't need to go out on my limited outbound bandwidth and come back in on what used to be also a very limited bandwidth (it's now just limited !).
It also means I can look at the logs - cf below about not knowing if stuff has been delivered. At least I can show that I sent stuff and if someone didn't receive it then that's their fault (either for setting up a broken system or using a broken external system).
More recently, it means my mail is IPv6 enabled :)
* Personally, I think the idea of building a whole system around the concept of "accept it, say you'll deliver it, then throw half of it away" is, struggling to stay polite, completely brain dead. It leads to all sorts of problems - notably the fact that these days (unlike the "good old days") you can have no confidence that when you send an email it'll either be delivered or you'll get a message back to say it wasn't. In other words, most mail service operators have deliberately broken their systems.
Ditto here: I run Exchange at home to keep my hand in (ok it's hardly a multi-site farm but it's useful!), and also CentOS Postfix MTAs to increase my knowledge, which all helps with the day job. Only this morning I was setting up greylisting on them to try and reduce the incoming spam. Not that I get that much but I take any as a personal insult ;)
"Ditto here: I run Exchange at home to keep my hand in (ok it's hardly a multi-site farm but it's useful!), and also CentOS Postfix MTAs to increase my knowledge, which all helps with the day job. Only this morning I was setting up greylisting on them to try and reduce the incoming spam. Not that I get that much but I take any as a personal insult ;)"
I've been running Postfix + Greylisting for a few years, it's been working very well for a few years. Next step is to see what I can do with CARP. :)
"Why do you need to run an email server anyway?"
I can't speak for the author, but minimising the number of third parties harvesting private email for fun+profit was enough motivation for me. Spending hours reading EULAs and re-reading them every time they changed was wasting a lot of time. Also a third party service can be terminated/changed/dropped so you have to be prepared to shift everything over to another one - quite often at very short notice (that has happened twice - both times with zero notice, zero mitigation and no refund).
At least when you are running the server yourself you have some control about what changes and when.
Hi AC, he might do it because he has a domain and wants to use the email address me@mydomain. Having his own server, he can do that for free without paying somebody else for it. He can also give other people their own email address @mydomain - either friends/family or on a commercial basis (probably not commercial as it is on a Sheevaplug). Perhaps mydomain is the name of a company he runs, for example. Or he wants to deepen his experience and knowledge for professional reasons, or just as a "hobby".
I think he's running the mail server as an education exercise, a way to keep up-to-date when you don't have a client or company test system to experiment with. It is better than testing in production where other people will be affected.
Otherwise there are all kinds of places that will host your @domain mail server (plus web site, etc.) for you. Problem is you don't learn server configuration and protection that way. But then not needing to learn mail server configuration can be a good thing if your real business or job is application programming.
the question should have read "what benefits are there to you in running an email server"? Then you are trying to elicit a response and further information, rather than sounding dismissive. I haul up people from time to time in meetings to try and get them to ask "the right questions"; ie. ones that don't drive an agenda.
BTW, the answer for me would be "I can experiment and train with mail setups and security in a non-Production environment".
Well, it makes it more difficult for the NSA or anyone else to read my mail without asking me or breaking into my house. They can still do it, but it means they have to try that little bit harder and use another method. It's all where I get at it, I can easily keep backups and I don't get bothered with adverts if I read it using webmail.
I use a Sheevaplug for my mail server too, although mail is stored on an external USB hard drive, I wouldn't trust the SD card for that (had a few of those go down due to bit rot).
Of course, it's fun to learn how to set it all up and make it work, knowing how to run such things can be useful on occasion. I run two DHCP servers and two DNS servers on my home network too, in a master/slave arrangement so if one goes down, everything falls back to the other one. Sheevaplugs are good for this, extra resources, low power consumption.
FYI the link to Action Fraud is wrong, it should be http://www.actionfraud.police.uk/. It's run by the City of London Police who have overall responsibility for fraud and internet crime and pass (mostly statistics) on to people like the National Fraud Intelligence Bureau, other police forces, the NCA and the National Intelligence Machinery. Action Fraud are worth contacting if you have been the victim of any fraud, even if it's just someone making an unauthorised charge on your credit card.
There needs to be a single international body with umbrella organizations run by police in each country where you can send details of the online attack and sample files, there is no one is even trying to keep statistics on hacked websites or other forms of attacks its is discraseful.
No where to report online?
No where to send sample files i.e fake PayPal log files etc?
No standardization of addresses to report fraud/phishing emails to even from the big banks and websites
Though at least Google and Microsoft will drop you an email if they detect your website has been hacked (if you have register with their Web Master Tools service.
Your mail system is being enumerated; the attacking network is atetmpting to obtain a list of email addresses for which your MTA will accept email.
There are some very poor attack lists doing the rounds, but since this is all zombie traffic, it's free for the attacker...
HTH
Vic.
If the random ip addresses make a connection to the smtp server and either drop the connection or do anything invalid then they can be blocked by fail2ban. True if there are enough it may cause issues with iptables but I've never had a problem yet and get random attacks like this multiple times a day now and then from blocks of random addresses which are blocked within seconds
I had a similar 'attack' a few months ago. I thought I ran a pretty tight ship but I was proven wrong. Firstly, I'd suggest running postfix instead of sendmail; It has far more easy-to-configure features for staving off attacks like these (such as anvil). I had to block the backscatter to the fake <never used addresses>@mydomain.com using the following lines. In my case the mails were not coming from any common source IP block (i.e. it was a DDoS) so I had to block on the targeted fake recipients:
smtpd_recipient_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_destination
check_sender_access hash:/etc/postfix/sender_whitelist
check_sender_mx_access cidr:/etc/postfix/mx_access
check_recipient_access hash:/etc/postfix/recipient_block
reject_unknown_recipient_domain
reject_rbl_client zen.spamhaus.org
reject_rbl_client bl.spamcop.net
/etc/postfix/recipient_block contained the following (e.g.):
fakeuser@mydomain.com DISCARD
fakeuser2@mydomain.com DISCARD
This rejects and effectively blackholes these emails while dropping the client connection at the earliest opportunity without causing any more error emails. It does seem that the attack attempts have now gone away. I was getting hundreds per minute before.
PS: I have a load of other settings to prevent spammers (none actually examine the email body) and as a result I get next to no spam (< 10 a year) on all of my 10 active email domains.
Another way of locking out the 'tards is to note several "nosuchuser" delivery attempts and simply blackhole the sending IP for 6 days. Suitable tweaking of "fail2ban" or related packages can do the trick and this will drop the double bounces back into the postmaster box of the malformed mailserver (said box is likely to be malconfigured and spin itself into the ground or die of a full disk as a result)
If these really are malformed bounces then the odds of getting only 1 or 2 from any given srver are slim to negligable
You might also want to think about blocking IP addresses based on Xtables and Geo IP. Not much use if you have an international friend list but for many of us large blocks of countries can be denied access to port 25 by shoving a xtables enabled Linux firewall (running something like shorewall) in front of your Mail server.
Expect many thousands of intrusion attempts a day for an address with a domain name. The worst bot armies are smart enough to evade adaptive firewalls by having hundreds of bots attempt only four or five attacks each per day.
Your best bet for a tiny server is to determine which networks have the most bots and hackers then block them. My small business router pretends that nobody's home when Taiwan, China, or S. Korea calls because they'd otherwise saturate my computer and bandwidth. Anything mentioning "yahoo" to my mail server gets rejected so there's no need for a trip to a Spamhaus to confirm that it's junk. I occasionally sift through what remains and send out abuse complaints when it gets bad. An invalid network abuse contact gets the network in the firewall.
Back when mail was sent on paper the Spam was limited to a few items once or twice a day. Now that mail is electronic and effectively free there is nothing to stop the Spammer from constantly blitzing all of us with it. Blocking certain IP ranges will cut some of your spam, but much of it comes from distributed botnets so good luck blocking that. Personally, I gave up on running a home mail server years ago as it is far too complicated, expensive and time consuming these days. Either that or else get a more capable server.
Self-determination generally, and greater control of one's own service, specifically. It's a personal decision.
I will refrain from citing specific capabilities, because I don't care to convince you and you might be a troll anyway. This is easily researched or imagined if you are motivated.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
