The Microsoft SMTP service has endured a lot of criticism for its lack of security out of the box (in the case of Windows 2000), its relatively lackluster performance (in the case of Windows Server 2003) and its unbreakable integration with Exchange (in both versions of the OS). However, its security need not be a problem.
Here are four things you can do to help improve performance and secure your SMTP service:
Limit the size of permitted messages: Many denial of service (DoS) attacks are designed to prevent normal access to a certain mail server. If an attacker sends tremendously large messages to a specific SMTP server, it may become otherwise unavailable to service regular incoming and outgoing mail, making the attack successful. Limiting the size of messages can significantly reduce this risk. If a mail client sends a message that exceeds the threshold you set, the client will get an error; if a sending SMTP server performs a standard "EHLO"-style lookup and detects that its pending outbound message is too big, the original sender will get an NDR. The default maximum is 2,048 KB, and you can't prevent messages smaller than 1 KB from being sent.
Limit the total size of a single session: Another angle to a DoS attack is to send thousands upon thousands of messages in rapid-fire succession to your SMTP server during one session. The spammer makes that server so busy processing those messages that it can't spare the CPU cycles to answer other legitimate mail requests. When you limit the total size of the session, you protect against this happening. One note: Make sure you set the size to more than the minimum message size as described above; otherwise, no mail will pass. I recommend setting it to the same threshold as the above option.
Limit the number of messages per connection: You can also protect against a spammer or other cracker using your SMTP server to flood recipients internal to your organization with unwanted spam by setting a maximum number of messages per connection. As a matter of fact, there are some performance benefits to setting this to an appropriate value -- between one and five. And, if you have some money left over in your IT budget -- yeah, right -- you might even consider adding a third-party software package to perform "tarpitting," which adds a significant delay between messages to hopefully slow down address harvesters and spammers.
Limit the number of permitted recipients per message: This tactic guards against another problem, that of junk mail using a single message that is addressed to many people. Surely you've seen spam that has 40, sometimes even 50 recipients, all internal to your organization, advertising something strange. The minimum required number to comply with the SMTP RFC (number 821, if you're curious) is 100. When the message's number of recipients exceeds 100, the Microsoft SMTP service will deliver the first 100 and then open another connection for delivering the remainder of the messages.
Ensure relaying is closed: An open relay is an SMTP server that accepts mail from any server and will send it to any server, without restriction. This is a fairly old problem, but a surprising number of home servers and machines in small businesses are configured with an SMTP service that, by default, is open to relaying. This is especially a problem if you are still running Windows 2000 (and a surprising number of people are, given the latest usage statistics I've seen). Depending on the speed of the connection that the spammer and the SMTP relay computer are using, it is possible to send millions of spam messages via an open SMTP relay within just an hour or so. Luckily, there are several resources available for disabling open relays in the Microsoft SMTP service, most notably the document on Microsoft TechNet: Securing Your Exchange Server.
About the author: Jonathan Hassell is author of Hardening Windows (Apress LP) and is a SearchWindowsSecurity.com site expert. Hassell is a systems administrator and IT consultant residing in Raleigh, N.C., who has extensive experience in networking technologies and Internet connectivity. He runs his own Web-hosting business, Enable Hosting. His previous book, RADIUS (O'Reilly & Associates), is a guide to implementing the RADIUS authentication protocol and overall network security.
