securing windows 2003 server for mailmarshal 2006 ......whitepaper – mailmarshal smtp 2006 –...

24
WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 1 Securing Windows 2003 Server for MailMarshal 2006 Standalone Installations June, 2006 Contents Introduction 2 Security Best Practices 3 Installation Considerations 3 Ports and Services 4 Firewall Port Requirements 5 Securing MailMarshal 6 Anti-Spoof protection 7 Attack Prevention 9 Receiver Greeting 10 Received Stamp 11 Securing Server Networking 12 Appendix A 20 References 23 This document is a guide to securing a Windows Server2003 server for use with MailMarshal SMTP 2006 in a standalone server installation. MailMarshal has a secure and resilient SMTP receiver; however there are other aspects of Windows that need to be addressed, particularly if you intend to deploy MailMarshal email processing nodes in the DMZ. This document is not intended to replace, or be used instead of, good business practices, nor is it the intention to be used as the only a guide to security measures you should take in your organization. Authored by Kevin McFall

Upload: others

Post on 11-Aug-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

1

Securing Windows 2003 Server for MailMarshal 2006 Standalone Installations June, 2006

Contents

Introduction 2

Security Best Practices 3

Installation Considerations 3

Ports and Services 4

Firewall Port Requirements 5

Securing MailMarshal 6

Anti-Spoof protection 7

Attack Prevention 9

Receiver Greeting 10

Received Stamp 11

Securing Server Networking 12

Appendix A 20

References 23

This document is a guide to securing a Windows Server2003 server for use with MailMarshal SMTP 2006 in a standalone server installation. MailMarshal has a secure and resilient SMTP receiver; however there are other aspects of Windows that need to be addressed, particularly if you intend to deploy MailMarshal email processing nodes in the DMZ.

This document is not intended to replace, or be used instead of, good business practices, nor is it the intention to be used as the only a guide to security measures you should take in your organization.

Authored by Kevin McFall

Page 2: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

2

Introduction The following document describes techniques and procedures designed to provide a secure environment for deploying MailMarshal 2006 on a Standalone Windows 2003 Server into a DMZ.

Note that this is not the recommended solution (Recommended Solution is described in the whitepaper Securing Windows 2003 Server for MailMarshal 6 Array).

The design and security inherently built into MailMarshal 2006 is for deployment of the Array Manager and SQL in the Trusted Network and the MailMarshal Email Processing Nodes in the DMZ. However if budget, current policy or other factors require all MailMarshal components to be deployed in the DMZ, this document will assist in making the environment as secure as possible.

The document will present only a single scenario; however the policies and concepts used can be applied to a wide range of environments by a competent engineer or technician.

Target Audience The intended audience for this document is an engineer or technician with a good understanding of Windows Server 2003 and general Windows networking environments.

The reader should also have a good understanding of messaging systems, networking architecture and TCP/IP. An understanding of MailMarshal SMTP or at the very least, SMTP gateways would also be advantageous.

Assumptions The environment assumed in this document is that of an Intranet Web Server and Windows Domain Controller within the trusted network. A single DMZ is also assumed with a single MailMarshal server and SQL Server (or MSDE) deployed within this DMZ.

As the entire MailMarshal Server is to be deployed in the DMZ for the described environment, and with the current implementation of MailMarshal the Configuration tool requires Remote Registry access to the Array Manger. As such it is recommended that Terminal Services be enabled to the MailMarshal Server to perform Configuration tasks.

Page 3: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

3

For the purpose of this document it is also assumed that MailMarshal and SQL Server (MSDE) are the only applications running on the server in the DMZ. Additional applications may require additional ports or operating system functionality to be made available.

Preparation Some of the items in this White Paper refer to changes in the Windows registry. If performed incorrectly these could have adverse effects on the performance and reliability of Windows 2003 Server. Ensure that adequate backups of system on which the changes are to be made have been performed before any of the changes suggested in this document are performed.

Disclaimer This document does not provide information about securing a Windows 2003 Server for every conceivable threat or attack. It offers information and policies around securing a server based on information available at the time of publishing. Marshal Limited, neither its employees, associated organizations nor anyone else warrant this information in any way, and will not be held responsible for breaches in security within any organization where the information in this document was deemed inaccurate, insufficient or in any way responsible for allowing such a security breach.

Security Best Practices It is important to understand that it is in the nature of malicious attackers to continually attempt to find ways to disrupt systems. As such the security measures put in place today may not be adequate in the future. The more prolific the deployment of an operating system the more it is targeted by these individuals, as it increases the breadth of vandalism that they can initiate.

Fortunately the more prolific the deployment of an operating system the quicker issues are recognized and dealt with. Such is the case with Windows. In most cases the reluctance in deploying Windows in the DMZ arises from poor education regarding the Operating System and the fact that its widespread use causes greater media publication of issues. In many ways this assists in ensuring that the prudent administrator is aware of, and secures against any new threat with greater agility.

It is always best to keep any Operating System up-to-date with security patches and fixes made available from the manufacturer. These are published soon after any new threat is discovered and help to ensure that the Operating System is protected from any new malicious behavior from external and internal sources.

The recommended method of ensuring that security updates are applied as soon after release as practical is to ensure that the MailMarshal Server is included within some form of patch management environment within the site. At the very least Automatic Updates in some form should be configured on the Server.

Installation Considerations The following are the considerations that need to be taken when installing the various MailMarshal components on the Servers. Primarily the focus is on the MailMarshal Server in the DMZ; however depending on the target environment and security policies of the site there may be considerations for the IIS Server if deployed.

As a general rule it is recommended that only the services required to perform the application functions are installed on a server, this ensures that other services are not available to be exploited by a malicious attacker, or are inadvertently neglected when security updates are applied.

Page 4: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

4

MailMarshal Server The following considerations should be made when installing the Operating System for the MailMarshal Server in the DMZ:

1. Install only the default options for Windows Server 2003. No other services or functionality provided with Windows are required.

2. Once SQL 2000 or SQL 2005 is installed, apply the latest Service Pack, or at least SQL 2000 SP3a or SQL 2005 SP1

3. Use a fixed IP address and manual parameters for the default gateway. No DNS servers need to be configured at the TCP level; MailMarshal has its own configuration for DNS.

4. Once installed, Services that are not required for the functionality of the MailMarshal Server can be stopped and disabled. The following list are services that can be safely stopped and disabled without affecting MailMarshal:

Automatic Updates,1 Computer Browser, Cryptographic Services, DHCP Client Distributed File System, Distributed Link Tracking Client, Distributed Transaction Coordinator IPSEC Services, Print Spooler, Remote Registry, Server, Shell Hardware Detection, Task Scheduler, TCP/IP NetBIOS Helper, Windows Time,2 Wireless Configuration, Workstation

IIS Server The following considerations should be made when installing the Operating System for the IIS within the Trusted Network:

1. An IIS Server only needs to be deployed if you intend to implement the MailMarshal End User Spam Management component.

2. Install Windows Server 2003 and ensure that IIS and ASP .Net components are also installed. No other services or functionality provided with Windows are required unless IIS is not the only application on this server.

3. As this server is deployed within the trusted network, any other security considerations or organizational security policy should be configured for this server, such as Anti-Virus, Patch Management etc.

Ports and Services When deploying MailMarshal in the environment described above, certain ports need to be available to allow the services on the server to communicate and provide the functionality they were designed for. In most cases this will require configuration of rules within a sites Internet Firewall to allow inbound and outbound connectivity to Internet services.

Consideration needs to be made with regard to Firewall rules and policy. It is recommended that if possible the MailMarshal server be allowed to receive inbound SMTP from the Internet directly through a NAT or Transparent Proxy rule on the Firewall. This ensures that MailMarshal receives messages from the apparent sender and assists with the detection of Spam and performing reverse DNS lookups on inbound connections.

1 Stop and Disable this Service only if you are not using the Microsoft Automatic Updates Service and you are using some other form of Security Patch Management.

2 Stop and Disable this Service if you are not performing Time Synchronization with Windows Time.

Page 5: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

5

It is also recommended that the MailMarshal server be allowed to deliver SMTP messages directly to the intended destination gateway through a NAT or Transparent Proxy rule on the Firewall. This ensures that the reporting and message tracking facilities of MailMarshal are fully available and messages do not always appear to be delivered to another responsible gateway for delivery.

Firewall Port Requirements The following ports are required to be opened on the firewall to allow MailMarshal services and message flow:

1. SMTP (port 25/TCP) inbound from the Internet to the MailMarshal Server in the DMZ. Depending on the environment this may require a NAT or Transparent Proxy rule to allow SMTP inbound directly to the MailMarshal Server, or may require an SMTP proxy rule if the Firewall or internal policy requires SMTP to have a proxy on the firewall.3

2. SMTP (port 25/TCP) outbound to the Internet from the MailMarshal Server in the DMZ. Depending on the environment this may also require a NAT or Transparent Proxy rule to allow SMTP outbound directly to the destination gateway, or may require an SMTP proxy rule if the Firewall or internal policy requires SMTP to have a proxy on the firewall outbound.3

3. SMTP (port 25/TCP)4 is required to be open in a bi-directional fashion between the MailMarshal Server in the DMZ and the Email Server in the Trusted Network. This may be through NAT or SMTP proxy as site policy dictates.

4. DNS (port 53/TCP, 53/UDP)5 is required outbound from the MailMarshal server in the DMZ and the DNS hosts at your ISP. In some cases this may simply be to redundant DNS servers within the DMZ or on the Firewall itself.

5. MailMarshal Management (port 19001/TCP) is required to be open in a bi-directional fashion between the MailMarshal Server in the DMZ and the Workstations running the MailMarshal Management Console inside the Trusted Network.6

6. Remote Desktop Protocol (port 3389/TCP) is required to allow Terminal Service access between selected workstations in the Trusted Network to the MailMarshal Server in the DMZ. This will allow access to run the MailMarshal Configuration Tool.7

7. HTTP (port 80/TCP) and HTTPS (port 443/TCP) outbound to the Internet from the MailMarshal Server. This is to allow MailMarshal to retrieve Spam Updates from Marshal. Depending on the integrated Anti-Virus used with the MailMarshal, the MailMarshal server may have a requirement to allow these ports outbound to acquire updates from the Anti-Virus vendor’s site as well.

3 If possible it is recommended that a NAT rule is used and no SMTP proxy is enforced by the firewall. See the MailMarshal Ports and Services heading for further clarification.

4 This port can be changed to any port required as the SMTP traffic into the trusted network can use non-standard ports for greater security. The outbound port used must match that used in the local domains configuration in MailMarshal, as well as that defined within the internal email environment for outbound SMTP relay to MailMarshal.

5 DNS is typically only performed using UDP; however if systems have large numbers of defined MX records and the data can not be contained in a single UDP packet then a TCP lookup should be performed.

6 This port can be changed to any port required. The default is 19001. See Appendix A at the end of this document for details on how to change this value.

7 If you use a product other than Terminal Services (such as VNC or PCAnywhere) then change this port to reflect that of the product in use.

Page 6: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

6

8. SQL Server (port 1433/TCP) would be required between selected workstations in the Trusted Network and the MailMarshal Server in the DMZ to facilitate the running of Reporting from the Trusted Network.

Securing MailMarshal By default MailMarshal is very secure. The receiver in particular is well designed and hardened against buffer overrun and other SMTP based attacks. However there are a few additional changes that can optionally be made to increase the security of the gateway.

Anti-Relay protection Anti-Relay protection is on by default in MailMarshal. In fact the only gateways that are able to route messages through MailMarshal that are destined for the Internet are those identified in the Local Domains table in MailMarshal.

In some circumstances inbound messages may be delivered to multiple gateways but outbound messages may only originate from a single apparent host. In this case the Anti-Relay settings can be changed to reflect this.

1. Open the MailMarshal Configurator. On the Tools menu choose Server and Array Properties and click the Anti-Relaying tab.

Ensure that the Prohibit Relaying option is checked. To limit the hosts that have the ability to relay to a specific host, click New.

Page 7: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

7

Enter the IP address of the host you wish to allow to relay, with a Network Mask of 32 bits. Choose to Include as part of the local network and click OK. To specifically exclude any host, or to exclude hosts listed in the Local Domain table from being able to relay, click New again.

2. Enter the IP address of the host you wish to disallow from relaying, with a Network Mask of 32 bits. Choose to Exclude as part of the local network and click OK. Hosts allowed to relay will appear in the Anti-Relay list, with excluded hosts having a cross through them.

Anti-Spoof protection Anti-Spoof protection is on by default in MailMarshal. There is a rule within the Spam and Junk Mail Policy Group called Block Spoofed Messages.

In some circumstances you will be able to modify this rule to force the Anti-Spoof detection to be more rigid.

By default the Anti-Spoofing detection assumes that a message is spoofed if it has a FROM address indicating a local domain, but did not originate from a local server as determined by the anti-relay settings. This can be changed to considering that it is spoofed if it has a FROM address indicating a local domain, but did not come from the IP address associated with that specific local domain (i.e. the specific host for a local domain defined in the local domains table). To change this setting follow the instructions below:

Page 8: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

8

1. Open the MailMarshal Configurator. Open the Email Policy Node and highlight the Anti-Spam Policy Group.

2. Click the Where message spoofing analysis is based on anti-relay link in the Rule Description.

3. Click the Does not match the IP address for that specific domain radio button and then click OK.

4. Save the rule by clicking Finish and then commit the MailMarshal Configuration. Note: Ensure that the settings chosen here match up with you Anti-Relay settings. There is no point setting the Anti-Spoofing to force messages to originate from the specific host assigned for that domain if the Anti-Relay settings only allow outbound messages from a single, or different set of hosts.

Page 9: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

9

Attack Prevention Denial of Service (DoS) and Directory Harvesting Attack (DHA) prevention is built-in and available within MailMarshal 2006.

These functions are not enabled by default as they need some configuration within the organization to be effective.

DoS prevention is implemented by denying connections from IP addresses that make too many connection attempts in a specified amount of time.

DHA prevention is achieved by denying connections to IP addresses that exceed a defined number of invalid recipients in messages within a certain timeframe.

To configure the settings fort DoS and DHA follow the instructions below:

1. Open the MailMarshal Configurator. On the Tools menu choose Server and Array Properties and click the Attack Prevention tab.

2. The first configurable value is that of the total number of recipients per message. The default is 250 and should be more than adequate for most environments.

3. To enable DoS prevention, check the Enable DoS prevention checkbox. The default values of 10 connections from an individual host in 60 seconds will only be adequate for small to medium environments and as such you may need to increase the number of connections allowed, or extend the period.

Page 10: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

10

4. To enable DHA prevention check the Enable DHA prevention checkbox. You will be prompted to ensure that the All Employees group is populated. Click OK.

5. Before you commit any configuration changes you MUST ensure that the default All Employees group is populated with groups from Active Directory, or LDAP that contain the aliases for all users in your organization. MailMarshal uses this group to determine valid email address aliases for users within your company. In a simple Active Directory environment this would simply contain the Domain Users group for example.

6. The next three settings allow you to define how many invalid recipients in messages from the same host will be accepted within the evaluation period. You can also define for how long the sending host will have connection attempts blocked for if they breach the policy.

7. By clicking on the Safe List you can define IP addresses or subnets that are excluded from DoS and DHA analysis. A good example of where this may be necessary is ensuring that branch offices that may exchange large volumes of email over SMTP are not inadvertently blocked by the DoS policy.

Receiver Greeting When a foreign gateway connects to an SMTP gateway, the gateway responds initially with a greeting message. This message usually identifies that it is an SMTP gateway and the product name and version details of the gateway.

Page 11: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

11

The default greeting for a MailMarshal server is similar to this:

220 mail1.companyname.com ESMTP MailMarshal (v6.1.4.446) Ready

In many circumstances this is not appropriate, as the information provided gives a potential attacker two pieces of initial information. That being the Product being used, and the version number. This greeting can be changed to be generic and not display this information.

1. Open the MailMarshal Configurator. On the Tools menu choose Server and Array Properties and click the Advanced tab. Choose Additional Options and then click the Receiver tab.

2. In the Greeting string field enter something similar to: {Hostname} ESMTP Gateway Ready For reliability in communication with other gateways it is best to start the greeting with the Fully Qualified hostname and ESMTP.

3. Click OK.

Received Stamp When a SMTP gateway processes a message it stamps the message with a header that identifies what gateway it is, the date and time, product employed and version number if the gateway.

The default stamp for a MailMarshal server is similar to this:

Received: from {HelloName} ({VerifiedHost}[{SenderIp}]) by {Hostname} with {ProductName} (v{ProductVersion})

In many circumstances you may also not wish to advertise the product or version information of this gateway in the Received: stamp. This stamp can be changed to be generic and not display this information if desired.

Page 12: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

12

1. Open the MailMarshal Configurator. On the Tools menu choose Server and Array Properties and click the Advanced tab. Choose Additional Options and then click the Receiver tab.

2. In the Received header field enter something similar to: Received: from {HelloName} ({VerifiedHost}[{SenderIp}]) by {Hostname} with ESMTP Gateway For reliability in communication with other gateways it is best to leave the format of the first part of this header as is and only change the Product Name and Version information.

3. Click OK.

Securing Server Networking As well as the protection offered by the firewall between the MailMarshal Server in the DMZ and the Internet, it is prudent to secure the network interface of this server with the built-in firewall functionality offered by Windows Server 2003.

The only ports required to be opened inbound to the MailMarshal Server are SMTP (25/TCP), MailMarshal Management (19001/TCP), SQL Server (1433/TCP) and RDP (3389/TCP).

There are a number of optional changes that can be made to the MailMarshal Receiver to afford additional security, and these will be described below.

Additionally there are other registry based settings that can be selected to provide additional security from attack.

Network Interface Firewall The following details how to secure the opened ports on the Network interface:

1. On the MailMarshal Server open Network Connections, Right Click the Local Area Connection icon and choose Properties.

Page 13: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

13

2. Click the Advanced Tab and select the Protect my computer and network by limiting or preventing access to this computer from the Internet option.

3. Click Settings and check the box beside Internet Mail Server (SMTP). In the dialog that opens enter the name of this server into the field provided and click OK.

Page 14: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

14

4. Click Add. In the Service Settings dialog enter values similar to the ones below; however use the name of this server in the Name or IP address field. Click OK.

5. Click Add. In the Service Settings dialog enter values similar to the ones below; however use the name of this server in the Name or IP address field. Click OK.

Page 15: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

15

6. Click Settings and check the box beside Remote Desktop. In the dialog that opens enter the name of this server into the field provided and click OK.

7. Once this is completed you should have only these four items in the Advanced Settings dialog.

Page 16: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

16

8. Select the ICMP Tab. Ensure that all of the ICMP protocol options are not selected. If any of these are required at a later date for diagnostic purposed they can be enabled at that time and disabled once the diagnostic process is complete. Click OK.

9. Click OK to close the Local Area Connection properties and apply these settings.

Network Client Settings

The following details how to remove additional Network services from operating on the MailMarshal Server. MailMarshal does not require these services, and only needs the TCP/IP Protocol:

Page 17: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

17

1. On the MailMarshal Server open Network Connections, Right Click the Local Area Connection icon and choose Properties.

2. Ensure that only the Internet Protocol (TCP/IP) item is selected in the General properties page. Click OK.

SYN flooding attack protection The SYN flooding attack protection feature of the TCP stack detects symptoms of SYN flooding and responds by reducing the time a server spends on connection requests that it cannot acknowledge.

Specifically, the TCP stack shortens the required interval between SYN-ACK (connection request acknowledgements) retransmissions. (TCP retransmits SYN-ACKS when they are not answered.) As a result, the allocated number of retransmissions is consumed quicker and the un-acknowledgeable connection request is discarded faster:

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Create a DWORD value called SynAttackProtect with a value of 1.

3. Create a DWORD value called TcpMaxConnectResponseRetransmissions with a value of 2.

4. Create a DWORD value called TCPMaxPortsExhausted with a value of 5.

ICMP redirects ICMP redirects can override OSPF generated routes. Internet Control Message Protocol (ICMP) redirects cause the stack to plumb host routes. These routes override the Open Shortest Path First (OSPF) – generated routes.

This behavior is expected; the problem is that the 10 minute time – out period for the ICMP redirect – plumbed routes temporarily creates a black hole for the network where traffic will no longer be routed properly for the affected host.

Page 18: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

18

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Modify the DWORD value called EnableICMPRedirect with a value of 0.

Dead Gateway Detection Allow automatic detection of dead network gateways could lead to Denial of Service. When dead – gateway detection is enabled, TCP may ask the IP to change to a backup gateway if a number of connections are experiencing difficulty.

An attacker could force the server to switch gateways, potentially to an unintended one.

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Ensure that the DWORD value called EnableDeadGWDetect has a value of 0.

MTU Discovery Automatic detection of maximum transmission unit (MTU) size is the default setting in Windows. When this value is enabled, the default setting, the TCP stack tries to automatically determine either the maximum transmission unit (MTU) or the largest packet size over the path to a remote host.

An attacker could force the MTU to a very small value and overwork the stack by forcing the server to fragment a large number of packets.

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Create a DWORD value called EnablePMTUDiscovery with a value of 0.

Keep Alive Time This value determines how often keep-alive packets are sent in milliseconds. Keep-alive packets control how often TCP attempts to verify that an idle connection is still intact by sending a keep–alive packet. If the remote computer is still reachable, it acknowledges the keep–alive packet.

An attacker who is able to connect to network applications could cause a Denial of Service condition by establishing numerous connections. (300,000 milliseconds is recommended).

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Create a DWORD value called KeepAliveTime with a value of 300000 (decimal).

IP Source Routing

IP source routing protection level (protects against packet spoofing. IP source routing is a mechanism allowing the sender to determine the IP route that a datagram should take through the network.

An attacker could use source routed packets to obscure their identity and location. Source routing allows a computer sending a packet to specify the route it takes.

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Create a DWORD value called DisableIPSourceRouting with a value of 2.

Page 19: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

19

Max Data Re-Transmissions This value determines how many times unacknowledged data is retransmitted. This parameter controls the number of times that TCP retransmits an individual data segment (non–connect segment) before aborting the connection. The retransmission time – out is doubled with each successive retransmission on a connection. It is reset when responses resume. The base time – out value is dynamically determined by the measured round–trip time on the connection.

A malicious user could exhaust resources by never sending any acknowledgement messages for data transmitted by the target computer.

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Create a DWORD value called TcpMaxDataRetransmissions with a value of 3.

Router Discovery

This value can allow Internet Router Discovery Protocol (IRDP) to detect and configure Default Gateway addresses (this could lead to a Denial of Service). IRDP allows the system to detect and configure Default Gateway addresses automatically.

An attacker who has gained control of a system on the same network segment could configure a computer on the network to impersonate a router. Other computers with IRDP enabled would then attempt to route their traffic through the already compromised system.

1. Start Regedit and open the key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Create a DWORD value called PerformRouterDiscovery with a value of 0.

Page 20: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

20

Appendix A This appendix provides addition details to functions referred to in the preceding document.

Changing the MailMarshal Management Port Use the following steps to change the MailMarshal management port from the default of 19001 to your own custom value. This will be required if you wish to use a different port between the MailMarshal Console in the Trusted Network and the MailMarshal Server in the DMZ. Note that if you change this value then all references to the Management Port in this document should be replaced with your custom value:

1. On the MailMarshal Array Manager server open the MailMarshal Server Tool8.

2. Go to the Array / Node Communication tab and click Change.

3. Enter the port9 that you wish to use for Array to Node communication. Click OK, and then click OK to close the MailMarshal Server Tool. When prompted to re-start the services choose Yes.

4. On the MailMarshal Server open the MailMarshal Server Tool.

5. Go to the Array / Node Communication tab and click Change.

8 This tool can be found by clicking on Start / All Programs / NetIQ MailMarshal / MailMarshal Tools and choosing the MailMarshal Server Tool icon.

9 Ensure that the port entered is not use by any other service on the Array Manager server or the node. Confirm this by running netstat from a command prompt on each server and ensuring that the port is not active.

Page 21: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

21

6. Enter the same port2 that you used on the Array Manager into the Array Manager port and the Node port fields. Click OK, and then click OK to close the MailMarshal Server Tool. When prompted to re-start the services choose Yes.

7. When you next open the MailMarshal Console or the MailMarshal Configurator they will, indicate that the Array Manager is not running. Click OK and then enter your new port number into the Connect to MailMarshal Server dialog.

8. If you have the MailMarshal Web Components installed you will also need to go to the Web Server where they are installed and run the MailMarshal Web Configuration Tool10.

10 This tool can be found by clicking on Start / All Programs / NetIQ MailMarshal and choosing the MailMarshal Web Configuration Tool icon.

Page 22: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

22

9. On both the Spam Quarantine Management tab and the Admin Web Console Tab change the Port value to your new port. Click OK.

Page 23: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone

23

References Additional information regarding information contained in this document can be located at the following locations:

Securing a Windows 2003 Server http://www.microsoft.com/technet/security/prodtech/windowsserver2003/secmod119.mspx

Windows Server 2003 Security Guide http://www.microsoft.com/downloads/details.aspx?FamilyId=8A2643C1-0685-4D89-B655-521EA6C7B4DB&displaylang=en

Threats and Countermeasures Guide http://www.microsoft.com/downloads/details.aspx?FamilyID=1b6acf93-147a-4481-9346-f93a4081eea8&DisplayLang=en

Windows TCP/IP Registry parameters http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/DepKit/56a33a88-a7b2-4f21-ab5e-5c62d728619f.mspx

Page 24: Securing Windows 2003 Server for MailMarshal 2006 ......WHITEPAPER – MailMarshal SMTP 2006 – Securing Windows 2003 Server Standalone 3 For the purpose of this document it is also

Marshal’s Worldwide and EMEA HQ Marshal Limited, Renaissance 2200, Basing View, Basingstoke, Hampshire RG21 4EQ United Kingdom Phone: +44 (0) 1256 848080 Fax: +44 (0) 1256 848060 Email:[email protected]

Americas Marshal Inc. 5555 Glenridge Connector, Suite 200, Atlanta, GA 30342 USA Phone: +1 404 459 2890 Fax +1 404 759 2549 Email: [email protected]

Asia-Pacific Marshal Software (NZ) Ltd Suite 1, Level 1, Building C Millennium Centre 600 Great South Road Greenlane, Auckland New Zealand Phone: +64 9 984 5700 Fax: +64 9 984 5720 Email: [email protected]

[email protected] | www.marshal.com

24

THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT ARE FURNISHED UNDER AND ARE SUBJECT TO THE TERMS OF A LICENSE AGREEMENT OR A NON-DISCLOSURE AGREEMENT. EXCEPT AS EXPRESSLY SET FORTH IN SUCH LICENSE AGREEMENT OR NON-DISCLOSURE AGREEMENT, MARSHAL LIMITED PROVIDES THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. SOME JURISDICTIONS DO NOT ALLOW DISCLAIMERS OF EXPRESS OR IMPLIED WARRANTIES IN CERTAIN TRANSACTIONS; THEREFORE, THIS STATEMENT MAY NOT APPLY TO YOU.

This document and the software described in this document may not be lent, sold, or given away without the prior written permission of Marshal, except as otherwise permitted by law. Except as expressly set forth in such license agreement or non-disclosure agreement, no part of this document or the software described in this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, or otherwise, without the prior written consent of Marshal. Some companies, names, and data in this document are used for illustration purposes and may not represent real companies, individuals, or data. This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. Marshal may make improvements in or changes to the software described in this document at any time.

© 2006 Marshal Limited, all rights reserved.

U.S. Government Restricted Rights: The software and the documentation are commercial computer software and documentation developed at private expense. Use, duplication, or disclosure by the U.S. Government is subject to the terms of the Marshal standard commercial license for the software, and where applicable, the restrictions set forth in the Rights in Technical Data and Computer Software clauses and any successor rules or regulations.

Marshal, MailMarshal, the Marshal logo, WebMarshal, Security Reporting Center and Firewall Suite are trademarks or registered trademarks of Marshal Limited or its subsidiaries in the United Kingdom and other jurisdictions. All other company and product names mentioned are used only for identification purposes and may be trademarks or registered trademarks of their respective companies.