Tweaking TCP for Fast (100mbps+) Connections and Transfers on Linux

We recently did some speed testing on a few of the servers on our network, and we were not receiving the speeds expected considering they were sitting on a physical 100mbps ethernet port. The servers were indeed on physical 100mbps connection, however wget (TCP/IP, HTTP Port 80) download tests showed only a max of about 1.5MB/sec (note the 8bit/byte conversion, so this translates to about 12mbits).

This is due to how TCP frames data packets and optimizes them for connections. I believe by default TCP on most systems assumes about a 10mbit max capable transfer rate, so it does not show performance gains on a larger pipe without modification to the kernel options which govern TCP/IP frame size and features. Some distributions may make this change for you automatically however many will not.

To keep things short and sweet, we took the following advice from Speedguide.net on tweaking TCP parameters on linux kernel systems. This will cover Linux 2.1 and above – which means CentOS, RedHat, Ubuntu, Debian and many more distributions.

The TCP Parameters we will change are:

• /proc/sys/net/core/rmem_max – Maximum TCP Receive Window
• /proc/sys/net/core/wmem_max – Maximum TCP Send Window
• /proc/sys/net/ipv4/tcp_timestamps – (RFC 1323) timestamps add 12 bytes to the TCP header…
• /proc/sys/net/ipv4/tcp_sack – tcp selective acknowledgements.
• /proc/sys/net/ipv4/tcp_window_scaling – support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535.

If you recall /proc/ is the volatile portion of kernel configuration, you can change it on the fly but it will be reset on reboot unless settings are changed via an init file or setting the options in /etc/sysctl.conf. To change the settings once (to test):

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

And to apply them for good, add the following lines to /etc/sysctl.conf:

net.core.rmem_default = 256960
net.core.rmem_max = 256960
net.core.wmem_default = 256960
net.core.wmem_max = 256960
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1

Use ‘sysctl -p’ to apply the changes in this file to your running Linux instance. Feel free to experiment with these numbers to see how they impact your transfers, it depends a lot on how many and how large the files are that you transferring. These changes must be made on the SERVER side, any change on the client side would not impact the download speed from the server.

There are several other variables to consider, and these all depend on your application so change them if you know what you are attempting to do. After changing these settings, you will see speeds of about 10MB/sec (80mbps) on a 100mbps connection. The other 20mbps are lost in TCP and other network layer overhead, which is unavoidable.