Have you ever come across a server that is doing a lot of traffic? Maybe you have logged in to see a process running at 100% CPU, so you know the culprit, but instead of kill -9ing it, wouldn’t it be great to see what exactly it is up to? Or even if you see a process and don’t know exactly what it is doing, and you are just curious what it is up to?

terminal-icon-64x64As with most issues there are several ways to skin this cat. You can use tcpdump or wireshark to sniff the all of the network traffic on the device. If you know the port the program is running on (you can use lsof for that), you can restrict traffic to that port. But what if the program is jumping ports, or even uses a side-port for some sort of data transmission (UDP?).

The main problem going down this route is that on a server that is doing any significant bit of traffic, it is like sorting through a needle in a haystack. If you have a single process that is taking up all of your bandwidth, you can probably find it pretty fast. But if the process is not doing a ton of traffic it can be hard to track down.

Strace to the rescue

You can use the great program strace to sniff the network data that an executed program is doing, or even a currently running program. This works well because if you are trying to isolate the network traffic a currently running process, your options can be limited. Using strace is the only way that I know of to see ALL of the traffic coming from a process.

To check the traffic of a currently running process X:

strace -p X -f -e trace=network -s 10000

The command breaks down:

  • -p: process ID
  • -f: follow forks
  • -e: follow set of system calls. In our case, we use trace=network, which follows network system calls.
  • -s: set output string sizes. default is 32, which does not  give a lot of information.

Finally if you have a new program to execute and you want to watch the network traffic on it, you execute that command with strace. This would be good to use if you work in a highly secure environment and need to find out what sort of network traffic a distributed binary is doing. Checking for a program ‘Phoning home’ is a good example of that.

Here is the command that launches a new process:

strace -f -e trace=network -s 10000 /usr/bin/command arguments

Hopefully using strace in this manner will help you debug some issues on your server – I know I have used it several times.

You May Also Like

Visual.Syntax is my choice for code highlighting

I am using the Visual.Syntax code highlighting plugin by Matthew Delmarter. There…

Tweaking TCP for Fast (100mbps+) Connections and Transfers on Linux

We recently did some speed testing on a few of the servers on our network, and we were not receiving the speeds expected considering they were sitting on a physical 100mbps ethernet port. The servers were indeed on physical 100mbps connection, however wget (TCP/IP, HTTP Port 80) download tests showed only a max of about 1.5MB/sec (note the 8bit/byte conversion, so this translates to about 12mbits).

Joomla! 1.0.12 Double Pathway Bug

Joomla! 1.0.12 appears to have a pathway bug. If you use a…

Default Grub Boot Commands for Ubuntu 7.10

I recently formatted my laptop and installed Windows first, using half of…