Have you ever come across a server that is doing a lot of traffic? Maybe you have logged in to see a process running at 100% CPU, so you know the culprit, but instead of kill -9ing it, wouldn’t it be great to see what exactly it is up to? Or even if you see a process and don’t know exactly what it is doing, and you are just curious what it is up to?
As with most issues there are several ways to skin this cat. You can use tcpdump or wireshark to sniff the all of the network traffic on the device. If you know the port the program is running on (you can use lsof for that), you can restrict traffic to that port. But what if the program is jumping ports, or even uses a side-port for some sort of data transmission (UDP?).
The main problem going down this route is that on a server that is doing any significant bit of traffic, it is like sorting through a needle in a haystack. If you have a single process that is taking up all of your bandwidth, you can probably find it pretty fast. But if the process is not doing a ton of traffic it can be hard to track down.
Strace to the rescue
You can use the great program strace to sniff the network data that an executed program is doing, or even a currently running program. This works well because if you are trying to isolate the network traffic a currently running process, your options can be limited. Using strace is the only way that I know of to see ALL of the traffic coming from a process.
To check the traffic of a currently running process X:
strace -p X -f -e trace=network -s 10000
The command breaks down:
- -p: process ID
- -f: follow forks
- -e: follow set of system calls. In our case, we use trace=network, which follows network system calls.
- -s: set output string sizes. default is 32, which does not give a lot of information.
Finally if you have a new program to execute and you want to watch the network traffic on it, you execute that command with strace. This would be good to use if you work in a highly secure environment and need to find out what sort of network traffic a distributed binary is doing. Checking for a program ‘Phoning home’ is a good example of that.
Here is the command that launches a new process:
strace -f -e trace=network -s 10000 /usr/bin/command arguments
Hopefully using strace in this manner will help you debug some issues on your server – I know I have used it several times.
