I do however have access to a linux machine on campus. I am familiar with port forwarding using SSH but I had never used it to actually tunnel web traffic using a web browser on Windows. Turns out it is surprisingly easy!

The ssh command to use is:

This command sshes to the remote_host, and creates a tunnel on your localhost, port 8080. Note that you need to have private key authentication already set up for this host – it will not work with password authentication.

The description of the switches are (from the ssh man page):

• -C : Compression
• -2 : Use SSHv2
• -q : quiet!
• -T : Disable pseuto-tty allocation
• -n : Prevents reading from stdin (you need to have private key authentication set up, to prevent password authentication)
• -N : Do not execute a remote command (or launch a shell). Just use the ssh process for port forwarding
• -D : Allocate a socket to listen on the local side. When a connection is made to this port it is located to the remote machine. Makes SSH work as a SOCKS server. Only root can forward privileged ports like this.

From here, you set up Firefox or your browser of choice to use a Socks proxy on localhost:8080. The man page says that SOCKS4 and SOCK5 should both work but I had to use SOCKS v4, SOCKS v5 did not seem to work for me.

As with most issues there are several ways to skin this cat. You can use tcpdump or wireshark to sniff the all of the network traffic on the device. If you know the port the program is running on (you can use lsof for that), you can restrict traffic to that port. But what if the program is jumping ports, or even uses a side-port for some sort of data transmission (UDP?).

The main problem going down this route is that on a server that is doing any significant bit of traffic, it is like sorting through a needle in a haystack. If you have a single process that is taking up all of your bandwidth, you can probably find it pretty fast. But if the process is not doing a ton of traffic it can be hard to track down.

Strace to the rescue

You can use the great program strace to sniff the network data that an executed program is doing, or even a currently running program. This works well because if you are trying to isolate the network traffic a currently running process, your options can be limited. Using strace is the only way that I know of to see ALL of the traffic coming from a process.

To check the traffic of a currently running process X:

strace -p X -f -e trace=network -s 10000

The command breaks down:

• -p: process ID
• -f: follow forks
• -e: follow set of system calls. In our case, we use trace=network, which follows network system calls.
• -s: set output string sizes. default is 32, which does not  give a lot of information.

Finally if you have a new program to execute and you want to watch the network traffic on it, you execute that command with strace. This would be good to use if you work in a highly secure environment and need to find out what sort of network traffic a distributed binary is doing. Checking for a program ‘Phoning home’ is a good example of that.

Here is the command that launches a new process:

strace -f -e trace=network -s 10000 /usr/bin/command arguments

Hopefully using strace in this manner will help you debug some issues on your server – I know I have used it several times.

grep -lr -e '<oldword>' * | xargs sed -i 's/<oldword>/<newword>/g'

This command broken down:

• grep for the word in a files, use recursion (to find files in sub directories), and list only file matches
• | xargs passes the results from the grep command to sed
• sed -i uses a regular expression (regex) to evaluate the change: s (search) / search word / target word / g (global replace)

For more information, see man pages for grep, sed, and xarg. Also it is very handy to learn about regular expressions as they are a valuable tool to any command line programmer!

Update 2009/7/19:

Thanks to reader btr we have a great one-line perl command that will perform the same task:

Perl provides a really nice one-line for this kind of thing:

It also provides the option of creating a backup of each file changed:

mnemonic: PIE (”easy as pie”, etc.)
google “perl pie” and you’ll get lots of info for other uses of this technique.

http://www.linux.org/lessons/short/perlpie/perl_pie.html

Zabbix is an excellent system monitoring package. It does everything from basic availability checking to detailed system resource analysis. It is capable of graphing the variables pulled from the system, and alerting admins if there is a problem or something needed for attention.

Once you have the Zabbix server set up, you need to install the client on any systems you want to monitor. Windows systems have a precompiled binary to install. On linux, unix or freebsd systems you’ll need to compile binaries. If you have a range of systems that are homogeneous, you can port the binary to those or also compile it with static dependencies. Below are steps to compile, configure and install the zabbix client:

Steps to install a Zabbix Client

1. Download zabbix source code from www.zabbix.com; decompress with ‘tar zxvf’ and cd to directory
2. Configure the make program: ./configure –enable-agent
3. Compile and install the program: make install
4. Add zabbix group and user: groupadd zabbix; adduser -g zabbix -s /sbin/nologin -M -p RANDOMPASS zabbix
5. Create log file: touch /var/log/zabbix_agentd.log; chown zabbix.zabbix /var/log/zabbix_agentd.log
6. Copy init script to /etc/init.d. Scripts are located in ./misc/init.d/ and your distro directory.
7. Make sure bin directory in init script is where Zabbix actually compiled to.
8. chmod 755 /etc/init.d/zabbix_agentd
9. chkconfig zabbix_agentd on
10. Copy agent config script to /etc/zabbix/zabbix_agentd.conf. Current one is:

    # This is config file for zabbix_agentd
    # To get more information about ZABBIX, go http://www.zabbix.com
    
    # This is the ip and port of the main zabbix server
    Server=10.0.0.1
    ServerPort=10051
    Hostname=AGENTHOSTNAME
    ListenPort=10050
    # ListenIP=10.0.0.2
    StartAgents=5
    DisableActive=1
    DebugLevel=1
    PidFile=/var/tmp/zabbix_agentd.pid
    # LogFile=/var/log/zabbix_agentd.log
    LogFileSize=1
    Timeout=10

11. Start the zabbix service: service zabbix_agentd start
12. Open firewall for zabbix port (10050) if necessary.
13. Log into Zabbix on the server, Add server to hosts – use correct templates and groups depending on what type of server it is.
14. Add monitoring and notification as appropriate.
15. Consider if all necessary services are being monitored; test that detection of down services and notifications work properly.

I checked the memory, and sure enough, all of the VPSes were using up the memory. More specifically, dom0 (the main server) had ‘ballooned’ and begun to take up 3GB of RAM, this is more than the system should need:

Basically when memory ballooning is set, the memory available to the main hypervisor will have a minimum value set and will balloon according to the current requirements. This setting is in the config file which for me is at /etc/xen/xend-config.sxp:

I changed this line to:

I also modified grub to set an absolute amount of memory to dom0:

The value for dom0_mem sets the system memory at boot time. This will be set on the kernel line. Make sure this is large enough to handle system tasks; 1.5G seems to be good for me right now.

Reboot the server to set the memory for dom0.

You could also set the memory usage for dom0 temporarily by using the ‘xm mem-set’ command:

This would not apply after a reboot. Good luck!