Shell code samples (BASH)

How to Install SNMP on Tomato Router Firmware and Graph Traffic with Cacti

You’ve flashed your old WRT54G or other vanilla router with the Tomato firmware. This itself turns your router into a lean, mean routing machine with QOS, SSH and more, but let’s say we want to take it a bit further. What it we want to get some more stats out of it?

In order to do this, we first need to set up a way to pull this information from the router. The best way to do this is to install an SNMP (Simple Network Management Protocol) daemon on the system.

The main roadblock we face here is that the system mainly runs in volatile system memory, meaning that every time the system is rebooted the filesystem is reset. Fortunately Tomato provides a way to get around this using CIFS shares. Follow the steps below (as modified from here) to install an SNMP server on a Tomato router.

  1. Create a network (samba, CIFS) share somewhere on the network. This computer must be on all of the time in order for Tomato to run the SNMP server.
  2. Download the file from one of these locations:

    expand the binary and .conf file into the share or a subdirectory (for example, <share name>/snmp)

    MD5 for snmpd binary is ae0d622648efdb8dceb7b3b5a63e23ac

  3. Set up the shared directory on the router. Visit Administration->CIFS Client. Add the share as follows, with your correct share information:cifs1
  4. Log into the Tomato router via ssh, and start SNMPd on the router by issuing the command:
    /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf &
  5. Test that SNMP is running and can be accessed on another computer on the network. To test it, you can use snmpwalk like so:
    snmpwalk -c public -v 2c <IP Address of Router>

    If it works properly, it will list the available OIDs from the router. You do not need to take note of these, but they will be used in the graphing software later.

  6. Finally, we need to launch the SNMP server when the router is restarted. You do this by adding the command to start it in the area Administration -> Scripts -> Firewall:
    sleep 30
    /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf -s &

    This launches the snmp server 30 seconds after the router is started or rebooted.

Thats it! SNMP is now running on the router.

Now to add this SNMP host to your graphing software. For this example, I will use Cacti, which I will assume you have already set up. If you need to set it up, please follow the directions on the Cacti site for installation.

First, add the router as a new device, using the information below (change IP to suite your needs):


After adding the device, you have several options depending on what sort of data you are looking for. For system information on the router – for example CPU usage, memory usage, etc; you can go directly to Create -> New Graphs. Select your device and then add the graph you are looking for.

The graph will show as a broken image at first, or a blank graph with “NaN” as the data source. Give it a few minutes to update, and the information should start to flow through. The ucd/net options work best, but feel free to experiment.

To get traffic stats on the interface, you first need to “Walk” the device.  Go back to your device list, and edit the device you added. Under “Associated Data Queries”, Add Data Query, add “SNMP – Interface Statistics” with Re-Index period as “Uptime goes backwards”. After adding it you should see under status something like: Success [39 Items, 6 Rows].

Since these data sources are now added, you can go back to Add a new Graph. After selecting the device, you should see a list of these new interfaces. Select the interfaces you wish to graph, and select the graph type (I suggest In/Out bits with Total).

After a few minutes, the data should start filling in. After a while, you will get a graph like this:


In conclusion, with a little work, you can get enterprise class graphing from your consumer router. The total project took me about 45 minutes, and I was trying to figure out all of the data sources and the correct way to enter everything.

Let me know your experiences, suggestions and corrections!

Command Line Packet Sniff Existing Running Process in Linux

Have you ever come across a server that is doing a lot of traffic? Maybe you have logged in to see a process running at 100% CPU, so you know the culprit, but instead of kill -9ing it, wouldn’t it be great to see what exactly it is up to? Or even if you see a process and don’t know exactly what it is doing, and you are just curious what it is up to?

terminal-icon-64x64As with most issues there are several ways to skin this cat. You can use tcpdump or wireshark to sniff the all of the network traffic on the device. If you know the port the program is running on (you can use lsof for that), you can restrict traffic to that port. But what if the program is jumping ports, or even uses a side-port for some sort of data transmission (UDP?).

The main problem going down this route is that on a server that is doing any significant bit of traffic, it is like sorting through a needle in a haystack. If you have a single process that is taking up all of your bandwidth, you can probably find it pretty fast. But if the process is not doing a ton of traffic it can be hard to track down.

Strace to the rescue

You can use the great program strace to sniff the network data that an executed program is doing, or even a currently running program. This works well because if you are trying to isolate the network traffic a currently running process, your options can be limited. Using strace is the only way that I know of to see ALL of the traffic coming from a process.

To check the traffic of a currently running process X:

strace -p X -f -e trace=network -s 10000

The command breaks down:

  • -p: process ID
  • -f: follow forks
  • -e: follow set of system calls. In our case, we use trace=network, which follows network system calls.
  • -s: set output string sizes. default is 32, which does not  give a lot of information.

Finally if you have a new program to execute and you want to watch the network traffic on it, you execute that command with strace. This would be good to use if you work in a highly secure environment and need to find out what sort of network traffic a distributed binary is doing. Checking for a program ‘Phoning home’ is a good example of that.

Here is the command that launches a new process:

strace -f -e trace=network -s 10000 /usr/bin/command arguments

Hopefully using strace in this manner will help you debug some issues on your server – I know I have used it several times.

Error While Mounting an ISO Image in Linux: ioctl: LOOP_SET_FD: Invalid argument

I recently received this error while mounting an iso image in a CentOS 5.3 install:

[root@host ~]# mount -t iso9660 -o loop /mnt/glusterfs/ecp-spe-4867.iso /mnt/iso/
ioctl: LOOP_SET_FD: Invalid argument

For more detail:

[root@host ~]# mount -v -t iso9660 -o loop /mnt/glusterfs/ecp-spe-4867.iso /mnt/iso/
mount: going to use the loop device /dev/loop0
ioctl: LOOP_SET_FD: Invalid argument
mount: failed setting up loop device

So what could this mean? The confusing error message “ioctl: LOOP_SET_FD: Invalid argument” means that your ISO image is on a filesystem that is not supported for the loopback device on your system. In my case, that was a GlusterFS mount that was hosting the image.

I copied it into another directory on my root ext3 filesystem and it mounted just fine!

Ubuntu Server in Place Network Upgrade From 8.10 to 9.04

Ubuntu Upgrade

It is easy to do an in-place upgrade of Ubuntu Server from 8.10 ‘Intrepid Ibex‘ to 9.04 ‘Jaunty Jackalope‘. You can do this remotely over ssh or whatever you use to control your server. Best practices say to make sure to backup your server before doing the upgrade. I’ve done several servers this way with no issues!

Issue the command:

sudo apt-get update; sudo apt-get upgrade; sudo apt-get install update-manager-core; sudo do-release-upgrade

Follow any prompts to first upgrade the current distribution with the newest packages, then do the release upgrade.

Easy Search and Replace in Multiple Files on Linux Command Line

I recently came across a typo that existed in a bunch of html files on my web server. I thought it should be easy enough to change, but since it was in a number of files, editing it by hand would be time consuming. Fortunately, there is an easy, one liner command to replace the text in multiple files in a sub directory using recursion.

grep -lr -e '<oldword>' * | xargs sed -i 's/<oldword>/<newword>/g'

This command broken down:

  • grep for the word in a files, use recursion (to find files in sub directories), and list only file matches
  • | xargs passes the results from the grep command to sed
  • sed -i uses a regular expression (regex) to evaluate the change: s (search) / search word / target word / g (global replace)

For more information, see man pages for grep, sed, and xarg. Also it is very handy to learn about regular expressions as they are a valuable tool to any command line programmer!

Update 2009/7/19:

Thanks to reader btr we have a great one-line perl command that will perform the same task:

Perl provides a really nice one-line for this kind of thing:

perl -p -i -e ’s///g’ *

It also provides the option of creating a backup of each file changed:

perl -p -i.bak -e ’s///g’ *

mnemonic: PIE (”easy as pie”, etc.)
google “perl pie” and you’ll get lots of info for other uses of this technique.

How To Turn Off Your Monitor Via Command Line in Ubuntu

As previously written on this blog, I have set up a display in our lobby at work to display the day’s current events and meetings using Ubuntu and a tiny PC. Since this is a display which is on all day, the screensaver and monitor blanking (and other Energy Star features) are all turned off.

Under the auspice of wanting to save energy and also extending the life of a new monitor, someone suggested that we turn off the monitor at night using an electrical timer. A lightbulb went off in my head, that there must be a better way to do this via command line and then run it in the cron.

It turns out the solution is very simple. The xset command is the X server preferences command. It has a simple command to turn off the monitor:

$ xset dpms force off

and to turn the monitor back on:

$ xset dpms force on

You can also check the status of the X server settings by using:

$ xset -q

Also, when dpms turns off the monitor, it will turn back on after a keypress or by moving the mouse. Since this is a lobby display, there is no keyboard or mouse installed in the system.

I’ve rolled this into a little bash script with on, off, and status switches:

export DISPLAY=:0.0

if [ $# -eq 0 ]; then
  echo usage: $(basename $0) "on|off|status"
  exit 1

if [ $1 = "off" ]; then
  echo -en "Turning monitor off..."
  xset dpms force off
  echo -en "done.\nCheck:"
  xset -q|grep "Monitor is"
elif [ $1 = "on" ]; then
  echo -en "Turning monitor on..."
  xset dpms force on
  echo -en "done.\nCheck:"
  xset -q|grep "Monitor is"
elif [ $1 = "status" ]; then
  xset -q|sed -ne 's/^[ ]*Monitor is //p'
  echo usage: $(basename $0) "on|off|status"

You can then use cron to turn off the monitor at night, and back on in the morning:

0 20 0 0 0 /home/lobby/ off
0 7 0 0 0 /home/lobby/ on

This script will turn it off at 8pm and back on at 7am.

Note that this was written for an Ubuntu system, but the xset command is pretty generic so any system that runs Xserver like RedHat, CentOS, Debian, Fedora, etc should be able to use the script as well.

Apt-get Update GPG Key Errors and Fix

Running sudo apt-get upgrade, I started getting this error:

Reading package lists... Done

W: GPG error: intrepid Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 313D312748A22A95

W: You may want to run apt-get update to correct these problems

Ah ha! But apt-get update is the command causing this problem.

The solution is to import this key from the gpg servers; I don’t know why this isn’t done automatically, but here is it:

sudo apt-key adv --recv-keys --keyserver 313D312748A22A95; gpg --export --armor 313D312748A22A95 | sudo apt-key add -

Resulting in:

Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --recv-keys --keyserver 313D312748A22A95
gpg: requesting key 48A22A95 from hkp server
gpg: key 48A22A95: public key "Launchpad PPA for Filip Brcic" imported
gpg: Total number processed: 1
gpg: unchanged: 1

Congrats! sudo apt-get update now works properly!