After figuring out that I had 2 out of 3 of my 1TB Seagate Barracuda hard drives in the array fail, I had to give the entire array up for a loss and test out my backup strategy. Fortunately it worked and there was no data loss. After receiving the replacement drives in from Seagate, I rebuilt the ZFS array (using raidz again) and went along my merry way. After another 6 months or so, I started getting some funky results from my other drive. Thinking it might have some issue as with the others, I removed the drive and ran Seatools on it (by the way, Seatools doesn’t offer a 64-bit Windows version – what year is this?).

The drive didn’t show any signs of failure, so I decided to wipe it and add it back into the array to see what happens. That, of course, is easier said than done.

One of the problems I ran into is that I am using Ubuntu and fuse to run zfs. Ubuntu has this nasty habit of changing around drive identifiers when USB devices are plugged in. So now when this drive is plugged in, it is on /dev/sde instead of /dev/sdd, which is now a USB attached drive.

No problem, I figure, I’ll offline the bad drive in the zpool and replace it with the new drive location. No such luck.

First I offlined the drive using zpool offline media /dev/sdd:

Now that it’s offline, I thought you should be able to detach it. No such luck – since it is a ‘primary’ device of the zpool it does not allow you to remove it.

What they want you to do is replace the drive with another drive. This drive (the same drive, with all info wiped from it) is now on /dev/sde. I try to replace it:

Even with -f it doesn’t allow the replacement, because the system thinks that the drive is part of another pool.

So basically you are stuck if trying to test a replacement with a drive that already been used in the pool. I’m sure I could replace it with another 1TB disk but what is the point of that?

I ended up resolving the problem by removing the external USB drive, therefore putting the drive back into the original /dev/sdd slot. Without issuing any commands, the system now sees the drive as the old one, and starts resilvering the drive.

It is interesting to see what it looks like from an i/o perspective. The system reads from the two good drives and writes to the new (bad) one. Using iostat -x:

It seems that ZFS is able to identify a hard drive by GID somehow but doesn’t automatically use it in the pool. This makes it so that you can’t test a drive by removing it, formatting it, and putting it into a new location. Basically, zfs assumes that your drives are always going to be in the same /dev location, which isn’t always true. As soon as you attach a USB drive in Ubuntu things are going to shift around.

After the resilver is complete, the zpool status is:

You can now clear the error with:

Zpool status now shows no errors:

So now the question I have is this: Are you able to manually update or remove the drive status somewhere in your system? How did zfs know that this drive already had a pool installed on it? I zeroed the drive and verified with fdisk there were no partition on it. Is there a file somewhere on the system that stores this information, or is it written somewhere on the drive?

ZFS is great, but it still has some little issues like this that give me pause before using it in a production system. Then again, I suppose all massive disk array systems have their little quirks!

You might see it running on your system as follows:

If your system is static and the devices do not change, you can actually disable this service using a policy entry.

Create a file in your policy directory, for example /etc/hal/fdi/policy/99-custom.fdi. Add the text:

Save and reload the hald using /etc/init.d/haldaemon restart.

And you will find that service no longer is polling your hardware.

Of course to turn it back on, remove that policy entry and restart the haldaemon again, it will be back in service.

Solution Credit: Linuxforums User cn77

The goal of this is to add a little quote, at random, every time you log into your system. My thoughts were to do it not only as a little source of inspiration but also to add perspective to what I’m doing sitting in front of the computer all of the time.

Originally I was going to try to write the script solely in bash since it is so flexible (and just as a proof of concept) but dealing with RSS in bash isn’t exactly pretty and I just wanted to get this together as quick as possible. PHP makes parsing XML easy, there are a number of ways to accomplish it. I chose to use the ready-made script at rssphp.net to do this, if you are curious about how you can handle this yourself using SimpleXML check out this tutorial over at Pixel2Life. The end result of my solution is a bash script calling a php script to grab the quote.

The Code

First create a file named /etc/update-motd.d/10-quote. The name does not matter much – the number will decide what order the script is called in of all the scripts in /etc/update-motd.d. Do an ls on that directory to see what all is being called when you log in. Add the following lines to this file, assuming you are placing your scripts in /etc/scripts/:

Download v1 of rssphp and extract it to the /etc/scripts/ directory. We will require that file in our php code.

Create the file /etc/scripts/getquote.php and add the following:

I am using the RSS source from QuoteDB as the source of my quotes. Of all the places I checked (and I checked a lot) they seemed to have the most appropriate ones for this use. Feel free to use any source you wish – as long as the XML fields title/description hold the quote you will be able to use it. The RSS url was not obvious from the site and I had to do some digging to find it, in the end I am using http://www.quotedb.com/quote/quote.php?action=random_quote_rss.

We also add the if statement to allow it to degrade nicely in case you have no network connectivity to the server. After a short period – a second or two – it will time out and let you log in.

The end result is a pretty quote in our motd:

It should be pretty strait forward; let me know if you run into any problems!

I manage a server which has a sole purpose: serving DNS requests. We use PowerDNS, which has been great. It is a DNS server whose backend is SQL, making administration of large numbers of records very easy. It is also fast, easy to use, open source and did I mention it is free?

The server has been humming along for years now. The traffic graphs don’t show a lot of data moving through it because it only serves DNS requests (plus MySQL replication) in the form of tiny UDP packets.

We started seeing these spikes in traffic but everything on the server seemed to be working properly. Test connections with dig proved that the server was accurately responding to requests, but external tests showed the server going up and down.

The First Clue

I started going through logs to see if we were being DoSed or if it was some sort of configuration problem. Everything seemed to be running properly and the requests, while voluminous, seemed to be legit. Within the flood of messages I spied error messages such as this:

Ah ha! A clue! Let’s check the current numbers of ip_conntrack, which is a kernel function for the firewall which keeps tabs on packets heading into the system.

Continuing this line of logic, lets check our current value for this setting:

So it looks like we are hitting up against this limit. After the number of connections reaches this number, the kernel will simply drop the packet. It does this so that it will not overload and freeze up due to too many packets coming into it at once.

This system is running on CentOS 4.8, and since then newer versions of RHEL5 have the default set at 65536. For maximum efficiency we keep this number at multiples of 2. The top size depends on your memory, so just be careful as overloading it may cause you to run out of it.

Fixing The ip_conntrack Bottleneck

In my case I decided to go up 2 steps to 131072. To temporarily set it, use sysctl:

Test everything out, if you have some problems with your network or system crashing, a reboot will set the value back to normal. To make the setting permanent on reboot, add the following line to your /etc/sysctl.conf file:

My theory is that since the server was dropping packets, remote hosts were re-sending their DNS requests causing a ‘flood’ of traffic to the server and the spikes you see in the traffic graph above whenever traffic was mildly elevated. The bandwidth spikes were caused by amplification of traffic due to resending of the requests. After increasing ip_conntrack_max I immediately saw the bandwidth resume to normal levels.

Your server should now be set against an onslaught of tiny packets, legitimate or not. If you have even more connections than what you can safely track with ip_conntrack you may need to move to the next level which involves hardware firewalls and other methods for packet inspection off-server on dedicated hardware.

Some resources used in my investigation of this problem:
[1] http://wiki.khnet.info/index.php/Conntrack_tuning
[2] http://serverfault.com/questions/111034/increasing-ip-conntrack-max-safely
[3] http://www.linuxquestions.org/questions/red-hat-31/ip_conntrack-table-full-dropping-packet-615436/

The image of the kittens used for the featured image has nothing to do with this post. There are no known good photos of a “UDP Packet”, and I thought that everyone likes kittens, so there it is. Credit flickr user mathias-erhart.

From Christian:

find /home/backup/ -maxdepth 1 -type d -mtime +7 -exec rm -r {} \;

From Michael:

find /home/backup/ -maxdepth 1 -type d -mtime +7 -exec echo “Removing Directory => {}” \; -exec rm -rf “{}” \;

The first one works quietly, while the second one will display what is being deleted. These are probably faster than putting it into a for loop, so feel free to use whatever works best in your particular situation!

Seinfeld-(The Wig Master)-2003-09-26-0(1).avi
Seinfeld-(The Wink)-2004-01-03-0.avi
Seinfeld-(The Wizard)-2004-02-26-0.avi
Seinfeld-(The Yada Yada)-2003-06-04-0.avi
Seinfeld-(The Pen)-2004-06-22-0.avi

I created a CSV file, and from the powers of the Amazon Mechanical Turk services, generated an entire list of the correct filenames, in the format:

Seinfeld.7x19.The.Wig.Master.avi
Seinfeld.7x04.The.Wink.avi
Seinfeld.9x15.The.Wizard.avi
Seinfeld.8x19.The.Yada.Yada.avi
Seinfeld.3x03.The.Pen.avi

Now I just needed a way to actually do the rename.

The final CSV file I generated is in the format:

Seinfeld-(The Wig Master)-2003-09-26-0(1).avi,Seinfeld.7x19.The.Wig.Master.avi
Seinfeld-(The Wink)-2004-01-03-0.avi,Seinfeld.7x04.The.Wink.avi
Seinfeld-(The Wizard)-2004-02-26-0.avi,Seinfeld.9x15.The.Wizard.avi
Seinfeld-(The Yada Yada)-2003-06-04-0.avi,Seinfeld.8x19.The.Yada.Yada.avi
Seinfeld-(The Pen)-2004-06-22-0.avi,Seinfeld.3x03.The.Pen.avi

Note that in my format, there could be no commas in the file names; but you can use any other delimiter such as = or ;, it would work equally as well with a character change in the script.

Using awk, you can get a correct command line output from this file:

awk -F',' '{print("mv \"" $1 "\" \"" $2 "\"")}' input.txt

Results in a preview of your commands, like so:

mv "Seinfeld-(The Wig Master)-2003-09-26-0(1).avi" "Seinfeld.7x19.The.Wig.Master.avi"
mv "Seinfeld-(The Wink)-2004-01-03-0.avi" "Seinfeld.7x04.The.Wink.avi"
mv "Seinfeld-(The Wizard)-2004-02-26-0.avi" "Seinfeld.9x15.The.Wizard.avi"
mv "Seinfeld-(The Yada Yada)-2003-06-04-0.avi" "Seinfeld.8x19.The.Yada.Yada.avi"
mv "Seinfeld-(The Pen)-2004-06-22-0.avi" "Seinfeld.3x03.The.Pen.avi"

Note that we put the file names in quotes because they contain spaces, and linux won’t recognize the filenames if you don’t include that. The -F command in awk is where you specify your delimiter, so feel free to use another character. Once you have ‘proofed’ the script to make sure it is doing what you expect it to, you execute it by adding |/bin/sh to the end, to pipe it directly to bash.

awk -F',' '{print("mv \"" $1 "\" \"" $2 "\"")}' input.txt | /bin/sh

This won’t result in any output, but you will see that it changed the names of your files, just as if you would have typed in the commands (proofed above) yourself.

If you don’t want to go through the bother of generating a CSV file, but you still want to use bash to rename files using awk, you can use a similar command which I found over at Snipplr.

ls foo*.jpg | awk '{print("mv "$1" "$1)}' | sed 's/foo/bar/2'

As with the other one, this will output the command to move the file but instead uses sed to do a search/replace of the string in the filename. Append |/bin/sh to execute the commands on your system.

dd if=/dev/zero of=zero.filename bs=1024 count=1000

You do the same using /dev/urandom:

dd if=/dev/urandom of=random.filename bs=1024 count=1000

Resulting in a 1MB file:

1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.0294247 s, 34.8 MB/s

This is transferring random data from the virtual device urandom to the output file. We use /dev/urandom instead of /dev/random because the /dev/random source generates random data very slowly. urandom is much faster at this but remains very random, if not quite a random as /dev/random. This should work with any system with dd and /dev/urandom.

This is due to how TCP frames data packets and optimizes them for connections. I believe by default TCP on most systems assumes about a 10mbit max capable transfer rate, so it does not show performance gains on a larger pipe without modification to the kernel options which govern TCP/IP frame size and features. Some distributions may make this change for you automatically however many will not.

To keep things short and sweet, we took the following advice from Speedguide.net on tweaking TCP parameters on linux kernel systems. This will cover Linux 2.1 and above – which means CentOS, RedHat, Ubuntu, Debian and many more distributions.

The TCP Parameters we will change are:

• /proc/sys/net/core/rmem_max - Maximum TCP Receive Window
• /proc/sys/net/core/wmem_max – Maximum TCP Send Window
• /proc/sys/net/ipv4/tcp_timestamps - (RFC 1323) timestamps add 12 bytes to the TCP header…
• /proc/sys/net/ipv4/tcp_sack – tcp selective acknowledgements.
• /proc/sys/net/ipv4/tcp_window_scaling – support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535.

If you recall /proc/ is the volatile portion of kernel configuration, you can change it on the fly but it will be reset on reboot unless settings are changed via an init file or setting the options in /etc/sysctl.conf. To change the settings once (to test):

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

And to apply them for good, add the following lines to /etc/sysctl.conf:

net.core.rmem_default = 256960
net.core.rmem_max = 256960
net.core.wmem_default = 256960
net.core.wmem_max = 256960
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1

Use ‘sysctl -p’ to apply the changes in this file to your running Linux instance. Feel free to experiment with these numbers to see how they impact your transfers, it depends a lot on how many and how large the files are that you transferring. These changes must be made on the SERVER side, any change on the client side would not impact the download speed from the server.

There are several other variables to consider, and these all depend on your application so change them if you know what you are attempting to do. After changing these settings, you will see speeds of about 10MB/sec (80mbps) on a 100mbps connection. The other 20mbps are lost in TCP and other network layer overhead, which is unavoidable.

I do however have access to a linux machine on campus. I am familiar with port forwarding using SSH but I had never used it to actually tunnel web traffic using a web browser on Windows. Turns out it is surprisingly easy!

The ssh command to use is:

This command sshes to the remote_host, and creates a tunnel on your localhost, port 8080. Note that you need to have private key authentication already set up for this host – it will not work with password authentication.

The description of the switches are (from the ssh man page):

• -C : Compression
• -2 : Use SSHv2
• -q : quiet!
• -T : Disable pseuto-tty allocation
• -n : Prevents reading from stdin (you need to have private key authentication set up, to prevent password authentication)
• -N : Do not execute a remote command (or launch a shell). Just use the ssh process for port forwarding
• -D : Allocate a socket to listen on the local side. When a connection is made to this port it is located to the remote machine. Makes SSH work as a SOCKS server. Only root can forward privileged ports like this.

From here, you set up Firefox or your browser of choice to use a Socks proxy on localhost:8080. The man page says that SOCKS4 and SOCK5 should both work but I had to use SOCKS v4, SOCKS v5 did not seem to work for me.

In order to do this, we first need to set up a way to pull this information from the router. The best way to do this is to install an SNMP (Simple Network Management Protocol) daemon on the system.

The main roadblock we face here is that the system mainly runs in volatile system memory, meaning that every time the system is rebooted the filesystem is reset. Fortunately Tomato provides a way to get around this using CIFS shares. Follow the steps below (as modified from here) to install an SNMP server on a Tomato router.

1. Create a network (samba, CIFS) share somewhere on the network. This computer must be on all of the time in order for Tomato to run the SNMP server.
2. Download the snmpd.zip file from one of these locations:
   [xs4all.nl]
   [systembash]

   expand the binary and .conf file into the share or a subdirectory (for example, <share name>/snmp)

   MD5 for snmpd binary is ae0d622648efdb8dceb7b3b5a63e23ac

3. Set up the shared directory on the router. Visit Administration->CIFS Client. Add the share as follows, with your correct share information:
4. Log into the Tomato router via ssh, and start SNMPd on the router by issuing the command:

   /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf &

5. Test that SNMP is running and can be accessed on another computer on the network. To test it, you can use snmpwalk like so:

   snmpwalk -c public -v 2c <IP Address of Router>

   If it works properly, it will list the available OIDs from the router. You do not need to take note of these, but they will be used in the graphing software later.

6. Finally, we need to launch the SNMP server when the router is restarted. You do this by adding the command to start it in the area Administration -> Scripts -> Firewall:

   sleep 30
   /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf -s &

   This launches the snmp server 30 seconds after the router is started or rebooted.

Thats it! SNMP is now running on the router.

Now to add this SNMP host to your graphing software. For this example, I will use Cacti, which I will assume you have already set up. If you need to set it up, please follow the directions on the Cacti site for installation.

First, add the router as a new device, using the information below (change IP to suite your needs):

After adding the device, you have several options depending on what sort of data you are looking for. For system information on the router – for example CPU usage, memory usage, etc; you can go directly to Create -> New Graphs. Select your device and then add the graph you are looking for.

The graph will show as a broken image at first, or a blank graph with “NaN” as the data source. Give it a few minutes to update, and the information should start to flow through. The ucd/net options work best, but feel free to experiment.

To get traffic stats on the interface, you first need to “Walk” the device.  Go back to your device list, and edit the device you added. Under “Associated Data Queries”, Add Data Query, add “SNMP – Interface Statistics” with Re-Index period as “Uptime goes backwards”. After adding it you should see under status something like: Success [39 Items, 6 Rows].

Since these data sources are now added, you can go back to Add a new Graph. After selecting the device, you should see a list of these new interfaces. Select the interfaces you wish to graph, and select the graph type (I suggest In/Out bits with Total).

After a few minutes, the data should start filling in. After a while, you will get a graph like this:

In conclusion, with a little work, you can get enterprise class graphing from your consumer router. The total project took me about 45 minutes, and I was trying to figure out all of the data sources and the correct way to enter everything.

Let me know your experiences, suggestions and corrections!