After figuring out that I had 2 out of 3 of my 1TB Seagate Barracuda hard drives in the array fail, I had to give the entire array up for a loss and test out my backup strategy. Fortunately it worked and there was no data loss. After receiving the replacement drives in from Seagate, I rebuilt the ZFS array (using raidz again) and went along my merry way. After another 6 months or so, I started getting some funky results from my other drive. Thinking it might have some issue as with the others, I removed the drive and ran Seatools on it (by the way, Seatools doesn’t offer a 64-bit Windows version – what year is this?).

The drive didn’t show any signs of failure, so I decided to wipe it and add it back into the array to see what happens. That, of course, is easier said than done.

One of the problems I ran into is that I am using Ubuntu and fuse to run zfs. Ubuntu has this nasty habit of changing around drive identifiers when USB devices are plugged in. So now when this drive is plugged in, it is on /dev/sde instead of /dev/sdd, which is now a USB attached drive.

No problem, I figure, I’ll offline the bad drive in the zpool and replace it with the new drive location. No such luck.

First I offlined the drive using zpool offline media /dev/sdd:

Now that it’s offline, I thought you should be able to detach it. No such luck – since it is a ‘primary’ device of the zpool it does not allow you to remove it.

What they want you to do is replace the drive with another drive. This drive (the same drive, with all info wiped from it) is now on /dev/sde. I try to replace it:

Even with -f it doesn’t allow the replacement, because the system thinks that the drive is part of another pool.

So basically you are stuck if trying to test a replacement with a drive that already been used in the pool. I’m sure I could replace it with another 1TB disk but what is the point of that?

I ended up resolving the problem by removing the external USB drive, therefore putting the drive back into the original /dev/sdd slot. Without issuing any commands, the system now sees the drive as the old one, and starts resilvering the drive.

It is interesting to see what it looks like from an i/o perspective. The system reads from the two good drives and writes to the new (bad) one. Using iostat -x:

It seems that ZFS is able to identify a hard drive by GID somehow but doesn’t automatically use it in the pool. This makes it so that you can’t test a drive by removing it, formatting it, and putting it into a new location. Basically, zfs assumes that your drives are always going to be in the same /dev location, which isn’t always true. As soon as you attach a USB drive in Ubuntu things are going to shift around.

After the resilver is complete, the zpool status is:

You can now clear the error with:

Zpool status now shows no errors:

So now the question I have is this: Are you able to manually update or remove the drive status somewhere in your system? How did zfs know that this drive already had a pool installed on it? I zeroed the drive and verified with fdisk there were no partition on it. Is there a file somewhere on the system that stores this information, or is it written somewhere on the drive?

ZFS is great, but it still has some little issues like this that give me pause before using it in a production system. Then again, I suppose all massive disk array systems have their little quirks!

You might see it running on your system as follows:

If your system is static and the devices do not change, you can actually disable this service using a policy entry.

Create a file in your policy directory, for example /etc/hal/fdi/policy/99-custom.fdi. Add the text:

Save and reload the hald using /etc/init.d/haldaemon restart.

And you will find that service no longer is polling your hardware.

Of course to turn it back on, remove that policy entry and restart the haldaemon again, it will be back in service.

Solution Credit: Linuxforums User cn77

The goal of this is to add a little quote, at random, every time you log into your system. My thoughts were to do it not only as a little source of inspiration but also to add perspective to what I’m doing sitting in front of the computer all of the time.

Originally I was going to try to write the script solely in bash since it is so flexible (and just as a proof of concept) but dealing with RSS in bash isn’t exactly pretty and I just wanted to get this together as quick as possible. PHP makes parsing XML easy, there are a number of ways to accomplish it. I chose to use the ready-made script at rssphp.net to do this, if you are curious about how you can handle this yourself using SimpleXML check out this tutorial over at Pixel2Life. The end result of my solution is a bash script calling a php script to grab the quote.

The Code

First create a file named /etc/update-motd.d/10-quote. The name does not matter much – the number will decide what order the script is called in of all the scripts in /etc/update-motd.d. Do an ls on that directory to see what all is being called when you log in. Add the following lines to this file, assuming you are placing your scripts in /etc/scripts/:

Download v1 of rssphp and extract it to the /etc/scripts/ directory. We will require that file in our php code.

Create the file /etc/scripts/getquote.php and add the following:

I am using the RSS source from QuoteDB as the source of my quotes. Of all the places I checked (and I checked a lot) they seemed to have the most appropriate ones for this use. Feel free to use any source you wish – as long as the XML fields title/description hold the quote you will be able to use it. The RSS url was not obvious from the site and I had to do some digging to find it, in the end I am using http://www.quotedb.com/quote/quote.php?action=random_quote_rss.

We also add the if statement to allow it to degrade nicely in case you have no network connectivity to the server. After a short period – a second or two – it will time out and let you log in.

The end result is a pretty quote in our motd:

It should be pretty strait forward; let me know if you run into any problems!

From Christian:

find /home/backup/ -maxdepth 1 -type d -mtime +7 -exec rm -r {} \;

From Michael:

find /home/backup/ -maxdepth 1 -type d -mtime +7 -exec echo “Removing Directory => {}” \; -exec rm -rf “{}” \;

The first one works quietly, while the second one will display what is being deleted. These are probably faster than putting it into a for loop, so feel free to use whatever works best in your particular situation!

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' \
-e 's/^/ /' -e 's/-/|/'

What’s going on here?

• 1

  ls -R

  – list files and directories recursively

• 1

  grep ":$"

  – find lines with : at the end (so only the directories)

• 1

  sed -e

  – evaluate expressions on the lines

• 1

  s/:$//

  – remove ‘:’ at the end of the line

• 1	s/[^-][^\/]*\//--/g

  – replaces text between / / lines (parent directories) with — , globally on each line

• 1

  s/^/ /

  – add space at the beginning of the lines

• 1

  s/-/|/

  – replace first – of the line with |

I reduced this using the following command. The most notable difference is that I use find instead of ls, which results in also viewing .hidden directories. I’m not sure which command is faster.

find ./ -type d | sed -e 's/[^-][^\/]*\//--/g;s/--/ |-/'

Both commands result in a formatted directory listing, demonstrated below:

|-sitetransfer
 |---redacteddomain.com
 |-----cache
 |-----templates
 |-------skidoo_too
 |---------images
 |-----------_vti_cnf
 |---------css
 |-----------_vti_cnf
 |---------js
 |-----------scriptaculous
 |-------------src
 |-------------lib
 |---------admin_templates

Seinfeld-(The Wig Master)-2003-09-26-0(1).avi
Seinfeld-(The Wink)-2004-01-03-0.avi
Seinfeld-(The Wizard)-2004-02-26-0.avi
Seinfeld-(The Yada Yada)-2003-06-04-0.avi
Seinfeld-(The Pen)-2004-06-22-0.avi

I created a CSV file, and from the powers of the Amazon Mechanical Turk services, generated an entire list of the correct filenames, in the format:

Seinfeld.7x19.The.Wig.Master.avi
Seinfeld.7x04.The.Wink.avi
Seinfeld.9x15.The.Wizard.avi
Seinfeld.8x19.The.Yada.Yada.avi
Seinfeld.3x03.The.Pen.avi

Now I just needed a way to actually do the rename.

The final CSV file I generated is in the format:

Seinfeld-(The Wig Master)-2003-09-26-0(1).avi,Seinfeld.7x19.The.Wig.Master.avi
Seinfeld-(The Wink)-2004-01-03-0.avi,Seinfeld.7x04.The.Wink.avi
Seinfeld-(The Wizard)-2004-02-26-0.avi,Seinfeld.9x15.The.Wizard.avi
Seinfeld-(The Yada Yada)-2003-06-04-0.avi,Seinfeld.8x19.The.Yada.Yada.avi
Seinfeld-(The Pen)-2004-06-22-0.avi,Seinfeld.3x03.The.Pen.avi

Note that in my format, there could be no commas in the file names; but you can use any other delimiter such as = or ;, it would work equally as well with a character change in the script.

Using awk, you can get a correct command line output from this file:

awk -F',' '{print("mv \"" $1 "\" \"" $2 "\"")}' input.txt

Results in a preview of your commands, like so:

mv "Seinfeld-(The Wig Master)-2003-09-26-0(1).avi" "Seinfeld.7x19.The.Wig.Master.avi"
mv "Seinfeld-(The Wink)-2004-01-03-0.avi" "Seinfeld.7x04.The.Wink.avi"
mv "Seinfeld-(The Wizard)-2004-02-26-0.avi" "Seinfeld.9x15.The.Wizard.avi"
mv "Seinfeld-(The Yada Yada)-2003-06-04-0.avi" "Seinfeld.8x19.The.Yada.Yada.avi"
mv "Seinfeld-(The Pen)-2004-06-22-0.avi" "Seinfeld.3x03.The.Pen.avi"

Note that we put the file names in quotes because they contain spaces, and linux won’t recognize the filenames if you don’t include that. The -F command in awk is where you specify your delimiter, so feel free to use another character. Once you have ‘proofed’ the script to make sure it is doing what you expect it to, you execute it by adding |/bin/sh to the end, to pipe it directly to bash.

awk -F',' '{print("mv \"" $1 "\" \"" $2 "\"")}' input.txt | /bin/sh

This won’t result in any output, but you will see that it changed the names of your files, just as if you would have typed in the commands (proofed above) yourself.

If you don’t want to go through the bother of generating a CSV file, but you still want to use bash to rename files using awk, you can use a similar command which I found over at Snipplr.

ls foo*.jpg | awk '{print("mv "$1" "$1)}' | sed 's/foo/bar/2'

As with the other one, this will output the command to move the file but instead uses sed to do a search/replace of the string in the filename. Append |/bin/sh to execute the commands on your system.

dd if=/dev/zero of=zero.filename bs=1024 count=1000

You do the same using /dev/urandom:

dd if=/dev/urandom of=random.filename bs=1024 count=1000

Resulting in a 1MB file:

1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.0294247 s, 34.8 MB/s

This is transferring random data from the virtual device urandom to the output file. We use /dev/urandom instead of /dev/random because the /dev/random source generates random data very slowly. urandom is much faster at this but remains very random, if not quite a random as /dev/random. This should work with any system with dd and /dev/urandom.

In order to do this, we first need to set up a way to pull this information from the router. The best way to do this is to install an SNMP (Simple Network Management Protocol) daemon on the system.

The main roadblock we face here is that the system mainly runs in volatile system memory, meaning that every time the system is rebooted the filesystem is reset. Fortunately Tomato provides a way to get around this using CIFS shares. Follow the steps below (as modified from here) to install an SNMP server on a Tomato router.

1. Create a network (samba, CIFS) share somewhere on the network. This computer must be on all of the time in order for Tomato to run the SNMP server.
2. Download the snmpd.zip file from one of these locations:
   [xs4all.nl]
   [systembash]

   expand the binary and .conf file into the share or a subdirectory (for example, <share name>/snmp)

   MD5 for snmpd binary is ae0d622648efdb8dceb7b3b5a63e23ac

3. Set up the shared directory on the router. Visit Administration->CIFS Client. Add the share as follows, with your correct share information:
4. Log into the Tomato router via ssh, and start SNMPd on the router by issuing the command:

   /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf &

5. Test that SNMP is running and can be accessed on another computer on the network. To test it, you can use snmpwalk like so:

   snmpwalk -c public -v 2c <IP Address of Router>

   If it works properly, it will list the available OIDs from the router. You do not need to take note of these, but they will be used in the graphing software later.

6. Finally, we need to launch the SNMP server when the router is restarted. You do this by adding the command to start it in the area Administration -> Scripts -> Firewall:

   sleep 30
   /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf -s &

   This launches the snmp server 30 seconds after the router is started or rebooted.

Thats it! SNMP is now running on the router.

Now to add this SNMP host to your graphing software. For this example, I will use Cacti, which I will assume you have already set up. If you need to set it up, please follow the directions on the Cacti site for installation.

First, add the router as a new device, using the information below (change IP to suite your needs):

After adding the device, you have several options depending on what sort of data you are looking for. For system information on the router – for example CPU usage, memory usage, etc; you can go directly to Create -> New Graphs. Select your device and then add the graph you are looking for.

The graph will show as a broken image at first, or a blank graph with “NaN” as the data source. Give it a few minutes to update, and the information should start to flow through. The ucd/net options work best, but feel free to experiment.

To get traffic stats on the interface, you first need to “Walk” the device.  Go back to your device list, and edit the device you added. Under “Associated Data Queries”, Add Data Query, add “SNMP – Interface Statistics” with Re-Index period as “Uptime goes backwards”. After adding it you should see under status something like: Success [39 Items, 6 Rows].

Since these data sources are now added, you can go back to Add a new Graph. After selecting the device, you should see a list of these new interfaces. Select the interfaces you wish to graph, and select the graph type (I suggest In/Out bits with Total).

After a few minutes, the data should start filling in. After a while, you will get a graph like this:

In conclusion, with a little work, you can get enterprise class graphing from your consumer router. The total project took me about 45 minutes, and I was trying to figure out all of the data sources and the correct way to enter everything.

Let me know your experiences, suggestions and corrections!

As with most issues there are several ways to skin this cat. You can use tcpdump or wireshark to sniff the all of the network traffic on the device. If you know the port the program is running on (you can use lsof for that), you can restrict traffic to that port. But what if the program is jumping ports, or even uses a side-port for some sort of data transmission (UDP?).

The main problem going down this route is that on a server that is doing any significant bit of traffic, it is like sorting through a needle in a haystack. If you have a single process that is taking up all of your bandwidth, you can probably find it pretty fast. But if the process is not doing a ton of traffic it can be hard to track down.

Strace to the rescue

You can use the great program strace to sniff the network data that an executed program is doing, or even a currently running program. This works well because if you are trying to isolate the network traffic a currently running process, your options can be limited. Using strace is the only way that I know of to see ALL of the traffic coming from a process.

To check the traffic of a currently running process X:

strace -p X -f -e trace=network -s 10000

The command breaks down:

• -p: process ID
• -f: follow forks
• -e: follow set of system calls. In our case, we use trace=network, which follows network system calls.
• -s: set output string sizes. default is 32, which does not  give a lot of information.

Finally if you have a new program to execute and you want to watch the network traffic on it, you execute that command with strace. This would be good to use if you work in a highly secure environment and need to find out what sort of network traffic a distributed binary is doing. Checking for a program ‘Phoning home’ is a good example of that.

Here is the command that launches a new process:

strace -f -e trace=network -s 10000 /usr/bin/command arguments

Hopefully using strace in this manner will help you debug some issues on your server – I know I have used it several times.

For more detail:

So what could this mean? The confusing error message “ioctl: LOOP_SET_FD: Invalid argument” means that your ISO image is on a filesystem that is not supported for the loopback device on your system. In my case, that was a GlusterFS mount that was hosting the image.

I copied it into another directory on my root ext3 filesystem and it mounted just fine!