From Christian:

find /home/backup/ -maxdepth 1 -type d -mtime +7 -exec rm -r {} \;

From Michael:

find /home/backup/ -maxdepth 1 -type d -mtime +7 -exec echo “Removing Directory => {}” \; -exec rm -rf “{}” \;

The first one works quietly, while the second one will display what is being deleted. These are probably faster than putting it into a for loop, so feel free to use whatever works best in your particular situation!

Seinfeld-(The Wig Master)-2003-09-26-0(1).avi
Seinfeld-(The Wink)-2004-01-03-0.avi
Seinfeld-(The Wizard)-2004-02-26-0.avi
Seinfeld-(The Yada Yada)-2003-06-04-0.avi
Seinfeld-(The Pen)-2004-06-22-0.avi

I created a CSV file, and from the powers of the Amazon Mechanical Turk services, generated an entire list of the correct filenames, in the format:

Seinfeld.7x19.The.Wig.Master.avi
Seinfeld.7x04.The.Wink.avi
Seinfeld.9x15.The.Wizard.avi
Seinfeld.8x19.The.Yada.Yada.avi
Seinfeld.3x03.The.Pen.avi

Now I just needed a way to actually do the rename.

The final CSV file I generated is in the format:

Seinfeld-(The Wig Master)-2003-09-26-0(1).avi,Seinfeld.7x19.The.Wig.Master.avi
Seinfeld-(The Wink)-2004-01-03-0.avi,Seinfeld.7x04.The.Wink.avi
Seinfeld-(The Wizard)-2004-02-26-0.avi,Seinfeld.9x15.The.Wizard.avi
Seinfeld-(The Yada Yada)-2003-06-04-0.avi,Seinfeld.8x19.The.Yada.Yada.avi
Seinfeld-(The Pen)-2004-06-22-0.avi,Seinfeld.3x03.The.Pen.avi

Note that in my format, there could be no commas in the file names; but you can use any other delimiter such as = or ;, it would work equally as well with a character change in the script.

Using awk, you can get a correct command line output from this file:

awk -F',' '{print("mv \"" $1 "\" \"" $2 "\"")}' input.txt

Results in a preview of your commands, like so:

mv "Seinfeld-(The Wig Master)-2003-09-26-0(1).avi" "Seinfeld.7x19.The.Wig.Master.avi"
mv "Seinfeld-(The Wink)-2004-01-03-0.avi" "Seinfeld.7x04.The.Wink.avi"
mv "Seinfeld-(The Wizard)-2004-02-26-0.avi" "Seinfeld.9x15.The.Wizard.avi"
mv "Seinfeld-(The Yada Yada)-2003-06-04-0.avi" "Seinfeld.8x19.The.Yada.Yada.avi"
mv "Seinfeld-(The Pen)-2004-06-22-0.avi" "Seinfeld.3x03.The.Pen.avi"

Note that we put the file names in quotes because they contain spaces, and linux won’t recognize the filenames if you don’t include that. The -F command in awk is where you specify your delimiter, so feel free to use another character. Once you have ‘proofed’ the script to make sure it is doing what you expect it to, you execute it by adding |/bin/sh to the end, to pipe it directly to bash.

awk -F',' '{print("mv \"" $1 "\" \"" $2 "\"")}' input.txt | /bin/sh

This won’t result in any output, but you will see that it changed the names of your files, just as if you would have typed in the commands (proofed above) yourself.

If you don’t want to go through the bother of generating a CSV file, but you still want to use bash to rename files using awk, you can use a similar command which I found over at Snipplr.

ls foo*.jpg | awk '{print("mv "$1" "$1)}' | sed 's/foo/bar/2'

As with the other one, this will output the command to move the file but instead uses sed to do a search/replace of the string in the filename. Append |/bin/sh to execute the commands on your system.

dd if=/dev/zero of=zero.filename bs=1024 count=1000

You do the same using /dev/urandom:

dd if=/dev/urandom of=random.filename bs=1024 count=1000

Resulting in a 1MB file:

1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.0294247 s, 34.8 MB/s

This is transferring random data from the virtual device urandom to the output file. We use /dev/urandom instead of /dev/random because the /dev/random source generates random data very slowly. urandom is much faster at this but remains very random, if not quite a random as /dev/random. This should work with any system with dd and /dev/urandom.

This is due to how TCP frames data packets and optimizes them for connections. I believe by default TCP on most systems assumes about a 10mbit max capable transfer rate, so it does not show performance gains on a larger pipe without modification to the kernel options which govern TCP/IP frame size and features. Some distributions may make this change for you automatically however many will not.

To keep things short and sweet, we took the following advice from Speedguide.net on tweaking TCP parameters on linux kernel systems. This will cover Linux 2.1 and above – which means CentOS, RedHat, Ubuntu, Debian and many more distributions.

The TCP Parameters we will change are:

• /proc/sys/net/core/rmem_max - Maximum TCP Receive Window
• /proc/sys/net/core/wmem_max – Maximum TCP Send Window
• /proc/sys/net/ipv4/tcp_timestamps - (RFC 1323) timestamps add 12 bytes to the TCP header…
• /proc/sys/net/ipv4/tcp_sack – tcp selective acknowledgements.
• /proc/sys/net/ipv4/tcp_window_scaling – support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535.

If you recall /proc/ is the volatile portion of kernel configuration, you can change it on the fly but it will be reset on reboot unless settings are changed via an init file or setting the options in /etc/sysctl.conf. To change the settings once (to test):

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

And to apply them for good, add the following lines to /etc/sysctl.conf:

net.core.rmem_default = 256960
net.core.rmem_max = 256960
net.core.wmem_default = 256960
net.core.wmem_max = 256960
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1

Use ‘sysctl -p’ to apply the changes in this file to your running Linux instance. Feel free to experiment with these numbers to see how they impact your transfers, it depends a lot on how many and how large the files are that you transferring. These changes must be made on the SERVER side, any change on the client side would not impact the download speed from the server.

There are several other variables to consider, and these all depend on your application so change them if you know what you are attempting to do. After changing these settings, you will see speeds of about 10MB/sec (80mbps) on a 100mbps connection. The other 20mbps are lost in TCP and other network layer overhead, which is unavoidable.

In order to do this, we first need to set up a way to pull this information from the router. The best way to do this is to install an SNMP (Simple Network Management Protocol) daemon on the system.

The main roadblock we face here is that the system mainly runs in volatile system memory, meaning that every time the system is rebooted the filesystem is reset. Fortunately Tomato provides a way to get around this using CIFS shares. Follow the steps below (as modified from here) to install an SNMP server on a Tomato router.

1. Create a network (samba, CIFS) share somewhere on the network. This computer must be on all of the time in order for Tomato to run the SNMP server.
2. Download the snmpd.zip file from one of these locations:
   [xs4all.nl]
   [systembash]

   expand the binary and .conf file into the share or a subdirectory (for example, <share name>/snmp)

   MD5 for snmpd binary is ae0d622648efdb8dceb7b3b5a63e23ac

3. Set up the shared directory on the router. Visit Administration->CIFS Client. Add the share as follows, with your correct share information:
4. Log into the Tomato router via ssh, and start SNMPd on the router by issuing the command:

   /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf &

5. Test that SNMP is running and can be accessed on another computer on the network. To test it, you can use snmpwalk like so:

   snmpwalk -c public -v 2c <IP Address of Router>

   If it works properly, it will list the available OIDs from the router. You do not need to take note of these, but they will be used in the graphing software later.

6. Finally, we need to launch the SNMP server when the router is restarted. You do this by adding the command to start it in the area Administration -> Scripts -> Firewall:

   sleep 30
   /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf -s &

   This launches the snmp server 30 seconds after the router is started or rebooted.

Thats it! SNMP is now running on the router.

Now to add this SNMP host to your graphing software. For this example, I will use Cacti, which I will assume you have already set up. If you need to set it up, please follow the directions on the Cacti site for installation.

First, add the router as a new device, using the information below (change IP to suite your needs):

After adding the device, you have several options depending on what sort of data you are looking for. For system information on the router – for example CPU usage, memory usage, etc; you can go directly to Create -> New Graphs. Select your device and then add the graph you are looking for.

The graph will show as a broken image at first, or a blank graph with “NaN” as the data source. Give it a few minutes to update, and the information should start to flow through. The ucd/net options work best, but feel free to experiment.

To get traffic stats on the interface, you first need to “Walk” the device.  Go back to your device list, and edit the device you added. Under “Associated Data Queries”, Add Data Query, add “SNMP – Interface Statistics” with Re-Index period as “Uptime goes backwards”. After adding it you should see under status something like: Success [39 Items, 6 Rows].

Since these data sources are now added, you can go back to Add a new Graph. After selecting the device, you should see a list of these new interfaces. Select the interfaces you wish to graph, and select the graph type (I suggest In/Out bits with Total).

After a few minutes, the data should start filling in. After a while, you will get a graph like this:

In conclusion, with a little work, you can get enterprise class graphing from your consumer router. The total project took me about 45 minutes, and I was trying to figure out all of the data sources and the correct way to enter everything.

Let me know your experiences, suggestions and corrections!

As with most issues there are several ways to skin this cat. You can use tcpdump or wireshark to sniff the all of the network traffic on the device. If you know the port the program is running on (you can use lsof for that), you can restrict traffic to that port. But what if the program is jumping ports, or even uses a side-port for some sort of data transmission (UDP?).

The main problem going down this route is that on a server that is doing any significant bit of traffic, it is like sorting through a needle in a haystack. If you have a single process that is taking up all of your bandwidth, you can probably find it pretty fast. But if the process is not doing a ton of traffic it can be hard to track down.

Strace to the rescue

You can use the great program strace to sniff the network data that an executed program is doing, or even a currently running program. This works well because if you are trying to isolate the network traffic a currently running process, your options can be limited. Using strace is the only way that I know of to see ALL of the traffic coming from a process.

To check the traffic of a currently running process X:

strace -p X -f -e trace=network -s 10000

The command breaks down:

• -p: process ID
• -f: follow forks
• -e: follow set of system calls. In our case, we use trace=network, which follows network system calls.
• -s: set output string sizes. default is 32, which does not  give a lot of information.

Finally if you have a new program to execute and you want to watch the network traffic on it, you execute that command with strace. This would be good to use if you work in a highly secure environment and need to find out what sort of network traffic a distributed binary is doing. Checking for a program ‘Phoning home’ is a good example of that.

Here is the command that launches a new process:

strace -f -e trace=network -s 10000 /usr/bin/command arguments

Hopefully using strace in this manner will help you debug some issues on your server – I know I have used it several times.

It is easy to do an in-place upgrade of Ubuntu Server from 8.10 ‘Intrepid Ibex‘ to 9.04 ‘Jaunty Jackalope‘. You can do this remotely over ssh or whatever you use to control your server. Best practices say to make sure to backup your server before doing the upgrade. I’ve done several servers this way with no issues!

Issue the command:

sudo apt-get update; sudo apt-get upgrade; sudo apt-get install update-manager-core; sudo do-release-upgrade

Follow any prompts to first upgrade the current distribution with the newest packages, then do the release upgrade.

Zabbix is an excellent system monitoring package. It does everything from basic availability checking to detailed system resource analysis. It is capable of graphing the variables pulled from the system, and alerting admins if there is a problem or something needed for attention.

Once you have the Zabbix server set up, you need to install the client on any systems you want to monitor. Windows systems have a precompiled binary to install. On linux, unix or freebsd systems you’ll need to compile binaries. If you have a range of systems that are homogeneous, you can port the binary to those or also compile it with static dependencies. Below are steps to compile, configure and install the zabbix client:

Steps to install a Zabbix Client

1. Download zabbix source code from www.zabbix.com; decompress with ‘tar zxvf’ and cd to directory
2. Configure the make program: ./configure –enable-agent
3. Compile and install the program: make install
4. Add zabbix group and user: groupadd zabbix; adduser -g zabbix -s /sbin/nologin -M -p RANDOMPASS zabbix
5. Create log file: touch /var/log/zabbix_agentd.log; chown zabbix.zabbix /var/log/zabbix_agentd.log
6. Copy init script to /etc/init.d. Scripts are located in ./misc/init.d/ and your distro directory.
7. Make sure bin directory in init script is where Zabbix actually compiled to.
8. chmod 755 /etc/init.d/zabbix_agentd
9. chkconfig zabbix_agentd on
10. Copy agent config script to /etc/zabbix/zabbix_agentd.conf. Current one is:

    # This is config file for zabbix_agentd
    # To get more information about ZABBIX, go http://www.zabbix.com
    
    # This is the ip and port of the main zabbix server
    Server=10.0.0.1
    ServerPort=10051
    Hostname=AGENTHOSTNAME
    ListenPort=10050
    # ListenIP=10.0.0.2
    StartAgents=5
    DisableActive=1
    DebugLevel=1
    PidFile=/var/tmp/zabbix_agentd.pid
    # LogFile=/var/log/zabbix_agentd.log
    LogFileSize=1
    Timeout=10

11. Start the zabbix service: service zabbix_agentd start
12. Open firewall for zabbix port (10050) if necessary.
13. Log into Zabbix on the server, Add server to hosts – use correct templates and groups depending on what type of server it is.
14. Add monitoring and notification as appropriate.
15. Consider if all necessary services are being monitored; test that detection of down services and notifications work properly.

Under the auspice of wanting to save energy and also extending the life of a new monitor, someone suggested that we turn off the monitor at night using an electrical timer. A lightbulb went off in my head, that there must be a better way to do this via command line and then run it in the cron.

It turns out the solution is very simple. The xset command is the X server preferences command. It has a simple command to turn off the monitor:

$ xset dpms force off

and to turn the monitor back on:

$ xset dpms force on

You can also check the status of the X server settings by using:

$ xset -q

Also, when dpms turns off the monitor, it will turn back on after a keypress or by moving the mouse. Since this is a lobby display, there is no keyboard or mouse installed in the system.

I’ve rolled this into a little bash script with on, off, and status switches:

#!/bin/bash
export DISPLAY=:0.0

if [ $# -eq 0 ]; then
  echo usage: $(basename $0) "on|off|status"
  exit 1
fi

if [ $1 = "off" ]; then
  echo -en "Turning monitor off..."
  xset dpms force off
  echo -en "done.\nCheck:"
  xset -q|grep "Monitor is"
elif [ $1 = "on" ]; then
  echo -en "Turning monitor on..."
  xset dpms force on
  echo -en "done.\nCheck:"
  xset -q|grep "Monitor is"
elif [ $1 = "status" ]; then
  xset -q|sed -ne 's/^[ ]*Monitor is //p'
else
  echo usage: $(basename $0) "on|off|status"
fi

You can then use cron to turn off the monitor at night, and back on in the morning:

0 20 0 0 0 /home/lobby/monitorControl.sh off
0 7 0 0 0 /home/lobby/monitorControl.sh on

This script will turn it off at 8pm and back on at 7am.

Note that this was written for an Ubuntu system, but the xset command is pretty generic so any system that runs Xserver like RedHat, CentOS, Debian, Fedora, etc should be able to use the script as well.

xx.xx.125.240 spc240.xx.xx^M$
xx.xx.125.241 spc241.xx.xx^M$

The ^M is a Windows ‘End of Line’ character.

Fortunately there is an easy way to fix these for using them in a unix based system.

Install ‘dos2unix’, and then run the file through them:

dos2unix filename

You can even run these on a bunch of files:

[root@ns1 ~]# dos2unix RDNS-*
dos2unix: converting file RDNS-xx.xx.81 to UNIX format …
dos2unix: converting file RDNS-xx.xx.82 to UNIX format …
dos2unix: converting file RDNS-xx.xx.85 to UNIX format …
dos2unix: converting file RDNS-xx.xx.95 to UNIX format …
dos2unix: converting file RDNS-xx.xx.100 to UNIX format …

‘cat -e’ will show that these files are fixed.

You can also use the program ‘recode’.