Easy Search and Replace in Multiple Files on Linux Command Line

I recently came across a typo that existed in a bunch of html files on my web server. I thought it should be easy enough to change, but since it was in a number of files, editing it by hand would be time consuming. Fortunately, there is an easy, one liner command to replace the text in multiple files in a sub directory using recursion.

grep -lr -e '<oldword>' * | xargs sed -i 's/<oldword>/<newword>/g'

This command broken down:

  • grep for the word in a files, use recursion (to find files in sub directories), and list only file matches
  • | xargs passes the results from the grep command to sed
  • sed -i uses a regular expression (regex) to evaluate the change: s (search) / search word / target word / g (global replace)

For more information, see man pages for grep, sed, and xarg. Also it is very handy to learn about regular expressions as they are a valuable tool to any command line programmer!

Update 2009/7/19:

Thanks to reader btr we have a great one-line perl command that will perform the same task:

Perl provides a really nice one-line for this kind of thing:

perl -p -i -e ’s///g’ *

It also provides the option of creating a backup of each file changed:

perl -p -i.bak -e ’s///g’ *

mnemonic: PIE (”easy as pie”, etc.)
google “perl pie” and you’ll get lots of info for other uses of this technique.


How To Turn Off Your Monitor Via Command Line in Ubuntu

As previously written on this blog, I have set up a display in our lobby at work to display the day’s current events and meetings using Ubuntu and a tiny PC. Since this is a display which is on all day, the screensaver and monitor blanking (and other Energy Star features) are all turned off.

Under the auspice of wanting to save energy and also extending the life of a new monitor, someone suggested that we turn off the monitor at night using an electrical timer. A lightbulb went off in my head, that there must be a better way to do this via command line and then run it in the cron.

It turns out the solution is very simple. The xset command is the X server preferences command. It has a simple command to turn off the monitor:

$ xset dpms force off

and to turn the monitor back on:

$ xset dpms force on

You can also check the status of the X server settings by using:

$ xset -q

Also, when dpms turns off the monitor, it will turn back on after a keypress or by moving the mouse. Since this is a lobby display, there is no keyboard or mouse installed in the system.

I’ve rolled this into a little bash script with on, off, and status switches:

export DISPLAY=:0.0

if [ $# -eq 0 ]; then
  echo usage: $(basename $0) "on|off|status"
  exit 1

if [ $1 = "off" ]; then
  echo -en "Turning monitor off..."
  xset dpms force off
  echo -en "done.\nCheck:"
  xset -q|grep "Monitor is"
elif [ $1 = "on" ]; then
  echo -en "Turning monitor on..."
  xset dpms force on
  echo -en "done.\nCheck:"
  xset -q|grep "Monitor is"
elif [ $1 = "status" ]; then
  xset -q|sed -ne 's/^[ ]*Monitor is //p'
  echo usage: $(basename $0) "on|off|status"

You can then use cron to turn off the monitor at night, and back on in the morning:

0 20 0 0 0 /home/lobby/monitorControl.sh off
0 7 0 0 0 /home/lobby/monitorControl.sh on

This script will turn it off at 8pm and back on at 7am.

Note that this was written for an Ubuntu system, but the xset command is pretty generic so any system that runs Xserver like RedHat, CentOS, Debian, Fedora, etc should be able to use the script as well.

Apt-get Update GPG Key Errors and Fix

Running sudo apt-get upgrade, I started getting this error:

Reading package lists... Done
W: GPG error: http://ppa.launchpad.net intrepid Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 313D312748A22A95
W: You may want to run apt-get update to correct these problems

Ah ha! But apt-get update is the command causing this problem.

The solution is to import this key from the gpg servers; I don’t know why this isn’t done automatically, but here is it:

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 313D312748A22A95; gpg --export --armor 313D312748A22A95 | sudo apt-key add -

Resulting in:

Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --recv-keys --keyserver keyserver.ubuntu.com 313D312748A22A95
gpg: requesting key 48A22A95 from hkp server keyserver.ubuntu.com
gpg: key 48A22A95: public key "Launchpad PPA for Filip Brcic" imported
gpg: Total number processed: 1
gpg:              unchanged: 1

Congrats! sudo apt-get update now works properly!

HOWTO: Installing ZFS and setting up a Raid-Z array on Ubuntu

Readers should note that this applies to Ubuntu 8.10 Intrepid Ibex only!

ZFS is a relatively new filesystem created by Sun. It is released under the CDDL License which is incompatible with Linux’s GPL License, meaning that it can not be installed natively in the kernel. Therefore, for not it is relegated to addon packages and is brought to Ubuntu via the Fuse framework.

For more information on this see the Ubuntu Wiki article on ZFS.

The wiki article also explains this, but getting ZFS installed on Ubuntu is actually pretty straightforward by issuing these commands:

$ sudo echo "deb http://ppa.launchpad.net/brcha/ubuntu intrepid main" >> /etc/apt/sources.list.d/zfs-fuse.list
$ sudo echo "deb-src http://ppa.launchpad.net/brcha/ubuntu intrepid main" >> /etc/apt/sources.list.d/zfs-fuse.list
$ sudo apt-get update
$ sudo apt-get install zfs-fuse

This installs zfs onto your system. Now to create your raid-z array! Its dead simple.

$ sudo zpool create media -m /storage raidz /dev/sda /dev/sdb /dev/sdc

In the above command:

  • media is the name of the pool,
  • /storage is the mount point (make sure it already exists),
  • raidz is the type of mirroring/striping we are going to run (see more below), and
  • the rest of the options are the devices you are adding into the pool.

Make sure the mount directory already exists.

The type of array you are creating can be the following:

  • mirror will mirror the data across disks
  • raidz will mirror and stripe data across all disks, allowing 1 drive to fail without losing data
  • raidz2 will mirror and stripe data across all disks, allowing 2 drives to fail without losing data (but creating more overhead)

That really is it! You’ll then have a raid array, mounted to your indicated mount point. You can do all kinds of cool stuff with your pool, such as add disks, export the array to a file, import an exported file to the disk, create hot spares, and more.

After far as speed goes, I tested the pool performance with bonnie and also via real world estimates, and I’m only receiving about 20Mb/sec. Its not stellar and I plan to do some more testing as I haven’t tweaked my disks at all and I think I may be able to push it further. This slowness is caused by the checksumming that zfs does — it makes your data extremely safe but it does cause a lot of CPU overhead.

Hopefully ZFS will get more and more stable with time (although I haven’t had any problems with it so far!) and will be included in the Linux kernel, because it really is a great performing, easy to use file system. The uses for it could be tremendous, such as creating a desktop system as a zfs pool and creating instant disk backups from it with minimal overhead. Think: Time Machine for Linux!

Using Ubuntu as a 24/7 Lobby Display Driver

I recently took upon the task of setting up a presentation display in the lobby of our building. This display had previously used an old computer running Windows 2000, and displayed a fairly simple PowerPoint presentation with a few frames, mostly a schedule for anyone coming into the building.

We did a renovation and this display moved from a desk to on the wall. Therefore we needed to figure out a way to drive this display. We had 3 main options:

  1. Run the VGA cable upstairs to a room where the PC would sit.
  2. Purchase an expensive display driver box that could show PowerPoint along with other things like RSS feeds and video streams. I quoted this out and it was about $1800 for this fancy box.
  3. Install a small computer behind the display and use that to drive the presentation.

I decided to purchase a FitPC Slim to drive this display. This computer is surprisingly small and fits right behind the monitor. It is also extremely power efficient, so that it saves you money and also helps the environment.

When it came to the OS choice, I wanted something that would run on a slower machine (It only clocks in at 500Mhz) but could handle PowerPoint. Also, of course, the budget was a concern as well. So to me,the obvious choice was to use Ubuntu for this.

The stock Ubuntu installation is great on your desktop, with a few exceptions. I thought I would list the big changes you need to make in order to turn it into a keyboardless, mouseless display driver.

The following is a list of the major modifications I made to turn an Ubuntu system into a 24/7 display driver for a lobby television display.

Modify the Power Saving Features

First think you need to do is to turn off the screensaver, and also to disable it from blanking the screen after 10 minutes.

This is done via the System->Preferences->Screensaver tab.

  • Uncheck “Activate screensaver when computer is idle”.
  • Click “Power Management”, Make sure “Put computer to sleep when inactive for:” is set to Never, and change “Put display to sleep when inactive for:” to Never.

Close to save preferences.

Replace Network Manager

This is only important if you are using a wireless card, as I was. By default, network-manager will ask for your key to turn on wireless at every boot. This is really a strange thing to do, I have no idea what they are thinking with this! I would imagine that in the future this will be turned off. But in the mean time, it is totally disabling to us in this situation since there will be no keyboard attached to this unit.

The wireless network manager wicd is the perfect replacement for Ubuntu’s network-manager. It has a lot more features and is very customizable, and most importantly, doesn’t require a password to sign on to the default wireless network.

First, add the wicd repository to your sources.lst. In the menu system, it is: System-> Administration-> Synaptic Package Manager-> Settings-> Repositories-> Third Party Software.

Add (depending on your version):

8.10: deb http://apt.wicd.net intrepid extras
8.04: deb http://apt.wicd.net hardy extras

Reload the repositories.

Go to command line and issue this command to install the GPG key:

wget -q http://apt.wicd.net/wicd.gpg -O- | sudo apt-key add -

Install wicd via Synaptic, or via command line:

sudo apt-get install wicd

This will uninstall network-manager and network-manager-gnome.

Run wicd, Applications->Internet->Wicd. This is only for the first time, it will load on boot.

Configure your network settings, and check off “Connect to this network automatically.”

In some situations, I have also had to reboot the system to get wicd working normally to control the wired or wireless connection.

Enable automatic desktop login for your user

Go to: System > Administration > Login Window > Security and check “Enable Automatic Login” with your preferred user. This is not good from a security standpoint, but it does allow the machine to start up and log in without someone being there to type in the username and password.

Install Power Point Viewer

If you want to display a power point presentation, you will need to install a viewer. OpenOffice will display a powerpoint slide, however as far as I know, you need to launch it from the desktop and it can not be launched from the command line. Since in my parameters, I knew I wanted to launch a presentation from the command line via a script, I decided to go with Powerpoint Viewer.

There is a version that already exists in the Ubuntu archives. You can install it either with Synaptic or via command line:

sudo apt-get install pptview

However, this version had some display problems that I could not seem to figure out. Namely, it would place a white box in the center of all images I had in the powerpoint presentation.

I fixed this by installing wine, then installing the 2007 version of PowerPoint viewer – PowerPointViewer.exe. This version of the program displayed my presentation much nicer.

Another small hiccup I had was that the existing presentations were done using a proprietary font – Arial Unicode MS. I fixed this issue by editing the presentation on my local desktop with Power Point 2007, and changing all fonts to vanilla “Arial”. Make sure the font you use is available in wine on your system if you have problems with that.

Install Remote Desktop Program

First thing you need to do is to allow remote logins, or desktop sharing, to your display system. Go to System-> Preferences-> Remote Desktop.

Remote Desktop Ubuntu Preferences

Remote Desktop Ubuntu Preferences

Check off “Allow other users to view your desktop” and “Allow other users to control your desktop”.

Uncheck “Ask for your confirmation”. The field “Require the user to enter this password:” does as it says, it requires a VNC password before allowing a connection to the machine. This is recommended for the security of the OS, otherwise anyone on the network has full control of the PC.

For the remote desktop server, you have a few choices here – among the best choices are VNC and NXDesktop. I prefer VNC because it allows me to control the display that is currently on the screen.

You can use this remote control to change the current display or otherwise access the machine. Its easy to install in Ubuntu with the command:

sudo apt-get install vnc4server

or use Synaptic to install.

After it is installed on the display system, go to UltraVNC and install the client on your chosen platform. For your local PC, you only need to install the VNC Viewer to control this, or any, remote system. Access your remote computer by typing in it’s IP address. Note that VNC is not considered a secure protocol, so if this display PC is located on a public network, make sure to use a VPN tunnel to secure your VNC connection. I prefer Hamachi.

Set command line to output to the main display screen

This is actually a really easy trick that doesn’t seem obvious at first. Normally if you login to a system remotely, there is no X Display variable set. This means that if you launch a program with a gui component, you will get the error: Error: Can’t open display. This is because the $DISPLAY variable is not set in a normal ssh session.

To fix this, type the following into your command prompt:

export DISPLAY=:0.0

Or, to have it apply every time you log in, add it to ~/.bashrc.

As a side note, to launch a program remotely via ssh, you use “ssh -x” to have it execute on the remote system.

Advanced: Set up a cron script to launch a daily powerpoint presentation

This probably isn’t the best script in the world to do this but it gets the job done. What it does it kill existing Powerpoint Viewer processes, then launches a new one with a slide that has the current date as it’s name. Feel free to use and modify to your uses!

export DISPLAY=:0.0
DATE=`date +'%B %d'`
"/home/lobby/.wine/drive_c/Program Files/Microsoft Office/Office12/PPTVIEW.EXE" "$FILENAME" &

Add this script to a cron entry according to your schedule, and your powerpoint presentation will update even when you aren’t in the building!

sudo crontab -e

Add a line like these:

0 8 * * 1-5 /bin/su - lobby /home/lobby/launchppt.sh
30 16 * * 1-5 /bin/su - lobby /home/lobby/displayoffhours.sh

The first line will run the command under the “lobby” user at 8:00am on every weekday. The second command will launch an off-hours presentation at 4:30pm every weekday.

In summary:

The final result:

Those are about all the tips I have for now. It’s been running for a few days without problems.

Let me know if you have any tips or tricks you would like to share or if you have any success stories using Ubuntu to run a presentation on a lobby display screen!

Default Grub Boot Commands for Ubuntu 7.10

I recently formatted my laptop and installed Windows first, using half of the hard drive, and then installing Ubuntu 7.10 on the other half. It had been a while since I tried Ubuntu – it has a come a long way – but that is another story.

The install worked fine, however at the end, it just sort of hung while installing grub. GREAT. I reboot and it kicks me to a (grub) standard prompt.

It took me a while to figure this out, but you can manually boot Ubuntu via the grub prompt. The tricky part is finding out the right commands since your system is totally inaccessible.

You can find your available hard drive name by typing:

> root (   <tab>

pressing tab will list your available hard drives and partitions and hopefully your Ubuntu ext3 partition.

Continue setting the root boot partition. This includes your partition with all /boot files. For example, mine was installed onto the root / filesystem, and not a separate filesystem.

> root (hd0,2)

This sets your root that grub uses.

You then need to set the kernel. Use:

> kernel /boot/vmlin <tab>

Tab will show you the available files to use. Also you can use this at any level to explore your filesystem. So for example:

> kernel /boot/vmlinuz-2.6.15-20-386

But wait before you hit enter!

You will get a pivot root error – the kernel doesn’t know where the rest of your file system is. In a file called device.map in your /boot directory, this location should be specified. In my case, it is listed as /dev/sda3. This is important for your kernel. Fortunately, GRUB has a ‘cat’ command you use to get the text output of this file. We also set this to a read-only filesystem – Ubuntu takes care of setting it back to rw when it boots.

So the final kernel line is:

> kernel /boot/vmlinuz-2.6.15-20-386 root=/dev/sda3 ro

Now we need to set the initrd file – hopefully you by now know to use to find this if you do not know it off of the top of your head:

> initrd /initrd.img-2.6.15-20-386

And finally, type ‘boot’ to begin the boot process:

> boot

Hopefully ubuntu will boot for you now without problems. Again, if you get a pivot root or ‘unable to mount root VFS’ error – you need to check the root=/dev/sda3 line part of the grub commands. This means that the kernel could not find your main filesystem.

Once you are logged into Ubuntu – use the following command to regenerate menu.lst for you. Once this is rebuilt, you will be presented with the normal Grub menu and you won’t need to follow the process above!

sudo update-grub