Useful Programs

How to Turn PHP into an RPM The Easy Way with FPM

So you have a custom PHP binary that you wish to distribute across multiple servers.

The “old” way to do this is to create a bash script in order to pull down the source code, run ./configure and then make; make install. But this is 2013, we have a much more civilized way to create this package now, so that you can distribute that RPM across multiple nodes or even include it in your own yum repository.

We’re going to use FPM (Effing Package Manager) to do the job for this. It is a little bit confusing because PHP also includes the acronym FPM for FastCGI Process Manager. Our package manager FPM is a ruby gem, so install the prerequisites first (CentOS 6 directions here), along with fpm:

1
2
# yum install ruby ruby-devel rubygems
# gem install fpm

To start, compile PHP the way you normally would, ie:

1
2
3
4
5
6
'./configure'  '--program-prefix=' '--prefix=/usr' \
'--exec-prefix=/usr' '--bindir=/usr/bin' '--enable-ftp' \
'--enable-calendar' '--with-libxml-dir=/usr' '--enable-mbstring' \
'--enable-mbregex' '--with-gd' '--enable-zip' '--with-mcrypt' \
'--with-mhash' '--with-libdir=lib64' '--enable-fpm' '--with-fpm-user=www' \
'--with-fpm-group=www' '--with-mysql=mysqlnd' '--with-mysqli=mysqlnd'

Then make it as normal.

1
2
# make
... Lots of output

Now instead of running make install like normal, we tell it to output to another directory, just some temporary directory on the server:

(Note, if compiling another program, the switch to make it install into another directory may be different, such as DESTDIR=/tmp/installdir)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
make install INSTALL_ROOT=/tmp/installdir
Installing PHP SAPI module:       fpm
Installing PHP CLI binary:        /tmp/installdir/usr/bin/
Installing PHP CLI man page:      /tmp/installdir/usr/share/man/man1/
Installing PHP FPM binary:        /tmp/installdir/usr/sbin/
Installing PHP FPM config:        /tmp/installdir/etc/
Installing PHP FPM man page:      /tmp/installdir/usr/share/man/man8/
Installing PHP FPM status page:      /tmp/installdir/usr/share/fpm/
Installing build environment:     /tmp/installdir/usr/lib/build/
Installing header files:          /tmp/installdir/usr/include/php/
Installing helper programs:       /tmp/installdir/usr/bin/
  program: phpize
  program: php-config
Installing man pages:             /tmp/installdir/usr/share/man/man1/
  page: phpize.1
  page: php-config.1
/usr/local/src/php-5.3.25/build/shtool install -c ext/phar/phar.phar /tmp/installdir/usr/bin
ln -s -f /usr/bin/phar.phar /tmp/installdir/usr/bin/phar
Installing PDO headers:          /tmp/installdir/usr/include/php/ext/pdo/

Now PHP is happily installed in /tmp/installdir. Which is great, but we can’t use it until it we package it up for regular installation. Run the FPM command to pack it all up:

1
2
3
4
5
# fpm -s dir -t rpm -n php-fpm -v 5.3.25 -C /tmp/installdir \
   -p php-fpm_VERSION_ARCH.rpm -d openssl -d pcre -d bzip2 -d curl \
   -d libjpeg -d libpng -d freetype -d gmp -d libmcrypt -d libmhash \
   -d libxml2 usr/bin usr/share/man/man1 usr/sbin etc \
   usr/share/man/man8 usr/share/fpm usr/lib/build usr/include/php

What do the options mean?

I’ll go through these options:

  • -s dir: source is a directory
  • -t rpm: we are creating an rpm
  • -n php-fpm: we are creating a package with the name php-fpm
  • -v 5.3.25: the version of the rpm (so you can manage upgrades)
  • -C /tmp/installdir: the base location to take the files from
  • -p php-fpm_VERSION_ARCH.rpm: fpm will replace the VERSION and ARCH macros automatically with your version number and architecture
  • -d openssl -s pcre …: these are all of the rpm dependencies that you’ll want on the machines installing the package. If you don’t include the proper ones (to match your source system, and which are included in ./configure, you’ll get library errors when you try to launch the program.
  • usr/bin usr/share/man/man1 usr/sbin etc usr/share/man/man8 usr/share/fpm usr/lib/build usr/include/php: the directories you wish to include in your package file

When all is done, you should have your rpm file located in /tmp/installdir. You can install it directly on any system with rpm -i <rpm name>

Before you package up your files, you can include any other files you want. For example, you could include the init script or php.ini file. But if you are managing the system using puppet or chef, it would be better to manage this configuration file through there rather than from the rpm file. This works really well as a ‘building block’ for your programs when you add these custom rpms into your own repository. That way, you can have your system install custom versions of the binaries where applicable but still have the niceties that come from an systems automation with puppet or chef.

Enjoy!

Evaluating FTP Servers: ProFTPd vs PureFTPd vs vsftpd

ftp

Usually, I will try to push clients towards using SCP (via a client such as WinSCP), however inevitably there are clients who do not understand this new method of accessing their files securely online, and who for one reason or another insist on using FTP for their online file access. As they say – the customer is always right?

Anyway, there are currently 3 mainstream FTP servers available via the yum command on CentOS 6.x. PureFTPd, ProFTPd and vsftpd. So which FTP server is the best? I will summarize the servers below, or skip to the summary.

ProFTPd

ProFTPd is a modular FTP server which has been around for a long time. The large control panels (cPanel, DirectAdmin) all support ProFTPd and have for years.

The most feature rich of the bunch is certainly ProFTPd. There are a ton of plugins available for it, and the creator of it modeled its configuration architecture much like Apache’s – it is also using the GPL for licensing.

Configuration of ProFTPd is fairly straight forward, and example configuration files abound at a quick search of Google.

ProFTPd is available on a wide variety of system architectures and operating systems.

ProFTPd Security

Of the bunch, ProFTPd has the most CVE vulnerabilities listed. The high number is most likely an indicator of ProFTPd’s widespread use which makes it a target of hackers.

ProFTPd CVE Entries: 40
Shodan ProFTPd entries: 127

PureFTPd

PureFTPd‘s mantra is ‘Security First.’ This is evident in the low number of CVE entries (see below).

Licensed under the BSD license, PureFTPd is also available on a wide-range of operating systems (but not Windows).

Configuration of PureFTPd is simple, with a no-configuration file option. Although not as widely used as ProFTPd, PureFTPd has many configuration examples listed online.

PureFTPd Security

PureFTPd’s “Security First” mantra puts it at the lead in the security department with the fewest security vulnerabilities.

PureFTPd CVE Entries: 4
Shodan Pure-FTPd Entries: 12

vsftpd

vsftpd is another GPL-licensed FTP server, which stands for “Very Security FTP daemon.” It is a lighweight FTP server built with security in mind.

Its lightweight nature allows it to scale very efficiently, and many large sites (ftp.redhat.com, ftp.debian.org, ftp.freebsd.org) currently utilize vsftpd as their FTP server of choice.

vsftpd Security

vsftpd has a lower number of vulnerabilities listed in CVE than ProFTPd but more than PureFTPd. This could be because, since its name implies it is a secure FTP service, or because it is so widely used on large sites – that it is under more scrutiny than the others.

vsftpd CVE Entries: 12
Shodan vsftpd entries: 41

Summary & FTP Server Recommendations

#

Considering the evaluations above, any server would work in a situation, however generally speaking:

  • If you want a server with the most flexible configuration options and external modules: ProFTPd
  • If you have just a few users and want a simple, secure FTP server: PureFTPd
  • If you want to run a FTP server at scale with many users: vsftpd

Of course, everyone’s requirements are different so make sure you evaluate the options according to your own needs.

Disagree with my assessment? Let me know why!

UbunTOS – Ubuntu 9.10 + TinyOS 2.x VirtualBox Image

vboxubuntos

This is my admittedly minor but I hope useful contribution to the TinyOS development community. TinyOS is an Operating System and development framework for Wireless Sensor Networks and other platforms which has a small footprint and is very energy conscious.

The TinyOS source code is available for free online for many operating systems, however it takes a long time to get the environment set up and it is not portable at all. I came across XubunTOS but it did not seem to be in active development anymore, so I endeavored to install TinyOS 2.1 and 2.x from source into a regular Ubuntu image. The most help came from Matt Keally’s Blog. While doing this, I thought it might be useful to many others who wish to develop in the TinyOS framework but might not have the skills necessary to install it. Therefore, I developed this VirtualBox image so that you can install it on any system for which VirtualBox is available and supports USB passthrough for the programming of the motes. I’ve tested on Windows 7, Windows XP and it should work on any other host OS, but I would love to hear your feedback. All funny business aside, I present to the world UbunTOS: Read more

Copy Files to Many USB Drives Quickly and Easily

photo

I was recently tasked with copying speaker’s presentations, files and handouts onto 100s of USB Drives (key drives) for a conference that work is hosting down in Washington, D.C. My first thought was that it was going to be a pain to have to copy/paste the files to each drive. I thought about creating a batch script to copy the files with a double click. But really, who wants to be doing all of that clicking and/or typing? Work smarter, not harder.

Then I remembered a neat feature that SyncBackSE, a program I use at home for backups, has available. The backup program – which is basically a file copy process – can be triggered based on the insert of a drive, whether that be a USB Key Drive or an External Hard Drive. Using the program, the only action you need to do to trigger the copy process is literally plug the drive in. After the machine recognizes the drive and mounts it to a drive letter, the copy process starts automatically.

Here are the directions on how to set this up.

  1. USB Hub

    Prepare your system. Install SyncBackSE. I have purchased the full program at home, but since I am using this for a limited task you can just install the 30-day trial. Create a folder, on your desktop or location of choice, with the files you want to be copied to the filesystem. Copy files to that folder, this is your “source directory”. Plug in as many USB drives as you can, in my case I used a USB hub with two handy top USB ports. Note which letter these drives assign themselves as.

  2. Set up the Source and Destination. The Source will be the “source directory” of any files you wish to copy. The Destination will be the USB Drive Letter. If you are able to plug more than one drive in at a time, you will need to create separate profiles for each one, with differing drive letters. Create them at first or just copy the first profile you make, and switch the Destination Drive on each one.
  3. Source and Destination Settings

  4. Modify the profile. This is where the magic happens. Under profile setup, go to “Click for Options” and check off “Expert”. Select When and then “Insert”. Check off “Run this profile when:” and select the drive letter under “is inserted into drive”. Also check off “Run unattended” – this will ensure it does not prompt you when you plug in your drive each time. The other options should remain with . A summary of these settings is in the figure below.

Settings for Auto Copy

That is it! Test by inserting a USB drive. You should see the profile change to “Running” then then “Success” after files have been completed. The screen will look as follows:

Final Product

In my experience, the part that takes the longest was Windows XP discovering the new drive and then assigning the drive letter. This PC is slow, so I imagine on a faster machine this process would actually be much quicker. I ended up having to re-copy the drives since someone added their handouts in the last minute, but the drive discovery process happened much faster the second time around. I was able to do about 100 drives in 30 minutes, so this process is actually very speedy and works very well.

How to Install SNMP on Tomato Router Firmware and Graph Traffic with Cacti

You’ve flashed your old WRT54G or other vanilla router with the Tomato firmware. This itself turns your router into a lean, mean routing machine with QOS, SSH and more, but let’s say we want to take it a bit further. What it we want to get some more stats out of it?

In order to do this, we first need to set up a way to pull this information from the router. The best way to do this is to install an SNMP (Simple Network Management Protocol) daemon on the system.

The main roadblock we face here is that the system mainly runs in volatile system memory, meaning that every time the system is rebooted the filesystem is reset. Fortunately Tomato provides a way to get around this using CIFS shares. Follow the steps below (as modified from here) to install an SNMP server on a Tomato router.

  1. Create a network (samba, CIFS) share somewhere on the network. This computer must be on all of the time in order for Tomato to run the SNMP server.
  2. Download the snmpd.zip file from one of these locations:
    [xs4all.nl]
    [systembash]

    expand the binary and .conf file into the share or a subdirectory (for example, <share name>/snmp)

    MD5 for snmpd binary is ae0d622648efdb8dceb7b3b5a63e23ac

  3. Set up the shared directory on the router. Visit Administration->CIFS Client. Add the share as follows, with your correct share information:cifs1
  4. Log into the Tomato router via ssh, and start SNMPd on the router by issuing the command:
    /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf &
  5. Test that SNMP is running and can be accessed on another computer on the network. To test it, you can use snmpwalk like so:
    snmpwalk -c public -v 2c <IP Address of Router>

    If it works properly, it will list the available OIDs from the router. You do not need to take note of these, but they will be used in the graphing software later.

  6. Finally, we need to launch the SNMP server when the router is restarted. You do this by adding the command to start it in the area Administration -> Scripts -> Firewall:
    sleep 30
    /cifs1/snmp/snmpd -c /cifs1/snmp/snmpd.conf -s &

    This launches the snmp server 30 seconds after the router is started or rebooted.

Thats it! SNMP is now running on the router.

Now to add this SNMP host to your graphing software. For this example, I will use Cacti, which I will assume you have already set up. If you need to set it up, please follow the directions on the Cacti site for installation.

First, add the router as a new device, using the information below (change IP to suite your needs):

adddevice

After adding the device, you have several options depending on what sort of data you are looking for. For system information on the router – for example CPU usage, memory usage, etc; you can go directly to Create -> New Graphs. Select your device and then add the graph you are looking for.

The graph will show as a broken image at first, or a blank graph with “NaN” as the data source. Give it a few minutes to update, and the information should start to flow through. The ucd/net options work best, but feel free to experiment.

To get traffic stats on the interface, you first need to “Walk” the device.  Go back to your device list, and edit the device you added. Under “Associated Data Queries”, Add Data Query, add “SNMP – Interface Statistics” with Re-Index period as “Uptime goes backwards”. After adding it you should see under status something like: Success [39 Items, 6 Rows].

Since these data sources are now added, you can go back to Add a new Graph. After selecting the device, you should see a list of these new interfaces. Select the interfaces you wish to graph, and select the graph type (I suggest In/Out bits with Total).

After a few minutes, the data should start filling in. After a while, you will get a graph like this:

graph_image.php

In conclusion, with a little work, you can get enterprise class graphing from your consumer router. The total project took me about 45 minutes, and I was trying to figure out all of the data sources and the correct way to enter everything.

Let me know your experiences, suggestions and corrections!

Installing And Compiling Zabbix Client / Agent

Zabbix

Zabbix is an excellent system monitoring package. It does everything from basic availability checking to detailed system resource analysis. It is capable of graphing the variables pulled from the system, and alerting admins if there is a problem or something needed for attention.

Once you have the Zabbix server set up, you need to install the client on any systems you want to monitor. Windows systems have a precompiled binary to install. On linux, unix or freebsd systems you’ll need to compile binaries. If you have a range of systems that are homogeneous, you can port the binary to those or also compile it with static dependencies. Below are steps to compile, configure and install the zabbix client:

Steps to install a Zabbix Client

  1. Download zabbix source code from www.zabbix.com; decompress with ‘tar zxvf’ and cd to directory
  2. Configure the make program: ./configure –enable-agent
  3. Compile and install the program: make install
  4. Add zabbix group and user: groupadd zabbix; adduser -g zabbix -s /sbin/nologin -M -p RANDOMPASS zabbix
  5. Create log file: touch /var/log/zabbix_agentd.log; chown zabbix.zabbix /var/log/zabbix_agentd.log
  6. Copy init script to /etc/init.d. Scripts are located in ./misc/init.d/ and your distro directory.
  7. Make sure bin directory in init script is where Zabbix actually compiled to.
  8. chmod 755 /etc/init.d/zabbix_agentd
  9. chkconfig zabbix_agentd on
  10. Copy agent config script to /etc/zabbix/zabbix_agentd.conf. Current one is:
    # This is config file for zabbix_agentd
    # To get more information about ZABBIX, go http://www.zabbix.com
    
    # This is the ip and port of the main zabbix server
    Server=10.0.0.1
    ServerPort=10051
    Hostname=AGENTHOSTNAME
    ListenPort=10050
    # ListenIP=10.0.0.2
    StartAgents=5
    DisableActive=1
    DebugLevel=1
    PidFile=/var/tmp/zabbix_agentd.pid
    # LogFile=/var/log/zabbix_agentd.log
    LogFileSize=1
    Timeout=10
  11. Start the zabbix service: service zabbix_agentd start
  12. Open firewall for zabbix port (10050) if necessary.
  13. Log into Zabbix on the server, Add server to hosts – use correct templates and groups depending on what type of server it is.
  14. Add monitoring and notification as appropriate.
  15. Consider if all necessary services are being monitored; test that detection of down services and notifications work properly.

Installing VirtualBox on a PAE Kernel System

I recently got stuck while trying to install Headless Virtualbox on a Centos 5.2 server per the directions at Howtoforge here. The RPM failed with the error that it couldn’t find the kernel source. I thought I had already installed kernel-devel, and confirmed that I did. The instructions were to define the KERN_DIR variable like so:

$ export KERN_DIR=/usr/src/kernels/2.6.18-92.1.22.el5-i686/
$ /etc/init.d/vboxdrv setup
Stopping VirtualBox kernel module [  OK  ]
Removing old VirtualBox netflt kernel module [  OK  ]
Removing old VirtualBox kernel module [  OK  ]
Recompiling VirtualBox kernel module [  OK  ]
Starting VirtualBox kernel module [FAILED]
(modprobe vboxdrv failed. Please use 'dmesg' to find out why)

dmesg shows this:

vboxdrv: disagrees about version of symbol struct_module

Interesting!

After several missteps, it turns out there is a separate development rpm for the PAE kernel:

$ yum install kernel-PAE-devel

After that installs, we are good:

$ export KERN_DIR=/usr/src/kernels/2.6.18-92.1.22.el5-PAE-i686/
$ /etc/init.d/vboxdrv setup
Stopping VirtualBox kernel module [  OK  ]
Removing old VirtualBox netflt kernel module [  OK  ]
Removing old VirtualBox kernel module [  OK  ]
Recompiling VirtualBox kernel module [  OK  ]
Starting VirtualBox kernel module [  OK  ]

Golden.