I’ve been using the excellent fpm (Effing package manager!) tool for automatically generating rpms from source (for example, creating a custom compiled php-fpm binary and then wrapping it for install in an rpm for distribution from our own repository). FPM also creates .debs and other binaries as you need them.

To ensure we have a secure infrastructure we wanted to make sure that the rpms are correctly signed so that the gpg check can be completed on them when installed.

First, generate the gpg public/private key pair:

Fill out the required information. The entropy generation step may take a long time to complete.

Export a text version of the public key with this command, for later use in your repository:

fpm docs state that you just need to add the –rpm-sign to your fpm command. However, this generated the following error for me:

The reason for this is that it uses the settings located in your ~/.rpmmacros file. Find your key ID:

In this sample, the KEY_ID would be “94E8C1F6″

From here, the fpm command runs without issue.

This is an alias of the rpm --addsign command.

Now let’s check the signature on this rpm.

Whoops, it doesn’t seem to be signed properly. Well, that is because the rpm system doesn’t know to trust this GPG key yet. Import with the following command:

Note that this is the plain text file from before. Your RPM system knows to trust this key, and now if you verify the file:

This RPM is now ready for use in a repo. In the .repo file, use the following to ensure it follows the correct signature:

All key names have been changed to protect the innocent.

Usually, I will try to push clients towards using SCP (via a client such as WinSCP), however inevitably there are clients who do not understand this new method of accessing their files securely online, and who for one reason or another insist on using FTP for their online file access. As they say – the customer is always right?

Anyway, there are currently 3 mainstream FTP servers available via the yum command on CentOS 6.x. PureFTPd, ProFTPd and vsftpd. So which FTP server is the best? I will summarize the servers below, or skip to the summary.

ProFTPd

ProFTPd is a modular FTP server which has been around for a long time. The large control panels (cPanel, DirectAdmin) all support ProFTPd and have for years.

The most feature rich of the bunch is certainly ProFTPd. There are a ton of plugins available for it, and the creator of it modeled its configuration architecture much like Apache’s – it is also using the GPL for licensing.

Configuration of ProFTPd is fairly straight forward, and example configuration files abound at a quick search of Google.

ProFTPd is available on a wide variety of system architectures and operating systems.

ProFTPd Security

Of the bunch, ProFTPd has the most CVE vulnerabilities listed. The high number is most likely an indicator of ProFTPd’s widespread use which makes it a target of hackers.

ProFTPd CVE Entries: 40
Shodan ProFTPd entries: 127

PureFTPd

PureFTPd‘s mantra is ‘Security First.’ This is evident in the low number of CVE entries (see below).

Licensed under the BSD license, PureFTPd is also available on a wide-range of operating systems (but not Windows).

Configuration of PureFTPd is simple, with a no-configuration file option. Although not as widely used as ProFTPd, PureFTPd has many configuration examples listed online.

PureFTPd Security

PureFTPd’s “Security First” mantra puts it at the lead in the security department with the fewest security vulnerabilities.

PureFTPd CVE Entries: 4
Shodan Pure-FTPd Entries: 12

vsftpd

vsftpd is another GPL-licensed FTP server, which stands for “Very Security FTP daemon.” It is a lighweight FTP server built with security in mind.

Its lightweight nature allows it to scale very efficiently, and many large sites (ftp.redhat.com, ftp.debian.org, ftp.freebsd.org) currently utilize vsftpd as their FTP server of choice.

vsftpd Security

vsftpd has a lower number of vulnerabilities listed in CVE than ProFTPd but more than PureFTPd. This could be because, since its name implies it is a secure FTP service, or because it is so widely used on large sites – that it is under more scrutiny than the others.

vsftpd CVE Entries: 12
Shodan vsftpd entries: 41

Summary & FTP Server Recommendations

#

Considering the evaluations above, any server would work in a situation, however generally speaking:

• If you want a server with the most flexible configuration options and external modules: ProFTPd
• If you have just a few users and want a simple, secure FTP server: PureFTPd
• If you want to run a FTP server at scale with many users: vsftpd

Of course, everyone’s requirements are different so make sure you evaluate the options according to your own needs.

Disagree with my assessment? Let me know why!

Did you ever have a situation where you needed to access a website that had an IP restriction in place? I recently had a situation where I needed to access the web via my university connection (due to IP restrictions placed on accessing databases of research papers). They do not have a VPN setup so it is hard to do this off-campus.

I do however have access to a linux machine on campus. I am familiar with port forwarding using SSH but I had never used it to actually tunnel web traffic using a web browser on Windows. Turns out it is surprisingly easy!

The ssh command to use is:

This command sshes to the remote_host, and creates a tunnel on your localhost, port 8080. Note that you need to have private key authentication already set up for this host – it will not work with password authentication.

The description of the switches are (from the ssh man page):

• -C : Compression
• -2 : Use SSHv2
• -q : quiet!
• -T : Disable pseuto-tty allocation
• -n : Prevents reading from stdin (you need to have private key authentication set up, to prevent password authentication)
• -N : Do not execute a remote command (or launch a shell). Just use the ssh process for port forwarding
• -D : Allocate a socket to listen on the local side. When a connection is made to this port it is located to the remote machine. Makes SSH work as a SOCKS server. Only root can forward privileged ports like this.

From here, you set up Firefox or your browser of choice to use a Socks proxy on localhost:8080. The man page says that SOCKS4 and SOCK5 should both work but I had to use SOCKS v4, SOCKS v5 did not seem to work for me.

Have you ever come across a server that is doing a lot of traffic? Maybe you have logged in to see a process running at 100% CPU, so you know the culprit, but instead of kill -9ing it, wouldn’t it be great to see what exactly it is up to? Or even if you see a process and don’t know exactly what it is doing, and you are just curious what it is up to?

As with most issues there are several ways to skin this cat. You can use tcpdump or wireshark to sniff the all of the network traffic on the device. If you know the port the program is running on (you can use lsof for that), you can restrict traffic to that port. But what if the program is jumping ports, or even uses a side-port for some sort of data transmission (UDP?).

The main problem going down this route is that on a server that is doing any significant bit of traffic, it is like sorting through a needle in a haystack. If you have a single process that is taking up all of your bandwidth, you can probably find it pretty fast. But if the process is not doing a ton of traffic it can be hard to track down.

Strace to the rescue

You can use the great program strace to sniff the network data that an executed program is doing, or even a currently running program. This works well because if you are trying to isolate the network traffic a currently running process, your options can be limited. Using strace is the only way that I know of to see ALL of the traffic coming from a process.

To check the traffic of a currently running process X:

strace -p X -f -e trace=network -s 10000

The command breaks down:

• -p: process ID
• -f: follow forks
• -e: follow set of system calls. In our case, we use trace=network, which follows network system calls.
• -s: set output string sizes. default is 32, which does not  give a lot of information.

Finally if you have a new program to execute and you want to watch the network traffic on it, you execute that command with strace. This would be good to use if you work in a highly secure environment and need to find out what sort of network traffic a distributed binary is doing. Checking for a program ‘Phoning home’ is a good example of that.

Here is the command that launches a new process:

strace -f -e trace=network -s 10000 /usr/bin/command arguments

Hopefully using strace in this manner will help you debug some issues on your server – I know I have used it several times.

I recently came across a typo that existed in a bunch of html files on my web server. I thought it should be easy enough to change, but since it was in a number of files, editing it by hand would be time consuming. Fortunately, there is an easy, one liner command to replace the text in multiple files in a sub directory using recursion.

grep -lr -e '<oldword>' * | xargs sed -i 's/<oldword>/<newword>/g'

This command broken down:

• grep for the word in a files, use recursion (to find files in sub directories), and list only file matches
• | xargs passes the results from the grep command to sed
• sed -i uses a regular expression (regex) to evaluate the change: s (search) / search word / target word / g (global replace)

For more information, see man pages for grep, sed, and xarg. Also it is very handy to learn about regular expressions as they are a valuable tool to any command line programmer!

Update 2009/7/19:

Thanks to reader btr we have a great one-line perl command that will perform the same task:

Perl provides a really nice one-line for this kind of thing:

It also provides the option of creating a backup of each file changed:

mnemonic: PIE (”easy as pie”, etc.)
google “perl pie” and you’ll get lots of info for other uses of this technique.

http://www.linux.org/lessons/short/perlpie/perl_pie.html

Zabbix is an excellent system monitoring package. It does everything from basic availability checking to detailed system resource analysis. It is capable of graphing the variables pulled from the system, and alerting admins if there is a problem or something needed for attention.

Once you have the Zabbix server set up, you need to install the client on any systems you want to monitor. Windows systems have a precompiled binary to install. On linux, unix or freebsd systems you’ll need to compile binaries. If you have a range of systems that are homogeneous, you can port the binary to those or also compile it with static dependencies. Below are steps to compile, configure and install the zabbix client:

Steps to install a Zabbix Client

1. Download zabbix source code from www.zabbix.com; decompress with ‘tar zxvf’ and cd to directory
2. Configure the make program: ./configure –enable-agent
3. Compile and install the program: make install
4. Add zabbix group and user: groupadd zabbix; adduser -g zabbix -s /sbin/nologin -M -p RANDOMPASS zabbix
5. Create log file: touch /var/log/zabbix_agentd.log; chown zabbix.zabbix /var/log/zabbix_agentd.log
6. Copy init script to /etc/init.d. Scripts are located in ./misc/init.d/ and your distro directory.
7. Make sure bin directory in init script is where Zabbix actually compiled to.
8. chmod 755 /etc/init.d/zabbix_agentd
9. chkconfig zabbix_agentd on
10. Copy agent config script to /etc/zabbix/zabbix_agentd.conf. Current one is:

    # This is config file for zabbix_agentd
    # To get more information about ZABBIX, go http://www.zabbix.com
    
    # This is the ip and port of the main zabbix server
    Server=10.0.0.1
    ServerPort=10051
    Hostname=AGENTHOSTNAME
    ListenPort=10050
    # ListenIP=10.0.0.2
    StartAgents=5
    DisableActive=1
    DebugLevel=1
    PidFile=/var/tmp/zabbix_agentd.pid
    # LogFile=/var/log/zabbix_agentd.log
    LogFileSize=1
    Timeout=10

11. Start the zabbix service: service zabbix_agentd start
12. Open firewall for zabbix port (10050) if necessary.
13. Log into Zabbix on the server, Add server to hosts – use correct templates and groups depending on what type of server it is.
14. Add monitoring and notification as appropriate.
15. Consider if all necessary services are being monitored; test that detection of down services and notifications work properly.

Recently I started seeing this error while provisioning Xen VPSes on a server with 8GB of RAM:

I checked the memory, and sure enough, all of the VPSes were using up the memory. More specifically, dom0 (the main server) had ‘ballooned’ and begun to take up 3GB of RAM, this is more than the system should need:

Basically when memory ballooning is set, the memory available to the main hypervisor will have a minimum value set and will balloon according to the current requirements. This setting is in the config file which for me is at /etc/xen/xend-config.sxp:

I changed this line to:

I also modified grub to set an absolute amount of memory to dom0:

The value for dom0_mem sets the system memory at boot time. This will be set on the kernel line. Make sure this is large enough to handle system tasks; 1.5G seems to be good for me right now.

Reboot the server to set the memory for dom0.

You could also set the memory usage for dom0 temporarily by using the ‘xm mem-set’ command:

This would not apply after a reboot. Good luck!