Tips & Tricks for Technologists & System Administrators | About & Contact

Category: System Administration


SSH – Unspecified GSS failure

Posted 10th September in CentOS, Linux, Shell, System Administration. No Comments

Recently came across a problem with one system authentication to another via ssh.

I added the public ssh key to ~/.ssh/authorized_keys entry. Changed ownership to the proper user and also chmod 600 ~/.ssh/authorized_keys. Still no dice.

Using ssh -vvvv the following error returned on pubkey authentication:

1
2
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

Well, that is pretty non-specific.

Since permission problems are the number one issue I have with getting ssh authentication working, and how I thought I set permissions right, my mind then wondered if SELinux permissions were causing a problem. Most providers disable SELinux right off the bat because of the ‘problems’ it causes, but some don’t. It turned out, this one has SELINUX=enforcing. So, let’s fix the SELinux permissions:

1
/sbin/restorecon -r /root/.ssh

This sets the context as follows:

1
2
# ls -Z authorized_keys
-rw-------. root root unconfined_u:object_r:ssh_home_t:s0 authorized_keys

SSH should now authenticate.


How to Turn PHP into an RPM The Easy Way with FPM

So you have a custom PHP binary that you wish to distribute across multiple servers.

The “old” way to do this is to create a bash script in order to pull down the source code, run ./configure and then make; make install. But this is 2013, we have a much more civilized way to create this package now, so that you can distribute that RPM across multiple nodes or even include it in your own yum repository.

We’re going to use FPM (Effing Package Manager) to do the job for this. It is a little bit confusing because PHP also includes the acronym FPM for FastCGI Process Manager. Our package manager FPM is a ruby gem, so install the prerequisites first (CentOS 6 directions here), along with fpm:

1
2
# yum install ruby ruby-devel rubygems
# gem install fpm

To start, compile PHP the way you normally would, ie:

1
2
3
4
5
6
'./configure'  '--program-prefix=' '--prefix=/usr' \
'--exec-prefix=/usr' '--bindir=/usr/bin' '--enable-ftp' \
'--enable-calendar' '--with-libxml-dir=/usr' '--enable-mbstring' \
'--enable-mbregex' '--with-gd' '--enable-zip' '--with-mcrypt' \
'--with-mhash' '--with-libdir=lib64' '--enable-fpm' '--with-fpm-user=www' \
'--with-fpm-group=www' '--with-mysql=mysqlnd' '--with-mysqli=mysqlnd'

Then make it as normal.

1
2
# make
... Lots of output

Now instead of running make install like normal, we tell it to output to another directory, just some temporary directory on the server:

(Note, if compiling another program, the switch to make it install into another directory may be different, such as DESTDIR=/tmp/installdir)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
make install INSTALL_ROOT=/tmp/installdir
Installing PHP SAPI module:       fpm
Installing PHP CLI binary:        /tmp/installdir/usr/bin/
Installing PHP CLI man page:      /tmp/installdir/usr/share/man/man1/
Installing PHP FPM binary:        /tmp/installdir/usr/sbin/
Installing PHP FPM config:        /tmp/installdir/etc/
Installing PHP FPM man page:      /tmp/installdir/usr/share/man/man8/
Installing PHP FPM status page:      /tmp/installdir/usr/share/fpm/
Installing build environment:     /tmp/installdir/usr/lib/build/
Installing header files:          /tmp/installdir/usr/include/php/
Installing helper programs:       /tmp/installdir/usr/bin/
  program: phpize
  program: php-config
Installing man pages:             /tmp/installdir/usr/share/man/man1/
  page: phpize.1
  page: php-config.1
/usr/local/src/php-5.3.25/build/shtool install -c ext/phar/phar.phar /tmp/installdir/usr/bin
ln -s -f /usr/bin/phar.phar /tmp/installdir/usr/bin/phar
Installing PDO headers:          /tmp/installdir/usr/include/php/ext/pdo/

Now PHP is happily installed in /tmp/installdir. Which is great, but we can’t use it until it we package it up for regular installation. Run the FPM command to pack it all up:

1
2
3
4
5
# fpm -s dir -t rpm -n php-fpm -v 5.3.25 -C /tmp/installdir \
   -p php-fpm_VERSION_ARCH.rpm -d openssl -d pcre -d bzip2 -d curl \
   -d libjpeg -d libpng -d freetype -d gmp -d libmcrypt -d libmhash \
   -d libxml2 usr/bin usr/share/man/man1 usr/sbin etc \
   usr/share/man/man8 usr/share/fpm usr/lib/build usr/include/php

What do the options mean?

I’ll go through these options:

  • -s dir: source is a directory
  • -t rpm: we are creating an rpm
  • -n php-fpm: we are creating a package with the name php-fpm
  • -v 5.3.25: the version of the rpm (so you can manage upgrades)
  • -C /tmp/installdir: the base location to take the files from
  • -p php-fpm_VERSION_ARCH.rpm: fpm will replace the VERSION and ARCH macros automatically with your version number and architecture
  • -d openssl -s pcre …: these are all of the rpm dependencies that you’ll want on the machines installing the package. If you don’t include the proper ones (to match your source system, and which are included in ./configure, you’ll get library errors when you try to launch the program.
  • usr/bin usr/share/man/man1 usr/sbin etc usr/share/man/man8 usr/share/fpm usr/lib/build usr/include/php: the directories you wish to include in your package file

When all is done, you should have your rpm file located in /tmp/installdir. You can install it directly on any system with rpm -i <rpm name>

Before you package up your files, you can include any other files you want. For example, you could include the init script or php.ini file. But if you are managing the system using puppet or chef, it would be better to manage this configuration file through there rather than from the rpm file. This works really well as a ‘building block’ for your programs when you add these custom rpms into your own repository. That way, you can have your system install custom versions of the binaries where applicable but still have the niceties that come from an systems automation with puppet or chef.

Enjoy!


Simple Guide To Signing RPMs with FPM

I’ve been using the excellent fpm (Effing package manager!) tool for automatically generating rpms from source (for example, creating a custom compiled php-fpm binary and then wrapping it for install in an rpm for distribution from our own repository). FPM also creates .debs and other binaries as you need them.

To ensure we have a secure infrastructure we wanted to make sure that the rpms are correctly signed so that the gpg check can be completed on them when installed.

First, generate the gpg public/private key pair:

1
gpg --gen-key

Fill out the required information. The entropy generation step may take a long time to complete.

Export a text version of the public key with this command, for later use in your repository:

1
gpg --export -a 'Dave Drager' > ~/RPM-GPG-KEY-reponame

fpm docs state that you just need to add the –rpm-sign to your fpm command. However, this generated the following error for me:

1
2
error: You must set "%_gpg_name" in your macro file {:level=>:error}
Pass phrase check failed {:level=>:error}

The reason for this is that it uses the settings located in your ~/.rpmmacros file. Find your key ID:

1
2
3
4
5
6
# gpg --list-keys
/user/.gnupg/pubring.gpg
------------------------
pub   2048R/94E8C1F6 2013-04-24
uid                  Dave Drager <ddrager@xxx>
sub   2048R/94E8C1F6 2013-04-24

In this sample, the KEY_ID would be “94E8C1F6″

1
2
3
4
%_signature gpg
%_gpg_path /path/to/.gnupg
%_gpg_name KEY_ID
%_gpgbin /usr/bin/gpg

From here, the fpm command runs without issue.

1
2
3
4
# fpm -s dir -t rpm -n php-fpm-xda -v 5.2.23 -C /tmp/installdir -p php-fpm-xda_VERSION_ARCH.rpm -d openssl -d pcre -d bzip2 -d curl -d libjpeg -d libpng -d freetype -d gmp -d libmcrypt -d libmhash -d libxml2 --rpm-sign usr/bin usr/share/man/man1 usr/sbin etc usr/share/man/man8 usr/share/fpm usr/lib/build usr/include/php
Enter pass phrase:
Pass phrase is good. {:level=>:error}
Created rpm {:path=>"php-fpm-xda_5.2.23_x86_64.rpm"}

This is an alias of the rpm --addsign command.

Now let’s check the signature on this rpm.

1
2
# rpm --checksig php-fpm-xda_5.2.23_x86_64.rpm
php-fpm-xda_5.2.23_x86_64.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK (MISSING KEYS: (MD5) PGP#94E8C1F6)

Whoops, it doesn’t seem to be signed properly. Well, that is because the rpm system doesn’t know to trust this GPG key yet. Import with the following command:

1
rpm --import ~/RPM-GPG-KEY-reponame

Note that this is the plain text file from before. Your RPM system knows to trust this key, and now if you verify the file:

1
2
# rpm --checksig php-fpm-xda_5.2.23_x86_64.rpm
php-fpm-xda_5.2.23_x86_64.rpm: rsa sha1 (md5) pgp md5 OK

This RPM is now ready for use in a repo. In the .repo file, use the following to ensure it follows the correct signature:

1
2
gpgcheck=1
gpgkey=http://repo.xxxx.com/RPM-GPG-KEY-reponame

All key names have been changed to protect the innocent.


Track TCP and UDP Connections With Zabbix

There is a very easy way to track opened TCP connections in Zabbix. Although it does not come built in to the Zabbix agent, it is simple enough to get. This method will allow you track the open number of TCP and UDP connections, as well as track TCP connections in TIME_WAIT.

TCP UDP Sockets Zabbix

Rather than using

1
grep

to count numbers in netstat, which can take an extraordinary amount of time on busy servers, I’ve used /proc/net/sockstat to get this data.

Add the following parameters to zabbix_agentd.conf:

1
2
3
4
5
6
7
8
UserParameter=sockstat.sockets,cat /proc/net/sockstat|grep sockets|cut -d' ' -f 3
UserParameter=sockstat.tcp.inuse,cat /proc/net/sockstat|grep TCP|cut -d' ' -f 3
UserParameter=sockstat.tcp.orphan,cat /proc/net/sockstat|grep TCP|cut -d' ' -f 5
UserParameter=sockstat.tcp.timewait,cat /proc/net/sockstat|grep TCP|cut -d' ' -f 7
UserParameter=sockstat.tcp.allocated,cat /proc/net/sockstat|grep TCP|cut -d' ' -f 9
UserParameter=sockstat.tcp.mem,cat /proc/net/sockstat|grep TCP|cut -d' ' -f 11
UserParameter=sockstat.udp.inuse,cat /proc/net/sockstat|grep UDP:|cut -d' ' -f 3
UserParameter=sockstat.udp.mem,cat /proc/net/sockstat|grep UDP:|cut -d' ' -f 5

Then you can either manually import these UserParameters into Zabbix, or use this template (if you’re using Zabbix 2.x).

Download the sockstat Zabbix template

It includes the items and a graph for the socket stats. Enjoy!


Using rsync on an Alternate SSH Port or with OpenSSH Keys

Posted 9th October in Linux, Shell, System Administration. 4 Comments

Best practices state that you should run ssh on an non-standard port. Unfortunately some programs use port 22 by default and it isn’t obvious what the switch is to change this port.

One of these programs is the eminently useful rsync. It states how to do this in the man file, but it is hidden and non-obvious. You simply add the option via the -e command to pass ssh options. -e is the shell rsync uses to connect to a remote host.

Using rsync on an alternate ssh port

1
rsync -avz -e "ssh -p $port" username@ip:/path/to/files/ /local/files/

Using rsync with an openssh key

1
rsync -avz -e "ssh -i /path/to/private/key" username@ip:/path/to/files/ /local/files/

Or with both an alternate port and openssh key:

1
rsync -avz -e "ssh -i /path/to/private/key -p $port" username@ip:/path/to/files/ /local/files/

This can be used in a bash script if you set the $port variable or directly on the command line by using the set port instead of $port.

Here are some other useful pages regarding using rsync. If you don’t use rsync for remote file transfer – I highly recommend it. The transfer speed is much faster than scp over ssh.


Simple Sysadmin Trick: Using tcpdump To Sniff Web Server Traffic

Posted 2nd October in Linux, Shell, System Administration. 1 Comment

Sometimes, you just have to look into the raw data to see what your web server is doing. The logs might not show you enough detail or you suspect something is going on which is just not shown in the log files. Or, as in my case, logging is turned off because of too much activity.

The excellent tcpdump utiliy comes to the rescue here. I recommend you get more familiar with the tcpdump man page. Here is the command you can use, in a nutshell:

1
tcpdump -nl -w - -i eth1 -c 500 port 80|strings

or alternatively with just tcpdump (Thanks Chris!):

1
tcpdump -nl -s 0 -A -i eth1 -c 500 port 80

Your command line will print out all traffic exiting your server from port 80, headers and all. Lets look at the options in more detail.

  • -n: Don’t convert addresses (i.e., host addresses, port numbers, etc.) to names.
  • -l: Make stdout line buffered. Useful if you want to see the data while capturing it.
  • -w: Write the raw packets to file rather than parsing and printing them out. (Sent to stdout)
  • -i: Interface you want to sniff on, usually eth0 or eth1, but depends on your system.
  • -c: Number of packets to capture
  • port: port 80, duh :)
  • -A: Print each packet (minus its link level header) in ASCII.
  • -s: size

Now, depending on your web server configuration, you will probably have gzipped content which comes out as garbled characters. To strip all that out, just pipe it through strings.

The output will look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
GET /about/comment-page-1 HTTP/1.0
Accept: image/gif, image/jpeg, image/pjpeg, application/x-ms-application, application/vnd.ms-xpsdocument, application/xaml+xml, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*
User-Agent: Mozilla/2.0 (compatible; MSIE 3.02; Windows CE; 240x320)
Referer: http://quittingsoda.com/about/comment-page-1#comment-6637
Host: quittingsoda.com
Cookie: comment_author_xxx=sakyjartory; comment_author_email_26e707905b5fd6e7139333eb1dab208f=olfaexxxx; comment_author_url_26e707905b5fd6e7139333eb1dab208xxx
HTTP/1.1 200 OK
Date: Wed, 03 Oct 2012 01:49:19 GMT
Server: Apache/2
X-Powered-By: PHP/5.2.17
X-Pingback: http://quittingsoda.com/xmlrpc.php
Vary: Accept-Encoding,User-Agent
Connection: close
Content-Type: text/html; charset=UTF-8
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head profile="http://gmpg.org/xfn/11">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>About This Site | Quitting Soda</title>

Let me know if you have any suggestions for improving this simple command line program for sniffing your web server traffic! Netcat is an essential tool that every linux administrator should get to know better.


Evaluating FTP Servers: ProFTPd vs PureFTPd vs vsftpd

Usually, I will try to push clients towards using SCP (via a client such as WinSCP), however inevitably there are clients who do not understand this new method of accessing their files securely online, and who for one reason or another insist on using FTP for their online file access. As they say – the customer is always right?

Anyway, there are currently 3 mainstream FTP servers available via the yum command on CentOS 6.x. PureFTPd, ProFTPd and vsftpd. So which FTP server is the best? I will summarize the servers below, or skip to the summary.

ProFTPd

ProFTPd is a modular FTP server which has been around for a long time. The large control panels (cPanel, DirectAdmin) all support ProFTPd and have for years.

The most feature rich of the bunch is certainly ProFTPd. There are a ton of plugins available for it, and the creator of it modeled its configuration architecture much like Apache’s – it is also using the GPL for licensing.

Configuration of ProFTPd is fairly straight forward, and example configuration files abound at a quick search of Google.

ProFTPd is available on a wide variety of system architectures and operating systems.

ProFTPd Security

Of the bunch, ProFTPd has the most CVE vulnerabilities listed. The high number is most likely an indicator of ProFTPd’s widespread use which makes it a target of hackers.

ProFTPd CVE Entries: 40
Shodan ProFTPd entries: 127

PureFTPd

PureFTPd‘s mantra is ‘Security First.’ This is evident in the low number of CVE entries (see below).

Licensed under the BSD license, PureFTPd is also available on a wide-range of operating systems (but not Windows).

Configuration of PureFTPd is simple, with a no-configuration file option. Although not as widely used as ProFTPd, PureFTPd has many configuration examples listed online.

PureFTPd Security

PureFTPd’s “Security First” mantra puts it at the lead in the security department with the fewest security vulnerabilities.

PureFTPd CVE Entries: 4
Shodan Pure-FTPd Entries: 12

vsftpd

vsftpd is another GPL-licensed FTP server, which stands for “Very Security FTP daemon.” It is a lighweight FTP server built with security in mind.

Its lightweight nature allows it to scale very efficiently, and many large sites (ftp.redhat.com, ftp.debian.org, ftp.freebsd.org) currently utilize vsftpd as their FTP server of choice.

vsftpd Security

vsftpd has a lower number of vulnerabilities listed in CVE than ProFTPd but more than PureFTPd. This could be because, since its name implies it is a secure FTP service, or because it is so widely used on large sites – that it is under more scrutiny than the others.

vsftpd CVE Entries: 12
Shodan vsftpd entries: 41

Summary & FTP Server Recommendations

#

Considering the evaluations above, any server would work in a situation, however generally speaking:

  • If you want a server with the most flexible configuration options and external modules: ProFTPd
  • If you have just a few users and want a simple, secure FTP server: PureFTPd
  • If you want to run a FTP server at scale with many users: vsftpd

Of course, everyone’s requirements are different so make sure you evaluate the options according to your own needs.

Disagree with my assessment? Let me know why!