Posts

SSH – Unspecified GSS failure

Recently came across a problem with one system authentication to another via ssh.

I added the public ssh key to ~/.ssh/authorized_keys entry. Changed ownership to the proper user and also chmod 600 ~/.ssh/authorized_keys. Still no dice.

Using ssh -vvvv the following error returned on pubkey authentication:

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

Well, that is pretty non-specific.

Since permission problems are the number one issue I have with getting ssh authentication working, and how I thought I set permissions right, my mind then wondered if SELinux permissions were causing a problem. Most providers disable SELinux right off the bat because of the ‘problems’ it causes, but some don’t. It turned out, this one has SELINUX=enforcing. So, let’s fix the SELinux permissions:

/sbin/restorecon -r /root/.ssh

This sets the context as follows:

# ls -Z authorized_keys 
-rw-------. root root unconfined_u:object_r:ssh_home_t:s0 authorized_keys

SSH should now authenticate.

Dead linux users?

Not dead as in dead, but dead as in the user has logged out of the system and for some reason their shell is still open. This might happen if your system crashes before you can log out, there are network problems and you are disconnected, or a number of other reasons. This article explains how to log these “dead” shell users out.

I use the command “w” to find out who is logged in and how long they have been idle. Compare this to the “who” command:

# w
07:47:18 up 73 days, 9:15, 2 users, load average: 0.43, 0.17, 0.11
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 pool-ip-addr 07:44 0.00s 0.04s 0.00s w
root pts/2 pool-ip-addr 07:45 1:26 0.02s 0.02s -bash
# who
root pts/0 Sep 22 07:44 (pool-ip-addr.phlapa.fios.verizon.net)
root pts/2 Sep 22 07:45 (pool-ip-addr.phlapa.fios.verizon.net)
#

You can also see the command they are currently running, and the TTY they are on. In this case you can see I ran the “w” command, so you can tell that pts/0 is the current session – and pts/2 is the “other” connection. You get this information from the “who” command as well, but “w” adds the idle time (which you can get with who -i, actually). Anyway you can can use either one, it is just a matter of preference.

First find the process list:

# ps fax

10585 ? Ss 0:05 /usr/sbin/sshd
16984 ? Ss 0:00 \_ sshd: root@pts/0
16986 pts/0 Ss 0:00 | \_ -bash
17068 pts/0 R+ 0:00 | \_ ps fax
17033 ? Ss 0:00 \_ sshd: root@pts/2
17035 pts/2 Ss+ 0:00 \_ -bash

Look for the sshd line, this is the ssh server (hopefully you aren’t using telnet any more!)

You can see it shows the pts/0 login, and the pts/2 login.

Find the parent process number of the “other” shell that is logged in. In this particular case, the number is “17033”

Kill that process (Note: you must be the root user to do this, use “sudo” or the like if you are in Ubuntu):

# kill 17033
or
# kill -9 17033

Which will force the other idle shell to log out.

Check “w” again to make sure they are logged out.

Block brute force password attempts via SSH

If you are a system administrator of a linux system, you may find the following log entries familiar:
Sep 15 02:00:30 sol sshd[16364]: Failed password for invalid user test from ::ffff: 61.167.x.x port 53382 ssh2
Sep 15 02:00:30 sol sshd[16365]: Failed password for invalid user test from ::ffff: 61.167.x.x port 53394 ssh2
Sep 15 02:00:30 sol sshd[16366]: Failed password for invalid user test from ::ffff:61.167.x.x port 53396 ssh2
Sep 15 02:00:28 sol sshd[16366]: Invalid user test from ::ffff: 61.167.x.x
Sep 15 02:00:28 sol sshd[16370]: Invalid user test from ::ffff:61.167.x.x

Many, many times over. These are caused by an brute force attack from the remote host. Most likely this is another compromised machine, checking your machine for easy to guess username and password combinations. It could be someone manually trying to run a password cracking program on your ssh server too. In either case, the remote system really has no business touching your machine. This situation needs an automated solution to block this IP from even getting to your machine. Doing this real-time is essential as well.

Enter the Free APF + BFD scripts from R-fx Networks. These programs work in conjunction with one another to monitor for brute password attempts on your system, then ban the attacking host.

First install the APF (Advanced Policy Firewall) script [Download]

Then install the BFD (Brude Force Detection) script [Download]

When it finds a host that has tried and failed to log in too many times, or has tried too many users who don’t exist on your system, it blocks them in your firewall and e-mails you a message:

The remote system 61.167.x.x was found to have exceeded acceptable login
failures on somehost.com; there was 63 events to the service sshd. As such the
attacking host has been banned from further accessing this system. For the integrity
of your host you should investigate this event as soon as possible.

Executed ban command:
/etc/apf/apf -d 61.167.x.x {bfd.sshd}

The following are event logs from 61.167.x.x on service sshd (all time stamps are GMT -0400):

Sep 15 02:00:27 sol sshd[16364]: Invalid user test from ::ffff:61.167.x.x
Sep 15 02:00:27 sol sshd[16365]: Invalid user test from ::ffff: 61.167.x.x
Sep 15 02:00:28 sol sshd[16366]: Invalid user test from ::ffff: 61.167.x.x
Sep 15 02:00:28 sol sshd[16370]: Invalid user test from ::ffff:61.167.x.x
Sep 15 02:00:30 sol sshd[16364]: Failed password for invalid user test from ::ffff: 61.167.x.x port 53382 ssh2
Sep 15 02:00:30 sol sshd[16365]: Failed password for invalid user test from ::ffff: 61.167.x.x port 53394 ssh2
Sep 15 02:00:30 sol sshd[16366]: Failed password for invalid user test from ::ffff:61.167.x.x port 53396 ssh2
Sep 15 02:00:31 sol sshd[16370]: Failed password for invalid user test from ::ffff:61.167.x.x port 53412 ssh2
Sep 15 02:00:31 sol sshd[16372]: Invalid user test from ::ffff:61.167.x.x
Sep 15 02:00:32 sol sshd[16373]: Invalid user test from ::ffff: 61.167.x.x

In my experience it works great and is a very easy to install!