OK - it was my fault, I admit it and I'll take my licks.

I was in the midst of doing a number of site updates of glFusion and got distracted midway through one of my own sites - and left the installation directory in place for over a month. No wonder the site was hacked. I should know better and now I ensure that this tool repository is removed no matter what, and that permissions are changed and things tidied up before I let myself get pulled away.

I've informed the glFusion support as well as SANS - and dumped a copy of the code on them. It turned out to be a couple of fairly well known tools, c99shell and fx29shell - with their names changed to css.php and cyber.php respectively.

I twigged to the exploit because the number of emails the hack created to one user at Yahoo got the Yahoo email system hot and bothered, and it slowed down reception of the stream long enough for a timeout message to be generated (4 hours) by the system - and I got the message since I'm the recipient of last resort for all such messages on the server.

That got me looking and I found the hack and disabled it. I spent most of the rest of the day inspecting the machine and documenting the hack - a nice sunny Sunday I'd rather have back thank you.

You may find some nuggets in the rest of the story

I monitor the activities of a number of servers. They send me a log report daily and various special activities such as backups also send me e-mail after they've finished, showing what was done.

Today a strangeness caught my eye.

/root/bin/single-roller1.sh georgia
mv: cannot stat `5/georgia': No such file or directory
mv: cannot stat `4/georgia': No such file or directory
mv: cannot stat `3/georgia': No such file or directory
mv: cannot stat `2/georgia': No such file or directory
mv: cannot stat `1/georgia': No such file or directory
mv: cannot stat `0/georgia': No such file or directory
job 578 at 2009-03-14 01:28

 

Over the past few months I've been moving many of the Linux systems I look after over to CentOS 5.2 - the latest free version of Red Hat's system.

One of the ongoing problems has been intermittant timeouts by some of the users of ftp. All of them use Proftpd.

After doing some tcpdump analysis, one customer and I noted that no matter what the settings in the proftpd.conf file, the system was doing a IDENT callout which was taking up to 30 seconds to time out.

Over the years I've run into all manner of problems where systems have run out of memory and swap space. With the latest versions of the Linux kernel there are some new tools that allow you to control what the system does when this happens.

A recent discussion on the Exim (mail transport agent) mail list got me to looking around a bit as I've had a problem with my workstation running out of swap/RAM (and it has lots of both) when I keep lots of Firefox windows open. One of the comments lead me to do a search for "overcommit" on Google, and that lead me to an article in Red Hat's magazine and from there things got interesting.

As many of you may know, I have a lot of computers in my home. I deal with huge amounts of data (mostly video but a lot of other stuff too) and just having it all online means I have more than 10 systems here.

But I have a core of 4 systems that I work with daily and that make up my primary set of working files: My old workstation (pacdat), my new workstation (video), my file server (NFS1) and my backup and domain name master (NETFS)

A few months ago I decided to move much of the data that still resided on my old workstation (P4 2.0GHz- called "pacdat") to a NFS file server (NFS1), including my home directory which is huge.

The old machine had several sets of mirrored drives of various sizes - usually the "sweet spot" size for whenever I purchased them - from 160 Gigs to 300 Gigs. My home directory has grown to outstrip each of these and in fact now has links to several such pairs of RAID 1 arrays. It was my intention to build a RAID 5 array of 320Gig drives that would do me for at least a year or so of growth at present rate - and host them on a single computer that I could mount from several of the systems in my home as needed.

All was going well - until Mother Nature stepped in a couple of weeks ago.

read on for the tail of woe

I was setting up one of my servers to be a workstation for my wife, and had to bring in some video drivers. The latest version of the drivers referenced a new kernel, and of course I have not been updating the kernel because on this machine I also run VMWare's free server instance. VMWare requires that their software be linked against the latest kernel development kit so updating the kernel all the time requires re-linking the software, so I use the "exclude=kernel-*" line in /etc/yum.conf to not automatically update the kernel each time I do other updates.

So I updated the kernel - version 2.6.25.4-10.fc8-i686 - and rebooted.

Then I went to re-link the VMWare software and it wouldn't link :(

I use the "vmware-any-any" patches - version 116 being the latest from http://www.miscreant.org/files/rpms/ - but in this case the compile failed and nothing I did would bring it back to working status.

So I put the incantation "vmware-any-any 2.6.25.4-10.fc8-i686" into Google and came up with a single hit - a French language blog entry at forums.fedora-fr.org which Google kindly offered to translate for me

It turns out I would have saved myself a whole mess of trouble if I'd kept up with the updates from VMWare - seems their latest version, 1.0.6, no longer needs the patches. Thank you REMI

I downloaded the Linux server file from www.vmware.com and the install went without a hitch.

While installing a test copy of Ubuntu on a server the other day I noticed a package called SSHFS being installed which struck me as a "good idea"

I do almost 100% of my system administration on remote systems (even if only in the next room or downstairs) via ssh - and with my work doing the video grabbing of various nest-cams and such for Hancock Wildlife Foundation I end up moving fairly large amounts of video files around from server to server to archive, etc. Mostly I use rsync and scp - but the thought of being able to use such simple tools as "mv" to move a file from one server to another over a secure link (of course you can do this with NFS but mostly that's not secure) made me stop and take notice.

Just trying to keep files from being duplicated on machines is one of the hardest parts. We accumulate them on systems that tend to have limited disk space - then after a day or two - move them to archive servers that are not nearly as well connected to the net (lower speed links) but which have massive amounts of storage on them. To date this has been pretty much a manual process - albeit one that doesn't take much time for me to do, but takes quite a bit of time once set to doing it - so I have to check every day or two - at least before the main systems run out of space.

And of course there have been times when I've missed checking and the systems have run out of space :(

So... sshfs looks very interesting!

A quick Google for sshfs took me to fuse.sourceforge.net/sshfs.html

It appears that the latest kernels, including the ones I have on my workstations, already have the fuse kernel module in them. I have a /dev/fuse device file already for instance.

I downloaded the fuse source from the handy link provided, then thought "hmmm... if the fuse kernel mod is there, maybe fuse is available via YUM" and lo and behold it was!

"yum install fuse" works as advertised

so I downloaded the sshfs code and ran configure. It was looking for some fuse modules so I returned to yum.

"yum install fuse-devel" got me two more modules and sshfs was happy!

after "./configure ; make ; make install" I now have /usr/local/bin/sshfs and a  module (sshnodelay.so) in /usr/local/lib

The docs suggest not running this as root. Now I have to admit that I tend to do everything as root on my machines, but then I've already gone through the "oops" stage of Unix/Linux system admin and both keep lots of backups and know how to re-install/restore - and have not had to do so in quite a number of years (over 15). You may want to take their suggestion.

I also have a number of "blank password" ssh keys set up to allow unattended backups between various machines. They're protected in other ways too so not much of a security hole. Again, you may not want to do this with your systems unless you really know what you are doing with firewalls and such.

Hey - this is great! Another arrow in the admin quiver.