The Digital Rag
Real World Information in a Virtual World
Sign Up!
Login
Welcome to The Digital Rag
Thursday, June 20 2013 @ 04:59 AM PDT

Computer System Administration

I am not a programmer. In fact my long-time partner from back in the days of our ISP, Wimsey, told me one day that if he ever again caught me trying to program, he'd break my fingers; this despite the fact that I've been taught programming formally at university levels. You see, I just don't do it often enough to be really good at it. In fact, when I do get into doing much programming I turn into a real ogre and don't like myself much. That's because true programming requires "being at one with the code" which in turn requires focus for long periods of time on that code - and when I do that I become no longer very human or social.

Having said all that, it turns out that I'm actually a pretty good system administrator and trouble shooter. In fact that's been a large part of what I've done over the past 30+ years of being self employed. The thing is I've done the system admin on Unix and Linux systems rather than the "more popular" Windows systems. That way I have time to do other things since the systems I look after do much of the work themselves and only require me when something really breaks.

There has been lots written about the difference between administering Windows systems and Unix/lLinux systems - both from the cost and complexity point of view. My opinion is that Windows needs more personal hand-holding than *nix systems - and that many of those whose job it is to administer Windows systems simply are not worth what they're paid - but then maybe that's because I sometimes end up being called in when they have a real problem, even though I've never taken any MCSE courses.

Here you'll find some of the more interesting items I come across. There's another topic only available to members where some of the real secrets are kept :)


 

eMail Article To a Friend View Printable Version

Of Mirrors and Luck - a Tail of Losing The Coin Flip

Unlike Harrison Ford in Indiana Jones and the Last Crusade, I didn't choose wisely. In fact, there was no obvious information on which to base the choice, I simply picked one of the two RAID 1 drives (mirrors) and removed it - and the system immediately started to run faster; at least for about a day.

A lot of analysis had been done up to that point, and finally one screen, updating every 1/10 second, showed me that the process that was sticking the system up was to do with the RAID 1 drives on the root partition.

I'd had no hardware errors reported by smartd or RAID errors by mdmonitor.

The only thing I had done recently was update the kernel to the latest version on Fedora Core 14, the OS version on the system. Could this have done something? I found an obscure reference to running RAID and problems with Western Digital Caviar Green drives - this system uses Caviar Blacks, but... maybe.

So I broke the mirror. 

mdadm /dev/md0 --fail /dev/sda3

I could have chosen /dev/sdc3, but that 50/50 chance somehow made me type the first drive's name instead. It didn't seem to be a specific hardware problem with one drive but instead simply using the type of drive in a RAID, so no obvious reason to choose one over the other.

Then the problem surfaced again, but by this time I'd re-formatted and tested the removed partition (thank goodness I didn't do this to the rest of the drive), and there was simply no way back. Two days later, I'm still recovering files from the other partitions and getting the system back running all the various things it ran. Looking back, I'm not sure that there was anything I could have done differently except had the good luck to pick the correct drive - in general I don't gamble because "if you don't play, you can't lose" - in this case I lost.

It all started with the Linux server system I host becoming slower and slower. The real load on it at this time of year is trivial, so load should not have been a factor at all. I started looking for other reasons: denial of service attack, huge directories (some of which are growing to contain millions of photos at this point) and a number of other things went through my head and were tested and didn't show as the real cause.

Meanwhile, the all-year core members of Hancock Wildlife Foundation (the major tenant on the system) were starting to really complain - their sessions were disappearing, posts were duplicated, response to simply looking at some of the pages was measured in minutes at times.

The steps along the way to full recovery of the data should prove interesting if you're faced with similar problems, no matter whether caused by choosing wrongly as I did, or by real hardware errors. Sometimes the best data is on the "failed" drive - and getting it off once the other drive is toast is not an easy task.


eMail Article To a Friend View Printable Version

Firefox 5.0 is Out - Installing the x86_64 Version on Linux

System Administration Tidbits

I was reading an article on Slashdot that said Firefox 4 is being discontinued. Since I don't use Firefox by default, I've been pretty much using Chrome, this caused me to take a look at what version I was running on my workstation which is Fedora 11 at the moment; version 3.5.9, which is the one the repository has for yum install.

So, I guess I completely missed out on Firefox 4 - what about Firefox 5?

So - off to the Mozilla home page where I was immediately redirected to a page with a big green "Firefox - free download" button. Click the button, download the file; hmmm - where to put it?

Ooops - the button, in small print, says Linux i686 and I'm running X86_64 on my AMD quad core - wrong version??? Maybe they put both in the one download?

Un-tar the files into /usr/lib64, where the current version lives - ensuring not to crash over top of the ones already there - and use "file firefox-bin" to peer inside the file - nope, it's 32 bit - but where is the 64 bit version?

Back to Google


eMail Article To a Friend View Printable Version

Free is "free" - WIFI Security (updated)

System Administration Tidbits

WIFI hotspots in the world that are otherwise "open" should be set with a WEP2 password of "free" to protect your information from snooping. Here's why.

Every now and then someone comes up with an idea that is a "head smacker" - you know, you smack your head and say "why didn't I think of that?"

So it is with an article by Chester Wsniewski in the Naked Security section of Sophos.com (Sophos is a vendor of computer/network security products)

As Chester points out, the recently crafted "Firesheep" plugin for Firefox browser that lets you see other WIFI users' Facebook login info and which is somewhat mitigated by "Blacksheep," would be fully foiled if all the public (and free) WIFI systems had at least a known WEP2 password, because then all the sessions would be encrypted over the airwaves - and snooping would not be possible at all.

Chester proposes, and I fully back this, that the password for any/all WIFI systems that are otherwise "default" be the simple word "free"

It's so simple!

Manufacturers should start NOW - make sure your WIFI has a password by default, and set that password to "free" - it's so simple, just do it!

Coffee shops and others with free WIFI services - set your system to use WEP2 encryption and set the password to "free"

Laptop and smart-phone users - tell your favorite locations that you want this and tell them why. You don't want to have others snooping your clear-text passwords.

Update: OK - I should have read the comments at the bottom of the article before I posted this but... the basic idea is still sound - ensure that all WIFI is using encryption, one way or another. Read on for more...


eMail Article To a Friend View Printable Version

WatchOCR - A Linux Bootable OCR System

System Administration Tidbits

Sometimes serendipity steps in just as you think you're going to have to re-invent the wheel or (shudder) pay for some proprietary software and run it on Windoze.

This happened late last week as I looked around for ways of converting the publishing PDF of a huge reference book (made using Quark) into something I could put up online as a reference work with good search engine coverage and potential for crowd-sourced editing and extension.

The project is about to be announced as an adjunct to the work being done by Hancock Wildlife Foundation, an organization I've been part of in both technical and managerial roles since its founding in 2006.

The book Raptor Research and Management Techniques is a compendium of papers that deal with all manner of the life management of raptors; eagles, hawks, falcons, ospreys, owls, vultures, etc. - birds of prey.

The problem I ran into is that once the manuscript was in Quark (where all the final editing had been done), the only way to get it out in any "portable" fashion is via conversion to PDF - just one of those things you find when you deal with proprietary software it seems; nothing reads Quark files because they are not documented and the format is protected jealously by the company.

OK - so I have a PDF. I also have tools that can take one apart and do other "interesting things" like convert to Postscript (pdf2ps) which can then be converted to ASCII (ps2ascii) and all the stuff in the PDF Toolkit (pdftk) but... none of them work properly with the format of this book with its two columns and lots of diagrams, etc.

What I needed was some method of running OCR on the file and getting things out that way. I've seen some impressive facilities that accompany scanners and such - and of course only run on their output it seems. 

I and many others have been looking for some open source facility that would do the trick. One that appeared to be "interesting" was Cuneiform, but the information on this Russian software is sparse in English. I had downloaded the source and had a short session trying to get it to run on my Fedora Core 11 box but was missing some libraries, and the docs are for Debian systems with "apt get" instead of "yum" and obviously different package names so I had put it aside for the time being.

The alternative was a Windows binary already configured - low on my list for now but a fall-back.

And lo and behold, along comes a note about a bootable Linux disk with all the things necessary:

WatchOCR.

It's web page contains a link to Cuneiform so maybe the problem is resolved.

Read on to find out how the system performed but the docs differ from reality - and how to make the system at least a bit better and do some other interesting things such as create HTML output from your PDFs.


eMail Article To a Friend View Printable Version

Forgotten Update Leads to Compromised Site

System Administration Tidbits

OK - it was my fault, I admit it and I'll take my licks.

I was in the midst of doing a number of site updates of glFusion and got distracted midway through one of my own sites - and left the installation directory in place for over a month. No wonder the site was hacked. I should know better and now I ensure that this tool repository is removed no matter what, and that permissions are changed and things tidied up before I let myself get pulled away.

I've informed the glFusion support as well as SANS - and dumped a copy of the code on them. It turned out to be a couple of fairly well known tools, c99shell and fx29shell - with their names changed to css.php and cyber.php respectively.

I twigged to the exploit because the number of emails the hack created to one user at Yahoo got the Yahoo email system hot and bothered, and it slowed down reception of the stream long enough for a timeout message to be generated (4 hours) by the system - and I got the message since I'm the recipient of last resort for all such messages on the server.

That got me looking and I found the hack and disabled it. I spent most of the rest of the day inspecting the machine and documenting the hack - a nice sunny Sunday I'd rather have back thank you.

You may find some nuggets in the rest of the story


eMail Article To a Friend View Printable Version

Batch Command Relies on Load Being Low Enough to Run

System Administration Tidbits

I monitor the activities of a number of servers. They send me a log report daily and various special activities such as backups also send me e-mail after they've finished, showing what was done.

Today a strangeness caught my eye.

    /root/bin/single-roller1.sh georgia
    mv: cannot stat `5/georgia': No such file or directory
    mv: cannot stat `4/georgia': No such file or directory
    mv: cannot stat `3/georgia': No such file or directory
    mv: cannot stat `2/georgia': No such file or directory
    mv: cannot stat `1/georgia': No such file or directory
    mv: cannot stat `0/georgia': No such file or directory
    job 578 at 2009-03-14 01:28

 


eMail Article To a Friend View Printable Version

Proftpd timeout problem - solved

System Administration Tidbits

Over the past few months I've been moving many of the Linux systems I look after over to CentOS 5.2 - the latest free version of Red Hat's system.

One of the ongoing problems has been intermittant timeouts by some of the users of ftp. All of them use Proftpd.

After doing some tcpdump analysis, one customer and I noted that no matter what the settings in the proftpd.conf file, the system was doing a IDENT callout which was taking up to 30 seconds to time out.


eMail Article To a Friend View Printable Version

Running Out of Swap on Linux

System Administration Tidbits

Over the years I've run into all manner of problems where systems have run out of memory and swap space. With the latest versions of the Linux kernel there are some new tools that allow you to control what the system does when this happens.

A recent discussion on the Exim (mail transport agent) mail list got me to looking around a bit as I've had a problem with my workstation running out of swap/RAM (and it has lots of both) when I keep lots of Firefox windows open. One of the comments lead me to do a search for "overcommit" on Google, and that lead me to an article in Red Hat's magazine and from there things got interesting.


eMail Article To a Friend View Printable Version

A Tail of Woe - Playing Hardware God and Losing Data

System Administration Tidbits

As many of you may know, I have a lot of computers in my home. I deal with huge amounts of data (mostly video but a lot of other stuff too) and just having it all online means I have more than 10 systems here.

But I have a core of 4 systems that I work with daily and that make up my primary set of working files: My old workstation (pacdat), my new workstation (video), my file server (NFS1) and my backup and domain name master (NETFS)

A few months ago I decided to move much of the data that still resided on my old workstation (P4 2.0GHz- called "pacdat") to a NFS file server (NFS1), including my home directory which is huge.

The old machine had several sets of mirrored drives of various sizes - usually the "sweet spot" size for whenever I purchased them - from 160 Gigs to 300 Gigs. My home directory has grown to outstrip each of these and in fact now has links to several such pairs of RAID 1 arrays. It was my intention to build a RAID 5 array of 320Gig drives that would do me for at least a year or so of growth at present rate - and host them on a single computer that I could mount from several of the systems in my home as needed.

All was going well - until Mother Nature stepped in a couple of weeks ago.

read on for the tail of woe


eMail Article To a Friend View Printable Version

VMWare updates their package to no longer need patching on Fedora

System Administration TidbitsI was setting up one of my servers to be a workstation for my wife, and had to bring in some video drivers. The latest version of the drivers referenced a new kernel, and of course I have not been updating the kernel because on this machine I also run VMWare's free server instance. VMWare requires that their software be linked against the latest kernel development kit so updating the kernel all the time requires re-linking the software, so I use the "exclude=kernel-*" line in /etc/yum.conf to not automatically update the kernel each time I do other updates.

So I updated the kernel - version 2.6.25.4-10.fc8-i686 - and rebooted.

Then I went to re-link the VMWare software and it wouldn't link :(

I use the "vmware-any-any" patches - version 116 being the latest from http://www.miscreant.org/files/rpms/ - but in this case the compile failed and nothing I did would bring it back to working status.

So I put the incantation "vmware-any-any 2.6.25.4-10.fc8-i686" into Google and came up with a single hit - a French language blog entry at forums.fedora-fr.org which Google kindly offered to translate for me

It turns out I would have saved myself a whole mess of trouble if I'd kept up with the updates from VMWare - seems their latest version, 1.0.6, no longer needs the patches. Thank you REMI

I downloaded the Linux server file from www.vmware.com and the install went without a hitch.

What's New

Stories

No new stories

Comments last 2 days


Trackbacks last 2 days

No new trackbacks

Older Stories

Monday 28-Nov


Friday 07-Oct


Tuesday 04-Oct


Thursday 15-Sep


Saturday 10-Sep


Tuesday 30-Aug


Saturday 20-Aug


Thursday 18-Aug


Sunday 14-Aug

?

Ads by Clickochet

G+ Public Posts

There was a problem reading this feed (see error.log for details).
?

G+

?

Facebook Page

RSS Feed

Richard's Digital Rag

Poll

How do you like to find out news about the internet and computers?

  •  Newspaper
  •  Radio
  •  TV
  •  Web Search
  •  Favourite Web Site(s)
  •  Pod Cast
  •  Video Online
  •  Email List(s)
  •  RSS - Syndication
  •  Word of mouth
This poll has 0 more questions.
Results
Other polls | 54 votes | 0 comments