AIX SFTP Best Practices

SFTP is a functional part of SSH which replaces the behavior of FTP in a secure fashion. This is great on AIX for transferring files, batch job uploads and downloads, and much more secure using SSL on the wire and with a variety of authentication options.

Unfortunately when left in the default configuration, the SSH server on AIX allows all users to use SFTP to access any files on the system (subject to filesystem permissions). It's common to see my customers be surprised when an unprivileged application account can SFTP in with WINSCP and browse the entire systems.

Read more…

AIX SSH Best Practices

In recent years insecure and unencrypted protocols have been deprecated because they pose an unacceptable security risk on any network.

For daily usage systems administrators should use SSH to connect to AIX. SSH is encrypted on the wire and supports additional options for using secure keys instead of simple passwords. It completely replaces telnet and ftp, and all of the rsh tools.

IBM ships and supports their own OpenSSH compiled for AIX. I intend to review settings which should be configured in order to be secure.

Read more…

AIX User Security Best Practices

Did you know that the original AIX crypt implementation 1 only allows 8 character passwords? That's really unacceptable by today's standards. AIX now supports several modern hashing algorithms for password storage and default crypt should no longer be used.

I recommend using SHA512, which is the strongest currently supported. With SHA512 password hashing, passwords up to 255 characters long are supported. That means your important root password should now be 32 characters long or more!

Read more…

VIOS v3.1 upgrades failing from 2.2.6.51

Recently one of my customers had difficulty upgrading to PowerVM v3.1 using the alt_disk method. IBM's instructions are to upgrade your v2 VIO to the latest to ensure a smooth transition to v3, and then the alt_disk upgrade method was added in late 2.2.6.30.

Unfortunately in this case, there's a poorly documented bug in the installer. I decided to document it here to help others who may encounter it in the future.

Read more…

Merging NMON files and the new nmonchart

I frequently have to review performance data for customers while doing performance troubleshooting and capacity planning. Kudos to Nigel Griffiths for his excellent NMON tool and associated programs. NMON makes collecting performance data on AIX and Linux a breeze.

Analyzing that data is fairly simple. There are a variety of tools, and I have typically used the Excel based NMON Analyzer 1. I feel that the graphs are quite good and because it is in Excel it's easy for me to annotate and share with customers. It has the option to merge data files, unfortunately due to Excel's constraints it is limited to about five days of data.

Recently I was learning about Nigel's new efforts with JSON and web based graphing, and came across his nmonchart 2 tool. This new tool has dynamically zooming graphs via javascript directly in your browser from a single file! I had to try it, and I'm very impressed.

Read more…

Verifying IBM downloads using XSLTPROC

I often download updates from IBM FixCentral using FTPS/SFTP instead of IBM's Download Director. It's just easier to do on a NIM server rather than my laptop or customer desktop. Unfortunately it makes it difficult to validate the checksums of the downloads.

IBM doesn't typically publish a simple text file of checksums with any of their POWER or AIX downloads. They do include an XML file for Download Director.

Read more…

Simple error reporting

In a production environment there should be no silent failures. AIX has an excellent centralized error reporting facility whose messages are viewed using the errpt command. Compared to other logging sources like syslog, the messages in errpt are of much higher quality and low volume. They are worth reviewing!

Logs always suffer from inattention if they must be checked manually so here's a simple way to email errpt entries to yourself in real time as soon as they happen. This method has two components, forwarding root's email and then using the errdaemon ODM to make errdaemon email any new log entries when they are created.

Read more…

Working with Snap files

I frequently work with customer systems where I need a systems inventory. This could be for troubleshooting or just to save the final state of a system for later reference.

I have worked with many consultants who have an inventory script they give customers but I have found that I prefer to use the tools native to the platform when they are available. On AIX I use IBM's native snap command. If you've ever been on the phone with IBM support before, you know they barely wait to ask your name before they ask for you to upload a snap.

IBM has created an excellent tool for troubleshooting AIX in the snap utility which is distributed as part of the OS. In my experience it captures about 90% of what I need to know about a system, including:

  • Installed software

  • Devices and attributes

  • LVM details and disk layout

  • Network statistics and configuration data

Rather than ask a customer to run commands for me and capture the output, or ask them to run a script from an untrusted source as root on their production server, I always ask for a snap.

Read more…