I frequently have to review performance data for customers while doing performance troubleshooting and capacity planning. Kudos to Nigel Griffiths for his excellent NMON tool and associated programs. NMON makes collecting performance data on AIX and Linux a breeze.

Analyzing that data is fairly simple. There are a variety of tools, and I have typically used the Excel based NMON Analyzer [1]. I feel that the graphs are quite good and because it is in Excel it's easy for me to annotate and share with customers. It has the option to merge data files, unfortunately due to Excel's constraints it is limited to about five days of data.

Recently I was learning about Nigel's new efforts with JSON and web based graphing, and came across his nmonchart [2] tool. This new tool has dynamically zooming graphs via javascript directly in your browser from a single file! I had to try it, and I'm very impressed.

Read more…

I often download updates from IBM FixCentral using FTPS/SFTP instead of IBM's Download Director. It's just easier to do on a NIM server rather than my laptop or customer desktop. Unfortunately it makes it difficult to validate the checksums of the downloads.

IBM doesn't typically publish a simple text file of checksums with any of their POWER or AIX downloads. They do include an XML file for Download Director.

Read more…

In a production environment there should be no silent failures. AIX has an excellent centralized error reporting facility whose messages are viewed using the errpt command. Compared to other logging sources like syslog, the messages in errpt are of much higher quality and low volume. They are worth reviewing!

Logs always suffer from inattention if they must be checked manually so here's a simple way to email errpt entries to yourself in real time as soon as they happen. This method has two components, forwarding root's email and then using the errdaemon ODM to make errdaemon email any new log entries when they are created.

Read more…

I frequently work with customer systems where I need a systems inventory. This could be for troubleshooting or just to save the final state of a system for later reference.

I have worked with many consultants who have an inventory script they give customers but I have found that I prefer to use the tools native to the platform when they are available. On AIX I use IBM's native snap command. If you've ever been on the phone with IBM support before, you know they barely wait to ask your name before they ask for you to upload a snap.

IBM has created an excellent tool for troubleshooting AIX in the snap utility which is distributed as part of the OS. In my experience it captures about 90% of what I need to know about a system, including:

• Installed software

• Devices and attributes

• LVM details and disk layout

• Network statistics and configuration data

Rather than ask a customer to run commands for me and capture the output, or ask them to run a script from an untrusted source as root on their production server, I always ask for a snap.

Read more…