Working with Snap files

I frequently work with customer systems where I need a systems inventory. This could be for troubleshooting or just to save the final state of a system for later reference.

I have worked with many consultants who have an inventory script they give customers but I have found that I prefer to use the tools native to the platform when they are available. On AIX I use IBM's native snap command. If you've ever been on the phone with IBM support before, you know they barely wait to ask your name before they ask for you to upload a snap.

IBM has created an excellent tool for troubleshooting AIX in the snap utility which is distributed as part of the OS. In my experience it captures about 90% of what I need to know about a system, including:

  • Installed software

  • Devices and attributes

  • LVM details and disk layout

  • Network statistics and configuration data

Rather than ask a customer to run commands for me and capture the output, or ask them to run a script from an untrusted source as root on their production server, I always ask for a snap.

The snap tool has a large number of arguments, see:

https://www.ibm.com/support/knowledgecenter/api/content/nl/en-us/ssw_aix_72/com.ibm.aix.cmds5/snap.htm

I prefer to specify explicit flags for the data to be collected instead of using snap -ac default. Be aware that the "all" data collection (-a) includes the user password database files with usernames and password hashes, which I don't want and should not be sent over an untrusted network. As a result I commonly use:

snap -cfgGiknLt

Using these options quoted below from the IBM link above:

-c Creates a compressed pax image (snap.pax.Z file) of all files in
 the /tmp/ibmsupt directory tree or other named output
 directory. Note: Information not gathered with this option should
 be copied to the snap directory tree before using the -c flag. If a
 test case is needed to demonstrate the system problem, copy the
 test case to the /tmp/ibmsupt/testcase directory before compressing
 the pax file.

-g Gathers the output of the lslpp -hac command, which is required
 to recreate exact operating system environments. Writes output to
 the /tmp/ibmsupt/general/lslpp.hac file. Also collects general
 system information and writes the output to the
 /tmp/ibmsupt/general/general.snap file.

-G Includes predefined Object Data Manager (ODM) files in general
 information collected with the -g flag.

-f Gathers file system information.
-i Gathers installation debug vital product data (VPD) information.
-k Gathers kernel information
-L Gathers LVM information.
-n Gathers Network File System (NFS) information.
-t Gathers Transmission control protocol information.

This gives a good overview of the system without including a system dump. Most of the time the snap files range from 5 MB to 25 MB.

Always run snap -r to clear the snap cache in /tmp/ibmsupt before taking a new snap. This is generally safe as the only files it will remove are files snap knows that it wrote.

The snap file is always stored in /tmp/ibmsupt/snap.pax.Z. This is a compressed PAX file which the pax utility or GNU tar can read.

It's easy to rename them before downloading via:

mv /tmp/ibmsupt/snap.pax.Z /tmp/ibmsupt/$(hostname)_$(date +%Y%m%d).snap.pax.Z

The format of the snap file is pretty simple. In the base directory there are several directories, each contains copied files from the host and files ending with .snap which contain commands and output. For instance:

client_collect

This is where data about the virtual client devices is saved, including kernel debug dumps.

filesys

Common commands like 'df' are stored here, all focused at the filesystem layer.

kernel

Copies of kernel parameters, 'vmo', etc.

lvm

Summary .snap files for every volume group, all the way down to the LP mappings.

tcpip

Common configuration files from etc (/etc/hosts, /etc/resolv.conf), and networking commands in tcpip.snap.

general

This includes many items about the system, including ODM data exported to text, console logs, errpt output, environment variables, inittab, LPAR stats, lsdev of several device types, packet filter rules, SSH config, and general.snap. That snap file has many commands including lsdev, bootinfo, lsps, lslpp, lscfg, and more.

I have two scripts I use to enable me to work with snap files. The first untars the snap into a more regular format, renaming for the host, OS, and date taken. It also creates a bzipped tar for archival. The second allows me to query the .snap files in each snap for many host snaps stored in the same directory.

I use these scripts on Ubuntu Linux with common Linux utilities. I'm sure they would run on AIX with GNU Awk installed and some modifications.

The uncompress and normalize script:

NormalizeSnap.sh (Source)

#!/bin/sh

######################################################################
# ExtractCmd.sh
#
#  Copyright (c) 2016, Russell Adams <Russell.Adams@AdamsSystems.nl>
#  All rights reserved.
#
#  Redistribution and use in source and binary forms, with or without
#  modification, are permitted provided that the following conditions are
#  met:
#
#   - Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
#
#   - Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the
#     distribution.
#
#   - Neither the name of Russell Adams, Adams Systems Consultancy,
#     nor the names of its contributors may be used to endorse or promote
#     products derived from this software without specific prior written
#     permission.
#
#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
#  HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# This script uncompresses and unarchives an IBM AIX snap file stored
# in the PAX format. It create a new directory named by host, oslevel
# and date, performs recursive PAX unpacking (ie: HA snaps), and makes
# a final bzipped archive. The original .Z file can be removed
# afterward.

openpax () {

    # openpax DestinationDirectory Filename
    OLDDIR=$PWD
    cd "$1"

    pax $TAR_FLAGS -Orvf "${OLDDIR}/${2}" 2>&1 | tail -n1

    cd "$OLDDIR"
}

fixperms () {

        # there's a frequent hiccup in hacmp snaps where the subsnap has no execute privs
        echo "Fixing perms."
        find $1 -type d -exec chmod 755 '{}' \;
        find $1 -type f -exec chmod 755 '{}' \;
        find $1 -type d -exec chmod 755 '{}' \;
        find $1 -type f -exec chmod 755 '{}' \;

}


for TARGET in "$@" ; do {

    [ -f "${TARGET}" ] || {
            echo "Snap file does not exist"
            exit 1
    }

    echo "${TARGET}" | egrep '\.(tar|pax)\.(Z|gz)$'   > /dev/null && TAR_FLAGS="-z"
    echo "${TARGET}" | egrep '\.(tar|pax)\.bz2$'      > /dev/null && TAR_FLAGS="-j"

    [ -z "$TAR_FLAGS" ] && {
            echo "Unrecognized snap suffix, supports .{tar,pax}.{Z,gz,bz2}"
            exit 1
    }

    TMPDIR=$(mktemp -d "./snapXXXXXXXX")

    # Untar into the scratch dir
    echo "========================================"
        echo -n "Untarring, # of files: "
        echo "${TARGET}" | grep '\.tar\.' && tar --no-same-permissions -C $TMPDIR $TAR_FLAGS -xvf "${TARGET}" | wc -l
    echo "${TARGET}" | grep '\.pax\.' && openpax "$TMPDIR" "${TARGET}"

        # Check is this a snap?
        echo "Checking general exists."
    find $TMPDIR -name general -type d | grep general || {
                echo "Not a snap file, aborting."
                exit 1
        }

    # Fix perms
        fixperms $TMPDIR

    # Find snap/general/... and move them into the base of the scratch dir
        echo "Moving subdirs."
        [ -d ${TMPDIR}/general ] || mv $(dirname $(find $TMPDIR -name general -type d | tail -n1))/* $TMPDIR

    # Check for sub-snaps
    echo "Opening subsnaps."
    find $TMPDIR -type f -name '*.pax.Z' | while read Z; do
        SUBTMP="$(echo ${Z} | sed 's/\.pax\.Z$//g')"
        mkdir "${SUBTMP}"
        echo "Opening subsnap $Z"
        openpax "${SUBTMP}" "${Z}"
        rm "${Z}"
    done
    find $TMPDIR -type f -name '*.pax' | while read Z; do
        SUBTMP="$(echo ${Z} | sed 's/\.pax$//g')"
        mkdir "${SUBTMP}"
        echo "Opening subsnap $Z"
        TAR_FLAGS="" openpax "${SUBTMP}" "${Z}"
        rm "${Z}"
    done
    find $TMPDIR -type f -name '*.tar.Z' | while read Z; do
        SUBTMP="$(echo ${Z} | sed 's/\.tar\.Z$//g')"
        mkdir "${SUBTMP}"
        echo "Opening subsnap $Z"
        tar --no-same-permissions -C $SUBTMP -zxvf "${TARGET}" | wc -l
        rm "${Z}"
    done

        # Fix perms
        fixperms $TMPDIR

    # Clean empty dirs
        echo "Cleaning empty dirs."
        find $TMPDIR -type d -empty -print0 | xargs -0t rmdir

    # Attributes
        echo "Collecting data."
        SYSTEM=$(awk   'BEGIN {RS="\n\n.....\n"} /lsattr -El sys0/  {print $0}' < ${TMPDIR}/general/general.snap | awk '/^modelname +/ {print $2}' | cut -f2 -d,)
        SERIAL=$(awk   'BEGIN {RS="\n\n.....\n"} /lsattr -El sys0/  {print $0}' < ${TMPDIR}/general/general.snap | awk '/^systemid +/  {print $2}' | cut -f2 -d,)

        OSLEVEL=$(tail -n1 $TMPDIR/general/oslevel.info \
                                  || grep :bos.rte: ${TMPDIR}/general/lslpp.hac | sort | cut -f3 -d: | tail -n1)

        SNAPHOST=$(awk 'BEGIN {RS="\n\n.....\n"} /lsattr -El inet0/ {print $0}' < ${TMPDIR}/general/general.snap | awk '/^hostname +/  {print $2}' | tr -d _)
        SNAPDATE=$(date +%Y%m%d_%H%M%S -d "$(head -n1 $TMPDIR/general/general.snap)")

        NEWNAME="./${SYSTEM}_${SERIAL}_${SNAPHOST}_${OSLEVEL}_${SNAPDATE}.snap"

    # Cleanup
        echo "Cleaning dump and security."
        find $TMPDIR -name 'unix.Z'  -print0 | xargs -0 rm -f
        find $TMPDIR -name 'dump.Z'  -print0 | xargs -0 rm -f
        find $TMPDIR -name 'dump.BZ' -print0 | xargs -0 rm -f
        find $TMPDIR -name 'passwd'  -print0 | xargs -0 rm -f

    # Rename logically
    echo "Renaming to final destination"
        mv $TMPDIR $NEWNAME

    # Recompress
    echo "Retarring files into: ${NEWNAME}.tar.bzip2 "
    echo -n "Number of files compressed: "
        tar -jcvf "${NEWNAME}.tar.bzip2" "${NEWNAME}" | wc -l

    # Echo completion
    echo "Successfully extracted snap to: $NEWNAME"

} 2>&1 | tee -a ./NormalizeSnap.log

done

The command extraction script:

ExtractCmd.sh (Source)

#!/bin/sh -e

######################################################################
# ExtractCmd.sh
#
#  Copyright (c) 2016, Russell Adams <Russell.Adams@AdamsSystems.nl>
#  All rights reserved.
#
#  Redistribution and use in source and binary forms, with or without
#  modification, are permitted provided that the following conditions are
#  met:
#
#   - Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
#
#   - Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the
#     distribution.
#
#   - Neither the name of Russell Adams, Adams Systems Consultancy,
#     nor the names of its contributors may be used to endorse or promote
#     products derived from this software without specific prior written
#     permission.
#
#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
#  HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# ***** Requires GAWK to be installed. See below for details. *****
#
# This script uses awk to pull a command's output from an already
# untarred IBM AIX snap file. The output is in a format suitable for
# piping to grep for further analysis.
#
# For example:
#
#  Place a series of untarred snap files in your working directory,
#  with each snap untarred into an individually named subdirectory,
#  like:
#
#      ./host1.snap/
#      ./host2.snap/
#
#  Then to lookup a common command like 'lsattr -El sys0' use:
#
#      $ $PATH/ExtractCmd.sh 'lsattr -El sys0'
#
#  This returns:
#
#      host1.snap:     .....
#      host1.snap:     .....    bootinfo -K
#      host1.snap:     .....
#      host1.snap:
#      host1.snap:     32
#      host1.snap:
#      host1.snap:     .....
#      host2.snap:     .....    bootinfo -K
#      host2.snap:     .....
#      host2.snap:
#      host2.snap:     64
#      host2.snap:
#      host1.snap:     .....
#      host3.snap:     .....    bootinfo -K
#      host3.snap:     .....
#      host3.snap:
#      host3.snap:     64
#      host3.snap:
#
# Regular expressions are used to locate the commands, so you can use
# regexp to specify one command or a series of commands, like:
#
#  Return all attributes on EVERY hdisk on the system, including
#  hdiskpower devices:
#
#      lsattr -El hdisk
#
#  Return only ent1 attributes, noting that ent1 could match ent10:
#
#      lsattr -El ent1\$
#

# require a command
[ -z "$1" ] && {
    echo "Specify a quoted command as the first parameter"
    exit -1
}

CMD="$1"

# Each hostname.snap directory has subdirectories underneath with a .snap file in it
# We search those only for command output
for SNAP in *.snap ; do
    gawk "BEGIN {RS=\"\n\n.....\n\"} /${CMD}/ { print \"\n.....\n\" \$0 }" ${SNAP}/*/*.snap \
        | gawk "{printf \"%-20s %s\n\", \"${SNAP}:\", \$0 }"
done

Below is an example of extracting a snap file and running some basic commands:

% ls -l snap.pax.Z
-rw-rw-r-- 1 adamsrl adamsrl 6748366 May 26 17:20 snap.pax.Z

% ~/scripts/NormalizeSnap.sh snap.pax.Z
========================================
Untarring, # of files: snap.pax.Z
pax: ustar vol 1, 199 files, 5775360 bytes read, 0 bytes written.
Checking general exists.
./snapRtvZKUtV/general
Moving subdirs.
Opening subsnaps.
Fixing perms.
Cleaning empty dirs.
rmdir ./snapRtvZKUtV/testcase ./snapRtvZKUtV/scraid ./snapRtvZKUtV/other ./snapRtvZKUtV/hacmp
Collecting data.
Cleaning dump and security.
Renaming to final destination
Retarring files into: ./7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap.tar.bz2
Number of files compressed: 194
Successfully extracted snap to: ./7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap

% ls -ld 7044*
drwxr-xr-x 19 adamsrl adamsrl    4096 May 26 17:21 7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap
-rw-rw-r--  1 adamsrl adamsrl 2492457 May 26 17:21 7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap.tar.bz2

% cd 7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap
% ls -l
total 72
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 async
drwxr-xr-x 3 adamsrl adamsrl 4096 May 19  2008 dump
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 filesys
drwxr-xr-x 2 adamsrl adamsrl 4096 May 26 17:21 general
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 getRtasHeap
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 install
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 kernel
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 lang
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 lvm
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 nfs
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 pcixscsi
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 printer
-rw-r--r-- 1 adamsrl adamsrl  690 May 19  2008 script.log
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 sna
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 ssa
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 tcpip
drwxr-xr-x 5 adamsrl adamsrl 4096 May 19  2008 wlm
drwxr-xr-x 2 adamsrl adamsrl 4096 May 19  2008 XS25

% ~/scripts/ExtractCmd.sh 'lsattr -El sys0'
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap:
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: .....
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: .....    lsattr -El sys0
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: .....
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap:
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: SW_dist_intr    false              Enable SW distribution of interrupts              True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: autorestart     true               Automatically REBOOT OS after a crash             True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: boottype        disk               N/A                                               False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: capacity_inc    1.00               Processor capacity increment                      False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: capped          true               Partition is capped                               False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: conslogin       enable             System Console Login                              False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: cpuguard        disable            CPU Guard                                         True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: dedicated       true               Partition is dedicated                            False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: ent_capacity    1.00               Entitled processor capacity                       False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: frequency       90000000           System Bus Frequency                              False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: fullcore        false              Enable full CORE dump                             True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: fwversion       IBM,SPH05195       Firmware version and revision levels              False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: id_to_partition 0X03708091236A2C00 Partition ID                                      False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: id_to_system    0X03708091236A2C00 System ID                                         False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: iostat          false              Continuously maintain DISK I/O history            True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: keylock         normal             State of system keylock at boot time              False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: max_capacity    1.00               Maximum potential processor capacity              False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: max_logname     9                  Maximum login name length at boot time            True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: maxbuf          20                 Maximum number of pages in block I/O BUFFER CACHE True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: maxmbuf         0                  Maximum Kbytes of real memory allowed for MBUFS   True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: maxpout         0                  HIGH water mark for pending write I/Os per file   True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: maxuproc        128                Maximum number of PROCESSES allowed per user      True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: min_capacity    1.00               Minimum potential processor capacity              False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: minpout         0                  LOW water mark for pending write I/Os per file    True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: modelname       IBM,7044-170       Machine name                                      False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: ncargs          6                  ARG/ENV list size in 4K byte blocks               True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: nfs4_acl_compat secure             NFS4 ACL Compatibility Mode                       True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: pre430core      false              Use pre-430 style CORE dump                       True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: pre520tune      disable            Pre-520 tuning compatibility mode                 True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: realmem         1048576            Amount of usable physical memory in Kbytes        False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: rtasversion     1                  Open Firmware RTAS version                        False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: sed_config      select             Stack Execution Disable (SED) Mode                True
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: systemid        IBM,0110BDC8C      Hardware system identifier                        False
7044-170_0110BDC8C_GILSAIX_5300-06-00-0000_20071205_203526.snap: variable_weight 0                  Variable processor capacity weight                False

Now I can also use standard UNIX text utilties to run aggregate reports on the data from the snaps. Here's a good example, showing all of the queue depth values in use on all snaps in the current working directory.

% ~/scripts/ExtractCmd.sh 'lsattr -El hdisk' | grep queue_depth | cut -f2 -d: | sort -u
queue_depth   1                                Queue DEPTH                      True

Imagine checking 15 hosts for no_reserve on hdisks, or iostat set to true on sys0. This method of working with snaps can be very powerful even in an offline manner.

On a final note, here's a highlight of most of the commands included in the snap.

./dump/dump.snap:.....    creation date
./dump/dump.snap:.....    sysdumpdev -L -v
./dump/dump.snap:.....    uname -a
./filesys/filesys.snap:.....    df -k
./filesys/filesys.snap:.....    lsattr -El hdisk0
./filesys/filesys.snap:.....    lsfs -l
./filesys/filesys.snap:.....    lslv -l fslv00
./filesys/filesys.snap:.....    lslv -l hd1
./filesys/filesys.snap:.....    lslv -l hd10opt
./filesys/filesys.snap:.....    lslv -l hd2
./filesys/filesys.snap:.....    lslv -l hd3
./filesys/filesys.snap:.....    lslv -l hd4
./filesys/filesys.snap:.....    lslv -l hd5
./filesys/filesys.snap:.....    lslv -l hd6
./filesys/filesys.snap:.....    lslv -l hd8
./filesys/filesys.snap:.....    lslv -l hd9var
./filesys/filesys.snap:.....    lspv -l hdisk0
./filesys/filesys.snap:.....    lsvg -o
./filesys/filesys.snap:.....    lsvg -o| xargs lsvg -l
./filesys/filesys.snap:.....    lsvg -p rootvg
./filesys/filesys.snap:.....    mount
./general/general.snap:.....    bootinfo -K
./general/general.snap:.....    bootinfo -r
./general/general.snap:.....    env
./general/general.snap:.....    lsattr -El aio0
./general/general.snap:.....    lsattr -El cd0
./general/general.snap:.....    lsattr -El en0
./general/general.snap:.....    lsattr -El ent0
./general/general.snap:.....    lsattr -El et0
./general/general.snap:.....    lsattr -El fd0
./general/general.snap:.....    lsattr -El fda0
./general/general.snap:.....    lsattr -El fslv00
./general/general.snap:.....    lsattr -El gxme0
./general/general.snap:.....    lsattr -El hd1
./general/general.snap:.....    lsattr -El hd10opt
./general/general.snap:.....    lsattr -El hd2
./general/general.snap:.....    lsattr -El hd3
./general/general.snap:.....    lsattr -El hd4
./general/general.snap:.....    lsattr -El hd5
./general/general.snap:.....    lsattr -El hd6
./general/general.snap:.....    lsattr -El hd8
./general/general.snap:.....    lsattr -El hd9var
./general/general.snap:.....    lsattr -El hdisk0
./general/general.snap:.....    lsattr -El inet0
./general/general.snap:.....    lsattr -El isa0
./general/general.snap:.....    lsattr -El kbd0
./general/general.snap:.....    lsattr -El L2cache0
./general/general.snap:.....    lsattr -El lft0
./general/general.snap:.....    lsattr -El lo0
./general/general.snap:.....    lsattr -El lvdd
./general/general.snap:.....    lsattr -El mem0
./general/general.snap:.....    lsattr -El moj0
./general/general.snap:.....    lsattr -El mouse0
./general/general.snap:.....    lsattr -El paud0
./general/general.snap:.....    lsattr -El pci0
./general/general.snap:.....    lsattr -El pci1
./general/general.snap:.....    lsattr -El posix_aio0
./general/general.snap:.....    lsattr -El ppa0
./general/general.snap:.....    lsattr -El proc0
./general/general.snap:.....    lsattr -El pty0
./general/general.snap:.....    lsattr -El rcm0
./general/general.snap:.....    lsattr -El rootvg
./general/general.snap:.....    lsattr -El sa0
./general/general.snap:.....    lsattr -El sa1
./general/general.snap:.....    lsattr -El scsi0
./general/general.snap:.....    lsattr -El scsi1
./general/general.snap:.....    lsattr -El sioka0
./general/general.snap:.....    lsattr -El siokma0
./general/general.snap:.....    lsattr -El sioma0
./general/general.snap:.....    lsattr -El siota0
./general/general.snap:.....    lsattr -El sys0
./general/general.snap:.....    lsattr -El sysplanar0
./general/general.snap:.....    lscfg -pv
./general/general.snap:.....    lsdev -Ccprocessor
./general/general.snap:.....    lslpp -La
./general/general.snap:.....    lslpp -lc
./general/general.snap:.....    lsmcode -A
./general/general.snap:.....    lsps -a
./general/general.snap:.....    lsresource -al pci0
./general/general.snap:.....    lsresource -al pci1
./general/general.snap:.....    oslevel -r
./general/general.snap:.....    oslevel -s
./general/general.snap:.....    rpm -qa
./general/general.snap:.....    sysdumpdev -L -v
./general/general.snap:.....    trcnm
./kernel/kernel.snap:.....    bootinfo -K
./kernel/kernel.snap:.....    bootinfo -r
./kernel/kernel.snap:.....    env
./kernel/kernel.snap:.....    lssrc -a
./kernel/kernel.snap:.....    ps -ef
./kernel/kernel.snap:.....    ps -leaf
./kernel/kernel.snap:.....    vmstat
./kernel/kernel.snap:.....    vmstat -s
./kernel/kernel.snap:.....    vmstat -v
./lvm/lvm.snap:.....
./lvm/lvm.snap:.....    cat /tmp/gsclvmd.log
./lvm/lvm.snap:.....    getlvcb -AT fslv00
./lvm/lvm.snap:.....    getlvcb -AT hd1
./lvm/lvm.snap:.....    getlvcb -AT hd10opt
./lvm/lvm.snap:.....    getlvcb -AT hd2
./lvm/lvm.snap:.....    getlvcb -AT hd3
./lvm/lvm.snap:.....    getlvcb -AT hd4
./lvm/lvm.snap:.....    getlvcb -AT hd5
./lvm/lvm.snap:.....    getlvcb -AT hd6
./lvm/lvm.snap:.....    getlvcb -AT hd8
./lvm/lvm.snap:.....    getlvcb -AT hd9var
./lvm/lvm.snap:.....    lquerypv -h /dev/hdisk0
./lvm/lvm.snap:.....    lqueryvg -AtP -p hdisk0
./lvm/lvm.snap:.....    lqueryvg -g 000bdc8c00004c00000001143b80c737 -At
./lvm/lvm.snap:.....    lscfg -pv
./lvm/lvm.snap:.....    lsdev -C
./lvm/lvm.snap:.....    ls -l /dev
./lvm/lvm.snap:.....    ls -l /tmp/ch.log.*
./lvm/lvm.snap:.....    lslv fslv00
./lvm/lvm.snap:.....    lslv hd1
./lvm/lvm.snap:.....    lslv hd10opt
./lvm/lvm.snap:.....    lslv hd2
./lvm/lvm.snap:.....    lslv hd3
./lvm/lvm.snap:.....    lslv hd4
./lvm/lvm.snap:.....    lslv hd5
./lvm/lvm.snap:.....    lslv hd6
./lvm/lvm.snap:.....    lslv hd8
./lvm/lvm.snap:.....    lslv hd9var
./lvm/lvm.snap:.....    lslv -m fslv00
./lvm/lvm.snap:.....    lslv -m hd1
./lvm/lvm.snap:.....    lslv -m hd10opt
./lvm/lvm.snap:.....    lslv -m hd2
./lvm/lvm.snap:.....    lslv -m hd3
./lvm/lvm.snap:.....    lslv -m hd4
./lvm/lvm.snap:.....    lslv -m hd5
./lvm/lvm.snap:.....    lslv -m hd6
./lvm/lvm.snap:.....    lslv -m hd8
./lvm/lvm.snap:.....    lslv -m hd9var
./lvm/lvm.snap:.....    lspv
./lvm/lvm.snap:.....    lspv -p hdisk0
./lvm/lvm.snap:.....    lssrc -ls gsclvmd
./lvm/lvm.snap:.....    lsvg
./lvm/lvm.snap:.....    lsvg -l rootvg
./lvm/lvm.snap:.....    lsvg -o
./lvm/lvm.snap:.....    ps -aef | grep gsclvmd
./lvm/lvm.snap:.....    readvgda /dev/hdisk0
./nfs/nfs.snap:.....    exportfs
./nfs/nfs.snap:.....    lsnfsexp
./nfs/nfs.snap:.....    lsnfsmnt
./nfs/nfs.snap:.....    lssrc -a
./nfs/nfs.snap:.....    nfso -a
./nfs/nfs.snap:.....    nfsstat -c
./nfs/nfs.snap:.....    nfsstat -m
./nfs/nfs.snap:.....    nfsstat -n
./nfs/nfs.snap:.....    nfsstat -r
./nfs/nfs.snap:.....    nfsstat -s
./nfs/nfs.snap:.....    Note: If you want the /var/adm/ras/trcfile, snap -g will get it
./nfs/nfs.snap:.....    rpcinfo -p
./printer/printer.snap:.....    enq -AL
./printer/printer.snap:.....    lsdev -Ccprinter
./tcpip/tcpip.snap:.....    arp -a
./tcpip/tcpip.snap:.....    arp -t atm -a
./tcpip/tcpip.snap:.....    lssrc -a
./tcpip/tcpip.snap:.....    netstat -an
./tcpip/tcpip.snap:.....    netstat -in
./tcpip/tcpip.snap:.....    netstat -m
./tcpip/tcpip.snap:.....    netstat -nr
./tcpip/tcpip.snap:.....    netstat -s
./tcpip/tcpip.snap:.....    netstat -sr
./tcpip/tcpip.snap:.....    netstat -v
./tcpip/tcpip.snap:.....    no -a
./tcpip/tcpip.snap:.....    uname -xM

There are individual files that have to be examined separately, but are still useful:

./general/alog.boot
./general/alog.console
./general/devtree.out
./general/emgr.snap
./general/environ
./general/errlog
./general/errpt.out
./general/inittab
./general/instfix.i
./general/lsdev.adapter
./general/lsdev.disk
./general/lsdev.scsi
./general/lslpp.hac
./general/lsvpd.out
./general/oslevel.info
./general/services
./nfs/netsvc.conf.file
./printer/qconfig
./tcpip/bootptab
./tcpip/hosts
./tcpip/hosts.equiv
./tcpip/inetd.conf
./tcpip/mib.defs
./tcpip/mrouted.conf
./tcpip/protocols
./tcpip/rc.bsdnet
./tcpip/rc.net
./tcpip/rc.net.out
./tcpip/rc.net.serial
./tcpip/rc.tcpip
./tcpip/resolv.conf
./tcpip/sendmail.cf
./tcpip/services
./tcpip/snmpd.conf
./tcpip/syslog.conf
./tcpip/telnet.conf