VIOS v3.1 upgrades failing from 2.2.6.51

Recently one of my customers had difficulty upgrading to PowerVM v3.1 using the alt_disk method. IBM's instructions are to upgrade your v2 VIO to the latest to ensure a smooth transition to v3, and then the alt_disk upgrade method was added in late 2.2.6.30.

Unfortunately in this case, there's a poorly documented bug in the installer. I decided to document it here to help others who may encounter it in the future.

Source versions:
  • PowerVM 2.2.5.20

Target versions:
  • Update to PowerVM 2.2.6.51

  • Upgrade to PowerVM 3.1.0.21

The root cause of the problem is that the bos.alt_disk_install.rte and bos.alt_disk_install.boot_images filesets in 2.2.6.1 are newer than the versions on the installation media for 3.1.0.0 or 3.1.0.10.

Attempting to do a viosupgrade results in a Perl exception instead of an appropriate error message:

VIO2# viosupgrade -l -i /mnt/PowerVM31010.mksysb -a hdisk1
Welcome to viosupgrade tool.
Operation triggered for given node(s).
Broadcast message from root@VIO2 (vty0) at 14:34:26 ...
WARNING!!! VIOS Upgrade operation is in progress. Kindly Refrain from making any configuration changes...
Please wait for completion..
Upgrading from ioslevel '2.2.6.51' to '3.1.0.10'.
Verifying whether the MPIO software(s) is installed on the VIOS.
Following list of fileset(s) required for the VIOS meta data restore
seems to be not present in the provided installation image.
Continuing with the upgrade process may result in restore failure post installation:
1: devices.sddpcm.61.rte => IBM SDD PCM for AIX V61
2: devices.fcp.disk.ibm.mpio.rte => IBM MPIO FCP Disk Device
Choice[Y/N]:
y
Verification of the MPIO software(s) is successful.
Initiating VIOS configuration backup..
VIOS configuration backup successful.
Fileset installation failed: 'bos.alt_disk_install'.
Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2566.
Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2566.
Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2572.
Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2572.

IBM support said to either revert to backup and upgrade to 2.2.6.31, or upgrade to 3.1.1.x. Repeating the upgrade wasn't appealing, and 3.1.1.x is too new and has some HIPER APARs regarding HBA issues.

Once IBM confirmed the issue was that bos.alt_disk_install was too new, I suggested we just reject that new version. I confirmed we had the old version still installed. We had not performed a commit operation after the update to 2.2.6.31.

# lslpp -hac | grep alt_disk_install
/usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.200::COMMIT:COMPLETE:10/11/16:20;51;35
/usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.200::APPLY:COMPLETE:10/11/16:20;51;35
/usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.202::COMMIT:COMPLETE:02/11/20:11;58;24
/usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.202::APPLY:COMPLETE:04/17/17:11;34;13 <<< last committed
/usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.406::APPLY:COMPLETE:02/11/20:12;11;44 <<< applied level
/usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.200::COMMIT:COMPLETE:10/11/16:20;51;28
/usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.200::APPLY:COMPLETE:10/11/16:20;51;28
/usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.201::COMMIT:COMPLETE:02/11/20:11;58;24
/usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.201::APPLY:COMPLETE:04/17/17:11;41;20 <<< last committed
/usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.400::APPLY:COMPLETE:02/11/20:12;16;27 <<< applied level

IBM agreed that we could reject the new version, and proceed.

To fix the issue with the viosupgrade script and too new of bos.alt_disk_install filesets, we simply rejected those filesets using the installp command in oem_setup_env:

  • installp -r bos.alt_disk_install.boot_images

  • installp -r bos.alt_disk_install.rte

    # installp -r bos.alt_disk_install.boot_images
    +-----------------------------------------------------------------------------+
                            Pre-reject Verification...
    +-----------------------------------------------------------------------------+
    Verifying selections...done
    Verifying requisites...done
    Results...
    SUCCESSES
    ---------
      Filesets listed in this section passed pre-reject verification
      and will be rejected.
      Selected Filesets
      -----------------
      bos.alt_disk_install.boot_images 6.1.9.406  # Alternate Disk Installation ...
      << End of Success Section >>
    FILESET STATISTICS
    ------------------
        1  Selected to be rejected, of which:
            1  Passed pre-reject verification
      ----
        1  Total to be rejected
    +-----------------------------------------------------------------------------+
                              Rejecting Software...
    +-----------------------------------------------------------------------------+
    installp:  REJECTING software for:
            bos.alt_disk_install.boot_images 6.1.9.406
    Finished processing all filesets.  (Total time:  3 secs).
    +-----------------------------------------------------------------------------+
                                    Summaries:
    +-----------------------------------------------------------------------------+
    Installation Summary
    --------------------
    Name                        Level           Part        Event       Result
    -------------------------------------------------------------------------------
    bos.alt_disk_install.boot_i 6.1.9.406       USR         REJECT      SUCCESS
    
    # installp -r bos.alt_disk_install.rte
    +-----------------------------------------------------------------------------+
                            Pre-reject Verification...
    +-----------------------------------------------------------------------------+
    Verifying selections...done
    Verifying requisites...done
    Results...
    SUCCESSES
    ---------
      Filesets listed in this section passed pre-reject verification
      and will be rejected.
      Selected Filesets
      -----------------
      bos.alt_disk_install.rte 6.1.9.400          # Alternate Disk Installation ...
      << End of Success Section >>
    FILESET STATISTICS
    ------------------
        1  Selected to be rejected, of which:
            1  Passed pre-reject verification
      ----
        1  Total to be rejected
    +-----------------------------------------------------------------------------+
                              Rejecting Software...
    +-----------------------------------------------------------------------------+
    installp:  REJECTING software for:
            bos.alt_disk_install.rte 6.1.9.400
    Successfully updated the Kernel Authorization Table.
    Successfully updated the Kernel Role Table.
    Successfully updated the Kernel Command Table.
    Successfully updated the Kernel Device Table.
    Successfully updated the Kernel Object Domain Table.
    Successfully updated the Kernel Domains Table.
    Successfully updated the Kernel Authorization Table.
    Successfully updated the Kernel Role Table.
    Successfully updated the Kernel Command Table.
    Successfully updated the Kernel Device Table.
    Successfully updated the Kernel Object Domain Table.
    Successfully updated the Kernel Domains Table.
    Successfully updated the Kernel Authorization Table.
    Successfully updated the Kernel Role Table.
    Successfully updated the Kernel Command Table.
    Successfully updated the Kernel Device Table.
    Successfully updated the Kernel Object Domain Table.
    Successfully updated the Kernel Domains Table.
    Successfully updated the Kernel Authorization Table.
    Successfully updated the Kernel Role Table.
    Successfully updated the Kernel Command Table.
    Successfully updated the Kernel Device Table.
    Successfully updated the Kernel Object Domain Table.
    Successfully updated the Kernel Domains Table.
    Finished processing all filesets.  (Total time:  6 secs).
    +-----------------------------------------------------------------------------+
                                    Summaries:
    +-----------------------------------------------------------------------------+
    Installation Summary
    --------------------
    Name                        Level           Part        Event       Result
    -------------------------------------------------------------------------------
    bos.alt_disk_install.rte    6.1.9.400       ROOT        REJECT      SUCCESS
    bos.alt_disk_install.rte    6.1.9.400       USR         REJECT      SUCCESS

Once having reverted to the older version, viosupgrade ran normally.

$ viosupgrade -l -i /mnt/PowerVM31010.mksysb -a hdisk1
Welcome to viosupgrade tool.
Operation triggered for given node(s).
Broadcast message from root@VIO2 (vty0) at 15:13:07 ...
WARNING!!! VIOS Upgrade operation is in progress. Kindly Refrain from making any configuration changes...
Please wait for completion..
Upgrading from ioslevel '2.2.6.51' to '3.1.0.10'.
Verifying whether the MPIO software(s) is installed on the VIOS.
Following list of fileset(s) required for the VIOS meta data restore
seems to be not present in the provided installation image.
Continuing with the upgrade process may result in restore failure post installation:
1: devices.fcp.disk.ibm.mpio.rte => IBM MPIO FCP Disk Device
2: devices.sddpcm.61.rte => IBM SDD PCM for AIX V61
Choice[Y/N]:
y
Verification of the MPIO software(s) is successful.
Initiating VIOS configuration backup..
VIOS configuration backup successful.
Initiating installation on alternate disk(s)..
Installation on alternate disk(s) successful.
Copying files to altinst_rootvg.
Waking up altinst_rootvg successful.
VIOS will be rebooted after '60' seconds to boot from the newly installed disk.
Press contrl+c to terminate.
VIOS metadata restore (viosbr -restore) will be automatically resumed
after the reboot.
VIOS may be rebooted once during this restore process. Refrain from making
any changes to the VIOS virtual configurations during the restore process.
You can verify the restore status using 'viosupgrade -l -q' command and
resume your operation after the completion of the restore process.
portmir: Cannot unload mirror module
portmir: Mirroring is stopped.
Rebooting . . .