VIOS v3.1 upgrades failing from 2.2.6.51
Recently one of my customers had difficulty upgrading to PowerVM v3.1 using the alt_disk method. IBM's instructions are to upgrade your v2 VIO to the latest to ensure a smooth transition to v3, and then the alt_disk upgrade method was added in late 2.2.6.30.
Unfortunately in this case, there's a poorly documented bug in the installer. I decided to document it here to help others who may encounter it in the future.
- Source versions:
PowerVM 2.2.5.20
- Target versions:
Update to PowerVM 2.2.6.51
Upgrade to PowerVM 3.1.0.21
The root cause of the problem is that the bos.alt_disk_install.rte and
bos.alt_disk_install.boot_images filesets in 2.2.6.1 are newer than the
versions on the installation media for 3.1.0.0 or 3.1.0.10.
Attempting to do a viosupgrade results in a Perl exception instead of an appropriate error message:
VIO2# viosupgrade -l -i /mnt/PowerVM31010.mksysb -a hdisk1 Welcome to viosupgrade tool. Operation triggered for given node(s). Broadcast message from root@VIO2 (vty0) at 14:34:26 ... WARNING!!! VIOS Upgrade operation is in progress. Kindly Refrain from making any configuration changes... Please wait for completion.. Upgrading from ioslevel '2.2.6.51' to '3.1.0.10'. Verifying whether the MPIO software(s) is installed on the VIOS. Following list of fileset(s) required for the VIOS meta data restore seems to be not present in the provided installation image. Continuing with the upgrade process may result in restore failure post installation: 1: devices.sddpcm.61.rte => IBM SDD PCM for AIX V61 2: devices.fcp.disk.ibm.mpio.rte => IBM MPIO FCP Disk Device Choice[Y/N]: y Verification of the MPIO software(s) is successful. Initiating VIOS configuration backup.. VIOS configuration backup successful. Fileset installation failed: 'bos.alt_disk_install'. Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2566. Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2566. Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2572. Use of uninitialized value in numeric eq (==) at /usr/ios/sbin/viosupg.pl line 2572.
IBM support said to either revert to backup and upgrade to 2.2.6.31, or upgrade to 3.1.1.x. Repeating the upgrade wasn't appealing, and 3.1.1.x is too new and has some HIPER APARs regarding HBA issues.
Once IBM confirmed the issue was that bos.alt_disk_install was too new, I
suggested we just reject that new version. I confirmed we had the old version
still installed. We had not performed a commit operation after the update to
2.2.6.31.
# lslpp -hac | grep alt_disk_install /usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.200::COMMIT:COMPLETE:10/11/16:20;51;35 /usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.200::APPLY:COMPLETE:10/11/16:20;51;35 /usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.202::COMMIT:COMPLETE:02/11/20:11;58;24 /usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.202::APPLY:COMPLETE:04/17/17:11;34;13 <<< last committed /usr/lib/objrepos:bos.alt_disk_install.boot_images:6.1.9.406::APPLY:COMPLETE:02/11/20:12;11;44 <<< applied level /usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.200::COMMIT:COMPLETE:10/11/16:20;51;28 /usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.200::APPLY:COMPLETE:10/11/16:20;51;28 /usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.201::COMMIT:COMPLETE:02/11/20:11;58;24 /usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.201::APPLY:COMPLETE:04/17/17:11;41;20 <<< last committed /usr/lib/objrepos:bos.alt_disk_install.rte:6.1.9.400::APPLY:COMPLETE:02/11/20:12;16;27 <<< applied level
IBM agreed that we could reject the new version, and proceed.
To fix the issue with the viosupgrade script and too new of
bos.alt_disk_install filesets, we simply rejected those filesets using the
installp command in oem_setup_env:
installp -r bos.alt_disk_install.boot_images
installp -r bos.alt_disk_install.rte# installp -r bos.alt_disk_install.boot_images +-----------------------------------------------------------------------------+ Pre-reject Verification... +-----------------------------------------------------------------------------+ Verifying selections...done Verifying requisites...done Results... SUCCESSES --------- Filesets listed in this section passed pre-reject verification and will be rejected. Selected Filesets ----------------- bos.alt_disk_install.boot_images 6.1.9.406 # Alternate Disk Installation ... << End of Success Section >> FILESET STATISTICS ------------------ 1 Selected to be rejected, of which: 1 Passed pre-reject verification ---- 1 Total to be rejected +-----------------------------------------------------------------------------+ Rejecting Software... +-----------------------------------------------------------------------------+ installp: REJECTING software for: bos.alt_disk_install.boot_images 6.1.9.406 Finished processing all filesets. (Total time: 3 secs). +-----------------------------------------------------------------------------+ Summaries: +-----------------------------------------------------------------------------+ Installation Summary -------------------- Name Level Part Event Result ------------------------------------------------------------------------------- bos.alt_disk_install.boot_i 6.1.9.406 USR REJECT SUCCESS # installp -r bos.alt_disk_install.rte +-----------------------------------------------------------------------------+ Pre-reject Verification... +-----------------------------------------------------------------------------+ Verifying selections...done Verifying requisites...done Results... SUCCESSES --------- Filesets listed in this section passed pre-reject verification and will be rejected. Selected Filesets ----------------- bos.alt_disk_install.rte 6.1.9.400 # Alternate Disk Installation ... << End of Success Section >> FILESET STATISTICS ------------------ 1 Selected to be rejected, of which: 1 Passed pre-reject verification ---- 1 Total to be rejected +-----------------------------------------------------------------------------+ Rejecting Software... +-----------------------------------------------------------------------------+ installp: REJECTING software for: bos.alt_disk_install.rte 6.1.9.400 Successfully updated the Kernel Authorization Table. Successfully updated the Kernel Role Table. Successfully updated the Kernel Command Table. Successfully updated the Kernel Device Table. Successfully updated the Kernel Object Domain Table. Successfully updated the Kernel Domains Table. Successfully updated the Kernel Authorization Table. Successfully updated the Kernel Role Table. Successfully updated the Kernel Command Table. Successfully updated the Kernel Device Table. Successfully updated the Kernel Object Domain Table. Successfully updated the Kernel Domains Table. Successfully updated the Kernel Authorization Table. Successfully updated the Kernel Role Table. Successfully updated the Kernel Command Table. Successfully updated the Kernel Device Table. Successfully updated the Kernel Object Domain Table. Successfully updated the Kernel Domains Table. Successfully updated the Kernel Authorization Table. Successfully updated the Kernel Role Table. Successfully updated the Kernel Command Table. Successfully updated the Kernel Device Table. Successfully updated the Kernel Object Domain Table. Successfully updated the Kernel Domains Table. Finished processing all filesets. (Total time: 6 secs). +-----------------------------------------------------------------------------+ Summaries: +-----------------------------------------------------------------------------+ Installation Summary -------------------- Name Level Part Event Result ------------------------------------------------------------------------------- bos.alt_disk_install.rte 6.1.9.400 ROOT REJECT SUCCESS bos.alt_disk_install.rte 6.1.9.400 USR REJECT SUCCESS
Once having reverted to the older version, viosupgrade ran normally.
$ viosupgrade -l -i /mnt/PowerVM31010.mksysb -a hdisk1 Welcome to viosupgrade tool. Operation triggered for given node(s). Broadcast message from root@VIO2 (vty0) at 15:13:07 ... WARNING!!! VIOS Upgrade operation is in progress. Kindly Refrain from making any configuration changes... Please wait for completion.. Upgrading from ioslevel '2.2.6.51' to '3.1.0.10'. Verifying whether the MPIO software(s) is installed on the VIOS. Following list of fileset(s) required for the VIOS meta data restore seems to be not present in the provided installation image. Continuing with the upgrade process may result in restore failure post installation: 1: devices.fcp.disk.ibm.mpio.rte => IBM MPIO FCP Disk Device 2: devices.sddpcm.61.rte => IBM SDD PCM for AIX V61 Choice[Y/N]: y Verification of the MPIO software(s) is successful. Initiating VIOS configuration backup.. VIOS configuration backup successful. Initiating installation on alternate disk(s).. Installation on alternate disk(s) successful. Copying files to altinst_rootvg. Waking up altinst_rootvg successful. VIOS will be rebooted after '60' seconds to boot from the newly installed disk. Press contrl+c to terminate. VIOS metadata restore (viosbr -restore) will be automatically resumed after the reboot. VIOS may be rebooted once during this restore process. Refrain from making any changes to the VIOS virtual configurations during the restore process. You can verify the restore status using 'viosupgrade -l -q' command and resume your operation after the completion of the restore process. portmir: Cannot unload mirror module portmir: Mirroring is stopped. Rebooting . . .