The Request:
I have a client with multiple 6807 VSS pairs that required an IOS upgrade. All of the pairs have a single SUP2-T in each chassis and were in the 15 code train. Although the ISSU process is very straight forward I wanted to put this quick process up as I had to search through multiple documents to gather all the pieces I needed to knock it out.
The Solution:
Since these switches were in the proper code train to utilize ISSU I decided that was the best route to go. It also helps that everything was already dual-homed. This process is for VSS pairs with only one SUP per chassis! If you have another configuration you can reference the Cisco document provided at the bottom of the post. Some example text was taken from the Cisco Document referenced below One of the first things you want to verify is that there is a current boot variable configured on the VSS pair pointing to the version of code that is running currently. Some devices only have one version of code on the bootdisk so there is not a boot variable configured. For the ISSU to perform properly you MUST configure the boot variable:
Router(config)# boot system flash bootdisk:s72033-oldversion.v1
Next you will want to download the new image from your favorite file transfer spot to your bootdisk. I prefer to use FTP:
Router# copy ftp: bootdisk:
Address or name of remote host []? test.ftp.local
Source filename []? s72033-newversion.v2
Destination filename [s72033-newversion.v2]?
Accessing ftp://test.ftp.local/s72033-newversion.v2...!!!!! Complete
Once the image is downloaded you will want to copy the image over to the slavebootdisk:
Router# copy bootdisk: slavebootdisk:
Source filename []? s72033-newversion.v2
Destination filename [s72033-newversion.v2]?
Copy in progress...CCCCCCCCCCCCCCC Complete
After you have the images on both bootdisks you can begin the ISSU process. The first step is to verify the VSS pair is ready for the ISSU upgrade:
Router# show issu state detail
Slot = 1/3
RP State = Active
ISSU State = Init
Boot Variable = bootdisk:s72033-oldversion.v1,12;
Operating Mode = sso
Primary Version = N/A
Secondary Version = N/A
Current Version = bootdisk:s72033-oldversion.v1
Variable Store = PrstVbl
Slot = 2/3
RP State = Standby
ISSU State = Init
Boot Variable = bootdisk:s72033-oldversion.v1,12;
Operating Mode = sso
Primary Version = N/A
Secondary Version = N/A
Current Version = bootdisk:s72033-oldversion.v1
Router# show redundancy states
my state = 13 -ACTIVE
peer state = 8 -STANDBY HOT
Mode = Duplex
Unit = Secondary
Unit ID = 18
Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Communications = Up
client count = 132
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 9000 milliseconds
keep_alive count = 0
keep_alive threshold = 18
RF debug mask = 0x0
Once the pair is verified to be good to go you will want to load the new image onto the standby chassis. This will load the new code on the standby and reload the chassis. If you have EVERYTHING DUAL HOMED you will not see an interruption in traffic.
Router# issu loadversion bootdisk:s72033-newversion.v2
000133: Aug 6 16:17:44.486 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/3/4, changed state to down
000134: Aug 6 16:17:43.507 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/3/4, changed state to down
000135: Aug 6 16:17:43.563 PST: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/3/4, changed state to down
000136: Aug 6 16:17:44.919 PST: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/3/4, changed state to down
(Deleted many interface and protocol down messages)
%issu loadversion executed successfully, Standby is being reloaded
(Deleted many interface and protocol down messages, then interface and protocol up messages)
0000148: Aug 6 16:27:54.154 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/3/5, changed state to up
000149: Aug 6 16:27:54.174 PST: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/3/5, changed state to up
000150: Aug 6 16:27:54.186 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/3/5, changed state to up
000151: Aug 6 16:32:58.030 PST: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
During the process you will want to run the following command until you see that the ISSU Sub-State is Load Version Complete:
Router# show issu state detail
Slot = 1/3
RP State = Active
ISSU State = Load Version
Boot Variable = bootdisk:s72033-oldversion.v1,12
Operating Mode = sso
Primary Version = bootdisk:s72033-oldversion.v1
Secondary Version = bootdisk:s72033-newversion.v2
Current Version = bootdisk:s72033-oldversion.v1
Variable Store = PrstVbl
Slot = 2/3
RP State = Standby
ISSU State = Load Version
Boot Variable = bootdisk:s72033-newversion.v2,12;bootdisk:s72033-oldversion.v1,12
Operating Mode = sso
Primary Version = bootdisk:s72033-oldversion.v1
Secondary Version = bootdisk:s72033-newversion.v2
Current Version = bootdisk:s72033-newversion.v2
Router# show redundancy status
my state = 13 -ACTIVE
peer state = 8 -STANDBY HOT
Mode = Duplex
Unit = Secondary
Unit ID = 18
Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Communications = Up
client count = 132
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 9000 milliseconds
keep_alive count = 1
keep_alive threshold = 18
RF debug mask = 0x0
Once this state is reached you will want to go into the next step to force a switchover to the standby chassis that is running the new code and being upgrading the remaining chassis. Once you issue the runversion command it will start a rollback timer that is by default set to 45 minutes. If you do not commit the version (next step) before the timer runs out the upgrade will be rolled back!
Router# issu runversion
This command will reload the Active unit. Proceed ? [confirm]
(Deleted many lines)
Download Start
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(Deleted many lines)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Download Completed! Booting the image.
Self decompressing the image : ##########################################################################################
(Deleted many lines)
running startup....
(Deleted many lines)
000147: Aug 6 16:53:43.199 PST: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
Once the chassis has rebooted we will once again want to verify the ISSU state and redundancy state:
Router# show issu state detail
Slot = 2/3
RP State = Active
ISSU State = Run Version
Boot Variable = bootdisk:s72033-newversion.v2,12;bootdisk:s72033-oldversion.v1,12
Operating Mode = sso
Primary Version = bootdisk:s72033-newversion.v2
Secondary Version = bootdisk:s72033-oldversion.v1
Current Version = bootdisk:s72033-newversion.v2
Variable Store = PrstVbl
Slot = 1/3
RP State = Standby
ISSU State = Run Version
Boot Variable = bootdisk:s72033-oldversion.v1,12
Operating Mode = sso
Primary Version = bootdisk:s72033-newversion.v2
Secondary Version = bootdisk:s72033-oldversion.v1
Current Version = bootdisk:s72033-oldversion.v1
Router# show redundancy status
my state = 13 -ACTIVE
peer state = 8 -STANDBY HOT
Mode = Duplex
Unit = Primary
Unit ID = 39
Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Communications = Up
client count = 134
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 9000 milliseconds
keep_alive count = 1
keep_alive threshold = 18
RF debug mask = 0x0
You will now want to commit the new version to reload the standby chassis and have it run the new image:
Router# issu commitversion
Building configuration...
[OK]
000148: Aug 6 17:17:28.267 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/3/4, changed state to down
000149: Aug 6 17:17:28.287 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/3/4, changed state to down
(Deleted many interface and protocol down messages)
%issu commitversion executed successfully
(Deleted many interface and protocol down messages, then interface and protocol up messages)
000181: Aug 6 17:41:51.086 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/3/5, changed state to up
000182: Aug 6 17:42:52.290 PST: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
Once this has completed your entire VSS pair will be upgraded. You can verify the upgrade by once again checking the ISSU & redundancy state:
Router# show issu state detail
Slot = 2/3
RP State = Active
ISSU State = Init
Boot Variable = bootdisk:s72033-newversion.v2,12;bootdisk:s72033-oldversion.v1,12
Operating Mode = sso
Primary Version = N/A
Secondary Version = N/A
Current Version = bootdisk:s72033-newversion.v2
Variable Store = PrstVbl
Slot = 1/3
RP State = Standby
ISSU State = Init
Boot Variable = bootdisk:s72033-newversion.v2,12;bootdisk:s72033-oldversion.v1,12
Operating Mode = sso
Primary Version = N/A
Secondary Version = N/A
Current Version = bootdisk:s72033-newversion.v2
Router# show redundancy status
my state = 13 -ACTIVE
peer state = 8 -STANDBY HOT
Mode = Duplex
Unit = Primary
Unit ID = 39
Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Communications = Up
client count = 134
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 9000 milliseconds
keep_alive count = 1
keep_alive threshold = 18
RF debug mask = 0x0
Conclusion:
Once I found all of the information I needed this was a very easy process and actually did not take as long as I expected. The boot variable was one issue I ran into but once I figured out what it was asking for that was easy to fix. Another thing to be mindful of is that after the upgrade process your original active processor will be the standby processor.
THANKS!
I would like to say a quick thank you to the following references while I was working through this:
17 comments
Chris
Monday, Mar 16, 2015
Dan
In reply to Chris
Tuesday, Mar 17, 2015
Hk
Thursday, Jun 11, 2015
Dan
In reply to Hk
Thursday, Jun 11, 2015
Shehzad
Tuesday, Sep 22, 2015
brad wright
Thursday, Jul 6, 2017
thanks for sharing this
this has saved me loads of time searching Cisco’s mind field of support files
helped loads
regards
brad
Dan
In reply to brad wright
Wednesday, Jul 12, 2017
christian
Thursday, Nov 23, 2017
hi! how long does the loadversion take? i’m upgrading right now and i have no log message from the standby since 20 minutes now…
thanks, christian
Dan
In reply to christian
Thursday, Nov 23, 2017
Hello!
I have not done this in a while, but I do recall it taking long enough to make me REALLY uncomfortable.
What was the last message you received?
christian
In reply to christian
Friday, Nov 24, 2017
christian
Tuesday, Jan 23, 2018
Hi,
does this work for a DOWNGRADE in the same way?
Christian
christian
Tuesday, Jan 23, 2018
Dan
In reply to christian
Tuesday, Jan 23, 2018
Christian,
I have never used this method to perform a downgrade. I do not have access to my 6807 lab any longer so I am not able to test.
As far as email notifications, you will receive notifications whenever there are any comments on the post as a whole, not just replies to your comment.
If you happen to test the downgrade please let us know how it goes.
Thanks!
Ahmed
Thursday, May 17, 2018
Hi What is the way forward if you are upgrading 6509 with SUP 2T for this path 15.1(2)X versions are not compatible with 15.2(1)X IOS versions. Cisco says these are different trails and do not support ISSU/ e FSU. So I am left with standard upgrade procedure which involves a complete downtime.
My Question - Can I upgrade one switch at a time (eg. Standby Switch) in VSS and then do a switchover of Active - Standby and upgrade the other switch? this way will it work with pair running different versions during the process. ?
Valky
Monday, Feb 25, 2019
Umair
Saturday, Nov 16, 2019
Nawaz
Sunday, Dec 4, 2022
Hi Williams,
thanks for you detailed description over here, I have a sort of same issue with C6807-XL Modular Switch, when I try to save the configurations with commands (Write memory, copy running-config) following error came up, also the switch is Rommon mode always when I reload it or power it up. I have to put the boot command manualy to get the switch command prompt. could you please help to fix this issue and suggest the possible solution. thank you in advance.
Router(config)#do wr Building configuration…
*Jan 1 01:18:33.439: % Secondary config IOCTL failed
nvfs_ifs_ioctl:Nvram device needs a reformat. Please run erase /all nvram: to format the device
*Jan 1 01:18:39.643: % Secondary config IOCTL failed
nvfs_ifs_ioctl:Nvram device needs a reformat. Please run erase /all nvram: to format the device
*Jan 1 01:18:45.847: % Secondary config IOCTL failed
nvfs_ifs_ioctl:Nvram device needs a reformat. Please run erase /all nvram: to format the device
*Jan 1 01:18:52.051: % Secondary config IOCTL failed
nvfs_ifs_ioctl:Nvram device needs a reformat. Please run erase /all nvram: to format the device
*Jan 1 01:18:58.051: %HA_CONFIG_SYNC-3-GENERAL: Failed to sync startup-config to standby RP
*Jan 1 01:18:58.051: %PFREDUN-3-STANDBY_OUT_OF_SYNC: Active and Standby are out of sync. -Traceback= 241B0C9z 5E30C05z 66E8BDEz F8864Cz F8850Ez 23BEFAEz 234ED9Bz 234E42Cz 234CD2Ez 234B3FFz 234ED9Bz 234E42Cz 234CD2Ez F97FECz 2364848z 23638A7z *Jan 1 01:18:58.055: %RF-5-RF_RELOAD: Peer reload. Reason: Config sync fails *Jan 1 01:18:58.055: %OIR-3-PWRCYCLE: Card in module 4, is being power-cycled ‘Config sync fails’ *Jan 1 01:18:58.175: %C6KENV-4-LOWER_SLOT_EMPTY: The lower adjacent slot of module 3 might be empty. Airdam must be installed in that slot to be NEBS compliant *Jan 1 01:18:58.991: %PFREDUN-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode Router(config)# Router(config)# Router(config)# Router(config)# Router(config)# *Jan 1 01:20:04.175: %HA_CONFIG_SYNC-3-GENERAL: Failed to sync private-config to standby RP
*Jan 1 01:20:04.175: %PFREDUN-3-STANDBY_OUT_OF_SYNC: Active and Standby are out of sync. -Traceback= 241B0C9z 5E30C05z 66E8BDEz F8864Cz F88699z 23BEFF5z 234ED9Bz 234E42Cz 234CD2Ez 234B3FFz 234ED9Bz 234E42Cz 234CD2Ez F97FECz 2364848z 23638A7z *Jan 1 01:20:04.179: %RF-5-RF_RELOAD: Peer reload. Reason: Config sync fails *Jan 1 01:20:04.179: %OIR-3-PWRCYCLE: Card in module 4, is being power-cycled ‘Config sync fails’ *Jan 1 01:20:04.279: %C6KENV-4-LOWER_SLOT_EMPTY: The lower adjacent slot of module 3 might be empty. Airdam must be installed in that slot to be NEBS compliant
Say something
Thank you
Your post has been submitted and will be published once it has been approved.
OK