Checklist – CUCM Upgrade 11.5 to 12.5.x

Deploy/Upgrade, Quick How-to 7 Minutes

This checklist gives you quick overview of all the steps you need to do for CUCM upgrade 11.5.x to 12.5.x. I will list existing resources available on the Internet and add few things from my experience.
Before making any changes, please consult with Cisco TAC or do your own research /testing in lab environment.

Please dont consider this as step by step guide and be open to possibility that I may have missed few points:

  • I performed the upgrade few months ago so I might have forgot few things. I am human 🙂
  • You may not have identical environment.
  • You may even hit different issues which I didnt come across.

In short, I am not responsible for any disaster 🙂

Look for BRKUCC-2011 on Cisco live library, there are 2 prez/sessions contain all screenshots/details. Cisco live on demand library is very useful, informative and free. Just logon with your cco account or signup.

https://www.ciscolive.com/global/on-demand-library.html?#/

Keep Official upgrade guide handy for exhaustive list:

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/12_5_1/cucm_b_upgrade-migration-guide-125x/cucm_b_upgrade-guide-1251su2_chapter_00.html

Note – On CUCM12.5 SU2, we are facing the bug CSCth35648 ( and a few others), causing ITL issues.

Run this command and if you see the error on the last line, then you are facing the bug. Contact Cisco TAC as they need to fix the permission issue using the root account. Also a few other things maybe.
In version 12.X we expect to have itl and ctl signed by ITLRecovery cert, if you don’t see it then it also needs to be fixed.

Rather than explain in detail, I let you contact Cisco TAC with bug ID and then will help you with actions, based on your scenario/specific issue.

admin: show itl
“Failed or interrupted I/O operations.
Error parsing the ITL File.”

Pre-req

  1. Figure out your target version. Know your WHY. Check release notes for open bug / open security vulnerabilities.
  2. Check your Platform / server hardware compatibility with target version.
    https://www.cisco.com/c/en/us/support/unified-communications/unified-communications-manager-version-12-5/model.html#CompatibilityInformation
  3. Check your esxi version compatibility with target version. ( check BRKUCC-2011 prez).
  4. Check if any OVA template changes required ( RAM, CPU increase changes recommended or required).
    https://www.cisco.com/web/software/283088407/145119/cucm.ova.README.pdf
    https://docwiki.cisco.com/docwiki-eol.html#VMware_vSphere_ESXi_Version_Support_for_Call_Processing_and_System_Management_Applications
  5. Check compatibility of other applications relying on CUCM/CUC/UCCX with target version ( SBC, Voice GW, 3rd party apps..).
  6. Check the list of Deprecated Phone models listed in document, if you have any, take care of that in advance ( upgrade, replacement ..).
  7. Migrate your ELM/PLM licensing to Smart licensing. Check Cisco live BRKUCC-2725 for more details.
  8. If you decide, Deploy CSSM in advance.

https://www.cisco.com/go/smartlicensing

For some servers/devices only Smart licensing is the option after specific version ( e.g. CUCM 12.5.x) , for some its still optional ( UCCX 12.5.x).
In Plain English, With Smart licensing, Cisco sync your license status/inventory to cloud rather than PAK based/ tied to your device offline mode ( so no pain when you change general setting and license mac changes). You need to decide whether you want to sync your Infra devices to Cisco licensing portal directly ( device having internet access), via http proxy or you can deploy CSSM on prem ( lightweight VM) which can sync license from Cisco and with your infra ( Man in the middle 🙂 ).

Smart licensing goes beyond UC servers You can deploy 1 CSSM and use for entire Org. You will need to organize and clarify the Smart Account Structure in advance. Check Cisco live BRKUCC-2725 for more details.

Example – Top – Smart account = ABC.com
Sub level:
Virtual account = AP Collab
Virtual account = EU Collab

CSSM is simply a router/proxy which talks to internal devices with identifier and sync available licenses for your Smart accounts.

If you decide to use CSSM with Smart licensing, I suggest create a Smart account with Cisco and deploy/configure CCSM in advance then focus on license migration.

BRKUCC-2725 has all details on how to do license migration. If you get the error while doing the migration, open a case with licensing ( few times I got an error and licensing team helped doing from back end), they will deposit the licenses to your smart account. Probably you will need to provide, Smart account name, Smart account domain name, Virtual account name.

Plan your upgrade

  1. Plan your change window, open a change etc etc.
  2. Keep note that with RU upgrade, Server is down while upgrade is taking place on inactive partition. RU = refresh upgrade = Underlying Linux version is changing. Plan your downtime window accordingly.
  3. Decide your upgrade method – Direct or PCD. PCD = Prime Collaboration Deployment, VM you can deploy to orchestrate the migration/upgrades for you.
  4. Download the Software images and keep on sftp server. Notice the difference in between Restricted VS Unrestricted version, use the correct image.
    https://software.cisco.com/download/home/286322286/type/286319236/release/12.5(1)SU3
  5. Keep a OVA file/ bootable image of your existing version on sftp just in case you will need to rebuild the server. If you have Enterprise Agreement with Cisco, you can download by yourself or else, open a Cisco TAC and ask for images.
    Bootable images arent available/same under download page.
    From 12.x CUCM and Unity connection ISO are separate.
  6. Either verify that you have rights to deploy new VM or keep your team ready who can create a new VM in change window, if you have to rebuild a server ( worst case scenario).
  7. If your server has a uptime of more than 365 days, Its nice to plan and give it a reboot in advance to your upgrade.
  8. Verify you have rights to remove existing/old VM snapshots before upgrade.
  9. Verify the free space. You will need ~43 GB space. Some ways to clean space are:
    Run a cop file which cleanup the space( may need to do in downtime). Ask Cisco TAC before you do it, it might delete your inactive partition sw.
    Reduce the log size.
    https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200581-Procedure-to-Adjust-WaterMark-in-RTMT-of.html
    Delete unused load files from CUCM O.S admin –> load management.
  10. Open a Cisco TAC case for support in advance and inform them to requeue the case to engineer who is available in your change window. Put basic details about what you are doing in that case, it will save your time in change window.
    Note that you may need to call TAC frontline and may or may not need to raise a Sev in change window as Cisco TAC could be busy with multiple cases.
  11. Read the upgrade guide carefully.
  12. Review the Recommended Sequence in advance from upgrade doc. See some notes here:
    Publisher node MUST be Upgraded first.
    Switchover options depends on the collaboration product.
    CUCM Publisher can be switched over before Subscriber(s) Upgrade Startup.
    DRU Only –> Switchover IM&P Publisher after CUCM Cluster Database Replication Completes.
    CUC Publisher must NOT be switched over before Subscriber Upgrade Completes.
  13. Try the upgrade in lab/ Pre-pod to avoid any surprises.
  14. Prepare your test plan in advance. What you are gonna test and how are you gonna test it.

1 day before or in same change window

  1. Verify/take your fresh backup. Back Up Custom Ringtones and Background Images.
  2. Ensure SFTP is up. Please use SFTP in same location to CUCM to minimize the delay( upgrade iso is ~4G).
  3. Keep your O.S /Cluster password and SFTP password handy.

In change window ( Pre-upgrade tasks)

  1. Run Pre-upgrade check cop file. It checks many pre-req but common issue you may see is NTP/Free space/Expire cert/ Simple password, fix issues before upgrade.
    If you are using Win NTP or Stratum 4 or above, you may see error. Fix your cert. Change of password. Check upgrade guide for more.
  2. Verify your backup. Take details/screenshots of registered device count/ VG/ SIP trunks.
  3. Make VM/OVA related changes RAM/CPU if needed.
  4. Change VM compatibility. CUCM 12.5 needs ESXi 6.5 ( 13). Note that once you change, you might not be able to roll this back.
  5. Change Guest OS type. See the doc, you will nee to change it to CentOS 7 (64 bit). Its under VM general setting, if its greyed out maybe because you need to shutdown the server first. Should be done before CUCM upgrade.
  6. Change adapter type to VMXNet3 ( if its not already).
  7. Update to the latest Vmware tools.
  8. Remove old VM snapshots.

Upgrade

  1. You can disable TFTP services and do few other tweaks mentioned in document to expedite/smooth the upgrade.
  2. Do the upgrade from O.S admin. Follow the seq mentioned in doc ( Upgrade Pub –> Sub1 –> Sub2 …, Switch version Pub –> Sub1 –> Sub2 ..). Be aware of some known issues, with CUC, even if you select do not reboot after upgrade, it might 🙂
    For CUC, you may need to run a pre-req cop file as ISO file is changing. Ask TAC.
    When you do uccx es release, verify the procedure from the doc, it maybe recommended to do from cli ( as there maybe known issue with tomcat).

    Note – In RU, DB replication may take some time, be patient or ask TAC.

Post upgrade

  1. Run the post upgrade COP.
  2. Reset any parameters you changed to optimize the upgrade ( TFTP, enterprise parameters). Check upgrade doc for more details.
  3. Upgrade your RTMT.
  4. Check Device count/ Trunk/ Vg status.
  5. Check all the services. Check all the integration ( CCX, CUC …)
  6. Optional : Switch from Native to Open Vmware Tools via OS Admin CLI –> utils vmtools switch open
  7. Change Transport Settings for CUCM for Smart Licensing ( you can find the URL from CSSM or from doc or google it).
    Common issue you might face is DNS resolution for CSSM , so use IP address and port 80. If your DNS and Cert is all good then it could be different issue, like reachability ( IP/Port).
  8. Create or copy existing registration token from CSSM or Smart Software Manager satellite server to CUCM.
  9. Do your usual testing.

If you find have any feedback, suggestions, please feel free to put the comment or DM me on LinkedIn.

Published by Lalit Pamnani

Published

2 thoughts on “Checklist – CUCM Upgrade 11.5 to 12.5.x”

  1. good work mate .
    Question : why Run the post upgrade COP ? what this will do ? what’s the need of this file and for which we need to run CUC or CUCM (do we need to run on all nodes) ?

    Regards

    Reply

    1. hello Harikesh,
      Thank you, I hope it helps 🙂

      Post upgrade COP would compare data taken from Pe-upgrade COP and tell you if any errors are post-upgrade. In short, it saves you manual efforts of verifying state pre-upgrade VS post-upgrade. I still advise to take some screenshots from rtmt about registration count, existing errors just in case, but this cop file should give you the report of what’s broken (if anything) after the upgrade.
      This is just for CUCM and I think running it on Pub should be enough as it does run the check for the entire cluster ( I don’t remember 100%)

      Common issues could be network services, NTP, cluster or db related issue, Registration / GW/ trunk active count down .. Look for this section on the doc for the detailed list.

      Run Upgrade Readiness COP File (Post-upgrade)

      https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/12_5_1/cucm_b_upgrade-migration-guide-125x/cucm_b_upgrade-guide-1251su2_chapter_01101.html#task_F186272DFEA2E2DA20A81C23D3F96CE3

      Look at this doc for more explanations, it’s old but still explains the use/background of it.

      https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/214137-upgrade-readiness-checks-cop-file-for-cu.html

      Reply

Leave a comment