CUCM is a key component in Cisco collab deployment and the deployment/dependencies can be very different from Org to Org. So changing anything on CUCM needs a lot of prep and checks.
Recently we did the CUCM migration from one DC to another. So I’ve put together this checklist that gives you a quick overview of all the steps you need to do for CUCM migration. I will list existing resources available on the Internet and add a few things from my experience.
Before making any changes, please consult with Cisco TAC or do your own research /testing in lab environment.
Please dont consider this as step by step guide and be open to possibility that I may have missed few points:
- You may not have an identical environment.
- You may even hit different issues which I didn’t come across.
In short, I am not responsible for any disaster 🙂
Background
IP / Subnet is usually tied to location/DC. So when you change DC, you may have to changes the IP address of the server too.
Changing IP address VS hostname ( or both) has different effects. If possible, try to change a fewer thing at a time.
If you can keep the hostname/fqdn, its good ( less work in one change window).
If for some reason you can keep the same IP address and system setting then you even have less work to do. You could use DRS (Backup- Restore) or the method I am going to mention here ( using PCD).
What are the options?
VMotion isn’t supported for this task (by Cisco, or so I was told) and isn’t easy – taking the whole VM over the WAN can take time, and its unpredictable/scary when you have limited downtime.
3rd part tools – Zerto or Commvault may copy the entire VM in advance and sync the changes just before the cutover window, but can create other issues and can be complicated and have more dependencies.
DRS (Backup – Restore) and then re-addressing – Could work with lots of nerd knobs, but can be complicated and need lots of manual work. If you can have the same system settings at time of backup restore etc etc. And then change all the system/enterprise parameters, URLs in config wherever you have the CUCM IP.
Cisco PCD (Prime Collaboration Deployment) – In plain English, its a free Orchestration tool by Cisco, which you can deploy on VM and it can do many things for UC (CUCM, IM&P, CUC, UCCX)) servers like Migration, Installation, upgrade etc.
Have a quick look at Chapter: Cisco Prime Collaboration Deployment Features
Admin guide covers all details on How to deploy, maintain ..
I have used PCD in the past for CUCM P2V migration (From MCS to VM) as well. Note that Migration task is only supported for CUCM and IM&P. So for other apps:
- CUC and UCCX – you will need to use DRS method.
- Expressway, VMotion is recommended by Cisco.
- Cisco Meeting Server – You can use Backup and Restore.
In this blog, I will only focus on CUCM using PCD.
Why PCD?
- Supported by Cisco, Tested and Super easy.
- Doesn’t cost anything. Less dependencies on other teams.
- Can do migration ( installation, readdress , upgrade .. even updates enterprise parameters after the migration).
High Level Plan
- Deploy PCD server. Try to deploy at new DC. Check this chapter on how to.
- Add/Discover your existing CUCM cluster on PCD. It will need O.S admin password.
- It will fetch the server info. At the time of migration, it will take the backup, restore at destination, migrate all user facing feature, change all the parameters at new clusters and shutdown the existing cluster.
- It will fetch the server info. At the time of migration, it will take the backup, restore at destination, migrate all user facing feature, change all the parameters at new clusters and shutdown the existing cluster.
- Deploy OVF (Create VM) on destination (new DC) side.
- Add an ESXi host server on PCD where these new VMs (New DC) are created.
- To communicate with an ESXi host server, Cisco Prime Collaboration Deployment requires either root access to the ESXi software or a nonroot user with Host(Configuration, Storage Partition Configuration) and Virtual Machine(Interaction, Configure CD Media, Configure Floppy Media, Device Connection, Power Off, and Power On) privileges enabled.
- If this root account password is 16+ characters, Be aware of this bug
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva95404/?rfs=iqvred
- Download bootable image for installation of new CUCM.
- Define your sftp or use the local sftp within PCD, for copying the bootable image for installation. If using sftp built-in to PCD , use WinSCP and copy the file at this location:
- For Migration > ISO files should be copied into /fresh_install directory.
- The Username for SFTP will be “adminsftp” and the password will be PCD OS Administration Password.
- Plan a change window and create a task in PCD to do the migration. If you have SMTP configured, it will send you email when migration is done 🙂 Or you can monitor the progress from GUI itself.
Plan your migration
- Check all the compatibilities with new HW, ESXi ….
- Figure out all the dependencies. Discuss and work with your Network, System/DC/AD team. This is not the exhaustive list here but I suggest that you put all the things in writing and keep updating/discussing it with all the stakeholders.
- Get your new IP address, Def GW, System settings, OS admin password …. ready.
- Figure out what will be the new DNS, NTP, LDAP server config will be at new DC.
- When you create a new VM at new DC, all the network connectivity/firewall rules within new DC, from new DC to All branch locations and from New to OLD DC ( as some servers still be at old DC). Vlan config on server side etc etc
- If you are keeping some apps/server in old DC, figure out if its meeting all the latency requirements.
- Access/process to update the DNS A records of CUCM with new IP address, if you are keeping the hostname.
- Write down all the DHCP scope where you need to update option 150 (TFTP Ip address for IP Phones). Discuss who has access, verify the access and that you understand where the DHCP scope is for all branch locations, and who will update it?
- Discuss with your local IT support about manual update of TFTP IP address if some phones have static IP address. If you have phone remote tool, then its easier to do it by yourself.
- Check all your Infra is using the FQDN for this CUCM cluster, if not then on other infra side use FQDN instead of IP address. Such as other CUCM having SIP Trunks. VC endpoints configuration having fqdn of CUCM server in provisioning config , Expressway neighbor zones, Voice GW, media resources, 3rd part servers …
If any of these are using IP address of your CUCM, plan a change window in advance and make it on FQDN so it will save you some time.
- Have your PCD ready and get yourself familiar with it. Deploy the VMs, add existing cluster, new esxi to PCD.
Try to create a migration task, figure out all the steps it shows and all the details you need handy to put. See the option of “Re-addressing with new IP” with migration. I suggest to avoid patching/upgrade in combination to migration as if you hit some bug, it will be complicated to troubleshoot. - Keep your SFTP server ready, get the bootable image ready.
- Plan about your licensing. If its 12.5.x with smart licensing then it should be okay and may not need any additional effort.
- Read the documentation carefully and look for any existing bug with your PCD/CUCM version, related to migration. I faced this unexpected bug in between migration, had to push through the errors. It went all fine but scared/confused me and wasted some time. In this case, its asking to delete those last few steps where PCD shutdown the old cluster.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut43030/?rfs=iqvred
If your CUCM version is 12.0.x, PCD Migrations require a COP file: ciscocm-slm-migration.k3.cop.sgn
Clarify with Cisco TAC in advance or check the document. - Plan your change/downtime window carefully.
For example, on weekend there maybe a backup job running and chocking all the BW at the same time you are doing the migration. It will add more time when PCD is trying to copy the backup from source to destination.
Talk to your Network team in advance.
Another point is, Initial tasks such as copying the backup and installation to new location of a cluster (3 nodes )may take up to approx. 8 hours ( depends on size, BW ) so maybe you can even start at Friday evening. Schedule it and do something else ( maybe sleep 🙂 ). Take this advice lightly, It depends on your situation, process and lot of things. - Open a change, Fix the date and have people ready to support.
- Open a TAC case, ask all the questions. Leave the case open as Sev3 and inform them about change window to pass onto Engineer who will be available that day. So if you need any help, you will need to call TAC frontline and raise the Sev to get the Engineer on Webex right away.
- Write down your rollback plan. In this case, you could simply Turn on your Old cluster and undo all the network/DNS changes.
- Write down your test plan thoroughly.
- Do a test of entire change, if possible.
1 day before or in same change window
- Have all the credentials ready/handy.
- Verify all your access.
- Take a fresh backup.
- Give a heads-up to all people involved.
In change window
- Take a fresh backup. take all the screenshots from RTMT, CUCM config.
For example, when IP phone/ MGCP GW/ endpoint isn’t registered, it doesnt show IP address on CUCM admin page. So, with new cluster, if you want to check any particular IP phone GUI, its difficult to find IP from CUCM side. Having a screenshot or recording might help.
Screenshots from RTMT are super helpful, dont miss it. - Before you start the migration, If your cluster is in mixed mode, I recommend to use “Prepare Cluster for Rollback to Pre 8.0” in enterprise parameter. This effectively disables the ITL functions on the phone. So phones blindly trusts the next CTL or ITL file.
Check with Cisco TAC to be 100 % sure. - Start the migration task in PCD, with readdressing of CUCM. Wait for it to finish. Once its done, note that it should have updated the IP address in all the parameters/URLs in CUCM config as well.
- Make other changes ( DNS record update, DHCP scope update), Remove the Prepare Cluster for Rollback to Pre 8.0. Check the status/health. Shutdown the old cluster if not done automatically.
Check all the registrations, services. Note that when you update DNS record, it may take time to propagate everywhere. Update in between AD/DNS servers take some time, this may add some delay for other servers to update the SIP trunk with new cluster. So, perhaps you can update the DNS records when you start (or in mid) the PCD task itself. It depends on your downtime window planning.
If you are checking from you Win PC cmd, make sure to do “ipconfig /all flushdns” or else, it will show your cached entry and you will be confused.
If any SIP trunk/registration is stuck, you may need to reset it from other side. We had the issue from Skype side and had to reset the trunk from there. - Verify all the registrations, compare your RTMT screenshots and everything. Test all the services thoroughly. In my case, extension mobility was broken as PCD messed up the URLs (maybe some bug).
It added something in the end and had other issues, so had to find the URL from CUCM doc and put it again. - If all checks/testing goes good, make sure to check if all tests are passed in “utils diagnose test”.
If you see NTP stratum error, could be either you are using stratum 4 or above or Win server as NTP. Cisco UC apps throws error and recommend to use Linux based ntp.
Do check all the usual things like services up, DB cluster is good etc etc. - If you suspect some registrations are missing. Turn on the old cluster and see if you have registration still hitting to it. If so, you will know which all devices still need some work. Make sure to shut the old cluster down when you are done.
If you didnt do the re-address, then it should be relatively easy. If you also changed the hostname then there could be some difference in steps, instead of updating the DNS records, you will create new and points to new fqdn. You will also need to do other steps such as Cert regeneration, and maybe more.
Post Migration
Take a fresh backup, VM snapshot of new cluster.
Keep you old VM for month or so.
You can keep the PCD for usual tasks or can get rid of it. From 12.5.x, CUCM has mini-pcd built in for its own cluster upgrade orchestration. But you can keep this PCD VM for other installation, upgrade tasks (CUC, UCCX ..).
Conclusion
Migration of CUCM cluster isn’t easy task and you may see some different challenges based on your deployment, dependencies and process. So plan it good. Write things down. Communicate to all stakeholders. Have people on standby.
PS: I avoid putting/copy-paste the info which is already in the Cisco doc and prefer to provide the URL.
I hope this helps. If you find have any feedback, suggestions, please feel free to put the comment or DM me on LinkedIn. Thank you for your time.
