Cisco Expressway(VCS) migration from one DC to another

Deploy/Upgrade 10 Minutes

Cisco expressway server is usually deployed in a pair (Exp-C and Exp-E). In a Cisco collab deployment scenario, it’s a piece that connects your internal environment to the outer world (Internet, Webex cloud, Partners …).
Formerly known as VCS, these servers are now Virtualized. The underlying software of VCS and Expressway is the same.

On a Cisco VCS/Expressway you can run a service wizard (Introduced in X8.x I guess), choose what Services/features you want to use for VCS/Expressway (MRA, WebRTC, Webex hybrid, b2b …), get/deploy the licenses on it and you can convert VCS to the expressway or vice versa. I think, VCS option will be going away eventually and you will have to stick with Expressway.
Be careful with changes/setup wizard as you will need to first understand your requirement, as some feature/service cant co-exist and you have to get licensing figured out in advance.

Anyway, the whole point of explaining the above details is, VCS and Expressway are the same software so this migration scenario applies to either of them.

I’ve put together this checklist that gives you a quick overview of all the steps you need to do for Cisco Expressway/VCS migration. I will list existing resources available on the Internet and add a few things from my experience.
Before making any changes, please consult with Cisco TAC or do your own research /testing in the lab environment.
Software version – X12.5.9

Please dont consider this as step by step guide and be open to possibility that I may have missed few points:

  • You may not have an identical environment.
  • You may even hit different issues which I didn’t come across.

In short, I am not responsible for any disaster 🙂

Background

IP / Subnet is usually tied to location/DC. So when you change DC, you may have to changes the IP address of the server too. Changing IP address VS hostname ( or both) has a different impact. If possible, try to change a fewer thing at a time.
If you can keep the hostname/fqdn, it’s good ( less work in one change window, no change in certs etc).

A support contract (Smartnet) isn’t tied to the serial number but with a contract number, in the case of virtualized software. So even if you have to rebuild it, support shouldn’t be an issue.
Licensing (Release keys and option keys) are tied to the serial number, but if you have EA with Cisco, you can generate/regenerate/add-remove new serial as you want. There is also an option to resend the license (if you have the same serial but you lost the licences for some reason).
I prefer to go to the self-service portal but you can also contact the licensing team.

https://software.cisco.com/software/csws/ws/platform/home?locale=en_US#

Under the license tab, click on Enterprise Agreements.
You should see your ELA, when you expand you should be able to generate an Option/release key or add a new serial number. You can also get the bootable images at the same portal.

I am not a licensing expert and your contract/access could be different with Cisco, so I let you check with Cisco licensing/Account manager directly. But a lot of people are simply just not aware and opening a case with the licensing team and going back n forth create more dependency, time-consuming. So, I verify my access in advance and if I can, then I prefer the self-service portal and only reach out to the licensing team as last resort.

Changing the IP address (or System setting) doesn’t change the serial number of the Expressway. So, no impact on licensing.

What are the options?

Backup and restore are one of the options. You will need to build a new server in the new DC, take a backup of the existing one and restore it to the new one. Having a difference of serial might not cause the issue but you will need to keep the same licenses, SW version and System settings to restore the backup.
At least at the time of VCS P2V migration, I was able to do it this way, it gave me a warning but it worked. Please do the test or consult with TAC before you do this.

Even if it won’t let you restore from GUI, there may be another way to do the same thing. Backup consist of multiple things, config, certs and maybe a few other things. If you can get the config using the xconfig command and export/re-downloaded the certs, it should be ok. But strictly, consult with Cisco TAC or test it.
The backup file is way smaller than copying the whole VM, and the downtime window, rollback plan is fast 🙂

VMotion is supported and recommended by Cisco TAC. Taking the whole VM over the WAN can take time, and it’s unpredictable when you have limited downtime, so think accordingly.
It also creates a dependency on your compute team (if someone else manages your VMware/server infra) and need lots of checks about compatibility, vCenter etc. You may need to shut down the VM if live migration isn’t supported and if your WAN is choked then you have a much larger downtime window.
And if it fails and you retry, well, practice some meditation 🙂

Why I chose VMotion?

First of all, recommended by Cisco TAC in my case. It also saved me time for not having to build a new VM and do all licensing and testing.
A longer downtime window wasn’t much of an issue in my case. This expressway was just used for b2b and have GEO DNS redundancy for important call flow, so if this pair was down then another GEO expressway will be used in meantime (even if there is a rare b2b meeting on weekend).
I also didn’t have a patience issue, because I had 3 migrations in the same change window as well as some F5 config to do. So while it was happening, I could do some other thing.

High-Level Plan

  • Identify the impact, dependency, downtime window based on your deployment, and the migration plan, and schedule the change window.
  • Discuss/Communicate with all the stakeholders about the dependencies, pre-req, access and actions.
  • Figure out each step for migration, write down the plan including all commands. Write down your test and rollback plan as well.
  • If possible, test the plan in the lab before your change window.

Plan your migration

  • Check all the compatibilities with new HW, ESXi ….
  • Figure out all the dependencies. Discuss and work with your Network, System/DC/AD team. This is not an exhaustive list here but I suggest that you put all the things in writing and keep updating/discussing it with all the stakeholders.
    • Get your new IP address, Def GW, System settings, OS/admin password …. ready.
    • Figure out what will be the new DNS, NTP, LDAP server config will be at the new DC.
    • Access/process to update the DNS records of Expressway with a new IP address if you are keeping the hostname.
      • You will need to update the DNS record for internal as well for the Internet side. Figure out and verify access for both.
      • On the public DNS side, You may need to update A record for your expressway in the change window. Check the TTL as well, if it’s 24 hours (86400) and you might want to reduce it to 3600 for some time, so you don’t have to wait longer?
    • Whether you create a new VM or migrate using vmotion at/to new DC, all the network connectivity/firewall rules within new DC, from new DC to All branch locations and from New to OLD DC ( as some servers still are at old DC). Vlan config on server-side etc.
  • Check all your Infra is using the FQDN for this Expressway, if not then on the other infra side use FQDN instead of IP address. 
  • If you have an Expressway/VCS in the cluster, verify that you have the password handy.
  • Check your config wherever you need a manual change of IP address within config, and be prepared or write it down in advance.
    • You may have a lot of (or few) search rules and transformation rule using regex and matching to your Expressway IP address. You will need to update these with the new IP scheme.
    • On Exp-E, you may be having a local CPL and it needs an update.
    • the list goes on .. you get the idea. Identify, plan and be prepared.
  • Download the .ova file in advance for your version, keep it with you if you need to rebuild it.
  • Verify access for licenses.
  • Plan your change/downtime window carefully keeping all the points/dependencies in mind – DNS, network etc.
  • Open a change, Fix the date and have people ready to support. Yes, you need to make sure you have someone to call or have access for Network troubleshooting or maybe for rebuilding the VM.
  • Write down your test and rollback plan thoroughly.
  • Open a TAC case, ask all the questions. Leave the case open as Sev3 and inform them about the change window to pass onto the Engineer who will be available that day. So if you need any help, you will need to call TAC frontline and raise the Sev to get the Engineer on Webex right away.
  • Important point – figure out how you will change the IP address of the expressway in the new DC. Or if even if you are building an expressway with an old IP scheme in new DC, how will you have access?
    • You can access the Expressway using the console from Vcenter and change the IP using the command. Unlike CUC, when you change the IP of the expressway it doesn’t need access to DNS/NTP.
      But if you are doing a config/restore method then having network access is key.
    • I use the Win server (you can use Linux if you want)and add extra NIC to it. This extra NIC, I put in the same vlan and in the same IP range as this Expressway.
      I use it as an SFTP server, or to access the GUI/ssh/WinSCP of the expressway. I put Exp and this Win server in the same vlan temp and then put Expway back to their actual vlan from vcenter.

      Without going too much in networking basics, understand this.
      • 2 hosts in the same vlan (broadcast domain) can communicate with each other without having to go to the default gateway.
      • If the host is sending traffic to another host, it checks whether the destination IP is in the same subnet or not. If it’s the same subnet then it does the ARP, get the MAC address and send the traffic, Default GW doesn’t matter in this case. If it’s not in the same subnet not then it does the ARP for default GW.
      • The same subnet within a vlan is used, to avoid going to Default GW but it doesn’t mean that technically you can’t have a different subnet in the same vlan 😛
        I don’t recommend it for production but it just for a few minutes.
      • Just remember, within the same broadcast domain. Vlan20 is just an identifier for Switch, in the same broadcast domain.
        You can create a new vlan just for this purpose or use the existing one with the dummy IP address in the same range, doesn’t matter.
      • Be aware of your DC network, If you have microsegmentation or some security rules which restrict this, then you will need to prepare for it.

        Look at this diagram. Win server and VCS in the same range (10.1.1.X/24) but different IP scheme than DC2 (DC2 range is 192.168.x.x) will be reachable to each other. If you need RDP to Win server then add extra NIC in the same scheme as DC2 so you have network connectivity. Or, you could also use a VM console to this Win server.

        This diagram shows the basic network just to give you an idea.

1 day before or in the same change window

  • Have all the credentials ready/handy.
  • Verify all your access.
  • Take a fresh backup. Take the screenshots from GUI. Also, take xconfig and xstatus from Expressway.
  • Give a heads-up to all people involved.

In change window

  • Put your Expressway in maintenance mode(Maintenance –> maintenance mode). It will drain all the existing connections(if any) and stop taking new connection requests.
  • Shutdown the VM, Remove your existing snapshots and make a clone of this VM.
  • Migrate using vMotion.
  • Once migration is done, turn on the VM at the new DC. Change the IP address and system setting.
  • Do other config changes such as search rules, cpl changes.
  • Update the DNS records and verify all the zone, status and if any alarm.
    • Remember, DNS changes/updates take time.
  • Check all your Zones, services, registrations. Do your planned testing.

    If you didn’t do the re-address, then it should be relatively easy. If you also changed the hostname then there could be some difference in steps, instead of updating the DNS records, you will create new and points to new fqdn. You will also need to do other steps such as Cert regeneration, and maybe more.

Post Migration

  • Take a fresh backup, VM snapshot of the new server.
  • Keep your old VM (or VM clone )for a month or so.

Conclusion

Migration of UC servers isn’t an easy task and you may see some different challenges based on your deployment, dependencies and process. So plan it good. Write things down. Communicate with all stakeholders. Have people on standby.
I tend to write down all the background in my post rather than a simple command because it helps you know all the context. So, even if your situation/scenario isn’t exactly the same, you will know my constraint/options and can make an informed decision based on your situation.

I hope this helps. If you find have any feedback, suggestions, please feel free to put the comment or DM me on LinkedIn. Thank you for your time.

Published by Lalit Pamnani

Published

Leave a comment