Migrating Cisco Smart Software Manager On-Prem from Version 8 to 9

Throughout this blog post, I’ll refer to the Smart Software Manager On-Prem as CSSM for brevity.

Upgrading CSSM from Version 8 to 9 requires a migration process, as the inline upgrade is not supported. Specifically, you’ll need to migrate from CSSM On-Prem version 8-202404 to version 9-202406, which runs on AlmaLinux. You’ll need to deploy a new Virtual Machine with CSSM On-Prem version 9-202406.

Here are the high-level steps:

1. Upgrade your current CSSM On-Prem to version 8-202404. This is a pre-requisite for migrating to 9-202406. Refer to my previous blog post for detailed upgrade instructions.
2. Deploy a new Virtual Machine with CSSM On-Prem version 9-202406.
3. Run migration scripts on both the old and new VMs to create and restore the migration backup.

After migration, most of your settings will be copied from the 8-202404 (CentOS) VM to the 9-202406 (AlmaLinux) VM. This includes all product-related and license-related data. However, you’ll need to perform a sync to ensure all licensing information is up to date.

Keep in mind that some data won’t be part of the migration. This includes existing pcap files and any Password Policies you’ve added using the On-Prem console in the 8-202404 VM. You’ll need to reconfigure these manually in the new system.

Now that we understand the migration process, let’s get our hands on the necessary software.

Download the CSSM Software upgrade file

1. Visit Cisco’s software download page.
2. Search for “Smart Software Manager” in the “Select a Product” field.
3. Choose “Smart Software Manager On-Prem.”
4. Under “Latest Release” in the left column, select Release 9-202406.
5. Download the file named SSM_On-Prem-9-202406_AlmaLinux.zip.
6. Once downloaded, unzip the file. You’ll find 10 files inside.
   
   

Now, Let’s get our current CSSM (8-202404) ready for migration.

Preparing the current CSSM (8-202404)

Before starting the migration process, we must prepare our current CSSM.
These steps are intended for a single-node CSSM. If you’re running a High Availability setup, you’ll need to tear down the HA first and perform some additional steps. For HA setups, make sure to consult the SSM_On-Prem_9_Migration_Tool_Quick_Start_Guide.

Back-Up Your Data and Take a Snapshot

Ensure you’ve backed up your database before executing an upgrade. A backup will help if something goes wrong and you need to rebuild. 

To create a database backup:

1. SSH into your CSSM using your admin account.
2. At the prompt (e.g., [admin@abc-cssm01 ~]$), enter the command onprem-console.
3. At the new prompt (>>), enter database_backup.
4. Wait for the confirmation message: “Database successfully backed up to [path/filename].”
5. Use WinSCP to copy the backup from /var/files/backups on CSSM to your PC or admin server.

For other commands in CSSM, refer to the SSM On-Prem 8 Console Guide. As an additional precaution, consider taking a VM snapshot from VCenter.

Copy the Migration Script on the current CSSM (8-202404)

Next, we need to copy the Migration Script to the patches folder (/var/files/patches) on CSSM On-Prem 8-202404.

For security reasons, we can’t directly copy files to the /var/files/patches/ path anymore. So, we’ll use a two-step process:

• Use WinSCP to copy both files to /var/tmp/ on your CSSM.
• SSH into your CSSM.
• If you’re at the >> prompt, type exit to return to the standard prompt.
• Enter sudo -s and provide your admin password.
• Change to the tmp directory: cd /var/tmp/
• Move the files to the patches directory:
  
  mv migrate_configs.sh /var/files/patches/
mv migrate_configs.sh.sha256 /var/files/patches/
  
  
• Verify the files are in /var/files/patches/ using the command line or WinSCP.

Run the Migration Script on the current CSSM (8-202404)

Now, we’re ready to run the migration script.

• If you’re still in the root shell, type exit to return to the admin user.
• Enter onprem-console to access the CSSM console.
• Run the migration script: upgrade patches:migrate_configs.sh
  This may take 5-15 minutes. You’ll see output similar to this:
  
  Migration from CentOS SSM On-Prem to Alma-Linux SSM-OnPrem is a 2 step process.
Step 1: Creation of Migration Backup on CentOS SSM On-Prem(8-202404).
Step 2: Applying Migration Backup on AlmaLinux SSM On-Prem(9-202406).
CURRENTLY EXECUTING STEP 1.
Copying Redis DB file...
Copying Redis DB file: Successful
Copying Docker daemon.json file...
Copying Docker dameon.json file: Successful
Creating Database Backup...
Database successfully backed up to [path/filename]
Creating Database Backup: Successful
Copying Database Backup...
Copying Database Backup: Successful
Migration Backup Succesfully created
Migration Backup at Location [path/filename]
STEP-1 in Migration of CentOS SSM-OnPrem to Alma Linux SSM-OnPrem is now complete.
  

A migration backup file is created in the /var/files/backups folder. You’ll see a “Migration is complete” message with the backup file and path.

Copy the Migration Backup

Now, you need to copy this migration backup file (e.g., onprem-migration-123456789.tar.gz) to a safe external server. You can use WinSCP for this, but… surprise! You might hit a permission error. 🙂

This is due to a known bug (CSCwm06286). The migration script creates the backup with root as the owner and no read/write permissions for others. Here’s an example of what you might see for file permission:

-rw-r-----. 1 root root 1.2M Aug 9 18:47 onprem-migration-123456789.tar.gz

You can fix this permission issue using the workaround steps mentioned in the bug details. Or follow these steps:

1. SSH into your CSSM.
   After the migration script, I had issues with my SSH password. After a few tries, I rebooted the CSSM from the VM console. It worked fine after that. It may be another bug; I’m not sure. If you are not able to log in and get a password error, consider rebooting.
2. Enter sudo -s and provide your admin password.
3. Change to the backups directory: cd /var/files/backups/
4. Run ls -l to see the file permissions.
5. Change the file permissions: sudo chmod o+r <filename>
   For example: sudo chmod o+r onprem-migration-2024080912345.tar.gz
6. Run ls -l again. You should now see:
   
   -rw-r--r--. 1 root root 1.2M Aug 9 18:47 onprem-migration-2024080912345.tar.gz
   
   
7. Now, you should be able to copy the file using WinSCP. Save this file; we’ll need it in the next step.

Gather Network Information

Before we deploy the new CSSM VM, we need to gather some network info. The new AlmaLinux-based VM (CSSM On-Prem Version 9-202406) requires the same network configuration as the old CentOS-based VM (CSSM On-Prem 8-202404).

Here’s what you need to note down:

• Login to the CSSM GUI.
• Go to the admin workspace (at the top right, it opens in a new tab).
• Under Network > General, note the name, IPv4 gateway, and DNS settings.
• Under Network > Network Interface, note the IP address and mask assigned to your CSSM.
• Keep this information and your existing admin password handy.

You can power down the old CSSM On-Prem server now or wait until your new VM is ready (but do it before installation).

With all that done, we’re ready to deploy our new VM!

Deploying a New Virtual Machine with CSSM On-Prem version 9-202406

The system requirements (RAM, CPU, etc.) for the new AlmaLinux machine are the same as the 8-202404 release. Check out the SSM_On-Prem_9_Installation_Guide, pages 10-13 for detailed installation steps.

Here’s a high-level overview:

• Create a new VM with the same specs and VM config (VLAN, etc.) as your existing one.
• Copy the software package (ISO file) to the VMware Datastore and mount the ISO. New CD/DVD Drive: Select DataStore ISO from the list, then select the uploaded ISO and connect at power on.
  Make sure your CentOS-based CSSM On-Prem server (8-202404) is powered down at this stage.
• Start the new VM from vCenter. You’ll see three options – choose the installation one.
• Access the console. In 2-3 minutes, you’ll see the Kickstart Screen. Enter your initial configuration (Hostname, IP address, DNS, and password) and click OK.
  Keep an eye on the console for any issues. I learned this the hard way when I put a space after a comma in the DNS config and had to restart the installation!
• Wait for the installation to complete (about 10-15 minutes). After installation, dismount the ISO image and reboot the server. CSSM On-Prem will boot up automatically on restart.
• Now Copy the migration backup file obtained earlier (CSSM On-Prem server 8-202404) to AlmaLinux-based CSSM On-Prem server (9-202406) to the location “/var/files/backups” using WinSCP.
• Copy the migration script (both files) to the patches directory.
  
  
  The migration backup goes to the backups directory, and the scripts (two files) go to the patches directory. The new version should let you copy the files without permission errors.
  
• Keep the migration backup filename handy.
• SSH into your new CSSM and enter onprem-console
• Run upgrade patches:migrate_configs.sh
  
  It’ll ask for the admin password and show you this:
  
  Migration from CentOS SSM On-Prem to Alma-Linux SSM-OnPrem is a 2 step process.
STEP 1: Creation of Migration Backup on CentOS SSM On-Prem(8-202404).
STEP 2: Applying Migration Backup on AlmaLinux SSM On-Prem(9-202406).
CURRENTLY EXECUTING STEP 2.
You are about to proceed with Migration, Kindly ensure you have taken a snapshot. Do you wish to proceed?(y/n)y
Proceeding with Migration....
  
• When prompted, enter the name of the migration backup file.
• After the migration script finishes successfully, the CSSM On-Prem server will reboot.
• After the server has restarted, verify that the On-Prem server is communicating with your Infrastructure devices (CUCM, CUC etc).
• At this point, take a database backup and a VM snapshot.

I recommend keeping the old CSSM VM for a few months (in a shutdown state), just in case.

A Heads Up on Potential Bug

After migration, you might notice something odd in your GUI – a warning about high RAM usage and incorrect uptime. Don’t panic! This is actually due to a known bug (CSCwk63542).

Here’s what you need to know about the bug.

Symptom: The System Health display in the Admin workspace isn’t updating correctly. You’ll see the Uptime getting stuck at a random value and not increasing. Also, the memory usage appears constant, not changing with the On-prem load. If you want to verify this, you can use the CLI commands “uptime”, “free”, and “top”.

Conditions: This issue affects On-prem version 9-202406 and will likely occur when the server is reloaded.

Workaround: Unfortunately, there isn’t one at the moment. Cisco is aware of the issue, so we’ll have to wait for a fix in a future release.

Remember, while this might look concerning, it doesn’t affect the CSSM’s actual performance. It’s just a display issue in the GUI.

Upgrading to Version 9-202407 (Optional)

If you want to go one step further and upgrade to the latest version 9-202407, follow these steps:

• Download the upgrade files from the Cisco site.
• Unzip the files.
• Using WinSCP, copy these two files to your CSSM’s /var/files/patches/ directory:
  
  SSM_On-Prem-9-202407_Upgrade.sh
SSM_On-Prem-9-202407_Upgrade.sh.sha256
  
  
• SSH into your CSSM and enter onprem-console
• Run the command: upgrade patches:SSM_On-Prem-9-202407_Upgrade.sh
• The upgrade process will run several scripts and should take 5-10 minutes.
• Upon completion, the system should automatically reboot. If it doesn’t, manually reboot with the reboot command.

And that’s it! You’re now on the latest version of CSSM.

Did you find this guide helpful? Did you encounter any different challenges? I’d love to hear your feedback!