Upgrade Procedures

This document includes the procedures for successfully upgrading the following:

  • Evolve IMSWorkX Service Delivery Platform

  • Evolve IMSWorkX Applications

  • PostgreSQL ® Database System


Evolve IMSWorkX Service Delivery Platform

This update can be done successfully without service interruption or impact on system traffic. In general, updates should still be performed at a time when call traffic is minimal or during a maintenance window.

Before Upgrade

Perform the following steps a few days prior to the upgrade window.

  1. Download the Service Delivery Platform update RPM file from the Customer Portal.

  2. Upload the Service Delivery Platform update RPM file to each NIU and AS.

    1. Upload the RPM file to a directory, such as /home, on each server.

    2. Log in to each server via SSH and go to the directory where the RPM file was uploaded.

    3. Run the ls command to make sure that the uploaded RPM appears in the list.

    4. Run the md5sum <filename> command to verify the checksum. If the checksum is not valid, download and upload the RPM file again.

  3. Verify that the primary NIU is active and the secondary NIU is standby by running the ps_hastatus command on the primary NIU. You should see output similar to the following:

Local node status:
Services: Online
Cluster: Running
Network: Up
Heartbeat: Up
Private network address: 111.22.333.444
Private heartbeat address: 111.22.333.444

Partner node status:
Services: Offline
Cluster: Running
Network: Up
Heartbeat: Up
Private network address: 111.22.333.555
Private heartbeat address: 111.22.333.555

  1. Optional. Verify that the web UI of any installed application is showing CDRs with timestamps that are within the configured CDR materialized view cron rate. Note that timestamps in the web UI are based on Universal Time Coordinated (UTC).

During Upgrade

Perform the following steps during the upgrade window.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

    1. Select the service host in the Object panel.

    2. Expand the Logs folder.

    3. Select the PS_SysLog.txt file.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Optional, but recommended. Take a pre-upgrade VM snapshot for each NIU and AS.

  2. Perform a failover to the secondary NIU.

    1. On the primary NIU, run the ps_haswitch utility.

    2. On the primary NIU, run the ps_hastatus utility to ensure that the local node has a state of offline and the partner node has a state of online.

Note

Do not proceed to the next step if the local node has a state of offlining because database replication will be incomplete.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Install the RPM file on the primary NIU, which is now the offline NIU.

    1. Go to the directory where the RPM file was uploaded before the upgrade.

    2. Run the yum install <filename> command.

    3. Verify the install and then wait for the install to complete.

  2. Restart the Service Delivery Platform on the primary NIU.

    1. Run the service XpressPlatform stop command and then verify that it is stopped with the ps_cksys command. You should see output similar to the following:

cat: /var/run/ps_init.pid: No such file or directory
ps_init running as
ps_init not running for root!
But you still have these processes running:
USER GROUP PID PPID ELAPSED COMMAND

  1. Run the service XpressPlatform start command and then verify that it is started the ps_cksys command. You should see output similar to the following:

ps_init running as 7079
The following processes are running under ps_init: 7079 for root
USER GROUP PID PPID ELAPSED COMMAND
USER GROUP PID PPID COMMAND
root root 7079 1 /usr/sipxpress/bin/ps_init
root root 7090 7079 /usr/sipxpress/bin/ps_eventdispatcher -offline -local 172.31.0.169 -partner 172.31.0.168 -virtual 172.31.0.170
root root 7091 7079 /usr/sipxpress/bin/ps_httpd -offline -local 172.31.0.169 -partner 172.31.0.168 -virtual 172.31.0.170
postgres root 7092 7079 python2 /usr/sipxpress/bin/xpressworkx_data_services -offline -local 172.31.0.169 -partner 172.31.0.168 -virtual 172.31.0.170
root root 7093 7079 /sbin/runuser -p -s /bin/sh - tomcat -c /usr/sbin/tomcat_icr start
root root 7094 7079 /bin/bash /usr/sipxpress/bin/xpressworkx_web_services

Note

In the Object panel of Console, the primary NIU should show a yellow icon because it is standby and the secondary NIU should show a green icon because it is active.

Note

If the restart is not successful, try stopping and starting Service Delivery Platform again. If the restart fails again, contact Customer Support.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Perform a failover back to the primary NIU.

    1. On the secondary NIU, run the ps_haswitch utility.

    2. On the secondary NIU, run the ps_hastatus utility to ensure that the local node has a state of offline and the partner node has a state of online.

Note

Do not proceed to the next step if the local node has a state of offlining because database replication will be incomplete.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Install the RPM file on the secondary NIU, which is now the offline NIU.

    1. Go to the directory where the RPM file was uploaded before the upgrade.

    2. Run the yum install <filename> command.

    3. Verify the install and then wait for the install to complete.

  2. Restart the Service Delivery Platform on the secondary NIU.

    1. Run the service XpressPlatform stop command and then verify that it is stopped with the ps_cksys command.

    2. Run the service XpressPlatform start command and then verify that it is started the ps_cksys command.

Note

In the Object panel of Console, the primary NIU should show a green icon because it is active and the secondary NIU should show a yellow icon because it is standby.

  1. Install the RPM file on an AS.

    1. Go to the directory where the RPM file was uploaded before the upgrade.

    2. Run the yum install <filename> command.

    3. Verify the install and then wait for the install to complete.

  2. In Console, individually stop each application that is running on the AS.

    1. In the Object panel, right-click the application that you want to stop.

    2. Click Stop.

    3. Select the Stop Application Gracefully check box to allow each running session to complete its current iteration before stopping.

    4. Click Yes.

    5. Verify that another AS has active application sessions.

  3. Restart the Service Delivery Platform on the AS.

    1. Run the service XpressPlatform stop command and then verify that it is stopped with the ps_cksys command.

    2. Run the service XpressPlatform start command and then verify that it is started the ps_cksys command.

  4. In Console, verify that the applications restarted and have active sessions on the AS.

Note

You may have to manually restart each application if not configured for automatic restart in the ps_master_config.xml file.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Repeat step 12 through step 16 for each additional AS one at a time.

After Upgrade

Perform the following steps after the upgrade is complete.

  1. Verify that the primary NIU is active and the secondary NIU is standby by running the ps_hastatus command on the primary NIU.

  2. Verify access to the web UI by logging in to any installed application using a supported web browser. If you get an error when trying to access the web UI, try running the systemctl restart xpressworkx_web_service command on the primary NIU.

  3. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If the system is not responsive after an upgrade, contact Customer Support.


Evolve IMSWorkX Applications

This update can be done successfully without service interruption or impact on system traffic. In general, updates should still be performed at a time when call traffic is minimal or during a maintenance window.

Before Upgrade

Perform the following steps a few days prior to the upgrade window.

  1. Download the application update RPM file from the Customer Portal.

  2. Upload the application update RPM file to the primary NIU.

    1. Upload the RPM file to a directory, such as /home, on the primary NIU.

    2. Log in to the primary NIU via SSH and go to the directory where the RPM file was uploaded.

    3. Run the ls command to make sure that the uploaded RPM appears in the list.

    4. Run the md5sum <filename> command to verify the checksum. If the checksum is not valid, download and upload the RPM file again.

  3. Verify that the primary NIU is active and the secondary NIU is standby by running the ps_hastatus command on the primary NIU. You should see output similar to the following:

Local node status:
Services: Online
Cluster: Running
Network: Up
Heartbeat: Up
Private network address: 111.22.333.444
Private heartbeat address: 111.22.333.444

Partner node status:
Services: Offline
Cluster: Running
Network: Up
Heartbeat: Up
Private network address: 111.22.333.555
Private heartbeat address: 111.22.333.555

  1. Create a copy of cron by running the crontab -l command on the primary NIU and AS. Copy the displayed output, paste it to a text editor or notepad, and save the file.

  2. Optional. Verify that the web UI is showing CDRs with timestamps that are within the configured CDR materialized view cron rate. Note that timestamps in the web UI are based on Universal Time Coordinated (UTC).

  3. Confirm that the Ansible version on both the primary NIU and the secondary NIU is later than 2.4 and earlier than 2.9 using the ansible --version command.

  4. Confirm that the Application Manager version on the primary NIU is 1.4 or later using the xpressworkx_app_manager version command.

During Upgrade

Perform the following steps during the upgrade window.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

    1. Select the application in the Object panel.

    2. Expand the Logs folder.

    3. Select the PS_SysLog.txt file.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Optional, but recommended. Take a pre-upgrade VM snapshot for each NIU and AS.

  2. Create database backups.

    1. Log in to the primary NIU as root.

    2. Run the sudo -u postgres pg_dump --verbose --clean --create -f /var/opt/xpressworkx/app-manager/database/postgresql/$(date +%Y-%m-%dT%H-%M-%S)-<application short name>.pgdump <application short name> command.

Note

The backup files created by this command will be located in /var/opt/xpressworkx/app-manager/database/postgresql.

  1. Install the RPM file on the primary NIU.

    1. Go to the directory where the RPM file was uploaded.

    2. Run the yum install <filename> command.

    3. Verify the install and then wait for the install to complete.

  2. Perform the application update on the primary NIU.

    1. Run the xpressworkx_app_manager install <application short name> <version number> command.

    2. Wait for the update to complete.

If the update is successful, a list of IP addresses for each AS will be displayed with the status of ok or changed. If the status is failed, contact Customer Support.

Important

Read the release notes for the specific application and version that was upgraded and follow the steps, if any, in the Upgrade Procedure Notes section. These additional steps may include restarting Platform, restarting the application, or just running a command. If additional steps are required on the NIUs, it is important that these steps are completed first before performing any additional steps required on each AS.


If it is required to restart Platform on the NIUs:

  1. Perform a failover to the secondary NIU.

    1. On the primary NIU, run the ps_haswitch utility.

    2. On the primary NIU, run the ps_hastatus utility to ensure that the local node has a state of offline and the partner node has a state of online.

Note

Do not proceed to the next step if the local node has a state of offlining because database replication will be incomplete.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Restart the Service Delivery Platform on the primary NIU.

    1. Run the service XpressPlatform stop command and then verify that it is stopped with the ps_cksys command. You should see output similar to the following:

cat: /var/run/ps_init.pid: No such file or directory
ps_init running as
ps_init not running for root!
But you still have these processes running:
USER GROUP PID PPID ELAPSED COMMAND

  1. Run the service XpressPlatform start command and then verify that it is started the ps_cksys command. You should see output similar to the following:

ps_init running as 7079
The following processes are running under ps_init: 7079 for root
USER GROUP PID PPID ELAPSED COMMAND
USER GROUP PID PPID COMMAND
root root 7079 1 /usr/sipxpress/bin/ps_init
root root 7090 7079 /usr/sipxpress/bin/ps_eventdispatcher -offline -local 172.31.0.169 -partner 172.31.0.168 -virtual 172.31.0.170
root root 7091 7079 /usr/sipxpress/bin/ps_httpd -offline -local 172.31.0.169 -partner 172.31.0.168 -virtual 172.31.0.170
postgres root 7092 7079 python2 /usr/sipxpress/bin/xpressworkx_data_services -offline -local 172.31.0.169 -partner 172.31.0.168 -virtual 172.31.0.170
root root 7093 7079 /sbin/runuser -p -s /bin/sh - tomcat -c /usr/sbin/tomcat_icr start
root root 7094 7079 /bin/bash /usr/sipxpress/bin/xpressworkx_web_services

Note

In the Object panel of Console, the primary NIU should show a yellow icon because it is standby and the secondary NIU should show a green icon because it is active.

Note

If the restart is not successful, try stopping and starting Service Delivery Platform again. If the restart fails again, contact Customer Support.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Perform a failover back to the primary NIU.

    1. On the secondary NIU, run the ps_haswitch utility.

    2. On the secondary NIU, run the ps_hastatus utility to ensure that the local node has a state of offline and the partner node has a state of online.

Note

Do not proceed to the next step if the local node has a state of offlining because database replication will be incomplete.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Restart the Service Delivery Platform on the secondary NIU.

    1. Run the service XpressPlatform stop command and then verify that it is stopped with the ps_cksys command.

    2. Run the service XpressPlatform start command and then verify that it is started the ps_cksys command.

Note

In the Object panel of Console, the primary NIU should show a green icon because it is active and the secondary NIU should show a yellow icon because it is standby.


If it is not required to restart Platform on the NIUs:

  1. Run the specified command or commands on both the primary NIU and the secondary NIU.


If it is required to restart Platform on each AS:

  1. In Console, individually stop each application that is running on the AS.

    1. In the Object panel, right-click the application that you want to stop.

    2. Click Stop.

    3. Select the Stop Application Gracefully check box to allow each running session to complete its current iteration before stopping.

    4. Click Yes.

    5. Verify that another AS has active application sessions.

  2. Restart the Service Delivery Platform on the AS.

    1. Run the service XpressPlatform stop command and then verify that it is stopped with the ps_cksys command.

    2. Run the service XpressPlatform start command and then verify that it is started the ps_cksys command.

  3. In Console, verify that the applications restarted and have active sessions on the AS.

Note

You may have to manually restart each application if not configured for automatic restart in the ps_master_config.xml file.

  1. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Repeat step 1 through step 4 for each additional AS one at a time.


If it is required to restart only the application on each AS:

  1. In Console, restart the upgraded application that is running on the AS.

    1. In the Object panel, right-click the application that you want to restart.

    2. Click Restart.

    3. Select the Stop Gracefully and Restart on a new XTML server check box to allow each running session to complete its current iteration before stopping.

    4. Click Restart.

  2. In Console, verify that the application restarted and has active sessions on the AS.

  3. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

Note

If calls are not being completed successfully, do not proceed to the next step.

  1. Repeat step 1 through step 3 for each additional AS one at a time.

After Upgrade

Perform the following steps after the upgrade is complete.

  1. Verify that the primary NIU is active and the secondary NIU is standby by running the ps_hastatus command on the primary NIU.

  2. Update cron on each NIU and AS.

    1. Run the crontab -e command.

    2. Update cron to match the copy of cron output that was saved prior to upgrading.

  3. Verify access to the web UI by logging in to the updated application using a supported web browser. If you get an error when trying to access the web UI, try running the systemctl restart xpressworkx_web_service command on the primary NIU.

  4. Make tests calls or view the PS_SysLog.txt file in Console to ensure successful call traffic.

  5. Optional. Verify that the web UI is showing CDRs.

Note

CDRs will not appear in the web UI until the xpressworkx-cdr-synchronize cron job has run for that application.

Note

If the system is not responsive after an upgrade, contact Customer Support.

Rolling Back the Installation

Should something go wrong with the upgrade and it is required to rollback the installation using the database backup, perform the following steps.

Note

This will be service disrupting, if the failed upgrade hasn’t already affected service.

  1. Run the service XpressPlatform stop command on both NIUs.

  2. On the Primary NIU, run the sudo -u postgres /usr/sipxpress/bin/xpressworkx_data_services command to start the database separately from the rest of the Platform.

Note

The database will run continuously in the window the command is entered into, so another window will be required for the following step.

  1. On the Primary NIU, run the sudo -u postgres psql -f /var/opt/xpressworkx/app-manager/database/postgresql/<backup>.pgdump command, where <backup.pgdump is the name of the database backup that was saved previously.

  2. In the window from step 2, use ctrl+C to stop the data services.

  3. Run the service XpressPlatform start command on the Primary NIU. Once the Platform is running, run the command on the Secondary NIU.

  4. Follow the steps for installation, using the previously installed version of the application.

Note

It may be required to run the yum downgrade command using the RPM for the previously installed version in order for that version to be installed using xpressworkx_app_manager.


PostgreSQL Database System

After upgrading the Evolve IMSWorkX application as normal, perform the following steps to complete the PostgreSQL upgrade to version 12. This upgrade is service affecting, so it is recommended to complete it during a maintenance window. New installations will automatically use PostgreSQL version 12.

  1. Verify that the system is operating normally and that the primary NIU is online with the secondary NIU replicating.

  2. Run the service XpressPlatform stop command on the secondary NIU to stop the Service Delivery Platform.

  3. Run the xpress_pg_upgrade command on the primary NIU to stop the Service Delivery Platform and begin the database upgrade.

Note

The amount of time that it will take for the upgrade to complete will vary based on database size, but allow for 30 minutes.

Note

An SQL dump will be written automatically in case of a failure after upgrading that requires a rollback. This requires disk space for a full database dump in /var/lib/pgsql/imsworkx_dump.

  1. After the upgrade is complete, run the service XpressPlatform start command on the primary NIU to start the Service Delivery Platform.

Note

An application restart may be required. Read the release notes for the specific application and version that was upgraded and follow the steps, if any, in the Upgrade Procedure Notes section.

  1. Verify that the primary NIU is online by running the ps_hastatus command, that logging from the xpressworkx_data_service shows the database as online but not replicating yet, and that you can sign in to the web UI and make changes without errors.

Note

If the system is not responsive at this point, contact Customer Support for assistance with performing a system rollback.

  1. Run the service XpressPlatform start on the secondary NIU to start the Service Delivery Platform. The database version will be detected as 12 and the database will be resynchronized for replication.

  2. Verify that the secondary NIU is offline by running the ps_hastatus command and that logging from the xpressworkx_data_service shows the database as offline and replicating from the primary NIU.

Note

If database replication is not occurring successfully, contact Customer Support.


Perform a System Rollback

If the database upgrade is not successful, you may need to perform the following steps to terminate the upgrade and rollback the system before the secondary NIU replicates the new version of the database from the primary NIU. Contact Customer Support for assistance with performing a system rollback.

  1. Run the service !XpressPlatform stop command on the primary NIU to stop the Service Delivery Platform.

  2. Run the service !XpressPlatform start command on the secondary NIU to start the Service Delivery Platform.

  3. Verify that the secondary NIU is online with PostgreSQL version 9.4 by running the ps_hastatus command and that all application functionality is available.

  4. Run the service !XpressPlatform start command on the primary NIU to start the Service Delivery Platform. The database version will be detected as 9.4 and the database will be resynchronized for replication.

  5. Verify that the primary NIU is offline with by running the ps_hastatus command.

  6. Optional. Perform a failover back to the primary NIU once the system appears stable.