Server (VM) Configuring and Monitoring Recommendations¶

Minimum System Guidelines¶

• Each AS should have a minimum dual core CPU.

• Each AS should have a minimum of 4G memory.

• Each AS should have a minimum of a 2Ghz processor.

• Each AS should have a minimum of 10G file system storage.

• Each NIU should have enough file system storage for the databases of all applications.

• Each AS should have CentOS 7.x or Red Hat Enterprise Linux 7.x operating system.

• Each NIU should have HTTP and HTTPS as the only publicly accessible ports. This is only necessary if the provisioning interface is public.

Network Guidelines¶

• Each NIU needs passwordless SSH access for the root user to all machines.

• Each AS should have a host name that is locally resolvable.

• Data, OAM, RTP, SIGTRAN, and SIP should all be on different segmented (subnetted) networks.

• Data, OAM, RTP, SIGTRAN, and SIP should all be in different firewall realms. This primarily relates to firewalld.

• Segmented networks should be octet bound.

• The /var directory should be separately partitioned.

• End-to-end, one-way latency does not exceed 100ms.

• Jitter does not exceed 20ms.

• Packet loss is less than 0.5%.

• Each subnet should be at least Gigabit Ethernet.

• Each subnet should be privately addressed or privately accessible.

• Each interface should be physically redundant.

• Security-Enhanced Linux (SELinux) should be set as Enforcing.

• Use Quality of Service (QoS) to grant VoIP traffic the highest priority.

• An MPLS-based network is recommended.

• Net-SNMP should be enabled with Agent X connections allowed.

For Clustered Pairs

• There must be a private IP address for communication between both NIUs. This must be a layer 2 link.

• All IP addresses on the secondary NIU must be consecutive from the primary NIU addresses. For example, if the primary NIU address is x.y.z.a, the secondary MUST be configured as x.y.z.(a+1).

• The host name “niu” must resolve to the SIP IP HA address.

Virtual Machine Cloning Guidelines¶

It is not recommended to clone VMs as a shortcut during or after installation.

If you do choose to use cloned machines, the clone should be taken when only the following is installed:

• Operating system

• All dependencies for the Platform and applications that you will be installing

• Service Delivery Platform

• Application Server audio files

The clone should not be taken after any of the following:

• Making changes to any configuration files

• Installing the Application Manager

• Installing any other applications

After a machine has been cloned, complete the following steps:

1. Edit configurations files on the new machine. For more information, see Application Server Configuration.

2. Add the new machine to the cluster using the xpressworkx-app-manager add role address command on the NIU. For more information, see High Availability Services.

3. If the cloned machine is added to a cluster that already has applications installed, run the xpressworkx-app-manager synchronize command on the NIU.

Rsyslog Configuration Guidelines¶

While not necessary for proper functionality of the system, enabling network socket listening in rsyslog can result in the production of useful logging across applications that are using the Database Access Component version 2.0 or later.

To enable network socket listening, you will need to uncomment the following lines in /etc/rsyslog.conf on the NIU and each AS:

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

PostgreSQL Autovacuum Guidelines¶

The following are general purpose recommendations to configure more aggressive thresholds for postgres cleanup and analytics activities for optimal performance. The default values are conservative to deal with the wide range of hardware capabilities, but contemporary server hardware should benefit from these changes.

As statements to be applied to a system:

ALTER SYSTEM SET autovacuum_vacuum_threshold = 0;
ALTER SYSTEM SET autovacuum_vacuum_scale_factor = 0.01;
ALTER SYSTEM SET autovacuum_analyze_threshold = 0;
ALTER SYSTEM SET autovacuum_analyze_scale_factor = 0.01;
ALTER SYSTEM SET autovacuum_vacuum_cost_limit = 1000;

Followed by SELECT pg_reload_conf(); to apply the changes.

As items to directly set in the template file:

These changes must be made to the /etc/opt/xpressworkx/app-manager/database/postgresql.conf.j2 template file on each NIU in the system.

#------------------------------------------------------------------------------
# AUTOVACUUM PARAMETERS
#------------------------------------------------------------------------------
autovacuum = on                 # Enable autovacuum subprocess?  'on'
                                        # requires track_counts to also be on.
#log_autovacuum_min_duration = -1       # -1 disables, 0 logs all actions and
                                        # their durations, > 0 logs only
                                        # actions running at least this number
                                        # of milliseconds.
autovacuum_max_workers = 3              # max number of autovacuum subprocesses
                                        # (change requires restart)
autovacuum_naptime = 1min               # time between autovacuum runs
autovacuum_vacuum_threshold = 0         # min number of row updates before
                                        # vacuum
autovacuum_analyze_threshold = 0        # min number of row updates before
                                        # analyze
#autovacuum_vacuum_scale_factor = 0.01  # fraction of table size before vacuum
#autovacuum_analyze_scale_factor = 0.01 # fraction of table size before analyze
autovacuum_freeze_max_age = 200000000   # maximum XID age before forced vacuum
                                        # (change requires restart)
#autovacuum_vacuum_cost_delay = 20ms    # default vacuum cost delay for
                                        # autovacuum, in milliseconds;
                                        # -1 means use vacuum_cost_delay
autovacuum_vacuum_cost_limit = 1000     # default vacuum cost limit for
                                        # autovacuum, -1 means use
                                        # vacuum_cost_limit

Followed by restarting the Service Delivery Platform to apply the changes.

Additional Guidelines¶

• Each user should have a unique user name and password to access the system.

• No one should log in as root to the NIU or AS.

• Users requiring privileged access should acquire access using the sudo command.

• Logs should be rotated, including /var/log/xpress.log and /var/log/xpressworkx/*.

• For applications using the Web component, /var/log/xpressworkx/app-manager/web/access_log and /var/log/xpressworkx/app-manager/web/error_log will also need to be rotated. To reload or recreate these log files after rotating them, run the systemctl reload xpressworkx_web_service command.

• Log files should be monitored.

System Monitoring Guidelines¶

The following commands can be used to monitor the overall health of the system, including application database health.

ps ax | grep 'wal\sreceiver'

This command checks if the offline/offlining node is correctly configured to stream transactions.

ps -ax | grep 'rsync' | grep -v 'grep'

This command shows if there is a running rsync process, which would be expected during replication establishment in the offlining state when a postgres process is not yet running on that node.

stat /var/opt/xpressworkx/app-manager/database/postgresql/data/backup_label

This command shows if the backup_label file is present, indicating the online node is in a backup state, which should correspond with a running rsync process to establish replication on the offlining node.

SQL commands to monitor status:

All of these can be invoked directly from a command prompt such as sudo -u postgres psql -c "SELECT 1;" where the quoted string is the SQL statement. It may be easier, especially when needing to use quotes or complex expressions, to directly run the PostgreSQL interactive tool and copy/paste the commands into it.

SELECT pg_is_in_recovery();

This query will return t if the node is replicating from a partner or f if it is not.

SELECT pg_is_in_backup();

This query returns t if the node is in the backup state or f if it is not.

SELECT * FROM pg_replication_slots;

This query will display a table of replication slots, for which there is expected to a single slot on the online node when replication is established and working correctly, which would be marked active. The offline/offlining node should not have any replication slots.

SELECT datname, pg_size_pretty(pg_database_size(datname)) as size
FROM pg_database;

This query returns a result of the database names and sizes on disk.

SELECT
        CASE
                WHEN pg_last_xlog_receive_location() = pg_last_xlog_replay_location() THEN 0
                ELSE EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))::INTEGER
        END
        AS replication_lag;

This query displays a replication lag value in seconds, showing if the offline node is behind in applying replicated transactions.

Dedicated Ports¶

The following ports are used by processes and should allow traffic through.

Disclaimer¶

This document presents information for users of Evolve Cellular Inc. (“Evolve”) products. Although the information contained within this document is considered accurate and characteristic of the subject product, Evolve reserves the right to make changes to this document and any products described herein to improve reliability, functionality, or design.

Evolve does not assume any liability arising out of the application or use of any product or service described herein. No part of this document may be copied or reproduced in any form or by any means without the prior permission of Evolve.

These commodities, technology or software may only be exported from the United States in accordance with the Export Administration Regulations and other applicable laws and regulations. Diversion contrary to U.S. law is prohibited.

Trademarks¶

This document is the sole property of Evolve Cellular Inc.

Evolve is a registered trademark of Evolve Cellular Inc.

Linux is the registered trademark of Linus Torvalds in the United States and other countries.

Oracle, Oracle Database Standard Edition, Oracle Database Standard Edition One, and Java are registered trademarks of Oracle and/or its affiliates.

Festival Speech Synthesis System is developed and maintained by the Center for Speech Technology Research at the University of Edinburgh.

Other names may be trademarks of their respective owners.