Web Monitoring

This is a service that allows the operator to monitor and manage X-PUSH in a web browser.

The server that monitors the XPush Instance information and status, and integrates the user and message management

Settings

Configuration

Environment Configuration

Open the $xpush-3.0.x/bin/set_xpush.bat file to configure JAVA_HOME and XPUSH_HOME.

set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_XX
set XPUSH_HOME=D:\xpush-3.x.x

Configuration Files

Configuration Files Used in Monitor Server

They are contained in the $XPUSH_HOME/conf directory.

monitor_conf.properties
monitor_jdbc.properties
monitor_user.properties
monitor_logback.xml

monitor_conf.properties

Default Settings of Monitor Server

Item	Description
monitor.agentListen.host	Address of monitor Server to which xpush will access 192.168.1.1
monitor.agentListen.port	Port of monitor Server to which xpush will access 8099
monitor.httpd.host	Address of monitor Server to which the client will access 192.168.1.1
monitor.httpd.port	Port of the monitor Server to which the client will access 8088

monitor_jdbc.properties

Database Settings of Monitor Server

Item	Description
spring.datasource.url	URL of Database to access
spring.datasource.username	User ID to access Database
spring.datasource.password	User Password to access Database
spring.datasource.driver-class-name	JDBC Class Name to access

monitor_user.properties

Monitor Server Access User Settings

Item	Description
user.id	Account ID to access
user.password	The password of the account
xpushInstance.id	The ID of the xpush server
xpushInstance.password	The password of the xpush ID

monitor_logback.xml

Log Settings of Monitor Server

For the log settings of the monitor server, use logback

http://logback.qos.ch/manual/configuration.html

Operation

Run with the $XPUSH_HOME/bin/monitor_startup.sh/monitor_startup.bat command

Function

login

Access to Monitoring Web Console

Refer to the monitor_user.properties configuration file to log in to the Monitor server.

Setting

xpush instance
System Property
Health check
Mail

XPush Instance

You can add XPush Instance using the " + " button.

System Property
항목	설명
Server Name	XPUSH Server alias
X-PUSH IP	XPUSH Server IP Address
X-PUSH PORT	XPUSH Server PORT xpush instance xpush_config.xml 의 SecureShellService PORT

$XPUSH_HOME/conf/xpush_config.xml

<service name="SecureShellService" ...>
    <attribute name="ServerBindAddress">0.0.0.0</attribute>
    <attribute name="Port">10089</attribute>
    ...
</service>

If XPush Instance is added, then you can access the monitor server

System Property
Item	Description
	Access the X-PUSH instance to the monitor server
	Stop the monitor server access
	Edit
	Delete

System Property

Item	Description
Receive interval (second)	Interval to receive information from the xpush instance (unit: second)
ConnectionRetry Count	The number of times the xpush instance attempts to access
Connection Retry Timeout(second)	ID of the xpush server
Default row Count	The grid row count shown in Monitor UI

Health check

Item	Description
Division	L4, L7 selection
Interval Time (second)	Interval to check
Status	Health check status

Mail

Item	Description
SMTP Server	SMTP server address
SMTP PORT	SMTP server port
SMTP Password	Outgoing user password
USER Mail	Outgoing user mail
To Mail	Incoming user mail
Status	Mail usage status

Summary

Integrated Dashboard screen
Screen for each XPush Instance

Integrated Dashboard Screen

Item	Description
1	PUSH server in operation
2	Accessing user Shows the number and distribution of users currently accessing. Shows the total number of users and the number of users accessing by servers.
3	Health Check Outputs the status of PUSH. (OK/Fail) Checks at regular intervals based on L4 / L7 Layer. - L4(Transport): Checks normality by TCP access - L7(Application): Checks normality by whether the actual message is sent or not
4	System information Shows CPU, Memory, Network (Input/Output) in the line chart.
5	Queue status Shows the number of messages to be processed in Queue.

Screen for Each XPush Instance

Item	Description
1	Outputs the system information of XPush Instance. - Information such as OS, JDK, Memory, IP, PORT
2	Shows the number of users currently accessing.
3	Health Check Outputs the status of PUSH. (OK/Fail) Checks at regular intervals based on L4 / L7 Layer. - L4(Transport): Checks normality by TCP access - L7(Application): Checks normality by whether the actual message is sent or not
4	System information Shows CPU, Memory, Network (Input/Output) in the line chart.
5	Queue status Shows the number of messages to be processed in Queue.

User Mananger

User Information Can Be Searched.

Online users
DB user information
DB mobile user information

Item	Description
1	Search condition If a value is not entered, then search all If a value is entered, then search by AND condition
2	Online user information - Shows the information of users currently accessing. - Able to send messages to the selected users.
3	DB user information - Shows the TOPIC information saved in DB. - Able to create and change the TOPIC information. - Able to send a message to the selected TOPIC.
4	DB mobile user information - Shows the Mobile Device information of the users stored in DB. - Able to change the Device information.

Message Mananger

Search Available by Message/User/Mobile Notification

Message
User
Notification

Message

Search is Performed by Message.

Item	Description
1	Search condition If a value is not entered, then search all If a value is entered, then search by AND condition
2	Message list for the condition
3	User message information for the Row selected in 2
4	Notification information for the Row selected in 2

User

Search is Performed by User-specific Message.

Item	Description
1	Search condition If a value is not entered, then search all If a value is entered, then search by AND condition
2	User message list for the condition
3	Message information for the Row selected in 2
4	Notification information for the Row selected in 2

Notification

Search is Performed by Mobile Message.

Item	Description
1	Search condition If a value is not entered, then search all If a value is entered, then search by AND condition
2	Notification list for the condition
3	Message information for the Row selected in 2
4	User message information for the Row selected in 2

Send

Message Transmission Function (Provider)

Messages can be sent directly from the web management console

Item	Description
Project ID	ProjectID to transmit
Topic Type	Topic Type to transmit
Topic ID	Topic ID to transmit
Available Period	Message validity period (unit: day)
PUSH, RELI	Select message type PUSH: General message RELI: Reliability message
Message	Message to send You can add the number of messages by using the "+" button.

Console Monitoring

X-PUSH provides a script for monitoring server operation.

show_status.bat/sh: Outputs current server status.
show_trace.bat/sh: Outputs contents from the console logger.

show status

This script outputs the server status. Go to the "$XPUSH_HOME/bin" directory and run "show_status.bat/sh" The following four parameters are required when running the script.

Parameter	Description
host	X-PUSH Server IP address.
port	X-PUSH Server monitor port.
id	Authentication account for the monitor.
password	Authentication password for the monitor.

show_status.bat localhost 10083 tobesoft xpush
./show_status.sh localhost 10083 tobesoft xpush

The following is a status example.

.module Main X-PUSH Server

RUNTIME

Item	Description
Startup Time	Server startup time.
Current Time	Current time.
Status Reset	Status reset time.

MEMORY

Item	Description
Max	Maximum available memory for JVM.
Total Heap	Maximum heap memory.
Used Heap	Used heap memory.
Free Heap	Free heap memory.

An X-PUSH Server does not operate normally due to memory shortage when the free heap memory reaches zero.

QUEUE

The queue value is a statistic collected at 0.1-second intervals. A total of 10,000 are stored. The most recent 1,000 seconds (approx. 16 minutes) of stats are provided.

Item	Description
Message Size in Queue	The number of messages currently waiting to be processed in the X-PUSH main queue
Average Messages in Queue	Average number of messages in the queue. If this value is greater than 1, a delay is occurring in pushing messages.
Waiting Ratio	Percentage of time that messages are waiting in the queue relative to the total server runtime.

Item

Description

Message Size in Queue

The number of messages currently waiting to be

processed in the X-PUSH main queue

Average Messages in Queue

Average number of messages in the queue. If this value is greater than 1, a delay is occurring in pushing messages.

Waiting Ratio

Percentage of time that messages are waiting in the queue relative to the total server runtime.

Message Size in Queue item being 0 or more and the number continuously being increased. X-PUSH messaging sending being delayed.

More supply than message processing

DB processing being delayed

Client Servant Container

Item	Description
Container Count	Number of containers managing servants.
Total Client Count	Total number of currently-connected clients.
Total Push Count	Total number of messages pushed to the client after starting the server. Sum of the values pushed by each container.
Average Tps	Average per-second number of messages transmitted by the server. The average tps value for all containers.

Container-related delays and tps-related values represent the most recent 10,000 message cases. When individual messages are transmitted to containers and processed, each delay is collected and the respective statistics are calculated.

The items provided per container are as follows:

Item	Description
#client	Number of clients managed by the container.
#pushed	Total number of messages pushed by the container.
#in Q:	Number of messages currently in the container queue.
avg delay	Average standby time for messages in the container queue. The unit is in seconds.
max delay	Maximum standby time for messages in the container queue. The unit is in seconds.
push tps	Per-second number of messages pushed by the container. The value is obtained from the number of messages processed by the container and the time. The number of messages processed by the container is independent from the number of messages actually pushed to each client. For example, certain messages are transmitted to a small number of clients while other messages are transmitted to all clients.

#in Q item being 0 or more and the number continuously being increased. X-PUSH message sending being delayed.

More supply than message processing

Socket Push Message Provider

Item	Description
Connected Provider Count	Number of connected message providers.

Per Servant

Each message provider's IP address, port, connected accounts, and the number of messages provided are displayed.

Provider Message Queue

Item	Description
Push Count	Number of messages pushed by using the X-PUSH Server’s internal queue.

Repository Service

QUEUE

It means the number of messages currently waiting / queue size.

QUEUE
Item	Description
Provide Message Queue Size	Queue with supplied reliable message waiting
Response Message Queue Size	Queue with confirmation response message waiting
Request Message Queue Size	Queue with unreceived sending request message waiting
Topic Message Queue Size	Queue with topic registration request message waiting

#Queue Size item being 0 or more and the number continuously being increased. X-PUSH messaging sending being delayed.

DB processing being delayed

ex) Provide Message Queue Size = 5000 / 20000 being continued or being continuously increased.

Message Process Time(milisec)

It means the time required to process a message by each Queue (Provide, Response, Request, Topic) and the status of the operating thread that processes the corresponding message.

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
#	Activation operating thread – code level / activation operating thread
Avg	Average time required
Min	Minimum time required
Max	Maximum time required

DB Connection Pool

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
Active	Number of Connection in use
Max	Maximum number of Connection
Idle	Number of Connection waiting

Notification Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
in Q	Number of messages accumulated in the corresponding queue

Notification Result Update Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
in Q	Number of messages accumulated in the corresponding queue

Apns Notification Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
in Q	Number of messages accumulated in the corresponding queue

FCM Notification Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
in Q	Number of messages accumulated in the corresponding queue

ApnsNotifier Service

Message Process Time(milisec)

It means the message processing time of the ApnsQueue’s operating thread and Provider sending data to Apns, and the status of the operating thread that processes the corresponding message.

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
#	Activation operating thread – code level / activation operating thread
Avg	Average time required
Min	Minimum time required
Max	Maximum time required

FcmNotifier Service

Message Process Time(milisec)

It means the message processing time of FcmQueue’s operating thread and Provider sending data to FCM, and the status of the operating thread that processes the corresponding message.

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
#	Activation operating thread – code level / activation operating thread
Avg	Average time required
Min	Minimum time required
Max	Maximum time required

Etc Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태
Item	Description
in Q	Number of messages accumulated in the corresponding queue

JMX

JConsole Connection

Windows environment setup

Run %JAVA_HOME%\bin\jconsole

Enter the IP address and port in the [Remote] tab, and click the [Connect] button.

A connection to JMX is not made if the X-PUSH Server is installed on Linux.

X-PUSH is bound to 0.0.0.0 and run for both Linux and Windows. For Windows, however, the routing information is registered in the routing table under its own IP address for access via the corresponding address. This is not possible for Linux, because the same information is not registered in the routing table by default.

Method 1: Assign the actual IP address instead of "0.0.0.0" under the "ServerBindAddress" section in xpush_conf.xml.

<service name="Publisher">
	...
	<attribute name="ServerBindAddress">192.168.0.1</attribute>
	...
</service>

Method 2: Assign the IP address to the "/etc/hosts" file.

192.168.0.1 localhost.localdomain localhost
::1 localhost6.localdomain localhost6

Item

X-PUSH MBean

MBean Information List	ObjectName	Description
ConnectionManagerService	X-PUSH:name=ConnectionManagerService	Information for currently-connected users.
NodeManagerService	X-PUSH:name=NodeManagerService	X-PUSH Node information.
PublisherService	X-PUSH:name=PublisherService	Publisher information.
RepositoryService	X-PUSH:name=RepositoryService	Repository information.
System	X-PUSH:name=System	System information.
Container	X-PUSH:type=container, name=TobePublishContainerService0 X-PUSH:type=container, name=TobePublishContainerService1 ......	Container information.
Notification	X-PUSH:type=notification, name=ApnsNotifierService	APNs and GCM information.
Provider	X-PUSH:type=notification, name=GcmNotifierService	Provider information.

ConnectionManagerService

MBean Information List	Description
ConnectUserCount	Number of connected clients.

MBean 정보 목록	설명
ConnectUserCount	Number of connected clients

NodeManagerService

MBean Information List	Description
MyInetInfo	X-PUSH node’s IP and port information.
AnotherNodeCount	Number of nodes.
PublishCount	Number published to other nodes.
AnotherNodeList	Connected node information.

PublisherService

MBean Information List	Description
ClientNumber	Number of connected clients.
PublishAddress	Publish IP address.
ContainerCount	Number of containers.
PublishPort	Publish port.
XpushVersion	X-PUSH version.
DateStart	X-PUSH start date/time.
DateReset	X-PUSH data reset time.
DateCurrent	Current time.

RepositoryService

MBean Information List	Description
UpdateMsgQueueMaxCapacity	Queue Size (Receipt confirmed message queue)
SelectMsgQueueMaxCapacity	Queue Size (Unreceived message queue)
InsertMsgQueueMaxCapacity	Queue Size (Reliable message queue)
SelectMsgQueueState	Queue status (0: open, 1: blocked)
UpdateMsgQueueState	Queue status (0: open, 1: blocked)
InsertMsgQueueState	Queue status (0: open, 1: blocked)
DbcpConnectionPoolActiveCount	Number of active pools.
DbcpConnectionPoolIdleCount	Number of remaining idle pools.
DbcpConnectionPoolMaxCount	Maximum number of pools.
SelectMsgQueueSize	The number of messages currently in the queue. (Unreceived message queue)
InsertMsgQueueSize	The number of messages currently in the queue. (Reliable message queue)
UpdateMsgQueueSize	The number of messages currently in the queue. (Receipt confirmed message queue)
ThreadPoolMax	Number of thread pools.
FailCount	Number of database processing failures.
FailCountLimit	FailCount limit.
TimeRetryDBPeriodMillis	DB check interval (milliseconds) when the queue is blocked.

System

MBean Information List	Description
CpuIdle	System CPU Idle
CpuSys	System CPU usage.
CpuUser	User CPU usage.
JavaHome	JAVA_HOME directory
MemoryCommittedHeap	X-PUSH maximum heap size
MemoryFreeHeap	X-PUSH free heap size
MemoryMaxHeap	X-PUSH maximum heap size
MemoryUsedHeap	X-PUSH used heap Size
ProcessCpu	CPU usage
ProcessName	Process name
ProcessPid	Process ID
ProcessRss	Actual memory usage
ProcessSize	X-PUSH process size
SystemMemoryTotal	Total system memory
SystemMemoryUsed	System memory usage.
SystemMemoryFree	Free system memory.
SysmemNetworkTotalrx	Network read traffic.
SystemNetworkTotaltx	Network write traffic.
SystemSwapFree	Free swap amount.
SystemSwapTotal	Total swap amount.
SystemSwapUsed	Swap usage.
XpushHome	X-PUSH Home directory.

Container

MBean Information List	Description
AverageDelay	Average standby time for messages in the container queue. The unit is in seconds.
CountInQueue	Number of messages in the container queue.
MaxCapacity	Maximum number of messages that can be held in a container queue.
MaxDelay	Maximum standby time for messages awaiting push in a container queue. The unit is in seconds.
MaxServantNum	Maximum number of servants.
PushCount	Number of pushed general messages.
PushTps	Per-second number of messages pushed from a container.
QueueAverageDelay	Average delay time.
PushTps	Per-second number of messages pushed from a container.
QueueAverageDelay	Average delay time.
QueueCurrentCount	Number of messages in queue.
QueueMaxDelay	Maximum delay time.
QueueMaxSize	Maximum number in queue.
ReliCount	Number of pushed reliability messages.
ServantNum	Number of servants.
ThreadPoolCurrentSize	Current thread pool size.
ThreadPoolSetMaxSize	Maximum thread pool size.
ThreadPoolSetMinSize	Minimum thread pool size.

Notification

ApnsNotifierService

MBean Information List	Description
FailureCount	Failure count.
KeystorePath	APNs certificate location.
Retries	Number of retry attempts for failed transmissions.
SuccessCount	Number of successful transmissions.
TotalSendCount	Total number of transmissions.

GcmNotifierService

MBean Information List	Description
Apikey	Authentication key.
FailureCount	Failure count.
Retries	Number of retry attempts for failed transmissions.
SuccessCount	Number of successful transmissions.
TotalSendCount	Total number of transmissions.

Provider

MBean Information List	Description
ConnectProviderList	Connected provider information. (IP, port)
Port	Provider port.
ProviderAddress	Provider IP address.
ProviderNumber	Number of connected providers.

Identifying Resource Shortages

Resource shortages for server operation can be identified through the values from show status.

Memory Shortage

Normal operation is not possible when the free heap memory is low. If the amount of free heap memory is less than 10% of the total available, the memory must be expanded, and the settings must be changed. Modify the -Xms and -Xmx values in the "startup.bat / startup.sh" for the memory settings.

startup.bat
set JAVA_OPTS=-Xms1024m –Xmx1024m –server

startup.sh
JAVA_OPTS=”-Xms1024m –Xmx1024m –server $JAVA_OPTS”

An X-PUSH Server does not operate normally due to memory shortage when the free heap memory reaches zero.

Connection Resource Shortage

The maximum number of clients that can connect to an X-PUSH Server is determined by the container number set in "xpush_config.xml" and the number of servants per container.

The default values at the time of deployment are 20 Servant Containers and 50 Servants per container. A total of 1,000 clients are able to connect. (20 containers x 50 servants) Additional client connections are no longer allowed when the Total Client Count reaches the maximum number of connections. In such cases, the settings need to be modified.

Modifying the Servant Container and Servant numbers can negatively affect the performance of an X-PUSH Server.

CPU/Network Resource Shortage

A Waiting Ratio value greater than 30% signifies that messages are not immediately delivered and are waiting in standby. This indicates a CPU or network resource shortage. In this case, the Average Messages in Queue value is usually large, and the Average Tps is at a low level. CPU and network usage should be monitored using external tools, and the necessary resources should be expanded.

Lack of Database Connection

When DB usage increases, there is no DB Connection to process other messages, resulting in messages to be accumulated in Queue. At this time, the Max Connection of DB can be checked to increase the value in maxActive in X-PUSH xpush_conf.xml

[show status Symptom] 
MODULE : Repository Service
          QUEUE
                    Provide Message Queue Size = 100 / 100
                    Response Message Queue Size = 500 / 20000
                    Request Message Queue Size = 1000 / -1
                    Topic Message Queue Size = 0 / -1
          Message Process Time(milisec)
                    Provide           # : 0 / 0        Avg : 0      Min : 0      Max : 0
                    Response        # : 0 / 0        Avg : 0      Min : 0      Max : 0
                    Request          # : 20 / 20     Avg : 0      Min : 0      Max : 0
                    Topic             # : 0 / 0        Avg : 0      Min : 0      Max : 0

If all 20 Connections are used in Request, messages are accumulated in Queue as other Queues cannot be processed.

<service name="DbcpService">
	...
	<attribute name="maxActive">50</attribute>
	...
</service>

If MaxActive is all in use, the X-PUSH server does not operate properly.

show trace

This script allows for remote viewing of logs output to the console. Go to the "$XPUSH_HOME/bin" directory and run "show_trace.bat/sh" The following four parameters are required when running the script.

Parameter	Description
host	X-PUSH Server IP address.
port	X-PUSH Server monitor port.
id	Authentication account for the monitor.
password	Authentication password for the monitor.

show_trace.bat localhost 10083 id pw
./show_trace.sh localhost 10083 id pw

The following is a status example.

The information will continue to be output until it's stopped with Ctrl+C.