Monitoring

Web Monitoring

This is a service that allows the operator to monitor and manage X-PUSH in a web browser.

The server that monitors the XPush Instance information and status, and integrates the user and message management

Settings

Configuration

Environment Configuration

Open the $xpush-3.0.x/bin/set_xpush.bat file to configure JAVA_HOME and XPUSH_HOME.

set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_XX
set XPUSH_HOME=D:\xpush-3.x.x

Configuration Files

Configuration Files Used in Monitor Server

They are contained in the $XPUSH_HOME/conf directory.

monitor_conf.properties

Default Settings of Monitor Server

Item

Description

monitor.agentListen.host

Address of monitor Server to which xpush will access

192.168.1.1

monitor.agentListen.port

Port of monitor Server to which xpush will access

8099

monitor.httpd.host

Address of monitor Server to which the client will access

192.168.1.1

monitor.httpd.port

Port of the monitor Server to which the client will access

8088

monitor_jdbc.properties

Database Settings of Monitor Server

Item

Description

spring.datasource.url

URL of Database to access

spring.datasource.username

User ID to access Database

spring.datasource.password

User Password to access Database

spring.datasource.driver-class-name

JDBC Class Name to access

monitor_user.properties

Monitor Server Access User Settings

Item

Description

user.id

Account ID to access

user.password

The password of the account

xpushInstance.id

The ID of the xpush server

xpushInstance.password

The password of the xpush ID

monitor_logback.xml

Log Settings of Monitor Server

For the log settings of the monitor server, use logback

http://logback.qos.ch/manual/configuration.html

Operation

Run with the $XPUSH_HOME/bin/monitor_startup.sh/monitor_startup.bat command

Function

login

Access to Monitoring Web Console

Refer to the monitor_user.properties configuration file to log in to the Monitor server.

Setting

XPush Instance

You can add XPush Instance using the " + " button.

System Property

항목

설명

Server Name

XPUSH Server alias

X-PUSH IP

XPUSH Server IP Address

X-PUSH PORT

XPUSH Server PORT

xpush instance xpush_config.xml 의 SecureShellService PORT

$XPUSH_HOME/conf/xpush_config.xml

<service name="SecureShellService" ...>
    <attribute name="ServerBindAddress">0.0.0.0</attribute>
    <attribute name="Port">10089</attribute>
    ...
</service>

If XPush Instance is added, then you can access the monitor server

System Property

Item

Description

Access the X-PUSH instance to the monitor server

Stop the monitor server access

Edit

Delete

System Property

Item

Description

Receive interval (second)

Interval to receive information from the xpush instance (unit: second)

ConnectionRetry Count

The number of times the xpush instance attempts to access

Connection Retry Timeout(second)

ID of the xpush server

Default row Count

The grid row count shown in Monitor UI

Health check

Item

Description

Division

L4, L7 selection

Interval Time (second)

Interval to check

Status

Health check status

Mail

Item

Description

SMTP Server

SMTP server address

SMTP PORT

SMTP server port

SMTP Password

Outgoing user password

USER Mail

Outgoing user mail

To Mail

Incoming user mail

Status

Mail usage status

Summary

Integrated Dashboard Screen

Item

Description

1

PUSH server in operation

2

Accessing user

Shows the number and distribution of users currently accessing.

Shows the total number of users and the number of users accessing by servers.

3

Health Check

Outputs the status of PUSH. (OK/Fail)

Checks at regular intervals based on L4 / L7 Layer.

- L4(Transport): Checks normality by TCP access

- L7(Application): Checks normality by whether the actual message is sent or not

4

System information

Shows CPU, Memory, Network (Input/Output) in the line chart.

5

Queue status

Shows the number of messages to be processed in Queue.

Screen for Each XPush Instance

Item

Description

1

Outputs the system information of XPush Instance.

- Information such as OS, JDK, Memory, IP, PORT

2

Shows the number of users currently accessing.

3

Health Check

Outputs the status of PUSH. (OK/Fail)

Checks at regular intervals based on L4 / L7 Layer.

- L4(Transport): Checks normality by TCP access

- L7(Application): Checks normality by whether the actual message is sent or not

4

System information

Shows CPU, Memory, Network (Input/Output) in the line chart.

5

Queue status

Shows the number of messages to be processed in Queue.

User Mananger

User Information Can Be Searched.

Item

Description

1

Search condition

If a value is not entered, then search all

If a value is entered, then search by AND condition

2

Online user information

- Shows the information of users currently accessing.

- Able to send messages to the selected users.

3

DB user information

- Shows the TOPIC information saved in DB.

- Able to create and change the TOPIC information.

- Able to send a message to the selected TOPIC.

4

DB mobile user information

- Shows the Mobile Device information of the users stored in DB.

- Able to change the Device information.

Message Mananger

Search Available by Message/User/Mobile Notification

  • Message

  • User

  • Notification

Message

Search is Performed by Message.

Item

Description

1

Search condition

If a value is not entered, then search all

If a value is entered, then search by AND condition

2

Message list for the condition

3

User message information for the Row selected in 2

4

Notification information for the Row selected in 2

User

Search is Performed by User-specific Message.

Item

Description

1

Search condition

If a value is not entered, then search all

If a value is entered, then search by AND condition

2

User message list for the condition

3

Message information for the Row selected in 2

4

Notification information for the Row selected in 2

Notification

Search is Performed by Mobile Message.

Item

Description

1

Search condition

If a value is not entered, then search all

If a value is entered, then search by AND condition

2

Notification list for the condition

3

Message information for the Row selected in 2

4

User message information for the Row selected in 2

Send

Message Transmission Function (Provider)

Messages can be sent directly from the web management console

Item

Description

Project ID

ProjectID to transmit

Topic Type

Topic Type to transmit

Topic ID

Topic ID to transmit

Available Period

Message validity period (unit: day)

PUSH, RELI

Select message type

PUSH: General message

RELI: Reliability message

Message

Message to send

You can add the number of messages by using the "+" button.

Console Monitoring

X-PUSH provides a script for monitoring server operation.

show status

This script outputs the server status. Go to the "$XPUSH_HOME/bin" directory and run "show_status.bat/sh" The following four parameters are required when running the script.

Parameter

Description

host

X-PUSH Server IP address.

port

X-PUSH Server monitor port.

id

Authentication account for the monitor.

password

Authentication password for the monitor.

show_status.bat localhost 10083 tobesoft xpush
./show_status.sh localhost 10083 tobesoft xpush

The following is a status example.

.module Main X-PUSH Server

RUNTIME

Item

Description

Startup Time

Server startup time.

Current Time

Current time.

Status Reset

Status reset time.

MEMORY

Item

Description

Max

Maximum available memory for JVM.

Total Heap

Maximum heap memory.

Used Heap

Used heap memory.

Free Heap

Free heap memory.

An X-PUSH Server does not operate normally due to memory shortage when the free heap memory reaches zero.

QUEUE

The queue value is a statistic collected at 0.1-second intervals. A total of 10,000 are stored. The most recent 1,000 seconds (approx. 16 minutes) of stats are provided.

Item

Description

Message Size in Queue

The number of messages currently waiting to be

processed in the X-PUSH main queue

Average Messages in Queue

Average number of messages in the queue. If this value is greater than 1, a delay is occurring in pushing messages.

Waiting Ratio

Percentage of time that messages are waiting in the queue relative to the total server runtime.

Message Size in Queue item being 0 or more and the number continuously being increased. X-PUSH messaging sending being delayed.


More supply than message processing

DB processing being delayed

Client Servant Container

Item

Description

Container Count

Number of containers managing servants.

Total Client Count

Total number of currently-connected clients.

Total Push Count

Total number of messages pushed to the client after starting the server. Sum of the values pushed by each container.

Average Tps

Average per-second number of messages transmitted by the server. The average tps value for all containers.

Container-related delays and tps-related values represent the most recent 10,000 message cases. When individual messages are transmitted to containers and processed, each delay is collected and the respective statistics are calculated.

The items provided per container are as follows:

Item

Description

#client

Number of clients managed by the container.

#pushed

Total number of messages pushed by the container.

#in Q:

Number of messages currently in the container queue.

avg delay

Average standby time for messages in the container queue. The unit is in seconds.

max delay

Maximum standby time for messages in the container queue. The unit is in seconds.

push tps

Per-second number of messages pushed by the container. The value is obtained from the number of messages processed by the container and the time. The number of messages processed by the container is independent from the number of messages actually pushed to each client. For example, certain messages are transmitted to a small number of clients while other messages are transmitted to all clients.

#in Q item being 0 or more and the number continuously being increased. X-PUSH message sending being delayed.

More supply than message processing

Socket Push Message Provider

Item

Description

Connected Provider Count

Number of connected message providers.

Per Servant

Each message provider's IP address, port, connected accounts, and the number of messages provided are displayed.

Provider Message Queue

Item

Description

Push Count

Number of messages pushed by using the X-PUSH Server’s internal queue.

Repository Service

QUEUE

It means the number of messages currently waiting / queue size.

QUEUE

Item

Description

Provide Message Queue Size

Queue with supplied reliable message waiting

Response Message Queue Size

Queue with confirmation response message waiting

Request Message Queue Size

Queue with unreceived sending request message waiting

Topic Message Queue Size

Queue with topic registration request message waiting

#Queue Size item being 0 or more and the number continuously being increased. X-PUSH messaging sending being delayed.


DB processing being delayed


ex) Provide Message Queue Size = 5000 / 20000 being continued or being continuously increased.

Message Process Time(milisec)

It means the time required to process a message by each Queue (Provide, Response, Request, Topic) and the status of the operating thread that processes the corresponding message.

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

#

Activation operating thread – code level / activation operating thread

Avg

Average time required

Min

Minimum time required

Max

Maximum time required

DB Connection Pool

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

Active

Number of Connection in use

Max

Maximum number of Connection

Idle

Number of Connection waiting

Notification Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

in Q

Number of messages accumulated in the corresponding queue

Notification Result Update Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

in Q

Number of messages accumulated in the corresponding queue

Apns Notification Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

in Q

Number of messages accumulated in the corresponding queue

FCM Notification Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

in Q

Number of messages accumulated in the corresponding queue

ApnsNotifier Service

Message Process Time(milisec)

It means the message processing time of the ApnsQueue’s operating thread and Provider sending data to Apns, and the status of the operating thread that processes the corresponding message.

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

#

Activation operating thread – code level / activation operating thread

Avg

Average time required

Min

Minimum time required

Max

Maximum time required

FcmNotifier Service

Message Process Time(milisec)

It means the message processing time of FcmQueue’s operating thread and Provider sending data to FCM, and the status of the operating thread that processes the corresponding message.

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

#

Activation operating thread – code level / activation operating thread

Avg

Average time required

Min

Minimum time required

Max

Maximum time required

Etc Message Queue

Message Process Time(milisec) Provider, Response, Request, Topic 메시지의 처리에 소요되는 시간과 해당 메시지를 처리하는 작업 쓰레드의 상태

Item

Description

in Q

Number of messages accumulated in the corresponding queue

JMX

JConsole Connection

Windows environment setup

  1. Run %JAVA_HOME%\bin\jconsole

  1. Enter the IP address and port in the [Remote] tab, and click the [Connect] button.

A connection to JMX is not made if the X-PUSH Server is installed on Linux.

X-PUSH is bound to 0.0.0.0 and run for both Linux and Windows. For Windows, however, the routing information is registered in the routing table under its own IP address for access via the corresponding address. This is not possible for Linux, because the same information is not registered in the routing table by default.

Method 1

Assign the actual IP address instead of "0.0.0.0" under the "ServerBindAddress" section in xpush_conf.xml.

<service name="Publisher">
	...
	<attribute name="ServerBindAddress">192.168.0.1</attribute>
	...
</service>
Method 2

Assign the IP address to the "/etc/hosts" file.

192.168.0.1 localhost.localdomain localhost
::1 localhost6.localdomain localhost6

Item

X-PUSH MBean

MBean Information List

ObjectName

Description

ConnectionManagerService

X-PUSH:name=ConnectionManagerService

Information for currently-connected users.

NodeManagerService

X-PUSH:name=NodeManagerService

X-PUSH Node information.

PublisherService

X-PUSH:name=PublisherService

Publisher information.

RepositoryService

X-PUSH:name=RepositoryService

Repository information.

System

X-PUSH:name=System

System information.

Container

X-PUSH:type=container, name=TobePublishContainerService0

X-PUSH:type=container, name=TobePublishContainerService1

......

Container information.

Notification

X-PUSH:type=notification, name=ApnsNotifierService

APNs and GCM information.

Provider

X-PUSH:type=notification, name=GcmNotifierService

Provider information.

ConnectionManagerService

MBean Information List

Description

ConnectUserCount

Number of connected clients.

MBean 정보 목록

설명

ConnectUserCount

Number of connected clients

NodeManagerService

MBean Information List

Description

MyInetInfo

X-PUSH node’s IP and port information.

AnotherNodeCount

Number of nodes.

PublishCount

Number published to other nodes.

AnotherNodeList

Connected node information.

PublisherService

MBean Information List

Description

ClientNumber

Number of connected clients.

PublishAddress

Publish IP address.

ContainerCount

Number of containers.

PublishPort

Publish port.

XpushVersion

X-PUSH version.

DateStart

X-PUSH start date/time.

DateReset

X-PUSH data reset time.

DateCurrent

Current time.

RepositoryService

MBean Information List

Description

UpdateMsgQueueMaxCapacity

Queue Size (Receipt confirmed message queue)

SelectMsgQueueMaxCapacity

Queue Size (Unreceived message queue)

InsertMsgQueueMaxCapacity

Queue Size (Reliable message queue)

SelectMsgQueueState

Queue status (0: open, 1: blocked)

UpdateMsgQueueState

Queue status (0: open, 1: blocked)

InsertMsgQueueState

Queue status (0: open, 1: blocked)

DbcpConnectionPoolActiveCount

Number of active pools.

DbcpConnectionPoolIdleCount

Number of remaining idle pools.

DbcpConnectionPoolMaxCount

Maximum number of pools.

SelectMsgQueueSize

The number of messages currently in the queue. (Unreceived message queue)

InsertMsgQueueSize

The number of messages currently in the queue. (Reliable message queue)

UpdateMsgQueueSize

The number of messages currently in the queue. (Receipt confirmed message queue)

ThreadPoolMax

Number of thread pools.

FailCount

Number of database processing failures.

FailCountLimit

FailCount limit.

TimeRetryDBPeriodMillis

DB check interval (milliseconds) when the queue is blocked.

System

MBean Information List

Description

CpuIdle

System CPU Idle

CpuSys

System CPU usage.

CpuUser

User CPU usage.

JavaHome

JAVA_HOME directory

MemoryCommittedHeap

X-PUSH maximum heap size

MemoryFreeHeap

X-PUSH free heap size

MemoryMaxHeap

X-PUSH maximum heap size

MemoryUsedHeap

X-PUSH used heap Size

ProcessCpu

CPU usage

ProcessName

Process name

ProcessPid

Process ID

ProcessRss

Actual memory usage

ProcessSize

X-PUSH process size

SystemMemoryTotal

Total system memory

SystemMemoryUsed

System memory usage.

SystemMemoryFree

Free system memory.

SysmemNetworkTotalrx

Network read traffic.

SystemNetworkTotaltx

Network write traffic.

SystemSwapFree

Free swap amount.

SystemSwapTotal

Total swap amount.

SystemSwapUsed

Swap usage.

XpushHome

X-PUSH Home directory.

Container

MBean Information List

Description

AverageDelay

Average standby time for messages in the container queue. The unit is in seconds.

CountInQueue

Number of messages in the container queue.

MaxCapacity

Maximum number of messages that can be held in a container queue.

MaxDelay

Maximum standby time for messages awaiting push in a container queue. The unit is in seconds.

MaxServantNum

Maximum number of servants.

PushCount

Number of pushed general messages.

PushTps

Per-second number of messages pushed from a container.

QueueAverageDelay

Average delay time.

PushTps

Per-second number of messages pushed from a container.

QueueAverageDelay

Average delay time.

QueueCurrentCount

Number of messages in queue.

QueueMaxDelay

Maximum delay time.

QueueMaxSize

Maximum number in queue.

ReliCount

Number of pushed reliability messages.

ServantNum

Number of servants.

ThreadPoolCurrentSize

Current thread pool size.

ThreadPoolSetMaxSize

Maximum thread pool size.

ThreadPoolSetMinSize

Minimum thread pool size.

Notification

ApnsNotifierService

MBean Information List

Description

FailureCount

Failure count.

KeystorePath

APNs certificate location.

Retries

Number of retry attempts for failed transmissions.

SuccessCount

Number of successful transmissions.

TotalSendCount

Total number of transmissions.

GcmNotifierService

MBean Information List

Description

Apikey

Authentication key.

FailureCount

Failure count.

Retries

Number of retry attempts for failed transmissions.

SuccessCount

Number of successful transmissions.

TotalSendCount

Total number of transmissions.

Provider

MBean Information List

Description

ConnectProviderList

Connected provider information. (IP, port)

Port

Provider port.

ProviderAddress

Provider IP address.

ProviderNumber

Number of connected providers.

Identifying Resource Shortages

Resource shortages for server operation can be identified through the values from show status.

Memory Shortage

Normal operation is not possible when the free heap memory is low. If the amount of free heap memory is less than 10% of the total available, the memory must be expanded, and the settings must be changed. Modify the -Xms and -Xmx values in the "startup.bat / startup.sh" for the memory settings.

startup.bat
set JAVA_OPTS=-Xms1024m –Xmx1024m –server
startup.sh
JAVA_OPTS=”-Xms1024m –Xmx1024m –server $JAVA_OPTS”

An X-PUSH Server does not operate normally due to memory shortage when the free heap memory reaches zero.

Connection Resource Shortage

The maximum number of clients that can connect to an X-PUSH Server is determined by the container number set in "xpush_config.xml" and the number of servants per container.

The default values at the time of deployment are 20 Servant Containers and 50 Servants per container. A total of 1,000 clients are able to connect. (20 containers x 50 servants) Additional client connections are no longer allowed when the Total Client Count reaches the maximum number of connections. In such cases, the settings need to be modified.

Modifying the Servant Container and Servant numbers can negatively affect the performance of an X-PUSH Server.

CPU/Network Resource Shortage

A Waiting Ratio value greater than 30% signifies that messages are not immediately delivered and are waiting in standby. This indicates a CPU or network resource shortage. In this case, the Average Messages in Queue value is usually large, and the Average Tps is at a low level. CPU and network usage should be monitored using external tools, and the necessary resources should be expanded.

Lack of Database Connection

When DB usage increases, there is no DB Connection to process other messages, resulting in messages to be accumulated in Queue. At this time, the Max Connection of DB can be checked to increase the value in maxActive in X-PUSH xpush_conf.xml

[show status Symptom] 
MODULE : Repository Service
          QUEUE
                    Provide Message Queue Size = 100 / 100
                    Response Message Queue Size = 500 / 20000
                    Request Message Queue Size = 1000 / -1
                    Topic Message Queue Size = 0 / -1
          Message Process Time(milisec)
                    Provide           # : 0 / 0        Avg : 0      Min : 0      Max : 0
                    Response        # : 0 / 0        Avg : 0      Min : 0      Max : 0
                    Request          # : 20 / 20     Avg : 0      Min : 0      Max : 0
                    Topic             # : 0 / 0        Avg : 0      Min : 0      Max : 0

If all 20 Connections are used in Request, messages are accumulated in Queue as other Queues cannot be processed.
<service name="DbcpService">
	...
	<attribute name="maxActive">50</attribute>
	...
</service>

If MaxActive is all in use, the X-PUSH server does not operate properly.

show trace

This script allows for remote viewing of logs output to the console. Go to the "$XPUSH_HOME/bin" directory and run "show_trace.bat/sh" The following four parameters are required when running the script.

Parameter

Description

host

X-PUSH Server IP address.

port

X-PUSH Server monitor port.

id

Authentication account for the monitor.

password

Authentication password for the monitor.

show_trace.bat localhost 10083 id pw
./show_trace.sh localhost 10083 id pw

The following is a status example.

The information will continue to be output until it's stopped with Ctrl+C.