15 Distributed Monitoring
Zabbix can be configured to support hierarchical distributed monitoring.
1 Goals
There are several goals of the distributed monitoring:
-
Get control of whole monitoring from a single or several locations
Zabbix administrator may control configuration of all Nodes from a single Zabbix WEB front-end.
-
Hierarchical monitoring
This is for monitoring of complex multi-level environments.
-
Monitor large complex environments
This is especially useful when monitoring several geographical locations.
-
Offload the overhead from busy Zabbix server
Monitoring thousands of hosts using single Zabbix server? This may be for you!
2 Overview
Zabbix provides effective and reliable way of monitoring distributed IT infrastructure. Configuration of the whole distributed setup can be done from a single location via common WEB interface.
Zabbix supports up-to 1000 (one thousand) Nodes in a distributed setup. Each Node is responsible for monitoring of its own Location. Node can be configured either locally or by its Master node which has a copy of configuration data of all Child Nodes. Configuration of Child Nodes can be done in off line mode, i.e. when there are no connectivity between Master and Child Node.
Hierarchical distributed monitoring allows having tree-like structure of Nodes. Each Node reports to its Master Node only.
All Nodes may work even in case of communication problems. Historical information and events are stored locally. When communication is back, Child Nodes will optionally send the data to Master Node.
New Nodes can be attached to and detached from the Zabbix distributed setup without any loss of functionality of the setup. No restart of any Node required.
Each Node has its own configuration and works as a normal Zabbix Server.
3 Configuration
3.1 Configuration of Nodes
Node configuration is performed in Administration → DM section.
Parameters of a Node:
| Parameter | Description |
|---|---|
| Name | Unique node name. |
| Id | Unique Node ID. |
| Type | Local – Local node Remote – Remote node |
| Time zone | Time zone of the Node. Zabbix automatically converts time stamps to local timezone when transferring time related data across nodes. |
| IP | Node IP address. Zabbix trapper must be listening on this IP address. |
| Port | Node Port number. Zabbix trapper must be listening on this port number. Default is 10051. |
| Do not keep history older than (in days) | For non local historical data only. Zabbix won't keep history of the node longer than N days. |
| Do not keep trends older than (in days) | For non local trend data only. Zabbix won't keep trends of the node longer than N days. |
3.2 Simple configuration
Our simple configuration consists of a Central Node and a Child Node.
Central Node will have total control over configuration of Child Node. Child Node will report to central node events, history and trends.
Central Node will have NodeID=1, while Child Node's NodeID=2.
Central Node IP: 192.168.3.2
Child Node IP: 192.168.3.5
For Central Node
Step 1
Install Zabbix.
Follow standard installation instructions to create database, install Zabbix frontend and binaries.
Step 2
Setup NodeID in server configuration file.
In file zabbix_server.conf:
NodeID=1
Step 3
Convert database data.
Zabbix server has to be executed to convert unique IDs for use by first node.
cd bin ./zabbix_server -n 1 -c /etc/zabbix/zabbix_server.conf Converting tables .................................................................. done. Conversion completed.
Step 4
Configure Node parameters.
Step 5
Add child node.
Step 6
Start Master Node.
We should see NodeID in startup messages of server log file:
31754:20070629:150342 server #16 started [Node watcher. Node ID:1]
For Child Node
Step 1
Install Zabbix.
Follow standard installation instructions to create database, install Zabbix frontend and binaries.
Step 2
Setup NodeID in server configuration file.
In file zabbix_server.conf:
NodeID=2
Step 3
Convert database data.
Zabbix server has to covert all IDs to unique ones for the second node.
cd bin ./zabbix_server -n 2 -c /etc/zabbix/zabbix_server.conf Converting tables .................................................................. done. Conversion completed.
Step 4
Configure Node parameters.
Step 5
Add master node.
Step 6
Start Child Node.
We should see NodeID in startup messages of server log file:
27524:20070629:150622 server #9 started [Node watcher. Node ID:2]
Does it work?
Selection of active nodes will appear automatically after nodes are defined:
Add host for monitoring for Child Node node and see events coming to Master Node:
3.3 More complex setup
The setup consists of seven Nodes. Each Node may be configured either locally (using local WEB interface) or from one of its Master Nodes.
In this example, Riga (node 4) will collect events from all child nodes. It may also optionally collect historical information as well.
4 Platform independence
A node may use its own platform (OS, hardware) and database engine independently of other nodes. Also child nodes can be installed without Zabbix frontend.
It may be practical to use less powerful hardware with Zabbix server running SQLite or MySQL MyISAM while nodes of higher levels may use combination of a better hardware with MySQL InnoDB, Oracle or PostgreSQL backend.
5 Configuration of a single Node
Every Node in distributed environment must be properly configured to have a unique Node ID.
Additional steps
Step 1
Follow standard installation procedure.
Follow standard installation procedure but do not start Zabbix Server. Zabbix front end must be installed and configured. Zabbix database must be created and populated with data from data.sql.
Step 2
Configure zabbix_server.conf.
Add NodeID to Zabbix Server configuration file. NodeID must be a unique Node ID.
Step 3
Configure Master and Child Nodes.
Use Zabbix Frontend to configure details of Nodes having direct communication with the Node. Make sure that all IP addresses and port numbers are correct.
Step 4
Start Zabbix Node.
Start Zabbix Server:
shell> ./zabbix_server
If everything was configured properly, Zabbix node will automatically start configuration and data exchange with all nodes in distributed setup. You may see the following messages in server log file:
... 11656:20061129:171614 NODE 2: Sending data of node 2 to node 1 datalen 3522738 11656:20061129:171614 NODE 2: Sending data of node 2 to node 1 datalen 20624 ...
6 Switching between nodes
When connecting to a node in distributed setup, a list of available child nodes is accessible in right-upper corner of the GUI. It displays current node.
All information available in the GUI belongs to the selected node.
7 Data flow
7.1 Child to Master
Each Child Node periodically sends configuration changes, historical data and events to its Master Node.
| Data | Frequency |
|---|---|
| Configuration changes | Every 120 seconds. |
| Events | Every 10 seconds. |
| History | Every 10 seconds. |
Child Node will resend data in case of communication problems.
Trends are calculated locally based on received historical data.
Zabbix does not send operational data across the nodes. For example, item-related information (last check, last value, etc) exists only locally.
7.2 Master to Child
Each Master Node (a node with at least one child) periodically sends configuration changes to Child Nodes either directly or via other Child Nodes directly connected to the Master Node.
| Data | Frequency |
|---|---|
| Configuration changes | Every 120 seconds. |
Zabbix does not send configuration of a Master Node to Childs.
7.3 Firewall settings
Inter-node communications use TCP protocol only.
| Data flow | Source port | Destination port |
|---|---|---|
| Child to Master | Any | 10051 |
This is default port used by Zabbix trapper process.
8 Performance considerations
Any node requires more processing resources in a distributed setup. Master Node must be powerful enough to process and store not only local data but also data received from its all Child Nodes. Network communications must be also fast enough for timely transfer of new data.
Data source: Zabbix