Setting up node monitoring using Panic Bot

At some point, the number of installed nodes will exceed the allowable limit, and daily checks of their operability will turn into a routine you’ll want to get rid of. Over almost two years of node management, I’ve tried several tools, and Panic Bot from Simply VS has proven to be the most convenient to use.

In this guide, I will try to describe in detail the process of installing and configuring the bot for the following situations: missed blocks, changes in voting power, getting jailed, losing synchronization, and critical server disk space usage.

In this guide, I will use a universal solution for monitoring multiple nodes simultaneously, so I will install the bot on a separate server. If you need to monitor only one node, all actions can be performed on the server with that node, and you can use the notes in the purple frames.

Content

Server preparation

Content

Panic! doesn't use many resources, so it can be installed on the simplest server, such as Hetzner - CPX11.

Update your packages

sudo apt update && sudo apt upgrade -y

Check if Python3 is installed on your server. In most distributions, Python is pre-installed

python3 --version

Install PIP Package Manager and Redis Database Server

sudo apt-get install python3-pip redis-server -y

Install Pipenv Packaging Tool

sudo pip3 install pipenv
sudo systemctl enable redis-server.service

Installation of Panic!

Content

Create a new user for using Panic!

adduser panic --disabled-login

Create a folder and change access permissions

mkdir /opt/panic
chown -R panic:panic /opt/panic
su panic

Navigate to the folder and clone the repository

cd /opt/panic/
git clone https://github.com/SimplyVC/panic_cosmos.git
cd panic_cosmos
git checkout master

Update the packages and start the Panic! setup

pipenv update
pipenv run python run_setup.py

Enter a unique identifier for notifications. For example, your moniker

Setting up the Telegram bot

Content

Setting up Telegram notifications by pressing Y

Next, we need to enter the API token. To obtain it, follow these steps:

Open Telegram.
In the search bar, type @BotFather, select the bot, and press Start.
Use the command /newbot to create a new bot. Enter the bot's name and username. The username must end with "bot" (for example, tRDM_bot). After this, BotFather will send a message with a link to your bot and an API token. Copy the API token and save it in a notepad.
Follow the link to your bot's chat and press Start.
Go to the link api.telegram.org/bot<token>/getUpdates, replacing <token> with the API token of your bot from the previous step. Copy the ID and save it in a notepad.

Now that we have the bot, API token, and chat ID, let's return to the terminal and continue setting up Panic!

Paste the API token and press Enter.
Paste the chat ID and press Enter.
Press Y to test notifications. Go to Telegram and check for the notification. If set up correctly, you'll receive a message saying "Test Alert".
Press Y again to confirm that the notification was received.

Next, Panic! will ask if other notifications are needed (email, phone calls, or periodic alerts). Since we are setting up only the Telegram bot, respond with "n" to all three requests.

Setting up commands for the Telegram bot.

Press Y.
Paste the API token and press Enter.
Paste the chat ID and press Enter.
Press Y to test bot commands.
Go to your Telegram bot and enter the command /ping. A successful response should be "PONG!".
Press Y again to confirm that the command was executed successfully.

Setting up Redis

Content

To configure Redis, we will use the default values:

Redis host IP: localhost
Redis host port: 6379
Redis password: none
Press Y and then Enter three times.

Test Redis functionality by pressing Y.

Connecting a node to monitoring

Content

Navigate to the server with the node you want to connect to the bot and make changes to the configuration file. For example, I'll use the Haqq node. For connecting to other nodes, use your path to the configuration file

(usually - $HOME/.project_name/config/config.toml).

nano ~/.haqqd/config/config.toml

In the "TCP or UNIX socket address for the RPC server to listen on" block, change the value of "laddr".

laddr = "tcp://0.0.0.0:26657"

Save with: Ctrl + X > Y > Enter

If Panic! and Redis are installed on the same server as the node, leave it unchanged

laddr = "tcp://127.0.0.1:26657"

Reload node

sudo systemctl restart haqq

Make these changes to the configuration files of all nodes you want to monitor.

Return to the Panic! server!

Enter the node's name and RPC URL in the format http://NODE_IP:26657, confirm that the node is a validator. Additionally, you can add other nodes right away, but we will do that later.

Also, skip setting up GitHub updates. We will do this a bit later..

Start Panic!

Content

Creating a service file

printf "[Unit]
Description=P.A.N.I.C.
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
User=panic
TimeoutStopSec=90s
WorkingDirectory=/opt/panic/panic_cosmos
ExecStart=/usr/local/bin/pipenv run python /opt/panic/panic_cosmos/run_alerter.py

[Install]
WantedBy=multi-user.target" > /etc/systemd/system/panic.service

Start the service and check the logs

sudo systemctl daemon-reload
sudo systemctl enable panic
sudo systemctl start panic
sudo journalctl -u panic -f

Adding nodes for monitoring

Content

On the server with the node you want to connect to the bot, make changes to the configuration file.

nano ~/.<node_name>/config/config.toml
#or another path to config.toml

In the "TCP or UNIX socket address for the RPC server to listen on" block, change the value of laddr

laddr = "tcp://0.0.0.0:26657"

Save with: Ctrl + X > Y > Enter

Reload node

sudo systemctl restart haqq

Return to the Panic! server. Edit the file user_config_nodes.ini

nano /opt/panic/panic_cosmos/config/user_config_nodes.ini

Copy all existing lines and paste them below. Modify the number, name, and URL. Save the changes.

Reload Panic! bot

sudo systemctl restart panic
sudo journalctl -u panic -f

Check to ensure that the second node appears in the logs.

Repeat the procedure for all remaining nodes you are maintaining.

For additional verification, you can briefly stop the node (exercise caution here; only proceed if you are sure you can reconnect afterward or if you won't get jailed).

The bot will react as follows:

Changing settings

Content

To change the settings of the bot (API token, Redis password, etc.) and to enable email notifications, edit the file user_config_main.ini.

nano /opt/panic/panic_cosmos/config/user_config_main.ini

To configure notifications for changes in GitHub repositories, edit the file user_config_repos.ini.

nano /opt/panic/panic_cosmos/config/user_config_repos.in