Monitoring

This tutorial demonstrates how to set up infrastructure to monitor an instance of AvalancheGo. We will use:

• Prometheus to gather and store data
• node_exporter to get information about the machine,
• AvalancheGo's Metrics API to get information about the node
• Grafana to visualize data on a dashboard.
• A set of pre-made Avalanche dashboards

Prerequisites:

• A running AvalancheGo node
• Shell access to the machine running the node
• Administrator privileges on the machine

This tutorial assumes you have Ubuntu 20.04 running on your node. Other Linux flavors that use systemd for running services and apt-get for package management might work but have not been tested. Community member has reported it works on Debian 10, might work on other Debian releases as well.

Caveat: Security

The system as described here should not be opened to the public internet. Neither Prometheus nor Grafana as shown here is hardened against unauthorized access. Make sure that both of them are accessible only over a secured proxy, local network, or VPN. Setting that up is beyond the scope of this tutorial, but exercise caution. Bad security practices could lead to attackers gaining control over your node! It is your responsibility to follow proper security practices.

Monitoring Installer Script​

In order to make node monitoring easier to install, we have made a script that does most of the work for you. To download and run the script, log into the machine the node runs on with a user that has administrator privileges and enter the following command:

wget -nd -m https://raw.githubusercontent.com/ava-labs/avalanche-monitoring/main/grafana/monitoring-installer.sh ;\
chmod 755 monitoring-installer.sh;

This will download the script and make it executable.

Script itself is run multiple times with different arguments, each installing a different tool or part of the environment. To make sure it downloaded and set up correctly, begin by running:

./monitoring-installer.sh --help

It should display:

Usage: ./monitoring-installer.sh [--1|--2|--3|--4|--5|--help]
 
Options:
--help   Shows this message
--1      Step 1: Installs Prometheus
--2      Step 2: Installs Grafana
--3      Step 3: Installs node_exporter
--4      Step 4: Installs AvalancheGo Grafana dashboards
--5      Step 5: (Optional) Installs additional dashboards
 
Run without any options, script will download and install latest version of AvalancheGo dashboards.

Let's get to it.

Step 1: Set up Prometheus ​

Run the script to execute the first step:

./monitoring-installer.sh --1

It should produce output something like this:

AvalancheGo monitoring installer
--------------------------------
STEP 1: Installing Prometheus
 
Checking environment...
Found arm64 architecture...
Prometheus install archive found:
https://github.com/prometheus/prometheus/releases/download/v2.31.0/prometheus-2.31.0.linux-arm64.tar.gz
Attempting to download:
https://github.com/prometheus/prometheus/releases/download/v2.31.0/prometheus-2.31.0.linux-arm64.tar.gz
prometheus.tar.gz                           100%[=========================================================================================>]  65.11M   123MB/s    in 0.5s
2021-11-05 14:16:11 URL:https://github-releases.githubusercontent.com/6838921/a215b0e7-df1f-402b-9541-a3ec9d431f76?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211105%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211105T141610Z&X-Amz-Expires=300&X-Amz-Signature=72a8ae4c6b5cea962bb9cad242cb4478082594b484d6a519de58b8241b319d94&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=6838921&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.31.0.linux-arm64.tar.gz&response-content-type=application%2Foctet-stream [68274531/68274531] -> "prometheus.tar.gz" [1]
...

You may be prompted to confirm additional package installs, do that if asked. Script run should end with instructions on how to check that Prometheus installed correctly. Let's do that, run:

sudo systemctl status prometheus

It should output something like:

● prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-11-12 11:38:32 UTC; 17min ago
Docs: https://prometheus.io/docs/introduction/overview/
Main PID: 548 (prometheus)
Tasks: 10 (limit: 9300)
Memory: 95.6M
CGroup: /system.slice/prometheus.service
└─548 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/con>
 
Nov 12 11:38:33 ip-172-31-36-200 prometheus[548]: ts=2021-11-12T11:38:33.644Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=81 maxSegment=84
Nov 12 11:38:33 ip-172-31-36-200 prometheus[548]: ts=2021-11-12T11:38:33.773Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=82 maxSegment=84

Note the active (running) status (press q to exit). You can also check Prometheus web interface, available on http://your-node-host-ip:9090/

If everything is OK, let's move on.

Step 2: Install Grafana ​

Run the script to execute the second step:

./monitoring-installer.sh --2

It should produce output something like this:

AvalancheGo monitoring installer
--------------------------------
STEP 2: Installing Grafana
 
OK
deb https://packages.grafana.com/oss/deb stable main
Hit:1 http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports focal InRelease
Get:2 http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports focal-updates InRelease [114 kB]
Get:3 http://us-east-2.ec2.ports.ubuntu.com/ubuntu-ports focal-backports InRelease [101 kB]
Hit:4 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu focal InRelease
Get:5 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [114 kB]
Get:6 https://packages.grafana.com/oss/deb stable InRelease [12.1 kB]
...

To make sure it's running properly:

sudo systemctl status grafana-server

which should again show Grafana as active. Grafana should now be available at http://your-node-host-ip:3000/ from your browser. Log in with username: admin, password: admin, and you will be prompted to set up a new, secure password. Do that.

Prometheus and Grafana are now installed, we're ready for the next step.

Step 3: Set up node_exporter ​

In addition to metrics from AvalancheGo, let's set up monitoring of the machine itself, so we can check CPU, memory, network and disk usage and be aware of any anomalies. For that, we will use node_exporter, a Prometheus plugin.

Run the script to execute the third step:

./monitoring-installer.sh --3

The output should look something like this:

AvalancheGo monitoring installer
--------------------------------
STEP 3: Installing node_exporter
 
Checking environment...
Found arm64 architecture...
Downloading archive...
https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-arm64.tar.gz
node_exporter.tar.gz                        100%[=========================================================================================>]   7.91M  --.-KB/s    in 0.1s
2021-11-05 14:57:25 URL:https://github-releases.githubusercontent.com/9524057/6dc22304-a1f5-419b-b296-906f6dd168dc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211105%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211105T145725Z&X-Amz-Expires=300&X-Amz-Signature=3890e09e58ea9d4180684d9286c9e791b96b0c411d8f8a494f77e99f260bdcbb&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=9524057&response-content-disposition=attachment%3B%20filename%3Dnode_exporter-1.2.2.linux-arm64.tar.gz&response-content-type=application%2Foctet-stream [8296266/8296266] -> "node_exporter.tar.gz" [1]
node_exporter-1.2.2.linux-arm64/LICENSE

Again, we check that the service is running correctly:

sudo systemctl status node_exporter

If the service is running, Prometheus, Grafana and node_exporter should all work together now. To check, in your browser visit Prometheus web interface on http://your-node-host-ip:9090/targets. You should see three targets enabled:

• Prometheus
• AvalancheGo
• avalanchego-machine

Make sure that all of them have State as UP.

- job_name: "avalanchego"
  metrics_path: "/ext/metrics"
  scheme: "https"
  tls_config:
    insecure_skip_verify: true
  static_configs:
    - targets: ["localhost:9650"]

All that's left to do now is to provision the data source and install the actual dashboards that will show us the data.

Step 4: Dashboards ​

Run the script to install the dashboards:

./monitoring-installer.sh --4

It will produce output something like this:

AvalancheGo monitoring installer
--------------------------------
 
Downloading...
Last-modified header missing -- time-stamps turned off.
2021-11-05 14:57:47 URL:https://raw.githubusercontent.com/ava-labs/avalanche-monitoring/master/grafana/dashboards/c_chain.json [50282/50282] -> "c_chain.json" [1]
FINISHED --2021-11-05 14:57:47--
Total wall clock time: 0.2s
Downloaded: 1 files, 49K in 0s (132 MB/s)
Last-modified header missing -- time-stamps turned off.
...

This will download the latest versions of the dashboards from GitHub and provision Grafana to load them, as well as defining Prometheus as a data source. It may take up to 30 seconds for the dashboards to show up. In your browser, go to: http://your-node-host-ip:3000/dashboards. You should see 7 Avalanche dashboards:

Select 'Avalanche Main Dashboard' by clicking its title. It should load, and look similar to this:

Some graphs may take some time to populate fully, as they need a series of data points in order to render correctly.

You can bookmark the main dashboard as it shows the most important information about the node at a glance. Every dashboard has a link to all the others as the first row, so you can move between them easily.

Step 5: Additional Dashboards (Optional)​

Step 4 installs the basic set of dashboards that make sense to have on any node. Step 5 is for installing additional dashboards that may not be useful for every installation.

Currently, there is only one additional dashboard: Subnets. If your node is running any Subnets, you may want to add this as well. Do:

./monitoring-installer.sh --5

This will add the Subnets dashboard. It allows you to monitor operational data for any Subnet that is synced on the node. There is a Subnet switcher that allows you to switch between different Subnets. As there are many Subnets and not every node will have all of them, by default, it comes populated only with Spaces and WAGMI Subnets that exist on Fuji testnet:

To configure the dashboard and add any Subnets that your node is syncing, you will need to edit the dashboard. Select the dashboard settings icon (image of a cog) in the upper right corner of the dashboard display and switch to Variables section and select the subnet variable. It should look something like this:

The variable format is:

Subnet name:<BlockchainID>

and the separator between entries is a comma. Entries for Spaces and WAGMI look like:

Spaces (Fuji) : 2ebCneCbwthjQ1rYT41nhd7M76Hc6YmosMAQrTFhBq8qeqh6tt, WAGMI (Fuji) : 2AM3vsuLoJdGBGqX2ibE8RGEq4Lg7g4bot6BT1Z7B9dH5corUD

After editing the values, press Update and then click Save dashboard button and confirm. Press the back arrow in the upper left corner to return to the dashboard. New values should now be selectable from the dropdown and data for the selected Subnet will be shown in the panels.

Updating​

Available node metrics are updated constantly, new ones are added and obsolete removed, so it is good a practice to update the dashboards from time to time, especially if you notice any missing data in panels. Updating the dashboards is easy, just run the script with no arguments, and it will refresh the dashboards with the latest available versions. Allow up to 30s for dashboards to update in Grafana.

If you added the optional extra dashboards (step 5), they will be updated as well.

Summary​

Using the script to install node monitoring is easy, and it gives you insight into how your node is behaving and what's going on under the hood. Also, pretty graphs!

If you have feedback on this tutorial, problems with the script or following the steps, send us a message on Discord.