Improved Monitoring

TL;DR: I am aggregating my monit data with kuma, have an import script with IaC and better notifications.

I’ve been using monit as my primary monitoring tool for years now. It is low on resources, runs on all my relevant systems (mainly debian/ubuntu servers, raspberries) and is able to monitor both exposed services like webservers and internal metrics, e.g. memory or cpu load. And what I really love is the ability to deploy monit via ansible-playbook including set up a webserver with certs and get and store all my configs in files. This way everything is in a source code repository (git) and can be re-used/restored.

But I was never really happy with the notifications and aggregation.

I had several ideas and tries with other software, but some would take to much resources, others where not capable of monitoring everything I wanted and last but not least proper notifications for my use case was a deal breaker. The most promising had been prometheus alongside monit but I was unable to transform the data for a useful aggregation.

First steps for a solution

As I wanted a solution with matrix for notifications, I stumbled over 🐻 uptime kuma some time ago. Matrix and a lot of other notification providers are already integrated, docker compose is available. So why not give it a try?
After setting up some test monitors and the matrix notification I was quite happy, but did not find an easy solution to add more monitors, e.g. via CSV upload or other config files like I was familiar with monit. And I was (which is comprehensible) not able to monit system internal metrics.

kuma api, ansible vault

Some time later I’ve played around with python for another project and happened to come along a kuma api. As I only had a few needs (mainly importing monitors and tags), it was quite easy to build a small python script to import some monitors from a json file. Some more additions with groups (some monitor type), some tags and even edit entity by name were implemented in short time. Clearing data to set up everything from scratch was the easiest part. I even managed to use ansible vault, so I could separate secrets from the config and data files.

Infrastructure as Code

Storing my private data outside of the app repository was the last step to have this “finished”. git sparse is the solution to have one data repository but only check out the needed subfolder. I had the “remove” the submodule folder with git restore --staged dataand use git pull from it to update, but this is convenient enough.

Now my monitoring setup is massively improved.