With a few projects spreading over a mixture of networks and servers I though might need some monitoring. There is a few options like Cacti, phpmonitoring and Librenms. Issue is a lot do not cover multi site / the routers involved. Librenms looked very much like a good thought. Now here comes a dive in to the problems I faced.
1) Multiple sites that do not have IPsec between them. With two of the locations the public ip addresses change at random and so messes up the ipsecs. Most the time the ip addresses seem to appear from the same /16. To get round this on the master poller’s router I allow a /16. Big range but likely chances of attack becomes low. This means I can now use a public IP address for MYSQL, REDIS, RRDCAHCED and the web interface.
2) Next issue Librenms documentation is shocking and bad. It half tells you how to do it but it very disjoined to follow and over complicated. Even throwing it through ChatGPT made the matter more confusing. In the end a few changes to the config on pollers that can not use IPsec fixed this issue. It was easier to then tell the remote pollers to communicate over IPsec. This meant putting the server DNS entry in (This would be X.X.X.domainname.co.uk here to deal with the issue
3) Great getting data but RRDcached had stopped working. In the end this became it’s own Proxmox LXC which actually improved the master pollers job. Seeing as the master poller is only there to access the data.
4) The fact you have to add pollers to a group via a config file. It would of been nicer if this can be handled via the GUI or something. I found this to be a pain when deploying new pollers as I could clone a LXC container once it worked.
This is only a bit of a dive in to it as so far I got it working to a extent that I am happy but there is more I want to do with it.