Forget manual actions, autoregister your hosts
This simple idea saved countless amounts of work-hours for our team.
If you’re anything in the means of a traditional system administrator, you most likely have dealt with Zabbix along the way of your career path. Zabbix can be a useful tool for Sys Admins, DevOps, and people that handle infrastructure at any scale of companies.
My career in monitoring has led me to see many things in the given field. However, most of them might’ve not been the best possible ways of doing monitoring or could have been the worst possible ways of doing so.
Today I would like to have a deep-dive into something close to my heart — automation.
Back when I was a green-head-in-the-clouds monitoring administrator at Microsoft’s Skype sub-division we used to add graphs and hosts manually into our systems. As you might imagine, this was a time-consuming endeavor that needed doing on a day-to-day basis. Fast forward a couple of years, our team automated ourselves out of the positions we worked. You might see this as a set-back, but for us, it was a solid improvement, we saw this as a victory. I set my steps to a lesser-known Estonian company that was active in the real estate market as a Linux System Administrator. We used Zabbix, quite heavily, all of our servers were in Zabbix, but the one thing we didn’t have was automation. All of the hosts (50–100) were added there manually over half a year.
Moving me on to the current position I’m filling at Pipedrive as Monitoring Team lead, where we had some sort of automation in place when I joined.
This automation wasn’t perfect, rarely anything is. I do not believe that our current system (which is heavily automated) is perfect, there is always room for additional improvements, automating, even more, making systems more and more resilient, and so forth.
It was pretty early on when we decided to make changes to the current automation which was done via Chef Automation software to move the host adding and coupling with templates to Zabbix native way. And that, folks, is what I want to preach today.
How did it help us?
At first, I would say that the registration of the hosts more dynamical and fluid. Also, we ensured that no host would be left without attention, and hosts were added in a consistent matter, which was the essence.
We started by gathering the attributes, like what host group has what kind of FQDN’s (Fully Qualified Domain Names) — it was easy to make automation rules if all of your MySQL databases start with “db”. From there it was already easier to move towards the next challenges, like what templates should we include to these hosts and how to split them logically.
NOTE! You would have to have an active agent installation of Zabbix if you want to use the auto-registration feature in Zabbix.
How can it help you?
It’s fairly simple.
We provision our hosts with Chef — I would recommend you to do something similar to provision the Zabbix agents to the servers in a controlled way for improved management of the agent configuration.
In your zabbix_agentd.conf file define the Zabbix server address, so the agent would know whom to contact and give the server a name.
ServerActive=YOUR_ZABBIX_SERVER_IP_ADDRESS
Hostname=your-server.domain.comI promised it would be fairly simple, didn’t I? This will be enough to get the basic functionality working.
Now what you would want to do is to navigate your way towards the Zabbix UI and from there to Configuration > Actions > Event source: Auto Registration.
Click on the Create action and let the configuration begin.
Follow the written lines on the image above to fill in all the blanks, like what is the common denominator for your FQDN’s.
Then move forward to the Operations tab of the Action window and click on New.
Configure the Operation types you would like to perform and add the Action.
Now, Zabbix will auto-discover all the servers you have reporting to it and will make sure that each and every one of them is with the same configuration.
There are ways to make these actions count even more, and that would be to utilize the power of HostMetadata and HostMetadataItem, but these can remain for the next time.