SNMP Trap Monitoring With Zabbix - "It's a (SNMP) Trap !"

Jay

• 5 min read
SNMP Trap Monitoring With Zabbix - "It's a (SNMP) Trap !"

So this post is the result of spending some time (more than I care to admit) working out how to get SNMP traps sent by an application (Veeam Backup & Recovery) inside my homelab into Zabbix monitoring.

Why would you do this ?

Well, there are a couple of reasons you might want to do this, not everything supports SNMP polling, applications usually being a good example of this where they can sometimes send traps but dont have a dedicated SNMP server built in.

They are also real-time notifications, no waiting for the next poll from the monitoring server, with traps as soon as the event happens it gets fired through to the trap receiver.

The nature of this means that it reduces network traffic and active processes (pollers), as there is no frequent fetching of the data that needs to happen, depending how many and the type devices being monitored this could be significant. I've always found monitoring iLo interfaces causes my poller to run for longer than feels necessary to collect that many metrics.

Being a push event, server does not require direct access to the device sending the SNMP traps, you could send the data to a server in the cloud without having monitoring inside the network.

Sounds complicated ?

Well, actually no, technically yes - or is that the other way round. It's not so much difficult but there are quite a few things to tie together, and while there are some excellent resources out there, I found they either didn't go from start to finish or missed out something that meant tripping over and needing to troubleshoot - hence this post !

How do we do it ?

  • Device or application creates SNMP trap and sends it
  • Trap is received by snmptrapd
  • snmptrapd processes the event data and outputs to log file
  • zabbix proxy reads the logfile and sends to Zabbix server

I'm using a Zabbix proxy here, but it really doesn't change the setup if you dont use this and are sending the alerts straight to the server. I'm using the proxy as I have it running on a low power single board computer doing some other light monitoring duties.

Lets get started !

Okay, so first we need to have a device that we're going to run Zabbix on, I'm using a SBC running Debian arm.

Download and install Zabbix-proxy, please see the Zabbix download page for the most recent instructions.

# wget repo.zabbix.com/zabbix/7.2/release/raspbian/pool/main/z/zabbix-release/zabbix-release_latest_7.2+debian12_all.deb
# dpkg -i zabbix-release_latest_7.2+debian12_all.deb
# apt update

# apt install zabbix-proxy-sqlite3

Edit /etc/zabbix/zabbix_proxy.conf

Server=[[Enter FQDN hostname of Zabbix server]]
Hostname=[[Enter this proxy name as used in zabbix]]
DBName=/var/lib/zabbix/zabbix_proxy.db

SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
StartSNMPTrapper=1

We are defining the location of the log file we are going to create and telling Zabbix to start the SNMPTrapper.

Now make the directories, set the ownership and restart and enable the services.

# sudo mkdir /var/lib/zabbix
# sudo chown zabbix:zabbix /var/lib/zabbix

# sudo systemctl restart zabbix-proxy
# sudo systemctl enable zabbix-proxy

Okay, with that out of the way, we can now move on to setting up the snmptrap part of the process. Install the snmp tools.

# apt update
# apt install snmp snmptrapd -y

Edit /etc/snmp/snmptrapd.conf to have the following configuration.

createUser -e 0x800000020109840301 monitoring SHA512 zabbix_trap AES256 %Password123%
authUser execute monitoring
authCommunity execute public
perl do "/usr/bin/zabbix_trap_receiver.pl";

Create a log rotation. Create /etc/logrotate.d/snmptrap

/var/log/snmptrap/snmptrap.log {
    weekly
    rotate 12
    compress
    delaycompress
    missingok
    notifempty
}

Lets start and enable the snmptrap service.

sudo systemctl restart snmptrapd
sudo systemctl enable snmptrapd

What we've done so far. Well, we've installed snmptrapd and told it to use the script /usr/bin/zabbix_trap_receiver.pl to process the trap. This doesn't exist yes, so we need to download that and set it to parse the SNMPTrapperFile that we defined in the Zabbix config.

Lets download it, set it as executable and make the change to the script to output the logfile to the location /var/log/snmptrap.

# sudo apt install libsnmp-perl -y

# sudo curl -o /usr/bin/zabbix_trap_receiver.pl https://git.zabbix.com/projects/ZBX/repos/zabbix/raw/misc/snmptrap/zabbix_trap_receiver.pl

# chmod +x /usr/bin/zabbix_trap_receiver.pl

Now edit /usr/bin/zabbox_trap_receiver.pl and set

$SNMPTrapperFile = '/var/log/snmptrap/snmptrap.log';

Debian has some slightly odd user and group names for the snmp, lets create the snmptrap log directory and set the ownership.

#sudo mkdir /var/log/snmptrap
#sudo chown Debian-snmp:Debian-snmp /var/log/snmptrap/

Okay, we should be ready to test, lets run the following command to generate an interface link down trap.

# snmptrap -v 2c -c public localhost '' SNMPv2-MIB::snmpMIB IF-MIB::linkDown  s eth0

# tail --lines=42 /var/log/snmptrap/snmptrap.log

You should see output like this in the log file.

2025-01-15T11:33:37+0000 ZBXTRAP 127.0.0.1
PDU INFO:
  messageid                      0
  errorindex                     0
  receivedfrom                   UDP: [127.0.0.1]:49670->[127.0.0.1]:162
  version                        1
  requestid                      1667996388
  community                      public
  transactionid                  1
  notificationtype               TRAP
  errorstatus                    0
VARBINDS:
  iso.3.6.1.2.1.1.3.0            type=67 value=Timeticks: (5358463) 14:53:04.63
  iso.3.6.1.6.3.1.1.4.1.0        type=6  value=OID: iso.3.6.1.6.3.1
  iso.3.6.1.6.3.1.1.5.3          type=4  value=STRING: "eth0"

MIBS glorious MIBS ....
I'd rather not be looking at OIDs all day, so now install and configure the mibs to do this, you need to ensure that the repo "non-free" is enabled in your /etc/apt/sources.list. I'm also going to provide the system with the Veeam mib which I've copied from the C:\Program Files\Veeam installation.

# ensure non-free enabled in /etc/apt/sources.list
deb https://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware

# sudo apt update
# sudo apt install snmp-mibs-downloader -y
# sudo cp VeeamBackup.mib /usr/share/snmp/mibs/VEAAM-MIB.txt

And I'm going to tell snmp to enable all the mibs, you might want to just enable specific ones and this detailed in the snmp documentation.

Edit /etc/snmp/snmp.conf

mibs +ALL

Restart snmp.

 sudo systemctl restart snmpd && sudo systemctl restart snmptrapd

Lets re-run the test and see if the OID is translated.

# snmptrap -v 2c -c public localhost '' SNMPv2-MIB::snmpMIB IF-MIB::linkDown  s eth0

# tail --lines=42 /var/log/snmptrap/snmptrap.log

And now we should have descriptors

2025-01-15T11:42:48+0000 ZBXTRAP 127.0.0.1
PDU INFO:
  receivedfrom                   UDP: [127.0.0.1]:54391->[127.0.0.1]:162
  messageid                      0
  notificationtype               TRAP
  version                        1
  community                      public
  errorindex                     0
  errorstatus                    0
  transactionid                  1
  requestid                      1648331411
VARBINDS:
  DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (5413583) 15:02:15.83
  SNMPv2-MIB::snmpTrapOID.0      type=6  value=OID: SNMPv2-MIB::snmpMIB
  IF-MIB::linkDown               type=4  value=STRING: "eth0"

Note how "iso.3.6.1.6.3.1.1.4.1.0 type=6 value=OID: iso.3.6.1.6.3.1" has been resolved to "SNMPv2-MIB::snmpTrapOID.0 type=6 value=OID: SNMPv2-MIB::snmpMIB". Much better.

Lets enable snmp to listen on all the interfaces. Edit /etc/snmp/snmp.conf

# agentaddress: The IP address and port number that the agent will listen on.
# agentaddress  127.0.0.1,[::1]
agentaddress  0.0.0.0,[::1]
systemctl restart snmptrapd

Okay, this completes the setup for receiving and parsing the SNMP traps. Next we'll move onto configuring Zabbix, but I'm going to create another post for that linked here ......