Troubleshooting remote syslog with tcpdump

Remote syslog is the easiest way to forward all your logs to a remote log server. It is agentless and supported by most operating systems. For example, if you are using syslogd or rsyslog, all you have to do is edit the /etc/rsyslog.conf (or /etc/syslog.conf) and add the following:

*.* @SYSLOGSERVERIP:PORT

To have all your logs sent to SYSLOGSERVERIP. But what to do when you don't see the logs on the remote server?

Using tcpdump to look at syslog

Syslog is a clear text protocol, so perfect to inspect with tcpdump. To start, you can just run tcpdump restricted to UDP and the remote server IP:

$ sudo tcpdump -nnn udp and host y.y.18.158
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
17:54:01.479811 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
17:54:01.479977 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 90
..


In our case, the SYSLOGSERVERIP is y.y.18.158 running on port 514. So we can see that the packets are being sent over there. If you want to inspect deeper, you can pass the -A flag to see the content:



$ tcpdump -A -nnn udp and host y.y.18.158
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
18:00:01.708077 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
E.....@.@...k.(......E...wv.<86>May 17 18:00:01 testserver CRON[19937]: pam_unix(cron:session): session opened for user ngix by (uid=0)
18:00:01.711938 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
E.....@.@...k.(......E...wv.<86>May 17 18:00:01 testserver CRON[19939]: pam_unix(cron:session): session opened for user nginx by (uid=0)


And it will display the content. On our example, the cron job being executed. Note that just because the server is sending the syslog, it doesn't mean the syslog server is receiving it. You might have a firewall in between that could be blocking it. You can run the same command on the syslog server to see if the packet is arriving and investigate the issue from both angles.



















Posted in   logging   tcpdump   syslog     by trunc_team

Simple, affordable, log management and analysis.