Troubleshooting Remote Syslog with TCPDUMP
May 16, 2025
Daniel Cid (@dcid)

Remote syslog is the easiest way to forward all your logs to a remote log server. It is agentless and supported by most operating systems. For example, if you are using syslogd or rsyslog, all you have to do is edit the /etc/rsyslog.conf (or /etc/syslog.conf) and add the following:

*.* @SYSLOGSERVERIP:PORT

To have all your logs sent to SYSLOGSERVERIP. But what to do when you don't see the logs on the remote server?

Using tcpdump to look at syslog

Syslog is a clear text protocol, so perfect to inspect with tcpdump. To start, you can just run tcpdump restricted to UDP and the remote server IP:

$ sudo tcpdump -nnn udp and host y.y.18.158
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
17:54:01.479811 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
17:54:01.479977 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 90
..


In our case, the SYSLOGSERVERIP is y.y.18.158 running on port 514. So we can see that the packets are being sent over there. If you want to inspect deeper, you can pass the -A flag to see the content:



$ tcpdump -A -nnn udp and host y.y.18.158
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
18:00:01.708077 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
E.....@.@...k.(......E...wv.<86>May 17 18:00:01 testserver CRON[19937]: pam_unix(cron:session): session opened for user ngix by (uid=0)
18:00:01.711938 IP x.x.40.29.45637 > y.y.18.158.514: UDP, length 111
E.....@.@...k.(......E...wv.<86>May 17 18:00:01 testserver CRON[19939]: pam_unix(cron:session): session opened for user nginx by (uid=0)


And it will display the content. On our example, the cron job being executed. Note that just because the server is sending the syslog, it doesn't mean the syslog server is receiving it. You might have a firewall in between that could be blocking it. You can run the same command on the syslog server to see if the packet is arriving and investigate the issue from both angles.

Logging Guides

We love logs. In this section we will share some articles from our team to help you get better at logging.

Trunc Logging

Logging for fun and a good night of sleep.

  • Real time search
  • Google simple
  • Cheap
  • Just works
  • PCI compliance
Contact us!

Do you have an idea for an article that is not here? See something wrong? Contact us at support@noc.org

Tired of price gouging
  • Clear pricing
  • No need to guess
  • Real people
  • Real logging

Simple, Affordable, Log Management and Analysis.

14 days free trial. No credit card required.