Threat Hunting: A Basic Guide to Log Analysis for Cybersecurity

Over the years, we’ve helped countless organizations recover from compromises. Surprisingly, many breaches are discovered by chance rather than deliberate detection. A common scenario involves finding anomalies during routine investigations, leading to the discovery of hidden threats.

Take this classic example: “While looking for A, I stumbled upon B, which shouldn’t be there, and uncovered C, revealing we’ve been hacked.”

In some cases, breaches aren’t even discovered internally. External parties, such as law enforcement or attackers themselves (e.g., during ransom demands), often alert organizations to their vulnerabilities. This occurs even in companies with robust cybersecurity budgets and cutting-edge tools.

To stay ahead of such scenarios, organizations must adopt exploratory or unstructured threat hunting. This involves dedicating time to actively search for anomalies, suspicious activities, or indicators of compromise. While traditional threat hunting can start with alerts from tools like IDS/IPS or WAF, this guide emphasizes exploratory hunts, where you start from scratch to uncover hidden threats.

The Basics of Threat Huntings

Before diving into threat hunting, you must establish a strong foundation. Without proper groundwork, your efforts may lack critical data. Here’s what you need to prepare:

These are some of the items you need to have before you start your threat hunting:

1. Inventory Management
• List all employee devices (laptops, desktops, phones).
• Document servers, routers, firewalls, and externally hosted sites.
• Track domain names and owned IP addresses.
2. Update Management
• Ensure all devices and systems are updated automatically or through scheduled updates.
3. Centralized Log Collection
• Use a centralized logging server to gather logs from: Windows, Linux, Cloud, Mail, etc...
• Tools like Trunc can simplify centralized logging.
4. Enable Logging Across All Systems
• Ensure all relevant logs are enabled, configured, and sent to the logging server.

7 Key Threat Hunting Techniques

Effective threat hunting begins with knowing what to look for and where to find it. By examining patterns and behaviors in your logs, you can uncover hidden threats that automated tools might miss. This section focuses on practical techniques to identify anomalies, unauthorized access, and other suspicious activities. Whether it’s tracking unusual system user logins or detecting failed logins followed by success, these strategies will help you proactively protect your organization from cyber threats.

1. Investigate Suspicious System User Logins

System users (e.g., daemon, bin, sys, nobody) should never log in. Investigate any instances where they do. Use tools like grep to identify anomalies in authentication logs:

zcat /var/log/central.auth.log*.gz | grep -E "Accepted [a-z]+ for (daemon|bin|sys|www-data|nobody|...)"

2. Identify Logins to Unauthorized Locations

Review successful logins to systems where users have no business need. For example:

cat all-logs* | grep -E "Accepted (password|publickey) for" | cut -d ":" -f 3- | cut -d " " -f 2,7 | sort | uniq -c

These logs reveal user-to-server relationships, helping you identify unauthorized access attempts.

3. Analyze Logins Outside Business Hours

For local organizations, track logins outside standard business hours (e.g., 10 PM to 5 AM). This can uncover malicious activity with minimal false positives. Use SIEM tools or custom scripts to create time-based alerts.

4. Detect SSH Logins Using Passwords

Encourage SSH logins via keys and flag password-based logins as potential violations of security policy:

grep "Accepted password for" /var/log/auth.log

5. Monitor Logins from Proxies or Tor Nodes

Attackers often mask their IP addresses using proxies or Tor. Leverage reputation databases (e.g., NOC Reputation API) to flag suspicious IPs:

for ip in $(cat list-of-ips); do curl -s "https://reputation.noc.org/api/?ip=$ip" | grep true; done

6. Flag Logins from Foreign or "Impossible" Locations

Use GeoIP data to flag logins from unexpected countries or impossible locations (e.g., logins from the US and Europe within hours). Anomalies in IP ranges can also reveal unauthorized activity.

7. Investigate Failed Logins Followed by Success

Track repeated failed login attempts followed by a successful login. This could indicate brute-force attacks:

grep "Failed password for" /var/log/auth.log | cut -d " " -f 1-2 | uniq -c

Why Threat Hunting Matters

Threat hunting is proactive, enabling organizations to uncover hidden threats before they escalate into breaches. Unlike reactive measures, which rely on alerts triggered after malicious activity, threat hunting actively seeks anomalies in patterns, behaviors, and network logs. This approach uncovers:

• Advanced persistent threats (APTs)
• Insider threats
• Subtle compromises that bypass traditional defenses

Threat hunting is not a one-time task—it’s an ongoing process that evolves as your organization’s infrastructure and threat landscape change. By analyzing logs creatively and methodically, you can stay ahead of attackers and strengthen your security posture.

If you’re looking for a reliable centralized logging solution, check out Trunc to streamline your log collection and analysis. Have your own threat-hunting tips? Share them with us!

Stay vigilant and happy hunting!