Over the years, we have assisted countless organizations in handling compromises. Surprisingly, many of these breaches are discovered by what I would call "luck." Often, during routine investigations, something unusual catches attention elsewhere, and that's when realize they've been hacked.
The famous: "while looking for A, I tripped into B, which should not be there, and found C on the floor, showing we have been hacked".
In other cases, it's not even an internal detection at all, but external parties, such as law enforcement or even the attackers themselves demanding ransom, that inform the organization of the breach. This can happen even in companies with significant infosec budgets and "cutting-edge" (aka expensive) security tools.
To stay ahead of such scenarios, we recommend adopting a practice known as "exploratory or unstructured threat hunting." This involves dedicating time each day or week for your or your analysts to start with a clean slate and actively search for anomalies, suspicious behaviors, or subtle indicators of compromise. While threat hunting can be initiated following an alert from a security product like an IDS/IPS or WAF, we will focus on the exploratory hunt in this article, where you start from zero.
Before you can do any threat hunting, you need to have the basics done right. What are the basics? First, you need to have your systems properly updated, all your devices accounted for (aka systems inventory) and all your logs being sent to a centralized place for analysis. If you don't have those, your threat hunting will be incomplete and you will be missing data that could be relevant.
These are some of the items you need to have before you start your threat hunting:
And those are just suggestions of things to look for. Every organization is different, but it should cover things that are often overlooked.
Try to check them before you jump into threat hunting.
You have all your logs in place. You have a blank "paper" and a few hours to hunt. What to look for first? There are many areas you could start, from examining trends, graphs or peaks, but for this article, we will cover auth (login) logs.
The first idea is to look for "system user" logins. What are system users? If you use Linux, for example, system users would be those used by the daemons that should never, ever login (you can find all users on /etc/passwd).
daemon
bin
sys
sync
games
man
lp
mail
news
uucp
proxy
www-data
backup
list
irc
gnats
nobody
_apt
systemd-network
systemd-resolve
messagebus
systemd-timesync
pollinate
sshd
syslog
uuidd
tcpdump
Any login from them is highly suspicious and should be investigated. If you have your logs in clear text, you can use grep with this syntax:
$ zcat /var/log/central.auth.log*.gz |grep -E "Accepted [a-z]+ for (daemon|bin|sys|games|www-data|backup|nobody|systemd-[a-z]+|sshd|syslog|tcpdump) "
To find all sshd logins from any of those users. You can also find all users with the bash shell (cat /etc/passwd |grep bash or /bin/sh or zsh for example) and see if there are any anomalies there.
Another idea to look for is successful logins, but to unauthorized locations. Let's say a valid user logins to your DC (domain controller), but he has no business reason to do so.
In theory, it should not be allowed in the first place, but often a user is allowed to more places than it actually needs. And a common post compromise tactic is to find additional servers and locations where the attacker can get in.
If you have your logs in clear text, this command should show you all users / servers relation for sshd logins:
$ cat all-logs* | grep -E "Accepted (password|publickey) for" | cut -d ":" -f 3- | cut -d " " -f 2,7 | sed -E 's/(.*) (.*)/\2 \1/' | sort | uniq -c
3 dcid server-la3
1 dcid server4
1 john server1
1 john server4
1 mike server1
1 mike server2
3 mike server3
4 mike server4
2 mike server5
If you are using Trunc, you can see this data easily as well by going to "Security Insights" and click on Review. We are using SSHD logs for the examples, but you should look for all types of authentication success events.
Next, I would recommend looking for successful (and even failed logins) outside business hours. Depending on your company or how global it is, it might not be possible, but for smaller organizations with a local team, the chance of false positives are small.
For example, if you are in the US West coast, look for any logins from 10pm - 5am (pacific time). Specially local logins (exclude email), VPN access, or failed logins to any of your servers.
If you have a night own that often logins late, you can exclude them from your queries.
I personally like to create buckets of time (23to5, 5to12, 12to20 and 20to23) and fit users into those. Most SIMs/SIEMs support time of the day on their queries and you can leverage that on your search. Open source tools like OSSEC (or Wazuh) also support <time> to allow you to log those events.
Another example to look for is SSH logins using passwords. The recommended way to login via SSH is via keys, so for most companies, a password login should be prohibited by the security policy.
Finding all password logins (grep "Accepted password for") should be an easy way to see users who are not following the policy or a possible compromised account.
The same concept should be used for other products. If a specific login method is mandated, flag any other login types.
Logins from proxies, external VPNs or the Tor network. Attackers often like to hide their IP addresses and connecting via Tor or proxies is actually very common.
I recommend going through all IPs and using the free NOC reputation API to check if any of them flags as proxy, Tor or malicious (used in other attacks). For example, you can run:
$ for i in `cat list-of-ips`; do curl -s "https://reputation.noc.org/api/?ip=$i" |grep -E "is_tor|is_proxy|is_hosting|web_at|web_spam" |grep true; done
And use that to dig deeper into the login, the user and what happened.
This is an easy one. Use GeoIP to flag any login from foreign countries. If you are a USA organization, flag any login from outside of the USA. I would also flag any logins from hosting companies (AWS, GCE, Linode, etc) that are from the US, but should not be used by normal users.
This one goes hand in hand with the previous one. If you can use GeoIP and match the countries (or state) from all the logins from an IP address, you can determine impossible logins. Someone in the east coast of the US, can never login within the period of a few hours from the west coast (without using a VPN or proxy).
Or a login from the US and Europe within a few hours. If you have a global company, those anomalies can help you identify a possible attacker.
I also like to flag logins for the same user from a different IP address range (different /16) as a start (specially if within a small period of time).
This one is pretty standard and most SIEM/SIM tools have support for multiple failed logins. I recommend getting all those IP addresses and checking them against the success logins in case of brute force attempts worked at the end.
If you are using OSSEC/Wazuh, they have a rule for that, but it requires the success to happen within a few minutes from the failures.
For thread hunting, I recommend dumping all IP address for a longer period and checking them for success logins. That helps detect slower attackers that automated tools can miss.
And that's just some suggestions of activities you can look for on your authentication (login) logs. On the next articles, we will talk about other categories of logs and ideas on what to look for in them. Again, those are just ideas. With unstructure / exploratory threat hunting, your goal is to think outside of the box, "play" with your logs and see what you find.
If your boss needs an explanation on what you are doing, you can summarize as: "threat hunting provides proactive security by identifying threats (issues) before they can cause significant damage. Unlike reactive measures, which rely on alerts triggered after malicious activity is detected, threat hunting involves actively searching for hidden threats that may bypass traditional defenses.
By analyzing patterns, behaviors, and anomalies in network and log data, organizations can discover advanced persistent threats, compromises and insider attacks that automated systems might overlook. "
Happy logging!
If you have other ideas, let us know!