Simple, Affordable, Log Management and Analysis.
14 days free trial. No credit card required.
Hunting is what you do about the attacks your alerts did not catch. This page covers how to approach it with logs you already have, with specific hypotheses you can test this afternoon and the reasoning behind each one.
Detection is reactive. You write a rule for something you know is bad, and when it happens the rule fires. That covers the attacks somebody has already characterized, which is most of them, and it is the right place to spend most of your effort.
Hunting covers the rest. It starts from the assumption that something got past the rules, and goes looking. There is no alert to respond to, so the work begins with a question rather than an event.
The practical difference is that hunting needs the boring logs. Detection can discard everything that did not match a rule, which is what OSSEC does by default and what Wazuh does unless you turn on archives. Hunting is a search through exactly that discarded material, because a successful login is not an alert and neither is a DNS query, and both are how you find someone who is already inside.
A hunt is a hypothesis, a search that would confirm or deny it, and a decision about what to do with the result. Writing the hypothesis down first is worth the thirty seconds, because it stops the session drifting into aimless browsing.
A well formed hypothesis is specific enough to be wrong. "Are we compromised" is not one. "An attacker who obtained valid credentials would be logging in outside working hours from an address we have not seen before" is one, and it suggests its own query.
Most hunts find nothing. That is a normal outcome rather than a failed session: you have ruled out a scenario, and if the hunt is worth repeating it becomes a saved search or an alert so nobody has to hunt for that particular thing again. Turning successful hunts into detections is how the alert coverage grows.
Each of these is a hypothesis, why it works, and what to look at. The searches are written in Trunc syntax, which is full text with field filters, and translate to any log platform.
1. A successful login after a run of failures
Hypothesis: somebody guessed a password and is now using it.
This is the highest value single pattern in authentication logs, and it is invisible to search because it is a property of a sequence rather than a line. Fifty failures is an attempt. Fifty failures followed by a success from the same address is a breach, and the two look identical if you only search for failures.
In Trunc this is a built in rule and lands as category:auth_brute_force_success. If you are hunting manually, pull the failure sources, then check each against successful authentications in the following hour.
2. Authentication from somewhere your people are not
Hypothesis: credentials are being used from outside the countries your staff work in.
Cheap and effective, particularly for VPN and webmail. Start with the country breakdown of successful authentications and look at anything outside your normal set. Expect false positives from travel, VPN exit nodes and mobile carriers routing oddly, which is why this is a hunt rather than an alert for most organizations.
Worth pairing with impossible travel: the same account authenticating from two countries within an interval no aircraft covers. That one has far fewer false positives and is worth promoting to an alert once you have seen how your data behaves.
3. Service accounts behaving like people
Hypothesis: a service account has been taken over and is being used interactively.
Service accounts have narrow, predictable behaviour. They authenticate from the same handful of hosts, at regular intervals, often without a session. An interactive logon by a service account, a logon at an hour it has never used, or a logon from a workstation rather than a server are all worth a look.
On Windows this means watching logon type. Type 3 is network, which is normal for a service account. Type 2 is interactive and type 10 is remote interactive, and neither has any business appearing for an account that exists to run a scheduled task.
4. New user agents against your web estate
Hypothesis: a tool is being used against us that has not been used before.
Attack tooling is often careless about its user agent. Take the distinct user agents hitting your sites this week and compare against last month. Most of the new ones will be a browser version that just shipped or a crawler that changed its string. The interesting ones announce themselves: scanners with their own name in the agent, obviously synthetic strings, or an agent claiming to be a very old browser.
A specific case worth knowing: a user agent claiming to be Googlebot from an address that does not reverse resolve to Google. Real Googlebot passes a reverse and forward DNS check, and a great deal of scraping does not bother to fake that.
5. The shape of your 404s
Hypothesis: somebody is mapping our application before attacking it.
A handful of 404s is a broken link. Two hundred from one address inside a minute, against paths you have never had, is reconnaissance. The paths themselves tell you what they were hoping to find: /wp-login.php on a site that does not run WordPress, .env, /.git/config, backup files, administrative panels.
The follow up matters more than the finding. For each scanning source, check whether anything they requested returned a 200. A scanner that got a 404 for everything found nothing. A scanner that got one 200 found something, and that is the thing to go and look at.
6. Outbound connections nobody explained
Hypothesis: something on our network is talking to infrastructure it should not be.
Inbound gets the attention and outbound holds the evidence. Command and control, exfiltration and tool downloads are all outbound. Look for servers initiating connections to the internet when their job does not require it, connections on odd ports, and regular beacons at suspiciously consistent intervals.
Beaconing is the pattern with the clearest signature. Malware that checks in every sixty seconds produces a regularity that human traffic does not have. If you have flow data or firewall accept logs, group by source and destination and look at the interval distribution.
7. A source that went quiet
Hypothesis: logs stopped because somebody stopped them.
Turning off logging is a standard step after gaining access, and it produces no alert anywhere, because the absence of data is not an event. A host that reported steadily for a year and sent nothing since Tuesday deserves an explanation, and most of the time the explanation is dull. Occasionally it is not.
Trunc watches for this and reports silent sources on the overview, partly because PCI DSS 10.7.2 requires detecting the failure of logging mechanisms. If you are doing it by hand, compare the set of hosts sending logs today against the set from a month ago.
8. New administrative accounts and group changes
Hypothesis: an attacker gave themselves persistence.
Account creation, group membership changes and permission grants are low volume and high value. Windows event 4720 for account creation, 4728 and 4732 for group additions. On Linux, useradd and usermod through sudo or the audit log.
These are worth reviewing exhaustively rather than hunting through, because there are few enough of them that you can read every one. Anything you cannot tie to a change request is worth a phone call.
9. Cleared audit logs
Hypothesis: somebody removed evidence.
Windows event 1102 means the security log was cleared. There are legitimate reasons and they are rare, and the event should be treated as significant until somebody accounts for it. The equivalent question on Linux is a gap in a log file, or a log file whose inode changed outside a rotation window.
This is the argument for shipping logs off the host as they are written. An attacker who clears the local log cannot reach the copy that already left.
10. Volume that changed shape
Hypothesis: something changed and nobody mentioned it.
The least specific hunt and often the most productive. Look at events per hour by category over the last month. Steps, cliffs and new plateaus all mean something changed: a service was deployed, a configuration was altered, a log source was reconfigured, or something is generating traffic that was not there before.
Most of what this turns up is operational rather than security relevant, and it is still worth the ten minutes, because a team that knows the shape of its own log volume notices anomalies far faster than one that does not.
An hour a week beats a day a quarter. Hunting is a skill that depends on familiarity with your own data, and familiarity comes from frequency. A short regular session where you work through two or three hypotheses builds the instinct for what normal looks like, and that instinct is what makes the difference when something is wrong.
Keep notes. What you hunted, what you searched, what you found. Six months later the value is in knowing what you have already ruled out and what was normal at the time, and nobody remembers that unaided.
When a hunt finds something twice, make it an alert. The point of hunting is not to keep hunting for the same thing.
Hunting needs everything stored, categorized and searchable quickly, which is a description of what Trunc is for.
If you are already running OSSEC or Wazuh for host detection, they answer a different question and the two work together.
14 days free trial. No credit card required.
Get
Started