The ELK Stack, explained

What is the ELK Stack?

ELK is an acronym for three open source projects that are usually deployed together: Elasticsearch, Logstash and Kibana. Elastic, the company behind them, now calls the combination the Elastic Stack, partly because a fourth family of components called Beats became part of the standard deployment and the acronym stopped fitting.

Worth knowing up front: none of the three was built for logs. Elasticsearch is a search engine, Kibana is a visualization tool, and Logstash is a data pipeline. Log management is the use case they became famous for, and a lot of what follows comes down to that history.

The components

A production deployment, piece by piece

Filebeat runs on each server and tails the log files. It ships to Logstash, which parses the lines into fields. Logstash writes into Elasticsearch, which is a cluster of three or more nodes. Kibana runs somewhere with access to the cluster, usually behind a reverse proxy that handles authentication. Index lifecycle policies roll older indices onto cheaper storage and eventually delete them.

Somewhere between Logstash and Elasticsearch you need a buffer. During a burst, Logstash will produce faster than Elasticsearch accepts writes, and without something in the middle you drop data. Most production deployments put Kafka or Redis there, which is a fourth and fifth system to run and monitor.

That is a sound architecture. It scales past volumes almost nobody reaches, and if you have the team for it there is a lot to like. But it is an architecture, and it becomes yours the moment you deploy it.

Where the time goes

The download is free. That gets discussed. What gets discussed less is everything below, which is where the cost lives.

Writing the parsers

Elasticsearch does not know what an sshd line means. Take this one:

Failed password for invalid user admin from 203.0.113.5 port 44122 ssh2

To Elasticsearch that is a string. Turning it into a username, a source address and a port means writing a grok pattern, which is a named regular expression. Then you write one for your firewall, your web server, your mail server, your Windows events, your load balancer, and every application your team has ever deployed. Grok patterns are slow when written carelessly, and a pattern that fails silently produces a document with a _grokparsefailure tag and no fields, which nobody notices until they search for something that should be there.

They also rot. Vendors change log formats in point releases. We have hit this ourselves: a FortiGate traffic rule that had worked for years quietly stopped capturing addresses on ICMP events, because ICMP lines carry an identifier field where TCP and UDP lines carry a source port. Roughly fifteen hundred events a day were being stored with no source address attached, and nothing anywhere reported an error. Somebody has to notice that, and on a self-hosted stack that somebody is you.

Tuning the cluster

Heap size is the first thing everyone gets wrong. The JVM stops using compressed object pointers somewhere around 32GB, so a node given 40GB of heap can address less usable memory than one given 30GB. Elastic's guidance is to stay under the threshold and add nodes instead, which is not obvious to anyone who has not been bitten by it.

After that: shard count and shard size, replica count, refresh interval, field mappings, and mapping explosion when an application starts logging JSON with unpredictable keys. The usual shard guidance is tens of gigabytes each, so an index strategy that was reasonable at fifty gigabytes a day becomes a problem at five hundred. Most ELK incidents are a cluster going yellow or red, and most of those trace back to a shard strategy that was correct when it was chosen.

Building the dashboards

Kibana starts empty, and every saved search and visualization is something a person builds and then maintains against field names that change. The stack has no opinion about which of your logs matter, since it does not know what they are.

Doing the security part twice

ELK is log infrastructure. It will store an authentication failure and a successful login and let you search for both, but nothing in the base stack notices that fifty failures from one address were followed by a success from that same address. That single pattern is the difference between an attempt and a breach, and expressing it requires correlation rather than search.

Elastic sells a security product built on the stack and the free tier covers some of it, though detection rules and case management have sat behind paid tiers. The common alternative is to add Wazuh, a fork of OSSEC that uses Elasticsearch for storage. That works, and it also means running two systems with two rule sets, two agents and two upgrade cycles.

Owning it

A self-hosted ELK deployment is a production system that other production systems depend on. Patching, capacity planning, upgrade testing across major versions that have broken compatibility more than once, and somebody who understands it well enough to fix it at an unhelpful hour. A pattern we see often enough to mention: the engineer who built the deployment becomes the only person who can maintain it, and then changes jobs.

What ELK is good at

• Scale. Elasticsearch handles volumes most log platforms will not. At terabytes a day, with engineers to run it, little else competes.
• Flexibility. It indexes arbitrary JSON. If you can express it as a document you can query it, whether or not it is a log.
• Control. Your hardware, your network, your data. For some organizations that is a requirement rather than a preference.
• Ecosystem. Twelve years of grok patterns, dashboards and integrations somebody else has already written.

Trunc is your ELK and OSSEC stack, without the stack

Trunc does what most teams are after when they set out to build an ELK deployment: logs from everything in one place, searchable immediately, with the security relevant ones already identified. There is nothing to run. No cluster, no heap tuning, no grok patterns, no dashboards to build, no queue in the middle, and nobody paged when the logging platform itself has a bad night.

Parsers that already exist

Trunc ships with parsers for the things people run: Linux and Windows, sshd, sudo, Apache, nginx, IIS, Postfix, Dovecot, pfSense, FortiGate, Sophos, Cisco, Ubiquiti, Office 365, AWS, WordPress and a long list beyond that. A log arrives already broken into source address, user, action and result. When a vendor changes a format, updating the parser is our problem. That FortiGate ICMP case above is one of ours, and fixing it fixed it for everybody sending FortiGate logs.

Categories, not just fields

Every event is tagged with what it means as well as what it says. Authentication failure, brute force, web attack, firewall block, configuration change, PCI DSS 10.2. That taxonomy is what makes the interface useful, because you can ask for everything security relevant across the whole estate without knowing which device produced it or what its format looks like.

Detection inherited from OSSEC

Rules that correlate across events instead of matching one at a time. Repeated authentication failures from an address, and separately the case that matters more, repeated failures followed by a success from that same address. A source that stops sending when it should still be sending. Scanning across your web servers. These fire without configuration, and can publish a blocklist your firewall pulls directly.

An interface that only does logs

Kibana is a visualization tool that can be pointed at logs. Trunc only ever does one thing, so it can assume what Kibana cannot: that you want volume over time split by severity, that you care which hosts went quiet, that a source address should link through to everything else it has done. Search is full text with field filters and returns while you are still looking at the screen.

Common questions

Is the ELK Stack free?

The software can be downloaded and run at no cost, and for many deployments that is enough. Elastic moved away from Apache 2.0 in 2021 and added AGPL as an option in 2024, so it is worth checking the current terms against your own policy rather than relying on what you remember. Either way the licence is rarely the expensive part next to hardware and engineering time.

What about Elastic Cloud or OpenSearch?

Elastic Cloud is Elastic's managed offering, and it removes the cluster operations problem, which is a real portion of what is described above. It does not remove the parsing, the dashboard building or the detection rules. You get a managed general purpose search engine that you then configure for logs. OpenSearch is the fork AWS created after the licence change and the same reasoning applies.

Can Trunc replace Wazuh?

For log analysis and security event detection, yes, and the rule logic shares an ancestor. Wazuh also does file integrity monitoring and vulnerability detection on the endpoint, which are agent side features. If those are central to what you need, compare properly rather than taking our word for it.

Can I migrate from an existing ELK deployment?

Run both for a while. Point your shippers at Trunc alongside Elasticsearch, confirm you are seeing what you expect, then decommission. Trunc accepts standard syslog, so anything already forwarding to Logstash can forward to Trunc without touching the source.