The Importance of Remote Logging

Logging is a critical piece of the security and development ecosystem. Applications can use logs to track application behaviors, using them to debug and improve experiences, and security professionals can use them to better understand what happened and perform appropriate forensics. It is so important in the world of Security that multiple regulations, across multiple industries, exist.

One aspect of logging that needs more focus, however, is the storage of those logs. Granted, we're partial here, but in recent weeks we have seen how bad actors will always work to delete their tracks. Remote logging helps ensure the integrity of logs so that they are not manipulated, intentionally or unintentionally. The most notable incident highlighting the importance of this came from the Cisco Hack:

The attacker also took steps to remove evidence of activities performed on compromised systems by deleting the previously created local Administrator account. They also used the “wevtutil.exe” utility to identify and clear event logs generated on the system.

A more recent comes in the form of our research at NOC where we have been following a bad actor as they work through one of our honeypots. In the process, it appears the attacker became aware of our monitoring and tried to get rid of the evidence. This article shares that experience.

A Remote Logging Scenario

This whole thing started a few weeks back. We configured a honeypot designed to lure bad actors to one of our websites. It was a pretty standard thing for us, we're constantly looking at ways to better understand bad actors and their TTPs.

As part of this process, we built our loggers to record user activity. This helps us analyze data coming across via POST requests, and vis INPUT fields. When you're doing research, this is important because by default most web access logs are lacking in good data.

Something we were keenly aware of is the importance of log integrity. We wanted to track the users' behavior, but we also wanted to make sure they couldn't cover their tracks. We did this by pushing the logs to Trunc:

This little configuration would allow us to aggregate all our logs in one place, from the access and error logs generated by Apache, as well as our logs. While it presents a unique way to parse and analyze activity, it also protects us from situations like - a hacker trying to delete their logs.

Removing Incriminating Evidence in Logs

You will have to read the articles to better understand the whole scenario, but throughout the week we have been writing and sharing what the hacker was doing. From how they logged in, to how they took control, to the various payloads they were deploying.

In this scenario, as they were deploying payloads we were saving and analyzing what they were pushing to our servers. One day, we decided to wipe everything clean to see what they would do and that is where I think it occurred to them that something was up because when we logged in our key loggers were gone from the server.

Because of the remote configuration, however, we were able to trace the attackers' steps and we were able to correlate across other log files to understand their actions.

The first thing they did was log in via the normal access control (i.e., wp-admin) here:

Notice that it is going to the postlogger.php file, this is the log we're using to intercept inbound POST requests and save them to a log.

They then access a plugin in WordPress called File Manager and added their backdoor back: skuy.php

After adding the payload, they go back to the FileManager and remove these files:

What you see in that image is the ID's associated with those images from the DB. They correspond with three files: postlogger.php, postlogger.log, and postlogger.bkp. Yes, we were not very sneaky about what we were doing, we left it wide open in the root folder to see what would happen.

What was not accounted for was how the logger was working. In the process of removing the logs the attacker killed the site kililng any further work for the night:

Why Logs Were Removed - Our Version

It is impossible to know exactly what was going through their head, but here is what we think happened:

After a wonderful evening, the attacker came to work ready to check on his new treasure throve of sites. One of which was our honeypot. Upon checking his sensors, they likely realized our site was no longer connected to their ad / spam / malware network.

"What the heck?!?!?" they cried out loud...

Further investigation showed their payloads were off line as well. They quickly go to the site, logging in via the main access control (i.e., wp-admin), opened the File Manager plugin and saw these very peculiar files with postlogger in the name. They quickly realized what the files contained, an archive of all their activities.

"Blasphemy!!!" they scream.. "I'll show them..".. to which they then proceeded to remove the logs.

Maybe we took some liberties here, but the general gist is probably the same. Upon returning, they realized the site had been cleaned. Upon further inspection, they noticed these peculiar files recording their activity. They remove them in the hopes of deleting all traces of the evidence, but in the process the site dies killing eveyrone's exploits that evening. Regardless of the attack sophistication, the latter part of this process is what we should all be thinking about.

How do we ensure that if a bad actor tries to remove any evidence we preserve the data?

That's where things like remote logging come into play. Whether a solution like Trunc, or someone else, every organizations should consider a remote logging solution as a way to maintain the integrity and safe keeping of their logs.