The Cisco Hack - Learning from the Cisco Hack, Trail of Evidence Found in Logs

Cisco Talos recently shared a very interesting article detailing how Cisco was compromised. Companies rarely provide this level of detail, and when they do it serves as a great learning opportunity. These details are instrumental in helping the community get better by helping us understand TTPs being employed by bad actors, but also by highlighting the things we should be monitoring to help keep our enviornments safe. It also provides a great playbook other organizations can use to a) identify issues, b) how to mitigate and c) how to handle a security event.

We thank the Cisco team for how they shared, and what they shared. Before reading more, we encourage you to read their report first: Cisco - Recent Cyber Attack 2022.

As researchers, we were able to use the scenario they described to replicate the events in our labs in an effort to continue to the education and help the community as a whole. We focus on the traces of evidence left in the logs and how to make sense of what is going on. This should help security teams detect and mitigate similar issues if it effects their organization.

Through the use of an effective logging strategy and platform you should be able to identify a series of events, tracks left by bad actors, and alert on key events that can be strong indicators of a compromise (IoC) allowing your team to respond quickly.

1- Password compromise + Phishing success

The attackers were able to steal the Cisco VPN password from an employee through a compromise of that employee's personal Google account. Password sync was enabled in Google Chrome, giving the attacker access to all their passwords. They phished the employee using a series of phishing techniques to get their MFA token, the one that seemed to be most successful was voice phishing (vphish), but more on this in another post.

Up to this point, what could Cisco have really done when the vector was out of their control? Let's assume, not much - short of Phishing awareness in their training. Let's also assume that Phishing is a tough thing to prevent (as it is) and is especially more complicated when we're talking about personal accounts and devices. It is why humans are, and will continue to be, the weakest link in the security chain.

So up to this point, there is little that we could have done as defenders. Ok, an employee fell victim to a Phishing event. No way we could have detected that, they now have access to our network. What's next? How do you detect it? How do you minimize the damage? the impact?

2- New service (remote control tools) installed

According to Cisco Talos, as soon as the attacker got in, they installed a number of tools for remote access:

"The actor in question dropped a variety of tools, including remote access tools like LogMeIn and TeamViewer, offensive security tools such as Cobalt Strike, PowerSploit, Mimikatz, and Impacket, and added their own backdoor accounts and persistence mechanisms. "

When you install an application, Windows will create a new log event (event ID 7045) saying that a new service was installed. Here are some more details on this event ID for Windows:

Here is an example of the event and what it might look like:

If users are not allowed to install services on the internal servers, systems (which they probably should not), you can alert against these events.

3- New User Added

Next, the attackers added new users to the system:

"Consistent with activity we previously observed in other separate but similar attacks, the adversary created an administrative user called “z” on the system using the built-in Windows “net.exe” commands. This account was then added to the local Administrators group."

This is another activity that generates logs that can be alerted in real time. Nobody should be adding new users without going through the proper channels, so that's an easy one to keep an eye out for and alert on if triggered. Windows will use event ID 4720 for new users.

Here is an example of the event and what it might look like:

Windows will also log event ID 4732 for user group changes that can be used to detect whenever a user is added to an admin (or security-enabled) group.

Here is an example of what a trigger on that event looks like inside of the Trunc logging platform:

Note that Windows logs the subject (who did the change) and the target. In our tests, user truncroot did the change. That can also help filter events when only certain administrators can add/delete users.

On AD networks, local user creations should be flagged for extra attention. This seems to be what they did, as it was the escalation of privileges that alert the securtiy team to an issue:

"The attacker then escalated to administrative privileges, allowing them to login to multiple systems, which alerted our Cisco Security Incident Response Team (CSIRT).. "

4- Event log cleared

After adding new users, modifying groups, and installing new software, the attackers tried to clear their tracks by modifying the event log. A great example of why remote logging is so important.

"The attacker also took steps to remove evidence of activities performed on compromised systems by deleting the previously created local Administrator account. They also used the “wevtutil.exe” utility to identify and clear event logs generated on the system.

But, if you don't have remote logging in place, Windows does create an event for this: event ID 1102 for security, so you can at least see that it was cleared. We can not emphasize how important it is track these and alert on them when triggered.

Here is what that log looks like:

Again, a great example of the importance of remote logging that can help centralize all your logs and maintain its integrity, preventing log tampering and missing information.

5- User deleted

After clearing the logs, they deleted the users they had created:

we observed the attacker removing the previously created local administrator account.
net user z /delete

This event is also logged by default in Windows with event ID 4726. If users should not be randomly deleted on your organization (which should not be), that's a good event the monitor:

Here is what that log looks like:

2022/08/12 19:24:26 WinEvtLog: Security: AUDIT_SUCCESS(4726): Microsoft-Windows-Security-Auditing: no_user: no_domain: Windows-Trunc: A user account was deleted. Account Name: z Account Domain: Windows-Trunc

Note that Windows also logs who did the change. If only some users are allowed to delete accounts, the events from these users can be ignored to minimize the amount of log noise.

6- Firewall rule modified

To enable their remote access, the attackers modified the Windows Firewall to allow their connections:

"We observed them modifying the host-based firewall configurations to enable RDP access to systems.
netsh advfirewall firewall set rule group=remote desktop new enable=Yes"

Which is logged by default under Windows/Windows Firewall with Advanced Security event ID 2004. The only problem with this event ID is that it is very noisy.

Here is what that log looks like:

Even in our test system, it has generated over hundreds of logs in the past few hours. However, it is another thing to pay attention to, even if not the best rule to setup automatic alerts for.

What The Cisco Hack Teaches us About Events

At the core of the message is how important it is to have a good system in place to identify, notify and response to events. There team was able to effetively mitigate a security event because they had systems in place, but more importantly because someone was paying attention to the alerts.

From this one event, and from their awesome disclosure, we can improve our security by tuning our environments to pay special attention to the following window events: 7045, 4720, 4732 and 1102. What other events can we connect to their disclosure? Would love more ideas to improve our list here.

Thanks for that!