Looking at your logs - when under a DDoS attack

One of our sites got under a DDoS attack this past week and we thought it would be interesting to share the logs on how a DDoS attack looks like. The attack was on layer 7 ( HTTPS ), where thousands of IPs were generating thousands of requests per second against this one site.

The site was behind the NOC waf/cdn and it automatically blocked the attack using their javascript challenge, keeping their network and the site up during the whole time, but the logs still show all what was going on.



Attack duration

The attack wasn't very long and it only lasted about 1 hour. We noticed from previous experiencies that when the attacker can't take down the site, they will give up after 30 min to 1 hour. Specially if they are using a booter service where they are paying per minute of DDoS. If it is not working, they likely don't want to keep paying for it.

Attack timeline:

24/Sep/2024 02:31:22 started
24/Sep/2024 03:36:18 ended



Attack size

When it started, the attack wasn't very big, with only a few hundred requests per second. For example, for the first few minutes, it went from 101 to 107, 137 and 192 requests per second:

#requests_per_second #time
101 24/Sep/2024 02:31:22
107 24/Sep/2024 02:31:23
137 24/Sep/2024 02:31:24
192 24/Sep/2024 02:31:25
146 24/Sep/2024 02:31:26
156 24/Sep/2024 02:31:27
158 24/Sep/2024 02:31:28
176 24/Sep/2024 02:31:29
170 24/Sep/2024 02:31:30
195 24/Sep/2024 02:31:31
230 24/Sep/2024 02:31:32


But it kept slowly growing to thousands per second later. In total, the average was only 7,077 requests per second, but during the peak of the attack it reached over 120,000 requests per second. And stayed at over 60,000 for a while:

#requests_per_second #time
115394 24/Sep/2024:03:14:54
124034 24/Sep/2024:03:14:55
87017 24/Sep/2024:03:14:56
70828 24/Sep/2024:03:14:57
65735 24/Sep/2024:03:14:58
81757 24/Sep/2024:03:14:59


It also had many cycles, where it would go up to almost 100,000 requests per second and back down to just a couple hundred for a while. That's enough power to get most sites and servers down - and likely get the server disabled by their hosting providers. As all popular hosting providers take down (null route) the site IP in case they detect a DDoS attack against it.

This graph gives an idea on how it looks like:



Attack sources and IPs

The attack happened over IPv4 from 1,924 different IP addresses pretty distributed across hosting providers and locations. Just in the top IPs used in the attack, we can see the most popular hosting / cloud providers:



#1 34.218.33.x 782178 (2.77%) "16509 | US | AMAZON-02, US"
#2 104.42.135.x 704956 (2.49%) "8075 | US | MICROSOFT-CORP-MSN-AS-BLOCK, US"
#3 50.6.193.x 622940 (2.20%) "19871 | US | NETWORK-SOLUTIONS-HOSTING, US"
#4 65.111.28.x 513419 (1.81%) "200373 | DE | DREI-K-TECH-GMBH, DE"
#5 65.111.16.x 512278 (1.81%) "200373 | DE | DREI-K-TECH-GMBH, DE"
#6 65.111.0.x 467351 (1.65%) "200373 | DE | DREI-K-TECH-GMBH, DE"
#7 172.121.142.x 442910 (1.56%) "54252 | US | SP-NYJ, US"
#8 45.56.102.x 408374 (1.44%) "63949 | SG | AKAMAI-LINODE-AP Akamai Connected Cloud, SG"
#9 18.181.186.x 398936 (1.41%) "16509 | US | AMAZON-02, US"
#10 3.92.228.x 391173 (1.38%) "14618 | US | AMAZON-AES, US"
#11 64.176.15.x 331604 (1.17%) "20473 | US | AS-VULTR, US"
#12 138.2.58.x 294393 (1.04%) "31898 | US | ORACLE-BMC-31898, US"
#13 68.183.167.x 289234 (1.02%) "14061 | US | DIGITALOCEAN-ASN, US"
#14 186.49.65.x 286425 (1.01%) "6057 | UY | Administracion Nacional de Telecomunicaciones, UY"
#15 15.235.12.x 285354 (1.01%) "16276 | FR | OVH, FR"
#16 67.220.85.x 237468 (.84%) "63023 | US | AS-GLOBALTELEHOST, US"
#17 203.80.138.x 234872 (.83%) "23764 | HK | CTGNET CTGNet, HK"
#18 118.27.1.x 233294 (.82%) "7506 | JP | INTERQ GMO Internet,Inc, JP"
#19 72.167.46.x 230723 (.81%) "26496 | US | AS-26496-GO-DADDY-COM-LLC, US"
#20 128.199.7.x 225406 (.79%) "14061 | US | DIGITALOCEAN-ASN, US"


AWS, Azura, GoDaddy, Network Solutions, Digital Ocean, OVH, Linode, Vultr.. They are all there. So the attackers were clearly using cloud servers (most likely powerful servers) instead of end user devices for the ddos. Looking at the ASNs, we got a total of 780 different networks, with the top ones being:


62 AMAZON-02, US
46 DIGITALOCEAN-ASN, US
33 OVH, FR
33 EMERALD-ONION, US"
32 AKAMAI-LINODE-AP Akamai Connected Cloud, SG
29 VIETEL-AS-AP Viettel Group, VN
28 MICROSOFT-CORP-MSN-AS-BLOCK, US
27 TELKOMNET-AS-AP PT Telekomunikasi Indonesia, ID
27 DREI-K-TECH-GMBH, DE
26 IPG-AS-AP Philippine Long Distance Telephone Company, PH
23 ATT-INTERNET4, US
22 KIXS-AS-KR Korea Telecom, KR
22 HETZNER-AS, DE
20 AS-VULTR, US
19 ORACLE-BMC-31898, US
18 PINDC-AS, RU
18 AMAZON-AES, US
15 HINET Data Communication Business Group, TW
15 GLOBE-MOBILE-5TH-GEN-AS Globe Telecom Inc., PH


When we break down by countries, we can see requests from 97 countries (pretty well distributed) with the top ones being:


916 US
422 ID
295 RU
174 DE
142 CN
108 AR
107 BR
100 VN
100 PH
90 GB
84 FR
82 CO
78 TH
74 IN
74 HK


What else about the attacking IP addresses

We spent a bit more time looking at the attacking IP addresses and their reputation and we noticed that many of them had web servers running, some of them were running Plesk or cpanel or just the standard nginx on port 80.

Based on the reputation tests, we can see:

Proxy categorized: 25%

Running webserver: 34%

Previously tagged as a malicious IP: 7%

Server responding to pings: 81%

Checking them on DNSrepo, we can see thousands of domains hosted on those attacking IPs.

Based on our experience with hacked sites, I am assuming the majority of those are likely compromised. Some of them have open proxies installed, which can also lead to this problem.



How does the logs look like?

The logs by itself are very boring with nothing in special. The attackers were using random referers (like amazon, google, kijiji, etc) and random users agents (160 total) in an attempt to make it a bit harder to block the attack. The attack logs itself look like this:


204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
204.199.117.x 200 1163 - waf:js_challenge 24/Sep/2024:03:32:41 +0000 "GET / HTTP/1.1" "https://kijiji.ca" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"


But multiply by many million and from almost 2,000 IP addresses. Overall, it is always fun to parse logs to try to find patterns and see if there is anything novel on an attack. In this specific case, it was a basic HTTP flood attack, going against the root of the site.

If there is anything else you would like to see, let us know. All those IPs are tagged on the NOC reputation site now.



Posted in   ddos   logging     by trunc_team

Simple, affordable, log management and analysis.