HTTP User agents

User agents are one of the most interesting parts of the HTTP protocol. It has been there since the beginning and is used to identify the browser (or software doing the request) to the web server.

They are often misunderstood, misused, and more complicated than what it should be, but very useful from an analytics and security perspective.

In this article, we will talk more about user agents, and how the major browsers use it to identify themselves.

What are user agents?

If you are not familiar with the HTTP protocol, the USER AGENT is a header sent during the HTTP request that tells the server what kind of application (agent) is connecting to it.

If you look at the old HTTP/1.1 protocol, that's how a request would look like under the hood:

GET / HTTP/1.1
Host: trunc.org
User-Agent: MY_TEST_USER_AGENT


And on your logs, you would see that the web server accepted the request for the "/" URL with the user agent "MY_TEST_USER_AGENT":

192.168.1.1 - - [26/Jun/2022:00:45:53 +0000] "GET / HTTP/1.1" 301 409 "-" "MY_TEST_USER_AGENT"

The client software specifies what the user agent looks like, in pretty much any format they choose*. For example, if I use the popular curl software to connect to a URL:

$ curl 127.0.0.1/
$ tail -n 1 /var/log/apache2/access.log
127.0.0.1 - - [25/Jun/2022:17:46:53 +0000] "GET / HTTP/1.1" 403 344 "-" "curl/7.81.0"


It uses the "curl/7.81.0" USER AGENT to tell my server (at 127.0.0.1) that Curl version 7.81.0 was used to make the request.

*Different RFCs provide different guidelines (or recommendations), but they are not enforced.

Setting the User Agent

Since the USER AGENT is defined by the client software, you can set it to anything you want. For example, using curl, you can pass the "-A" argument to set the user agent. If we want to tell the server the request came from an iPhone, we could do:

$ curl -A "Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1" 127.0.0.1
$ tail -n 1 /var/log/apache2/access.log
127.0.0.1 - - [25/Jun/2022:18:18:34 +0000] "GET / HTTP/1.1" 403 344 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1"


And the user agent would mimic the iPhone one. Some times I like to change my own *browser USER AGENT to something arbritary to see if anything would break. And often gmail, web.whatsapp and other sites block access because I am using an "old or unsupported browser".

*To change the USER AGENT on Firefox, you need to type about:config and set "general.useragent.override" to the new user agent you want to use.

User Agents used by all the major browsers

All the major browsers use a well-formatted USER AGENT that help us identify the browser being used.

Mosaic Browser

Mosaic is a very popular browser and the only one to support images. If you want to display images on your server, you can parse the USER AGENT and look for NCSA_Mosaic, which is how it sets the USER AGENT. For example, on Windows 3.1, this is how it looks like:

NCSA_Mosaic/2.0 (Windows 3.1)

Ok, we are joking on Mosaic, but had to include it in there. Mosaic is a very old browser, the first one ever invited that supported web sites like they look today. If you want to have some fun online, try changing your USER AGENT to:

NCSA_Mosaic/2.6 (X11; SunOS 4.1.3 sun4m)

or

Mozilla/2.0

And see what breaks. The Mozilla one was used by Netscape 2.0 (Mozilla means Mosaic Killer - a good old Internet competition).

Now to the real browsers...

Chrome on Windows 8, 10 and 11

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36


Chrome on iOS

Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/102.0.5005.87 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/96.0.4664.116 Mobile/15E148 Safari/604.1


Chrome on MacOS

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36


Chrome on Android Pixel 6, Pixel 2

Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.98 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 11; Pixel 2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.98 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 11; SM-M127F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36


Chrome on Chromebook

Mozilla/5.0 (X11; CrOS x86_64 13904.97.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.167 Safari/537.36

Microsoft Edge Browser on Windows 11

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36 Edg/102.0.1245.39
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36 Edg/102.0.1245.39
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Edg/100.0.1185.44


Microsoft Edge on MacOS

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.124 Safari/537.36 Edg/102.0.1245.41

Firefox on Win 10 and 11

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:101.0) Gecko/20100101 Firefox/101.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:101.0) Gecko/20100101 Firefox/101.0


Firefox on MacOS

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:99.0) Gecko/20100101 Firefox/99.0


Firefox on Linux

Mozilla/5.0 (X11; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0

Safari on iOS

Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 14_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1


Safari on MacOS

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Safari/605.1.15


Brave Browser on MacOS

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36

Opera Browser on Windows

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36 OPR/87.0.4390.45

DuckDuckGo Browser on Android and iPhone

Mozilla/5.0 (Linux; Android 11) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/90.0.4430.210 Mobile DuckDuckGo/5 Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 DuckDuckGo/7 Safari/605.1.15



Other interesting USER Agents

There are some other fun things you can look at the USER AGENT. For example, if someone is using Slack and sharing a link there, you would see this USER AGENT:

Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)

And funny enough, if they are using Signal, you would see the "WhatsApp" user agent:

47.x.y.z - - [30/Jun/2022:23:00:02 +0000] "GET /learning/http-user-agents HTTP/1.1" 406 5386 "-" "WhatsApp/2"

With the IP address of the person sharing the link. Those can be useful if you are trying to track engagement - or even as a honeypot if you share a secret link with someone - to see if they will share anywhere else. Note that if you paste a URL on Twitter, the Twitterbot would also contact your server:

199.59.150.181 - - [30/Jun/2022:22:59:50 +0000] "GET /learning/http-user-agents HTTP/1.1" 200 10633 "-" "Twitterbot/1.0"

And if you share a link on WhatsApp, you get the "WhatsApp" user agent, like signal, but sharing your real WhatsApp App version + the device you are using (A for Android on this case):

47.x.y.z - - [30/Jun/2022:23:05:39 +0000] "GET /learning/http-user-agents HTTP/1.1" 200 6369 "-" "WhatsApp/2.22.13.76 A"

And that's pretty much it - you get the idea on the things you can do. If you have any questions about user agents, let us know!



Posted in   user_agents   http     by trunc_team

Simple, affordable, log management and analysis.