HTTP User agents

User agents are one of the most interesting parts of the HTTP protocol. It has been there since the beginning and is used to identify the browser (or software doing the request) to the web server.

They are often misunderstood, misused, and more complicated than what it should be, but very useful from an analytics and security perspective.

In this article, we will talk more about user agents, and how the major browsers use it to identify themselves.

What are user agents?

If you are not familiar with the HTTP protocol, the USER AGENT is a header sent during the HTTP request that tells the server what kind of application (agent) is connecting to it.

If you look at the old HTTP/1.1 protocol, that's how a request would look like under the hood:

GET / HTTP/1.1
Host: trunc.org
User-Agent: MY_TEST_USER_AGENT

And on your logs, you would see that the web server accepted the request for the "/" URL with the user agent "MY_TEST_USER_AGENT":

192.168.1.1 - - [26/Jun/2022:00:45:53 +0000] "GET / HTTP/1.1" 301 409 "-" "MY_TEST_USER_AGENT"

The client software specifies what the user agent looks like, in pretty much any format they choose*. For example, if I use the popular curl software to connect to a URL:

$ curl 127.0.0.1/
$ tail -n 1 /var/log/apache2/access.log
127.0.0.1 - - [25/Jun/2022:17:46:53 +0000] "GET / HTTP/1.1" 403 344 "-" "curl/7.81.0"

It uses the "curl/7.81.0" USER AGENT to tell my server (at 127.0.0.1) that Curl version 7.81.0 was used to make the request.

*Different RFCs provide different guidelines (or recommendations), but they are not enforced.

Setting the User Agent

Since the USER AGENT is defined by the client software, you can set it to anything you want. For example, using curl, you can pass the "-A" argument to set the user agent. If we want to tell the server the request came from an iPhone, we could do:

$ curl -A "Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1" 127.0.0.1
$ tail -n 1 /var/log/apache2/access.log
127.0.0.1 - - [25/Jun/2022:18:18:34 +0000] "GET / HTTP/1.1" 403 344 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Mobile/15E148 Safari/604.1"

And the user agent would mimic the iPhone one. Some times I like to change my own *browser USER AGENT to something arbritary to see if anything would break. And often gmail, web.whatsapp and other sites block access because I am using an "old or unsupported browser".

*To change the USER AGENT on Firefox, you need to type about:config and set "general.useragent.override" to the new user agent you want to use.

User Agents used by all the major browsers

All the major browsers use a well-formatted USER AGENT that help us identify the browser being used.

Mosaic Browser

Mosaic is a very popular browser and the only one to support images. If you want to display images on your server, you can parse the USER AGENT and look for NCSA_Mosaic, which is how it sets the USER AGENT. For example, on Windows 3.1, this is how it looks like:

NCSA_Mosaic/2.0 (Windows 3.1)

Ok, we are joking on Mosaic, but had to include it in there. Mosaic is a very old browser, the first one ever invited that supported web sites like they look today. If you want to have some fun online, try changing your USER AGENT to:

NCSA_Mosaic/2.6 (X11; SunOS 4.1.3 sun4m)

or

Mozilla/2.0

And see what breaks. The Mozilla one was used by Netscape 2.0 (Mozilla means Mosaic Killer - a good old Internet competition).

Now to the real browsers...

Chrome on Windows 8, 10 and 11

Chrome on iOS

Chrome on MacOS

Chrome on Android Pixel 6, Pixel 2

Chrome on Chromebook

Microsoft Edge Browser on Windows 11

Microsoft Edge on MacOS

Firefox on Win 10 and 11

Firefox on MacOS

Firefox on Linux

Safari on iOS

Safari on MacOS

Brave Browser on MacOS

Opera Browser on Windows

DuckDuckGo Browser on Android and iPhone

Other interesting USER Agents

There are some other fun things you can look at the USER AGENT. For example, if someone is using Slack and sharing a link there, you would see this USER AGENT:

Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)

And funny enough, if they are using Signal, you would see the "WhatsApp" user agent:

47.x.y.z - - [30/Jun/2022:23:00:02 +0000] "GET /learning/http-user-agents HTTP/1.1" 406 5386 "-" "WhatsApp/2"

With the IP address of the person sharing the link. Those can be useful if you are trying to track engagement - or even as a honeypot if you share a secret link with someone - to see if they will share anywhere else. Note that if you paste a URL on Twitter, the Twitterbot would also contact your server:

199.59.150.181 - - [30/Jun/2022:22:59:50 +0000] "GET /learning/http-user-agents HTTP/1.1" 200 10633 "-" "Twitterbot/1.0"

And if you share a link on WhatsApp, you get the "WhatsApp" user agent, like signal, but sharing your real WhatsApp App version + the device you are using (A for Android on this case):

47.x.y.z - - [30/Jun/2022:23:05:39 +0000] "GET /learning/http-user-agents HTTP/1.1" 200 6369 "-" "WhatsApp/2.22.13.76 A"

And that's pretty much it - you get the idea on the things you can do. If you have any questions about user agents, let us know!



Posted in   log-guide   user-agents   http     by Daniel Cid (@dcid)