DuckDuckGo Lite now requires a User-Agent header
8 comments
Yeah, but your 2nd example (with the UA header included) also returns a 400 response code. I am able to POST a search query to lite.duckduckgo.com just fine using another tool while also completely omitting the user agent request header (not empty string, completely omit the UA header field). I think your openssl s_client syntax is wrong, and thats why your getting a 400 status code, 400 is usually malformed request syntax.
That's not what causes the malformed request, but thanks for the heads up. Apologies for the inadvertence.
Try this:
If do not want to compile/install daemontools, try something like:
Try this:
addcr <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
POST /lite/ HTTP/1.1
host: lite.duckduckgo.com
content-length: 6
content-type: application/x-www-form-urlencoded
user-agent: test
connection: close
q=test
EOF
If do not want to compile/install daemontools, try something like:
x=$(printf '\r\n');
sed "s/$/$x/" <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
POST /lite/ HTTP/1.1
host: lite.duckduckgo.com
content-length: 6
content-type: application/x-www-form-urlencoded
user-agent: test
connection: close
q=test
EOFNB. If using HTTP/2, then no UA header is needed. The examples I provided use HTTP/1.1
HTTP/2 has some fingerprinting enablers that HTTP/1.1 does not. The above is from the HTTP/2 RFC.
Further reading: https://www.blackhat.com/docs/eu-17/materials/eu-17-Shuster-...
curl --http1.1 --resolve lite.duckduckgo.com:443:52.142.124.215 -d "q=test" -A "" https://lite.duckduckgo.com/lite/
HTTP/2 has some fingerprinting enablers that HTTP/1.1 does not. The above is from the HTTP/2 RFC.
Further reading: https://www.blackhat.com/docs/eu-17/materials/eu-17-Shuster-...
Ok... The above example works now, and with user-agent omitted, I still get search results. My point to others watching/following this thread is that the initial claim is false, lite.duckduckgo.com does indeed return search results if your request header does not have a user agent field.
So to be clear, there is no issue here.
So to be clear, there is no issue here.
[deleted]
What are the exact consequences of this for the end-user compared to no user agent header? Does it only reveal your used browser and version no.?
Is that supposed to imply something to me?
There are a variety of uses that a website operator might make of a User-Agent header.
Today, many of those uses are associated with identifying computer users, usually for tracking and/or advertising purposes, sometimes for access control, e.g., limiting access to only "acceptable" clients that run Javascript.
In this case, lite.duckduckgo.com, which can be used to make DDG search requests via POST instead of GET, has returned search results without requiring a User-Agent header since 2021. This is common. Fun fact: The majority of websites on the internet will return requested pages absent a User-Agent header.^1
Given that DDG is in the business of marketing itself as a privacy solution, it is curious to me why DDG is choosing now to collect more data from its users, expecially an item of data that is frequently (mis)used for purposes that contravene "privacy".
DDG was allegedly founded to "respect users' privacy" and to offer an alternative to "filter bubble" search results. The web search providers that DDG is supposedly competing against make heavy use of the User-Agent header for so-called fingerprinting. Fingerprinting can be used for a variety of purposes, but "respecting user privacy" and avoiding "filter bubbles" are not among them.^2
1. I have been using the web this way for many years now, sending only the minimal HTTP headers. I use non-browser clients and a text-only browser, so I have little need for graphical design, "responsive" or otherwise. As such, I send comparitively less data to web servers than if I was using a popular graphical browser. I avoid graphical browsers not for "privacy" reasons, but due to personal preference, although I welcome any "privacy" benefits that come as a result of this choice.
For folks who are concerned about graphics (cf. text), sending a UA header might be useful. YMMV.
Some amount of "fingerprinting" is inevitable if using the internet. Every remote access, regardless of the software used, has a fingerprint. However not all fingerprints are created equal. Some might be composed of less or more data, and thereby less or detail. Arguably, a more detailed fingerprint is generally higher fidelity. Therefore the more details the fingerprinter can collect, the more useful the fingerprint, e.g., for commercial purposes, and consequently, the larger the loss of "internet privacy".
IMHO, it would be a mistake to conclude that the inability to avoid any amount of fingerprinting (something that has never been possible) means all software is sending the same amount of "fingerpint" data to remote servers and thereby all TCP/HTTP clients, including but not limited to web browsers, are the same with respect to their "fingerprintability". They are not. See, e.g., https://arxiv.org/pdf/1703.05066.pdf
2. The UA header can of course be used for "browser sniffing", but keep in mind that in this case lite.duckduckgo.com is very basic HTML. IMHO, it is not a highly stylised web page. Consider also that UA "spoofing" has been common since the early days of the www and today Google is actvely trying to "deprecate" use of this header in favour of "Client Hints". See RFC 8942.
Today, many of those uses are associated with identifying computer users, usually for tracking and/or advertising purposes, sometimes for access control, e.g., limiting access to only "acceptable" clients that run Javascript.
In this case, lite.duckduckgo.com, which can be used to make DDG search requests via POST instead of GET, has returned search results without requiring a User-Agent header since 2021. This is common. Fun fact: The majority of websites on the internet will return requested pages absent a User-Agent header.^1
Given that DDG is in the business of marketing itself as a privacy solution, it is curious to me why DDG is choosing now to collect more data from its users, expecially an item of data that is frequently (mis)used for purposes that contravene "privacy".
DDG was allegedly founded to "respect users' privacy" and to offer an alternative to "filter bubble" search results. The web search providers that DDG is supposedly competing against make heavy use of the User-Agent header for so-called fingerprinting. Fingerprinting can be used for a variety of purposes, but "respecting user privacy" and avoiding "filter bubbles" are not among them.^2
1. I have been using the web this way for many years now, sending only the minimal HTTP headers. I use non-browser clients and a text-only browser, so I have little need for graphical design, "responsive" or otherwise. As such, I send comparitively less data to web servers than if I was using a popular graphical browser. I avoid graphical browsers not for "privacy" reasons, but due to personal preference, although I welcome any "privacy" benefits that come as a result of this choice.
For folks who are concerned about graphics (cf. text), sending a UA header might be useful. YMMV.
Some amount of "fingerprinting" is inevitable if using the internet. Every remote access, regardless of the software used, has a fingerprint. However not all fingerprints are created equal. Some might be composed of less or more data, and thereby less or detail. Arguably, a more detailed fingerprint is generally higher fidelity. Therefore the more details the fingerprinter can collect, the more useful the fingerprint, e.g., for commercial purposes, and consequently, the larger the loss of "internet privacy".
IMHO, it would be a mistake to conclude that the inability to avoid any amount of fingerprinting (something that has never been possible) means all software is sending the same amount of "fingerpint" data to remote servers and thereby all TCP/HTTP clients, including but not limited to web browsers, are the same with respect to their "fingerprintability". They are not. See, e.g., https://arxiv.org/pdf/1703.05066.pdf
2. The UA header can of course be used for "browser sniffing", but keep in mind that in this case lite.duckduckgo.com is very basic HTML. IMHO, it is not a highly stylised web page. Consider also that UA "spoofing" has been common since the early days of the www and today Google is actvely trying to "deprecate" use of this header in favour of "Client Hints". See RFC 8942.