Sansec logo

Why ordering HTTP headers is important

Sansec

by Sansec Forensics Team

Published in Threat Research − May 02, 2017

RFC 2616 in decay - picture by Olli Homann @ https://flic.kr/p/RyDVkc

RFC 2616 in decay - picture by Olli Homann @ https://flic.kr/p/RyDVkc

If you code against Akamai hosted sites, you could be rejected because your HTTP library sends request headers in the wrong order. In fact, most libraries use undefined order, as the IETF specification says it doesn't matter.

In casu:

$ URL=http://www.bulgari.com
$ UA="User-Agent: Mozilla/5.0 My API Client"
$ ACCEPT="Accept: */*"

$ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP'
< HTTP/1.1 403 Forbidden

$ curl -v -H "$ACCEPT" -H "$UA" $URL |& grep '< HTTP'
< HTTP/1.1 302 Moved Temporarily

My guess: they identified that major browsers send HTTP headers in a specific order, and they implemented this trick to fend off spammers.

Update After some more experimenting, it appears that this behaviour depends on order and the Accept header:

$ ACCEPT="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"

$ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP'
< HTTP/1.1 302 Moved Temporarily

$ curl -v -H "$ACCEPT" -H "$UA" $URL |& grep '< HTTP'
< HTTP/1.1 302 Moved Temporarily

Also, no block without Mozilla/5.0 in the User-Agent.

Conclusion: they will block your request if:

  • Mozilla User-Agent, and
  • */* Accept, and
  • the Accept is sent after the User-Agent header

Update 2 Other sites at Akamai don't expose this behaviour, so it could be a single site issue and/or a configurable setting.

Read more

Scan your store now
for malware & vulnerabilities

$ curl ecomscan.com | sh

eComscan is the most thorough security scanner for Magento, Adobe Commerce, Shopware, WooCommerce and many more.

Stay up to date with the latest eCommerce attacks

Sansec logo

experts in eCommerce security

Terms & Conditions
|
Privacy & Cookie Policy
Company Reg 77165187
|
Tax NL860920306B01