www.sciencedirect.com is slow on Linux (tcp window scaling)

December 2010


Table of Contents

The Symptoms
The First, Somewhat Rigorous Solution
The background and the drawback
The Somewhat Subtler Solution

The initial symptom was: using a 64-bit Ubuntu Linux PC, fetching any articles from www.sciencedirect.com is very slow, while on Windows XP it works normally.

Trying to quantify 'slow' and 'normally' is somewhat difficult: every third or so fetch of the index page takes minutes instead of seconds, the same goes for articles. This may be a measure against NATting sites or just against DOS, but it makes timing hard. And when wget is used to more precisely time fetches, the site throws an HTTP error 400 (bad request) unless the user agent is either Internet Explorer or Firefox.

On the site itself it says in addition that only Windows is being supported. So it looks like our computer scientists will have to put aside their Macs and Linuxes and convert to Windows to please Elsevier if they want to do any reading.

So we'll have to make do with some rather inaccurate measurements: fetching random articles typically completes in minutes.