XMLHttpRequest is a JavaScript function that allows the core dynamic data part of AJAX (Asynchronous JavaScript And XML) to work. It lets JavaScript running in a web browser call a URL on the same server and receive data back again to process. Either normal HTML, XML or any format such as JSON can be returned and the JavaScript can do what it likes with it. This lets the page update itself or submit a form without making the whole page refresh which typically gives the user a much better experience. Obvious examples are a news website with up-to-date news ‘ticker’-style headlines or live sports commentary, Viewing more comments in YouTube, Search engine pop-up suggestions as you type, and so on.

The Misuse

“When you have a shiny new hammer, every problem looks like a nail.”

As a way to update a page or get new data according to user actions or a timer this is incredibly powerful. But I see so many sites using it as a way to fill in the initial content during the initial page load. Apart from slowing down the overall page load itself by making additional requests to the server, it gives a jarring wait when it looks like it should have finished but then goes on to display ‘Loading…’ or worse, nothing. I am not writing this to give a flashy solution, it should be obvious for developers. I wrote it to raise awareness that it is a problem.

As an example, and not picking on him in any way, Ben Marsh made a hit website last year that became very popular during this winters snow and I’m sure will again in the future – http://uksnow.benmarsh.co.uk/. As a social content aggregator and public service it worked fantastically and I spent a large amount of time there myself watching the weather move around the UK. But watch what happens when you load/refresh the page… The main part loads, the Google Maps interface comes up and the tweet panel sits there empty with a message at the top saying ‘Updating tweets’… He had to cache the messages on his server to avoid the twitter API locking him out so it is all stored there to use. So why are we waiting? The data can’t have changed significantly in the few seconds since the page loaded so why isn’t the list pre-populated as part of the page load itself? It makes the page extremely slow to load, especially when the server load was very high in the worst of the cold spell.

A worst-case example was a mapping component I saw on an internal government department website. In IE8 and Firefox it cached things and worked reasonably but in IE6, which we are forced to use at work, it reloaded every icon and menu item multiple times (over a thousand requests for a single refresh) after the page load. It sometimes took a full minute to bring the map system up fully… This wasn’t even dynamic content! The map part worked fine once it was working, it was the map layer menu and fixed icons that it was reloading which would have stayed the same for months at a time. It was live for about a week before they had so many complaints that they had to go back to the old ActiveX version which they still use six months later. They also have a site main menu that loads via AJAX after the page load and a few forms that reload several seconds after the page has loaded, resetting the focus and making me type into nothing until I notice half a blank sentence later.

Why it happens

The reason for this, I believe, is a misguided attempt to avoid duplicate code – One block for the AJAX handler and another almost identical on the page load. As developers it is easy to look for ways to re-use code (or just avoid re-writing it again), put two and two together and decide to use the AJAX version for both purposes. Running on a local unstressed web server it is pretty responsive and people may either not test it remotely or work in a company where the production server is also local. This means they never notice how slow and/or visually odd it is for real world users.

Another reason that this may sadly become more common is Googles recent announcement that they may take page load time into account for rankings. Some web spiders (although not Googles I believe) don’t do JavaScript and so will see the empty page as the full load time. Misusing XMLHttpRequest this way will let them get a better ’score’ on some phrase if that search engine adopts the time measurement also.

This does cause another  SEO problem with bots that can’t follow AJAX requests – If the initial data never gets loaded, it can’t be indexed. So no free secondary ’splash’ search engine ranking when a phrase matches something in their content. Plenty of non-techy website owners that use cheap outsourcing facilities would put page rank on a specific page/phrase above this other method of getting free incoming traffic as they simply don’t know about it. The 2nd world ultra-low budget bulk outsourcing firms aren’t going to care which method they use if they can claim to boost pagerank.

How to avoid

Very easily -

Make both your AJAX handler and the main page generating code call a common function which generates the content. Tada! – code reuse and faster, smooth page loads for users.

You may already have a function for it but it outputs XML or JSON. That’s usable too – output it into a JavaScript variable as part of the main page and once the page is loaded call the same JavaScript routine to parse and display it that the XMLHttpRequest callback uses. It’s not quite as smooth that way but still vastly preferable to a return trip to the server + processing time.

Your page size will go up. That is saved by not having to do a second server request so it actually ends up about the same bandwidth used apart from in one case – If your original page can be cached. It’s up to you if the slight bandwidth saving is worth giving your users/customers a worse experience for. If you have an e-commerce site you should already know how fickle a large percentage of customers can be and a very slow page load can be a good enough reason for them to leave your site and go to the next one the search engine suggests.

Conclusion

Test your sites remotely on a stressed server to see it the same way a user does.

There are valid reasons to use XMLHttpRequest this way, mainly if the user expects it not to be instant anyway. Google Maps is a good reason to load content just after the page load as it can’t tell browser dimensions until then.

Are there any others? Have you noticed this as a problem as well or do you think it isn’t worth worrying about?

Leave feedback in the comments if you want.

I do write on blogs for two projects I work on, however both are fairly specific so I wanted one I can ramble on about anything I am interested in or up to. There is no specific theme to this other than general geekery. I’ll list each of my interests below so you have fair warning of what I may post about -

Programming

Nowadays I mostly use PHP for hobbies, but I also like to keep my toe dipped into the C/C++ pond and have VBA inflicted upon me at work sometimes. I like to think I am keeping up with coding standards and methodologies however you wouldn’t know that from reading my code…

I run two small open source projects, one active (Moa, a web-based image gallery) and one currently inactive but still used (Titan, a C++ image loading/saving library). I am working on a third at weekends – a raid DKP tracking system for World of Warcraft, no link yet. And a fourth hopefully launched before the end of the year for a web-based astronomy observation log.

I have always wanted to write games, though other things always get in the way. So no surprise I have one of those in development as well. Only a simple asteroids clone but with a few extras thrown in for good measure. It has been on hold for over a year now due to concentrating on Moa.

Digital Electronics

I mean making devices, not buying them. Modern microcontrollers such as the Microchip PIC and some experience of low-level assembler from my early PC days made this a fairly easy transition from PC programming.

I have two projects so far in this field. First is a camera intervalometer that triggers a camera via infra-red at a set frequency to make time-lapse video (link to my first test). Second is a device I made for my telescope to enable auto-guiding (working) and periodic error correction (semi-working) on my telescope mount with a webcam. I need to rewrite the software for the latter project when I am able to get the scope out again.

I couldn’t do analogue electronics to save my life.

Photography

I got into this as I wanted to do astro-photography. I bought an old film Olympus OM10 SLR camera but ended up using it more for regular pictures than astronomy ones. I upgraded to an Olympus C-5050 Zoom digital compact soon afterwards and now use a Nikon D60 digital SLR. I make no claims to being any good at photography and I have a lot to learn still, but I enjoy it and do get some good pics occasionally. My personal gallery (using Moa) is here.

Astronomy

I took a course at the Open University a few years back in Planetary Science and bought a cheap 30mm refractor from Cash Convertors as I thought it might help. I enjoyed looking at the night sky and trying to find those faint fuzzy galaxies/nebulae so much a few months later I splashed out and got a 6 inch reflector. A 9am start at work with bad weather and heavy light pollution due to living in an estate play havoc with being able to use it often now though. I should be moving in a couple of months though and a ground floor flat with dark(ish) garden will be high on my list of requirements. I use a webcam to take pictures now, being far more convenient than an SLR. It is low-resolution and grainy, but I get some OK pics considering its those limitations (gallery).

Gaming

You think I have time left over to play games as well! Well, OK I do. Mainly World of Warcraft a few nights a week and Battlefield 2 about the same. So much crap is pumped out these days though that I don’t play many other modern games. When I get time I like to play the old classics that blow a lot of todays games out of the water – XCOM and Terror From the Deep, Thief, Final Fantasy 7-9, Elite, Civilisation, Wizball, Head Over Heels, Sim City and sometimes even The Sims. Two new games I am looking forward to when they are eventually released are Thief 4 and Half Life 2: Episode 3.

I would also like to get into paper and pen role playing games, however convincing friends that they want to as well is harder than I thought it would be. I’ve never played D&D but would like to try. I have played both Heroes Unlimited and Call of Cthulhu and would happily play both again regularly. I am still surprised with things like Skype, Social Media and much more Internet-savvy people around that there isn’t a app to do this and let people find other like-minded players rather than being pre-arranged amongst friends. There is one simple one I found but was very dated and clunky to use. Maybe I should launch a fifth open source project as well…

Technology

I like trying new things, within the range of my wallet at least. And so when possible I try new software and hardware to see how things work. I try not to follow the crowd, not out of stubborness but because someone else will always get one I can have a play with instead. One thing seems to be universal though – Manufacturers are appear to be very ignorant of what people will actually be doing with their device/software and the usability suffers a lot.

In theory I use Linux as my desktop OS. In reality I spend about half my time in Windows due to Linux just plain not being ready for full-time use yet. More about that to come in a post shortly.

Fiction (mainly Science Fiction and Horror)

I like to read hard sci-fi and get both Analog and Asimovs magazine most months. Novels as well of course and I am currently re-reading Red Mars by Kim Stanley Robinson. Most TV and movies are soft-sci-fi though and rightly so or their ratings would plummet. They do tend to end up getting trapped in formulaic ruts or using the sci-fi technology as a replacement for a good plot though. Some of the supposedly geek fan-favourite sci-fi classics I couldn’t stand though, Enders Game particularly, also Dune and Snow Crash.

Horror I like also, although a good modern horror novel seems hard to find now. Back in school during the 80’s it was great with James Herbert, Shaun Hutson, Steven King, etc bringing new books out fairly regulary. So I tend to re-read those or go further back in time to the great grand-daddies of them all – HP Lovecraft and Edgar Allen Poe.

I do read other genres as well. I like the Harry Potter series even though I generally don’t like fantasy. The Beach, Fight Club, On the Road, Fear and Loathing in Las Vegas, etc are all great books. One plot-type that I especially like, regardless of genre, is the castaway. Not sure why but it seems to be a recurring theme in some of my favourite books/movies.

Science Fact

I learn for pleasure. Always have done and so perhaps it isn’t surprising that I have taken a few courses and try and keep up to date in stuff I want to know more about. So far with the Open University I have done courses in Planetary Science, Oceanography, Geology, Optics and Artificial Intelligence. Not for any reason or qualification, just to know more about the subject.

One thing that I may write about is conspiracy theories. I don’t believe any of them personally, but I find the mindset behind them bizarre. 9/11, Moan Hoax, Climate Change Denial, New World Order, Reptilians, Area 51, etc. All demonstrably bullshit yet people still actively find excuses and even make stuff up just so they can carry on believing.

Travel

Something I wish I had the opportunity to do more often. My last four holidays have been to Thailand and Japan, twice each. All have been with backpacks staying in hostels and guest houses and I wouldn’t trade them for a hotel/resort holiday for anything. For me the purpose of the holiday is to see other countries/cultures not just get a tan, get drunk and come home with some souvenirs.

I am currently planning and saving  for a 5-6 month trip around the world probably to be taken in the summer of 2011. None of this ‘fly around the world stopping in 3-4 countries  and say you have travelled around the world’ crap either.  Over land/sea from the UK through Europe, across Russia, Mongolia and into China, down through southeastern Asia to Australia. Then New Zealand over to South America and up by bus into the US and across to the east coast. It does involve four flights I can’t really avoid though – Bali->Darwin, Sydney->Christchurch, Auckland->Santiago and Boston->Glasgow but I worked out the land-based travel is in-excess of 30000 miles, 10000 more for the flights. I did hope to do it this year but work issues mean it would be better put off for now. Thinking of spending 3 weeks or so InterRailing in Europe during October at the moment for this years holiday.

About me

Not a subject I will write about, but for info.

I am 33, male and live near Aldershot in the UK. I work for a geotechnical department of a global company in Woking doing none of the above although occasionally get to do some Visual Basic coding within Microsoft Access.

Dan Brown (no, not that one)