How to Improve Loading Time with basket.js

For modern Web pages, scripts are obviously a crucial element. Their weight is growing in terms of importance and the amount of bytes. For instance, think of single page applications and the amount of code they need to give us all the features and animations required, without having to hit the Reload button on our browser. The total size of scripts in a page is far from the one of video and audio files, but it’s getting closer to the size of images. In this article, I’ll test the performance of two of the most visited websites, Facebook and Twitter, highlighting some of their major issues. Then, I’ll focus on how to improve the loading time of the scripts of a page by introducing you to a library called basket.js.

Measuring the Performance of Facebook and Twitter

Before covering what basket.js is and what problem it solves, let’s start with a real-world example. Let’s use a tool like the Chrome Developer Tools or F12 Developer Tools to measure the number of bytes transferred and the loading time of the Facebook wall. To perform this task in a reliable way, you have to hard reload the page in your browser and empty the cache, of course. If you need more instructions, this tweet by Addy Osmani will help you. Said that, what’s the total size of scripts loaded for the average Facebook wall (ballpark) and what’s the ratio images/scripts? And regarding the challenge of CSS versus JavaScript, which one is going to be heavier? Take a guess and write your answers down but… don’t cheat! Let’s take a look at the results: And here you can find a summary of them:

• Scripts: Total size: 785 KB, Number of requests: 49, Total sum of loading time: ~9s
• CSS: Total size: 125 KB, Number of requests: 18, Total sum of loading time: ~2.2s
• Images: Total size: 251 KB, Number of requests: 68, Total sum of loading time: ~8s

Keep in mind that files are loaded in parallel (up to a given limit), thus the loading time for the page is smaller than the sum of the time to download individual files. Let’s see another example by taking a look at the Twitter timeline:

• Scripts: Total size: 417 KB, Number of requests: 4, Total sum of loading time: ~650ms
• CSS: Total size: 114 KB, Number of requests: 3, Total sum of loading time: ~200ms
• Images: Total size: 647 KB, Number of requests: 36, Total sum of loading time: ~3.5s

Although the Twitter approach to minification looks different, the size of the scripts is still close to the sum of the size of all the loaded images. At this point, you might think: “What are you talking about? It’s just less than 1 MB, we shouldn’t even worry about that!”. Undoubtedly, on a broadband connection (or even a 4G one), the latency for loading the scripts could be (almost) negligible. However, the situation isn’t the same in all the countries. In many of them, no broadband connection is available outside of the urban areas. In Italy, for example, in the countryside you might find yourself stuck with a 56K modem, and last generation mobile connection has become a reality only in recent times. Although Italy doesn’t cover a big chunk of the market (“only” ~60 million potential users), some bigger countries are affected by the same issues. According to Akamai “State of the Internet” reports, in India the vast majority of the population doesn’t have access to a fast connection. In addition, according to the same report, Brazil is one of the countries having the lowest average connection speed. Based on this discussion, you can understand that caching scripts it’s a good idea. basket.js tackles this problem for scripts, both statically and dynamically loaded, storing them into the browser’s local storage. It also allows fine-grain control of caching and its expiration time. You might object that browser cache already takes care of that, and you’d be right. However, local storage is faster and this is especially important on mobile devices. We’ll deepen this topic in the following sections, but the fact that Google and Microsoft are using this technique might already give you a good reason to read this article.

What’s Basket.js

As stated on its website, basket.js is a small JavaScript library supporting localStorage caching of scripts. This quote summaries very well the aim of this project. Once the library is loaded into the memory, then it sends asynchronously requests to retrieve the other scripts needed by the page. It injects them into the document and then caches them into the browser’s local storage. Doing so, the next time the page is loaded, the scripts will be loaded locally without performing any HTTP request. Recalling the Facebook examples above, it means that you’ll save 49 HTTP requests, almost 800 KB, and a total (summed) loading time of ~9 seconds (on a broadband connection! You can reasonably expect this to be much slower on a 56K one).

LocalStorage vs Browser Cache vs Indexed DB

As mentioned before, research brought on by Google and Microsoft concur that localStorage is much faster than browser cache. On SitePoint we have covered this topic recently with the article HTML5 Local Storage Revisited, where Luis Vieira also covered some of the limitation of localStorage. In addition IndexedDB is (surprisingly) slower than localStorage, both for reading and writing. Obtaining exact measurements is quite challenging, and at the moment no extensive research is available – although it is one of the priorities for the project.

How to Use basket.js

Using the library is really simple. It provides four main methods:

• basket.require(): require remote scripts and inject them into the page (with or without caching them)
• basket.get(): inspect localStorage for scripts
• basket.remove(): remove a cached script
• basket.clear(): remove all cached scripts

Require Scripts

To require a script, we can write a statement like the following:

basket.require({ url: 'jquery.js' });

This method can be used to require one or more scripts with one call. It takes a variable number of arguments, one object for each script. You can also pass fields for the scripts’ URL and a few options for each script. The call always return a promise. This promise is fulfilled once the script is loaded, or rejected on error. This is convenient for several reasons:

• it becomes easy to handle dependencies using a promise chain to set the loading order
• it is possible to handle when scripts can’t be loaded, and hence fail gracefully
• as a plus, you can cache a file without executing it on load – you’ll be able to retrieve it with .get() at a later point, if you actually need it

The options that can be passed to the script allow to set:

• an alias to reference it
• if the script has to be execute once loaded
• the amount of hours after which the script will expire or…
• …if it has to skip cache altogether.

Handling Dependencies

In case none of your scripts has dependencies, you can simply require them all at once:

basket.require(
    { url: 'jquery.js' },
    { url: 'underscore.js' },
    { url: 'backbone.js' }
);

Otherwise, basket.js‘ promise-oriented API makes your life easy:

basket
    .require({ url: 'jquery.js' })
    .then(function () {
        basket.require({ url: 'jquery-ui.js' });
    });

Fine Grain Script Cache Expiry Management

As mentioned above, scripts can be kept out of cache individually, or expiry time can be set for each of them separately.

basket.require(
    // Expires in 2 hours
    { url: 'jquery.js', expire: 2 },
    // Expires in 3 days
    { url: 'underscore.js', expire: 72 },
    // It's not cached at all
    { url: 'backbone.js', skipCache: true },
    // If you later change this value the older version from cache will be ignored
    { url: 'd3.js', unique: 'v1.1.0' }
);

Manually Clearing the Cache

You can remove a single item from the cache:

basket
    .remove('jquery.js')
    .remove('modernizr');

Or, you can remove only the expired items, all at once, without explicitly listing them

remove basket.clear(true);

Finally, it’s also possible to clear all the scripts for your page:

remove basket.clear();

Manually validate items in cache

You can even provide your own custom function to validate items in cache and decide when to mark them as stale. You can overwrite basket.isValidateItem with a function that returns true when the cached item is valid, and false when the script has to be loaded from source again. This does not overwrite existing check for the expiry and unique options, but adds on top of it. Moreover, even if overwriting isValidateItem is a powerful option, it’s unlikely you will really need it, ever.

Hands-on: Let’s build an Example

I used basket.js to refactor scripts loading for TubeHound, replacing RequireJS as a script manager. This is what the main script header looked like before:

requirejs.config({
    "baseUrl”: "js/",
    "paths": {
      "jquery": "./lib/jquery-2.0.3.min",
      "Ractive": "./lib/Ractive",
      "utility": "utility",
      "fly": "./lib/Ractive-transitions-fly",
      "fade": "./lib/Ractive-transitions-fade",
      "bootstrap": "./lib/bootstrap.min",
      "jquery-ui": "./lib/jquery-ui-1.10.4.custom.min",
      "jquery-contextmenu": "./lib/jquery.contextmenu"
    },
    "shim": {
      "jquery": {
        exports: 'jquery'
      },
      "Ractive": {
            exports: 'Ractive'
      },
      "utility": {
            deps: ['jquery'],
            exports: 'utility'
      },
      "bootstrap": {
            deps: ['jquery'],
            exports: 'bootstrap'
      },
      "jquery-ui": {
            deps: ['jquery'],
            exports: 'jquery-ui'
      },
      "jquery-contextmenu": {
            deps: ['jquery'],
            exports: 'jquery-contextmenu'
      }
    }
});

require([
  'jquery',
  'Ractive',
  'utility',
  'bootstrap',
  'fly',
  'jquery-ui',
  'jquery-contextmenu',
  'fade'
], function ($, Ractive, utility) {
  ...
});

Now I removed all of that except for the function declaration, stripped of all its arguments. Then I added a new small script called loading.js:

(function () {
  function requireScriptsDependingOnJQueryAndRactive () {
    return basket.require(
      { url: 'js/lib/bootstrap.min.js'},
      { url: 'js/lib/Ractive-transitions-fly.js', key: 'fly' },
      { url: 'js/lib/Ractive-transitions-fade.js', key: 'fade' },
      { url: 'js/lib/jquery-ui-1.10.4.custom.min.js', key: 'jquery-ui' },
      { url: 'js/lib/jquery.contextmenu.js', key: 'jquery-contextmenu' },
      { url: 'js/utility.min.js', key: 'utility', unique: 1 }
    );
  }

basket.require(
    { url: 'js/lib/jquery-2.0.3.min.js', key: 'jquery' },
    { url: 'js/lib/Ractive.js', key: 'Ractive' }
  ).then(requireScriptsDependingOnJQueryAndRactive)
  .then(function () {
    basket.require({ url: 'js/thound.min.js', unique: 1 });  //unique is to make sure we can force a reload, in case of bugs
  });
}());

This is now loaded via the <script> tag in the HTML page (right after basket.js):

<script src="js/lib/basket.min.js"></script>
<script src="js/loader.js"></script>

I performed a similar refactoring for utility.js. Before, there was some plumbing needed by RequireJS:

requirejs.config({
    "baseUrl": "js/",
    "paths": {
      "jquery": "./lib/jquery-2.0.3.min"
    },
    "shim": {
      "jquery": {
        exports: 'jquery'
      }
    }
});

define([
  'jquery'
], function ($) {
        "use strict";
        ...
});

After, I “export” the module using a global variable as shown below:

var utility = (function () {
    "use strict";
    ...
}());

Performance

Let’s get to the grain: how much of an improvement did I get? Here it is the baseline, a hard reload of the existing page: It took 6.06s to download 904KB with 28 requests. Next, I hard-reloaded the new version of the page, and measured again: Since it’s loading the page from scratch, all the scripts are loaded via HTTP requests. It took 4.01s to download 899KB with 27 requests (requireJS was left out and replace by basket.js). At this point, when you hard-reload the page again, everything is flushed from browser’s cache but scripts are kept in localStorage: the delta will measure the actual gain provided by caching scripts. The result is: 2.01s to download the 352KB needed with 18 requests. So for a page that is JS-intensive you actually have a pretty good saving. Finally, let’s see the final loading time for a normal access to the homepage: Using browser cache and basket.js, the page can be loaded in 771ms, and only 5.3KB are actually loaded (17 requests, mostly served from cache).

Conclusions

This library is a good idea, with the one flaw of relying on a less than perfect data API. The considerations that led to choosing localStorage are totally understandable. It’s trying to improve performance, and experience has shown that localStorage is the fastest solution available. On the other hand, as Donald Knuth loves to say, “premature optimization is the root of all evil”! Without extensive and rigorous performance comparisons it’s hard to weight the limitations caused by quota restrictions. Unfortunately issues with localStorage are not going away any time soon, at least not for Chrome, where augmenting the quota would require some non-trivial rewriting. The good news is that basket.js‘ authors are considering several alternatives, including a tiered solution that will try to use the best persistence API available on the browser: Service Workers, Cache API (on Chrome), or FileSystem API. I was a bit surprised to see that Service Workers were not initially considered, but apparently this is going to change soon. And, even better news, there are a number of emerging libraries working on similar ideas from different angles. Shed, for instance, looks promising: an even broader range solution that make Service Workers super easy to use. A few problems I could touch first-hand (and get burned by) when trying to use it on a real project:

1. Feedback can be largely improved: it is hard to tell what’s going on when it fails to load your scripts. If you are lucky, you might see some kind of errors, but the messages is far from meaningful. For example, I was passing an actual array to the require() method: all I got was a generic TypeError from the lib’s code, so it took me a lot of trials and errors to realize my mistake.
2. Now if you are not lucky: a script might not be loaded at all because you have a typo (f.i. basker.require) inside a callback along the promises chain. Since your error message gets swallowed, it will take you some time to realize it.
3. In case you have dependencies for your scripts, you lose the clear declarative syntax you have using RequireJS, where you can list dependencies for each of them.
4. On the other hand, you have all of your dependencies listed and ordered in a single place. And, let’s face it, RequireJS is a little verbose and redundant.
5. Once they are cached, the scripts loaded asynchronously won’t show up in the Network panel of Chrome development tools (nor Firefox’s). Moreover, you won’t see them listed in the sources even when they are loaded from the remote source. This makes debugging a bit harder, but can be worked around during development, if you use basket.js only in production, when optimization is needed.

Truth to be told, this project hasn’t hit version 1 and it is clearly stated it is just an experiment, for now. And indeed, basket.js is a very promising idea, and the results looks really good – but my take is it needs a little extra step to be ready to be used for the development of a complex project – or in production for a huge page. (This would be true for any project that has not reached version 1, due to possible changes in its interface/structure). For a small-to-medium project, instead, this could be a nice way to cut your users’ loading time and frustration. I for one will keep an eye on it, and I’ll be happy to champion its adoption, as soon as the project reaches maturity.

Frequently Asked Questions about Improving Loading Time with Basket.js