How Asset Precompile Works, Part II

This is part 2 of How Asset Precompile works in Rails. In part 1 we started digging into Rails built-in support of packaging assets, how it compiles static assets (images), and how a digest is generated based on their content etc. This article contains a bit more in-depth coverage of the Rails asset pipeline.

Types of Assets

If you remember, there are three types of assets according to Sprockets

• Bundled Assets
• Processed Assets
• Static Assets

We covered about Static Assets in part 1. Bundled Assets are assets which require some processing in-order to get created. Assets such as CSS files and Javascript files are example of Bundled Assets. Processed Assets are parts of Bundled Assets. In simple words, a Bundled Asset is composed of different Processed Assets and is used to store the file after processing is taken from Bundled Asset.

For example, application.js is a Bundled Asset at first and it consists of different Processed Assets (including itself) which are defined in it according to the Sprockets structure. Bundled Assets are saved on disk and contains contents of Processed Assets. Bundled Assets serve as a container for different Processed Assets.

Now, let’s deep dive and see how Rails Asset Precompile accomplishes this work. For this article default application.js is used as a reference asset and article focuses on how application.js gets precompiled.

Before proceeding, please open following files in your favorite editor. If you like, you can click below to view them in the browser.

• actionpack-3.2.15/lib/sprockets/railtie.rb
• actionpack-3.2.15/lib/sprockets/assets.rake
• actionpack-3.2.15/lib/sprockets/static_compiler.rb
• sprockets-2.2.2/lib/sprockets/base.rb
• sprockets-2.2.2/lib/sprockets/index.rb
• sprockets-2.2.2/lib/sprockets/bundled_asset.rb
• sprockets-2.2.2/lib/sprockets/processed_asset.rb
• sprockets-2.2.2/lib/sprockets/context.rb
• sprockets-2.2.2/lib/sprockets/directive_processor.rb

rake assets:precompile

To precompile assets we type rake assets:precompile. This is a rake task, and is added in actionpack-3.2.15/lib/sprockets/railtie.rb at line 13 where the sprockets/asset.rake file is loaded. This code is in Sprockets::Railtie which is a subclass of Rails::Railtie. As discussed in part 1, that Railtie is the core of the Rails framework and provides several hooks to extend Rails and/or modify the initialization process. It is also used to add rake tasks.

In actionpack-3.2.15/lib/sprockets/assets.rake at lines 59-67, there is a rake task. This is the rake task which gets called when we run rake assets:precompile. This rake task, in turn, call another rake task, which calls the internal_precompile method.

In internal_precompile at line 50 an instance of Sprockets::StaticCompiler is created by passing different options and assigned to compiler. The env that is passed to Sprockets::StaticCompiler is an instance of Sprockets::Environment which was assigned to Rails.application.assets in actionpack-3.2.15/lib/sprockets/railtie.rb at line 23 (Which we discusses in Part 1.)

internal_precompile calls the compile on Sprockets::StaticCompiler. This is the method that performs all necessary operations to precompile our assets and write them to disk.

Let’s head to the compile method in actionpack-3.2.15/lib/sprockets/static_compiler.rb. In this method you will see the following line:

env.each_logical_path(paths) do |logical_path|

env is Rails.application.assets which is instance of Sprockets::Environment. In the above line paths is basically Rails.application.config.assets.precompile which was passed to Sprockets::StaticCompiler in actionpack-3.2.15/lib/sprockets/assets.rake.

each_logical_path is defined in sprockets-2.2.2/lib/sprockets/base.rb at line 332. This method, in turn, calls each_file which is defined just above each_logical_path.

each_file performs an each on paths. paths is list of directories which contain our assets. The paths for a newly created Rails 3.2.15 application in which you haven’t added any extra gems might look like this.

/home/ubuntu/Desktop/work/asset_pipeline_article/app/assets/images
 /home/ubuntu/Desktop/work/asset_pipeline_article/app/assets/javascripts
 /home/ubuntu/Desktop/work/asset_pipeline_article/app/assets/stylesheets
 /home/ubuntu/Desktop/work/asset_pipeline_article/vendor/assets/javascripts
 /home/ubuntu/Desktop/work/asset_pipeline_article/vendor/assets/stylesheets
 /home/ubuntu/.rvm/gems/ruby-1.9.3-p194@exporter-imranlatif/gems/jquery-rails-3.0.1/vendor/assets/javascripts
 /home/ubuntu/.rvm/gems/ruby-1.9.3-p194@exporter-imranlatif/gems/coffee-rails-3.2.2/lib/assets/javascripts

Your absolute paths will be different based on your Rails application path. The first 5 paths are from app/assets and vendor/assets. The last two paths are from the jquery-rails and coffee-rails gems, respectively.

If you are a gem author and want to include your gem’s assets paths in Rails.application.config.assets.paths, then you need to create an engine class that inherits from Rails::Engine. This is described here. paths are first assigned to Rails.application.config.assets.paths and later assigned to Rails.application.assets.paths. You can check the code that assigns Rails.application.config.assets.paths to Rails.application.assets.paths here.

Back to each_file in sprockets-2.2.2/lib/sprockets/base.rb. each_file performs each on paths and in that block each_entry is called.

each_entry is defined just above each_file. each_entry recursively iterates over each directory and collects all files in a that path. After collecting all files from a given path it executes the block passed to it. You can see following line in each_entry method:

paths.sort_by(&:to_s).each(&block)

The above line calls the block which was passed to the each_file method from each_logical_path. That block calls logical_path_for_filename, which is defined in the same file.

logical_path_for_filename calls matches_filter which, also defined in the same file. matches_filter matches filename according to the filters passed to it. filters are, basically, Rails.application.config.assets.precompile, an array containing different filters according to which an asset should be processed.

By default, the array has two items. First, a Proc that matches the name for Static Assets. The second item contains a RegExp object that matches CSS and Javascript files.

If a filename is not matched, then there will be no asset for it. For Static Assets, like images, it should match the filter defined by the proc in the filters array. For Bundled Assets, the filename should match the RegExp defined in filters.

If we want to bundle CSS or Javascript files that do not match the regular expression, then we have to add it to Rails.application.config.assets.precompile on our configuration files. We do this in config/environment/production by using following line:

config.assets.precompile += %w( sitepoint.js )

The proc in the filters array only matches Static Assets or assets which don’t have a .css or .js extension. The RegExp in filters only matches files which are either application.css or application.js. Since “sitepoint.js” does not match either of these criteria, it won’t be bundled. Appending sitepoint.js to Rails.application.config.assets.precompile forces a match and bundling of the asset will be performed.

After getting a successful match for the filename passed to the filters_matches method, logical_path_for_filename returns that result to the block passed to each_file from each_logical_path. This block yield the filename to another block passed to it from compile in actionpack-3.2.15/lib/sprockets/static_compiler.rb file. That block calls find_asset defined in sprockets-2.2.2/lib/sprockets/index.rb. This method sets options[:bundle] to true and calls find_asset from its Base class in sprockets-2.2.2/lib/sprockets/base.rb.

find_asset resolves the path based on the filename and assigns absolute path of file to pathname.

logical_path is just name of the file i.e. application.js. find_asset calls build_asset, which decides how to handle the asset. If there are no processors to run then it treats it like a StaticAsset.

If there are some processors to run, then it treats theasset as a BundledAsset. The first time build_asset is called, options[:bundle] is true, so a BundledAsset is created. find_asset is called again with options[:bundle] set to false, so a ProcessedAsset is created.

Remember our discussion that BundledAsset is actually saved to disk containing different ProcessedAsset instances. Every BundledAsset has at-least one ProcessedAsset. In the case of a single CSS or Javascript file, such as sitepoint.js, both the BundledAsset and ProcessedAsset point to the same file.

The initializer for ProcessedAsset executes context.evaluate(pathname) to get the contents of the file in pathname. evaluate in sprockets-2.2.2/lib/sprockets/context.rb performs different operations on the files mentioned in path. It gathers processors to run on the file.

By default for Javascript file, the two processors are Sprockets::DirectiveProcessor and Sprockets::SafetyColons. Sprockets::DirectiveProcessor is used to run a directive processor on each CSS and Javascript file. It basically scans the CSS or Javascript file and captures files that are required in that file according to the Sprockets syntax. Popular directives are require, require_tree, requite_self etc.

evaluate runs each on the processors array and a new instance of the processor is created. The render method is then called on each processor template. Sprockets::DirectiveProcessor is defined in sprockets-2.2.2/lib/sprockets/directive_processor.rb and it is a subclass of Tilt::Template. When render is called on Sprockets::DirectiveProcessor it also calls evaluate on Sprockets::DirectiveProcessor. In evaulate, process_directives is called, which scans files and gathers directives that needs to run on the current file.

For our reference asset application.js result might look like this:

[[13, "require", "jquery"], [14, "require", "jquery_ujs"], [15, "require_tree", "."]]

After getting directives and assigning them to @directives, process_directives is executed. As you can see, this line of code sends a process_<name>_directive message to each directive:

send("process_#{name}_directive", *args)

All these methods belongs to Sprockets::DirectiveProcessor. Let’s see what happens when process_require_tree_directive is called.

process_require_tree_directive calls each_entry, which we now know returns paths of all files from path passed to it. require_tree means that all the files in the given path should be required. You might have noticed, in the each_entry block, that if a filename matches the current filename, then it is skipped with the next statement. What is the purpose of this statement? We will get back to this later.

context.require_asset is used to assign assets to the @_required_paths array. @_required_paths are paths of assets on which the current ProcessedAsset depends. After process_directives completes, process_source is called, which captures the source of the current ProcessedAsset by removing the directive processors.

After evaluate of all processors completes, source of the current ProcessedAsset is returned and saved to @source in ProcessedAsset.

Then, build_required_assets from sprockets-2.2.2/lib/sprockets/processed_asset.rb is called. In build_required_assets you will see following code:

@required_assets = resolve_dependencies(environment, context._required_paths + [pathname.to_s]) -
      resolve_dependencies(environment, context._stubbed_assets.to_a)

resolve_dependencies is used to resolve the dependencies of the current ProcessedAsset. Until this point, we only have paths of files that are required for our current ProcessedAsset. For application.js asset context._required_paths might look like this.

{{GEMPATH}}/jquery-rails-3.0.1/vendor/assets/javascripts/jquery.js
{{GEMPATH}}/jquery-rails-3.0.1/vendor/assets/javascripts/jquery_ujs.js

{{GEMPATH}} represents absolute path of gems directory which holds different gems. You have noticed that in above paths there is no application.js which should be there because of require_tree directive. Take a look at the parameters passed to resolve_dependencies. We are explicitly appending the current pathname to the list of context._required_paths. Remember, from our last discussion that, in process_require_tree_directive we skip the current file.

As we discussed, a BundledAsset will have at least one ProcessedAsset. By default, our files don’t include any processors to run, so if we want it’s path to be included in context._required_paths we have to add a directive processor to each file which is obviously not ideal. We picked the other path. Instead of including the path of the file during directive processing, we included it explicitly to context._required_paths while calling resolve_dependencies from build_required_assets. If there are some directive processors to run on the file, then we skip including that filename from there, because we know we are adding it manually during the call to resolve_dependencies.

resolve_dependencies loops on all paths that are passed to it and, if it matches the path of the current ProcessedAsset, then it is appended to assets array. If the filename doesn’t match, the elsif statement is executed calling the find_asset method.

To collect the source and run processors on any file, we need to create an instance of ProcessedAsset. The jquery.js file is one of these paths. Let’s see how it is recursively processed and appended to the assets array for the ProcessedAsset instance of application.js. Starting with the elsif in resolve_dependencies you will see following line

asset = environment.find_asset(path, :bundle => false)

As previously mentioned, when we call find_asset by setting bundle to false, it means that we are creating an instance of ProcessedAsset. A ProcessedAsset is created for jquery.js in the elsif of resolve_dependencies.

Remember, the initializer of ProcessedAsset calls context.evaluate which performs different operations on the asset. jquery.js is a JavaScript file and doesn’t contains any dependencies so no dependencies will be added to it by calling context.require_asset. After the processors have been run on jquery.js, resolve_dependencies is called in the same way as it was called for application.js a few steps earlier.

Since jquery.js doesn’t depend upon any asset, context._required_paths is empty and we are explicitly adding the current pathname and passing the array to resolve_depedencies. There is only one dependency of jquery.js, which is jquery.js itself. The if statement is true and the current ProcessedAsset instance which we can refer as self is added to the assets array, which is assigned to @required_assets in build_required_assets. For those assets that don’t have any dependencies and are part of some BundledAsset, @required_assets points to one item which is the asset itself.

After creating an instance of ProcessedAsset for jquery.js and building it’s required assets, the control returns back to the elsif in resolve_depedencies which is building the required assets for application.js. The ProcessedAsset instance of jquery.js is assigned to asset in resolve_depedencies. We then loop through required_assets of jquery.js and append an instance of ProcessedAsset to the assets array.

These steps are executed for jquery_ujs.js and other paths too. After appending instances of ProcessedAsset for all the paths of application.js to the assets array for application.js, we return that array from resolve_dependencies method which is assigned to instance variable @required_assets of application.js‘s ProcessedAsset.

ProcessedAsset will finish executing and will be returned to the initializer of BundledAsset. The instance of ProcessedAsset for application.js is assigned to @processed_asset and necessary assets are assigned to @required_assets. We know that each ProcessedAsset has it’s source so we start assigning the source of all ProcessedAsset instances to @source. to_a returns required_assets and we are getting a string representation of the ProcessedAsset instance, meaning, the source. This behavior is defined here. When we have gathered sources of all ProcessedAsset instances, we again run some processors based on the content_type of current file. For Javascript files, we run the Sprockets::Processor (js_compressor) processor, passing it the source. After processing @source and getting new @source, the digest is calculated based on the source / contents.

When the constructor of BundledAsset is finished, control eventually returns back to compile in actionpack-3.2.15/lib/sprockets/static_compiler.rb. This is where the magic started and we are back here with our completed BundledAsset. Next steps are easy, as BundledAsset is saved on disk in the same way a StaticAsset is saved in file_name-digest.file_ext format.

The whole process that I described above continues for each file in the paths array.

Unanswered Questions Answered

You might be wondering why we have created the digest for StaticAsset beforehand but the digest for BundledAsset only after performing all that processing. The answer to this question is very simple. As discussed in part 1, there is no processing on StaticAssets, so we just copy and paste them to the public/assets directory with a digest based on its contents. The contents of a StaticAsset such as image etc. will not changed between copying / pasting so we can calculate its digest beforehand. Whereas calculation of digest for BundledAsset require some steps to be taken before the digest can be calculated.

We discussed in part 1 of this article that the write_to method is used to create one asset, so how does Sprockets creates two versions of the same file? When we do rake assets:precompile the :all task is called from actionpack-3.2.15/lib/sprockets/assets.rake at lines 59-67. This task has two rake tasks, one is called to create the digested version of assets and the other is called to create the non-digested version of assets. You can check those two tasks on lines 69-71 and on lines 73-75. Both tasks call the internal_precompile method with the digest parameter being set accordingly. The non-digested call to internal_precompile doesn’t take much time because everything has already been cached.

Final Thoughts

I have done my best to explain the coding internals of the Rails Asset pipelines. Reading code is an art and fun at the same time. By reading other developer’s code, weocan increase our technical skills. It was a real pleasure to read the awesome code of Sprockets.

Frequently Asked Questions (FAQs) about Asset Precompilation