Cloud Four Blog

Technical notes, War stories and anecdotes

What a holy grail image format would mean for the browser’s lookahead pre-parser?

As long as I’m posting puzzles, riddle me this: what happens to the browser’s lookahead pre-parser if we find the holy grail of responsive images—an image format that works for all resolutions?

What is the holy grail responsive image format?

In nearly every discussion about responsive images, the conversation eventually turns to the idea that what we really need is a new image format that would contain all of the resolutions of a given image.

For example, JPEG 2000 has the ability to “display images at different resolutions and sizes from the same image file”.

A solution like this would be ideal for responsive images. One image file that can be used regardless of the size the image is in the page. The browser only downloads what it needs and nothing more.

As Ian Hickson recently wrote on the WhatWG mailing list regarding future srcset syntax complexities that might come with hypothetical 4x displays:

Or maybe by then we’ll have figured out an image format that magically does the right thing for any particular render width, and we can just do:

<img src="jupiter-holiday-home.smf">

…and not have to worry about any of this.

It seems that no matter where you’d like to see responsive images go—srcset, picture, whatever—that everyone agrees we’d all be happier with a new, magical image format.

The end of image breakpoints for most responsive images

If this mythical image format existed and was adopted by browsers, it would mean the end of image breakpoints for most responsive images. You can see this in Ian Hickson’s hypothetical code above.

Unless the image required editing at various viewport widths (see art direction use case), the document author could simply point to a single file and know that the browser would download the data necessary to display the image for anything from a small mobile screen to a large, high-density display.

What does this mean for the lookahead pre-parser?

This is where the thought experiment gets interesting for me. As noted before, the big challenge with responsive images is that the browser’s lookahead pre-parser wants to start downloading images before the layout is finished. This is the central conflict for responsive images which I described as:

How do we reconcile a pre-parser that wants to know what size image to download ahead of time with an image technique that wants to respond to its environment once the page layout has been calculated?

Image breakpoints are designed to give the lookahead pre-parser the information it needs to pick the right image before the browser knows the size of the element in the layout. This is how we get around the problem.

But we just saw that image breakpoints would go away if we had a new, magical image format. Without the image breakpoints and without knowing the size of the image in the page, how would the browser know when to stop downloading the image?

Unless I’m missing something, it wouldn’t. The browser would start downloading the image file and would only stop once the layout had been determined. In the meantime, it may download a lot more data for a given image than is necessary.

Flickr example

Let’s take a look at Flickr as an example. Flickr would be a huge beneficiary of a new image format. Right now, Flickr maintains a bunch of different sizes of images in addition to the original source image.

Sample of the different sizes that a Flickr image can be downloaded at.

Instead of having to resize every image, Flickr could use a single image format no matter where the image was used throughout the site.

Lets also assume that in this future world, Flickr was responsive design using flexible images so that the size of an image is dependent on the width of the viewport. What happens to a typical Flickr layout like the one below?

Screenshot of Flickr home screen for a logged in user. Shows lots of smaller images.

Because it would be a responsive design, the size of the thumbnails on the page will vary based on the screen size. But no matter how big the thumbnails got, they would never equal the full size of the original image which means that the browser would need to stop downloading the magic image file at some point or it would download extra data.

What would the lookahead pre-parser know at the moment it started downloading the images? All it would know is the size of the viewport and the display density.

Based on those two data points, the pre-parser might try to determine when to stop downloading the image based on the maximum size the display could support. That works ok if the display is small, but on a large, high density display, the maximum image could be gigantic.

And because our magical file format would contain multiple resolutions, the browser would likely keep downloading image data until the layout was calculated making it very likely excess image data would be downloaded and the download of other assets would be delayed.

Responsive images versus the lookahead pre-parser

Of course, this is all hypothetical. We don’t have a magical image format on the table. So for now we can avoid resolving what I see as a central conflict between the need for the pre-parser to start speculatively downloading images before it knows the layout of the page and the fact that the size of responsive images isn’t known until the page layout is determined.

But in the long run—if we find our holy grail—this conflict is likely to resurface which makes me wonder about our current efforts.

I whole-heartedly agree with Steve Souders that “speculative downloading is one of the most important performance improvements from browsers”, and until a new image format materializes, it seems we should do everything we can to accomodate the pre-parser.

And at the same time, I can’t help but wonder, if we all want this magical image format and if in some ways it seems inevitable, then are we jumping through hoops to save browser behavior that won’t work in the long run regardless?

A framework for discussing responsive images solutions

Over the last few weeks many more web developers and designers have become engaged in the conversation surrounding responsive images. On the whole, this is great news because the more people we have telling browser makers that this is a legitimate issue, the more likely it is to get addressed quickly.

However, some of the conversations about responsive images end up going in circles because people are talking past each other. I believe this is happening because we don’t have a common framework to look at the problem.

I believe there are two separate, but related issues that need to be solved regarding the use of the img element in responsive designs. They are:

1. How do we enable authors so that they can display different images under different conditions based on art direction?

To understand this issue, it helps to look at a specific use case. Take for example the following photo of President Obama speaking at a Chrysler plant.1

Obama speaking at Chrysler plant

When the image is displayed at larger sizes, it makes sense for the image to show the automobile factory in the background. The background helps explain where the event took place and adds to the image. But look what happens when we scale the image down to fit a smaller screen.

Obama speaking at Chrysler plant shrunk to 100 pixels wide. Obama himself is tiny in the picture at this size.

At that size, you can barely recognize Obama. You can’t make out his face. Instead of simply resizing the image, it may make sense to crop the image to get rid of some of the background and focus in on him. The end result is an image that works better at the smaller size:

Obama speaking at Chrysler plant shrunk to 100 pixels wide and cropped so Obama can be seen better.

This is what I refer to as enabling art direction. Authors need to be able to provide different sources for images at different sizes not based on resolution or based on network speed, but based on the judgment of the designer for what is the best image at a particular breakpoint.

As an aside, showing photographs at different sizes to illustrate a point is more difficult when you’re dealing with flexible images in a responsive design. If those examples don’t make sense on your phone, I’m afraid you may have to look at it on a wider screen to see what I’m talking about! :-)

2. Enabling authors to provide different resolutions of images based on a variety of conditions.

When people talk about how to handle images for retina displays, they are talking about this second issue. The same is true of making decisions about the size of images based on bandwidth. All are dealing with how to deliver different resolutions based on conditions (pixel density, network speed, etc.).

Apple’s proposed addition to CSS4, image-set, is designed to solve this issue in CSS. Authors define images at different densities and the browser picks the best one based on some criteria that could be as simple as pixel density or as complex as a combination of observed network speed, pixel density, and user preference. What the criteria is remains to be defined.

Where the <picture> element fits

The proposed <picture> element attempts to solve issue #1. It focuses on how to give authors the ability to specify different images, but doesn’t do anything about pixel density or bandwidth.

I’ve seen a lot of feedback on the <picture> element that says we should have a new type of image format to replace JPEG, PNG and GIF that would be resolution independent. That would be awesome. And it would solve issue #2, but it wouldn’t help with the art direction outlined in issue #1.

When we discuss various solutions, it behooves us to figure out which issue we’re trying to solve. We can also debate whether or not the two issues I outlined are legitimate or if there are other issues that aren’t addressed by them.

But in order to have a fruitful discussion about how to solve these issues, we need to be clear about which issues we’re talking about or we’ll end up wasting more time. What I often see in comment threads and on Twitter is two people debating different solutions for responsive images, both looking at different issues, and neither realizing that the other isn’t looking at the same problem.

My hope is that by defining these issues, we can stop spinning our wheels and have more successful discussions.


  1. Photo of Obama licensed under Creative Commons. CC BY-NC-SA 2.0 by BarackObamaDotCom

Preferred solutions for responsive images

Scott Jehl recently tweeted:

Time to shift the responsive imgs discussion to how we'd actually prefer to do things, and then make it happen. Attrs? CSS? JS? New headers?

I concur. A couple of months ago, I asked what you preferred as a solution for responsive images. At the time, I didn’t have a strong opinion. But over the last couple of weeks, I’ve become increasingly comfortable with a direction that Scott Jehl, Ethan Marcotte and I discussed on twitter recently.

Short run solution: preparse attribute

In September, Scott Jehl proposed a solution that might be fairly easy for browser makers to implement in the short run: adding a preparse attribute to the script tag.

@grigs silly: <script preparse> //document.src references raw HTML src document.src = document.src.replace( pattern, replacement ) </script>

Like the defer and async attributes, there is a general use case for document authors to be able to tell the browser that a piece of javascript will impact the loading of assets and thus should be executed before parsing begins. There are probably even use cases that have nothing to do with asset downloading where telling the browser to execute javascript before parsing would be beneficial.

One of the other suggestions for solving this was for browsers to standardize on how they load assets. I don’t like that idea because it prevents browsers from experimenting on ways to parse pages and load assets more quickly.

If authors explicitly tell the browser when javascript loading matters (preparse, defer or async), it allows browser makers to experiment freely in other situations.

Long run solution: improvements to img tag

In the long run, I don’t like the idea that solutions require javascript. Images are content or presentation which means it should be possible to handle it with HTML and CSS alone.

Therefore, I would like to see one of these two improvements to the img tag:

<img alt="butterfly"> <source src="butterfly-small.png" media="max-device-width:480px" /> <source src="butterfly-large.png" media="min-device-width:481px" /> </img>

Modified from Bryan and Stephanie Rieger’s Rethinking the Mobile Web talk.

This is my preferred option, but I’m unclear on how older browsers would handle an image tag that contains child elements.

Isaac Lewis put together a test page using this style of markup. It would be great to collect some feedback on old browser support to see if it works or causes problems.

If that won’t work because of legacy browsers, the following variation proposed by Anselm Hannemann should:

<img src="http://cdn.url.com/img/myimage_xs.jpg" media-xs="(min-device-width:320px and max-device-width:640px)" media-src-xs="http://cdn.url.com/img/myimage_xs.jpg" media-m="(min-device-width:640px and max-device-width:1024px)" media-src-m="http://cdn.url.com/img/myimage_m.jpg" media-xl="(min-device-width:1024px)" media-src-xl="http://cdn.url.com/img/myimage_xsl.jpg" />

Why not headers?

I would love the browser to send more data to the server about the device making the request, but I don’t think servers should be necessary for the image tag to work in a responsive design.

I also think there is a decent chance that screen size is just the first of many headers we’d like the browser to send along. I don’t want to open up pandora’s box, but it would be nice to get something that felt more like a comprehensive solution instead of a bandaid.

Why not a different, progressive image format?

I’d love this, but feel wholly unqualified to judge what it would take. I’m not sure how long something like that would take to implement and what sort of patent minefield might lie there.

So let’s start with the preparse attribute

Unless someone points a major flaw with the preparse idea, I’m going to submit it as a feature request to the appropriate people to get the ball rolling.

Which reminds me, does anyone know where I should start? :-)

When is Mobile Safari not Mobile Safari?

When it is AppleCoreMedia. Let me explain.

I wrote recently about a quiz that we built that included HTML5 audio files. In addition to the volume problems already discussed, we encountered intermittent issues where the audio either would not play or would play much later than it should.

For the quiz, we preloaded the HTML5 audio file via javascript. To make sure that the file was set up for caching, we set far future expires headers as well as making sure the file was sufficiently small.

Whenever we displayed the screen that told the user whether or not they got a question correct, we play the preloaded audio file. Most of the time this worked flawlessly, but occasionally the iPhone would play the sound long after the results screen was displayed or not play the file at all.

That’s when things started getting weird. We watched the log files and started seeing some odd behavior. Here’s what we saw:

"GET /trivia HTTP/1.1" 200 2346 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /trivia.css HTTP/1.1" 200 988 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"

See what happened there? The user agent string changed from the normal Mobile Safari one (AppleWebKit) to one I hadn’t seen before: AppleCoreMedia.

It makes sense when you think about it. All video and audio playback gets handled by a system component. That component is responsible for downloading the asset. Ergo, the user agent string should change accordingly.

Surprisingly, this is true on desktop Safari as well. The equivalent user agent string is “Apple Mac OS X v10.6.6 CoreMedia v1.0.0.10J567.”

AppleCoreMedia Caching

On its own, the change from Mobile Safari’s user agent string to AppleCoreMedia’s user agent string would hardly be worth noting. But we were encountering problems with the sound not playing all of the time.

Because we noticed the sound problems happened more frequently on 3G than on WiFi, we started to suspect network issues and wondered if the audio file that we preloaded was getting cached properly. Here’s what we found in the logs:

"GET /trivia HTTP/1.1" 200 2346 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /trivia.css HTTP/1.1" 200 988 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 5494 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 2597 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /js/trivia.js HTTP/1.1" 200 2618 -" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_right.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 2597 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 5494 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"

All of that occurs before the first quiz answer is submitted so it is only in relation to preloading the audio. The files get downloaded multiple times. Lest you think this is simply http chunking per the 206 responses, here are the file sizes:

-rw-r--r-- 1 cloudfour psacln 2256 Dec 10 17:33 quiz_right.m4a
-rw-r--r-- 1 cloudfour psacln 5153 Dec 10 17:33 quiz_wrong.m4a

Before the audio file is ever played, AppleCoreMedia has downloaded 11,662 bytes for quiz_wrong.m4a. At 5153 bytes, the source file is less than half of the total bytes download.

When we could replicate the audio problems, we would find that the server would return a 304 response to AppleCoreMedia letting it know that the m4a had not been modified, but then AppleCoreMedia would go ahead and download it anyways. Of course, this behavior was inconsistent making it difficult to troubleshoot.

To try to narrow down the behavior, I created a simple page with links to m4a and mp3 files. I didn’t do any HTML5 audio embedding and didn’t include javascript. The page was as vanilla as can be. The test file size was 30,196 bytes. Clicking on the m4a link with a clear cache had the following result:

"GET /examples/applecoremedia/trailer_iphone_sound/ HTTP/1.1" 200 653 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 200 30551 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 304 233 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 206 22410 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 304 233 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 206 22410 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"

This is even more confusing than the last example. The full m4a file is first downloaded by Mobile Safari. Then AppleCoreMedia asks the server if the file has been modified and is told by the server that it hasn’t. AppleCoreMedia then ignores this information and proceeds to download the file. Then it repeats the process for good measure.

And despite setting far future expires headers for everything, the next time you play the m4a file, the same process is repeated.

Because of the issues we had previously had with short audio files, I decided to try a longer video file to see what it would do. Again, I set up a simple page and linked to the video file. The video file was 3,995,176 bytes (3.8MB).

"GET /examples/applecoremedia/trailer_iphone/ HTTP/1.1" 200 729 "http://www.cloudfour.com/examples/applecoremedia/" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 386 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 200 70500 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 71910 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 386 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 3995571 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"

Mobile Safari still downloads some part of the m4v file, but it isn’t the full file size (70500 bytes). It is unclear what it does with this data chunk nor why it appears to only be part of the data, but the server reports a 200 response instead of 206.

The next time the video is played, the same pattern repeats. Nothing appears to be cached. Based on previous research into iPhone 4 cache sizes, I would have expect even the 3.8MB video to get cached.

What I Expected to See

To illustrate what I expected to see, I ran with my simple audio file test in Firefox. The only change I made was to use an mp3 file instead of an m4a.

"GET /examples/applecoremedia/trailer_iphone_sound/ HTTP/1.1" 200 653 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.mp3 HTTP/1.1" 200 28185 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"

No matter how many times I load the mp3 file and then hit the back button to the web page containing the link, neither the web page nor the mp3 file were downloaded again. They had been successfully cached.

What does all this mean?

I wish I knew. For our project, it meant that we could not consistently ensure that the audio file would play at the right moment when the message was displayed on the screen letting the user know if they had got the question right or not. This issue combined with the volume issues caused us to remove the feature.

For consumers using iPhones, it appears to mean that every time you watch a video or play an audio file, that your phone is going to download it again. This makes for a slower experience and puts a seemingly unnecessary burden on the carrier network.

For developers, it is important to realize that video and audio playback in Mobile Safari is not handled by Mobile Safari even if you’re controlling it via javascript and the system media player is never visible. Everything you’ve learned about how Safari and Mobile Safari handle the downloading of assets doesn’t apply when AppleCoreMedia is doing the work instead of Safari.

Finally, this is just one more clue about how mobile is still frontier land. Even something that seems simple like a quiz that plays sounds can bump up against the boundaries of what we know about how mobile browsers work and what they can reliably do.

JavaScript GZIP Compression in WordPress: What’s Possible and what Hurts

WordPress 2.8 introduced some constants and bits and bots that nominally make certain kinds of compression of JavaScript and CSS possible. This post looks at JavaScript compression and explains what is possible, and what is very, very hard.

In a nutshell, with a bit of tweaking, you can GZIP and concatenate the scripts that come packaged with WordPress. GZipping and concatenating anything else–plugin scripts or scripts in your own theme–is not currently supported in any obviously sane way (using WP’s WP_Scripts management class anyway).

I’m going to give away the ending to save you some time:

add_action('wp_enqueue_scripts', 'lyza_force_compress');
function lyza_force_compress()
{
    global $compress_scripts, $concatenate_scripts;
    $compress_scripts = 1;
    $concatenate_scripts = 1;
    define('ENFORCE_GZIP', true);
 
    /* enqueue your scripts here */
}

Putting the above in your functions.php file may result in insanely good compression on your subsequently-enqueued JavaScript files. I say may because GZIP is one of those fidgety server-side things that varies from hosting provider to hosting provider. I tested this approach on two vastly different environments and had good success, but as always with these things, YMMV.

While this may look rather silly and simple, figuring out exactly which constants and globals to set and when to set them took me just about forever.

The most important thing to note is that this only works on scripts that come pre-packaged with WordPress and are listed in the wp_default_scripts() function in wp-includes/script-loader.php. This is slightly irritating.

When it Does Work

The above will concatenate and GZIP all of the applicable enqueued JavaScript into a single request, served out through the wp-admin/load-scripts.php script.

  • By “applicable”, I mean scripts listed in wp-includes/script-loader.php in the wp_default_scripts() function.
  • If you have applicable JavaScript in both the head and the foot, you will end up with two script src tags. If everything is in the foot (or the head), you’ll get one tag. You not only get GZipping, you get fewer HTTP requests! Yay!
  • JavaScript in plugins and your own theme won’t get concatenated and GZipped in this approach, but it certainly doesn’t break it.

In testing, the following:

    wp_enqueue_script('jquery');
    wp_enqueue_script('thickbox');
    wp_enqueue_script('scriptaculous');
    wp_enqueue_script('editor');

Went from:

10 requests, 310kB

to

2 requests, 90kB

with the activation of the above function. Nice! I got two requests because the above enqueueing puts stuff in both the head and the footer.

That’s a 340% improvement on filesize alone.

Why Won’t It Work for All JavaScript?

Because there are no, nada, zilch, none hooks for making it do so without doing some deep surgery and writing a plugin. It boils down to a couple of places in code:

From wp-includes/class.wp-scripts.php in the do_item() function:

100
101
102
103
104
105
106
107
108
	if ( $this->in_default_dir($srce) ) {
		$this->print_code .= $this->print_scripts_l10n( $handle, false );
		$this->concat .= "$handle,";
		$this->concat_version .= "$handle$ver";
		return true;
	} else {
		$this->ext_handles .= "$handle,";
		$this->ext_version .= "$handle$ver";
	}

To get the built-in concatenation and GZIPped delivery, it is necessary that the if statement here evaluate true. I won’t explain the nitty gritty unless you really want me to.

So, my first approach was to add paths to the $wp_scripts->default_dirs Array to add entries for my theme’s scripts and my plugin dir’s scripts. Big fat fail.

The reason is that the actual file — wp-admin/load-scripts.php — that serves up the concatenated, zipped JavaScript is a standalone script. I imagine that this is both for security and for performance. It does not have the complement of WordPress constants and hooks that we are accustomed to. In fact, it is completely and absolutely unpluggable (unless I missed something). It only includes the files that define the WP_Scripts and WP_Dependencies classes, and utility functions for those. It does not include any theme- or plugin-level items.

And it does something that completely shuts down the game. It instantiates a new, clean WP_Scripts object and then calls wp_default_scripts().

If you’ll recall, the WP_Scripts object in a normal WordPress request is a global singleton that manages all of the enqueued scripts. By creating a new instantiation, the load-scripts.php script has obliterated any notion of any enqueued scripts. Further, the wp_default_scripts() function is a hard-coded list of the JavaScript files that WordPress comes packaged with. While this function nominally triggers the ‘wp_default_scripts’ action, this is not usable here because no files are getting included in which I could take advantage of that hook. Further, it redefines do_action() and other API functions as blank functions that do nothing.

The load-scripts.php file then iterates through the requested JavaScript handles (comma-delimited GET) and sees if it recognizes them from the wp_default_scripts() list. As anything that’s in your own theme or in a plugin won’t be in this list, FAIL.

Though I see some possible options for me in the writing-a-plugin realm, I also sadly see that it would replicate functionality and am highly curious as to what the WordPress developers had in mind here.