Skip to main content

Preferred solutions for responsive images

By Jason Grigsby

Published on November 22nd, 2011

Topics

Scott Jehl recently tweeted:

Time to shift the responsive imgs discussion to how we'd actually prefer to do things, and then make it happen. Attrs? CSS? JS? New headers?

I concur. A couple of months ago, I asked what you preferred as a solution for responsive images. At the time, I didn’t have a strong opinion. But over the last couple of weeks, I’ve become increasingly comfortable with a direction that Scott Jehl, Ethan Marcotte and I discussed on twitter recently.

Permalink to Short run solution: preparse attribute

Short run solution: preparse attribute

In September, Scott Jehl proposed a solution that might be fairly easy for browser makers to implement in the short run: adding a preparse attribute to the script tag.

@grigs silly: <script preparse> //document.src references raw HTML src document.src = document.src.replace( pattern, replacement ) </script>

Like the defer and async attributes, there is a general use case for document authors to be able to tell the browser that a piece of javascript will impact the loading of assets and thus should be executed before parsing begins. There are probably even use cases that have nothing to do with asset downloading where telling the browser to execute javascript before parsing would be beneficial.

One of the other suggestions for solving this was for browsers to standardize on how they load assets. I don’t like that idea because it prevents browsers from experimenting on ways to parse pages and load assets more quickly.

If authors explicitly tell the browser when javascript loading matters (preparse, defer or async), it allows browser makers to experiment freely in other situations.

Permalink to Long run solution: improvements to img tag

Long run solution: improvements to img tag

In the long run, I don’t like the idea that solutions require javascript. Images are content or presentation which means it should be possible to handle it with HTML and CSS alone.

Therefore, I would like to see one of these two improvements to the img tag:

<img alt="butterfly">
  <source src="butterfly-small.png" media="max-device-width:480px" />
  <source src="butterfly-large.png" media="min-device-width:481px" />
</img>

Modified from Bryan and Stephanie Rieger’s Rethinking the Mobile Web talk.

This is my preferred option, but I’m unclear on how older browsers would handle an image tag that contains child elements.

Isaac Lewis put together a test page using this style of markup. It would be great to collect some feedback on old browser support to see if it works or causes problems.

If that won’t work because of legacy browsers, the following variation proposed by Anselm Hannemann should:

<img src="http://cdn.url.com/img/myimage_xs.jpg"
     media-xs="(min-device-width:320px and max-device-width:640px)"
     media-src-xs="http://cdn.url.com/img/myimage_xs.jpg"
     media-m="(min-device-width:640px and max-device-width:1024px)"
     media-src-m="http://cdn.url.com/img/myimage_m.jpg"
     media-xl="(min-device-width:1024px)"
     media-src-xl="http://cdn.url.com/img/myimage_xsl.jpg"
/>
Permalink to Why not headers?

Why not headers?

I would love the browser to send more data to the server about the device making the request, but I don’t think servers should be necessary for the image tag to work in a responsive design.

I also think there is a decent chance that screen size is just the first of many headers we’d like the browser to send along. I don’t want to open up pandora’s box, but it would be nice to get something that felt more like a comprehensive solution instead of a bandaid.

Permalink to Why not a different, progressive image format?

Why not a different, progressive image format?

I’d love this, but feel wholly unqualified to judge what it would take. I’m not sure how long something like that would take to implement and what sort of patent minefield might lie there.

Permalink to So let’s start with the preparse attribute

So let’s start with the preparse attribute

Unless someone points a major flaw with the preparse idea, I’m going to submit it as a feature request to the appropriate people to get the ball rolling.

Which reminds me, does anyone know where I should start? 🙂

Jason Grigsby is one of the co-founders of Cloud Four, Mobile Portland and Responsive Field Day. He is the author of Progressive Web Apps from A Book Apart. Follow him at @grigs.

Comments

Matt Wilcox said:

I don’t think preparse is going to work.

The reason that things don’t work already is because browsers request an image as soon as the node has finished loading. It isn’t a script execution speed or order issue, it’s a resource-fetching issue.

It’s important to understand why browsers do what they do, they fetch resources as soon as they’re encountered to increase the speed of page load. The metric browser vendors care about is shaving the time to complete rendering the page as low as possible. Preparse would cobble that even if it could be made to work. I don’t see any browser manufacturer wanting to slow down the page load (this is why they’re so set against including more useful headers with all HTTP requests).

Further, I don’t see how it would work anyway. In order for preparse to work as we want that flag would have to effectively tell the browser engine to delay the load of all linked assets until after the HTML has been downloaded. Because how can the browser know which assets the script is going to effect if the script hasn’t run yet?

I will be extremely surprised if this ever happens.

Matt Wilcox said:

For what it’s worth, my backing goes to more complete and standardised headers as one part of the solution, and a re-imagined image tag that works like the video tag as another. Both are valid, both are needed, both do different jobs.

The issue with header expansion is the drawback of increasing page-load time – but we’re getting to the point now where we have to reconsider if the benefit of shaving milliseconds of a complete page load is worth the lobotomisation of server-client relationships.

Jason Grigsby (Article Author ) said:

@matt Correct this requires browser manufacturers to agree to let authors declare explicitly that JavaScript is important to process before assets start loading. As to the likelihood of it, I can only point to Scott’s tweet:
http://twitter.com/#!/scottjehl/status/134449813226930176

Matt Wilcox said:

@Jason – I just don’t see it as an effective solution. To block loading of ALL external assets just because one script tag has a preparse attribute? That script might only be effecting one image on a page, and yet we’ve blocked all external assets loading?

And what happens with a script that is included on the page that _doesn’t_ have a pre-parse on it? Does that script execute immediately or after assets load? Is that script loaded at all – given that it too will be an external asset referenced via URI?

Matt Wilcox said:

Or do we take the strange decision that “preparse” actually means “just in the case of images”? Which I think would be terrible for future proofing.

Maybe we could have a wait attribute that takes a string of element types as arguments. e.g.,

<script src=”whatever.js” wait=”img,video” />

which would tell the page to not request img or video files until the page has loaded and the script was executed.

But then, how does the browser engine know that the script finished executing?

There are a raft of problems with this idea of a preparse tag, or any attribute on a script element effecting things other than the script element itself.

Jason Grigsby (Article Author ) said:

@matt wrote:

“To block loading of ALL external assets just because one script tag has a preparse attribute? That script might only be effecting one image on a page, and yet we’ve blocked all external assets loading?”

The alternative is the browser goes ahead with parsing the page and acting as everything in hunky dory despite the fact that the document author knows that a particular bit of javascript is going to change the way the page is loaded.

“And what happens with a script that is included on the page that _doesn’t_ have a pre-parse on it? Does that script execute immediately or after assets load? Is that script loaded at all – given that it too will be an external asset referenced via URI?”

If it doesn’t have preparse, it works like it always has.

“But then, how does the browser engine know that the script finished executing?”

Browsers already do this. This is why the Yahoo performance guidelines recommend putting scripts at the bottom of the document. The behavior has been that browsers block parallel downloads and block progressive rendering of anything below the script (per @souders book).

My point is that this isn’t new behavior. This is actually the way browsers used to work. Put javascript in the head and it blocked things until it was done.

Anyways, you can see the challenges that script already poses for browsers by reading this article on the MSDN blog about how they handle it in IE: http://bit.ly/udPoI4

They’re doing all sorts of things already—including running two parsers simultaneously—in order to be fast but also deal with the fact that the javascript they already have may change the assets that need to be downloaded.

The only thing the preparse tag would do is tell the browser explicitly, “Hey, this script is going to change things.” If anything, it would help browser makers be more efficient by giving them more information that they could use to streamline how they handle assets.

Isaac Lewis said:

“It would help browser makers be more efficient by giving them more information that they could use to streamline how they handle assets.”

I agree. The only drawback is the waiting game while the browsers figure it out. Any Chrome devs reading this? =D

Also, thanks plugging my test page, @Jason… There’s a patch in the works to collect browser support information for people to use… hopefully we’ll find some interesting stuff!

Cheers,
Isaac

Matt Wilcox said:

@ Jason

I don’t disagree that we need some options for better handling of our problems, but I don’t think that the preparse tag is a good (enough) solution.

The presence of an attribute on one element changing the behaviour of the rest of the page simply feels wrong and not very HTML’y. Now, that’s a bit of a daft comment but it’s also something of a good flag to indicate it’s not quite right and could do with thinking about.

I think I’m missing a bit of understanding on how browsers work. Imagine you’ve got a script with preparse in it. The preparse stops the browser from loading all externally referenced objects until after the DOM has completed loading *and after the script has run*. How does the browser know that the script has finished manipulating the image tags and it’s now OK to request those resources? Baring in mind that the script may also be talking to external resources through an API and not just run locally?

The idea of *not* rendering things immediately as they are loaded (and adjusting page rendering as new things arrive) is completely against the way the web has always worked, and that to me is a big issue. If we introduce blocking mechanisms then the web becomes a lot more like a Flash site than a HTML site, where we could see long waits with nothing happenening just because one bit of JS wants to manipulate a couple of image uris and needs to wait for who-knows-how-long to ensure it can do so.

There have been plenty of studies showing that as we get used to faster load speeds we’re less willing to wait for things. I can’t help but think that any site using the preparse tag would be noticeably “slower” in perceived page load times and likely to put people off viewing the site.

Matt Wilcox said:

PS, thanks for the links! It looks like the look-ahead pre-parser introduced to “make things faster” is a serious error and harms performance when responsive images are needed.

Perhaps instead of preparse on the script element we should have a meta attribute to turn pre-parse off for given tags:

<meta preparse_off=”img video audio” />

Matt said:

Unfortunately, the test page does not work. There’s that part in the source code: “var testKey = ‘CHANGE-THIS-TO-YOUR-TEST-KEY’;

Besides, the buttons are divs with no event listener attached. There is that ‘saveVote’ function that does nothing and is quite obscure…

About the test: I saw the Image in IE7. That’s good news, isn’t it?

Isaac Lewis said:

Yes, I am in the middle of changes. Due to being gone from home, I don’t have my full dev environment, so I apologize for having to build on a production page. It will be done by Friday.

Thanks for the patience,
Isaac

Simon Matt said:

Hey!

nice article, thanks for that!

I don’t think the preparse solution is the best one. For me it would really be a more advanced IMG tag – like the example shown:
‹img src=”large-size.jpg” alt=”my image”›
‹source media=”max-width:640px;” src=”mobile-size.jpg”›
‹/img›

for me it’s the only way which makes sence because it would be somehow identical to the video tag of html5.
I think we should try to get the best and most consistent solution. think of young people who start learning html(5)…

so long…

James Beardmore said:

I really like the idea of multiple child elements inside an image tag, it makes sense looking at where other media tags are heading. The only thing about it that bothers me here is the semantics of the attributes. I just wish we’d ended up with something a little less redundant than … but that’s not the point here.

Scott Jehl said:

Jason! Thanks for another great post, and for continuing to push on this issue.

Like others here, I’m not at all tied to any particular suggestion above either – frankly, I’d be thrilled to see any one of them implemented in the near future!

I agree with you that an extension of the img element is most robust. Beforehand, I would love to see some serious testing on how this renders on popular older browsers and devices before we start using it (what do IE6-8 do with content nested in an img element?). Maybe Filament’s (jQuery Mobile) test lab can be of assistance here.

As an aside, the ability to preparse a document source text with JS could really allow us to innovate in areas beyond this one issue. Like any code added to a document, it would have the potential to add latency at page load. This is where best practices would come in – maybe you’d leave preparsing JavaScript in the head rather than external, for example.. who knows.).

So I’d love to see page source preparsing available to JS even if a more robust solution for img elements was standardized as well.

On headers (and cookies for that matter): I’m not wild about adding more data to our requests just to handle this issue, and I really don’t think such simple asset negotiation should be restricted to those with access (and skills) to create server-side rewrites and such. The video element solves this issue very elegantly and provides a great model to follow.

If headers were our only hope though, I’d like to see a way to set specific headers for front-end requests via attributes. For example, the img element could be extended with a “headers” attribute that could accept several pre-defined values, possibly space-delimited as we use classes. A value like “viewport” could tell the browser to send a particular image request with a header that contains information about the viewport dimensions. Maybe a “screen” value could send data about screen dimensions, assuming they’re different than the viewport. Anyway, just a thought!

Thanks again!

Isaac Lewis said:

@Scott – As Jason mentioned, I have launched a test page to discover how much legacy support there is for the img tag extension. You can see the results here: ikelewis.com/the-future.html

We are working on getting every browser tested, including the more obscure mobile browsers. Check back often to see how the results are coming, and be sure to test it yourself and submit results on whatever browsers you’d like to add.

Matt Wilcox said:

“I’m not wild about adding more data to our requests just to handle this issue” – I’d agree if it were just to handle this issue. But it isn’t. It’s about adding the ability for better server-client communication so we can handle future needs too. That’s half the problem with how the browser header has evolved – the “fix it for today” attitude, which doesn’t serve the future well.

I also agree that I’d like more than one solution. While server side is very appropriate in some cases, in others it’s more appropriate to be client-side. They do different things. Another issue I think we’re seeing is that a lot of people seem to be after “one solution” to this “one issue”. It isn’t “one issue” and there shouldn’t be “one solution”.

Hans Christian Reinl said:

Thanks for the article!

I recently wrote about this topic, too and found that there are some other possible ways to handle responsive images now and in the future.

The native approach of responsive images without using JS at any point (as suggested by Anselm for instance) is my favorite variant. Furthermore I think an image-format which is developed to handle different file-sizes would be great too.

Gunnar Bittersmann said:

Hm, ain’t that re-inventing the wheel? The basic technique to choose the best of various options (image sizes) already exists: content negotiation. It works for image formats: SVG, PNG, JPEG, … All you’d need is an entry in the HTTP header telling the server the viewport size of the client.

And BTW, if one would change the emptyness of the img element (which would be hard to do because of backwards compatibility), let’s correct the design error of HTML and get rid off that @alt attribute. The alternative text would be much better paced in the element’s content, as it is for iframe, object, etc.

Jason Grigsby (Article Author ) said:

@Matt Simon, preparse is the short term solution that can be implemented by browser makers immediately. I agree that long term multiple sources on the img tag would be better.

@Salman Abbas, the reason for seeking a different solution than just putting the different sizes in data-src and then using javascript is because that solution doesn’t work reliably. There are race conditions with browser lookahead parsers that cause multiple images to download. Yoav’s test can be found here:
http://blog.yoav.ws/2011/09/Preloaders-cookies-and-race-conditions

@Gunnar Bittersmann, as I mentioned above, I’d love to have more http headers, but I don’t think the img tag should be the only html element that *requires* a server in order to work in responsive designs. That solution will not work for widgets or other html apps designed to be bundled and work entirely offline.

Jason Grigsby (Article Author ) said:

@Matt Wilcox, noodled on your feedback for a couple of days. First, I absolutely agree with you about performance. It is a focus of mine and the reason I started investigating these issues in the first place.

But I don’t agree that preparse will be a problem. The reason I don’t see it as a problem is because the browsers are already doing this with every single page they load. They’ve built complex loading behavior to deal with the fact that badly written JavaScript may be inserted into random spots in the page, change assets loading, and block downloads.

This is the life of a browser. Take the crap someone wrote and figure out how to handle it as best as possible.

Adding a preparse attribute is making the situation better for browsers by giving them more information. How they handle that information could be to queue the javascript earlier. It could be to delay asset loading. It could be to evaluate the network connection speed and decide it would be more efficient to continue to download assets and risk downloading two sets of assets.

This opens up a lot of opportunities for browser makers to innovate on performance once they have this information.

My guess is that most HTML documents will still be authored without preparse just like most do not contain defer. Yes, there will be occasional authors who misuse preparse just like there are currently authors who unnecessarily block asset loading and do other bad practices with JavaScript.

But for those who know for certain that the JavaScript that they’ve included will change the way the HTML is parsed, flagging the JavaScript as such does not but give browsers information that can help them do their jobs better.

Brett Jankord said:

All of the solutions seem to hold a lot of potential in my mind. I’m not sure of what issues the preparse tag would raise, though I am hesitant to have my images dependent on JavaScript, especially for a mobile solution. If I were going to use a JS solution, I think it would be imperative to set a mobile sized image as default and if JS is enabled, use that to determine the correct image size.

This raises questions in my mind of what is a correct mobile sized image? Should we have multiple image sizes for multiple mobile screen sizes. Does a phone with a screen width of 533 or 480, need a different image than a phone with a screen width of 320, 240, 128, or can fluid images work for all mobile screen sizes. In my mind it depends how granular you want to get with your solution. I’d be fine with serving a 533px image to all mobiles and using Ethan’s fluid image technique to scale it down.

I really like the idea of adding a media attribute to the image tag. Though I like the JS and Header solution because they allow the ability to change an entire block of markup, rather then just the image.

As for the header solution, this seems like a excellent option too. One issue I see with this though is what dimensions will the header send? The device’s physical screen size, the visual viewport size, the layout viewport size, or all of the above? I would love for the header to send more information about each device, though it would need to be STANDARDIZED.

Like I said, I think all of these solutions hold a lot potential and it would nice to see all of them implemented in the future.