Cloud Four Blog

Technical notes, War stories and anecdotes

Responsive IMGs Part 3 — Future of the IMG Tag

The conversation in the comments on part 1 and 2 of this Responsive IMGs series have been exceptional. If you read the articles, but didn’t read the comments, I encourage you to go read them. There are people much smarter than me in those comment threads.

One of common conclusion from the commenters is that our current IMG tag isn’t going to cut it. That we need some sort of replacement that is future friendly. This isn’t the first time the idea of expanding or replacing the IMG tag has come up.

I promised to collect some of these proposals so we can discuss their relative merits.

Current proposals and discussions to replace or extend the IMG tag

As far as I know, there hasn’t been a formal proposal submitted to any standards bodies, but there have been several conversations worth highlighting. I was going to try to summarize the various proposals, but there hasn’t been a lot of consensus. I’m afraid that to get up to speed, you’re going to have to read the threads.

Adaptive Images on W3C HTML mailing list

At the end of May, Dom Hazael-Massieux kicked off a lengthy thread talking about Adaptive Images with two ideas: a srclist attribute and new image file format that would be a text file containing a list of images.

The thread continues with many other ideas including new http headers, progressive image formats, and general media formats with queries.

BTW, Adaptive Images has been added as a placeholder on the HTML.next list.

Discussing Alternatives For Various Sizes Of The Same Image & Introducing src Property In CSS As An Option by Robert Nyman
Robert asks the question, “whether various sizes of the same image is really content or presentation?” If it is presentation, then it should be in CSS. Again, great discussion in the comments.
Responsive images using CSS3 by Nicolas Gallagher
“This method relies on the use of @media queries, CSS3 generated content, and the CSS3 extension to the attr() function.” This means it relies on already existing standards without creating anything new. Unfortunately, it doesn’t prevent multiple images from being downloaded and currently only Opera supports the CSS3 content property. That said, if we could get browser makers to change download behavior, this could work with existing standards.
My take on adaptive images, Responsive images – hacks won’t cut it, and Simpler responsive images proposal by Yoav Weiss

No one has been blogging more about the need for a better solution than Yoav. His proposals have changed over time. The simpler responsive images proposal is his latest version, but they are all worth reading to see his thoughts and the feedback on each post.

add html-attribute for “responsive images” on WhatWG mailing list

Anselm Hannemann started a recent thread on the WhatWG mailing list proposing using media- and inline media queries to provide different images. One of the things notable about this thread is that there are some on the list who don’t think the current situation is a problem or if it is a problem, that it is a niché use case. We have some education to do.

Comments on Responsive IMGs Part 1 and Part 2

It seems weird to link to my own posts, but the comments on each post contain some good suggestion. Scott Jehl said, “It’s unfortunate the comment streams of these images posts are separated. Good stuff going on in both.” I agree. Best I can do is link them up with this post.

What are our goals for a replacement for the IMG tag?

As I read through the conversations about the potential replacements or extensions for the IMG tag, I’m struck by the fact that people seem to all see the same problem, but haven’t yet come up with common criteria that could be used to evaluate different potential solutions. Until that exists, we won’t have consensus.

For example, I would love to have the browser send more data via HTTP headers that would allow us to make determinations on the server about what size image to send. But I don’t think that will work as a replacement for the IMG tag because we need something that will work for HTML widgets where the server isn’t part of the picture.

(Psst… but seriously browser makers, more headers would be great!)

So what should be our criteria. Here is start from my perspective:

  • It should be an HTML element and not solely CSS because IMGs are often part of the content. IMGs are semantic.
  • The solution should work without server content negotiation so that it can be used for HTML widgets. This doesn’t preclude servers being part of the solution, but we need a solution that will work if servers aren’t in the mix.
  • Current caching and proxy strategies cannot be ignored. Because of this, different image sizes likely need to be at unique URLs or the wrong size image for a given user will end up cached at the edge of the network.
  • It should support arbitrary image sizes and pixel density.
  • It should support art direction decisions to change the image at different sizes.
  • It should be a framework that will allow for future expansion based on factors beyond screen size. For example, if the browser provides information about the speed of the network connection, the designer or developer should be able to decide on the most appropriate image. We don’t have to define this now, but we shouldn’t box ourselves in to only looking at screens size.

That’s my start on criteria. What did I get wrong? What would you add? And of the various IMG tag replacements that have been suggested, which do you think holds the most promise?

22 Comments on “Responsive IMGs Part 3 — Future of the IMG Tag”

  1. James Pearce says:

    I disagree. It’s the image that needs to be self-adaptable, not only the HTML around it.

    Every solution out there right now (including my own, no doubt) is a hack*

    But you’re presuming that images never need to be responsive outside of an HTML-centric document environment. What about images used in web apps, where there is of course a DOM, but no HTML to speak of. What about native apps, which need just as much equivalent flexibility?

    Perhaps we feel very comfortable specifying, hacking, overriding, polyfilling etc etc the behavior of servers or browsers at a textual (HTML? CSS?) abstraction, rather than at a (SVG? binary?) image format level. But the latter is the right architectural choice. I can feel it.

    I guess I need to think more specifically about what this means. But I urge us all to think about the asset challenge as an asset challenge. Not a classic-web-superstructure challenge.

    * Although this is not necessarily a bad thing – even the img tag itself had a quick and brutal birth: http://1997.webhistory.org/www.lists/www-talk.1993q1/index.html#182 (but wow, check out that author roster!)

  2. E.Casais says:

    Shooting from the hip, the requirements you state make me think the a refurbished img tag would look a bit like the video or object tags, with the possibility to default sequentially from one alternative to another depending on parameters associated to each element (possibly using some kind of media-queries like syntax to select each case), and with a catch-all fallback at the end. This would have some nice effects (discoverability, readability and maintainability for developers, possibility to associate CSS). Not that video/object are terrifically good examples of how to handle adaptive media.

    As far as inherently adaptive formats are concerned, JPEG2000 was an attempt in that direction — but I presume the lack of popularity has something to do with IPR.

  3. James King says:

    Given your criteria list, are we asking for a tag (or super tag) in this case that does too much?

    I see a very big danger with a tag this complex when it comes to asking the browser makers to implement this consistently.

    I am not sure that simply adding some header information would not resolve this problem in the majority of cases. This would retain the simplicity of the image tag.

  4. Jason Grigsby says:

    @james wrote:

    “But you’re presuming that images never need to be responsive outside of an HTML-centric document environment. What about images used in web apps, where there is of course a DOM, but no HTML to speak of. What about native apps, which need just as much equivalent flexibility?”

    Web apps with no HTML write tags to the DOM, right? That to me doesn’t seem like a big distinction. If we have a tag, it will be part of the DOM and web apps will be able to manipulate it via javascript.

    With regards to native apps, they already have the control they need to select different image sizes since they can control the order, timing, and assets retrieved from the network.

    I agree that a different asset format might be useful for native apps, but that’s not the problem set I’m concerned with.

    “But I urge us all to think about the asset challenge as an asset challenge. Not a classic-web-superstructure challenge.”

    Perhaps it is both. A new image format won’t solve the problem where someone wants to select a different image instead of simply a resized image for a particular context. A new asset format won’t solve that problem.

    That’s not to say a different asset format wouldn’t be helpful. I’m open to the idea. Would love to see it just like I’d love to see more header information.

  5. Jason Grigsby says:

    @jamesking wrote:

    “I am not sure that simply adding some header information would not resolve this problem in the majority of cases. This would retain the simplicity of the image tag.”

    Header information requires a server to make decisions based on the headers. What other HTML element requires a server?

    Do we want to start making that a requirement now when we’re just now seeing Windows 8, WAC, WebOS, etc. all adopt encapsulated web apps as a primary way of building apps? We’d cut that movement off at the knees.

  6. James Pearce says:

    > Web apps with no HTML write tags to the DOM, right?

    I see plenty of discussion around loading sequences, aggressive loading, race conditions, cookie setting and other malarkey. These are document problems. A DOM which is created programmatically can completely avoid these issues, imperatively.

    (Which is then indeed your own point w.r.t. native apps. No different.)

    > A new image format won’t solve the problem where someone wants to select a different image instead of simply a resized image for a particular context.

    Why not? Solving this would *be* the reason to come up with a new format.

    Imagine layered, vector art controlled with media queries, for example:

    “Here is the sky-at-night. When viewed at 320px (or some DPI, say), keep level 1 labels 10px or higher, stars at 2px or more, constellation lines at most 1px. When viewed at 980px or more, stars should be rendered sprites, constellation lines 2px or more, and show level 1-3 labels”

    etc etc

    So content should flow like water? Yes! But the image is the content.

    The image should flow. We need to want to be smarter than just putting different sized stones into a document river which flows (yah yah, like water) around them.

    Images themselves need to know how to adapt to their constraints and opportunities.

  7. Jason Grigsby says:

    @jamespearce wrote:

    > I see plenty of discussion around loading sequences, aggressive loading, race conditions, cookie setting and other malarkey. These are document problems. A DOM which is created programmatically can completely avoid these issues, imperatively.
    >
    > (Which is then indeed your own point w.r.t. native apps. No different.)

    Ah… I get it now. Thanks.

    > Imagine layered, vector art controlled with media queries, for example:

    I agree if we moved to vector-based art for appropriate images, we would have fewer problems. But not every image makes sense vector-based.

    How would the same sky-at-night example work with pixel-based images?

    Also, do you see the same issues for video?

  8. Hey Jason,

    Great series. I’ve definitely enjoyed reading this and getting “schooled up” on the various techniques for responsive images.

    Based on this series and your earlier comments around CSS media queries, I decided to tinker with the cookie based technique to see where I could get to. It’s very early on with what I’m trying to do (haven’t been able to spend a lot of time), but I think there is some promise.

    Basically, the idea is to extend the cookie detection technique to redirect the client to a capabilities detection page which utilizes a combination of Modernizr + JS screen size detection. After this detection process, the cookie not only contains screen size information but also results of any other feature detection checks that were requested. You can then appropriately apply some server side logic based on the client capabilities – well that’s the idea anyway…

    I’d definitely value both yours and James’s feedback on what I’m trying to do, given the depth of experience that you both have in the area.

    Anyway, a couple of Git repos for you to look at if you have time:

    https://github.com/devcaps/devcaps
    https://github.com/devcaps/connect-devcaps

    I’m going to have a chat to a few guys during Web Directions South to collect some additional thoughts, and then post something on the technique a bit later on…

    Cheers,
    Damon.

  9. Absolutely – I’d much rather work together than reinvent the wheel. I’ll hit you up on this in a week or two :)

  10. Guy Podjarny says:

    Great series of posts, I agree the img tag needs to get better. Looking at it with more a performance bias (as opposed to design), I’m not a big fan of adding extra request data for every image request by default.
    The vast majority of sites will do nothing with the extra info, resulting in wasteful uploaded bytes – which are especially expensive on mobile networks.

    My vote would be to support an additional attribute that will make the browser add this extra info, to let the server make the call. This way it’s equally easy, but you need to opt in.

    While you’re at it, we should add an attribute that states whether to only download the image when it come into view. Today you have to get creative with JavaScript to achieve that. I see that as a part of responsive design, since on smaller screens it’s far less likely you’ll see the bottom images.

  11. Hi all,

    first thanks for mentioning my post to the W3 List. As I read some of the comments here I would like to explain why I posted to this list:

    First when I came up with the problem of responsive images I thought of HTML and CSS only. My first approach was to introduce a new img attribute. Some mentioned it is a bad idea regarding backward-compatibility.
    So I thought with some other people that best would be to have a responsive file-format for images. This is already real with some niche formats but as we can see on VP8/h.264 etc this won’t be available to broad mass the next few years.
    So this was the point when I thought: If we can have the real solution not now we need a good solution now. This is when I proposed the HTML5 img-tag changes to the W3 List.

    Result was not very good as the author here wrote: Many people thought it isn’t necessary to have such a solution because we don’t need it or can use solutions like the cookie-js-solution. But this is not a native solution which, in my mind, doesn’t make sense then…

    -Anselm

  12. Mairead says:

    first off, thanks for the mention jason. This is an amazingly useful collection of techniques. if only I’d had this when I started building our site it could have saved us a world of pain.

    Our key learning from building a responsive site is that its frustrating, its like turning the clocks back 10 years. You’re back in the land of hacks and crappy browsers again. I will gladly admit the noscript is a hack. I’m not in love with it, but perhaps having elements in the HTML that aren’t part of the loading DOM is an idea with legs?

    fwiw, I hate the idea of having to do ua-detection if you’re building for mobile. Thats ridiculous, even more ridiculous than doing it for desktop because the platforms are increasing at an exponential rate, where on earth do you stop and as scott mentions its a backward looking move – not future proof in any way.

    The question I have is how do you get feature detection without resorting to client side javascript or media queries? Surely its got to be in the HTML spec or the browser vendors hands? I fully support the idea of better headers that give you density, screen width, bandwidth, I really like the idea of fallback src attributes like the new video tag and I think HTML5 custom attributes could also play a big part in adding specific control over the images.

    In certain circumstances we wanted cropping rather than resizing, but we’ve ended up with a pretty big image and a tiny cropped area because its all handled via CSS. The noscript was something we decided to use sparingly. It could potentially be a heavy reflow and its not suitable for directing at every image on your site. In short I want more control and I fully applaud any efforts to get this written into the spec.

  13. Mark says:

    In all the discussions of responsive images I’ve seen little about bandwidth: the assumption is that mobile devices have bad bandwidth and non-mobile have good. But my iPhone is more often than not connected to a WiFi network. And a laptop might be using a 3G dongle. Knowing how a device is connected and what speeds it is getting would be really useful information for a web-developer: you could send more compressed images to low-bandwidth devices and better quality ones to high-bandwidth – wouldn’t matter what its form-factor was (although thats important too – no pint sending huge image to a tiny screen).

    I’d love to see either headers or a JS API that would tell you how a device is connected to the internet: connection type, average connection speed, metered or unmetered.

  14. Jacob Swartwood says:

    I’d first like to say that layout changes (portrait vs landscape) and drastic size differences should probably always be guided by responsive CSS. There is often more than just the image resolution changing in those cases, but I would like to scope this comment back to purely image scaling.

    I’d also like temporarily side-step restrictions of time, availability, support, impossibilities, etc with this comment… and think about long-term ideals.

    Let us entertain the concept of an image format that responds. And by that I don’t mean we even send headers or similar to adjust what is served. I like caching and don’t want image-sizing logic on my server. But what if the image container actually serves the image progressively? Think more along the lines of streaming video, but instead of frames, we load additional pixel density over time.

    With this strategy, the browser can make more of the tough decisions:
    – Stop loading depth (streaming data) as the image fills its container dimensions
    – Request more depth (resume streaming) if CSS changes (increases) the dimensions of the image
    – Throttle or defer pixel depth on slower connections

    That last point also lends this proposal to solve two of the problems with loading large images… Meaning instead of solving “we are loading unnecessary data (due to size constraints) AND we can’t afford to load this data (due to speed constraints)”, we may solve “we are loading unnecessary data (due to size constraints) OR we can’t afford to load this data (due to speed constraints)”.

    I am sure there are huge holes in this idea besides obvious ones like: what happens when this image format is used as a CSS background or something similarly undefined in scale (or size is defined only for crop and not render dimensions). Anyone think something like this could grow some legs to stand on?

  15. So our problem now lies with the built-in prefetcher (prerenderer?) in the newer versions of webkit/firefox, right? Does that pre-empt link prefetching in the header?

    Basically, can we specify a link rel=prefetch with a URL before the browser itself attempts a prefetch of that same URL?

    Does a prefetch link statement override the browser’s own prefetch?

  16. Jason Grigsby says:

    @Dustin, no, it isn’t the declared prefetching via the rel attribute. It is the look ahead preparser functionality that happens in browsers. A good description of how it works can be found here:
    http://blogs.msdn.com/b/ieinternals/archive/2011/07/18/optimal-html-head-ordering-to-avoid-parser-restarts-redownloads-and-improve-performance.aspx

  17. @Jason, okay that makes sense. Now that I have a better understanding, here’s a thought; the documentation you provided does say that a change in charset or document mode would cause the browser to abort the current look aheads.

    So by placing the charset declaration after the script that sets the cookie should force the speculative parser to reset the look aheads, right? Thereby allowing a cookie to be set and the look aheads to continue on with a set cookie after the rerun is forced.

  18. Jason Grigsby says:

    @Dustin, the problems are:

    1. You don’t want any look ahead downloads before the cookie is set. Setting the cookie will also abort the downloads and restart them, but we’re trying to avoid the performance hit from the extra downloads.

    2. Whatever is done now is based simply on the way browsers happen to work now. There is no contract between the developer and the browser that says that it will continue to work that way in the future. That’s the biggest issue.

  19. @Jason

    1. I agree with that, but that may be unavoidable with any workaround/hack that we can come up with, right? It may not be ideal, but if the performance hit is negligible compared to the savings in data transfer wouldn’t that still be worth it?

    2. I guess this is a bigger problem. Guess we should have really pressed the issue concerning that contract, eh? ;-)

  20. btw; Thanks for the info on look ahead prefetching, I had been trying to find something like that, extremely helpful!

  21. northk says:

    Don’t we really just want to measure download speed for a device that is connecting to our site? If so, why don’t we just measure it once on the first page visit and then set a cookie?

    It doesn’t have to be very accurate– we just need to know if the connection is “fast” or “slow” (above or below a threshold we choose). If the connection is fast, we serve up large images scaled down to fit the device screen. If the connection is slow, we serve up small images.

    Problems with this approach?