Cloud Four Blog

Technical notes, War stories and anecdotes

Responsive IMGs — Part 1

In my post “Where are the Mobile First Responsive Web Designs”, I noted that one of the first things I look for when trying to determine whether or not a responsive web design is “mobile first” is whether or not it has a strategy for handling the IMG tag.

A recent Smashing Magazine round up of responsive web design techniques included several new approaches for handling IMG tags which makes it the perfect time to dig into this problem and the potential solutions in more depth.

Why IMG Tags Suck for Responsive Web Design

If you want your site to load as quickly as possible, you do not want to deliver larger files than are needed. Many responsive web design sites provide mobile devices images at sizes appropriate for desktop and ask the mobile device to resize the image.

In my research, I found nearly 80% decrease in file size by delivering images at the actual size they were going to be used on a mobile device.

So what’s the problem with the IMG element in responsive designs? Unlike CSS images which can provide different source files based on screen resolution using media queries, IMGs have a single source attribute.

What are Responsive IMGs?

Responsive IMGs are images delivered using the HTML IMG tag that come from different sources depending the screen size. There are many different techniques for accomplishing Responsive IMGs.

As far as I can tell, Scott Jehl first coined the phrase Responsive Images to describe a javascript solution to the img source problem. He also referred to Responsive IMGs as a general term recently so I’m hopeful he doesn’t mind the fact that I’m extending his definition to describe any technique that attempts to provide images at an appropriate size for a responsive design.

Responsive IMGs Challenges

There are some common challenges that any Responsive IMG technique will face. As we review the various techniques that have been proposed, we need to keep these challenges in mind.

Minimum Bar: Start with Mobile, No Extra Downloads

Scott Jehl set a minimum bar for Responsive IMGs by stating they must do the following:

  1. Start with mobile img
  2. Upgrade to larger size without downloading both

Both of these are worthy and necessary goals.

The First Page Load Problem

Any solution that relies on client-side scripting to make a decision about what image source to display will suffer from a first page load problem. The first time someone visits a site, the server won’t know what size image to provide.

Hmm, Huston - we have a problem... no idea what we can deliver to this client

Image from Bryan Rieger’s Muddling Through the Mobile Web presentation, photo by wscullin, licensed under Creative Commons.

If javascript is added that determines what image size is appropriate, then this information can be retained for the user session via cookies or similar techniques. In theory, on subsequent requests the server can make a decision about what size image to include in IMG tag.

FWIW, the speed of first load is a big deal. The speed of a person’s first experience can dictate their impression of a product and company. Google, Yahoo and others have talked about how minor speed differences makes a big difference in usage of their products.

Rendering Race Conditions

Techniques that rely on adjusting the image source attribute via javascript need to make sure that the modification happens before the image requests start.

Browser makers have done a lot of work to download as many assets as possible at the same time. Usually this is a good thing. But in the case of responsive imgs, the javascript needs to evaluate what size image to retrieve before any image requests start.

A lot of earlier work was done using dynamic base tags. This worked when the javascript was inline in the head tag, but failed to prevent images from downloading twice when an external javascript file is used.

The upshot is that nearly every client side technique requires deep understanding of the order in which different browsers process and download assets. Or more realistically, each approach needs to be tested extensively.

Content Delivery Networks and Caching

When you deliver different size images at the same url, you can run into problems with CDNs and other caching at the edge of the network. If the first person to request an image is on a mobile phone, people who follow via the same CDN or cache will also see the mobile-optimized image even if they are on desktop unless consider CDNs in your strategy.

Future Friendly Responsive IMGs

If we accept that the “quantity and diversity of connected devices—many of which we haven’t imagined yet—will explode”, then we need to consider look for solutions that are future friendly. In addition to the current experimentation, we need to start thinking about what a long term solution might look like.

For example, many of the early solutions for Responsive IMGs consist of two size images: one for desktop and one for mobile screen sizes. Will two image sizes really suffice for all of the devices that are coming?

Also, a lot of the solutions right now tackle one part of the problem. They may tackle the client side changes to switch the image source, but leave as an exercise for the developer to figure out how image resizing will be handled. For shared libraries, limited scope makes sense.

But as we look at what systems will need to do to be successful in the future, we need to think about what we want out of both the server and client side.

Here are some of the things that I think a future friendly technique will need to consider:

  • Support arbitrary image resizing — We cannot anticipate what screen sizes may be coming. We need systems that handle image resizing automatically and support any arbitrary size needed for a particular page.
  • Art direction can override automatic resizing — Not every image can be resized without losing the meaning of the image. Sometimes cropping an image may work better than resizing it. Automatic tools need to easily support manual override.

automagically scaling detailed images isn’t ideal

  • Support for higher resolution displays — What do we do with the iPhone 4’s retina display and other devices sharing similar high resolution screens? It is an open question about whether we should deliver higher resolution images to those devices given the performance hit that will occur if the person is on a slow connection.
     
    But regardless of how we chose to handle it right now, it is clear that the trend towards more pixels per inch on displays is not going away. If anything, we’re seeing indicators that higher density will soon be available on desktop displays as well.

    This means that our current definition of what is a large image for web is probably too small for future devices. With that in mind, it probably makes sense for systems to accept the highest resolution image possible—even if that resolution isn’t currently being used—so that when new devices become available the high resolution source is already available and hasn’t been lost.

  • Connection speed should be part of the criteria — We can be much smarter about the size of the image we deliver if we can tell something about the network connection. We need an easier way to get at this information.
  • A replacement for the IMG tag? — All of the responsive image solutions are attempting to deal with the fact that the image tag has only a single source. There have been various proposals recently to take a new look at what the tag should be and see if we can find a long term replacement.

That’s my short list. What would you add?

In part 2, I’ll take a closer look at the current alternatives for responsive imgs and which ones hold the most promise.

22 Comments on “Responsive IMGs — Part 1”

  1. Andy Hume says:

    Hi Jason – Great piece.

    I agree that what we’re implementing now will likely turn out to not be future-friendly, and it’s great that discussion about genuine long-term solutions is happening. Our current solutions are hugely dependant on the current (and undefined) behaviour of browsers in regard to the page-load race conditions you mention. For example, most responsive image implementations would be compromised if a particular type of look-ahead pre-parser (http://goo.gl/TyzTi) began to speculatively download images before actually parsing the HTML or executing any script. (I half expect us to get bitten by this any day.) One way or the other we need to consort with browser makers to get future-friendly.

    “There have been various proposals recently to take a new look at what the tag should be and see if we can find a long term replacement.”

    I don’t believe I’ve seen these. Can you point me towards the proposals?

    What I’d been imagining in this area isn’t a replacement for the img element as such, but for the browser to provide APIs to let us interact with them at a slightly lower level.

    For example, a mechanism for explicitly deferring requests made by certain img elements until we’ve determined the characteristics of the browsing environment and the type of image we want to load. Or perhaps a collection of pre-request events on the window that let us listen for and hijack requests made by the browser – then either cancel them or alter the path before they leave (putting aside the unexamined security implications). I wonder if this type of approach would be preferable from a backwards compatibility and progressive enhancement point of view?

    That’s not to say that optional attributes and other backwards compatible features couldn’t be added to the img element to cover common use cases using a more declarative syntax. Eg, for images targeted at retina displays?

    But maybe I’m being too conservative in my approach.

    Thanks for thinking about this stuff.

  2. Nice roundup of the challenges facing us. The problem, of course, isn’t just one of content, it’s context as well. As you point out, some images shouldn’t be scaled, nor am I looking forward to generating four versions of the same resource, batch processing or not.

    Perhaps a wiser course of action would be to identify and define the types of images we need, and then modify the existing img tag as we have with form inputs, so that the appropriate action is taken within the device based on image type.

  3. Jason Grigsby says:

    Minor edit. The post originally said:

    “We shouldn’t put in place solutions that we will have to replace in the near future.”

    I changed it to:

    “In addition to the current experimentation, we need to start thinking about what a long term solution might look like.”

    The previous sentence sounded too prescriptive compared to what I’m thinking. We “shouldn’t” blah blah blah. Jeez. It’s pretty early. We’re all building scaffolding right now. How did that sentence get past the editors?

  4. Jason Grigsby says:

    @andyhume There has been some discussion on W3C or WhatWG mailing lists and some blog posts. No formal proposals. I was planning to link to a couple of them in part 2 so I’ll make sure to round up what I’ve seen.

  5. Yoav Weiss says:

    Great article! A few things I have to add:
    * I’ve seen (on the filament group’s cookie solution) that they use a “Vary: cookie” header to tackle the intermediary caching issues. This requires special care if the domain serving the images has other cookies on it, since you might end up invalidating these images for both client-side & intermediary caches.
    * I have seen a race condition mainly in FireFox. Chrome (and probably other webkits) block the preload scanner for scripts that are in the “head” tag (http://gent.ilcore.com/2011/01/webkit-preloadscanner.html)

    @andyhume – I have made a very simple proposal on http://blog.yoav.ws/2011/07/Simpler-responsive-images-proposal .Basically I suggest to add media queries to “base” and “img” tags which will enable URL changes according to screen size.

  6. Scott Jehl says:

    Great writeup, Jason. Looking forward to part 2!

    In considering browser changes to allow for a better approach, I’m unsure which would be better: additional feature-or-environmental headers passed by the user agent, or a client-side approach like Andy mentioned above. Additional headers would provide a more sustainable mechanism for determining device classes on the server, either with named common media types or inferred through device resolution, and that would lessen the need to maintain device databases that are used to perform the same sort of logic today (with more work and potential for missteps).

    However, even if the server could be aware of such information in a device-agnostic way, I still think it’d be more flexible to have media-query syntax at our disposal for configuring an image src. Andy’s fork of responsive images has shown that it’s quite valid to consider layout conditions other than the screen or browser’s width to determine which image to request, and that’s something that would be very hard to replicate through additional headers.

    Essentially, I think we’ll see more and more that the asset we request will depend not only on device/window/connection-speed constraints, but also on the content and layout of the elements in play.

  7. Dave Smith says:

    Could a complimentary method of attack be added… An image format that allows devices to decide how much they need to download. One URL for an image no matter what size or resolution it is.

    A world without the need to create thumbnails (programmatically or manually) would be cool :)

    I imagine the images would be larger KB-wise on the whole than regular JPEGs etc. But overall less bandwidth may be used as only a proportion of one image is downloaded rather than two (thumbnail and larger version).

    Half wondered whether Progressive JPEGs could do the job, with a bit of an arm-wrestle and a kick.

    I would’t think suggesting a new format produces happy thoughts to many, but I maybe it’s worth a look considering future friendliness.

    Ha, I always do this, half way through a point and I try and defeat my own thoughts! Surely SVG is the way to go! :D

  8. albert says:

    have you considered/tested using sprites via clip:; in coordination with changing background-size,background-position and background-cover with media-queries?
    would be quite tedious at first, certainly would not work for all instances, but i bet that’s a good way to go for responsive images.

  9. Melissa says:

    Great article – though I can’t help noticing the image says “Huston, we have a problem” instead of Houston. Is this inside convo or a typo?

  10. In the short term, I believe a lot of wonderful effort has been put forth to try to address the “right image for the job” issue.

    From a big picture perspective, I think this issue extends beyond our noble efforts and is now a “browser issue”. That’s not to say that everything done to this point and beyond now is futile. Of course it’s valuable – if only to keep exposing the shortcomings of what exists now. These attempts further illustrate the need for the browser to adapt to these emerging needs and provide the solution as well (in the HTML spec would be super).

    I don’t think paving cowpaths in this particular regard will be beneficial. Reviewing all the attempts thus far and where they fall short serves as a great starting point for finding a reasonable, implementable, sustainable solution.

  11. karl says:

    Did you follow the discussion which happened on different mailing lists?

    http://lists.w3.org/Archives/Public/public-html/2011May/thread.html#msg386

    also on whatwg on August 2011, as well as the style mailing list. We haven’t reached yet an agreement on how to do it. Please follow up on these emails and share your opinions.

  12. Jason Grigsby says:

    @Melissa The image is from Bryan Rieger’s presentation. He is from Edinburgh. I thought it might be a British joke. It could just be a typo. I found it endearing either way. :-)

    @karl Thank you. That was one of the threads I mentioned in my comment to Andy. Saves me digging it up for part 2.

  13. James Pearce says:

    Yes we need more than scaling. This has been Bryan’s ambition for years (e.g. http://www.slideshare.net/bryanrieger/prime-sky slide #20).

    I toyed with the idea of tinySrc + http://en.wikipedia.org/wiki/Seam_carving for a while, but it’s hard for generalized use. (e.g. labeled vector diagrams; not just photographic images).

    I think the architecturally correct solution would not be a new image tag, but a content- and context-aware image format. (Imagine SVG with built-in media queries). I know of no such initiative.

    As for cookies, JavaScript, roundtrips, feature detection, blah blah blah… keep it simple. Dare I say I have a horse in this race? ;-)

  14. [...] friend Viktor at We Made You Look pointed me to this article at Cloud Four on Responsive IMGs. The article makes a case for what is the problem with IMG tags when doing responsive web design. [...]

  15. Stephen Kaiser says:

    Nice article, Jason. Thank you!

    I’ve been following this topic with great interest over the last few months and reading this article made me realize that while there are some ingenious ideas and work going on to solve the challenges, maybe we are simply over-thinking it. Maybe the path to a solution has been in front of us all the time (or at least since 1996), but we have been avoiding it because it was originally abused by browser vendors? User-agent request-headers. Wait! Here me out. In RFC 1945, Section 10.15 (http://tools.ietf.org/html/rfc1945#section-10.15), the authors brilliantly envisioned the usefulness of the user-agent request-header: “The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and *automated recognition* of user agents for the sake of *tailoring responses* to avoid particular user agent limitations.” Unfortunately, they made it optional and also did not specify a strict format. Maybe this is the thing we need to be clamoring for standardization on? Maybe we should be demanding that browser and OS vendors be conforming to a strict format and providing more detailed and useful information (like not identifying yourself as Mozilla/5.0, when you’re not). If we can get a standardized format which provides an *accurate* identification of the browser and its host platform, we could reliably use server-side detection to deliver the appropriate content with no need to worry about race conditions, yada, yada, yada. What if a user-agent string contained the device dimensions? What if it had information about some basic functionality supported by the browser (AJAX, etc.)? What if it *consistently* identified itself as ‘mobile’, ‘handheld’, ‘tablet’ or ‘desktop’? Now, of course this won’t help identify existing browsers, but one of the main arguments against the user-agent header detection is future-proofing. This would effectively eliminate that as new browsers would conform to the new standard. Old browsers already have known strings, so those could just be compiled into a lookup library (which they already have been – http://wurfl.sourceforge.net/). Then taking into account browser half lives (http://infrequently.org/2011/03/half-lives/), the problem would be solved in a short time moving forward (except IE – http://infrequently.org/2010/10/ie-8-is-the-new-ie-6/ and http://paulirish.com/2011/browser-market-pollution-iex-is-the-new-ie6/).

    Does this seem unreasonable? What do you think might be problems with this approach (constructively, please)? It just seems that we are just starting to make things more difficult than perhaps they need to be and that we simply need to step back and look at EVERYTHING objectively and without prejudice to see what solutions that maybe didn’t work before, do work now based on the ever-changing landscape.

    Thanks!

  16. Jason Grigsby says:

    @Stephen I don’t have the same aversion to device detection that others do. It’s a tool with benefits and disadvantages. I’m going to write about it separately sometime soon.

  17. James Pearce says:

    Stephen: indeed it shouldn’t be heresy! Thanks for the RFC reality check ;-)

  18. @Stephen – I could see “User-agent request-headers, with 100% less lies” as a viable solution for being able to determine which image is the right one for the job based on device. That’s about 50% of the problem solved if that were to become a reality.

    The rest of the problem comes from not being able to accurately determine the connection worthiness of the device. I see that as being another big element that needs to be tackled to deliver the correct image for the situation. I believe that the quality of the image could be adjusted, within reason, if we had access to that kind of information.

    I’ve played plenty of MMORPGs in my time and games seem to be able to provide lag information, I would like for us to know something like that for web sites and apps. :)

  19. Scott Jehl says:

    The UA vs. Features debate is certainly angrier than it needs to be! :)

    The reality of this is that both approaches have very real disadvantages today, and sometimes a blend of the two is required. What I’m most concerned with is how developers would prefer to build on the web, assuming all techniques were reliable. If developers would prefer device-agnostic tools as I suspect, then we are not there yet, and browser vendor and standards advocacy is needed.

    I think the primary aversion many people have with UA sniffing is that standards-oriented web developers are drawn towards techniques that are future-friendly. Basically, is there a good chance that what I build today will work tomorrow? Unfortunately, no matter how correct and reliable your UA detection logic and matching may be, the very nature of that detection process is rooted in a known past, relying on constant updates by humans or machines to keep it working properly. It’s reactive rather than proactive; obsolete at the moment of release, as new devices with new blends of features and capabilities are released every day. Even if perfectly automated, this is an awful lot of work, and it does not match the promise of a forward-looking, standards-based approach.

    And yet, UA is a tool like any other, and we do often need it today as some features are not simple to detect (perhaps even screen size, unfortunate as that may be) and the occasional false positive or negative is not a myth.

    Rather than arguing over whether we should use UA at all today, I’d rather see some agreement that we’d rather not NEED to use it tomorrow. And if that’s our goal, what features should we advocate to browser makers?

    Are additional headers enough? I’m not sure they will be (see first comment above).

  20. seutje says:

    No need to reinvent the img element, CSS spec already contains tools to easily solve this, they’re just not implemented yet.
    Have a look at http://nicolasgallagher.com/responsive-images-using-css3/

    (Holy fuck this form was annoying to use on my phone)

  21. seutje says:

    @Stephen: I spoof my UA string all the time, mainly due to ppl sniffing them and making silly assumptions based on that…
    It is impossible to detect all current and future browser, letalone know what they support and how well it’s supported

  22. Travis says:

    If you want your site to load as quickly as possible, you do want to deliver larger files than are needed. …you mean ‘don’t'?