Responsive IMGs Part 3 — Future of the IMG Tag

I disagree. It’s the image that needs to be self-adaptable, not only the HTML around it.

Every solution out there right now (including my own, no doubt) is a hack*

But you’re presuming that images never need to be responsive outside of an HTML-centric document environment. What about images used in web apps, where there is of course a DOM, but no HTML to speak of. What about native apps, which need just as much equivalent flexibility?

Perhaps we feel very comfortable specifying, hacking, overriding, polyfilling etc etc the behavior of servers or browsers at a textual (HTML? CSS?) abstraction, rather than at a (SVG? binary?) image format level. But the latter is the right architectural choice. I can feel it.

I guess I need to think more specifically about what this means. But I urge us all to think about the asset challenge as an asset challenge. Not a classic-web-superstructure challenge.

* Although this is not necessarily a bad thing – even the img tag itself had a quick and brutal birth: http://1997.webhistory.org/www.lists/www-talk.1993q1/index.html#182 (but wow, check out that author roster!)

Shooting from the hip, the requirements you state make me think the a refurbished img tag would look a bit like the video or object tags, with the possibility to default sequentially from one alternative to another depending on parameters associated to each element (possibly using some kind of media-queries like syntax to select each case), and with a catch-all fallback at the end. This would have some nice effects (discoverability, readability and maintainability for developers, possibility to associate CSS). Not that video/object are terrifically good examples of how to handle adaptive media.

As far as inherently adaptive formats are concerned, JPEG2000 was an attempt in that direction — but I presume the lack of popularity has something to do with IPR.

Given your criteria list, are we asking for a tag (or super tag) in this case that does too much?

I see a very big danger with a tag this complex when it comes to asking the browser makers to implement this consistently.

I am not sure that simply adding some header information would not resolve this problem in the majority of cases. This would retain the simplicity of the image tag.

@james wrote:

“But you’re presuming that images never need to be responsive outside of an HTML-centric document environment. What about images used in web apps, where there is of course a DOM, but no HTML to speak of. What about native apps, which need just as much equivalent flexibility?”

Web apps with no HTML write tags to the DOM, right? That to me doesn’t seem like a big distinction. If we have a tag, it will be part of the DOM and web apps will be able to manipulate it via javascript.

With regards to native apps, they already have the control they need to select different image sizes since they can control the order, timing, and assets retrieved from the network.

I agree that a different asset format might be useful for native apps, but that’s not the problem set I’m concerned with.

“But I urge us all to think about the asset challenge as an asset challenge. Not a classic-web-superstructure challenge.”

Perhaps it is both. A new image format won’t solve the problem where someone wants to select a different image instead of simply a resized image for a particular context. A new asset format won’t solve that problem.

That’s not to say a different asset format wouldn’t be helpful. I’m open to the idea. Would love to see it just like I’d love to see more header information.

@jamesking wrote:

“I am not sure that simply adding some header information would not resolve this problem in the majority of cases. This would retain the simplicity of the image tag.”

Header information requires a server to make decisions based on the headers. What other HTML element requires a server?

Do we want to start making that a requirement now when we’re just now seeing Windows 8, WAC, WebOS, etc. all adopt encapsulated web apps as a primary way of building apps? We’d cut that movement off at the knees.

> Web apps with no HTML write tags to the DOM, right?

I see plenty of discussion around loading sequences, aggressive loading, race conditions, cookie setting and other malarkey. These are document problems. A DOM which is created programmatically can completely avoid these issues, imperatively.

(Which is then indeed your own point w.r.t. native apps. No different.)

> A new image format won’t solve the problem where someone wants to select a different image instead of simply a resized image for a particular context.

Why not? Solving this would *be* the reason to come up with a new format.

Imagine layered, vector art controlled with media queries, for example:

“Here is the sky-at-night. When viewed at 320px (or some DPI, say), keep level 1 labels 10px or higher, stars at 2px or more, constellation lines at most 1px. When viewed at 980px or more, stars should be rendered sprites, constellation lines 2px or more, and show level 1-3 labels”

etc etc

So content should flow like water? Yes! But the image is the content.

The image should flow. We need to want to be smarter than just putting different sized stones into a document river which flows (yah yah, like water) around them.

Images themselves need to know how to adapt to their constraints and opportunities.

@jamespearce wrote:

> I see plenty of discussion around loading sequences, aggressive loading, race conditions, cookie setting and other malarkey. These are document problems. A DOM which is created programmatically can completely avoid these issues, imperatively.
>
> (Which is then indeed your own point w.r.t. native apps. No different.)

Ah… I get it now. Thanks.

> Imagine layered, vector art controlled with media queries, for example:

I agree if we moved to vector-based art for appropriate images, we would have fewer problems. But not every image makes sense vector-based.

How would the same sky-at-night example work with pixel-based images?

Also, do you see the same issues for video?

Hey Jason,

Great series. I’ve definitely enjoyed reading this and getting “schooled up” on the various techniques for responsive images.

Based on this series and your earlier comments around CSS media queries, I decided to tinker with the cookie based technique to see where I could get to. It’s very early on with what I’m trying to do (haven’t been able to spend a lot of time), but I think there is some promise.

Basically, the idea is to extend the cookie detection technique to redirect the client to a capabilities detection page which utilizes a combination of Modernizr + JS screen size detection. After this detection process, the cookie not only contains screen size information but also results of any other feature detection checks that were requested. You can then appropriately apply some server side logic based on the client capabilities – well that’s the idea anyway…

I’d definitely value both yours and James’s feedback on what I’m trying to do, given the depth of experience that you both have in the area.

Anyway, a couple of Git repos for you to look at if you have time:

https://github.com/devcaps/devcaps
https://github.com/devcaps/connect-devcaps

I’m going to have a chat to a few guys during Web Directions South to collect some additional thoughts, and then post something on the technique a bit later on…

Cheers,
Damon.

@Damon, looks very cool. We should probably team up on https://github.com/jamesgpearce/modernizr-server and https://github.com/jamesgpearce/monomi 🙂

Absolutely – I’d much rather work together than reinvent the wheel. I’ll hit you up on this in a week or two 🙂

Great series of posts, I agree the img tag needs to get better. Looking at it with more a performance bias (as opposed to design), I’m not a big fan of adding extra request data for every image request by default.
The vast majority of sites will do nothing with the extra info, resulting in wasteful uploaded bytes – which are especially expensive on mobile networks.

My vote would be to support an additional attribute that will make the browser add this extra info, to let the server make the call. This way it’s equally easy, but you need to opt in.

While you’re at it, we should add an attribute that states whether to only download the image when it come into view. Today you have to get creative with JavaScript to achieve that. I see that as a part of responsive design, since on smaller screens it’s far less likely you’ll see the bottom images.

Hi all,

first thanks for mentioning my post to the W3 List. As I read some of the comments here I would like to explain why I posted to this list:

First when I came up with the problem of responsive images I thought of HTML and CSS only. My first approach was to introduce a new img attribute. Some mentioned it is a bad idea regarding backward-compatibility.
So I thought with some other people that best would be to have a responsive file-format for images. This is already real with some niche formats but as we can see on VP8/h.264 etc this won’t be available to broad mass the next few years.
So this was the point when I thought: If we can have the real solution not now we need a good solution now. This is when I proposed the HTML5 img-tag changes to the W3 List.

Result was not very good as the author here wrote: Many people thought it isn’t necessary to have such a solution because we don’t need it or can use solutions like the cookie-js-solution. But this is not a native solution which, in my mind, doesn’t make sense then…

-Anselm

first off, thanks for the mention jason. This is an amazingly useful collection of techniques. if only I’d had this when I started building our site it could have saved us a world of pain.

Our key learning from building a responsive site is that its frustrating, its like turning the clocks back 10 years. You’re back in the land of hacks and crappy browsers again. I will gladly admit the noscript is a hack. I’m not in love with it, but perhaps having elements in the HTML that aren’t part of the loading DOM is an idea with legs?

fwiw, I hate the idea of having to do ua-detection if you’re building for mobile. Thats ridiculous, even more ridiculous than doing it for desktop because the platforms are increasing at an exponential rate, where on earth do you stop and as scott mentions its a backward looking move – not future proof in any way.

The question I have is how do you get feature detection without resorting to client side javascript or media queries? Surely its got to be in the HTML spec or the browser vendors hands? I fully support the idea of better headers that give you density, screen width, bandwidth, I really like the idea of fallback src attributes like the new video tag and I think HTML5 custom attributes could also play a big part in adding specific control over the images.

In certain circumstances we wanted cropping rather than resizing, but we’ve ended up with a pretty big image and a tiny cropped area because its all handled via CSS. The noscript was something we decided to use sparingly. It could potentially be a heavy reflow and its not suitable for directing at every image on your site. In short I want more control and I fully applaud any efforts to get this written into the spec.

In all the discussions of responsive images I’ve seen little about bandwidth: the assumption is that mobile devices have bad bandwidth and non-mobile have good. But my iPhone is more often than not connected to a WiFi network. And a laptop might be using a 3G dongle. Knowing how a device is connected and what speeds it is getting would be really useful information for a web-developer: you could send more compressed images to low-bandwidth devices and better quality ones to high-bandwidth – wouldn’t matter what its form-factor was (although thats important too – no pint sending huge image to a tiny screen).

I’d love to see either headers or a JS API that would tell you how a device is connected to the internet: connection type, average connection speed, metered or unmetered.

I’d first like to say that layout changes (portrait vs landscape) and drastic size differences should probably always be guided by responsive CSS. There is often more than just the image resolution changing in those cases, but I would like to scope this comment back to purely image scaling.

I’d also like temporarily side-step restrictions of time, availability, support, impossibilities, etc with this comment… and think about long-term ideals.

Let us entertain the concept of an image format that responds. And by that I don’t mean we even send headers or similar to adjust what is served. I like caching and don’t want image-sizing logic on my server. But what if the image container actually serves the image progressively? Think more along the lines of streaming video, but instead of frames, we load additional pixel density over time.

With this strategy, the browser can make more of the tough decisions:
– Stop loading depth (streaming data) as the image fills its container dimensions
– Request more depth (resume streaming) if CSS changes (increases) the dimensions of the image
– Throttle or defer pixel depth on slower connections

That last point also lends this proposal to solve two of the problems with loading large images… Meaning instead of solving “we are loading unnecessary data (due to size constraints) AND we can’t afford to load this data (due to speed constraints)”, we may solve “we are loading unnecessary data (due to size constraints) OR we can’t afford to load this data (due to speed constraints)”.

I am sure there are huge holes in this idea besides obvious ones like: what happens when this image format is used as a CSS background or something similarly undefined in scale (or size is defined only for crop and not render dimensions). Anyone think something like this could grow some legs to stand on?

So our problem now lies with the built-in prefetcher (prerenderer?) in the newer versions of webkit/firefox, right? Does that pre-empt link prefetching in the header?

Basically, can we specify a link rel=prefetch with a URL before the browser itself attempts a prefetch of that same URL?

Does a prefetch link statement override the browser’s own prefetch?

@Dustin, no, it isn’t the declared prefetching via the rel attribute. It is the look ahead preparser functionality that happens in browsers. A good description of how it works can be found here:
http://blogs.msdn.com/b/ieinternals/archive/2011/07/18/optimal-html-head-ordering-to-avoid-parser-restarts-redownloads-and-improve-performance.aspx

@Jason, okay that makes sense. Now that I have a better understanding, here’s a thought; the documentation you provided does say that a change in charset or document mode would cause the browser to abort the current look aheads.

So by placing the charset declaration after the script that sets the cookie should force the speculative parser to reset the look aheads, right? Thereby allowing a cookie to be set and the look aheads to continue on with a set cookie after the rerun is forced.

@Dustin, the problems are:

1. You don’t want any look ahead downloads before the cookie is set. Setting the cookie will also abort the downloads and restart them, but we’re trying to avoid the performance hit from the extra downloads.

2. Whatever is done now is based simply on the way browsers happen to work now. There is no contract between the developer and the browser that says that it will continue to work that way in the future. That’s the biggest issue.

@Jason

1. I agree with that, but that may be unavoidable with any workaround/hack that we can come up with, right? It may not be ideal, but if the performance hit is negligible compared to the savings in data transfer wouldn’t that still be worth it?

2. I guess this is a bigger problem. Guess we should have really pressed the issue concerning that contract, eh? 😉

btw; Thanks for the info on look ahead prefetching, I had been trying to find something like that, extremely helpful!

Don’t we really just want to measure download speed for a device that is connecting to our site? If so, why don’t we just measure it once on the first page visit and then set a cookie?

It doesn’t have to be very accurate– we just need to know if the connection is “fast” or “slow” (above or below a threshold we choose). If the connection is fast, we serve up large images scaled down to fit the device screen. If the connection is slow, we serve up small images.

Problems with this approach?