Cloud Four Blog

Technical notes, War stories and anecdotes

Sensible jumps in responsive image file sizes

Last year, I wrote about the challenges of picking responsive images breakpoints and how I found it a nearly unsolvable problem. It has vexed me since.

But I have a new idea on how we might be able to define responsive image breakpoints that is based on a performance budget.

Before I begin, I should note that a lot of this is a thought experiment. I don’t yet know how practical this approach would be.

Brief refresher on responsive image breakpoints

Without going into all of the details about responsive image breakpoints, the short version is that most people are picking the breakpoints for responsive images based on one of two criteria:

  1. Based on what Scott Jehl referred to as “sensible jumps in file size to match screen dimension and/or density” OR
  2. Simply matching the image breakpoints to the major breakpoints being used for the design.

While the first method is more efficient and will probably result in better image sizes, my suspicion is that defining “sensible jumps in file size” is so nebulous that most web developers are going to choose to do the second, easier option.

That is unless we can find a formula to calculate what constitutes a sensible jump in file size and that’s what got me thinking about performance budgets.

What is a performance budget?

I’m not sure how long the idea of a performance budget has been around, but I first became cognizant of the idea when Steve Souders talked about creating a culture of performance on the Breaking Development podcast.

Tim Kadlec expanded on the idea in a recent blog post. He cites the BBC which determined that “each page to be usable within 10 seconds on a GPRS connection and then based their goals for page weight and request count on that.

So that’s the basic idea. Establish a performance budget and stick to it. If you add a new feature to the page and you go over budget, then you have a three options according to Steve (and transcribed by Tim):

  1. Optimize an existing feature or asset on the page.
  2. Remove an existing feature or asset from the page.
  3. Don’t add the new feature or asset.

What is the performance budget for flexible images?

Let’s apply this idea of a performance budget to responsive design. In particular, let’s treat the idea of flexible images as a feature. Because flexible images are a feature, we need a budget for that feature.

And as long as we’re making up the rules, let’s establish a few more hypotheticals:

  • The page we’re working with has 10 images on it of varying formats and visual content.
  • We haven’t reached our performance budget yet so we don’t have to remove other features, but we still need to make sure that flexible images do not add too much to the page weight.
  • We’ve concluded that flexible images can add up to 200k to the page above what the size of the page would be if we provided fixed width images. We picked 200k because it is ~1 second at HSDPA (recent mobile) speeds. And well, 200k is a nice even number for this thought experiment.
  • Because this page has 10 images on it, each image has a 20k budget for flexible images.

One thing to keep in mind, 200k isn’t the cap for the file size of all ten images combined. Instead, it is the price we’re willing to pay for using flexible images instead of images that are perfectly sized for the size they are used in the page.

For example, say you had a responsive web page with the following image on it:

fish-and-chips

That image is 500×333 pixels and 58K in file size.

Now imagine a visitor views that web page and based on the size of their viewport, the image is displayed at 300×200 pixels, but the source image is still the same. The cost of using flexible images is the difference in file size between what the image would be saved and optimized at 300×200 versus the file size of the image downloaded at 500×333.

In this case, I’ve taken that example image and resized it to 300×200 and saved it with the same compression level as the 500×333 image to see what the file size cost is of using that flexible image.

Source Width Height File Size
Flexible image 500 333 58k
Image matching size used in page 300 200 24K
Total extra download - - 34K

In this example, the visitor is downloading an extra 34k of image data because they are downloading a flexible image instead of downloading one that had been resized to the exact size being used in the page.

Translating the budget into breakpoints

Let’s go back to the page we want to optimize—the one with ten images on it and a total performance budget of 200k for flexible images. How do we translate that into image breakpoints?

Thinking back to the example above, the price for using flexible images is the difference between the size of the file that is downloaded and the size the file would have been if perfectly sized for its use in the page.

Our budget says that we can only download up to an extra 20K per image. Therefore, we need to make sure that we have a new image breakpoint every time the size of the image increases 20K.

We now have a methodology for picking sensible jumps in image file size that is tied to user experience instead of picking them arbitrarily.

Finding the breakpoints

How would this translate into a heuristic that could be used to find the breakpoints? You would need the following:

  • What is the minimum size this image will be used at? (In our example, let’s say 320×213)
  • What is the file size between breakpoints? In other words, what is your per image budget? (20k for our example)
  • A high quality source file to use for resizing.
  • Optionally, the largest size the image will be used at. (990×660 for this example).

Once you have this information, the basic logic looks like this:

  1. Take the source image and resize it to the smallest size the image will be used at.
  2. Keep the file size of that image handy.
  3. Start a series of tests that create new image files from the source that are gradually getting bigger.
  4. Check each image created. If the difference between the file size of the new image and the image file size you stored is less than your budget, discard the new image.
  5. When you find an image that hits your budget, save that image and replace the previous file size that you stored with the new file size.
  6. Repeat steps 2 through 5 until you reach either the maximum resolution of the source image or the largest size the image will be used at.

I’m tickled to say that my co-founder John Keith got excited by this idea and built a rough prototype of how this might work.

Sample Page

Using the script that John built, I created a demo page containing ten images. The source images were 990 by 660 pixels and all but one of them were saved as JPEGs at 50% quality. The one exception is a PNG8 image with an optimized color palette.

I tried to pick a variety of images so we can see how each image might have different breakpoints using our budget. Let’s take a look at three sample images.

Time Square — 8 Image Breakpoints


This image has a lot of visual diversity. The variations in colors and textures means that JPEG’s lossy compression cannot do as much without damaging the image quality.

Because of this, there are eight image breakpoints—set at 20k intervals—between the smallest size of the image (320×213) and the largest size of the image (990×660).

Breakpoint # Width Height File Size
1 320 213 25K
2 453 302 44K
3 579 386 65K
4 687 458 85K
5 786 524 104K
6 885 590 124K
7 975 650 142K
8 990 660 151K

Morning in Kettering — 3 Image Breakpoints


Unlike the Times Square image, this image has a lot of areas with very similar colors and little variation. Because of this, JPEG can compress the image better.

On an image that can be compressed better, our 20K budget goes farther. For this image, we only need three image breakpoints to cover the full range of sizes that the image will be used at.

Breakpoint # Width Height File Size
1 320 213 9.0K
2 731 487 29K
3 990 660 40K

Microsoft Logo — 1 Image Breakpoint


Microsoft Logo

This is a simple PNG8 file. At its largest size (990×660), it is only 13K. Because of this, it fits into our 20K budget without any modifications.

On a recent consulting project with a company that has over 800,000 images on its site, we identified a class of images—some icons, little badges, etc.—where the size of the image on desktop retina was not much different than the size used on mobile either because the image resolution doesn’t vary much or because the image compresses well. For those images, we decided to deliver the same size image to all screen sizes.

Breakpoint # Width Height File Size
1 990 660 13K

Every image is different in the way it compresses

Take a look at the other images on the sample page we created. See how the number of breakpoints vary even through all the images start with the same resolution end points.

This diversity exists despite the fact that with the exception of the Microsoft logo, all of the images start at the same size with the same compression settings. On a real site, we would see even more diversity with varying levels of JPEG quality, PNG8 with gradients going horizontally instead of vertically, and PNG32 images in the mix.

But what intrigues me about this approach to setting breakpoints is that it we wouldn’t be setting one-size-fits-all image breakpoints. Instead, we would make decisions about where the breakpoints should exist based on the our goals for user experience—the performance budget—and the unique characteristics of the image and how will it can be compressed.

What conclusions can we draw from this thought experiment?

The point of this thought experiment wasn’t to provide a complete methodology to set responsive image breakpoints. I started by simply asking the question about whether we might be able to use performance budgets to come up with a way to calculate what are sensible jumps in image sizes.

But the outcome of this exercise has caused me to draw some interesting conclusions as well as sparking more questions about responsive images:

  • Images do contain clues that can tell us where the breakpoints should be.
    Last year I wrote that “the problem is there is nothing intrinsic to the image that would guide you in deciding where you should switch from one size of the image to another.” But this experiment shows that images do have intrinsic information—how well the image compresses, what type of compression is being used, the range in size between the smallest and largest use of an image—that can be used to decide when you should switch from one source file to another.
  • We can set a performance budget for flexible images.
    There’s no reason why we can’t treat the use of flexible images like any other feature that we add to a page and define a performance budget for its use. In fact, setting a performance budget for flexible images could be the key to making informed decisions about where image breakpoints should be set.
  • Automated image resizing and compression is a must.
    I have already written about how automated image services will likely be a must for sites in the future. If a company wanted to use a technique like this to set their breakpoints, they will need an automated way to do it.
  • Can we set a performance budget for flexible images across an entire site?
    For our sample page, setting the performance budget to 200K for the whole page worked well. But in the real world, we often don’t know how many images are going to be on a given page. Similarly, we may not know what pages a given image is going to be added to. It seems like it would be useful to be able to say that for any given flexible image on the site, we’ve established a 20K budget. It would be less precise than a per page limit, but it may be the only practical way to translate this thought experiment into a production environment.
  • An image and its breakpoints could be stored as a bundle.
    The outcome of this approach to image breakpoints is that the breakpoints could be specific to the image no matter what context the image is used in. You could store the calculated breakpoints with the image and whenever the image is displayed on a page, no matter what size the image is used at within the page, the same breakpoints could be used.
  • An image and breakpoint bundle would be difficult to use with the proposed picture and srcset standards
    Image breakpoints calculated this way depend on knowing the size of the element in the page. Both picture and srcset make the switching of image sources contingent on the size of the viewport instead of the element. This means that you’d have to find a way to translate your image breakpoints to viewport sizes which would undermine a lot of the utility of storing breakpoints with the image.

Crazy? Or crazy like a fox?

Phew, you made it to the end. So what do you think? Is there merit in using performance budgets for flexible images to determine responsive image breakpoints?


Thanks to John for creating the sample script and for being my partner in crime on this crazy idea and to Lyza for being an amazing photographer and publishing her photos under creative commons.

22 Comments on “Sensible jumps in responsive image file sizes”

  1. Yoav Weiss says:

    Hey Jason,
    Great post and we definitely need to give this subject more thought.

    One point regarding the picture/srcset’s reference to the viewport size rather than the image’s size. The reason these standards do that is because we would like to avoid waiting for layout until the image loading starts. Waiting for layout results in some delay to image download with HTTP/1.1, but can result in significant delays when using SPDY & (future) HTTP/2.0. We don’t want that delay burdening our performance budget if we can avoid it.

    In my opinion, layout-based responsive images is a real pain to people building 3rd party responsive front-end components, but for most people that build responsive Web sites, the gap between layout and viewport can probably be resolved in a build step, before deploying to production.

    • Jason Grigsby says:

      @Yoav I’m aware of why the standards use viewport (http://blog.cloudfour.com/the-real-conflict-behind-picture-and-srcset/), but there’s no denying that using the viewport (or worse device width) to determine what image source to use is several steps removed from what actual matters when picking the appropriate image source—namely, the size fo the image on the page.

      But more on this in another post. :-)

      • The issue of viewport size vs element size is critical to this discussion. We should try to quantify the potential savings. Seems like it wouldn’t be that hard: build a bookmarklet that iterates over the IMGs in a page and compares their size vs viewport. I don’t think the savings are going to be significant enough to offset the savings of requesting them earlier based on viewport size.

        • Jason Grigsby says:

          @Steve, it seems like the idea of using a performance budget to determine image breakpoints would be regardless of element vs. viewport size.

          If using a performance budget to set breakpoints makes sense but element size isn’t available to us, then we’ll just build some sort of complex mapping between viewport size and element size on a page by page basis. Oh, and we’ll have to hope the user hasn’t zoomed the page since that isn’t accounted for with viewport size.

          It’s ugly, painful and a bit stupid, but it could still be done and might still make sense given the alternative which is to pick breakpoints entirely arbitrarily.

  2. Yoav Weiss says:

    I wrote a short script output how many bytes can be saved for a given site by resizing its content images to their layout size. This can help to automate a build process, and see if the performance budget given to responsive images is met.
    https://github.com/yoavweiss/Sizer-Soze

    • Jason Grigsby says:

      This is cool. Can’t wait to try it out.

      Do you have any test runs that we can look at the output of? Yes, I’m at the coast and it is beautiful so I’m being lazy. Feel free to ignore. :-)

  3. Eric Portis says:

    The idea of basing breakpoints on the image content itself, and bandwidth (which is what we’re trying to save) rather than viewports and pixels is *great.* I was a little confused until I got to the extreme Microsoft logo example. But then of *course* there should only be one of those assets. Then it all fell into place.

    I think what was confusing me was the conceptual frame of a “budget,” and the idea of “spending” something to “get” something you didn’t have before. There are all sorts of starting points for how people have been dealing with images on flexible layouts… If the baseline is sending a low-res image, then every higher res version is an additional “cost” in kB & time. But for a site like the Big Picture, sending smaller versions of what had before been the large, “default” size is saving bytes, not spending them.

    I’m still a little fuzzy on the reasoning behind choosing a particular kB gap, and thinking about that in terms of a budget. If I send a 200kB image to a user that *could* have been 175kB if I’d sized it perfectly for the layout/viewport, I’ve still “spent” 200kB, not 25kB?

    I’ve certainly “wasted” that 25kB though…

    I guess I find thinking about concrete, individual pageloads and which bytes were useful or not to a particular user very useful, but there’s a muddying here between differential bandwidth “costs” and overall bandwidth “costs” that threw me for a loop. Maybe just my own connotations re: what a “budget” is.

    @Yoavv, re: the viewport sizes & waiting for layout, I just wrote a boatload about this here:

    http://ericportis.com/2013/scalables/

    (I saw your blog post re: a possible new image format structure / request process and thought there were some parallels?)

    “Just add a build step” is not simple for authors. I guess, re: the article, neither is setting up an automated system which finds breakpoints and generates a range of source files.

    The dream, the goal, has to be images feeling almost as native to the web as text does. I want it to be as easy to “drop an image into a website” as it is to drop it into an indesign file… If at all possible, I want the interface where I do that dropping to be HTML, and not some higher-level abstraction like a CMS on top of it. Backends and build steps might be the best we can achieve now and in the short term, but can you imagine a web where instead of browsers soft breaking & automatically reflowing text by default authors had to come up with automated build steps that manually inserted line-breaks at various line lengths, and mark up their text with some mechanism for deciding which text block to send to a given user based on the user’s viewport size, the text-size, the font-family, etc? Madness! That is exactly what we’re building for images, and it makes me sad.

    • Jason Grigsby says:

      @eric thanks for the comment. More later as the sun is shining here. Just wanted to drop a quick note that the URL you shared is broken. I found the article at;
      http://ericportis.com/posts/2013/scalables/

    • Yoav Weiss says:

      Hi Eric,

      Great post! I understand that the ideal is that Web developers would define their images based on their display dimensions.
      Unfortunately, if that definition will results in significant page load slow-down, I don’t think it is worth the price.
      “User pain > Web dev pain > browser dev pain” is the guiding design principle.
      In the short term, this may be our only choice, so Web developers will have to adopt a build process if they want to optimize their Web sites (Much like JS/CSS bundling & minification). Note that this can be done for static resources during build time (which can be a simple script running on your local machine before you deploy your code). For dynamic resources it’d require backend changes, which may be difficult with legacy CMSs that don’t support plugins. I understand the pain, but the alternative is inflicting it on the users.

      The long term option that may make everyone happy is a responsive image format (with a corresponding fetching mechanism) that will fetch the image’s first few kilobytes before layout, and continue to fetch the rest (if necessary) once it has layout. It may still introduce some delay, but it’d be less significant, and browsers may be able to minimize it with heuristics. Some sort of a manifest that outlines image resources and their resolution=>bytesize mapping may also help.

      That may be the ideal solution for the resolution switching use-case, but as ideals often are, it’s not simple to achieve. Image format adoption on the Web is extremely slow, partly because there is no client-side image type negotiation mechanism. So as much as I like this particular unicorn, I’ll settle for a pony, at least for now :)

    • Jason Grigsby says:

      @Eric wrote:

      If the baseline is sending a low-res image, then every higher res version is an additional “cost” in kB & time. But for a site like the Big Picture, sending smaller versions of what had before been the large, “default” size is saving bytes, not spending them.

      I agree. What I’m trying to quantify here is what cost is there for delivering a larger image and letting the browser resize it which is the definition of flexible images in a responsive design. That’s why I narrowed the cost to the difference between that larger image and the perfectly-sized one.

      As you point out, the difference could be small and the page weight could still be large. But I had to start somewhere when trying to quantify flexible images.

      Backends and build steps might be the best we can achieve now and in the short term, but can you imagine a web where instead of browsers soft breaking & automatically reflowing text by default authors had to come up with automated build steps that manually inserted line-breaks at various line lengths, and mark up their text with some mechanism for deciding which text block to send to a given user based on the user’s viewport size, the text-size, the font-family, etc? Madness! That is exactly what we’re building for images, and it makes me sad.

      I feel ya.

      • Yoav Weiss says:

        A short note: Seems like even waiting for layout doesn’t guaranty that the image display dimensions will be known. images without explicit or relative dimensions can affect the display dimensions of other images. Here’s an example that shows that.

        “waiting for layout” then becomes “waiting for layout and the download of all the images that may affect the image’s dimensions”. In other words, it’s complicated…

      • Eric Portis says:

        What I’m trying to quantify here is what cost is there for delivering a larger image and letting the browser resize it which is the definition of flexible images in a responsive design.

        Responsive image sources don’t *have* to be larger than necessary. The BBC Sport hero image upsizes & gets fuzzy on wide viewports. As you size your window up and that image gets fuzzier, should you frame it as incurring more and more “savings” towards some total feature budget? Seems weird.

        Should serving a 1.5x image to a 2x screen be considered a cost or a savings?

        I guess I’m still nitpicking the budget metaphor & the idea that you can or should quantify precisely what you’ve saved or lost vs. some ideal baseline … but I agree entirely that basing breakpoints on *performance* rather than *pixel dimensions* is A++ 100% certainly the way to go. That’s the whole point of providing multiple sources, vs. a single, giant, source, right? Performance!

        And despite the evidently-also-super-important factors that Vjeux brings up below (DOM weight, browser-resizing-induced sluggishness), bytes seem like they make the most sense as a single number to use to quantify the slippery fish of “performance.” They’re certainly better than pixels: “breakpoints every 20kB” makes a lot more sense than “breakpoints every 200px” (or breakpoints every 0.5x pixels, which is what I weakly advised in the “scalables” post).

        Picking breakpoints still seems more like art than science, but these ideas are a step in the right direction. Thanks!

  4. Eric Portis says:

    Oops! Well that certainly is embarrassing (: Enjoy the day!

  5. Evgenii says:

    Thank you for the research. I hope in near future we will get this idea implemented in web frameworks, so all we need to do is to drop a big original image and set a global “size budget” in settings. Then the server will generate all necessary image sizes and send the list of available choices to client. The client will then choose the optimal image for downloading. Until this technology is implemented I am choosing simpler approaches. Right now I am thinking of using 2 image sizes: 480×320 50% quality jpeg for “iPhones” and 800×600 50% jpeg for wide screens with picturefill JavaScript. It is far from optimal. But in a small company simplicity and maintainability of the system are key concerns. Thanks again, great article.

  6. Chris says:

    Is there any breakdown in how smaller screens perform in displaying non-optimized images? I know there is a performance cost to not only loading larger images, but resizing them for display.

  7. Balaji says:

    Thank you for this wonderful post… This is very interesting and I am loving it.

  8. Vjeux says:

    I’ve been working on this problem for few months now at Facebook and here are some more things I hit:

    1) Browser image resizing is a costly operation for the browser. Sending properly sized images greatly improved fps while scrolling in an image grid.

    2) Image size vary a lot for the same JPEG quality setting. The range can be up to 10x between image that compresses well and worse case scenarios.

    3) This article assumes that you know the file size of the images in advance. When you’re dealing with a lot of images, you want to resize on the fly and cache them, so this information is tricky to obtain.

    4) Sending urls for many different images is costly. This represents a very important part of our payload.

    Unfortunately, I don’t have a good solution either :(

  9. I wrote an article about the pixel density explosion (inspired by the BDConf talk) that’s relevant to the responsive images debate.
    Regarding the viewport vs context layout discussion, I think we need to stop thinking in black and white. There’s no good solution, so it’s time to compromise. I don’t think it’s essential that the image source be initially selected based off the layout, but I am certain that placing breakpoints and media queries into markup is a horrible, future-hating thing we must avoid. That kind of complexity tax is unacceptable. Also, I think it would be “OK” to offer both viewport and layout-based selection to authors, accompanied by performance warnings. There will be situations where it’s better for images to load later but at the correct size (not all images are photos, and some images don’t resize well at all).

  10. Chris Adams says:

    Just to back up @Vjeux’s 4th point, I’d argue that the metric needs to include latency as a key factor: even with a 3G connection an extra few hundred milliseconds waiting for a CDN miss, server-generated thumbnail, etc. can be the same as the 34KB savings mentioned above. This is particularly important for images which don’t see lots of traffic or where the traffic is geographically dispersed and the origin servers are not – a US East/West CDN miss starts at 200ms and gets worse if you’re thinking globally.

    The question of browser resizing performance is an interesting one: I’ve found that it’s generally quite acceptable even on relatively old phone / tablet hardware if you’re using a browser-friendly layout – in my case, that’s a grid of div tags which have fixed size to avoid forcing the browser to decode them just to flow the page and displaying the image as the CSS background-image which allows the browser to unload the image when it’s not visible, keeping total memory usage stable.

  11. bisho says:

    For jpg you should try to use sizes multiple of the block size used for compression, typically 8×8 for color images. This avoids padding and computing on clients, resizes better, and gives better compression per pixel with actual data.

  12. Rob Noble says:

    Jason

    My Lead Design Engineer wrote this blog post this week. I thought it may be of interest to you. Would love to hear your comments.

    http://blog.greatfridays.com/engineering/images-in-responsive-web-development/

    Rob

  13. I love the concepts behind most of these, but would it make sense to have the logo asset as an SVG. I certainly don’t know what that would amount to in file size, but I know I’m moving towards that with most logos I put on the internet (if they’re simple enough) because then scale doesn’t matter.