Cloud Four Blog

Technical notes, War stories and anecdotes

Beyond “Hello, World” with Gulp

(This post is for: web developers who have used, or thought about using, a build tool before, are JS-comfy and maybe have noodled around in node.js but wouldn’t, say, feel up to submitting a talk to a node conference.)

It would be hackneyed to call the last twelve months the “Year of Build and Workflow” for web developers, but there’s an element of reality to the contrivance. In 2013, there was grunt (heck, there still is; it didn’t die or anything) and a whole lot of blog posts about streamlining workflow around SASS and templating and build package management, etc.: topics that were—poof!—suddenly an imperative for general people working on the web. I wrote a thing about workflow for A List Apart earlier this year, a few weary weeks after the whole “Oh, you learned grunt? Sucker! Now you need to learn gulp, because whatever!” revolution happened.

Summarized as this: a whole lot of people (increasingly non-nerdcore) learned to use grunt to wrangle previously-tedious workflow tasks for building web sites. This represented a big jump in things we all needed to know. And then, kind of literally overnight, there was gulp. For me, this felt like a burden as well as an opportunity.

I wasn’t exactly a holdout (as you can see), but I was initially resistant to throw the grunt baby out with the 2013 bathwater simply because a new tool existed. And, like maybe some of you (?), I found the initial sell a bit baffling: “streaming build system?” I didn’t know I wanted that? Awesome syntax? I didn’t really have a gripe with grunt on that front. “Speed?” Maybe? I didn’t exactly have to go brew a cup of coffee while my grunt tasks ran.

Grieving and Surmounting

But smart people I know kept extolling its highlights and I decided to give it a whirl. The familiar stages of dev-new-tool excitement-grief happened:

  1. Find and install new tool that people crow about.
  2. Get satisfying example configuration/program running using new tool.
  3. Extend satisfying example configuration/program in a tiny way and find success in making it do something useless, but cool. HELLS YEAH.
  4. Extend awesome slightly-extended configuration/program in another way to make it do something you actually need it to do. Spend 3 hours/days progressively assuring yourself you are a moron—this should really be so easy! Two possible outcomes: rage quit or eventual, sweat-soaked breakthrough.
  5. Assuming you’ve made it this far, progress with a slight wariness, possibly even progressively assuring yourself that the tool devs or Stack Overflow or Google or a particular plugin or, hell, the entire language you’re working in is the real moron here. Also revel in the fact that extant examples of what you’ve now managed to do exist on the Internet at all—you’ve birthed something truly novel. This is often a necessary step to repair the psyche.
  6. If you are kind, you then help other people avoid step 4 (or, at least, put a reassuring hand on their shoulder).

Gulp: The Good

Here’s some of why you might want to use gulp instead of another alternative:

  • Pretty syntax and intuitive structure: There is nothing wrong with grunt’s syntax, but gulp’s is kinda pretty. Using Streams means you’ll be able to do a lot of chaining, which always feels pleasing to the code-scanning eye. I was never a huge fan of the double-step grunt convention of grunt.loadNpmTasks() and grunt.registerTask(). In gulp you can require() and define gulp.task()—that feels more natural to me. I find I don’t have to look up core API stuff for gulp; with grunt, I never memorized it.
  • Speed: There’s no doubt. gulp feels noticeably fast. That is not exactly a scientific observation, but I stand by it.
  • Streams: If you know node already, gulp’s Streams architecture will make you feel right at home.
  • Gulp has a nice clarity that it is about working with streams of files. As such, there is more symmetry between any two given gulp tasks than most given grunt tasks. gulp.src globs and gulp.dest paths become second nature. Stream in, though, out.
  • Works with lots of things, easily: Like grunt, gulp has plugins (many of them). But the first edict in the guidelines for creating a gulp plugin is: “Your plugin should not do something that can be done easily with an existing node module,” meaning that gulp encourages the use of non-plugin-i-fied node packages wherever possible (versus “[w]rapping every possible thing just for the sake of wrapping it”). The plugin landscape feels correspondingly more serene.

Avoiding Step 4: Tips for Getting along with Gulp

  • Make sure your gulp tasks finish correctly. If not, it will wreak havoc with dependencies and other tasks. Be sure to read the section of the docs about valid things to return from gulp tasks and how to handle async tasks. Trust me on this one.
  • In fact, read the entire API documentation. If this sounds like a tall order, take note that it is shorter than this blog post. The core API for gulp is outstanding in its simplicity.
  • Get an understanding of Streams, a core node construct, because this is the way that gulp works. Take a few minutes/a while and read a primer on Streams. It’s a bit to get your head around, but if you’re doing any heavy lifting with gulp (or other node, for that matter), you’ll be glad you learned.
  • Watch out for plugin quality. The plugin landscape may be straightforward, but it’s young, and there are some stinkers. I lost some serious time to a couple.
  • Dig hard for real-world (read: complex) examples. This situation has gotten better in the past few months. Dan Tello’s excellent post on Gulp + Browserify: The Everything Post is a must-read-right-now. When I first encountered his repo full of organized gulp tasks, it changed my everything. I ended up extending my fork to handle dependencies effectively, but his gulp-starter repo has since been updated to do that, also, so, win! There is more documentation and examples out there now than there was a few months ago. But documentation about gulp things can sometimes presuppose that you know more about various concepts than—gasp—perhaps you do. Don’t fret. That jargon sounds weird to other people, too. Read slowly and give yourself some time!
  • Watch out for plugins getting between you and what you’re doing. Like any build tool with plugins, gulp puts a layer between you and (in gulp and grunt’s case) node. Sometimes it’s hard to debug whether you’re doing it wrong, or whether the plugin between you and the wrapped node package is doing something wrong (or at least making things real complicated). If you don’t understand the syntax of a plugin or what it’s doing, consider that a red flag and slow down until you do. Or: see if you can accomplish what you’re doing with a node module instead.
  • Complexity explodes as you add more tools to the mix. We are using browserify for our application code, but it got outrageously onerous for managing vendor code, much of which was not modular in any sensical way (this was serious step-four territory). We ended up throttling down and using something a bit simpler. Don’t be afraid to over-simplify if it gets the job done. I find you can usually kick up complexity later, when needed, but that the opposite—dialing it down—is not so easy.

We do some really cool things at present with gulp. We use it to parse YAML front-matter out of templates (which get piped through underscore and have partials inserted and come out as static HTML files) to create an auto-index of project assets, adding automated screenshots (thanks, phantom.js!) at different widths to show responsive layout. This of course along with SASS and auto-prefixer and browserify and concatenation and and and…whew. Because each task in the works is totally modular, this ends up not feeling rickety and spaghetti, believe it or not—and that alone is worth the price of admission.

Go out and build! Let me know how it goes!

My thanks to Cloud Fourian Erik Jung for leading us out of step 4 hell vis-a-vis the whole vendor JS thing I mentioned above.


Slow and ugly

Many years ago at a previous job, I was responsible for pushing our organization to adopt a new ticket tracking system. We decided to use an open source project called Request Tracker (RT).

This was before the days of Zen Desk and similar services. RT matched our requirements best. I worked with our sysadmins and engineers to get the software running and rolled it out to the organization.

It was a failure. People were openly swearing at RT, and I’m sure they were privately cursing me for forcing them to use it.

While I sympathized with the complaints, I felt hamstrung because RT was deployed on an underpowered server and we had not been allowed to modify the look and feel. Back then RT was a bit of a dog both from a performance and aesthetics perspective.

As I worked with engineering to secure better hardware and get someone to assist with the design, the number of grievances continued to pile up.

One conversation stands out to me. A member of the management team told me that the fact that customers couldn’t reset their password in RT was unacceptable and as long as that was true, we couldn’t use RT.

I listened to the objections and kept a list of the issues that were raised, but I kept my focus on making RT faster and more attractive.

A few months later we relaunched RT. It was met with skepticism, but to the credit of my colleagues, they gave me and it a second chance.

It wasn’t too long after the relaunch that previous skeptics were swearing by the system. One of the team members became the master of RT and made a bunch of process improvements that I would have never considered.

What happened to that list of complaints that I gathered? We didn’t address a single one of them. Passwords worked exactly as they did before.

We simply made RT fast and attractive.

I have a simple rule when it comes to user experience: when something is slow and ugly, nothing else matters.

Until you address those two issues, you’re not going to be tell whether other complaints are real or red herrings.

Fantastic intro to new srcset and sizes

I normally don’t write a post simply to link to another article, but if you’ve enjoyed the things I’ve written about responsive images, you really should read what Eric Portis wrote about Srcset and Sizes.

Eric describes the logic behind srcset and sizes, how they are part of the new picture proposed standard, and how to use them.

Easy. Peas.

Responsive Web Design is Solid Gold

A few years ago I wrote an article entitled CSS Media Query for Mobile is Fool’s Gold. It garnered a lot of attention at the time, and I still see people reference it.

I’ve long wanted to write an update to that article, but never knew quite what to say until now. And because one inflammatory title deserves another, read on for why Responsive Web Design is Solid Gold.

Our default approach is now responsive web design

Back in 2010 when I wrote the Fool’s Gold article, our default approach for mobile was to use device detection. If a site was simple and the budget small, we might use responsive web design.

Some time ago our default approach switched. We now treat responsive web design as the default approach and look out for reasons why it won’t work instead of the other way around.

Why did our default approach change? Two reasons: performance and device diversity.

Update on performance

Much of my Fool’s Gold article focuses on performance problems with flexible images and CSS background images. One of the reasons why I’ve had trouble figuring out how to write a follow up is because the performance issues that I raised remain true today.

Let’s do a quick round up of the issues and updates on each:

Perhaps most importantly, most responsive web sites are still far too large on small screens.

So if most of these remain true and most responsive designs are bloated, why has my perspective changed?

Because it is possible to build responsive design responsibly and create fast experiences. The keys to doing so are:

  1. Build mobile first responsive designs
  2. Keep CSS background images in scoped media queries
  3. Conditionally load JavaScript and even HTML fragments based on screen size and capabilities
  4. Implement a responsive images solution
  5. Handle retina images very carefully and err on the side of performance

If you do these things and do the normal things you should do to make any web page fast, you will have a fast responsive design. It may not be easy, but it is possible.

But what about the fact that most responsive designs are bloated? As Tim Kadlec says, “blame the implementation, not the technique“.

I don’t blame device detection for the many sites that route people to the mobile home page instead of the article they’re looking for. And I don’t blame responsive web design for the fact that most implementations are bloated.

Can a responsive design ever be as fast as a page tailored for a specific device?

Probably not. But the web is a balancing act between many competing interests. A site that was completely tailored for search engine optimization would be unreadable by humans.

By the same token, performance is an extremely important factor, but is still just one of many factors that make a site successful. You can build responsive designs that are fast enough that the benefits of responsive design outweigh the potential performance improvements you might get from separate sites.

Especially when you consider device diversity.

Device diversity makes responsive design inevitable

Let’s assume for a moment that responsive design doesn’t work for your project. So you decide that you’re going to need to build mobile, tablet and desktop experiences. And let’s set aside for the moment the inevitability of new form factors like televisions, watches, etc.

Even in this scenario with different experiences for each of the three major form factors, you’re still going to end up needing responsive design—or at least responsive characteristics.

Yesterday, Samsung announced that it was launching a 6.3 inch phone. The range of screen sizes on mobile phones alone will likely force you to build something that will adjust from the small screen sizes of Blackberry Bolds and feature phones to the mammoth screens of some Samsung devices.

Tablets present a similar challenge ranging from 7 to 13 inch displays (and sometimes much bigger). And we’re all familiar with the large difference between ultrabook screens and cinema displays.

I wrote in much more detail about the impact of device diversity earlier this year and my conclusion that:

Any attempt to draw a line around a particular device class has as much permanence as a literal line in the sand. Pause for a moment and the line blurs. Look away and it will be gone.

So regardless of what techniques you use in addition to it, responsive web design will likely be part of your solution.

What about mobile context?

One of the other points I made in the Fool’s Gold article was related to mobile context. It is something I’ve struggled with for years.

I’m now firmly on the side that there is no mobile context. We have abundant data that shows that people use their mobile devices indoors and for a wide variety of things.

Luke captured it well when he wrote:

But if there’s one thing I’ve learned in observing people on their mobile devices, it’s that they’ll do anything on mobile if they have the need. Write long emails? Check. Manage complex sets of information? Check. And the list goes on. If people want to do it, they’ll do it on mobile -especially when it’s their only or most convenient option.

Amen.

So Jeremy Keith and Stephen Hay were right. There is no mobile web. Mea culpa.

Still No Silver Bullets

I concluded my Fool’s Gold article by stating the obvious: there are no silver bullets when it comes to adapting our existing apps and sites for mobile screens. At that point in time, people were touting media queries as a quick fix for mobile.

Since then, our profession has learned a lot more about the complexity of designing and building experiences for multiple devices. Now it is generally understood that supporting all the devices that may access our content and services isn’t easy, but that tackling problems that range from responsive images to legacy content management systems are the heavy-lifting that we must do in order to be future friendly.

And despite those challenges, I’m excited about where the web is heading and what we can do with progressive enhancement and responsive web design.

Responsive web design: it’s solid gold, baby. :-)

Sensible jumps in responsive image file sizes

Last year, I wrote about the challenges of picking responsive images breakpoints and how I found it a nearly unsolvable problem. It has vexed me since.

But I have a new idea on how we might be able to define responsive image breakpoints that is based on a performance budget.

Before I begin, I should note that a lot of this is a thought experiment. I don’t yet know how practical this approach would be.

Brief refresher on responsive image breakpoints

Without going into all of the details about responsive image breakpoints, the short version is that most people are picking the breakpoints for responsive images based on one of two criteria:

  1. Based on what Scott Jehl referred to as “sensible jumps in file size to match screen dimension and/or density” OR
  2. Simply matching the image breakpoints to the major breakpoints being used for the design.

While the first method is more efficient and will probably result in better image sizes, my suspicion is that defining “sensible jumps in file size” is so nebulous that most web developers are going to choose to do the second, easier option.

That is unless we can find a formula to calculate what constitutes a sensible jump in file size and that’s what got me thinking about performance budgets.

What is a performance budget?

I’m not sure how long the idea of a performance budget has been around, but I first became cognizant of the idea when Steve Souders talked about creating a culture of performance on the Breaking Development podcast.

Tim Kadlec expanded on the idea in a recent blog post. He cites the BBC which determined that “each page to be usable within 10 seconds on a GPRS connection and then based their goals for page weight and request count on that.

So that’s the basic idea. Establish a performance budget and stick to it. If you add a new feature to the page and you go over budget, then you have a three options according to Steve (and transcribed by Tim):

  1. Optimize an existing feature or asset on the page.
  2. Remove an existing feature or asset from the page.
  3. Don’t add the new feature or asset.

What is the performance budget for flexible images?

Let’s apply this idea of a performance budget to responsive design. In particular, let’s treat the idea of flexible images as a feature. Because flexible images are a feature, we need a budget for that feature.

And as long as we’re making up the rules, let’s establish a few more hypotheticals:

  • The page we’re working with has 10 images on it of varying formats and visual content.
  • We haven’t reached our performance budget yet so we don’t have to remove other features, but we still need to make sure that flexible images do not add too much to the page weight.
  • We’ve concluded that flexible images can add up to 200k to the page above what the size of the page would be if we provided fixed width images. We picked 200k because it is ~1 second at HSDPA (recent mobile) speeds. And well, 200k is a nice even number for this thought experiment.
  • Because this page has 10 images on it, each image has a 20k budget for flexible images.

One thing to keep in mind, 200k isn’t the cap for the file size of all ten images combined. Instead, it is the price we’re willing to pay for using flexible images instead of images that are perfectly sized for the size they are used in the page.

For example, say you had a responsive web page with the following image on it:

fish-and-chips

That image is 500×333 pixels and 58K in file size.

Now imagine a visitor views that web page and based on the size of their viewport, the image is displayed at 300×200 pixels, but the source image is still the same. The cost of using flexible images is the difference in file size between what the image would be saved and optimized at 300×200 versus the file size of the image downloaded at 500×333.

In this case, I’ve taken that example image and resized it to 300×200 and saved it with the same compression level as the 500×333 image to see what the file size cost is of using that flexible image.

Source Width Height File Size
Flexible image 500 333 58k
Image matching size used in page 300 200 24K
Total extra download - - 34K

In this example, the visitor is downloading an extra 34k of image data because they are downloading a flexible image instead of downloading one that had been resized to the exact size being used in the page.

Translating the budget into breakpoints

Let’s go back to the page we want to optimize—the one with ten images on it and a total performance budget of 200k for flexible images. How do we translate that into image breakpoints?

Thinking back to the example above, the price for using flexible images is the difference between the size of the file that is downloaded and the size the file would have been if perfectly sized for its use in the page.

Our budget says that we can only download up to an extra 20K per image. Therefore, we need to make sure that we have a new image breakpoint every time the size of the image increases 20K.

We now have a methodology for picking sensible jumps in image file size that is tied to user experience instead of picking them arbitrarily.

Finding the breakpoints

How would this translate into a heuristic that could be used to find the breakpoints? You would need the following:

  • What is the minimum size this image will be used at? (In our example, let’s say 320×213)
  • What is the file size between breakpoints? In other words, what is your per image budget? (20k for our example)
  • A high quality source file to use for resizing.
  • Optionally, the largest size the image will be used at. (990×660 for this example).

Once you have this information, the basic logic looks like this:

  1. Take the source image and resize it to the smallest size the image will be used at.
  2. Keep the file size of that image handy.
  3. Start a series of tests that create new image files from the source that are gradually getting bigger.
  4. Check each image created. If the difference between the file size of the new image and the image file size you stored is less than your budget, discard the new image.
  5. When you find an image that hits your budget, save that image and replace the previous file size that you stored with the new file size.
  6. Repeat steps 2 through 5 until you reach either the maximum resolution of the source image or the largest size the image will be used at.

I’m tickled to say that my co-founder John Keith got excited by this idea and built a rough prototype of how this might work.

Sample Page

Using the script that John built, I created a demo page containing ten images. The source images were 990 by 660 pixels and all but one of them were saved as JPEGs at 50% quality. The one exception is a PNG8 image with an optimized color palette.

I tried to pick a variety of images so we can see how each image might have different breakpoints using our budget. Let’s take a look at three sample images.

Time Square — 8 Image Breakpoints


This image has a lot of visual diversity. The variations in colors and textures means that JPEG’s lossy compression cannot do as much without damaging the image quality.

Because of this, there are eight image breakpoints—set at 20k intervals—between the smallest size of the image (320×213) and the largest size of the image (990×660).

Breakpoint # Width Height File Size
1 320 213 25K
2 453 302 44K
3 579 386 65K
4 687 458 85K
5 786 524 104K
6 885 590 124K
7 975 650 142K
8 990 660 151K

Morning in Kettering — 3 Image Breakpoints


Unlike the Times Square image, this image has a lot of areas with very similar colors and little variation. Because of this, JPEG can compress the image better.

On an image that can be compressed better, our 20K budget goes farther. For this image, we only need three image breakpoints to cover the full range of sizes that the image will be used at.

Breakpoint # Width Height File Size
1 320 213 9.0K
2 731 487 29K
3 990 660 40K

Microsoft Logo — 1 Image Breakpoint


Microsoft Logo

This is a simple PNG8 file. At its largest size (990×660), it is only 13K. Because of this, it fits into our 20K budget without any modifications.

On a recent consulting project with a company that has over 800,000 images on its site, we identified a class of images—some icons, little badges, etc.—where the size of the image on desktop retina was not much different than the size used on mobile either because the image resolution doesn’t vary much or because the image compresses well. For those images, we decided to deliver the same size image to all screen sizes.

Breakpoint # Width Height File Size
1 990 660 13K

Every image is different in the way it compresses

Take a look at the other images on the sample page we created. See how the number of breakpoints vary even through all the images start with the same resolution end points.

This diversity exists despite the fact that with the exception of the Microsoft logo, all of the images start at the same size with the same compression settings. On a real site, we would see even more diversity with varying levels of JPEG quality, PNG8 with gradients going horizontally instead of vertically, and PNG32 images in the mix.

But what intrigues me about this approach to setting breakpoints is that it we wouldn’t be setting one-size-fits-all image breakpoints. Instead, we would make decisions about where the breakpoints should exist based on the our goals for user experience—the performance budget—and the unique characteristics of the image and how will it can be compressed.

What conclusions can we draw from this thought experiment?

The point of this thought experiment wasn’t to provide a complete methodology to set responsive image breakpoints. I started by simply asking the question about whether we might be able to use performance budgets to come up with a way to calculate what are sensible jumps in image sizes.

But the outcome of this exercise has caused me to draw some interesting conclusions as well as sparking more questions about responsive images:

  • Images do contain clues that can tell us where the breakpoints should be.
    Last year I wrote that “the problem is there is nothing intrinsic to the image that would guide you in deciding where you should switch from one size of the image to another.” But this experiment shows that images do have intrinsic information—how well the image compresses, what type of compression is being used, the range in size between the smallest and largest use of an image—that can be used to decide when you should switch from one source file to another.
  • We can set a performance budget for flexible images.
    There’s no reason why we can’t treat the use of flexible images like any other feature that we add to a page and define a performance budget for its use. In fact, setting a performance budget for flexible images could be the key to making informed decisions about where image breakpoints should be set.
  • Automated image resizing and compression is a must.
    I have already written about how automated image services will likely be a must for sites in the future. If a company wanted to use a technique like this to set their breakpoints, they will need an automated way to do it.
  • Can we set a performance budget for flexible images across an entire site?
    For our sample page, setting the performance budget to 200K for the whole page worked well. But in the real world, we often don’t know how many images are going to be on a given page. Similarly, we may not know what pages a given image is going to be added to. It seems like it would be useful to be able to say that for any given flexible image on the site, we’ve established a 20K budget. It would be less precise than a per page limit, but it may be the only practical way to translate this thought experiment into a production environment.
  • An image and its breakpoints could be stored as a bundle.
    The outcome of this approach to image breakpoints is that the breakpoints could be specific to the image no matter what context the image is used in. You could store the calculated breakpoints with the image and whenever the image is displayed on a page, no matter what size the image is used at within the page, the same breakpoints could be used.
  • An image and breakpoint bundle would be difficult to use with the proposed picture and srcset standards
    Image breakpoints calculated this way depend on knowing the size of the element in the page. Both picture and srcset make the switching of image sources contingent on the size of the viewport instead of the element. This means that you’d have to find a way to translate your image breakpoints to viewport sizes which would undermine a lot of the utility of storing breakpoints with the image.

Crazy? Or crazy like a fox?

Phew, you made it to the end. So what do you think? Is there merit in using performance budgets for flexible images to determine responsive image breakpoints?


Thanks to John for creating the sample script and for being my partner in crime on this crazy idea and to Lyza for being an amazing photographer and publishing her photos under creative commons.