Cloud Four Blog

Technical notes, War stories and anecdotes

On the Device Context Continuum

The fact the Apple may soon release an App Store for TVs has me revisiting a couple of questions that have troubled me these last few years: Where does the common device context continuum start and end? And more importantly, how do we know?

But before I look at those questions in detail, let’s talk about device context.

The Device Context Continuum

We now design for a continuum of devices. Responsive web design provides us with the techniques we need to design for varying screen sizes.

But responsive web design techniques wouldn’t be effective if there wasn’t a common context—or perhaps more accurately, a lack of context—between devices.

Put a different way, if people did demonstrably different things on mobile phones than they did on desktop computers, then responsive web design wouldn’t be a good solution.

phone-tablet-desktop-continuum

We design for different screen sizes confident in our knowledge that people will do similar things whether they are on phone, tablet or desktop devices. This is our common device context and the continuum that it applies to.

But it hasn’t always been this way.

The Mobile Context Debate

In the early days of responsive web design, people often debated whether or not mobile context was a thing that should be considered in our designs.

At the time, I wrote about my conflicted thoughts on mobile context. I advocated for keeping context in mind. But by 2013, I had concluded mobile context didn’t exist.

Now we have a lot of experience to back up this perspective. Chris Balt, a Senior Web Product Manager at Microsoft, told Karen McGrane and Ethan Marcotte on the Responsive Web Design podcast:

Our data shows us quite plainly and clearly that the behavior of those on our mobile devices and the small screens is really not all that different than the behavior of those on the desktop. And the things they are seeking to do and the tasks they are seeking to accomplish are really quite the same.

Karen and Ethan have been doing a weekly podcast for a year. In that time, regardless of the company or industry being discussed, people say that they see no difference in what people want to do based on whether they are using a mobile, tablet or desktop.

I still think Luke Wroblewski nailed it when he wrote:

But if there’s one thing I’ve learned in observing people on their mobile devices, it’s that they’ll do anything on mobile if they have the need. Write long emails? Check. Manage complex sets of information? Check. And the list goes on. If people want to do it, they’ll do it on mobile -especially when it’s their only or most convenient option.

What about new devices? TVs? Watches?

It seems that not a day goes by without a new device form factor being introduced. Watches. TVs. Virtual reality goggles. Augment reality glasses.

Where do these new devices fit in on this device context continuum? Do they share the same context?

continuum-question

The consensus at the moment seems to be that they are not part of the same continuum as phones, tablets and computers. When you read the guidelines for designing for watches or TVs, designers are advised to take context into consideration.

At Responsive Day Out, Rosie Campbell, who works in Research and Development for the BBC, gave a compelling presentation entitled Designing for displays that don’t yet exist. She shared research on what it would take to build a compelling smart wallpaper experience in the future when such technology might become commonplace.

In the talk, Rosie made two comments that I’ve been thinking about ever since. She addressed what we need to do as screens get weirder:

It’s not just about making content look beautiful on those different screens. We also need to think about what is appropriate for each device because you’re probably not going to want the same kind of information on your smart watch as you want on your smart wallpaper.

This makes intuitive sense to me. For whatever reason, my Apple Watch feels very different than my phone or my computer.

But Rosie also used browsers on Smart TVs to illustrate a point that just because a technology makes something possible, doesn’t mean that we should design experiences around it:

Suddenly, we all got Smart TVs. And it was great. We got Internet on our TVs. But actually browsing the web on the TV was a really clunky experience. It was not very pleasant. And no one really wanted to do it especially when you’ve got a mobile or tablet next to you that makes it easier.

Again, what Rosie states here is the popular consensus that people won’t browse the web on their TVs. Steve Jobs famously said that:

[People] don’t want a computer on their TV. They have computers. They go to their wide-screen TVs for entertainment. Not to have another computer. This is a hard one for people in the computer industry to understand, but it’s really easy for consumers to understand. They get it.

I’ve spent the last three years researching the web on TVs wondering about exactly this question. And it isn’t clear cut to me whether or not people will browse the web on TV screens in the future.

The consensus on mobile context has changed

The popular consensus used to be that no one wanted to browse the web on their phones. If you dared to build something targeting phones, you were advised to keep the mobile context in mind:

  • People are on the go.
  • Devices are slow and clunky.
  • Phones are hard to type on.

Even after the iPhone came out, people argued that yes, the iPhone had a good browser, but that we’ve had browsers on phones for years and no one used them. People simply don’t want to browse the web on their phones.

This seems laughable now, but it was the accepted consensus at the time.

The reason we had a debate about mobile context when responsive design first arrived is because responsive design challenged the widely accepted idea that people wanted to do different things on their phones.

What do we know about how people will use new devices?

Rosie shared solid research on smart wallpaper. The BBC tested their theories and watched people interact with prototypes. Those observations led to their conclusions about where that future technology would go.

But I found myself wondering what researchers in the early 2000s found when they observed people using their phones. Might they have said something like this:

Suddenly, we all got Smart TVs phones. And it was great. We got Internet on our TVs phones. But actually browsing the web on the TV phone was a really clunky experience. It was not very pleasant. And no one really wanted to do it especially when you’ve got a mobile or tablet next to you computer that makes it easier.

I’m not picking on Rosie here. I do this myself. My gut instinct is to agree with her in many ways.

I find myself thinking, “Well clearly watches are a different thing.” I have similar thoughts about screens in odd places like refrigerators. They don’t feel like they part of the same device context continuum.

But how do I know? I used to think that phones were a different thing.

Predicting future behavior is difficult

Because I was on the wrong side of the mobile context debate, I’ve become leery of our ability to predict future behavior.

In 1994, the New York Times published an article asking “Has the Internet been overhyped?” People were looking at usage of AOL and Prodigy and trying to understand what the impact of the web was going to be.

On a smaller scale, we’re often told that a web site doesn’t need to worry about mobile because the analytics show that people don’t use that site on mobile.

To which I counter, “Why would anyone use your site on mobile if it isn’t designed to work well on those devices? How do you know what people will do after it has been designed to work well on small screens?”

I now have a fundamental rule: we cannot predict future behavior from a current experience that sucks.

Where does the device context continuum end?

All of which brings me to back to my original questions: Where does the common device context continuum start and end? And more importantly, how do we know?

continuum-happy

I’m uncomfortable with the current consensus. Particularly when it comes to TVs. It feels like Justice Potter Stewart saying “I know it when I see it.” It makes me wonder if we’re in the feature phone era of TVs.

I want some guidelines to help me know when something is going to be part of the device context continuum and when it won’t. Some questions we can ask ourselves about devices that will help us see if our current views on a particular device’s context are real or simply artifacts of a current, flawed implementation.

And I wonder if what I wish for is simply impossible in the present.

Thanks to Tyler Sticka for the illustrations.

The feature phone era of TVs ends next week

Three years ago after writing Head First Mobile Web with Lyza, I was burnt out on mobile and wanted to do something different. So I decided to start researching the web on TVs.

TVs seemed like the furthest thing from mobile devices. Before his death, Steve Jobs had told his biographer, Walter Isaacson, that he had “finally cracked” TVs. Rumors were flying that Apple was getting into TVs in a big way.

Photo of TV

So I set off to find out what it might be like to design and build for the web on TVs. I’ve given a few talks on TVs and it has shaped the way I look at the web.

But I’ve never found the time to write about it. With Apple rumored to release its new Apple TV next week, it seemed like the right time to share what I’ve learned.

When I’m looking at TVs and what is possible on them, it doesn’t matter much to me whether the features are being provided by a set top box, game console or the TV itself. I realize that is different when someone makes a purchasing decision.

I’ve focused on Smart TVs because Anna Debenham has done extensive testing on game consoles already and there is less known about Smart TVs. But most of what I’ve found still rings true when I’ve tested game consoles and set top boxes.

No one knows what they’re getting when they buy a Smart TV

The first thing I learned about TVs is how incredibly difficult it is to find TVs that you can test on. Best Buy has walls of TVs, but only three of them had accessible remote controls and were on the Internet.

I fared a little better at Fry’s where the TVs had remote controls, but no Internet. So I tethered the TVs to my phone and watched them start downloading long-overdue software updates:

One of the many TVs that updated firmware using my phone's connection

Eventually, I found a local store, Video Only, that had TVs on WiFi. They’ve been great. I’ve returned every year bearing a box of donuts and a series of tests to conduct.

I drove to several stores before I found one that made it easy to test the smart features of these TVs. Average consumers won’t do this. They have no idea how what their Smart TV can do nor how well it does it before they buy it.

Smart TVs are computers, but no one sells them that way

The discovery happened by accident.

I was at Best Buy using the only TV that was on the Internet that had a remote. I was digging around in some of the Smart TV settings when I happened to notice a tiny progress bar in the lower right corner:

Surprising storage usage indicator

Wait… what? This TV has a storage limit?

Of course it makes sense that the TV would have a storage capacity. The reason why this surprised me is because I had not seen a single sales tag or spec sheet list the storage capacity of the TV.

Even today TV manufacturers will brag about their app stores, they will tout their fancy Octo-core processors, but they don’t list the basic storage capacity.

Good luck even finding the name of the operating system they are using let alone what the current version is.

These TVs are computers. They have downloadable apps. They have various CPU speeds. They have storage limits.

And yet, not a single store nor manufacturer sells them as if they were computing devices. I’m not saying TVs should be sold like a Windows machine, but there is no difference between how TVs are sold now than how they were sold in the 80s.

There is no Smart TV market

Because I spent hours testing TVs in the stores, I was able to observe dozens of people shopping for TVs. In all that time, I never heard a single person ask specifically for a Smart TV.

They asked if the TV supported Netflix.

Remote control with Netflix button

Sometimes they would ask about Hulu or Amazon Video. But they’d never dive deep into the Smart TV features. Even at Video Only where the TVs were on WiFi, only a small percentage of people would ever check out the Smart TV menus.

So while it is difficult to find TVs that don’t have some Smart TV capabilities, I don’t believe you can have a Smart TV market when no one knows what they’re buying and no one is asking for Smart TV features.

Web rendering on TVs is surprisingly capable

I’m going to go into more detail about what I found when testing browsers on TVs another day, but the short version is that the rendering engines on Smart TVs are generally equivalent to same era iPhones.

Both the 2015 LG and Samsung TVs have higher scores on HTML5Test than my iPhone 6 running iOS 8.4.1 does.

2015 LG TV scores 495 on HTML5Test.com

That’s not to say that the web browsing experience is necessarily good. It can be clunky especially if the remote only features a d-pad. But in general, the problem isn’t the rendering engine.

Input remains the biggest challenge

The moment you move away from changing channels and start interacting with the Smart TV functionality, input becomes the biggest challenge. Remote controls are cumbersome and crude input devices.

Over the years I’ve seen TV manufacturers try all sorts of ways to make remote controls better including:

Sony Google TV remote featuring keyboard on back, touch pad on front, and every button you could ever want.

  • Full keyboards
  • Miniature keyboards
  • Motion detection
  • Gesture
  • Touchpads
  • Voice control
  • Smartphone apps

To their credit, TV manufacturers keep looking for ways to make input work better. But no one has cracked it yet.

The Feature Phone Era of TVs

The more I studied TVs, the more I was struck by the similarity between the TV market of today and the phone market before the iPhone was released.

Timeline of phones showing how they changed after iPhone release

Before the iPhone came out, nearly every phone manufacturer had their own operating system. It was often hard to know what the operating system was called and what version you were using.

Input was difficult, slow and frustrating. People advised those who built for phone to keep in mind the mobile context.

Companies touted the ability to install apps and browse the web, but the experience was terrible so few used those features.

The phone manufacturers believed they had a mature market and that they understood what people wanted from their phones. They laughed at the optimism for the iPhone saying that browsers have been on phones for years and no one used them.

There are echoes of each of these in the current TV landscape. And again Apple stands poised to enter the market in a big way led by what sounds like an innovative form of input.

Will Apple pull it off? I don’t know.

But I can tell you one thing, I’m ready for the feature phone era of TVs to end.


Mobile Web Accessibility and Design Beyond Our Devices

Responsive Field Day is only a few short weeks away, yet we’re still announcing awesome new topics. This week is no exception, so let’s get to it!

Marcy Sutton: Mobile Web Accessibility for Developers

Marcy SuttonWe are web developers creating responsive websites and hybrid mobile apps with our HTML, CSS and JavaScript skills. How can we ensure our mobile experiences are accessible to all people, including those with disabilities? In this talk, we’ll discuss mobile web accessibility fundamentals, pain points and development tips you can use in your next project.

Ethan Marcotte: Design Beyond Our Devices

Ethan MarcotteIn this age of device diversity, we’ve been focusing less on pages, and more on patterns: reusable bits of design and content we stitch together into responsive design systems. But those patterns bring puzzles: how should they adapt, and why? And how do we, well, design with them? Let’s look at a few answers to those questions, and start moving our design practices beyond the screens in front of us.

Responsive design, performance, accessibility, modern layouts, animation, pattern libraries, progressive enhancement… you will be seriously bummed if you aren’t at Revolution Hall on September 25. Tickets are still available, you should be there!

Will someone do this on mobile?

A frequent question that product owners face is whether not someone will do a particular activity on a mobile device or not.

In my experience, people often arrive at two answers based on context.

  1. When looking at your own product:
    Of course, no one would ever try to do this on a mobile device.
  2. When using someone else’s product:
    Of course, this should work on mobile.


Design accordingly.

Responsive Images, Part 10: Conclusion

Phew. We made it! We’ve come to the end of the Responsive Images 101 series.

Before we part ways with this series, I want to pass along some tips, resources and final thoughts on where responsive images are heading next.

Responsive image audits

Were this a book, I would have devoted a chapter to responsive image audits. It is the first thing we do when we start looking at how to convert a site to responsive images.

And it is most likely your next step towards taking what you’ve learned and applying it to your sites.

Fortunately, I wrote about these audits in great detail recently. Instead of repeating it in the 101 series, I encourage you to read what I wrote about responsive image audits now. Consider that article Part 9a.

Compatibility

Browser support for responsive images standards is growing rapidly. As of August 2015, Chrome, Opera and Firefox all support picture, srcset, sizes, and type.

Microsoft Edge and Safari support srcset with display density (x) descriptors, but not the width descriptors. Microsoft has started development to support the full responsive images standard.

Apple hasn’t committed to supporting the standard yet, but Apple knows responsive images support is important and Yoav Weiss has been contributing to the WebKit implementation.

When it comes to image-set(), there is still a lot more work to be done.

PictureFill

But even if all the browsers currently supported the responsive images standards, we’d still need a way to help older browsers understand the new syntax. That’s where the PictureFill polyfill comes in.

PictureFill will allow you to use the new responsive images syntax now.

Automating your image processing

In Part 9, I said that humans shouldn’t be picking image breakpoints. Instead, we should have software doing this automatically for us.

I want to expand on this point and say that most of what we deal with when it comes to responsive images is not something that designers and developers should be thinking about on a regular basis.

The goal for most organizations should be to centralize image resizing and processing and automate as much of their responsive images as possible.

In my ideal world, a responsive images workflow for resolution switching would look something like this:

  • Where possible, use resolution independent SVG images.
  • When creating or modifying the design of templates, the template author provides the sizes attribute for the various images in the template.
  • The srcset attribute with width descriptors is inserted by the server which does all of the heavy lifting of figuring out what image breakpoints to choose for each image.
  • Content authors never worry about any of this. Their only responsibility is to upload the highest quality source available and let the image resizing service take care of the rest.

This isn’t a far-fetched scenario. Many organizations already have image resizing services. And if your organization doesn’t, I maintain a spreadsheet of image resizing services and tools that you can consider (be sure to read the explanatory blog post as well).

And many content management systems are starting to look for ways to incorporate responsive images. The Responsive Images Community Group (RICG) maintains a WordPress plugin and they are currently looking at how to add it to WordPress core. Drupal 8 will ship with a responsive images module (more details).

The only thing these image resizing services need to add is support for figuring out how many image sources to supply for a given image and to output the proper markup for those image sources. They may not even need to worry about the markup if Client Hints takes off.

(More on Client Hints soon. This is a 201 topic.)

But regardless of how you automate it, I believe that centralizing image resizing and processing is essential to maintaining your sanity. When we talk to new companies exploring responsive images, one of the first things we assess is their image workflow and how much of it we can automate.

Future of responsive images

We’re just getting started when it comes to responsive images. We have thousands of sites to update to use the new image standards. Many organizations need to update how they handle images to centralize and automate what has until now been a manual process.

Even though there’s still a lot of work ahead of us, it feels like we’re finally on the downhill slope. We’re no longer struggling to find solutions that everyone can agree on. Implementations are landing in browsers. We have PictureFill to help us fill the gaps.

And now the wider web development community is beginning to look at how they are going to implement these new standards which means that we can start learning from each other.

If you’ve read this entire 101 series, you have everything you need to get started with responsive images. I can’t wait to see what you do with the new standards. Please share what you learn!

Thank you for reading.


Responsive Images 101 Series
  1. Definitions
  2. Img Required
  3. Srcset Display Density
  4. Srcset Width Descriptors
  5. Sizes
  6. Picture Element
  7. Type
  8. CSS Responsive Images
  9. Image breakpoints
  10. Conclusion