Apple announced the new Apple TV yesterday. As many expected, it didn’t come with Safari. What was unexpected is that it doesn’t appear to have WebKit at all.
The utility of WebKit for app developers seems straightforward. Apps often use embedded web views to display information that it doesn’t make sense to duplicate in native code or for rendering links that people share.
But without webkit available for tvOS, there will be no embedded web views and no third party browsers.
We now have both Apple TV and Android TV without the web, and it’s a bloody shame.
I’m well aware of the argument that people don’t want to browse the web on TVs. I believe the jury is still out on that one, but even if I concede that point, there is still tremendous utility in using web technology for building apps for TVs.
The reason I started researching the web on TVs dates back to the original Google TV Showcase. There was a Vimeo app in that showcase that I loved.
I used that app for several months before I accidentally hit a button that converted it from the TV app that I knew into the standard Vimeo web page.
I had inadvertently discovered that the Vimeo app was just a different view of Vimeo’s normal web page. Vimeo calls this couch mode.
Nintendo, Samsung, LG, and others have all built app platforms on top of HTML. For years, Netflix built all of their TV apps on HTML5 before recently going native.
Web apps on TVs can be great experiences. Maybe it is because we don’t notice the lag as much when we’re using remote controls instead of touching the interface directly. Maybe it is because the interfaces for most non-game, TV apps are fairly simple.
Whatever the reason, building TV apps using web technology just seemed to work. Bridging the gap between native and web apps on TVs was easier than it was on mobile.
But year after year at Google I/O when I’d try to talk to people about Chrome on TVs, I wouldn’t get anywhere. Google TV shipped with Chrome, but it was a forked version that the Chrome team wasn’t responsible for and grumbled about.
I remember trying desperately to figure out who to talk to about the browser on Google TV. I was repeatedly and humorously pointed to Chris Wilson. Chris hadn’t working on Google TV in months. It became a running joke between the two of us.
A couple years later, Google announced that their TV product would be called Android TV. Google touted how the TV would finally be running the same version of Android as phones and tablets. It would be kept up to date.
Except it would no longer have a browser.
At the time, Microsoft showed more interest in the web on TVs than Google. That may still be the case. I haven’t checked in awhile. Firefox OS has recently moved to TVs and Opera still has a TV browser. So all hope isn’t lost.
Back in 2012, I was trying to muster enthusiasm in browser makers for working on the web on TVs. It seemed likely that TVs were going to be the next platform and instead of playing catch up like the web did on phones, we could be ready for the web on TVs from the beginning.
I feared playing catch up again. In retrospect, I should have feared much worse.
The two biggest mobile operating systems are now on TVs. One started with a browser, but no longer has one. The other just shipped without even an embedded web view.
From what I’ve seen, the web on TV could have been a star. What a missed opportunity.
We’re about an hour away from the Apple event where they will announce the new Apple TV. Here are the things I’m going to be watching for based on my time researching Smart TVs, game consoles, and set top boxes.
How does the remote control work?
Input remains the biggest challenge for all attempts to bring computer smarts to the screens on our walls. While the software and content options for the new Apple TV will matter, if Apple truly revolutionizes TVs, I suspect if will come from an improvement in input.
Historically, improved input has accompanied Apple innovation. The Mac’s mouse. The iPod’s scroll wheel. The iPhone’s touch screen.
The other lesson here is that none of these inputs were wholly Apple inventions. In each case, the input technology had been used by other companies in the past. The iPhone’s touch screen seemed ho-hum until people actually used it and realized how much attention to detail Apple had put into perfecting the input.
So I’ll be surprised if the remote control has some feature that we haven’t seen on remote controls in the past, but I also suspect that if Apple TV is a game changer, it will be because of the remote control.
Is there a pointer? And where is it used?
The current Apple TV is limited to d-pad interactions—up, down, left, right. There are a lot of interactions that need the ability to select an arbitrary point on the screen instead of navigating to that point by successive d-pad button presses.
The most obvious need is in games. The rumors are strong that the new Apple TV will focus on games. The remote control has been described as Wii-like in its ability to detect motion.
The question is whether or not there will be any interfaces where you may see a pointer on the screen. I highly suspect app developers will build apps that include pointers, but do any of the Apple apps themselves include a pointer. And if so, where is it used and how does it work?
Where is web technology used? Is there a browser?
I have little doubt that the new TV operating system will support embedded web views. Web views are critical for many apps.
So the big question is whether or not Apple will include a browser as well. I’ve explored some of the arguments for and against a browser in the past.
And whatever other surprises Apple has in store.
I’ve been looking forward to today’s announcement since I started researching the web on TVs in 2012. I can’t wait to see what it looks like when Apple is no longer treating TVs as a hobby.
The fact the Apple may soon release an App Store for TVs has me revisiting a couple of questions that have troubled me these last few years: Where does the common device context continuum start and end? And more importantly, how do we know?
But before I look at those questions in detail, let’s talk about device context.
The Device Context Continuum
We now design for a continuum of devices. Responsive web design provides us with the techniques we need to design for varying screen sizes.
But responsive web design techniques wouldn’t be effective if there wasn’t a common context—or perhaps more accurately, a lack of context—between devices.
Put a different way, if people did demonstrably different things on mobile phones than they did on desktop computers, then responsive web design wouldn’t be a good solution.
We design for different screen sizes confident in our knowledge that people will do similar things whether they are on phone, tablet or desktop devices. This is our common device context and the continuum that it applies to.
But it hasn’t always been this way.
The Mobile Context Debate
In the early days of responsive web design, people often debated whether or not mobile context was a thing that should be considered in our designs.
At the time, I wrote about my conflicted thoughts on mobile context. I advocated for keeping context in mind. But by 2013, I had concluded mobile context didn’t exist.
Now we have a lot of experience to back up this perspective. Chris Balt, a Senior Web Product Manager at Microsoft, told Karen McGrane and Ethan Marcotte on the Responsive Web Design podcast:
Our data shows us quite plainly and clearly that the behavior of those on our mobile devices and the small screens is really not all that different than the behavior of those on the desktop. And the things they are seeking to do and the tasks they are seeking to accomplish are really quite the same.
Karen and Ethan have been doing a weekly podcast for a year. In that time, regardless of the company or industry being discussed, people say that they see no difference in what people want to do based on whether they are using a mobile, tablet or desktop.
I still think Luke Wroblewski nailed it when he wrote:
But if there’s one thing I’ve learned in observing people on their mobile devices, it’s that they’ll do anything on mobile if they have the need. Write long emails? Check. Manage complex sets of information? Check. And the list goes on. If people want to do it, they’ll do it on mobile -especially when it’s their only or most convenient option.
What about new devices? TVs? Watches?
It seems that not a day goes by without a new device form factor being introduced. Watches. TVs. Virtual reality goggles. Augment reality glasses.
Where do these new devices fit in on this device context continuum? Do they share the same context?
The consensus at the moment seems to be that they are not part of the same continuum as phones, tablets and computers. When you read the guidelines for designing for watches or TVs, designers are advised to take context into consideration.
At Responsive Day Out, Rosie Campbell, who works in Research and Development for the BBC, gave a compelling presentation entitled Designing for displays that don’t yet exist. She shared research on what it would take to build a compelling smart wallpaper experience in the future when such technology might become commonplace.
In the talk, Rosie made two comments that I’ve been thinking about ever since. She addressed what we need to do as screens get weirder:
It’s not just about making content look beautiful on those different screens. We also need to think about what is appropriate for each device because you’re probably not going to want the same kind of information on your smart watch as you want on your smart wallpaper.
This makes intuitive sense to me. For whatever reason, my Apple Watch feels very different than my phone or my computer.
But Rosie also used browsers on Smart TVs to illustrate a point that just because a technology makes something possible, doesn’t mean that we should design experiences around it:
Suddenly, we all got Smart TVs. And it was great. We got Internet on our TVs. But actually browsing the web on the TV was a really clunky experience. It was not very pleasant. And no one really wanted to do it especially when you’ve got a mobile or tablet next to you that makes it easier.
Again, what Rosie states here is the popular consensus that people won’t browse the web on their TVs. Steve Jobs famously said that:
[People] don’t want a computer on their TV. They have computers. They go to their wide-screen TVs for entertainment. Not to have another computer. This is a hard one for people in the computer industry to understand, but it’s really easy for consumers to understand. They get it.
I’ve spent the last three years researching the web on TVs wondering about exactly this question. And it isn’t clear cut to me whether or not people will browse the web on TV screens in the future.
The consensus on mobile context has changed
The popular consensus used to be that no one wanted to browse the web on their phones. If you dared to build something targeting phones, you were advised to keep the mobile context in mind:
- People are on the go.
- Devices are slow and clunky.
- Phones are hard to type on.
Even after the iPhone came out, people argued that yes, the iPhone had a good browser, but that we’ve had browsers on phones for years and no one used them. People simply don’t want to browse the web on their phones.
This seems laughable now, but it was the accepted consensus at the time.
The reason we had a debate about mobile context when responsive design first arrived is because responsive design challenged the widely accepted idea that people wanted to do different things on their phones.
What do we know about how people will use new devices?
Rosie shared solid research on smart wallpaper. The BBC tested their theories and watched people interact with prototypes. Those observations led to their conclusions about where that future technology would go.
But I found myself wondering what researchers in the early 2000s found when they observed people using their phones. Might they have said something like this:
Suddenly, we all got Smart
TVs phones. And it was great. We got Internet on our TVs phones. But actually browsing the web on the TV phone was a really clunky experience. It was not very pleasant. And no one really wanted to do it especially when you’ve got a mobile or tablet next to you computer that makes it easier.
I’m not picking on Rosie here. I do this myself. My gut instinct is to agree with her in many ways.
I find myself thinking, “Well clearly watches are a different thing.” I have similar thoughts about screens in odd places like refrigerators. They don’t feel like they part of the same device context continuum.
But how do I know? I used to think that phones were a different thing.
Predicting future behavior is difficult
Because I was on the wrong side of the mobile context debate, I’ve become leery of our ability to predict future behavior.
In 1994, the New York Times published an article asking “Has the Internet been overhyped?” People were looking at usage of AOL and Prodigy and trying to understand what the impact of the web was going to be.
On a smaller scale, we’re often told that a web site doesn’t need to worry about mobile because the analytics show that people don’t use that site on mobile.
To which I counter, “Why would anyone use your site on mobile if it isn’t designed to work well on those devices? How do you know what people will do after it has been designed to work well on small screens?”
I now have a fundamental rule: we cannot predict future behavior from a current experience that sucks.
Where does the device context continuum end?
All of which brings me to back to my original questions: Where does the common device context continuum start and end? And more importantly, how do we know?
I’m uncomfortable with the current consensus. Particularly when it comes to TVs. It feels like Justice Potter Stewart saying “I know it when I see it.” It makes me wonder if we’re in the feature phone era of TVs.
I want some guidelines to help me know when something is going to be part of the device context continuum and when it won’t. Some questions we can ask ourselves about devices that will help us see if our current views on a particular device’s context are real or simply artifacts of a current, flawed implementation.
And I wonder if what I wish for is simply impossible in the present.
Thanks to Tyler Sticka for the illustrations.
Three years ago after writing Head First Mobile Web with Lyza, I was burnt out on mobile and wanted to do something different. So I decided to start researching the web on TVs.
TVs seemed like the furthest thing from mobile devices. Before his death, Steve Jobs had told his biographer, Walter Isaacson, that he had “finally cracked” TVs. Rumors were flying that Apple was getting into TVs in a big way.
So I set off to find out what it might be like to design and build for the web on TVs. I’ve given a few talks on TVs and it has shaped the way I look at the web.
But I’ve never found the time to write about it. With Apple rumored to release its new Apple TV next week, it seemed like the right time to share what I’ve learned.
When I’m looking at TVs and what is possible on them, it doesn’t matter much to me whether the features are being provided by a set top box, game console or the TV itself. I realize that is different when someone makes a purchasing decision.
I’ve focused on Smart TVs because Anna Debenham has done extensive testing on game consoles already and there is less known about Smart TVs. But most of what I’ve found still rings true when I’ve tested game consoles and set top boxes.
No one knows what they’re getting when they buy a Smart TV
The first thing I learned about TVs is how incredibly difficult it is to find TVs that you can test on. Best Buy has walls of TVs, but only three of them had accessible remote controls and were on the Internet.
I fared a little better at Fry’s where the TVs had remote controls, but no Internet. So I tethered the TVs to my phone and watched them start downloading long-overdue software updates:
Eventually, I found a local store, Video Only, that had TVs on WiFi. They’ve been great. I’ve returned every year bearing a box of donuts and a series of tests to conduct.
I drove to several stores before I found one that made it easy to test the smart features of these TVs. Average consumers won’t do this. They have no idea how what their Smart TV can do nor how well it does it before they buy it.
Smart TVs are computers, but no one sells them that way
The discovery happened by accident.
I was at Best Buy using the only TV that was on the Internet that had a remote. I was digging around in some of the Smart TV settings when I happened to notice a tiny progress bar in the lower right corner:
Wait… what? This TV has a storage limit?
Of course it makes sense that the TV would have a storage capacity. The reason why this surprised me is because I had not seen a single sales tag or spec sheet list the storage capacity of the TV.
Even today TV manufacturers will brag about their app stores, they will tout their fancy Octo-core processors, but they don’t list the basic storage capacity.
Good luck even finding the name of the operating system they are using let alone what the current version is.
These TVs are computers. They have downloadable apps. They have various CPU speeds. They have storage limits.
And yet, not a single store nor manufacturer sells them as if they were computing devices. I’m not saying TVs should be sold like a Windows machine, but there is no difference between how TVs are sold now than how they were sold in the 80s.
There is no Smart TV market
Because I spent hours testing TVs in the stores, I was able to observe dozens of people shopping for TVs. In all that time, I never heard a single person ask specifically for a Smart TV.
They asked if the TV supported Netflix.
Sometimes they would ask about Hulu or Amazon Video. But they’d never dive deep into the Smart TV features. Even at Video Only where the TVs were on WiFi, only a small percentage of people would ever check out the Smart TV menus.
So while it is difficult to find TVs that don’t have some Smart TV capabilities, I don’t believe you can have a Smart TV market when no one knows what they’re buying and no one is asking for Smart TV features.
Web rendering on TVs is surprisingly capable
I’m going to go into more detail about what I found when testing browsers on TVs another day, but the short version is that the rendering engines on Smart TVs are generally equivalent to same era iPhones.
Both the 2015 LG and Samsung TVs have higher scores on HTML5Test than my iPhone 6 running iOS 8.4.1 does.
That’s not to say that the web browsing experience is necessarily good. It can be clunky especially if the remote only features a d-pad. But in general, the problem isn’t the rendering engine.
Input remains the biggest challenge
The moment you move away from changing channels and start interacting with the Smart TV functionality, input becomes the biggest challenge. Remote controls are cumbersome and crude input devices.
Over the years I’ve seen TV manufacturers try all sorts of ways to make remote controls better including:
- Full keyboards
- Miniature keyboards
- Motion detection
- Voice control
- Smartphone apps
To their credit, TV manufacturers keep looking for ways to make input work better. But no one has cracked it yet.
The Feature Phone Era of TVs
The more I studied TVs, the more I was struck by the similarity between the TV market of today and the phone market before the iPhone was released.
Before the iPhone came out, nearly every phone manufacturer had their own operating system. It was often hard to know what the operating system was called and what version you were using.
Input was difficult, slow and frustrating. People advised those who built for phone to keep in mind the mobile context.
Companies touted the ability to install apps and browse the web, but the experience was terrible so few used those features.
The phone manufacturers believed they had a mature market and that they understood what people wanted from their phones. They laughed at the optimism for the iPhone saying that browsers have been on phones for years and no one used them.
There are echoes of each of these in the current TV landscape. And again Apple stands poised to enter the market in a big way led by what sounds like an innovative form of input.
Will Apple pull it off? I don’t know.
But I can tell you one thing, I’m ready for the feature phone era of TVs to end.
Responsive Field Day is only a few short weeks away, yet we’re still announcing awesome new topics. This week is no exception, so let’s get to it!
Marcy Sutton: Mobile Web Accessibility for Developers
In this age of device diversity, we’ve been focusing less on pages, and more on patterns: reusable bits of design and content we stitch together into responsive design systems. But those patterns bring puzzles: how should they adapt, and why? And how do we, well, design with them? Let’s look at a few answers to those questions, and start moving our design practices beyond the screens in front of us.
Responsive design, performance, accessibility, modern layouts, animation, pattern libraries, progressive enhancement… you will be seriously bummed if you aren’t at Revolution Hall on September 25. Tickets are still available, you should be there!