Since Facebook announced their “fully native” app, some speculation has centered on the idea that Apple may have made a native app a precondition for the upcoming integration of Facebook with iOS 6.
I doubt Apple cares how much HTML Facebook uses in its app. And if it does, it is being hypocritical. All of the following Apple-made iOS apps use embedded webviews in some capacity:
- App Store
- iTunes U
- Apple Store
And that’s not counting the fact that both the iAd and iBooks formats are built on HTML5. Why aren’t people clamoring for Apple to create a “native” version of iBooks or the App Store?
Apple has a lot invested in HTML5. How quickly we forget that WebKit—the rendering engine used by Google Chrome, Android Browser, Samsung’s Dolfin Browser, Blackberry Browser, and numerous others I’m forgetting—was partially created by Apple because of the need to embed web content in a native application.
On the mobile front, Apple has pushed the browser more quickly and further than other companies. The advances still aren’t fast enough for my tastes, and I hope competitors catch up and turn the heat on Apple’s browser effort. Regardless, you’d be hard pressed to make a case that Apple isn’t a major contributor to HTML5.
I suspect Apple does what we do at Cloud Four when we look at apps. Apple likely takes a look at the features of the app and tries to determine if the feature is better as native, web or some combination thereof. When I took a look at the traffic from Apple’s own apps, I certainly saw indications of that thought process. I found:
- Screens that were full native and receiving binary plist files
- Screens that were native and received JSON data
- Screens that were embedded webviews with the full HTML document, associated CSS and JS downloaded
- Screens that were mostly native, but received JSON with HTML encapsulated inside of it for display in certain areas of the screen
So do I think Apple cares if Facebook uses HTML5 in their app? No. I think Apple cares that users have a great experience using Facebook on its platforms. And it is clear that the old Facebook app wasn’t a good experience, and they needed to improve it.
Does that mean that Facebook had to go “fully native” to create a great experience in the eyes of Apple? Obviously not given the fact Apple is using a mixture of web and native its own apps.
There’s a common saying in startups: Ideas don’t matter. Execution does.
For apps, a similar statement can be made: Languages don’t matter. The experience does.
Focus on a great experience, use whatever tools you need to create that experience, and if you succeed, no one will care how the app is built.
I recently did some research into the HTML that Facebook was using in the old version of its iOS app. More on that in a future post. In the meantime, I thought I’d share how to inspect what an iOS app is sending over the network using Charles Proxy.
Before I begin, I must disclose a few things:
- I am not an expert at using a proxy server nor even at how to use Charles Proxy. I am likely using Charles Proxy in naive ways. I have resisted writing this for some time because I know I am an amateurish hack.
- Charles Proxy runs on Windows, MacOS and Linux. I have only used the Mac version and will be talking about it.
- I am a HUGE fan of Charles Proxy. It gives me tremendous joy to be able to see into activity that I otherwise would have no insight into. I feel like the app gives me super powers. I am unabashedly biased about this product.
Ok, let’s start looking at an app shall we?
Setting up Charles Proxy for iOS
First things first, you have to buy Charles Proxy. You can get a free trial to begin with. But the full price for the app is $50 with discounts if multiple licenses are purchased.
You will be surprised to learn that I think it is well worth the $50 and one of the best purchases I’ve made. ;-)
After you install Charles and have it running, getting your iOS device to recognize Charles is fairly easy. First, make sure your iOS device is on the same network as the machine you’re running Charles on.
Go into your network settings on the iOS device and select the wifi network.
At the bottom of the network settings is the HTTP Proxy settings. They are likely off. Select Manual. Enter the IP address of the machine running Charles for the Server and 8888 for the Port.
If you open Safari (or anything else that makes a network connection), you should receive a prompt in Charles asking if it is ok to let the device connect to your proxy server.
After you approve your device, all future network traffic will be routed through Charles. You can record the traffic by hitting the record button in Charles.
If you see a lot of noise coming from your Mac, turn off the Mac OS X Proxy by unchecking it under the Proxy menu or by pressing Shift-Command-P.
Enabling SSL Proxying
Now you can launch the app you want to inspect and see what it downloads. I’ll use the profile page in the new Facebook app as an example.
When you first load the profile page in Charles proxy, things looked promising. You can see a series of requests, how long they took to download, and their size.
Unfortunately, if you try to see what the server sent in response to the request, or even any details on the request itself, you’ll find that you can’t see much or what you do see will be gibberish.
That’s because Facebook is using SSL to encrypt most of the communication between the app and the server. To see what is going on inside that communication, you need to trick your phone into thinking Charles Proxy’s SSL Certificate is valid for the domains you want to inspect.
Setting this up is a fairly easy two step process.
Step 1: Install the Charles Proxy SSL Certificate on your iOS device
If you’re using iOS 4 or above, visit http://charlesproxy.com/charles.crt on your device and install the certificate for Charles.
If you’re on an older version of iOS, want to use Charles as part of your app development, or want to use it with the simulator, check out Charles Proxy’s iOS SSL documentation.
Step 2: Configure Charles Proxy to support SSL proxies for the domains you need
In the menu under Proxy, select Proxy Settings. Select the SSL tab. Enable SSL Proxying if it isn’t enabled.
You can now add as many domains as you need to complete your task. Wildcards work. For examining the Facebook app, I added the following domains:
Basically, I watched the queries and saw what domains requests came in for that I couldn’t see because they were under SSL, and then I proceeded to add them to the list.
Looking at the requests again
Now that we’ve got SSL proxying in place, let’s take another look at those Facebook requests for the profile page and see if we can see more of what is going on.
Ah… that’s more like it. We can now see the full path requested. If we tap on the response section, we can also see what the content of the response was. In this case, the profile page is still an HTML page even in the new “native” Facebook app.
Now that you have the requests and responses, you can do some really cool stuff. Select all of the relevant requests to look at the total payload delivered.
The overview tab will tell you how many requests were made, how many errors occurred, etc.
The Summary tab will let you know the total size of the assets downloaded and the amount of time it took. You can easily sort the requests to find the largest files or the files that took the most time to download.
Exporting your results
Charles allows you to save your session in a Charles Session file or export it as a HAR (HTTP Archive Specification) file or other common formats like CSV which can be shared with others.
Remember to turn off the HTTP Proxy setting
After you’re done testing, you’ll need to turn off the HTTP Proxy settings on your phone. I know it seems obvious, but there have been multiple times where I’ve thought a site was down or my network wasn’t working only to realize I forgot to turn HTTP Proxy off.
X-ray vision for your little black rectangle with rounded corners
That’s it. Easy huh?
You can use Charles Proxy to examine mobile web sites or any network requests so long as the device you are testing on supports HTTP Proxies. It’s a great tool to have in your mobile toolbox.
As long as I’m posting puzzles, riddle me this: what happens to the browser’s lookahead pre-parser if we find the holy grail of responsive images—an image format that works for all resolutions?
What is the holy grail responsive image format?
In nearly every discussion about responsive images, the conversation eventually turns to the idea that what we really need is a new image format that would contain all of the resolutions of a given image.
For example, JPEG 2000 has the ability to “display images at different resolutions and sizes from the same image file”.
A solution like this would be ideal for responsive images. One image file that can be used regardless of the size the image is in the page. The browser only downloads what it needs and nothing more.
As Ian Hickson recently wrote on the WhatWG mailing list regarding future srcset syntax complexities that might come with hypothetical 4x displays:
Or maybe by then we’ll have figured out an image format that magically does the right thing for any particular render width, and we can just do:
…and not have to worry about any of this.
It seems that no matter where you’d like to see responsive images go—srcset, picture, whatever—that everyone agrees we’d all be happier with a new, magical image format.
The end of image breakpoints for most responsive images
If this mythical image format existed and was adopted by browsers, it would mean the end of image breakpoints for most responsive images. You can see this in Ian Hickson’s hypothetical code above.
Unless the image required editing at various viewport widths (see art direction use case), the document author could simply point to a single file and know that the browser would download the data necessary to display the image for anything from a small mobile screen to a large, high-density display.
What does this mean for the lookahead pre-parser?
This is where the thought experiment gets interesting for me. As noted before, the big challenge with responsive images is that the browser’s lookahead pre-parser wants to start downloading images before the layout is finished. This is the central conflict for responsive images which I described as:
How do we reconcile a pre-parser that wants to know what size image to download ahead of time with an image technique that wants to respond to its environment once the page layout has been calculated?
Image breakpoints are designed to give the lookahead pre-parser the information it needs to pick the right image before the browser knows the size of the element in the layout. This is how we get around the problem.
But we just saw that image breakpoints would go away if we had a new, magical image format. Without the image breakpoints and without knowing the size of the image in the page, how would the browser know when to stop downloading the image?
Unless I’m missing something, it wouldn’t. The browser would start downloading the image file and would only stop once the layout had been determined. In the meantime, it may download a lot more data for a given image than is necessary.
Let’s take a look at Flickr as an example. Flickr would be a huge beneficiary of a new image format. Right now, Flickr maintains a bunch of different sizes of images in addition to the original source image.
Instead of having to resize every image, Flickr could use a single image format no matter where the image was used throughout the site.
Lets also assume that in this future world, Flickr was responsive design using flexible images so that the size of an image is dependent on the width of the viewport. What happens to a typical Flickr layout like the one below?
Because it would be a responsive design, the size of the thumbnails on the page will vary based on the screen size. But no matter how big the thumbnails got, they would never equal the full size of the original image which means that the browser would need to stop downloading the magic image file at some point or it would download extra data.
What would the lookahead pre-parser know at the moment it started downloading the images? All it would know is the size of the viewport and the display density.
Based on those two data points, the pre-parser might try to determine when to stop downloading the image based on the maximum size the display could support. That works ok if the display is small, but on a large, high density display, the maximum image could be gigantic.
And because our magical file format would contain multiple resolutions, the browser would likely keep downloading image data until the layout was calculated making it very likely excess image data would be downloaded and the download of other assets would be delayed.
Responsive images versus the lookahead pre-parser
Of course, this is all hypothetical. We don’t have a magical image format on the table. So for now we can avoid resolving what I see as a central conflict between the need for the pre-parser to start speculatively downloading images before it knows the layout of the page and the fact that the size of responsive images isn’t known until the page layout is determined.
But in the long run—if we find our holy grail—this conflict is likely to resurface which makes me wonder about our current efforts.
I whole-heartedly agree with Steve Souders that “speculative downloading is one of the most important performance improvements from browsers”, and until a new image format materializes, it seems we should do everything we can to accomodate the pre-parser.
And at the same time, I can’t help but wonder, if we all want this magical image format and if in some ways it seems inevitable, then are we jumping through hoops to save browser behavior that won’t work in the long run regardless?
For a few months now, I’ve been puzzling over the question of how to pick responsive images breakpoints. It has become a koan to me—no more answerable than the sound of one hand clapping. And as long as I’m haunted by the question, I thought I would share the joy.
What responsive images am I talking about
First, let’s make sure we’re talking about the same thing. In this case, when I refer to responsive images, I’m not talking about:
- An image that is edited depending on the size of the screen. (what I previously called the art direction use case.)
- Different images for different display densities.
What I am talking about is:
- Selecting different image files with different resolutions of the same image based on the screen size.
- A desire to provide these different image files so that a browser doesn’t download an unnecessarily large file.
Basically, the classic use case that got us all looking at responsive images in the first place.
Defining image breakpoints
In responsive designs, we refer to breakpoints as the screen widths at which media queries are used to make changes to the layout. These breakpoints should be determined by the content, not by common device dimensions.
(For more on setting responsive design breakpoints, I highly recommend this excerpt from Tim Kadlec’s new book Implementing Responsive Design)
Image breakpoints are similar. What are the screen widths at which you switch from one image file to another?
Shouldn’t image and responsive design breakpoints be the same?
Not necessarily. If we were talking about the art direction use case, then it is likely that the breakpoints would be the same because changes in the layout might also indicate an edit to the image.
But in the case where we’re simply changing files to provide different resolutions and a faster download, the image breakpoints should be determined based on when the browser is downloading an unnecessarily large file.
Scott Jehl recently made this point on the WhatWG list:
In the responsive layouts I’ve worked on, content image sizes and their breakpoints were chosen for completely different reasons than the design (CSS) breakpoints: the former for sensible jumps in file size to match screen dimension and/or density, and the latter for how content modules are visibly designed at given viewport dimensions.
Design breakpoints can be plentiful, especially when factoring in all the minor and major tweaks in multi-column responsive layout. Yet for content images, I’ve found the need for fewer breakpoints, or even entirely different breakpoints than the design. In a site like the bostonglobe.com for example, 2-3 image sizes provide sensible jumps in file size, and because the images live a fluid layout, they scale to fill the layout gaps as the CSS breakpoints shift much more frequently around them.
So unless the image is changing in coordination with layout changes, the breakpoints for images do not need to coincide with the breakpoints for layout. In fact, coordinating them may be less ideal.
So how do we pick “sensible jumps in file size”?
This is the question that has been troubling me. If we’re going to tell people creating responsive designs that they shouldn’t tie their image breakpoints to their designs, what guidance can we give them on how many file sizes make sense and when should they switch to a different file size?
To wit, how do you determine what is an unnecessarily large file? Is that 1 byte? 10k? 20k?
And if you can answer that question, how do you determine the resolutions at which those file size jumps are going to occur? Depending on the content of an image and the compression algorithm used, images are likely to have different resolutions at which they experience significant changes in file size.
On a large site with a mixture of gif, png, and jpg images, can you pick common image breakpoints? If so, how?
As I wrote a few months ago:
The problem is there is nothing intrinsic to the image that would guide you in deciding where you should switch from one size of the image to another. Whatever we select will be entirely arbitrary unless we base it on our design breakpoints.
And basing our image breakpoints on our design breakpoints is not ideal.
The correctly sized image
The only thing we can absolutely say about what size image should be used for a given screen size is that the ideal solution is one where the resolution of the image matches precisely the size that the image is in the page.
Of course, this isn’t practical unless we have find the holy grail of responsive images—a magical new image format where a single file provides all resolutions and the browser only has to download what it needs to fill the space on the page. And even if we find the format, we’d have to hope it wasn’t legally encumbered and that all of the browser makers could agree on it.
In the meantime, I’m stuck pondering the question of what to tell people when they ask how many image sizes and what breakpoints should they use in their responsive designs.
If you have ideas, I’m all ears. But please explain how you come to that conclusion and how you would advise another designer, on a different project, to figure out what image breakpoints would make sense for them.
- 30 Jan 2013 — This is not a stable specification. It will likely change. Do not use it in a production environment. See Peter’s comment below.
- 30 Jan 2013 — Per Nicholas Shanks’s suggestion, I’ve removed examples for non-WebKit browsers because we don’t know how those browsers will implement the feature and if the syntax will be the same.
Safari 6 and Chrome 21 added vendor-prefixed support for the proposed CSS4 image-set specification. This proposed specification is designed to support displays with different pixel densities (read: retina displays).
I’ve put together a test page that demonstrates image-set and its syntax so you can see if it in action.
Why use image-set instead of media queries?
We can already address high density displays using media queries in CSS so why do we need image-set? Good question.
I see two main benefits for image-set:
Unlike a media query, image-set isn’t telling the browser what image to use, but instead provides options. In the future, I hope that someone using a high density device on a slow connection might be able to tell the browser that they only want low resolution images. It would be even better if the browser could be smart about what image to use based on the current network speed.
The problem with media queries for higher resolution displays is they don’t give the browser any discretion. They say explicitly that if the pixel density equals 1 or 2 or whatever, that the browser should (must?) use a particular image source.
I realize that right now this benefit is largely theoretical because the browsers that support image-set are simply matching the image source to the display density and not providing any additional functionality. But I believe that providing the browser options and letting it pick the right image is the direction we need to go for different resolutions of the same image.
The various resolutions of the image are all in the same place in your css. When you use media queries to specify different image resolutions, you can end up with the image sources separated by dozens of lines of css which makes tracking down the different image sources more challenging.
What about <img> tags?
Unfortunately, a solution for <img> tags is still a work in progress. Image-set only addresses CSS background images.
Browser support and cascade
Image-set is currently only support by Safari 6 and Chrome 21 using the webkit prefix. It is rumored to be coming in iOS 6. As far as I can tell, no other browsers have committed to supporting image-set and it remains a proposal, not part of the CSS4 spec.
BTW, I found the cascade for the image-set syntax to be interesting. As far as I can tell, the code should look like this:
background-image: -webkit-image-set(url(assets/test.png) 1x,
Because the prefix is on the property value instead of the property itself, the cascade looks a bit different than we are accustomed to seeing.
Should you use image-set now?
Well, it is prefixed for a reason. It will may change in the future. So if you have a way to change it easily in the future, then it may make sense to throw caution to the wind and start using it. Otherwise, consider using it in less critical places where you can experiment with new technology.