Skip to main content

When is Mobile Safari not Mobile Safari?

By Jason Grigsby

Published on January 12th, 2011

Topics

When it is AppleCoreMedia. Let me explain.

I wrote recently about a quiz that we built that included HTML5 audio files. In addition to the volume problems already discussed, we encountered intermittent issues where the audio either would not play or would play much later than it should.

For the quiz, we preloaded the HTML5 audio file via javascript. To make sure that the file was set up for caching, we set far future expires headers as well as making sure the file was sufficiently small.

Whenever we displayed the screen that told the user whether or not they got a question correct, we play the preloaded audio file. Most of the time this worked flawlessly, but occasionally the iPhone would play the sound long after the results screen was displayed or not play the file at all.

That’s when things started getting weird. We watched the log files and started seeing some odd behavior. Here’s what we saw:

"GET /trivia HTTP/1.1" 200 2346 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /trivia.css HTTP/1.1" 200 988 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
Code language: HTML, XML (xml)

See what happened there? The user agent string changed from the normal Mobile Safari one (AppleWebKit) to one I hadn’t seen before: AppleCoreMedia.

It makes sense when you think about it. All video and audio playback gets handled by a system component. That component is responsible for downloading the asset. Ergo, the user agent string should change accordingly.

Surprisingly, this is true on desktop Safari as well. The equivalent user agent string is “Apple Mac OS X v10.6.6 CoreMedia v1.0.0.10J567.”

Permalink to AppleCoreMedia Caching

AppleCoreMedia Caching

On its own, the change from Mobile Safari’s user agent string to AppleCoreMedia’s user agent string would hardly be worth noting. But we were encountering problems with the sound not playing all of the time.

Because we noticed the sound problems happened more frequently on 3G than on WiFi, we started to suspect network issues and wondered if the audio file that we preloaded was getting cached properly. Here’s what we found in the logs:

"GET /trivia HTTP/1.1" 200 2346 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /trivia.css HTTP/1.1" 200 988 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 5494 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 2597 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /js/trivia.js HTTP/1.1" 200 2618 -" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /media/quiz_right.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 337 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_right.m4a HTTP/1.1" 206 2597 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /media/quiz_wrong.m4a HTTP/1.1" 206 5494 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
Code language: HTML, XML (xml)

All of that occurs before the first quiz answer is submitted so it is only in relation to preloading the audio. The files get downloaded multiple times. Lest you think this is simply http chunking per the 206 responses, here are the file sizes:

-rw-r--r-- 1 cloudfour psacln 2256 Dec 10 17:33 quiz_right.m4a
-rw-r--r-- 1 cloudfour psacln 5153 Dec 10 17:33 quiz_wrong.m4a
Code language: HTML, XML (xml)

Before the audio file is ever played, AppleCoreMedia has downloaded 11,662 bytes for quiz_wrong.m4a. At 5153 bytes, the source file is less than half of the total bytes download.

When we could replicate the audio problems, we would find that the server would return a 304 response to AppleCoreMedia letting it know that the m4a had not been modified, but then AppleCoreMedia would go ahead and download it anyways. Of course, this behavior was inconsistent making it difficult to troubleshoot.

To try to narrow down the behavior, I created a simple page with links to m4a and mp3 files. I didn’t do any HTML5 audio embedding and didn’t include javascript. The page was as vanilla as can be. The test file size was 30,196 bytes. Clicking on the m4a link with a clear cache had the following result:

"GET /examples/applecoremedia/trailer_iphone_sound/ HTTP/1.1" 200 653 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 200 30551 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 304 233 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 206 22410 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 304 233 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.m4a HTTP/1.1" 206 22410 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
Code language: HTML, XML (xml)

This is even more confusing than the last example. The full m4a file is first downloaded by Mobile Safari. Then AppleCoreMedia asks the server if the file has been modified and is told by the server that it hasn’t. AppleCoreMedia then ignores this information and proceeds to download the file. Then it repeats the process for good measure.

And despite setting far future expires headers for everything, the next time you play the m4a file, the same process is repeated.

Because of the issues we had previously had with short audio files, I decided to try a longer video file to see what it would do. Again, I set up a simple page and linked to the video file. The video file was 3,995,176 bytes (3.8MB).

"GET /examples/applecoremedia/trailer_iphone/ HTTP/1.1" 200 729 "http://www.cloudfour.com/examples/applecoremedia/" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 386 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 200 70500 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 71910 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 386 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
"GET /examples/applecoremedia/trailer_iphone/trailer_iphone%20-%20iPhone.m4v HTTP/1.1" 206 3995571 "-" "AppleCoreMedia/1.0.0.8C148 (iPhone; U; CPU OS 4_2_1 like Mac OS X; en_us)"
Code language: HTML, XML (xml)

Mobile Safari still downloads some part of the m4v file, but it isn’t the full file size (70500 bytes). It is unclear what it does with this data chunk nor why it appears to only be part of the data, but the server reports a 200 response instead of 206.

The next time the video is played, the same pattern repeats. Nothing appears to be cached. Based on previous research into iPhone 4 cache sizes, I would have expect even the 3.8MB video to get cached.

Permalink to What I Expected to See

What I Expected to See

To illustrate what I expected to see, I ran with my simple audio file test in Firefox. The only change I made was to use an mp3 file instead of an m4a.

"GET /examples/applecoremedia/trailer_iphone_sound/ HTTP/1.1" 200 653 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
"GET /examples/applecoremedia/trailer_iphone_sound/trailer_iphone_trimmed.mp3 HTTP/1.1" 200 28185 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
Code language: HTML, XML (xml)

No matter how many times I load the mp3 file and then hit the back button to the web page containing the link, neither the web page nor the mp3 file were downloaded again. They had been successfully cached.

Permalink to What does all this mean?

What does all this mean?

I wish I knew. For our project, it meant that we could not consistently ensure that the audio file would play at the right moment when the message was displayed on the screen letting the user know if they had got the question right or not. This issue combined with the volume issues caused us to remove the feature.

For consumers using iPhones, it appears to mean that every time you watch a video or play an audio file, that your phone is going to download it again. This makes for a slower experience and puts a seemingly unnecessary burden on the carrier network.

For developers, it is important to realize that video and audio playback in Mobile Safari is not handled by Mobile Safari even if you’re controlling it via javascript and the system media player is never visible. Everything you’ve learned about how Safari and Mobile Safari handle the downloading of assets doesn’t apply when AppleCoreMedia is doing the work instead of Safari.

Finally, this is just one more clue about how mobile is still frontier land. Even something that seems simple like a quiz that plays sounds can bump up against the boundaries of what we know about how mobile browsers work and what they can reliably do.

Jason Grigsby is one of the co-founders of Cloud Four, Mobile Portland and Responsive Field Day. He is the author of Progressive Web Apps from A Book Apart. Follow him at @grigs.

Comments

James Pearce said:

I see a lot of these UAs in the tinySrc logs too, but each contains a strange brand name. My conclusion was that they are from (different) native apps downloading images via WebView components, and your suggestions seem to confirm that.

The wild west indeed.

James Abley said:

It’s been like this ever since the iPhone came out [1]. Unfortunately, that link contains broken links to the relevant Apple documentation.

The device makes HTTP Range requests [2] and your server is correctly returning a 206 Partial Content response. You’d need to look at the Range HTTP header to understand which parts of the entity actually get downloaded. The fact that it appears to be so inefficient with the network is news to me; good catch. It might be worth doing more analysis and maybe raising something in Radar?

[1],[2] Your spam protection stopped me adding any links to my comment?

Bill said:

I’m having a similar issue. I’ve got a hit counter on my podcast that tracks GET requests. Every day I get at least twenty hits that appear to be coming from bots, but when I look at the logs, it’s just a bunch of groups of six or seven requests from an AppleCoreMedia device, all within the space of a couple of seconds. After reading this article, I think I’ll be able to just filter that traffic out of the hit counter, and not sacrifice hit count accuracy.