Adventures on the mobile Web frontier: Wrangling Drupal, third-party APIs and server oomph into an enterprise-class experience

You’ll hear us at Cloud Four often use the metaphor that mobile Web development is still in the Wild West phase: there’s a lot of hooting and hollering and posturing, but it’s still a sparsely-settled, incompletely-explored and exhilarating place full of legends and buried treasure. Well, perhaps not that glamorous. It’s sort of the unsung, rough-and-tumble mountain man cousin compared to those dashing native apps. But there’s gold in them hills.

When Cloud Four recently helped Hautelook, a high-volume “flash sale” retail site, build their mobile Web strategy, we got a chance to see if some of our bolder theories about the mobile Web—context, content adaptation, technology stack—would hold water in a performance-critical environment.

Supporting large numbers of concurrent connections from widely disparate devices, handling the crush of shopping enthusiasm at 8AM Pacific every day (when new sales events go live), all while interfacing with a remote, third-party API meant we really had to push the envelope. And invent some neat tricks. And find a few things out the hard way.

It’s an enduring debate: should the savvy mobile Web developer adapt content for mobile devices on the server, or should most of that happen on the device (client) itself? Before everyone gets too riled, let me say this: I believe both approaches have solid merit. There.

For Hautelook, we’re letting the servers do the heavy lifting. In a nutshell, we use the Drupal Web content management system (CMS), along with the WURFL device library and some Cloud Four-brewed software to define and manage content for several categories of devices.

We’re able to put the brawn of the server hardware behind making decisions about incoming requests and are able to route the right devices, very quickly—utilizing memory-based caching and other performance tuning—to the right place, ensuring that only those bytes relevant to the user are delivered. In such a high-traffic environment, it simply won’t do to be pumping out markup, images or CSS that aren’t actively needed for the particular user.

In addition, Hautelook’s mobile Web site supports a rather dizzying array of devices. While the majority of users are on newer smartphones—iOS, Android, BlackBerry 5+ and WebOS are the major platform players here—we do see a consistent flow of folks using all sorts of older devices. For older phones and feature phones, it can be dangerous to rely on the client too much, and especially perilous to assume any sort of meaningful JavaScript or advanced CSS (ahem, media query) support. That puts the server in the best position to make certain kinds of decisions.

Another thing that sometimes precludes client-side adaptation is the need not just to provide adapted content to a given device class, but in fact to provide different content altogether. For example, in Hautelook’s case, product listings are output as HTML unordered lists on modern devices—the semantically obvious way to do it. However, due to some brain-dead-ness in certain devices (I’m looking at you, BlackBerry 4.x browsers, and you, certain weird flavors of IE Mobile), CSS for unordered lists was, well, predominantly it wasn’t behaving in any socially-acceptable way. In this case, we opted to deliver product listings as HTML tables for those devices: inelegant, smacks of 1997, but easier to rein in on ill-behaving phones. We’re able to do that server-side by leveraging Drupal’s theming system and our own content-adaptation framework.

Another situation that calls for not-just-adjusted-but-different content is in the handling of different user contexts: we know that browsing high fashion on a 128-pixel Samsung smartphone is not going to be as compelling as on a big, comfy desktop monitor. Instead we know that those users probably want to find stuff fast and buy it even faster. It’s not just about delivering mobile-shaped content, it’s about delivering mobile-relevant content, fresh and piping hot from the server.

Then there are images. Shoving a 153,600-pixel image at a phone that only has about 16,000 pixels of real estate to work with is like giving 4-by-8-foot sheets of plywood to a kid who wants to build a miniature birdhouse. Even if wood is cheap, it tastelessly wastes a lot of trees, and poor little underpowered Billy can barely lift the thing or cut it with his little toy handsaw. Not to stretch a metaphor too far or anything.

By extending our definition and handling of what a device class is and does, we can deliver the right kind of images and other media to the right kinds of devices.

That doesn’t mean that there is no room for client-side awesomeness! We’re pretty excited about stuff we can do with tools like Modernizr, especially for newer devices. The full solution certainly includes both client- and server-side elements.

1. Akamai sits in front of both sites (desktop and mobile). It’s the first-line mobile detection workhorse. It looks at incoming traffic and determines whether it is a mobile device or not. Mobile devices are sent to the mobile site*.
2. Sometimes you need an Abrams Tank, sometimes you want the brisk efficiency of a Hyundai (hey, newer Hyundais are pretty decent!). In our case, we’re putting the nginx web server (our high gas-mileage Hyundai) out on the front lines. nginx is dinky and quick, and can handle whole scads of simultaneous connections. Apache has big guns and horsepower on its side, but sprightly little nginx acts as gatekeeper, navigator and hand-holder, only sending for the tanks when it really needs the tanks. nginx handles Keep-Alive connections and load balancing, leaving apache to do what it does best. Even with apache, we’re being as efficient as possible, disabling Keep-Alives and using APC to keep application code in memory.
3. We can arbitrarily scale by adding more application instances (apache/Drupal) on more servers.
4. Now we’ve made it to the apache/Drupal piece of the puzzle. We know our requester is a mobile device, but now we want to know what kind. We’ve written a Drupal module that lets us define what characteristics constitute different devices, giving us control over what each device class “means.” This is a (very) similar approach to the available, third-party Drupal module mobile_tools. Our module communicates with the WURFL PHP API, but only when it needs to. Once a device has been identified, we cache what we need to know about it, keyed by User Agent. Any further requests from devices using that User Agent are handled super duper fast, using information cached in memory.
5. Intelligent caching, while not strictly a mobile-specific tactic, is absolutely core to our strategy. We cache as aggressively as we can, balancing performance against the need to have very fresh data out of the API. API calls are shared across users and multiple page requests, wherever possible.
6. While we aren’t delivering the same content to each device class, nor are we replicating markup between them. We define baseline content theming, from which each device class inherits, and has the opportunity to override. Again, we cache what we can—yes, even device-specific rendered markup in some cases—to push performance even further.

* Don’t worry! Mobile users can opt to view the desktop site and they won’t be redirected back to the mobile site.

Here’s some stuff that was hard (warning, highly Drupal-specific/tech-y):

• Drupal has, generally, a phenomenally-flexible hook system. I can, 98% of the time, find a pleasing way to implement something without ugliness. But this breaks down (in Drupal 6) when it comes to tight control of CSS and JS inclusion in pages. Drupal 7 introduces hook_js_alter() and hook_css_alter(), which will fix the exact problem I had to lightly hack around on the theme level for this project.
• WURFL is awesome sauce. It contains a veritable treasure trove of device-specific data. One thing that’s challenging still, however, is the conflation of devices and browsers. I hear they’re working on this as we speak! But at present, using the PHP WURFL API, Opera Mini on an iPhone is identified as…an iPhone. With Safari, and WebKit. Eep.
• OpenWave browsers (quite popular on older feature phones, but also still used on some recent ones) cause pain. Drupal forms with an action that is the same as the page they are on will create an infinite redirect loop, somehow. The only solution seems to be to tack on an interstitial #redirect page (via hook_form_alter()) for those forms that don’t have one. Also, I was only ever successful in getting them to correctly handle cookies if I used a TLD cookie domain. UGH.
• Some BlackBerries don’t take it well if you destroy and write to the same cookie in a single request.
• If you want to serve up valid XHTML Mobile Profile 1.2 content, you’ll need to override Drupal’s default theme implementation for ‘table’. It uses THEAD tags, which are a no-no in XHTML MP1.2.
• RFC 2109 is a curse. But I digress.