Cloud Four Blog

Technical notes, War stories and anecdotes

Adventures on the mobile Web frontier: Wrangling Drupal, third-party APIs and server oomph into an enterprise-class experience

You’ll hear us at Cloud Four often use the metaphor that mobile Web development is still in the Wild West phase: there’s a lot of hooting and hollering and posturing, but it’s still a sparsely-settled, incompletely-explored and exhilarating place full of legends and buried treasure. Well, perhaps not that glamorous. It’s sort of the unsung, rough-and-tumble mountain man cousin compared to those dashing native apps. But there’s gold in them hills.

When Cloud Four recently helped Hautelook, a high-volume “flash sale” retail site, build their mobile Web strategy, we got a chance to see if some of our bolder theories about the mobile Web—context, content adaptation, technology stack—would hold water in a performance-critical environment.

Supporting large numbers of concurrent connections from widely disparate devices, handling the crush of shopping enthusiasm at 8AM Pacific every day (when new sales events go live), all while interfacing with a remote, third-party API meant we really had to push the envelope. And invent some neat tricks. And find a few things out the hard way.

Adapting the Experience: Server or Client?

It’s an enduring debate: should the savvy mobile Web developer adapt content for mobile devices on the server, or should most of that happen on the device (client) itself? Before everyone gets too riled, let me say this: I believe both approaches have solid merit. There.

For Hautelook, we’re letting the servers do the heavy lifting. In a nutshell, we use the Drupal Web content management system (CMS), along with the WURFL device library and some Cloud Four-brewed software to define and manage content for several categories of devices.

We’re able to put the brawn of the server hardware behind making decisions about incoming requests and are able to route the right devices, very quickly—utilizing memory-based caching and other performance tuning—to the right place, ensuring that only those bytes relevant to the user are delivered. In such a high-traffic environment, it simply won’t do to be pumping out markup, images or CSS that aren’t actively needed for the particular user.

In addition, Hautelook’s mobile Web site supports a rather dizzying array of devices. While the majority of users are on newer smartphones—iOS, Android, BlackBerry 5+ and WebOS are the major platform players here—we do see a consistent flow of folks using all sorts of older devices. For older phones and feature phones, it can be dangerous to rely on the client too much, and especially perilous to assume any sort of meaningful JavaScript or advanced CSS (ahem, media query) support. That puts the server in the best position to make certain kinds of decisions.

Another thing that sometimes precludes client-side adaptation is the need not just to provide adapted content to a given device class, but in fact to provide different content altogether. For example, in Hautelook’s case, product listings are output as HTML unordered lists on modern devices—the semantically obvious way to do it. However, due to some brain-dead-ness in certain devices (I’m looking at you, BlackBerry 4.x browsers, and you, certain weird flavors of IE Mobile), CSS for unordered lists was, well, predominantly it wasn’t behaving in any socially-acceptable way. In this case, we opted to deliver product listings as HTML tables for those devices: inelegant, smacks of 1997, but easier to rein in on ill-behaving phones. We’re able to do that server-side by leveraging Drupal’s theming system and our own content-adaptation framework.

Another situation that calls for not-just-adjusted-but-different content is in the handling of different user contexts: we know that browsing high fashion on a 128-pixel Samsung smartphone is not going to be as compelling as on a big, comfy desktop monitor. Instead we know that those users probably want to find stuff fast and buy it even faster. It’s not just about delivering mobile-shaped content, it’s about delivering mobile-relevant content, fresh and piping hot from the server.

Then there are images. Shoving a 153,600-pixel image at a phone that only has about 16,000 pixels of real estate to work with is like giving 4-by-8-foot sheets of plywood to a kid who wants to build a miniature birdhouse. Even if wood is cheap, it tastelessly wastes a lot of trees, and poor little underpowered Billy can barely lift the thing or cut it with his little toy handsaw. Not to stretch a metaphor too far or anything.

By extending our definition and handling of what a device class is and does, we can deliver the right kind of images and other media to the right kinds of devices.

That doesn’t mean that there is no room for client-side awesomeness! We’re pretty excited about stuff we can do with tools like Modernizr, especially for newer devices. The full solution certainly includes both client- and server-side elements.

Nutshell: How it works

Mobile Web service overview

  1. Akamai sits in front of both sites (desktop and mobile). It’s the first-line mobile detection workhorse. It looks at incoming traffic and determines whether it is a mobile device or not. Mobile devices are sent to the mobile site*.
  2. Sometimes you need an Abrams Tank, sometimes you want the brisk efficiency of a Hyundai (hey, newer Hyundais are pretty decent!). In our case, we’re putting the nginx web server (our high gas-mileage Hyundai) out on the front lines. nginx is dinky and quick, and can handle whole scads of simultaneous connections. Apache has big guns and horsepower on its side, but sprightly little nginx acts as gatekeeper, navigator and hand-holder, only sending for the tanks when it really needs the tanks. nginx handles Keep-Alive connections and load balancing, leaving apache to do what it does best. Even with apache, we’re being as efficient as possible, disabling Keep-Alives and using APC to keep application code in memory.
  3. We can arbitrarily scale by adding more application instances (apache/Drupal) on more servers.
  4. Now we’ve made it to the apache/Drupal piece of the puzzle. We know our requester is a mobile device, but now we want to know what kind. We’ve written a Drupal module that lets us define what characteristics constitute different devices, giving us control over what each device class “means.” This is a (very) similar approach to the available, third-party Drupal module mobile_tools. Our module communicates with the WURFL PHP API, but only when it needs to. Once a device has been identified, we cache what we need to know about it, keyed by User Agent. Any further requests from devices using that User Agent are handled super duper fast, using information cached in memory.
  5. Intelligent caching, while not strictly a mobile-specific tactic, is absolutely core to our strategy. We cache as aggressively as we can, balancing performance against the need to have very fresh data out of the API. API calls are shared across users and multiple page requests, wherever possible.
  6. While we aren’t delivering the same content to each device class, nor are we replicating markup between them. We define baseline content theming, from which each device class inherits, and has the opportunity to override. Again, we cache what we can—yes, even device-specific rendered markup in some cases—to push performance even further.

* Don’t worry! Mobile users can opt to view the desktop site and they won’t be redirected back to the mobile site.

Challenges

Here’s some stuff that was hard (warning, highly Drupal-specific/tech-y):

  • Drupal has, generally, a phenomenally-flexible hook system. I can, 98% of the time, find a pleasing way to implement something without ugliness. But this breaks down (in Drupal 6) when it comes to tight control of CSS and JS inclusion in pages. Drupal 7 introduces hook_js_alter() and hook_css_alter(), which will fix the exact problem I had to lightly hack around on the theme level for this project.
  • WURFL is awesome sauce. It contains a veritable treasure trove of device-specific data. One thing that’s challenging still, however, is the conflation of devices and browsers. I hear they’re working on this as we speak! But at present, using the PHP WURFL API, Opera Mini on an iPhone is identified as…an iPhone. With Safari, and WebKit. Eep.
  • OpenWave browsers (quite popular on older feature phones, but also still used on some recent ones) cause pain. Drupal forms with an action that is the same as the page they are on will create an infinite redirect loop, somehow. The only solution seems to be to tack on an interstitial #redirect page (via hook_form_alter()) for those forms that don’t have one. Also, I was only ever successful in getting them to correctly handle cookies if I used a TLD cookie domain. UGH.
  • Some BlackBerries don’t take it well if you destroy and write to the same cookie in a single request.
  • If you want to serve up valid XHTML Mobile Profile 1.2 content, you’ll need to override Drupal’s default theme implementation for ‘table’. It uses THEAD tags, which are a no-no in XHTML MP1.2.
  • RFC 2109 is a curse. But I digress.

12 Comments on “Adventures on the mobile Web frontier: Wrangling Drupal, third-party APIs and server oomph into an enterprise-class experience”

  1. GREAT post.

    One more note on Drupal forms is that It likes to redirect you automatically after a form POST, this gives a generic error on many Nokia S40 without webkit. Back in the days of dotMobi we had to do some serious hacks to behave differently,

  2. james Pearce says:

    In my experience, nginx is more like a Lotus Elise.

  3. Jason Grigsby says:

    I like my Hyundai Elantra GT. :-p

  4. Lyza Gardner says:

    James,

    How about a Hyundai Genesis coupe? That sort of walks the pointy boundary of workaday mundanity and vroom. Elises are so…standoffish*. If you squint hard and try to forget that the Genesis is a Hyundai, you just saved a few tens of kilobucks.

    * Don’t get me wrong. I want one.

  5. John Keith says:

    Well I may have to take exception to the “Elises are so…standoffish” point. I do agree that nginx is a sporty little server though.

  6. Hi Jason,
    You mention the OpenWave browser :-) We don’t run into it that much in Europe any more (I actually thought it was no longer under development…which would be nice) and I hate testing on emulators if I can help it! Can you suggest a good/typical test device that’s available in the US?

  7. It might not be very clean, but in your templates.php you can create a phptemplate_preprocess_page(&$vars) function and access the CSS and JS in $vars and clean it up at your will.

    I did it when implementing the Nokia theme for Drupal 6: http://drupalcode.org/viewvc/drupal/contributions/themes/nokia_mobile/template.php?revision=1.6&view=markup

  8. Lyza Gardner says:

    Hi Stephanie,

    Ah, yes, OpenWave. A number of low- to mid-priced Samsungs still use OpenWave, for example, the A177 (Magnet) series and the A777 series. In the US, AT&T certainly carries a few. We also have an older Samsung SGH-A237 flip feature phone in the office–it was practically free from the used mobile phone store we occasionally frequent (and works with an AT&T SIM).

    As to what’s going on with OpenWave currently, that’s a bit more of a serpentine situation. They were bought by PurpleLabs in 2008, and the browser was re-branded Myriad. Purple claims that the next variant will be WebKit. I haven’t been paying particularly close attention to the specifics :).

  9. Lyza Gardner says:

    Hi Andrea,

    Yep, that’s what I do. I have a template_preprocess_page(&$vars) function for my mobile theme (which gets picked up by all of the sub themes) that pulls items out of the CSS and JS arrays that are not explicitly allowed.

    I just wish I could do it in a module. It feels inelegant. This week I built out module-based control, at least, of what the allowed/disallowed JS and CSS should be.

  10. Thanks Lyza!
    As I guessed, there’s no sign of those devices (new or used) anywhere on this continent. Will see if I can find an unlocked one on EBay :-p

  11. perusio says:

    Why not dispense with Apache altogether and just use nginx with FastCGI?

    Have you heard of Panels Everywhere? Gives *full* control of the Drupal rendering pipeline. It’s a smart choice for Mobile stuff. You just need to write a context plugin that does (mobile) device detection.

  12. John Keith says:

    Hi @perusio!

    You’re right about ditching apache and going full to nginx as being a smart option. We originally deployed on apache, using the preform MPM, because that’s the default environment that’s provisioned for us in our standard hosting setup. We’re pretty good at optimizing that particular environment, but in this case we evolved the approach to 1) serve static content from nginx to off-load apache, 2) to immediately release apache processes for subsequent requests, and 3) to provide a means to load balance with multiple server instances. With a bit more time we may very well remove apache entirely from this particular mix, though we are living under some deployment constraints that make that difficult at the moment. We are also evaluating the apache worker MPM because, well, we like investigating and thinking about this stuff.

    I’m not the designer here, so I can’t speak to the Panels Everywhere idea. In general I think it takes time and energy for frameworks and programming models to be adopted. I hadn’t come across this before so thanks for the pointer.