Mobile Device Detection Results
In a previous post I presented the performance numbers for mobile device detection using WURFL and Device Atlas. Our numbers were generated on a data set consisting of 1,572 unique user agents we collected as part of our mobile device concurrency test. At the time I was looking for information about the quality of our results – did we have accurate classification of mobile devices for the purposes of our test? I unintentionally wandered into performance for a while, but now I’m back onto the quality thing.
Here, once again, are the results for processing 1,572 unique user agents:
|WURFL Old API||1082||711||861|
|WURFL New API||20.8||1090||482|
|Mobile Device Detect||1.3||684||888|
As you can see, there are obviously some differences between results for each detection method. Interestingly, both the old and new WURFL API use the same device database (wurfl.xml file), but yield considerably different results. So, what’s going on here? Is there a clear quality winner in the group?
For the time being, we are using the old WURFL API as our classifier. The new WURFL API is giving us a lot of false positives – many different user agents are mapping to the wireless device “amoi_e72_ver1,” which could be some kind of fall-through condition. It’s possible that I’m doing something wrong, but the API usage is pretty simple: instantiate a class and call a method with the user agent as a parameter.
Device Atlas is interesting to me and it’s quite possible we’ll be shifting to that for device detection. It’s certainly much faster than WURFL in our environment. While there are a few misses, there is really just one issue I’m seeing, and that may not be a real issue at all. The Opera Mini user agents are not being classified as mobile. Is this because we’re talking to a transcoder? Or is it a mistaken classification? I’m not sure, but the other methods all classify these user agents as mobile devices. Take this user agent for example. Should this be classified as a mobile device? Device Atlas says no; WURFL says yes.
Opera/9.50 (J2ME/MIDP; Opera Mini/4.0.10406/298; U; de)
The mobile device detection script actually does a pretty good job. Rather than use a comprehensive user agent database, this script looks for specific, known string fragments in the user agent string. That’s also a bit of its undoing though. For example, I’m seeing a bunch of false positives for user agents containing the word “Java.” I think these false positives could be eliminated by also checking the HTTP_ACCEPT header for mobile content types.
So the implication here is that you need to be aware that device detection is not guaranteed to be 100% accurate. User agent strings are highly variable and non-standardized, so there’s a bit of artistry here.