Friday, October 4, 2013

Nerd News: Anandtechs's "They're (Almost) All Dirty: The State of Cheating in Android Benchmarks"

Really great article at Anandtech about the "cheating" going on in Android benchmarks.

Covers the details of this mess, and as Anand has said repeatedly this has happened before, he has seen it before.

But the key aspect, IMVHO, was not directly stated in the article, though it is in the comments by Geekfool & Anand's response source:

geekfool - Wednesday, October 02, 2013 - link

It seems like these cheats are an admission that that frequency scale-up under load happens too slowly, and that users will never get the speed they paid for because the battery / thermal dissipation can't support it. I would like a follow up that checks if the governor differences are noticeable to users as lag, dropped frames, etc.
The article touched on this slightly when mentioning the difference between the Nexus 4 and its LG branded equivalent. It seems especially likely to be noticeable with the A7 / A15 split on the Exynos

Anand Lal Shimpi - Wednesday, October 02, 2013 - link

Bingo! I had a whole section about the embarrassment that is software DVFS before culling it to keep the whole thing manageable.

I think that comment and response covers the meat of the issue, also explains why iOS with lower spec hardware tends to equal or exceed performance of stock skinned Android devices with twice the cores. 

For more on this issue comparing Moto X (dual core Android) to Samsung S4 (Quad Core Android) see

Anand & Brian cover this article and more in podcast that just came out few minutes before I started writing this blog post.


No comments:

Post a Comment