Egad! Could Samsung be cheating in Galaxy benchmark tests? Samsung has reportedly been cheating in benchmark tests, artificially boosting the scores of its latest and greatest system-on-chip, the Exynos 5 Octa, on those performance-ranking number generators so beloved by reviewers and product evaluators. "Oh hell Samsung, shame on you!" wrote a Beyond3D forum member in a posting on …
"What treatment would Apple have got if this was reversed??"
Well for a start the headline would, for sure, not have been represented as a question.
And for seconds they wouldn't have had The Register try to re-level the playing field already tipped so underhandedly in their favour with a statement like;
"And if they're cheating, there's a fair chance that others are, as well. But Samsung got caught."
Third, they would have got a "journalist" deliberately designated with a trolling brief to write the article and use only childishly disparaging language throughout, with copious references to "fanbois" (Jasper Hamill, before him Anna Leach)
Might I suggest The Register adjust their strap-line when publishing articles on Samsung to "licking the arse that shits on IT"
This post has been deleted by its author
No, actually all the current generation of ARMs are 32bit. Not that this has anything to do with comparing CPU performance by clock speed (several very different ARM designs in current use as well as independently designed ARM compatible cores.) It has even less to do with comparing CPU and GPU perf/clock. GPUs are wide -- often very wide -- SIMD machines that get a lot of work done in a single clock.
While Samsung's cheating is inexcusable, I hope the people running these benchmarks learn something from this and expand their repertoire a bit from choosing a handful of stupid benchmarks that don't measure anything useful versus what a person uses their phone. These type of toy benchmarks were mostly exterminated from articles about PCs about 10 years ago, but you still see people referring to crap like AnTutu, Geekbench and even Dhrystone for chrissakes.
Maybe there aren't any good mobile benchmarks out there, hopefully someone will fix that.
>Tomshardware, for example is often covered with them [synthetic benchmarks]
Yeah, But Tom's put their synthetic benchmarks in context, and always alongside 'real-world' tasks.... so if they are testing Workstation GPUs, for example, they run tasks in a variety of CAD and transcoding applications. If they are testing CPUs, they might run standard tasks in Photoshop and other productivity apps. Gaming hardware is tested on popular titles such as Crysis and Skyrim, since different games tax CPUs and GPUs differently. Seems reasonable.
Ah, but are they really cheating? The handset is actually running the benchmark, and if the battery life is worsened while benchmarks are being run, then surely it is a failure of the individual performing the tests that they assume stupid things about the linearity of current draw under varied conditions.
"Cheating" would be if the system didn't do the work, e.g. by optimizing out routines that it had been told weren't important to the benchmark.
All this is is a system that is optimized to run those benchmarks. Whoopie. If that really matters to you, is there anything fundamentally preventing you from applying those same optimizations to whatever app you care about (i.e. stuff the name of the app into the tweak file)?
So that is why your comment disparages the customers of another company instead of making any statement about the unethical and possibly illegal (Unfair Commercial Practices Directive may stretch this far and could it even constitute fraud) behaviour of Samsung. Well that's OK then because you didn't defend them.
This is an interesting idea. However, Samsung are not *that* stupid...
What Samsung did was to trick the reviewers and it was the reviewers who outright "lied" to the public, in part by not actually checking the "facts" of their findings. Will these reviewers be punished? I doubt that very much.
If Samsung were stupid enough to actually refer to those benchmarks in their advertising materials and thus directly deluded consumers, that would merit punishment. Otherwise, they are safe.
Let's hope this story will actually make the reviewers more wary and less prone to praise whoever vendor gives them new gadgets to play with.
Here's a li'l fairness v. bias test we suggest you might find personally illuminating. Read the story above one more time, except each time you see the word "Samsung", substitute "Apple".
Then ask yourself: "Is my response any different?"
Sure is.
With Samsung, my response is "Poor Samsung got caught doing what all major Android manufacturers are doing these days, because their market is saturated with people who take these benchmarks seriously. Stupid but understandable."
With Apple, my response would be "WTF Apple? Your target customers don't read benchmarks! They come to you for shiny and cool. Why'd you waste resources trying to game a system you don't depend on!?"
A defective CCI-400, or Cache Coherency Interconnect, which means the 5410 can only operate in "Cluster Migration" mode, which is big.LITTLE's least useful mode of operation.
Thanks to the broken CCI, the 5410 will only ever be running all A15s at once, or all A7s, but never a mixture of the two cores ("CPU Migration" mode), and when the 5410 migrates all of the tasks from one cluster to the other it has to flush all of the caches to main memory resulting in a significant power and performance penalty.
See Anandtech.
Basically, the 5410 is a bit of a cluster f**k, pardon the pun, and hopefully the 5420 has allowed Samsung to correct this glaring mistake. I wonder if they rushed the 5410 to market to meet the schedule of the international Galaxy S4 in which it is used (avoid the international GS4!)
I'm a consumer. Well - an ex mobile developer that used to care about this profoundly. No more.
There isn't all that much difference between Galaxy S2 and 3 or JesusPhone 4S and 5 to warrant bothering at this level of detail. Whoever is 'benchmarking' phones these days is probably a rather misguided effort. Nobody cares. It has been a PIII-733 with a GeForce 1 card by default for at least 2 years now.
Intel benchmarks have been inflated for decades but only in recent years have reviewer discovered and confirmed this conspiracy to dupe the gullible. If you want to see how Intel and AMD processors actually perform, test them with real applications not benchmarks that look at what processor brand is being tested.
Years ago I remember hearing how one now-gone computer manufacturer had a tweak in their compiler which was able to recognise the code for a common benchmark (SPECmark?) and, when it found it was compiling that code, it inserted an optimised block of machine code that could just all fit in the processor cache. It flew on that benchmark! Cheating or clever optimisation? You decide.
I suppose it really depends on whether that optimisation works in the main application the benchmark is based on - if there is one.
Optimising for Antutu for example, is a dirty trick as it's used as nothing but a performance benchmark.
Optimising for a benchmark based within an industry standard app, like Maya - If you use Maya, and the CPU is optimised for it, that's a good thing from the Maya users point of view.
Of course, if it only works in the benchmark within Maya and doesn't work for the main program in day to day tasks, then it's a dirty trick.
Context is king. But this instance *appears* to be dirty. It seems to be a workaround for the 'wait' as Exynos ramps up to it's big cores from the little ones (As flushing them out takes time that would skew the benchmark) but if that's what they were doing, they:
A: Should have said so
B: Should have implemented it in userland as a feature.
Otherwise it just looks underhand.
All IMHO.
Hoping someone (ideally AT with their deep dives) can try to replicate this on other devices to see who else is doing this, be interesting to see how widespread it is...
Steven R
Well, it is implemented in userland and managing DVFS frequencies from userland on Linux is a normal thing to do. Shockingly, you'll probably also find similar code in other 'optimised' android releases from most OEMs. You won't see such things in AOSP. Generally you'll require root to change DVFS governor or do similar things yourself.
Plus of course, there is a more general mechanism through power HAL where max frequencies are limited when the screen is off etc.
The fishy bits are that you can't achieve the same GPU clock yourself and that the increased thermal headroom is only allowed for specific apps. If both of those were time-limited features available on-demand to any app subject to device conditions, it would be all above board.
At least with Android being so open (don't laugh!) you can see what's going on. We have no clue what other OS' are doing in this regard.
And it seems confirmation that HTC use the same technique is already out there if anyone looks.
http://forum.xda-developers.com/showpost.php?p=36033515&postcount=744
I suspect all the Android OEMs will be doing something like this for their optimised ROMs, DVFS is not perfect and a lot of the mobile benchmarks have an execution model which is just about the worst case for a usage-based frequency monitor - sleep for a while, wake up and calculate for a short time, go back to sleep. Repeat for each sub-test.
Shitty benchmarks given too much weight by shitty reviewers leads to shitty optimisations.
thanks for the bootnote, a fairly good test, if my moral benchmarking hasn't been rigged I'm relieved to think, that I would have felt no more amused / outraged, had the hero of this article been Apple.
BUT wait! - this test is flawed - if I substitute Samsung with Apple, I already know it's a just a test, and the real (...) are Samsung. So... perhaps my internal benchmarking is rigged with anti-Apple strings anyway! :)
I think, the true test of "are we biased against Apple" (hell, of course we are! ;) would be to run such article claiming it was Apple - and THEN telling people to compare how they felt, as the real culprit is Samsung, as people's reactions to Apple are, generally, more... emotional and strong, than of any other company. But then, oh dear, Apple response to such a "test" article would have been swift and brutal, with the whole Register probably taken down by lunchtime, and claim for 100 billion £ in damages lodged by end of business day...
p.s. can, we, in the wake of the findings, expect an industry-wide wave of "minor" firmware updates coming up in the next few days from all major and minor phone manufacturers? ;)
Yet, Samsung stays mum... that has to tell us something.
But you are correct. If Apple were caught to do something like this, that would enrage a lot of people -- for no other reason, than the fact that Apple does not have to make such tricks to sell their stuff. Nor do Samsung, but for some reason they don't seem to understand it.
Although if you did an analogy of a Formula One car.
If there were benchmarks for the engines and pre season every car was showing off that their new engine was the fastest ever. They might set up a benchmark that measures, power and torque. Now for the pre-season benchmark they would run it at maximum for the three minutes to do the test (end then bin the engine or resign it to the test car). However during the season they run the cars at slightly lower performance due to the fact the engine has to last for a few races.
This doesn't mean the engine can't run at that benchmarked speed, just that they choose not to as other factors need to be considered.
Similar to a benchmark, it is saying that the processor is capable of running at that high speed and can be software activated to do it - for instance if a specific piece of software needed it, however for day-to-day use they optimise it for other factors. They aren't inserting fake results into the benchmark or writing over memory addresses with different stats for instance.
If you look at the battery stats for a device and it says 4 days - they have obviously optimised very few if any applications running, and closed down any background data or unneeded radios. In real-life however YMMV.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
