Large corproations fudge their numbers?
Shocking. Just shocking.
Samsung has reportedly been cheating in benchmark tests, artificially boosting the scores of its latest and greatest system-on-chip, the Exynos 5 Octa, on those performance-ranking number generators so beloved by reviewers and product evaluators. "Oh hell Samsung, shame on you!" wrote a Beyond3D forum member in a posting on …
Shocking. Just shocking.
So the bottom line is this,
You're not getting the performance you are paying for
You're not getting the battery life you have been promised
Nothing new really is there
Next you'll be telling me I can't expect the achieve the mpg figures in my car that are listed in the brochure!
And absolutely no-one in any business has, of course, "positively represented" their relevant stats to the General Public.
Getting caught out is embarassing, but really is this news?
They get caught with their hand in the cookie jar and it just okay - they must have been hungry. What treatment would Apple have got if this was reversed??
Of course this is news - how else do you expect to enforce a level playing field, if not by journalists reporting when companies cheat? What do you think qualifies as news, simply a new shiny thing?
"What treatment would Apple have got if this was reversed??"
Well for a start the headline would, for sure, not have been represented as a question.
And for seconds they wouldn't have had The Register try to re-level the playing field already tipped so underhandedly in their favour with a statement like;
"And if they're cheating, there's a fair chance that others are, as well. But Samsung got caught."
Third, they would have got a "journalist" deliberately designated with a trolling brief to write the article and use only childishly disparaging language throughout, with copious references to "fanbois" (Jasper Hamill, before him Anna Leach)
Might I suggest The Register adjust their strap-line when publishing articles on Samsung to "licking the arse that shits on IT"
My WiFi Access Point has a 400MHz processor in it.
I thought advanced phones would have a faster chip inside them.
The processors are 1.2GHz. 480Mhz is the GPU speed.
Oh, and clock speed is not the sole predictor of performance. The processor architecture and design have a great deal of impact.
Clock speed comparisons across different architectures are completely meaningless.
Mhz is just the oscillator speed. It's how much a CPU can do per cycle that matters.
Just like in a car where CC doesn't really matter, it is power to weight ratio.
You do know their all 64bit right.
"You do know their all 64bit right."
Top marks for ironic handle and use of icon though.
No, actually all the current generation of ARMs are 32bit. Not that this has anything to do with comparing CPU performance by clock speed (several very different ARM designs in current use as well as independently designed ARM compatible cores.) It has even less to do with comparing CPU and GPU perf/clock. GPUs are wide -- often very wide -- SIMD machines that get a lot of work done in a single clock.
While Samsung's cheating is inexcusable, I hope the people running these benchmarks learn something from this and expand their repertoire a bit from choosing a handful of stupid benchmarks that don't measure anything useful versus what a person uses their phone. These type of toy benchmarks were mostly exterminated from articles about PCs about 10 years ago, but you still see people referring to crap like AnTutu, Geekbench and even Dhrystone for chrissakes.
Maybe there aren't any good mobile benchmarks out there, hopefully someone will fix that.
"These type of toy benchmarks were mostly exterminated from articles about PCs about 10 years ago..."
You must be reading VERY different webpages to me! Tomshardware, for example is often covered with them.
Time to bring back the Sieve of Eratosthenes from Byte and the 8 benchmark tests from the early days of Personal Computer World!
Or even the list in ZD publications in the era of Whetstone and Dhrystone that included such gems as:
slyandthefamilystone - a measure of how funky your PC is
rollingstone - a measure of how well your aging PC can perform compared to modern units
>Tomshardware, for example is often covered with them [synthetic benchmarks]
Yeah, But Tom's put their synthetic benchmarks in context, and always alongside 'real-world' tasks.... so if they are testing Workstation GPUs, for example, they run tasks in a variety of CAD and transcoding applications. If they are testing CPUs, they might run standard tasks in Photoshop and other productivity apps. Gaming hardware is tested on popular titles such as Crysis and Skyrim, since different games tax CPUs and GPUs differently. Seems reasonable.
The benchmark should specifically set the Mhz/Ghz on the CPU and GPU to stop this being bodged.
Agreed! And they usually do!
But his statement is still plainly wrong.
Ah, but are they really cheating? The handset is actually running the benchmark, and if the battery life is worsened while benchmarks are being run, then surely it is a failure of the individual performing the tests that they assume stupid things about the linearity of current draw under varied conditions.
"Cheating" would be if the system didn't do the work, e.g. by optimizing out routines that it had been told weren't important to the benchmark.
All this is is a system that is optimized to run those benchmarks. Whoopie. If that really matters to you, is there anything fundamentally preventing you from applying those same optimizations to whatever app you care about (i.e. stuff the name of the app into the tweak file)?
Cheating, what you mean like BT Broadband does with speedtests? Shouldn't even be surprising theres a lot more of this goes on than people realise.
In the interest of fairness I'd also like to point out that Virgin Media have been suspected of the same. Brilliant speed test results but try to stream a movie - no chance. A pity it is so hard to find out the truth in that market though.
But this depends on the links your ISP has, how utilised they are, quality of the exchange etc...
Unlike iCultists, I'm disinclined to defend bad behaviour, or "evangelize" for any faceless corporation, for that matter.
So that is why your comment disparages the customers of another company instead of making any statement about the unethical and possibly illegal (Unfair Commercial Practices Directive may stretch this far and could it even constitute fraud) behaviour of Samsung. Well that's OK then because you didn't defend them.
This is an interesting idea. However, Samsung are not *that* stupid...
What Samsung did was to trick the reviewers and it was the reviewers who outright "lied" to the public, in part by not actually checking the "facts" of their findings. Will these reviewers be punished? I doubt that very much.
If Samsung were stupid enough to actually refer to those benchmarks in their advertising materials and thus directly deluded consumers, that would merit punishment. Otherwise, they are safe.
Let's hope this story will actually make the reviewers more wary and less prone to praise whoever vendor gives them new gadgets to play with.
Here's a li'l fairness v. bias test we suggest you might find personally illuminating. Read the story above one more time, except each time you see the word "Samsung", substitute "Apple".
Then ask yourself: "Is my response any different?"
With Samsung, my response is "Poor Samsung got caught doing what all major Android manufacturers are doing these days, because their market is saturated with people who take these benchmarks seriously. Stupid but understandable."
With Apple, my response would be "WTF Apple? Your target customers don't read benchmarks! They come to you for shiny and cool. Why'd you waste resources trying to game a system you don't depend on!?"
How many 'major' android manufacturers are still out there?
They didn't bother to disassemble the code? Sigh. Kids today...
A defective CCI-400, or Cache Coherency Interconnect, which means the 5410 can only operate in "Cluster Migration" mode, which is big.LITTLE's least useful mode of operation.
Thanks to the broken CCI, the 5410 will only ever be running all A15s at once, or all A7s, but never a mixture of the two cores ("CPU Migration" mode), and when the 5410 migrates all of the tasks from one cluster to the other it has to flush all of the caches to main memory resulting in a significant power and performance penalty.
Basically, the 5410 is a bit of a cluster f**k, pardon the pun, and hopefully the 5420 has allowed Samsung to correct this glaring mistake. I wonder if they rushed the 5410 to market to meet the schedule of the international Galaxy S4 in which it is used (avoid the international GS4!)
I'm a consumer. Well - an ex mobile developer that used to care about this profoundly. No more.
There isn't all that much difference between Galaxy S2 and 3 or JesusPhone 4S and 5 to warrant bothering at this level of detail. Whoever is 'benchmarking' phones these days is probably a rather misguided effort. Nobody cares. It has been a PIII-733 with a GeForce 1 card by default for at least 2 years now.
Intel benchmarks have been inflated for decades but only in recent years have reviewer discovered and confirmed this conspiracy to dupe the gullible. If you want to see how Intel and AMD processors actually perform, test them with real applications not benchmarks that look at what processor brand is being tested.
I think it's only men who would be shocked by this. Women automatically take about 30% off any measurement given to them in respect of performance or size.
I could agree more
I totally agree.
I generally don't mind certain Americanisms, the spelling is just a fun difference to make note of; the terms used for a few things are different, and that's OK.
But at literally making no logical sense in English at all is where the line has to be drawn.
You sure do done got that nail thar hit right upon its head, yessir.
While we're at it... 'Do you got xxxxx'
For %$£&*%$'s sake that's PAST TENSE. It's do you HAVE xxxx, not 'Do you "did acquire" xxxx'.
" is where the line has to be drawn."
do you mean, "is where the line ought to be drawn."?
Apple doesn't sell its products on specs, but "user experience", so it wouldn't bother to fake the benchmarks.
Surely an Apple benchmark would olny be really relevant to another Apple product unless anything else uses the same chips+architecture ?
And all Appleists need anyway is to be told it's faster so yep, no point benchmarking.
Is that sarcasm? Because it's hard to tell.
Years ago I remember hearing how one now-gone computer manufacturer had a tweak in their compiler which was able to recognise the code for a common benchmark (SPECmark?) and, when it found it was compiling that code, it inserted an optimised block of machine code that could just all fit in the processor cache. It flew on that benchmark! Cheating or clever optimisation? You decide.
I suppose it really depends on whether that optimisation works in the main application the benchmark is based on - if there is one.
Optimising for Antutu for example, is a dirty trick as it's used as nothing but a performance benchmark.
Optimising for a benchmark based within an industry standard app, like Maya - If you use Maya, and the CPU is optimised for it, that's a good thing from the Maya users point of view.
Of course, if it only works in the benchmark within Maya and doesn't work for the main program in day to day tasks, then it's a dirty trick.
Context is king. But this instance *appears* to be dirty. It seems to be a workaround for the 'wait' as Exynos ramps up to it's big cores from the little ones (As flushing them out takes time that would skew the benchmark) but if that's what they were doing, they:
A: Should have said so
B: Should have implemented it in userland as a feature.
Otherwise it just looks underhand.
Hoping someone (ideally AT with their deep dives) can try to replicate this on other devices to see who else is doing this, be interesting to see how widespread it is...
Well, it is implemented in userland and managing DVFS frequencies from userland on Linux is a normal thing to do. Shockingly, you'll probably also find similar code in other 'optimised' android releases from most OEMs. You won't see such things in AOSP. Generally you'll require root to change DVFS governor or do similar things yourself.
Plus of course, there is a more general mechanism through power HAL where max frequencies are limited when the screen is off etc.
The fishy bits are that you can't achieve the same GPU clock yourself and that the increased thermal headroom is only allowed for specific apps. If both of those were time-limited features available on-demand to any app subject to device conditions, it would be all above board.
At least with Android being so open (don't laugh!) you can see what's going on. We have no clue what other OS' are doing in this regard.
I agree generally - although to clarify, when I said userland, what I really meant was 'exposed to the user as a 'turbo boost' feature. My bad :D
That GPU clock thing is definitely shady, though.
And it seems confirmation that HTC use the same technique is already out there if anyone looks.
I suspect all the Android OEMs will be doing something like this for their optimised ROMs, DVFS is not perfect and a lot of the mobile benchmarks have an execution model which is just about the worst case for a usage-based frequency monitor - sleep for a while, wake up and calculate for a short time, go back to sleep. Repeat for each sub-test.
Shitty benchmarks given too much weight by shitty reviewers leads to shitty optimisations.