Feeds

back to article Egad! Could Samsung be CHEATING in Galaxy benchmark tests?

Samsung has reportedly been cheating in benchmark tests, artificially boosting the scores of its latest and greatest system-on-chip, the Exynos 5 Octa, on those performance-ranking number generators so beloved by reviewers and product evaluators. "Oh hell Samsung, shame on you!" wrote a Beyond3D forum member in a posting on one …

COMMENTS

This topic is closed for new posts.

Page:

Silver badge

Large corproations fudge their numbers?

Shocking. Just shocking.

Not.

4
1
Silver badge
IT Angle

Re: Large corproations fudge their numbers?

So the bottom line is this,

You're not getting the performance you are paying for

You're not getting the battery life you have been promised

Nothing new really is there

3
5

Re: Large corproations fudge their numbers?

Next you'll be telling me I can't expect the achieve the mpg figures in my car that are listed in the brochure!

0
1
Silver badge

Shocking....

And absolutely no-one in any business has, of course, "positively represented" their relevant stats to the General Public.

Getting caught out is embarassing, but really is this news?

1
8
Anonymous Coward

Re: Shocking....

They get caught with their hand in the cookie jar and it just okay - they must have been hungry. What treatment would Apple have got if this was reversed??

11
3

Re: Shocking....

Of course this is news - how else do you expect to enforce a level playing field, if not by journalists reporting when companies cheat? What do you think qualifies as news, simply a new shiny thing?

18
0
Anonymous Coward

Re: Shocking....

"What treatment would Apple have got if this was reversed??"

Well for a start the headline would, for sure, not have been represented as a question.

And for seconds they wouldn't have had The Register try to re-level the playing field already tipped so underhandedly in their favour with a statement like;

"And if they're cheating, there's a fair chance that others are, as well. But Samsung got caught."

Third, they would have got a "journalist" deliberately designated with a trolling brief to write the article and use only childishly disparaging language throughout, with copious references to "fanbois" (Jasper Hamill, before him Anna Leach)

Might I suggest The Register adjust their strap-line when publishing articles on Samsung to "licking the arse that shits on IT"

14
13

This post has been deleted by its author

Anonymous Coward

My WiFi Access Point has a 400MHz processor in it.

I thought advanced phones would have a faster chip inside them.

0
14
Silver badge
Facepalm

They do

The processors are 1.2GHz. 480Mhz is the GPU speed.

Oh, and clock speed is not the sole predictor of performance. The processor architecture and design have a great deal of impact.

14
0
Anonymous Coward

Not just "not the sole predictor"

Clock speed comparisons across different architectures are completely meaningless.

4
0
Gold badge

Mhz is just the oscillator speed. It's how much a CPU can do per cycle that matters.

Just like in a car where CC doesn't really matter, it is power to weight ratio.

0
0
Facepalm

Re: They do

You do know their all 64bit right.

0
3
Anonymous Coward

Re: They do

"You do know their all 64bit right."

*they're.

Top marks for ironic handle and use of icon though.

0
0
Anonymous Coward

Re: They do

No, actually all the current generation of ARMs are 32bit. Not that this has anything to do with comparing CPU performance by clock speed (several very different ARM designs in current use as well as independently designed ARM compatible cores.) It has even less to do with comparing CPU and GPU perf/clock. GPUs are wide -- often very wide -- SIMD machines that get a lot of work done in a single clock.

1
0
Silver badge

Proper mobile benchmarking

While Samsung's cheating is inexcusable, I hope the people running these benchmarks learn something from this and expand their repertoire a bit from choosing a handful of stupid benchmarks that don't measure anything useful versus what a person uses their phone. These type of toy benchmarks were mostly exterminated from articles about PCs about 10 years ago, but you still see people referring to crap like AnTutu, Geekbench and even Dhrystone for chrissakes.

Maybe there aren't any good mobile benchmarks out there, hopefully someone will fix that.

16
0

Re: Proper mobile benchmarking

"These type of toy benchmarks were mostly exterminated from articles about PCs about 10 years ago..."

You must be reading VERY different webpages to me! Tomshardware, for example is often covered with them.

2
3
Silver badge
Happy

Re: Proper mobile benchmarking

Time to bring back the Sieve of Eratosthenes from Byte and the 8 benchmark tests from the early days of Personal Computer World!

2
0
Silver badge

Re: Proper mobile benchmarking

Or even the list in ZD publications in the era of Whetstone and Dhrystone that included such gems as:

slyandthefamilystone - a measure of how funky your PC is

rollingstone - a measure of how well your aging PC can perform compared to modern units

1
0
Silver badge

Re: Proper mobile benchmarking

>Tomshardware, for example is often covered with them [synthetic benchmarks]

Yeah, But Tom's put their synthetic benchmarks in context, and always alongside 'real-world' tasks.... so if they are testing Workstation GPUs, for example, they run tasks in a variety of CAD and transcoding applications. If they are testing CPUs, they might run standard tasks in Photoshop and other productivity apps. Gaming hardware is tested on popular titles such as Crysis and Skyrim, since different games tax CPUs and GPUs differently. Seems reasonable.

3
0
Gold badge

Re: Proper mobile benchmarking

The benchmark should specifically set the Mhz/Ghz on the CPU and GPU to stop this being bodged.

0
0

Re: Proper mobile benchmarking

Agreed! And they usually do!

But his statement is still plainly wrong.

0
0
WTF?

Re: Proper mobile benchmarking

Ah, but are they really cheating? The handset is actually running the benchmark, and if the battery life is worsened while benchmarks are being run, then surely it is a failure of the individual performing the tests that they assume stupid things about the linearity of current draw under varied conditions.

"Cheating" would be if the system didn't do the work, e.g. by optimizing out routines that it had been told weren't important to the benchmark.

All this is is a system that is optimized to run those benchmarks. Whoopie. If that really matters to you, is there anything fundamentally preventing you from applying those same optimizations to whatever app you care about (i.e. stuff the name of the app into the tweak file)?

0
1

Cheating, what you mean like BT Broadband does with speedtests? Shouldn't even be surprising theres a lot more of this goes on than people realise.

6
0

In the interest of fairness I'd also like to point out that Virgin Media have been suspected of the same. Brilliant speed test results but try to stream a movie - no chance. A pity it is so hard to find out the truth in that market though.

6
0
Bronze badge
Unhappy

But this depends on the links your ISP has, how utilised they are, quality of the exchange etc...

0
0

re: bootnote

Unlike iCultists, I'm disinclined to defend bad behaviour, or "evangelize" for any faceless corporation, for that matter.

7
12
WTF?

Re: re: bootnote

So that is why your comment disparages the customers of another company instead of making any statement about the unethical and possibly illegal (Unfair Commercial Practices Directive may stretch this far and could it even constitute fraud) behaviour of Samsung. Well that's OK then because you didn't defend them.

9
0
Thumb Up

Re: Unfair Commercial Practices Directive

This is an interesting idea. However, Samsung are not *that* stupid...

What Samsung did was to trick the reviewers and it was the reviewers who outright "lied" to the public, in part by not actually checking the "facts" of their findings. Will these reviewers be punished? I doubt that very much.

If Samsung were stupid enough to actually refer to those benchmarks in their advertising materials and thus directly deluded consumers, that would merit punishment. Otherwise, they are safe.

Let's hope this story will actually make the reviewers more wary and less prone to praise whoever vendor gives them new gadgets to play with.

1
1
Silver badge
Holmes

Of course it is.

Here's a li'l fairness v. bias test we suggest you might find personally illuminating. Read the story above one more time, except each time you see the word "Samsung", substitute "Apple".

Then ask yourself: "Is my response any different?"

Sure is.

With Samsung, my response is "Poor Samsung got caught doing what all major Android manufacturers are doing these days, because their market is saturated with people who take these benchmarks seriously. Stupid but understandable."

With Apple, my response would be "WTF Apple? Your target customers don't read benchmarks! They come to you for shiny and cool. Why'd you waste resources trying to game a system you don't depend on!?"

21
1

Re: Of course it is.

How many 'major' android manufacturers are still out there?

3
0
Anonymous Coward

WTF?

They didn't bother to disassemble the code? Sigh. Kids today...

10
0
Facepalm

Something else the 5410 contains

A defective CCI-400, or Cache Coherency Interconnect, which means the 5410 can only operate in "Cluster Migration" mode, which is big.LITTLE's least useful mode of operation.

Thanks to the broken CCI, the 5410 will only ever be running all A15s at once, or all A7s, but never a mixture of the two cores ("CPU Migration" mode), and when the 5410 migrates all of the tasks from one cluster to the other it has to flush all of the caches to main memory resulting in a significant power and performance penalty.

See Anandtech.

Basically, the 5410 is a bit of a cluster f**k, pardon the pun, and hopefully the 5420 has allowed Samsung to correct this glaring mistake. I wonder if they rushed the 5410 to market to meet the schedule of the international Galaxy S4 in which it is used (avoid the international GS4!)

6
1

Meh

I'm a consumer. Well - an ex mobile developer that used to care about this profoundly. No more.

There isn't all that much difference between Galaxy S2 and 3 or JesusPhone 4S and 5 to warrant bothering at this level of detail. Whoever is 'benchmarking' phones these days is probably a rather misguided effort. Nobody cares. It has been a PIII-733 with a GeForce 1 card by default for at least 2 years now.

4
0
Anonymous Coward

This is nothing new

Intel benchmarks have been inflated for decades but only in recent years have reviewer discovered and confirmed this conspiracy to dupe the gullible. If you want to see how Intel and AMD processors actually perform, test them with real applications not benchmarks that look at what processor brand is being tested.

5
0
xyz
Paris Hilton

Re: This is nothing new

I think it's only men who would be shocked by this. Women automatically take about 30% off any measurement given to them in respect of performance or size.

4
1

This post has been deleted by a moderator

I could agree more

3
5
Silver badge

I totally agree.

I generally don't mind certain Americanisms, the spelling is just a fun difference to make note of; the terms used for a few things are different, and that's OK.

But at literally making no logical sense in English at all is where the line has to be drawn.

6
1
g e
Silver badge
Headmaster

You sure do done got that nail thar hit right upon its head, yessir.

While we're at it... 'Do you got xxxxx'

For %$£&*%$'s sake that's PAST TENSE. It's do you HAVE xxxx, not 'Do you "did acquire" xxxx'.

2
0
Anonymous Coward

" is where the line has to be drawn."

do you mean, "is where the line ought to be drawn."?

1
2
Bronze badge

Apple

Apple doesn't sell its products on specs, but "user experience", so it wouldn't bother to fake the benchmarks.

7
2
g e
Silver badge

Re: Apple

Surely an Apple benchmark would olny be really relevant to another Apple product unless anything else uses the same chips+architecture ?

And all Appleists need anyway is to be told it's faster so yep, no point benchmarking.

2
2
Trollface

Re: Apple

Is that sarcasm? Because it's hard to tell.

1
0
Anonymous Coward

Re: Apple @plrndl

"prndl" surely?

0
0
Silver badge

Nothing really new here

Years ago I remember hearing how one now-gone computer manufacturer had a tweak in their compiler which was able to recognise the code for a common benchmark (SPECmark?) and, when it found it was compiling that code, it inserted an optimised block of machine code that could just all fit in the processor cache. It flew on that benchmark! Cheating or clever optimisation? You decide.

3
0
Silver badge

Re: Nothing really new here

I suppose it really depends on whether that optimisation works in the main application the benchmark is based on - if there is one.

Optimising for Antutu for example, is a dirty trick as it's used as nothing but a performance benchmark.

Optimising for a benchmark based within an industry standard app, like Maya - If you use Maya, and the CPU is optimised for it, that's a good thing from the Maya users point of view.

Of course, if it only works in the benchmark within Maya and doesn't work for the main program in day to day tasks, then it's a dirty trick.

Context is king. But this instance *appears* to be dirty. It seems to be a workaround for the 'wait' as Exynos ramps up to it's big cores from the little ones (As flushing them out takes time that would skew the benchmark) but if that's what they were doing, they:

A: Should have said so

B: Should have implemented it in userland as a feature.

Otherwise it just looks underhand.

All IMHO.

Hoping someone (ideally AT with their deep dives) can try to replicate this on other devices to see who else is doing this, be interesting to see how widespread it is...

Steven R

2
0
Anonymous Coward

Re: Nothing really new here

Well, it is implemented in userland and managing DVFS frequencies from userland on Linux is a normal thing to do. Shockingly, you'll probably also find similar code in other 'optimised' android releases from most OEMs. You won't see such things in AOSP. Generally you'll require root to change DVFS governor or do similar things yourself.

Plus of course, there is a more general mechanism through power HAL where max frequencies are limited when the screen is off etc.

The fishy bits are that you can't achieve the same GPU clock yourself and that the increased thermal headroom is only allowed for specific apps. If both of those were time-limited features available on-demand to any app subject to device conditions, it would be all above board.

At least with Android being so open (don't laugh!) you can see what's going on. We have no clue what other OS' are doing in this regard.

0
0
Silver badge

Re: Nothing really new here

I agree generally - although to clarify, when I said userland, what I really meant was 'exposed to the user as a 'turbo boost' feature. My bad :D

That GPU clock thing is definitely shady, though.

Steven R

0
0
Anonymous Coward

Re: Nothing really new here

And it seems confirmation that HTC use the same technique is already out there if anyone looks.

http://forum.xda-developers.com/showpost.php?p=36033515&postcount=744

I suspect all the Android OEMs will be doing something like this for their optimised ROMs, DVFS is not perfect and a lot of the mobile benchmarks have an execution model which is just about the worst case for a usage-based frequency monitor - sleep for a while, wake up and calculate for a short time, go back to sleep. Repeat for each sub-test.

Shitty benchmarks given too much weight by shitty reviewers leads to shitty optimisations.

0
0

Page:

This topic is closed for new posts.