back to article Samsung denies benchmark cheating, despite evidence

Samsung says that despite well-documented reports to the contrary, it is not cheating in benchmark testing of its smartphone processors. "The Galaxy Note 3 maximises its CPU/GPU frequencies when running features that demand substantial performance," Sammy told CNET UK. "This was not an attempt to exaggerate particular …

COMMENTS

This topic is closed for new posts.

Page:

  1. This post has been deleted by its author

    1. Steve Todd

      What, the benchmarks that run ONLY on Android (like AnTuTu) are optimised for iOS?

      These "optimisations" are part of the power management software stack, but somehow Samsung building (not designing) the A7 means that they have been incorporated in iOS? The article even pointed out that Intel and Qualcomm weren't involved in the cheating, even though their CPUs were being gamed.

    2. ThomH

      Per the Ars article, it's a software hack. Samsung's build of Android spots when popular performance tools are being run and turns everything up to 11. The device couldn't run like that ordinarily without significantly shortening its lifetime, both per charge cycle and in general.

      What Ars did was to create two versions of the benchmarking tool Geekbench — one that the system could identify as Geekbench and one that it couldn't. The code remained identical.

      When it was identifiable by the system as Geekbench, the code ran up to 20% faster.

    3. Anonymous Coward
      Anonymous Coward

      ARM design the core, Apple designs the rest, Samsung build it. It's not a Samsung design.

      1. Steve Todd

        @AC 6:54

        No, ARM don't design Apple's CPU cores for them. Apple have an ARM Architecture license that allows them to design their own ARM compatible cores, just as QUALCOMM do. The Swift core introduced in the A6 was completely their own work. The A5 was, as best we can figure out, a modified ARM Cortex A9. The A7 seems to be an enhanced Swift.

    4. LarsG

      There would have been a little more respect had they put their hands up and said,

      "Yes it is true, we do cheat, but so does everyone else".

  2. This post has been deleted by its author

  3. mickey mouse the fith

    Hmm, from what iv read, it should be possible to rename an apk to antutu or whatever and get it to run full pelt, or add your apk name to the decompiled java file samsung use to trigger this `feature` and recompile. Could be great for running demanding games a bit faster (until the phone overheats and dies obviously).

    Its not really cheating though, as technically the phone CAN run at that speed, they arnt falsifying data, just being a bit dishonest about how long the device would last at that speed.

    It would be funny if reviews started reporting the degraded battery life these tests would produce, Sammy would soon stop the practice if consumers thought the quoted battery life was a third of the actual figure.

    1. Steve Todd

      What do you mean it's not cheating? If your bank ATM was to tell you that you had £500 in your account when you asked, but if you need to take money out you've only got £450 then you'd damned well think that was cheating.

      Any attempt to artificially boost benchmark numbers in ways that normal apps have no access too is cheating.

      1. Voland's right hand Silver badge

        Quote :"What do you mean it's not cheating? If your bank ATM was to tell you that you had £500..."

        The bank has more like 250. If it is for all customers. If not less (as per current liquidity rules).

      2. Roland6 Silver badge

        Re: ATM

        No what Samsung are doing is the equivalent of the cash machine telling you it is out of cash, only for the next person in the queue to draw out £1000.

    2. Chad H.

      Its cheating because its running in an overdrive that is beyond the phones normal operation.. If Sammy wanted to they could run it at full speed all the time, but then noone would buy it cos the battery life would be shot.

      1. Anonymous Coward
        Anonymous Coward

        Its cheating because its running in an overdrive that is beyond the phones normal operation.. If Sammy wanted to they could run it at full speed all the time, but then noone would buy it cos the battery life would be shot.

        So they're cheating because their phone uses automated features to try and optimise battery life?

        REALLY?

    3. Steven Raith

      Just to back up Steve Todd...

      If my car leaned out it's fuel mixture to give me 60mpg (rather than it's normal 30 under mixed conditions) when at a MIRA test track (as determined by GPS), but nowhere else, that'd be cheating.

      If it also adjusted it's fuelling map for more agressive fuelling and removed the timing retardation in first and second gear* when it detected it was at the Bedford Autodrome (to be figured for a car mag), and nowhere else, that would also be cheating.

      MPG and 0-60/0-100 times mean little to the average consumer - most people don't hypermile, and most people don't dump the clutch at 4500rpm then bury the throttle to the carpet at traffic lights, either.

      But they use those figures to determine the capabilities of the car, particularly against other cars in the same sector which have ostensible been tested under the same conditions. It's just that they didn't adjust their maps based on thier location...

      Of course, cars don't do this. Unless they're Ferrari, who regularly provide specifically modified versions of their cars for testing, and burn journos who point it out. http://jalopnik.com/5760248/how-ferrari-spins

      To get back to the point, when you have a benchmark that is (rightly or wrongly) being used to determine a devices capabilities against other devices using the same benchmarks, be that combined platform performance of an OEM mobile device, or time around a race circuit, in ways that the consumer cannot easily acheive (if at all - the average consumer, or even prosumer, isn't going to know how to set up adjustable suspension for Donnington Park, nor how to rename their APKs to take advantage of higher thermal limits) then yes, it absolutely is cheating. because most arguments about 'figurable' devices (CPUs, GPUs, cars, etc) refer to benchmarks of some kind, which are supposed to be representative of performance, as evidence.

      Of course, all that will happen if tech rags like AT, et al, can't work around this (renaming the packages will only work for so long if the OEMs are serious about gaming the benches) is that benchmarks will becomes effectively meaningless.

      I'll be honest, I'm not sure if that's a good thing or a bad thing.

      But it's definitely shenanigans, and pretty stupid ones at that. All they respective OEMs would have to do is put a widget on that says 'Fast and Hot' or 'Game Mode' to cause the CPU to be ramped up and thermal limits to be raised for playing games, and boom, controversy over.

      Steven R

      *no, it's not traction control - it has that too, useless as it is - the car retards timing in first and second gear to 'protect it's driveshafts' and makes it feel anaemic when pulling away. It also knocks a solid second from the 0-60 time, making the car slower than it needs to be. Getting it removed, however, is costly, and frankly, it saves me accidentally lighting the tyres up on wet junctions so I'll leave it be methinks...

      PS: Yes, I've had a crap day and am ranting. Normal jocular service will resume at the weekend when I have a working toilet in the house.

      1. frank ly
        Happy

        @Steven Raith

        So, at the weekend it reall will be a crap day. Enjoy!

      2. itzman

        If my car leaned out it's fuel mixture to give me 60mpg ....

        But cars are exactly optimised to do well on 'standard tests'

        They are driven to an exact pattern.

        This is detected and the engines leaned out to make it look good,.

        1. Voland's right hand Silver badge

          Re: If my car leaned out it's fuel mixture to give me 60mpg ....

          Quote: But cars are exactly optimised to do well on 'standard tests'

          Indeed. I can think of at least one manufacturer that has been caught more than once doing it. A french one.

          1. Adam T

            Re: If my car leaned out it's fuel mixture to give me 60mpg ....

            The car analogy doesn't work here (and when will people realise it never does) because in order for it to apply, your car would have to perform better on specific roads. I don't live anywhere near White Sands.

            Samsung's cheat is to use a Plist file that's inaccessible to other applications. The argument 'optimised' falls apart when it's an optimisation available only to them.

            Now, if they made it so we could hit turbo mode in our apps then it would be a different story, but then we'd probably have more stories about batteries catching fire.

            Which makes me wonder, what would happen if you let one of these tests run and run...

        2. Anonymous Coward
          Anonymous Coward

          Re: If my car leaned out it's fuel mixture to give me 60mpg ....

          But is it ONLY benchmarking apps. How do we know there aren't other non benchmarking apps that benefit?

          Either way it's a storm in a teacup. Apple aren't angels either......

        3. Philip Lewis

          Re: If my car leaned out it's fuel mixture to give me 60mpg ....

          It has long been public knowledge that EMSs are optimised for the target market EPA measurements.

          This means that optimum fuel economy is achieved at exactly 50km/hr & 100km/hr, but not the posted 110 or 130 (depending on country) speed that everyone drives.

        4. Steven Raith

          Re: If my car leaned out it's fuel mixture to give me 60mpg ....

          Frank Ly - ta. It's a rented property but I reckon that doing a bit of DIY with me dad (Landlord knows us both and is cool with it) is one of those things worth knowing - floorboards, seals, some minor plumbing etc. So it'll be interesting, at worst.

          itzman (and the rest referring to this) - yes, I know that cars are mapped to work best on the economy run tests - the first gen Porsche Cayman R (allegedly) had a flat spot in it's map where it would barely fuel at all, just at around 50mph in top....that was never proven though. But my point was about being very specific about changing the way the engine works based on parameters that the driver cannot mess with - hence the reference to it being kicked off by GPS location ;-)

          You are all, of course, correct, on the MPG point. My car is rated at somewhere around 45mpg at 50mpg. I can get about 40 out of it (it's got 120k on it, natch) if I pootle at 50. But when you have 86% of torque from 1200rpm to 6500 rpm...er....you don't pootle much.

          My normal average MPG is about 25-30, as I spend an inordinate amount of time at WOT* (where safe to do so, natch) because WOT is so much fun :D

          Steven R

          *WOT - Wide Open Throttle - AKA brrrrmmrrrRAAAAAARRR-crackle-pop-bang-bbrrrrmmmrrrRRRAAARR, repeat till fourth gear....

      3. Anonymous Coward
        Anonymous Coward

        Of course, all that will happen if tech rags like AT, et al, can't work around this (renaming the packages will only work for so long if the OEMs are serious about gaming the benches) is that benchmarks will becomes effectively meaningless

        On El Reg? A site which regularly tells us stories about how one storage vendor or another is gaming configurations of storage arrays to achieve amazing and completely unrealistic benchmarks. Do feel free to read the comments of some of those 'reports' where we all get to witness shills trying to defend the gaming of benchmarks. Most benchmarks are meaningless, I've even seen corporate shills admit (on this site) that benchmarks are a game for marketing purposes.

      4. Moeluk

        Sorry to back everyone else up here....but yeah, the cars they use to produce the spec figures for cars are whats known as 'blueprinted', which means not only are they all leaned out on the test, but the engine in that car is designed exactly to spec, and all tolerances are much finer than they are in the production car.

    4. julianh72

      RE: Battery Life

      Yes, including battery life in the reporting of benchmark scores is the way to combat this practice - just include battery life as a weieghted part of the overall test result scores (e.g. include battery life as a 50% weighting in the "Overall Score"). After all, battery life IS a big issue for real world buyers! To get the best possible benchmark score, the OEMs would then need to target a balance between raw computing / graphics speed and battery life - and isn't that what most of us REALLY want?

      If the OEMs turn up their processors to 11 to get 20% better performance, I'll bet the battery life takes a bigger hit than 20%. Who wants a phone that is the fastest beast on the block according the popular benchmarks, but will only run for a hour and a half according to the same benchmarks?

      1. Richard 22

        Re: RE: Battery Life

        The benchmarking done internally by our company is mostly concerned with performance / watt. Taking accurate current measurements while running these benchmark tests is the best way to weed out any cheating of this kind, since the increase in performance will be more than outweighed by the increase in power consumption.

    5. Voland's right hand Silver badge

      Probably not

      It most likely looks at the actual signed developer info in the apk metadata. So simple renaming will not work here.

    6. Michael Thibault
      Mushroom

      It would hilarious...

      if an exploit managed to defeat the PMS stack across all apps, just for shits-n-giggles...

      "Wow, this little bugger is HOT!"

    7. Roland6 Silver badge

      Re: Samsung Benchmark Overdrive mode

      I expect we'll soon see some app's that have figured out just what this feature is and provide 'turbo' settings for a user's favour app's...

  4. Nate Amsden

    as a geek

    and as someone who plans to buy a Note 3 - it doesn't matter to me either. I've never owned a Samsung phone or even Android phone(or tablet) before. I know the basic specs of the device, I think the bigger screen will make it easier to type with my bigger hands. Though I have not tried it yet. I have little doubt it will blow the pants off my HP Pre3 with a single core 1.4Ghz Snapdragon, 512MB of ram and 16G flash(w/no SD expansion) - which performance wise is all I'm really concerned with. The multitasking stuff looks neat too.

    What I do care about though is finding a 64G Note3, if I get a new phone I want as much local storage (128GB w/64G SD expansion would be nice) as I can get. Would be nice to have a back cover that has inductive charging too (I think there was a special cover option for the Note 2 which had that feature).

    UNFORTUNATELY I have not found anyone selling (even pre order) 64G versions in the U.S. yet.

    1. Anonymous Coward
      Anonymous Coward

      Re: as a geek

      As someone who has used both bigger (Nexus 4) and smaller (iPhone) screens extensively, I don't think screen size has anything to do with how easy it is to type.

      Basically it comes down to motor coordination. Personally, I have pretty big hands, but I can still move my thumbs and index fingers short distances quickly and with good accuracy. So I can type quickly and accurately on the iPhone's screen. Once the distances get a little larger (Nexus 4) I have a hard time moving my fingers that far with the same level of accuracy. Actually sometimes I have to move my entire hand, depending on how I'm holding the phone. Result is that I can't type nearly as fast on the bigger screen as the smaller screen.

      1. Pristine Audio

        Re: as a geek

        >So I can type quickly and accurately on the iPhone's screen. Once the distances get a little larger (Nexus 4) I have a hard time moving my fingers that far with the same level of accuracy. Actually sometimes I have to move my entire hand, depending on how I'm holding the phone. Result is that I can't type nearly as fast on the bigger screen as the smaller screen.

        So install Swype then. Leave the iPhone's typing speed in the dust...

      2. Mark .

        Re: as a geek

        Where as I can type better on a Galaxy Nexus screen than a smaller 3.5" Nokia screen. I don't think there's an overall winner here. But that doesn't mean size has no effect, it means different people will have different experiences. Plus he did say he had bigger hands (and I find bigger screens are nice for all sorts of other reasons).

        Different experiences is also why it's good to be able to install different keyboards - and I agree with the other poster, swiping (now standard in Android) can making things much faster (though not everyone gets on with that method).

        This point is also worth considering for all the people saying we should throw away our laptops for ipads - imagine, those people having to type the same speed as a small phone screen. I did notice myself that whilst a 7" tablet on my Nexus was better for typing than the Galaxy Nexus, it was only very slightly better (and still nowhere near as good as a real keyboard).

    2. Anonymous Coward
      Anonymous Coward

      Re: as a geek

      As someone who owns a Note-3 (32Gb) I can promise that you won't be disappointed.

      Despite the test 'cheating' it's still VERY quick.

      The 4K video is worth the price of entry alone!

  5. Roo
    Windows

    Reminds me of SUN... :)

    Ah, poor old Samsung, despite innovating they will be remembered for copying... I wonder if anyone else was reminded of when SUN knobbled the SPEC 179.art benchmark (over a decade a go - blimey!) ? :)

    Bizarrely one of the few simple references I found on Google was written by a Mr Orlowski for el Reg. :)

    http://www.theregister.co.uk/Print/2001/11/22/sun_defies_gravity_with_usiii/

  6. Henry Wertz 1 Gold badge

    All kinds of cheats

    "Its not really cheating though, as technically the phone CAN run at that speed, they arnt falsifying data, just being a bit dishonest about how long the device would last at that speed."

    Oh it's absolutely falsifying data and cheating. If the device will never *ordinarily* run at a speed, it is flat out cheating to make it run at this higher speed only when running certain benchmarks.

    That said, NVidia & ATI both got caught making illegal optimizations in OpenGL and D3D benchmarks (in the past, they know they'd be caught now), you could take some 3D game or app, rename it "foobenchmark2000.exe" (or whatever), and suddenly your game would get like 40% better FPS but look like crap (most of the illegal optimizations were turning quality options off while lying to the user and app and claiming they are turned on. A few cheats were odder, like recognizing a specific 3D sub-benchmark and just pulling a known set of triangles rather than doing proper visibility tests and hidden triangle removal.

    It's been practically an art among supercomputer vendors to try to make their machine as fast as possible on benchmarks, within the constraints of the supercomputer benchmarks. The rules are stricter and stricter now, but in the 1990s and 1980s? Weird compiler optimization flags, weird setups that could even be re-tuned for each benchmark (especially on cluster systems, tuning settings so one benchmark would run great while the next may run terribly, rather than a "compromise" setting that runs all of them pretty well.) Custom systems that closely resembled a shipping system (other than just happening to run benchmarks faster....) but a customer couldn't actually buy for any price.

    I could go on and on but I won't.

  7. Anonymous Coward
    Anonymous Coward

    Samsung are hurting themselves - what with this and the region locking and then making a deal out of some of the fugliest gold phones ever........

  8. Wang N Staines

    The little sympathy I had for Samsung had disappeared with the region locking & cheating.

    I would never buy their products again until they clean up the act.

  9. Anonymous Coward
    Anonymous Coward

    Benchmarks

    People lose track of the idea of benchmarks, saying they're irrelevant for regular users, or that they're supposed to somehow be artificial measurements of the performance of the "raw hardware."

    Fact is, a good benchmark IS designed to represent (or actually BE) a real-world scenario.

    I think at least one of the benchmarks in question is supposed to be a web browser benchmark. I don't know how accurately it represents web surfing, but it's VERY relevant to normal users how quickly they can surf the web using their phones.

    I really wish all this commentary would stop, on how benchmarks are irrelevant, nobody should trust them, cheating doesn't matter, blah blah blah. If we keep up this attitude, it will become a self-fulfilling prophecy.

  10. bigtimehustler

    The best way to optimise a Samsung Android device is to remove the samsung changed Android OS and install something more closely matching the vanilla Android OS, so much snappier without all the android stuff on top!

  11. itzman

    Once upon a toime ...

    I did a job for a twat who sold PCS and a business application.

    "Look" he said ;'The Apricot is 4 times faster than this clone"

    I frowned. Same CPU, same frequency..I edited the config.sys of the half price clone to the same as the Apricot's "Look, half price clone as fast as Apricot"

    He never employed me again.

  12. Anonymous Coward
    Anonymous Coward

    Another benchmark result for the Note

    Zero.

    It crossed a border and (being region locked) refused to work.

  13. Anonymous Coward
    Anonymous Coward

    What's next?

    Region locking the Note.

    Cheating on benchmarks.

    They're on a roll... What's next? A Samsung executive coughs up chunks of identifiable endangered species onto the head table at a vegan convention?

    1. Anonymous Coward
      Anonymous Coward

      Re: What's next?

      You forgot "Copying Apple", their most egregious sin.

    2. Armando 123

      Re: What's next?

      Denying court orders about document confidentiality?

    3. Moeluk

      Re: What's next?

      We can't be far off them allowing the phone to install a rootkit on any PC it's hooked up to.

  14. raving angry loony

    The real schock...

    Actually, the most shocking thing is that people are surprised by a large corporation cheating on benchmarks. I can't think of a single large corporation that HASN'T cheated on something like that in the last three decades (well, 33 years, but let's not get pedantic) years I've been in this business. Marketing drives these things, and marketing is run by liars and con artists. Always has been. Always will be. It's their function.

  15. Chairo
    Angel

    Market-speech

    In market speech "cheating" is translated to "optimising". It is just the same than "doping" in sports, btw.

    There are some variations, of course. "lying" would be translated to "optimising the truth".

    And stop talking about "locking" or "disabling"! The correct market-speech is "featuring".

    So Samsung optimised the truth by optimising the benchmark performance of their phones and even added a new feature regarding regional support. From marketing point of view everything is in great order, now please be a good consumer, close your eyes and open your mouth!

  16. Anonymous Coward
    Anonymous Coward

    I just hope Samsung get a class action against them for this. They're a really dishonest company.

  17. Anonymous Coward
    Anonymous Coward

    I love the whole Intel is not a problem. They have been known to doctor benchmarks in their favour in the past. True though I don't see them trying to do so on third party ones but still. It's questionable at best.

    1. Roo

      In the case of Intel's C Compiler cheats, once upon a time the binaries generated would disable optimised code at run time if NOT running on a Genuine Intel (tm) processor. This was discovered by folks benchmarking their application and then trying to work out why their code took a massive performance hit on AMD chips.

Page:

This topic is closed for new posts.

Other stories you might like