back to article Bad benchmarks bedevil boffins' infosec efforts

A group of operating systems specialists has said that sloppy benchmarking is harming security efforts by making it hard to assess the likely performance impact of security countermeasures. The researchers from the Netherlands and Australia, decided to take a look at the accuracy of security researchers' systems benchmark. As …

  1. a_yank_lurker Silver badge

    Benchmarks Valid?

    All the benchmarks I have ever seen are like EPA gas mileage figures; artificial numbers the are optimistic at best. But numbers because of how they are derived have some comparative value at least in a gross sense. EPA mileage figures do not actually account varying load, weather conditions, etc. that affect the real world numbers. Benchmarks have the same problems, what is the proper balance between CPU load, memory load, drive reads, etc. to mimic the mythical average user? The 'crime' is not that benchmarks are skewed but that they researches are misusing them to try predict behavior in the real world when they are only good tell general trends.

    1. The Count
      Facepalm

      Re: Benchmarks Valid?

      Can anyone spot the multiple crimes made by this bot against the English language?

  2. Archtech Silver badge

    Is there anyone out there who is good at performance and security?

    'As they explain in this paper at arXiv, security papers are littered with so-called “benchmarking crimes”'.

    Ah yes, just as the chip optimizing experts have been committing security crimes.

    The wonders of specialization...

  3. John Smith 19 Gold badge
    Unhappy

    "Ah yes, just as the chip optimizing experts have been committing security crimes."

    But it sure did hit their benchmark targets, didn't it?

    I'm curious. How many systems don't start with the assumption (and it is an assumption) that the OS code is absolutely correct, and should therefor have full access to the machine?

    Because IRL, over and over again, we've seen that is complete bo***ks.

    We know every substantial piece of code will have bugs in it, and they will escape detection whatever level of testing it was given before release (which appears to be f**k all in some cases).

    So what OS developers plan for bugs in the code?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019