back to article Blighty's GCHQ stashes away 50+ billion records a day on people. Just let that sink in

The enormous scale of GCHQ's surveillance was revealed on Friday by newly published Snowden documents. The files note the growth in capabilities enjoyed by the UK government's snoopers since intercepting communications in bulk from 2007. These details were revealed in a series of documents published by The Intercept including …

  1. moiety

    So the question I'm asking is is this genuine, or propaganda? Where, exactly, did the new Snowden documents come from would be the first question.

    1. Mark 85 Silver badge

      As I understand from the reportage, there's still a rather large pile awaiting release. Releasing a little at a time allows full media irritation at the NSA while dumping it out all at once would be overwhelming for the media to grasp and talk about.

      1. moiety

        I understand that too; but there has been lots of time for nearly everyone to get at that pile and -this is somewhat of an understatement- motivation to do so. Even if the documents and Snowden are unimpeachable; changing the order of release could destabilise countries at a critical time; change an important vote etc.

        It's "divide and conquer" whether the document is valid or not; and this does not make me happy.

      2. Christoph Silver badge

        They are, I gather, having to go through all the material first - and there's a lot of it!

        They can't just dump it on the net. The most obvious reasons being that it might expose and endanger agents, or reveal important secrets which are not related to spying. There's probably personal information that shouldn't be exposed. All sorts of things. And they can't be sure what the problems might be unto they've actually read through the stuff.

        At a guess, Snowden probably laid down some conditions of what could be done with it before he gave it to whoever it was - a list of journalists he felt were trustworthy? I don't know the details.

        1. Pliny the Whiner

          Of shoes and (sailed) ships and sealing wax

          "[T]he most obvious reasons being that it might expose and endanger agents, or reveal important secrets which are not related to spying. There's probably personal information that shouldn't be exposed."

          Yeah, well, that ship pretty much sailed when the Chinese nicked the Colonies' database of all federal employees hired from the beginning of time to the present, including -- we now learn -- 5-6 million associated fingerprints. About the only thing that Snowden's documents could add is that Secret Agent 53B has a Boston terrier named "Throckmorton," which doesn't really increase anyone's knowledge other than to marvel at the stupid names people give to their beleaguered pets.

          1. moiety

            Re: Of shoes and (sailed) ships and sealing wax

            @Pliny the Whiner - Yes, yes, that was the OPM. However there are some issues that aren't about America, and this is one of them. Well, you might have been caught in the cross-hairs of GCHQ's spying; but in that regard you can just feel like the rest of the world feels about the NSA.

            So back to the point. Is this genuine or is this propaganda? What verification is there?

          2. Smooth Newt

            Re: Of shoes and (sailed) ships and sealing wax

            To be fair the Chinese having a copy of the OPM database isn't the same as it being in the public domain. They will only trade bits of it, at some price, with their friends and clients.

    2. JeffyPoooh Silver badge
      Pint

      Browse websites that you're not really interested in...

      Keep them guessing.

      1. Anonymous Coward
        Anonymous Coward

        Re: Browse websites that you're not really interested in...

        I hate it when certain websites jump around on the screen just when one is about to select a category.

        Who knows where one will end up... ...a couple of rows down. YIKES!! The spies must be left with entirely the wrong impression of one's predilections.

        I hope that this clarifies things for them. Thank you.

      2. Hans 1 Silver badge

        Re: Browse websites that you're not really interested in...

        The other day I saw an ad inciting ppl to convert to Islam, I thought it would be a good idea to click the link ... especially since, if the US have kept the files, I am considered a Jedi by them ... at least, that is what I jotted on the Visa weaver last time I was there ... ;-)

        I am an dwJediAtheist, in case anybody on here did not know.

      3. amanfromMars 1 Silver badge

        Re: Browse websites that you're not really interested in...

        Browse websites that you're not really interested in... Keep them guessing. ... JeffyPoooh

        That can have them prepared to give one something quite juicy and enjoyable and interesting too, if they be tempted to act and thus reveal their targeted attention upon certain persons/virtual personae......... but for that to be in the national interest, rather than a gross negligent waste of time, effort and paper money, would they need to be Super XSSXXXX Smart and more fully cognizant of inherent systemic failings in subprime and subprime ministerial leaderships.

      4. Adam JC

        Re: Browse websites that you're not really interested in...

        Or just spout meaningless gobbledegook, heck it's been working for amanfrommars for years! ;-)

        1. amanfromMars 1 Silver badge

          Re: Browse websites that you're not really interested in... @ Adam JC

          Or just spout meaningless gobbledegook, heck it's been working for amanfrommars for years! ;-)...... Adam JC

          Meaningless, Adam JC? Oh, I don't think so. You need to pay greater attention to the shared details, methinks, but it is fair to say that secure steganography is something of an alien art form that makes effective phishing more than just simply challenging.

          1. Adam JC

            Re: Browse websites that you're not really interested in... @ Adam JC

            My comment comes from a deep source of respect, amanfrommars! ;-)

    3. Stuart 22

      All this and they didn't spot the VW 'defeat test' emails? Something that likely will result in more deaths in the UK than all the terrorist activity combined. And cost more.

      1 out of 10 GCHQ. Must try harder.

  2. elDog Silver badge

    It's getting harder and harder to fit all this data on my thumb drive

    As I exit the building.

    Fortunately the tech companies are surfing right in front of the wave.

    640TB should be enough for anyone!

  3. edge_e
    Unhappy

    Where there should be anger

    only apathy

    the icon should be crying

    1. Stuart 22

      Re: Where there should be anger

      Meanwhile real terrorists have moved on and probably reverted to coded classified ads in your local paper to communicate with their underground warriors or using invisible ink. Bit like our secret services did in WW2. Difficult to trap electronically. It worked then ...

      Only the stupid will get caught.

  4. Daggerchild Silver badge
    Terminator

    #include <std/quotes/spiderman>

    Yum. Data. I've thought about tools like those myself. Didn't realise they were quite so far along.

    I expect to be stoned to death for this, but I am not actually that bothered that this exists, or that GCHQ use it. It's what they'll have to have to do what they have to do. (Of course that doesn't mean we have to make their job easy)

    What does bother me is when they put down the Palantir and leave their world of fear and shadows to enter the real world. Paranoid humans staring at clouds will see the faces of monsters in them. The same will be true of net data. I think there was a family in America who each individually went shopping for ingredients that, combined, made for another Boston Bomber (pressure cooker, nails, rucksack). Pure random coincidence but it ended up as cops in their house. The maths says that for each loony you successfully dig up, you'll probably have distressed many more innocent families. It actually becomes a question; who spreads more terror?

    Using power like this outside their realm needs a high price. If you use it, and you miss, you need to suffer, *really* suffer. Can't handle it? Can't have it.

    1. Anonymous Coward
      Anonymous Coward

      Re: #include <std/quotes/spiderman>

      Which segues into the drone assassination program on the US side and now operational for your side as well. The maths says that for each loony you successfully dig up, you'll probably have distressed many more innocent families. It actually becomes a question; who spreads more terror? I don't like this calculus at all.

    2. Richard Jones 1
      Unhappy

      Re: #include <std/quotes/spiderman>

      The story about 'that family' sounds rather like urban myth. Unless you get enough qualifying markers no one, marketer, shop data miner, Google, Facebook or even the covert operators, absolutely no one should even think of pulling any sort of trigger on an operations. Whether to market something, 'I see you bought a pressure cooker, would you like some semtex to go with that?' or an investigator saying 'send a squad car there a subversive there'. In both cases sloppy processing could cause such stupid actions, but if processing is that sloppy any bad guys of any sort have nothing at all to fear. The roads would be traffic jammed with false events. The trick is always to screen out the false 'could be positives' in favour of the 'real yes they are villains'. The papers suggest that in at least one published case the 'reality (sanity?) checks' took to long, so someone died.

      Black holing data that has not proved useful so far, is one way to allow a second look if subsequent data come to light to allow more focused searches, while at the same time avoiding cluttering up on-line operations. That way a less real time operation might go back in search of sleepers who have not yet been triggered.

      Big IF time, if the operation is evolving via the routes the papers suggested it appears to have followed the well established road maps for detection activities,, so no real surprise. As ever the risks of fantasists and time wasters both inside and out side of operations is a real hazard. The shortage of skilled and well directed operatives is the real weak spot. This can lead to various nightmare outcomes as we have seen a bit too frequently.

      1. Uncle Slacky Silver badge
        Stop

        Re: #include <std/quotes/spiderman>

        It's no urban myth:

        http://www.theguardian.com/world/2013/aug/01/new-york-police-terrorism-pressure-cooker

        "Michele Catalano, who lives in Long Island, New York, said her web searches for pressure cookers, her husband's hunt for backpacks and her "news junkie" son's craving for information on the Boston bombings had combined somewhere in the internet ether to create a "perfect storm of terrorism profiling".

        Members of what she described as a "joint terrorism task force" descended on Catalano's home on Wednesday."

    3. Hollerith 1

      Re: #include <std/quotes/spiderman>

      Sadly, pressure cooker + nails + rucksack + Russian warning that these two brothers should be kept under observation + two American boys heading off to Chechen land for a summer vacation all did not add up to 'maybe we should send the police aroudn to have a word'. They do that later in the fear that there will be a copycat, but it would be really nice is they used their gazillion petayetaflops of data from us to, ya know, stop atrocities before they happen. You can call me a dreamer...

    4. Fat Northerner

      Re: #include <std/quotes/spiderman>

      If you want to control the government Daggerchild, you have to get them to do what you want. The same is true for facilitating the protection of our women folk.

      And you can't just tell them to do it, because they won't. Politicians have close protection, houses in Primrose hill, and total response of the police. They just don't give a f*ck about you for more than one day every five years.

      So if you want your womenfolk protected, from nutters which you know have unique abilities, then seed the idea with the intelligence services as to how they'll be caught, and act in that fashion against MPs. They will then fund the security service's ability to detect you, and by extension, the nutters.

      When they look closely into you, they'll see you're just some capricious average joe, and aside from listening to your daughter masturbate through bugs built into plug sockets, tracking your every movement, and following you on holidays abroad, there'll not be any really intrusive privacy concerns.

      The only real downside is you'll lose security clearance, but who wants it with so many Common Purpose chippy shouldered fourth wave feminist b1tches with humanities degrees working in the public sector.

  5. Anonymous Coward
    Anonymous Coward

    Security threats

    Doubtless much of the spooks' time is currently being spent on those people who have been clearly identified by the Prime Minister as "a threat to our national security, our economic security and your family's security".

    It would be highly remiss of them to ignore such a warning from the leader of our country.

    Cameron was referring to the Labour Party.

    Of course, the Intelligence Services may not turn up anything which would prevent a single act of terrorism but I'm sure that what they collect will be useful in the hurly-burly of political life in Westminster.

    1. graeme leggett

      Re: Security threats

      Short of finding out that a backbencher or a parliamentary assistant secretary actually cleaves to a foreign power, I don't think intelligence would have much to do with the cockpit that is parliament.

      While a minister might like the spooks to enumerate the opposition's peccadilloes to his own gain, the positions of power are fluid and within a year he might be cast across the chamber and his former target the new minister. A sort of MAD. Better to let journalists and the rumour mill bring skeletons to notice.

    2. Rich 11 Silver badge

      Re: Security threats

      I think it's reasonably unlikely that the spooks will be able to uncover anything suggesting that Corbyn fucks pigs. If they were to broaden their scope beyond one party, though...

    3. Smooth Newt
      Black Helicopters

      Re: Security threats

      There is a long tradition of the security services spying on Labour party politicians, going even further back than Wilson. http://www.theguardian.com/uk-news/2015/mar/25/police-spied-on-labour-mps-whistleblower etc ad nauseum.

      To many of the people attracted to a career in the security services, "a threat to public order" is synonymous with "interfering with the status quo". I expect Corbyn already has several terabytes from his career to date.

  6. Teiwaz Silver badge
    Facepalm

    If only they spent as much time, effort and money unlocking the secrets of the universe rather than peoples browser history...

    1. Lyndon Hills 1

      If only they spent as much time, effort and money unlocking the secrets of the universe rather than peoples browser history...

      Who was it paraphrased Howl with 'I watched the brightest minds of my generation, spend their time trying to persuade people to click on ads'?

  7. amanfromMars 1 Silver badge

    The Power of Knowledge is a Real Crazy Thing and Delivers Maddeningly Unintended Consequences

    So, GCHQ and MI5 are ultimately responsible and accountable for all abhorrent behaviour of folk using communications devices, the metadata and/or content of which they can so easily, apparently, access and interpret/understand ‽ .

    Is that a perversely convenient, government of the day, 'Stay out of Jail Free" card which can be played by assorted rogue elements which/who may have thought to act as with impunity?

  8. Anonymous Coward
    Anonymous Coward

    Imagine how many more lives would be saved

    if they spent a fraction of that money on road safety.

    1. Christian Berger Silver badge

      Re: Imagine how many more lives would be saved

      Yes or education. Good and widespread education is the best way to pacify a society and make your economy stay competitive. However that would also be a thread to the upper class. Educated people might question your decisions, like to kickstart a society bankrupted by banks by austerity measures, etc.

  9. John Smith 19 Gold badge
    Gimp

    "what is metadata extends to passwords"

    You know that is no accident

    Data fetishism

    It's a disease, not a sane policy.

  10. Your alien overlord - fear me

    Why doesn't GCHQ/MI5 start (or buy) a mobo business and also become an ISP - imagine the data they could slurp legitinately.

    1. Anonymous Coward
      Anonymous Coward

      what on earth makes you think they haven't?

  11. Fraggle850

    Is this the only instance of a successful UK government IT project?

    Odd that they apparently got this one right but can't seem to otherwise do large scale IT effectively. Couldn't we redeploy some of the spooks to GDS, a few more to the NHS, some to the police, etc... ?

    Might even find that their somewhat 'esoteric' knowledge would help push things forward e.g. : 'I know you don't want to spend the force's money upgrading from Windows XP chief constable but it would be 'unfortunate' if your Web surfing history got into the public domain...'

    1. Anonymous Coward
      Anonymous Coward

      Re: Is this the only instance of a successful UK government IT project?

      You're assuming it works properly.

      1. Rich 11 Silver badge

        Re: Is this the only instance of a successful UK government IT project?

        At least they managed to get *something* working. And I bet they didn't outsource it to Crapita.

        It almost makes you proud to be British...

        1. Fraggle850

          Re: Is this the only instance of a successful UK government IT project?

          Outsourcing to Crapita would be bad enough but given that our gubinmentalists are happy to share our personal medical data with commercial interests perhaps it's not too big a leap of faith to wonder at the commercial possibilities of GCHQs extensive dataset?

          No, I'm being excessively paranoid. That would never happen because our ruling elite are scrupulous when it comes to separation of public and commercial interests. The fact that the head of corporation X saw politician Y doing something dubious with a pigs head at some elitist university club initiation is unlikely to have any bearing whatsoever.

        2. Anonymous Coward
          Anonymous Coward

          Re: Is this the only instance of a successful UK government IT project?

          At least they managed to get *something* working. And I bet they didn't outsource it to Crapita.

          They got something running I'm not convinced it works.

  12. amanfromMars 1 Silver badge

    Wake up and Smell the Java and Cocoa

    Why doesn't GCHQ/MI5 start (or buy) a mobo business and also become an ISP - imagine the data they could slurp legitinately. ....Your alien overlord - fear me

    No need, Your alien overlord - fear me, whenever legitimacy is not a requisite for doing sensitive spooky business anywhere, near or far, here and now or later. Courts are sideshows for grandstanders playing to the intellectually challenged masses, simple details of events after the event. They don't actually do anything fundamentally groundbreaking and revolutionary like phorming and/or phishing to effectively impact and driver the future in a novel beta direction.

    The maintenance and retention of corrupt and perverse status quos, whenever systems are freely acknowledged as being far from perfect, are their bag with its bags of right dodgy tricks.

  13. ISYS

    How long do they store the data for?

    What this does not tell us is how long the metadata is stored for. 217TB sound like a lot of data but in the grand scheme of things it isn't. Banking systems hold nearly as much.

    Do they slurp up a days worth of metadata from across the globe, process it and then keep the bits they are interested in?

    Or do they just download the entire internet's browsing history and file it away?

    My guess is somewhere between the two. As another poster mentioned, this system would have been a govt IT project. The spooks probably wanted the first option and got something nearer the second.

  14. i steal your leccy

    Hold A Happy Thought!

    We can hope that the meals served in the canteen today contain Polonium:)

    Don't upvote whatever you do:(

    1. Anonymous Coward
      Anonymous Coward

      Re: Hold A Happy Thought!

      Terrorist! You'll be trying to add radioactive Potassium 40 isotopes to all the Bananas in Cheltenham & Vauxhall Cross & 'arrogate tesco's next, I hope that MODplod regularly sweep the soft sigint infrastructure underbelly for such criminal attacks on spooks

    2. Anonymous Coward
      Anonymous Coward

      Re: Hold A Happy Thought!

      Swallowing polonium does indeed safeguard your privacy. There's no surveillance over the other side yet.

      1. Madge
        Trollface

        Re: Hold A Happy Thought!

        How do you know? Could be worse if at all?

  15. Dan 55 Silver badge

    So...

    When's El Reg flicking the switch for https?

    1. Anonymous Coward
      Anonymous Coward

      Re: So...

      Do you really think SSL actually works?

  16. phil dude
    Coat

    a few random thoughts...

    We are paying for this - is it good value for money?

    Does it play Crysis?

    P.

    1. Anonymous Coward
      Anonymous Coward

      Re: a few random thoughts...

      It generates Crysis.

  17. BobRocket

    Data Source

    Presumably there is a big fat pipe going in to GCHQ that all this data slurp is carried on, how do they know that they are seeing real traffic emanating from this spigot and not some spurious crap injected into the stream by a foreign/malign/domestic entity.

    How much faith does the bod on the desk have that any analysis they do will be presented higher up in a manner representing any kind of reality ?

    1. amanfromMars 1 Silver badge

      Re: Data Source and Novel Noble Nobel IntelAIgent Supplies for CyberSpace Programs/Mass Projections

      Quite so, BobRocket, and those are the realities which virtual machines assure systems are true to and quite necessary for the maintenance of life as IT and media portray it/broadbandcast it to try and force and reinforce perceptions/realise imagined thoughts, both good and/or bad.

      Welcome to Greater IntelAIgent Games .... where nothing is really as it seems and everything is capable of being, and increasingly likely to be in the present deliveries of madness and mayhem, flash crashed and beautifully destroyed quite sublimely and remotely anonymously ...... by forces and sources way beyond any traditional ken and savvy.

  18. Destroy All Monsters Silver badge
    Trollface

    The right not to be forgotten?

    Maybe we should enshrine it as additional "Human RIght".

    That would just make it legit, liberals could take to the benches and defend the utter goodness of governmental oversight, "accidents by MP5" could lead to heartfelt expression of "he wanted too much of his right not to be forgotten" etc.

    Problem solved!

    Once we go full Oswald Mosley/hard left this can be morphed into the "right to be remembered".

  19. Anonymous Coward
    Anonymous Coward

    600k records a second

    I sure would love to get my hands on the hardware capable of storing pictures of cats so fast.

  20. phuzz Silver badge
    Happy

    HTTPS

    Dear el Reg, about time you provide an https connection to this site?

  21. Yugguy

    "Black Hole" is right.

    Cos I imagine the volume of data is such that until we get a POI-style Machine it'll be immensely difficult to get any useful output.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019