back to article So you've 'seen' the black hole. Now for the interesting bit – how all that raw data was stored

The black hole image released yesterday needed over a thousand helium-filled disk drives to record it. Regular drives were tried and tested but failed due to air pressure at the high altitudes of the radio telescope sites. Eight radio telescopes at six locations formed the Event Horizon Telescope (EHT) array, which …

  1. BebopWeBop
    Thumb Up

    I understand the maths involved in the reconstruction to create a virtual telescope, but what boggles me (and betrays my real experience) is the sheer scale of the organisation and admin needed to get all of this together and functional.

    1. Anonymous Coward
      Anonymous Coward

      If I might quote from one university's draft Research Data Management document...

      "[...] procuring, configuring and implementing storage solutions is not usually a core competency of a researcher – nor should it be."

      *shakes head in bewilderment*

      1. Anonymous Coward
        Anonymous Coward

        Re: *shakes head in bewilderment*

        "[...] procuring, configuring and implementing storage solutions is not usually a core competency of a researcher – nor should it be."

        *gets green with envy*

        Head of a R&D lab here. Just yesterday I spent an hour trying to find furniture that could fit in our monthly budget.

        1. Anonymous Coward
          Anonymous Coward

          Re: *shakes head in bewilderment*

          If you've seen the difference in price between SMOHO storage for ~100TB+ range, like a Synology, and Enterprise storage in the same range, you'd appreciate WHY a hard pushed for cash R&D lab head would choose a "comedy home-brew system" when the university's offering is non-existent.

        2. imanidiot Silver badge

          Re: *shakes head in bewilderment*

          Find furniture that fits in your overal budget and spread booking the payment over more than 1 month? Or buy a few tables and chair each month? This is mostly a problem that calls for creative bookkeeping to fit whatever arbitrary rules the organisation has put in place.

          1. Anonymous Coward
            Anonymous Coward

            Re: *shakes head in bewilderment*

            Or buy a few tables and chair each month?

            Local guvmint rules (I'm in South Bananistan) asks for three separate estimates for each unique item. Must get several documents for each seller. Lowest bidder wins. Allow 90-120 days for payment. Bidders know that so all proposals already are 50-100% overpriced.

            I've mentioned to our troll in charge of buying stuff that the time we spend on this is worth more than the chairs and desks. Got a lecture on "doing things the way they are supposed to be done".

            15 more years until retirement... then I'd have to face the HR trolls.

      2. Jamie Jones Silver badge

        Re: If I might quote from one university's draft Research Data Management document...

        <hippy mode>

        I was just thinking how great the human race can be when it works together.

        It makes Trumps walls and Moggys threats to throw tantrums in the EU even more pathetic.

        </hippy mode>

        Oh, and Checkmate, flat-eathers!

        1. 404

          Re: If I might quote from one university's draft Research Data Management document...

          'Checkmate, flat-eathers!'

          Takes more than an unfocused picture of the Eye of Sauron to fool a flat eather...

          1. This post has been deleted by its author

    2. Doctor Syntax Silver badge

      "what boggles me ...is the sheer scale of the organisation and admin needed to get all of this together and functional."

      Try this: https://www.bbc.co.uk/iplayer/episode/m00042l4/how-to-see-a-black-hole-the-universes-greatest-mystery

    3. rg287

      I understand the maths involved in the reconstruction to create a virtual telescope, but what boggles me (and betrays my real experience) is the sheer scale of the organisation and admin needed to get all of this together and functional.

      Probably why it took Shep Doeleman more than a decade to make it happen! Years and years of proposals, talks and not a little sucking up to the right people to get the funding and observing time. Huge amounts of preparation just to get the infrastructure in place for 5 days of observations.

      And all that is before you get into the intellectual feats to do anything useful with it and develop the mathematics and software to wrangle the raw data into a meaningful result!

    4. Bongwater

      It's definitely not the Imperium of Man..... they would have taken another 900 years....

      1. baud

        The Imperium of Man would have dismissed scientific research as heretic and burned down everyone involved. Remember that those guy don't believe in thing like "progress" or "science" and their technological caste worship technology because they can't understand it.

  2. 9Rune5
    Coat

    Flat Earth

    said to be good enough enough to read a newspaper in New York from a sidewalk in Paris

    You flat Earthers are everywhere!

    (that said; I was surprised to learn that regular drives aren't sealed. Good thing my kit is located 19m above sea level)

    1. BebopWeBop
      Joke

      Re: Flat Earth

      Well until global warming really hits

    2. xeroks

      Re: Flat Earth

      Pretty sure they'd need to be at the top to the Eiffel Tower to see that far.

    3. A.P. Veening Silver badge

      Re: Flat Earth

      My kit is slightly below sea level, just like the ground floor of my house. More importantly, computer manufacturers have been asked in the past to recertify their products to guarantee it from somewhere like 10 meters below sea level to their current maximum height above sea level, just to secure an order from Schiphol (Amsterdam Airport) as it is at minus 7 meters. Luckily for most pilots, air planes mostly use radio altimeters now.

      1. werdsmith Silver badge

        Re: Flat Earth

        Luckily for most pilots, air planes mostly use radio altimeters now.

        Only near the ground I think. Many planes don't have radio ones, just barometric. QFE/QNH etc.

        1. A.P. Veening Silver badge

          Re: Flat Earth

          Only near the ground I think. Many planes don't have radio ones, just barometric. QFE/QNH etc.

          Near the ground is the only time it is relevant. Planes used to fly and land at negative heights using barometric altimeters upon an approach of Schiphol, enough pilots got scared shitless the first time this happened to them.

          1. Anonymous Coward
            Anonymous Coward

            Re: Flat Earth

            "Planes used to fly and land at negative heights using barometric altimeters upon an approach of Schiphol, enough pilots got scared shitless the first time this happened to them."

            Not a pilot, but ISTR that pilots reference altitude relative to sea level at cruising altitudes and reference ground level during landing/takeoff phases. For barometric altimiters don't they set 0' to correspond to the pressure reading at the airport?

            Even if that's not the case, knowing that ambient pressure is always changing, landing at a negative altitude for a roughly sea level airport wouldn't be discomforting. If you're relying on a barometric altimiter reading, it'd be just as unnerving to land at a 3000' high airport on a day when the altimeter was reading 2900'. There's no magic about 0', the hard limit you're worried about is where the runway is supposed to be.

            1. Andre Carneiro

              Re: Flat Earth

              Above a transition altitude you use 1013.2 hPa as the altimeter reference. Below the transition level you use QNH (altimeter setting corresponding to the air pressure which makes it so that the altimeter will read the aeroplane ALTITUDE) or QFE (altimeter setting that makes it read the HEIGHT of the aeroplane in relation to the aerodrome).

              So if you're using QNH and your aerodrome is at 150' above mean sea level (MSL) then your altimeter will read 150' when you're on the ground.

              If you're using QFE then your altimeter will read 0 when you're on the ground.

              QNH is the setting used most commonly, as radio altimeters are more precise and are the source of data below approximately 2500'.

            2. Bill Michaelson

              Re: Flat Earth

              "For barometric altimiters don't they set 0' to correspond to the pressure reading at the airport?"

              They do. In the US, the first part of an airport condition report such as ATIS contains the phrase "altimeter XX.YY (in/Hg)" so that a pilot can appropriately calibrate the pressure altimeter via the Kollsman window prior to approach.

              But the pilot is typically quite relaxed anyway because they have another window in front of them that they can look through to see the runway. As long as the altimeter is calibrated reasonably, there will be adequate error margin for a precision approach with 200' AGL minimum to be clear of clouds, and a precision approach supplements vertical guidance via continuous glide slope indication. A non-precision approach will like have 500' minimums or higher. That's about a half inch of Hg slop.

    4. chivo243 Silver badge
      Pirate

      Re: Flat Earth

      Shit! My kit isn't even that high... I think my back garden is below sea level her in NL, now where's that boat that was fly tipped I was reading about, might come in handy when all that ice melts?

    5. Jamie Jones Silver badge

      Re: Flat Earth

      Regular drives *are* sealed. They are just not pressurised. (If you look carefully, you'll see a mini-condom lookalike on your drives which moves in and out to equalize air pressure.

      1. Gene Cash Silver badge

        Re: Flat Earth

        Hm. On all the drives I've disassembled, there's been a small fiber filter next to a very small hole in the enclosure. (granted, I've not disassembled any in the past 3 years or so. I've never seen a rubber flexure of any kind.

        1. Jamie Jones Silver badge

          Re: Flat Earth

          Hmmm. Ok, a very small membrane!

          https://www.pcmag.com/util_get_image/19/0,1468,i=194580,00.jpg

          I was led to believe it was like a tiny pin sized thing made of a balloon-like material. My bad.

  3. Drone Pilot

    My mom is proud I know HMTL. I'll not crush her by saying people who understand this type of data exist :)

    1. Valerion

      HMTL?

      Hyper Markup Text Language?

      1. Anonymous Coward
        Anonymous Coward

        Re: HMTL?

        "Hyper Markup Text Language?"

        Yes - it's commonly used to write Instagram and Tinder profiles.

      2. Vometia Munro Silver badge

        Re: HMTL?

        Only if you say it in Dobly.

    2. Paul Hovnanian Silver badge

      My mom tried to get me into drug intervention for using PHP.

      1. Anonymous Coward
        Anonymous Coward

        And she was right. PHP is a drug that causes immense social damage.

        1. asdf

          the malware portal

          And say hello to my little friend Wordpress - commonly heard in nightmares of corporate IT security people everywhere.

        2. J.G.Harston Silver badge

          I tell the scammers that ask if I have any problems with PPI that I'm perfectly happy with my Proton Pump Inhibitors.

  4. Dom 3

    Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

    That is all.

    1. OssianScotland

      Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

      Or in this case, a Hercules full of HDDs....

      … latency sucks, though.

      1. Tom 7

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        The start of streaming is indeed delayed but the end of streaming is a lot lot lot quicker. Which in this case is quite important. In this case latency can suck as much as it likes, In fact 'latency sucks' sucks.

      2. JJKing
        Coat

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        … latency sucks, though.

        Not if you get high enough to catch a Jet Stream. ☺

        Your Vint Cerf reference in the Title was actually referring to a motorcycle with a sidecar and a new Unix release. Mine's the one with the 300 baud modem in the pocket.

        1. Adrian Harvey
          Headmaster

          Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

          Title reference is to Tanenbaum in “Computer Networks”. Not Cerf.

          Quote details in the link. Tanenbaum was likely paraphrasing others here. Lots of people have made similar statements in various forms.

          Pedant icon as we’re both being pedantic!

      3. wallyhall

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        Obligatory XKCD reference: https://what-if.xkcd.com/31/

        "Ding dong" - "Internet's here!"

    2. TRT Silver badge

      Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

      Or a helicopter hauling helium.

      1. Arthur the cat Silver badge

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        Or a helicopter hauling helium.

        Or a rocket running RAID?

    3. Doctor Syntax Silver badge

      Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

      And don't sneer at "spinning rust": Western Digital's HGST 8TB helium-filled drives were used, whereas capacity and cost limitations ruled out SSDs.

      1. phuzz Silver badge

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        Mind you, they bought the equipment in late 2016 to early 2017. Two years on and spinning rust is still the cheapest per byte, but that gap is narrowing every week.

        1. Ian Michael Gumby
          Boffin

          Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

          Not really.

          While the cost per TB is getting closer, there is still a significant gap.

          Its going to be a while, although... I'd say go w SSDs because they will last longer under the stress imposed on them. (Less likely to fail)

          I tend to use Samsung pro drives and M.2 cards, New Egg has a 10pack of 3.8TB drives for $9,750USD

          That's ~975.00 a drive for 3.8TB which isn't bad. (Never tried them so I don't know how well they will last. ) Samsung has 4TB for around $1,140 USD.

          And Intel has a 7.6TB drive.

          YMMV

          1. Trollslayer

            Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

            Flash memory including the type sued in SSDs are less reliable.

            Flash relies on storing an electric charge and that can leak away over time.

            Spinning rust requires energy to flip the bit.

            The best example was when courts had USB sticks with evidence on and stored them without making certified copies.

            2-3 years later they found the USB sticks were unreadable because the charge had leaked away in too many places.

      2. Wellyboot Silver badge

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        I would think they will also have had a limited amount of time to use the disks at altitude before enough Helium escaped to cause head crashes again.

        1. Anonymous Coward
          Anonymous Coward

          Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

          Helium-filled disks are going to be really, really, well sealed, because helium has tiny molecules (single atoms, in fact, so much smaller than hydrogen) and leaks through everything.

          1. Jon 37

            Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

            Yep, one of the big challenges of making helium-filled drives was figuring out how to seal them so the helium doesn't escape before the warranty runs out. Of course, if you have a helium-filled drive, I would suggest replacing it sharpish when the warranty does run out, because it might be about to run out of helium.

          2. JJKing
            Coat

            Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

            helium has tiny molecules and leaks through everything.

            So that is why my farts stink the up the neighbourhood. They are made of helium and are not filtered by my grundies.

            Mine's the one with air freshener and gas mask in the pocket.

          3. J.G.Harston Silver badge

            Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

            Hmmm? A molecule of hydrogen is two atoms, 2 protons + 2 electrons = 2.001ish. A molecule of helium is one molecule 2 protons + 2 neutons + 2 electrons = 4.002ish. Doesn't that make helium bigger than hydrogen? Certainly I remember doing diffusion experiements in school, and H2 escaped faster than He.

            1. Stoneshop
              Boffin

              Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

              A helium atom is surrounded by a single, fully populated, electron shell; a hydrogen molecule consists of two atoms, each with their shell (which get intertwined to some extent). As the diameter of these shells are by and (not very) large equal, simple geometry would posit that a hydrogen molecule is larger by the distance between their nuclei. Except that at the atom scale simple geometry doesn't quite apply. But anyway, empirical data has shown that He is indeed a bit smaller than H2, 28pm (2.0248024e-13 brontosaurus) versus 31pm (2.2417455e-13 brontosaurus).

            2. Anonymous Coward
              Boffin

              Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

              Helium is not only smaller being monoatomic, it has the advantage from a diffusion point of view that it isn't sticky, like hydrogen (hydrogen bonds), which is why helium is used in leak detectors.

              But my solution for keeping the helium in the drives is simple: include some Pu-238, which is an alpha (helium nucleus) emitter. Not only is the supply of helium being constantly restored but even at altitude, where it's cold, your drive stays nice and warm.

              I can't see any obvious objections, and for some reason it isn't patented, so feel free.

            3. eromana
              Thumb Up

              Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

              You are correct a H molecule is lighter than a He atom, so the H2 sound speed is greater that of He by a factor of root. Diffusion on the other hand depends on 2 materials and their state. So for example, in air H2 has a longer diffusion length than He. It turns out that the molecular diameter of H2 is slightly larger than that of He, 297 vs 258 (picometer) [1]. Because He is an nonreactive noble gas the diffusion of He through a solid material barrier is much but much greater than the molecular diameter would insinuate. "At 512°C the rate of helium diffusion [in silica glass] is approximately 45 times that of hydrogen, and it is believed that chemical forces delay the progress of the hydrogen molecule through the silicate network"[2]

              [1] http://www.kayelaby.npl.co.uk/general_physics/2_2/2_2_4.html

              [2] https://aip.scitation.org/doi/pdf/10.1063/1.1750133?class=pdf

      3. dfsmith

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        Pfft. Spinning rust is terrible for PB of data. It would take almost 100 million "spinning rust" drives to store 14PB. Luckily, spinning cobalt alloy media is far superior. (The last iron oxide drives were sold in the early 1990s.)

        1. Ian Michael Gumby

          Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

          huh?

          Pfft. Spinning rust is terrible for PB of data. It would take almost 100 million "spinning rust" drives to store 14PB. Luckily, spinning cobalt alloy media is far superior. (The last iron oxide drives were sold in the early 1990s.)

          Oh, while you're getting to be a word nazi, the term spinning rust is meant as a reference to hard drives.

          But to your point spinning drives at 12TB per drive. 100 drives is 1.2PB. 1,167 drives is 14PB.

          Now I thought that there were larger drives... like 14TB so that would be 1000 drives.

      4. Wayland

        Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

        Western Digital's HGST?

        HGST used to be Hitachi which was the IBM drives. So WD has them now? Funny old corporate world.

    4. Antron Argaiv Silver badge
      Coat

      Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

      ...and when all else fails: sneakernet.

      (one HDD in each pocket...balances the load)

  5. Anonymous Coward
    Anonymous Coward

    BBC: what?

    I can't see anything useful in such an analogy but the switch from paper to disk.

    Everything else is orthogonal.

    I'm always impressed how the mass media can fail to understand almost anything about science and technology.

    1. Greg D

      Re: BBC: what?

      It was a feminism angle. A female wrote the code that got Apollo to the moon, and a female wrote the algorithm that got us the black hole pic.

      1. Tom 7

        Re: BBC: what?

        You misunderstand the BBC. They were trying hard to explain why this meant brexit was a good idea.

        1. Anonymous Coward
          Anonymous Coward

          Re: BBC: what?

          "You misunderstand the BBC. They were trying hard to explain why this meant brexit was a good idea."

          Using their standard technique of having 4 Remainers and 1 Brexiteer for balance? And when questioned over balance declaring that any government minister in-spite of any views said minister then chooses to express (or later follows up with by voting for in Parliament)?

          1. Anonymous Coward
            Anonymous Coward

            Re: BBC: what?

            So we have one person arguing that the BBC are pro-brexit, one against: could this imply they are actually unbiassed?

          2. Anonymous Coward
            Anonymous Coward

            Re: BBC: what?

            The BBC isn't biased against BREXIT - they often allow many morons on to spout their lies and fantasies without challenge.

          3. Anonymous Coward
            Anonymous Coward

            Re: BBC: what?

            Yet they permit 1 UKIP activist to appear on and ask questions on at least 4 different question time shows in one year and then plead ignorance when they are called on it, along with cutting the reply from a Scottish Govt minister to make her look incompetent.

            This along with quoting UKIP at every chance they get, failing to challenge blatant falsehoods by Brexiteers, having at least one senior writer hauled up for being biased against the main opposition (yet said writer is still in post)

            The BBC is pro-Brexit pretending to be impartial, look at the amount of times they've had Brexiteers on TV and given them soft ball questioning and allowed them to air nonsense and then act like its undisputed fact

            Read articles from BBC News from late 90s early 2000s and compare the standard of writing with the red top gutter "journalism" they emit today, where everything is dumbed down and their contradictory biases are apparent - Pro-Brexit, Anti- disabled, Pro-radical feminist, Pro religion, Anti-sex, pro-nostalgia (what products from the good old days are due a comeback as an example, all of it was stuff that anyone born after the mid 60s would have only seen in the grandparents homes)

            1. Anonymous Coward
              Anonymous Coward

              Re: BBC: what?

              Post-Brexit chaos will sell newspapers and get clicks, which is why the media are for it.

              The old joke was that every night at bedtime the editor of the Daily Express prayed for something terrible to happen to the Pope, Billy Graham or the Queen.

              1. CountCadaver Silver badge

                Re: BBC: what?

                IIRC an editor of the Daily Mail was quoted as saying if he hadn't raised the blood pressure of his readers by an average of 10 points he hadn't done his job

      2. Anonymous Coward
        Anonymous Coward

        Re: BBC: what?

        "A female wrote the code that got Apollo to the moon, and a female wrote the algorithm that got us the black hole pic."

        Mostly wrong on both counts, but whatever. I don't expect anything like "accuracy" out of the media or anybody else these days, anyway.

        1. Anonymous Coward
          Anonymous Coward

          "Mostly wrong on both counts..."

          <citation needed>

          1. scross

            Re: "Mostly wrong on both counts..."

            Nope, this isn't academia, so you'll have to do your own legwork here if you want to know the details. I've already done it, so you can too!

            1. Richard 12 Silver badge
              Facepalm

              Re: "Mostly wrong on both counts..."

              I've done the legwork, and the only possible inaccuracy is the word "the" instead of "some of the" or "most of the" or "one of the", as there were of course several different algorithms and many different "codez", all of which were needed to get to the moon/assemble the image.

              However, in this case, Katie was personally responsible for the algorithm and software which made it possible to run any of the others, so yes, she did write the thing that made it possible.

              Just not the one thing you looked at, scross. Do the legwork properly next time.

              1. Anonymous Coward
                Anonymous Coward

                Re: "Mostly wrong on both counts..."

                Reference the other article about this on The Register right now, which includes a link to Katie Bouman's own Facebook page, which states the following:

                "I'm so excited that we finally get to share what we have been working on for the past year! The image shown today is the combination of images produced by multiple methods. No one algorithm or person made this image, it required the amazing talent of a team of scientists from around the globe and years of hard work to develop the instrument, data processing, imaging methods, and analysis techniques that were necessary to pull off this seemingly impossible feat. It has been truly an honor, and I am so lucky to have had the opportunity to work with you all."

                https://www.theregister.co.uk/2019/04/12/astronomer_schools_sexists/

                https://www.facebook.com/photo.php?fbid=10213326021042929&set=a.10211451091290857&type=3&theater

                So at least Katie has the class to try and keep the record straight, despite both mainstream and social media's attempts to try to spin things differently.

                Similarly inaccurate statements were made about Margaret Hamilton's role in the Apollo program, but right off the top of my head I don't recall whether or not she tried to correct these statements. I specifically noted that the captions on Hamilton's photos usually tried to claim that the huge stack of folders (program code listings) were all her own personal work, while the articles that those photos were attached to usually pointed out (buried somewhere in the text) that this was in fact the product of a relatively large team.

                Now don't get me wrong here. I don't mind folks calling out these ladies' contributions, and maybe even playing them up a bit. But I do mind them just flat-out lying about the situation, as they are so wont to do these days.

                At this point I have a completely different question to ask, though: Why are we being shown the output from data collected back in 2017, when apparently there is much more current data available now?

                1. Antron Argaiv Silver badge

                  Re: "Mostly wrong on both counts..."

                  I think the object here is to show that women can make significant contributions too. And to remind us that they are under-represented, as a group, in the cutting edge workforce (those that do important and interesting things, like image black holes, design computers, and develop new medicines, etc). We need more women in science and engineering, and this serves as a reminder that we need to do more to encourage and support them in these fields (and others). It makes no sense to discourage half your potential talent pool because of their gender (or race, for that matter)

                  I don't think caption accuracy is tremendously important here, and Dr. Bouman seems somewhat taken aback by all the publicity, to which she has reacted, quite correctly, by pointing out that it was a team effort, and she was only part of a much larger team. I find it hard to fault her. She is an attractive, smart, young lady, working in a male-dominated field, which is why the photo is being circulated so widely by the media. Most of them are still using their lizard brains and are thinking first of what sells papers (or bits) and not too worried about the accuracy of their captions.

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: "Mostly wrong on both counts..."

                    Well, in general it's not just captions but also clickbait headlines and such. A lot of people probably don't read much beyond that stuff, and I doubt that many of them read entire articles through and through, where (as I said) the truth of such matters is often buried down somewhere near the middle or bottom. (I confess that I don't throughly read many articles today, either; rather I often just skim them looking for exactly these types of situations.)

                    As for Margaret Hamilton (and since social media is all the rage these days), compare these two things: A recent tweet from MIT (emphasis mine) vs. a tweet from the BBC. One is far more accurate than the other. You'd think that a place like MIT would at least try and get this right, and certainly her work is old enough now that there really should be no questions about it. But as I said earlier, I don't recall whether she ever tried to correct the record here; I'm guessing not.

                    MIT - Right: MIT computer scientist Margaret Hamilton w/the code SHE WROTE that helped put a man on the moon.

                    BBC - 1969: Margaret Hamilton alongside the code that got us to the moon

    2. Anonymous Coward
      Anonymous Coward

      Re: BBC: what?

      You obviously missed the adverts for BBC management trainees which suggested that an Oxbridge English degree was a jolly good starting point (unless you wanted to be stuck in a lab somewhere.)

  6. MonkeyBob

    Bigger, better

    Is the next step in this to use space based telescopes to create an even larger 'dish'? Or does the data collection make this unfeaseable?

    1. DJO Silver badge

      Re: Bigger, better

      Assuming the subject is roughly at a right angle to the ecliptic then observations 6 months apart will give an effective telescope of 2AU which should be big enough. Correlation of observations would be a bitch but not impossible.

      1. Anonymous Coward
        Anonymous Coward

        Re: Bigger, better

        I thought about this, but to do interferometry you need to be looking at the same time, so this doesn't work I think.

        1. Yet Another Anonymous coward Silver badge

          Re: Bigger, better

          True,you need something in space at the same time.

          There was a Japanese space VLBI mission halca

          but you really need a big dish (for sensitivity) and the satelite was pretty small

          1. Anonymous Coward
            Anonymous Coward

            Re: Bigger, better

            If you can put telescopes in heliocentric orbits, and solve the data & synchronisation issues (of which I guess the data volume one might be severe), then can you get away with less sensitivity because you can make very long observations? My guess is that perhaps you can't, for this purpose, because the accretion disk may be changing significantly on timescales of a few days, so really long observations won't work.

            1. MJB7
              Boffin

              Re: Bigger, better

              "the accretion disk may be changing significantly on timescales of a few days, so really long observations won't work."

              Exactly right. This is why they couldn't image the BH at the centre of the Milky Way galaxy. It's less massive (and hence smaller radius) than the one they did image, but it's a *lot* closer, so it would have twice the apparent size. The problem is that because it's physically so much smaller, the variations happen on a time-scale of hours, and they just couldn't accumulate enough photons to get the necessary sensitivity.

  7. mdubash

    What am I missing? I though radio telescopes were impervious to weather conditions.

    1. TRT Silver badge

      Space weather!

    2. Baldrickk

      To most weather, yeah.

      I imagine thunderstorms will add noise, and I've heard stories of nearby microwave ovens* but also people and birds on the dish causing noise too.

      *iirc, the issue with the microwave wasn't in it's use - the shielding was adequate, but the physicists hungry for their lunch would open the door while it was running instead of stopping the oven first. This led to microwave leakage as the door opened before it properly cut out. Not anything to worry about normally, but the dish cared.

    3. Tom 7

      Not at the frequencies involved.

      where liquid water in the form of (moving) clouds seriously messes up the signal.

    4. Anonymous Coward
      Anonymous Coward

      From what I understand, they needed to capture the same "image" at the same time across the radio dishes. They would need to capture the difference in timing depending on the location (so the ones closer to the equator would receive the same image as the south pole dish slightly ahead). Atmospheric conditions can slow light by measurable degrees, so you aren't able to identify the timing accurately.

      Plus the wider weather aspect was they couldn't ship out the south pole data until winter had passed.

    5. richardcox13

      These are technically radio, but is closer to infra-red rather than conventional radio wavelengths.

      "Sub-millimetre" is much shorter than the cm scale of microwaves and hundreds of metres of LW..

    6. Stephen Beynon

      Wind

      Steerable radio telescopes can certainly catch the wind. At high wind speeds this can damage the telescope, and they generally have to be put into a safe position to ride out storms.

  8. Conundrum1885

    He filled hard disks

    Also I think they did back up the data, though He filled drives are more reliable there were one or two problems.

    Its incredible work to be sure. May one day try out my version using LNBs and atomic references discussed in here a while back.

    1. Tom 7

      Re: He filled hard disks

      On the TV program I saw on the evening of the release they said that it was incredibly difficult to back up the drives. If you consider that a drive may fail on downloading the data then its as likely to fail when making a backup so that gets you nowhere* and costs a shitload more**.

      *unless you have an MBA in the loop who (by definition) doesnt understand the process.

      ** and the backup disk money would have been spent on the MBA anyway.

      1. ibmalone

        Re: He filled hard disks

        I'm not sure that's entirely true, the drives are being moved, so backing them up prior to that would escape any damage caused in transit. Also simultaneously writing to two sets of drives would avoid the fail-on-read problem (and also any fail on write issues, well, not avoid, but do (1-p)^2 on the failure rate). But with this amount of data some of it will be missing or corrupted anyway, so their algorithm for combining it likely has to deal with the missing observations and a corrupt hard-disc in petabytes of data may well just be another source of noise.

      2. Doctor Syntax Silver badge

        Re: He filled hard disks

        "the backup disk money"

        Tape?

        It was probably a financial decision. The bigger risk would have been loss in transit rather than H/W failure during transcription. Anything from rough handling via over-zealous customs official to a crash, and not forgetting the packages that disappear into a warehouse and are never found again due to theft or incompetence.

        1. Anonymous Coward
          Anonymous Coward

          Re: He filled hard disks

          CERN uses tape.*

          * Thanks, Reg!

        2. JJKing
          Joke

          Re: He filled hard disks

          packages that disappear into a warehouse and are never found again

          Try looking in Warehouse 13. Lots of items ended up in there.

    2. Anonymous Coward Silver badge
      Coat

      Re: He filled hard disks

      They tried to back up, but the disk arrays wouldn't fit on the photocopier.

    3. This post has been deleted by its author

    4. JJKing
      Facepalm

      Re: He filled hard disks

      Also I think they did back up the data

      Man, that is a shit (or ship) load of floppies required for that backup.

      1. Antron Argaiv Silver badge
        Linux

        Re: He filled hard disks

        [flashback: remembering downloading and then copying the 13 disk images necessary to install early Linux]

  9. Camilla Smythe

    Helium filled at high altitude.

    Pffft.

    Just stick normal drives in a sealed cabinet held at one atmosphere for half an hour to equalize prior to and then whilst they are operating. It's not Black Hole Science.

    1. Jellied Eel Silver badge

      Re: Helium filled at high altitude.

      First make your cabinet.. Then use helium so you can call the helpdesk and say your disks are squeaky. Or floating.

      I guess it boils down to budget, especially stuff like transport and construction costs in a pretty remote location.

    2. Anonymous Coward
      Anonymous Coward

      Re: Helium filled at high altitude.

      Now arrange cooling for all that, and make it easy to replace drives without taking the whole environment down &c &c.

    3. ibmalone

      Re: Helium filled at high altitude.

      Commercially available sealed He drive (and they tend to be the high capacity ones) vs building, maintaining and running a pressurised cabinet at every site. You'd need to get quotes, but I'll bet on the first option being cheaper.

      1. Yet Another Anonymous coward Silver badge

        Re: Helium filled at high altitude.

        It's was a pain keeping equipement working on Hawaii at 14,500ft -can't imagine how hard it is for ALMA.

        CPU fans don't work because there is only half as much air as they expect, we tried sealing cabinets and pressurising them but its expensive, tricky to get cables in/out and where does the heat go ?

        This was before water cooling was easily available. We generally just used slightly lower than cutting edge machines, speed wise, and put extra case fans everywhere we could

        1. harmjschoonhoven

          Not so high

          @Yet Another Anonymous coward

          ALMA is located at the uninhabitable Chilian Atacama desert at 5000 meters above sealevel. The operations Support Facility (OSF) where the primary observations are done and where the data are stored, is located at 2900 meters.

          1. Yet Another Anonymous coward Silver badge

            Re: Not so high

            >The operations Support Facility (OSF) where the primary observations are done and where the data are stored, is located at 2900 meters.

            But the support controls, power drives and all the detector and front end electronics needs to be on the telescope - in a place where even plastics break down in months. The UV levels were causing laminated windshields in cars to fog.

            1. TRT Silver badge

              Re: Not so high

              I believe the drives were totally sealed against sunlight, hence why the platters had shingles. Or something.

            2. John Brown (no body) Silver badge
              Coat

              Re: Not so high

              "The UV levels were causing laminated windshields in cars to fog."

              Good for working on your tan though!

              The extra thick one with the sunscreen bottle in all the pockets -------------->

              1. This post has been deleted by its author

        2. muhfugen

          Re: Helium filled at high altitude.

          It never occurred to you that fans are commodity components and are readily available at higher speeds?

          1. Yet Another Anonymous coward Silver badge

            Re: Helium filled at high altitude.

            > fans are commodity components and readily available at higher speeds?

            On your home made PC maybe, on a Sun workstation in the 90s they weren't, and every failure had to be shipped back to the mainland for repair.

            Plus a fan running at higher speed in lower pressure doesn't necessarily generate the same air flow - it tends to just burn out it's own motor - especially before they were speed controlled brushlless DC motors like today.

    4. Annihilator

      Re: Helium filled at high altitude.

      An untethered helium drive will naturally drift up to higher altitudes over time anyway, so I suspect that they had an abundance of drives that had piled up against the radio dishes, having long since drifted away from an inattentive child who was carrying it around on the end of a string but sadly let go.

      1. Andy The Hat Silver badge

        Re: Helium filled at high altitude.

        When my mate was working for GRO, I believe Canary Islands scopes (at around 2200m) used standard drives. I'm not sure what Mauna Kea have done b.h. (before helium) but they're at 4200m and I'm almost certain they had standard pcs with drives on the desk ... Perhaps it's a potentially high failure rate issue with low flying heads in modern drives rather than drives not working at all?

      2. Mystic Megabyte

        Re: Helium filled at high altitude. @Annihilator

        Firstly; LOL!

        I once spent a day selling Helium balloons at an exhibition (please don't ask!). I must have done the squeaky voice thing a hundred times that day. And yes you were correct, the ~20mtr high ceiling was populated with loads of balloons. They must employ someone with an air rifle to get them down.

  10. O RLY

    BBC stole that tweet about the two scientists without correct attribution.

    Here is the original tweet from MIT CSAIL:

    https://twitter.com/MIT_CSAIL/status/1116007460039483392

  11. John Savard

    Tape

    Using tape instead of hard drives to ship the data sounds like a good idea. But I suppose they could have had reasons. All the sites would have to be using the same kind of high-capacity tape drive. Plus, isn't tape capacity lagging behind hard drive capacity these days? And the tapes would only be useful for shipping data, whereas the hard drives could always be used later as more hard drive capacity.

    Still, for many of us, using the helium drives to only ship the data as far as down the mountain to the nearest city would make sense - as it might be easier to get tapes instead of disks through customs for ordinary people like you and I, and we wouldn't have petabytes to ship. Of course, the site in Antarctica didn't have that option.

    1. ibmalone

      Re: Tape

      Tape capacity is catching up again, and will probably pass HD soon as HD are at the limits of what's physically possible while tape spent a long time not being developed much, LTO-8 can take 12TB, though in 2017 they'd have likely had to go with LTO-7 and 6TB, or one of the proprietary formats which can take more. Bandwidth is comparable, though not as good, about 350MB/s (uncompressed, tape figures often quote compressed due to use as backup), vs about twice that for 12TB helium drives. Drives are expensive, but individual tapes are a lot cheaper than really big HDs of the same size.

      However I'd guess the need for a drive makes it less attractive for this application where you want to be able to maintain high bandwidth and continuous writing (also there's the wraps thing which I don't know how it plays out in practice, the wraps are written in alternating directions, but not sure if there's a delay at the change-over). Likely cheaper overall to have an array of HDs than an array of tape drives. That 350TB/day is about 4GB/s, so you'd need over 10 tape drives with two changes (three tapes) a day with LTO-8 to keep up (probably more, since you'd only just keep up), while about 30*12TB HDs will provide the capacity and easily have the bandwidth. A 12TB helium drive is about £400, a basic LTO-8 drive is about £2000 (tapes for it are ~£60), so the break-even cost-wise is about 6 tapes to one drive, which obviously has much less bandwidth. (Doesn't account for the extra hardware to host the HDs though, which will start to balance it out once you've got more drives than can connect to a single server.)

      1. Yet Another Anonymous coward Silver badge

        Re: Tape

        You don't need to keep the raw signal data for very long once you have analysed it. Some of the intermediate results are worth archiving for future improvements.

        The problem with tape is the time to read/write. The disk storage system is sized to barely keep up with the data rate, so having enough spare capacity to buffer it so that it could be written to tape at the same rate - together with the number of tape drives you would need - is expensive.

        Then you have the same problem at the processing site. The processing for these signals is read the data once into the correlator and then the drives can be sent back for reuse.

        Funnily enough the detectors seem to have kept the same rate of improvement as the drives. 20 years ago we shipped 6Gb scsi drives home with a few nights data, now they ship racks of 8Tb drives.

        For real data firehose nightmare - look at the Square Kilometer Array !

        1. Jellied Eel Silver badge

          Re: Tape

          For real data firehose nightmare - look at the Square Kilometer Array !

          Obviously big data researchers need to pool resources. Black hole storage is the obvious solution, once issues around 'read' cycles and error correction are overcome.

        2. John Brown (no body) Silver badge

          Re: Tape

          "You don't need to keep the raw signal data for very long once you have analysed it. "

          I'd not accept that as a blanket statement. There are still new discoveries coming from new analysis of old data.

          1. Yet Another Anonymous coward Silver badge

            Re: Tape

            But not the very raw signal - once it has gone through the correlator and filters it gets a lot smaller.

            Ironically we don't keep as much archive data as we once did because of the cost of analysis. Even poor grad students are expensive to keep

        3. ibmalone

          Re: Tape

          This is why I was looking at the data rate and talking about multiple tape drives, you can actually get decent bandwidth. Back of the envelope a tape drive has about half the write bandwidth of an HD (is there a longer write latency? anyone care to chip in?), compression can bring it up, but I'm guessing raw observation data is not very compressible. From the description they've had to do work to make sure the data is being distributed across the discs, it's certainly more than one or two could handle (while you could RAID-0 it instead that's unattractive because one missing disc would corrupt all the measurements), but it looks like the resulting bandwidth of the system is more than they need. There's two things determining the size of the disc array, the total capacity needed and the bandwidth needed, the difference with HD is adding capacity automatically adds bandwidth, with tapes you need to increase the number of more expensive drives.

          Correction to what I posted previously though, the bandwidth is 8GB/s (64Gb/s from the article) I've just realised why going from 350TB/day to 4.4GB/s is missing a factor of two... The 32 drives in one of those DBE should be able to handle something like 2-3 times that (may well be bottlenecks elsewhere), but it changes the numbers for my back of the envelope for tape as that's based on the minimum number of drives you can get away with, 24 would only just manage on paper, so 32 tape drives should have the bandwidth, but would cost more than the 128 HD for a site. Hadn't accounted for the read at the other end, though you don't need the same bandwidth there. The cost calculation may change for longer observation periods at inaccessible sites; the longer you need to keep the raw data before you can ship it the more storage you need, while your bandwidth requirements stay the same.

          By bizarre coincidence, the SKA was one of the projects I applied to do a PhD on, but didn't get that project (maybe just as well, hadn't realised they still haven't built it yet!).

  12. Anonymous Coward
    Anonymous Coward

    Has anyone made a "all those drives something something all the porn on the internet" joke yet?

    1. Korev Silver badge

      I used to work with someone who went to work for a "specialist video website", they have a huge amount of data. I haven't spoken to him for a while but I imagine they have is greater than what they have in this article. The big difference of course is that it doesn't arrive in five days.

      1. Glen 1

        It comes much quicker than that...

        1. Korev Silver badge
          Coffee/keyboard

          Very good Glen -->

  13. bodi_thung

    The usual exaggeration

    "locate an orange on the surface of the Moon from Earth"...

    You could presumably "see" an orange on the moon but find one? Not likely - as magnification increases, the field of view decreases. This would be like trying to find a needle in a haystack, in the dark, with a laser for a flashlight.

    1. lowwall

      Re: The usual exaggeration

      It's rolled under the sofa anyway.

      1. Yet Another Anonymous coward Silver badge

        Re: The usual exaggeration

        Depends how hot the orange is.

        A moon temperature orange on moon temperature rock would be tricky.

        A 1000 deg C infrared Orange is doable, a million degree orange you could see yourself

    2. Adam 1

      Re: The usual exaggeration

      > This would be like trying to find a needle in a haystack, in the dark, with a laser for a flashlight.

      So you're saying there's a chance?

  14. Deryk Barker

    Careless?

    Apparently somebody involved with this article cannot read.

    The image from the BBC clearly says that the large pile of fanfold listings is the *code* that got the Apollo astronauts to the moon, whereas the HDDs contain the *data* for the black hole image.

    Comparing apples and elephants.

  15. rmr

    Excellent article

    Really enjoyed reading the details behind the image.

  16. Uncle Ron

    More?

    I may sound like an old coot here, but please tell me that there is more benefit to this huge effort than just a fuzzy picture of a bagel that appears to be getting too toasted on one side. I'm not a physicist or a mathematician or even a photographer, but I'd like to see bullet points as to what is being learned from whatever else this Mount Everest of data shows. Like, why do they keep referring to it as a "shadow" and not an actual view into the gaping maw of eternity itself. What's going on here?

    1. ibmalone

      Re: More?

      It's a good question. The data is going to be mined for years I'm sure, to test different theories on black holes and accretion discs. For one thing, just the shape of that halo confirms theories about how accretion discs behave around black holes, before we just had models, now we know they are right (sometimes), it also allows confirmation of the mass of the black hole, again this was previously only known indirectly, so we can be more confident about those methods. One thing that's gone largely unmentioned, they've got images from four days (nights?), showing the structure is stable over that time period. The ring structure confirms a lot of things about gravitational lensing and how light behaves near the event horizon.

      The horse's mouth is a good place to start: https://eventhorizontelescope.org/. If you really want to get into the details, the first few papers are at IoP (open access). Papers V and VI are the most interesting for what they can actually tell from these images.

      Like, why do they keep referring to it as a "shadow" and not an actual view into the gaping maw of eternity itself..

      In the simple sense we can never see into a black hole, because no light can escape. There is the phenomenon of Hawking radiation, but it's produced outside the event horizon. So all you can see (if that's the right word), is nothing, a shadow. There's slightly more to it though, the shadow, the region from which you don't see light, is distorted by lensing effects and is bigger than the horizon, if the black hole was rotating significantly it would be noticeably oblate. Slightly disappointingly, M87 doesn't spin much, but they can at least confirm its shadow looks as expected.

      But the simplest bullet point? Ever since they were first hypothesized, people have thought black holes couldn't exist for one reason or another, it's only really in the last three decades they became widely accepted, and now you're looking at one. (Quasars were discovered about 1960, but it was a long time before black holes were accepted as a possible explanation.)

      1. Richard 12 Silver badge

        Re: More?

        They also have multiple frames, so the boffins can look at how the accretion disk changes over time.

        The ones I've seen appear to show the bright glowy part moving, which must be providing really interesting information to the people who study such things.

      2. ibmalone

        Re: More?

        An important point I missed (if you're feeling generous you can say 'lensing' covered it): the bright ring is due to accretion, but more fundamentally it's the exterior of the photon sphere https://astronomy.stackexchange.com/questions/30317/m87-black-hole-why-can-we-see-the-blackness/30360#30360. That is to say, we are actually looking at the effect of light orbiting a black hole, basically confirming the Schwarzschild or Kerr equation holds outside the event horizon (there's a bit in one of the papers about testing the Kerr nature of M87).

  17. Paul Johnson 1
    Thumb Up

    1/3 mm total error

    For me the most stunning thing is the required accuracy. The observations were taken at a wavelength of 1.3mm. A telescope works "perfectly" if it is accurate to 1/4 of a wavelength (the Rayleigh Limit, not to be confused with the equally important Ralyleigh Criterion which is about the aperture). So the total error in the system had to be kept down to 0.4mm across the Earth. If I mention that the moon creates tides of up to 1m in solid Earth every day, and one of the telescopes was on a moving ice sheet, you will have some idea of the practical difficulties.

    Timing was equally important. 0.3mm is 1 light-picosecond, so clocks accurate to 1ps had to be used to time the incoming signals.

  18. elgarak1

    Back when they started doing VLBI, they used video tape. Storing the data diagonally got them higher capacity, and was more error-correctible than any other storage available at the time.

    I gave a seminar talk on it in the late 1990s, and back then video tape was still pretty much state-of-the-art.

    1. Glen 1

      Storing the data diagonally

      Like VHS

  19. Anonymous Coward
    Anonymous Coward

    Imagine if they somehow were able to stream the data to S3

    ...it would have bankrupted them after one computed pixel.

  20. Bob.

    Re: I was just musing the other day that M$ might do this and low and behold!

    Turing got it right [near enough] in the 40s, when he pre-supposed Moore's law and forecast Gigabit memories/storage by the end of the century.

    Tape and HDDs have their quirks and amusing stories.

    They first did amazing things with paper tape loops and magnetic ferrite core arrays.

    And the Manchester William's Tube CRT phosphor dot thing. What was all that about?

    I remember seeing an IBM 5 platter[?} removable 5MB 'cassette' at school. Our teacher brought it in from a SysAdmin friend to show us.

    Tapes suffer 'Print Through' and stretching and Wow and Flutter.

    Ever seen a fingerprint or speck of dust on a hard drive platter?

    Probably only after the head has crashed to the surface. They don't like those, which is why they are sealed.

    I remember early reports of HDDs popping their seals in unpressurised aircraft holds. Dunno if true.

    The technology doesn't really matter in the end. It will always evolve.

    Backup, backup, backup. Multiple copies in multiple places and the data should survive. At least for a while.

    Whether you can find it in 10 years time is another matter.

    1. IJD

      Re: I was just musing the other day that M$ might do this and low and behold!

      How about mercury delay line memories, as used on UK's MOSAIC computer? (which my father-in-law Bill Chandler designed and built with Allen Coombs after Colossus). Five tons of mercury in a hundred steel tubes, kept temperature stable to better than one degree...

      https://mraths.org.uk/?page_id=582

  21. Anonymous Coward
    Anonymous Coward

    A quart into a pint pot?

    1024x8TB HDDs is only ~8PB so how did they store 14PB

  22. Milton

    One optimist, one cynic ...

    One optimist, one cynic ... are combined, somewhat unprepossessingly, in this tired old carcass.

    The optimist is delighted by this achievement. The world seems to be in full-on 'Lunatics are taking over the asylum mode', and this amazing effort is a grand antidote. It's good to be reminded of what the best of humanity can achieve.

    The cynic, on the other hand, observes that the world is full of dumb-shit politicians, liars, hypocrites and soulless greedmongers making life worse for everyone except themselves; and that the only people doing constructive, worthwhile, decent, positive things for the species are the generally humble, badly-rewarded ones: teachers, nurses, artists, writers and above all scientists.

    While the teacher who educates your child has to visit a food bank (maybe because she buys lunch for some of the most deprived kids from her own pocket) your useless, evasive, podgy, halfwitted, gob-on-a-stick Member of Parliament is bagging a £250k profit selling a house the taxpayers paid for.

    I am a believer in the importance of every human life, but I cannot pretend that some humans appear to be way better for our species than those who seem to be ... just a waste of oxygen. I've never forgotten the independent analysis done after the 2008 global financial crisis, comparing the social usefulness of top bankers with the people who cleaned their offices, taking into account their netted financial contributions and costs to society. The minimum-wage cleaners came out a clear winner; the bankers balanced out as highly costly parasites.

    If you asked me whether I'd sacrifice the entire contents of both houses of Congress in the US, and the Commons and the Lords in the UK, to save the life of a single Einstein-calibre scientist (let's say, a Fields or Nobel nominee)—hell, I'd really have to think about that. The difference, by both character and intellect, is a grim, depressingly vast gulf.

    And I can't help feeling that a species that hasn't the vision, wit or fortitude to recognise and rid itself of an existentially damaging parasitic infestation, in the shape of its most selfish, dishonest and foolish members, may deserve the fate that those people are bringing upon it.

    Yep: cynic.

  23. 89724102172714182892114I7551670349743096734346773478647892349863592355648544996312855148587659264921

    "Oh my God, it's full of stars..."

    I like that her name is Bouman

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like