back to article Sometimes shining a light on a nuclear problem just makes things worse

Welcome to On Call, The Register's receptacle of woe for those unfortunate enough to be at the beck and call of that species we call The Customer. Today's tale concerns a duo we will name "Pete" and "Dud" because, frankly, we're of a certain age and still remember Not Only... But Also with a fondness. There were, however, no …

  1. 2+2=5 Silver badge
    Coat

    Scintillating!

    .

    .

    .

    .

    .

    [icon: coat, gone]

    1. Anonymous Custard Silver badge
      Trollface

      Re: Scintillating!

      Now you're just being flash and showing off...

      1. Anne-Lise Pasch

        Re: Scintillating!

        I see what you did there.

    2. HildyJ Silver badge
      Pint

      Re: Scintillating!

      I come to el Reg for the emissions.

      1. ds6

        Re: Scintillating!

        You can't change the current reality and the zone kick you out.

        Oh, wrong emission?

    3. Anonymous Coward
      Anonymous Coward

      Re: Scintillating!

      Well, he's a bright spark, 'innt he?

  2. Pascal Monett Silver badge

    Solved by a stroke of sheer luck

    A component was supposed to be impervious to light, but wasn't. You can't guess that that is the problem - something has to reveal it. Had "Dud" done literally anything else, they would not have found the root of the issue and we would be reading about a product failure - or likely not reading about it at all.

    1. Ian K
      Stop

      Re: Solved by a stroke of sheer luck

      To be fair to "Pete" there's no indication from the article that he was stymied by the problem, or even that he'd got very far into troubleshooting after getting the diagnostics running. There's every chance that checking the integrity of the light shields was towards the top of his check list for investigating high readings from scintillation counters - Dud's theatrics simply gave him an early clue.

      1. hmv Silver badge

        Re: Solved by a stroke of sheer luck

        And even if he'd run out of items on the checklist, torn out all of his hair, and was completely stumped, the problem solving doesn't stop. Unless you can slopey-shoulder it onto someone else.

        1. swm Silver badge

          Re: Solved by a stroke of sheer luck

          I know a scientist who always liked to do "dirty" experiments. In other words, he didn't set up his experiments to measure just one thing but ran them in a somewhat uncontrolled environment. He said you discover more that way about things you don't expect.

          1. keithpeter
            Coat

            Re: Solved by a stroke of sheer luck

            How does your colleague separate out the factors once he has seen a promising result?

    2. Sgt_Oddball Silver badge
      Coat

      Re: Solved by a stroke of sheer luck

      Never underestimate the problem solving skills of sheer-luck, Holmes...

      Mines with one with pipe and magnifying glass in the pockets.

  3. Neil Barnes Silver badge

    A favourite trick used to be to 'help' a colleague trying to fix a recalcitrant umatic tape player by ensuring there was plenty of light on the job - thereby confusing all the IR sensors that tracked where the tape was loading...

    And I can recall as a very young engineer building radios with detector diodes that were light sensitive, so they worked in the daytime but at night picked up an impressive 100Hz hum.

    1. Benson's Cycle

      Ah, the joys of (inter alia) the OA47. You bring back memories.

      A lot of those old components were light sensitive. Then Mullard realised that the OC71 was being used instead of the very expensive OCP71 and started putting a blob of opaque silicone in the OC71 cases. And then people discovered that the blob was too small, and a few minutes in the centrifuge would restore the light sensitivity.

  4. Anonymous Coward
    Anonymous Coward

    Random crashing

    I used to work for a company that produced motor controllers for various platforms, including fork-lift trucks. A specially modified one was used for testing, with a plate mounted on the back so that it was easy to replace the controller, attach instrumentation and replace the CPU - this was in the days of one-time programmable devices and very, very expensive, gold-flashed EPROM parts for development.

    One of the engineers was driving this thing back-and-forward one day testing some new software. It was generally working fine, but every now and then the watchdog would trip, putting the system into a fail-safe state.

    Much time was spent reviewing instrumentation and code, but, as the cause of the problem could not be found I decided to offer some assistance. I asked to see the problem, and, sure enough, the controller would enter the fail-safe state once in a while. Strangely, this always seemed to happen when the truck was moving backwards - which was when I noticed the shaft of sunlight that travelled up the back of the truck, over the (uncased) controller and into the window of the EPROM CPU. There was just enough UV in the sunlight to cause the CPU to glitch, showing that those little stickers you're supposed to put over the window are important (though never used during "agile" development phases)!

    1. Anonymous Coward
      Anonymous Coward

      Re: Random crashing

      I discovered similar with UV Pic micro that was being used to develop an engine tuning device, we'd been called out to investigate the gas analyser and they were convinced it was faulty because it would sometimes show unusually high figures.

      Turns out the guys who designed the product had no idea about the well documented light sensitivity of those and many other MCU devices with UV windows, a simple piece of black tape and a large bill were all it took to solve the prpblem

  5. chivo243 Silver badge
    Facepalm

    Yes! Move the person out of the way!

    Long ago, when wireless networking was best effort (and on 2.4ghz), we had one problem area in the school, call it one problem room. When ever the students started all writing a test on the laptops, the wireless would drop out of this room. We busted our heads and asses trying to get our minds around this problem. After a few observations of the issue, the coin dropped. Every time there was test on the laptops, the instructor would position themselves squarely in the doorway(to observe for cheating etc). This instructor didn't miss any meals, and didn't know when to push away from the dinner table. The AP was directly across the hall from their door and their large carriage was blocking the signal. We asked them to move away from the door, and presto, connection to the network was restored! We couldn't bring ourselves to tell them it was their own fault...

  6. Empty1

    Mice too

    The rare appearance of bright sunshine at certain times of day caused some old rollerball mice to behave eratcally

    1. JassMan Silver badge
      Headmaster

      Re: Mice too

      Maybe they weren't mice but eRats. why else would they behave erat cally.

      Sorry. I didn't mean to be cruel.

      1. A.P. Veening Silver badge

        Re: Mice too

        You really should be pun-ished for that ;)

    2. Cessquill

      Re: Mice too

      Same happened on Sun Microsystems early optical mice (with a metal mouse-pad). Took a few days to work out why somebody's mouse crapped out at the same time every day.

      1. squigbobble

        Re: Mice too

        Even later optical mice could be banjaxed by a piece of paper. One of my housemates discovered this in the early 2000s when she tried using a blank piece of paper as a mouse mat so there was nothing for sensor to read as she moved the mouse.

        1. phuzz Silver badge

          Re: Mice too

          I had the opposite problem in a previous job.

          The mouse hooked up via the KVM to all our servers sat on a metal tray, with a crinkly white gloss finish. Even when left still, it would detect minor movements, so the cursor would jitter on the screen, and the screensaver, and more importantly, the screenlock, would never engage.

          I solved the problem by sticking a post-it note under where the mouse was left at rest, and less reflective surface banished the phantom movements.

          More fun can be had by sticking the post-it to the bottom of a colleagues mouse, but they usually work it out quite quickly.

          1. imanidiot Silver badge
            Angel

            Re: Mice too

            Clear cello-tape under the post-it works very well if they are expecting only the post-it. (Not that I've ever done that to anyone...)

            1. bpfh Silver badge

              Re: Mice too

              For wired rodents, a piece of post-it note strategically placed over the centre pins of the usb plug before carefully plugging it back in the back into the computer, isolating the 2 centre data connectors, but not the power lines on the left and right of the plug. Mouse lights up (if it’s a LED and not an IR laser), but no movement data is sent. That generally takes someone a few more minutes especially when they turn the mouse upside down, no post it and a shining red light saying all is well...

          2. Gene Cash Silver badge

            Re: Mice too

            That still happens... I solved it with xinput disable "$MOUSE" as part of the shell script to turn off all the monitors at the end of the day, and xinput enable "$MOUSE" after you hit enter to wake things up again.

        2. Steven Guenther

          Re: Mice too

          A company I worked for gave out mouse pads at trade events. The pads had very cool hologram logos on them. They worked great for ball mice. The results for optical mice were truly bizZaAAaaRRr.

      2. james_smith Silver badge

        Re: Mice too

        I remember when a new, non-technical manager too over our department. He decided the scruffy looking metal mouse mats had to go, and over one weekend they were replaced by company branded rubbery ones. Cue howls of protest on the Monday when we programmers and admins couldn't use or mice as they needed the metal mats! Worked out OK in the end, as we ordered new rollerball Sun mice that were more convenient. The cost was deducted from the new manager's salary, since Sun branded stuff was so exorbitantly priced.

      3. Venerable and Fragrant Wind of Change

        Re: Mice too

        Those mice were always crap. Poor, coarse, control; mouse mat that was unpleasant on the hand. Troubles with natural light were the least of its problems.

  7. ocelot

    I can remember similar : on a project requiring extreme frequency stability and type approval, we installed a state-of-the-art Temperature Compensated Crystal Oscillator with a ceramic encapsulation.

    We installed it on a PCB with a plastic case around it.

    On test, it functioned perfectly until somebody turned on the light in the test chamber and it hopped frequency.

    Turns out the plastic case and ceramic TCXO encapsulation was transparent to infrared light from an incandescent bulb.

    1. heyrick Silver badge
      Happy

      I can remember similar too.... Didn't a recent(ish) Pi freak out when people took flash photos of the board in use?

      1. Liam Proven

        It did indeed:

        https://www.theregister.co.uk/2015/02/08/raspberry_pi_2_camera_flash_glitch/

        1. swm Silver badge

          The SEAC computer used Williams storage tubes. During a demo a reporter took a flash picture and erased memory.

  8. Kevin Johnston

    Radiotherapy systems

    I was servicing a recently installed system at a hospital and they mentioned it would trip if they used the rotation option as part of the treatment. It was only in a small section of the arc but nothing was obvious until I noticed a gap between two panels with a ribbon cable behind which carried various monitor signals. Popped down the local Spar shop to get some tinfoil which I wrapped round the cable and then added an earth wire to and as if by magic, all now OK. A proper sheiled cable was sent to replace my homebrew efforts but I can honestly say it was no better and looked boring in comparison.

  9. STOP_FORTH Silver badge
    Unhappy

    Of course we 'ad it tough...

    scraping the paint off germanium thingamabobs to make 'em light sensitive devices. It were never the same wi' silicon.

    1. Symon Silver badge
      Joke

      Re: Of course we 'ad it tough...

      https://www.google.com/search?q=oc71+transistor

      Good old geranium transistors, blooming marvelous!

    2. Tom 7 Silver badge

      Re: Of course we 'ad it tough...

      Twere tho - we made some chips that worked fine when on the test bed. Packaged the buggers up and they stopped working. Had to show a demo of a repeater in a box with a light to keep it working while the cct was redesigned.

  10. Tashritu

    OC71

    I met, and cursed, the man at Mullard who changed the black paint on the OC71 to a black filling so you could not scrape the paint off and get an OCP71 (the photosensitive version). They sold the photosensitive version for a much higher price by leaving the paint off and relabelling them. A long time ago but I think 36 shillings and sixpence with no paint and nineteen shillings and sixpence painted! Bastards!

    1. STOP_FORTH Silver badge
      Happy

      Re: OC71

      That's the ones! Think positively, they charged you minus seventeen shillings for the extra paint!

    2. Anonymous Coward
      Anonymous Coward

      Re: OC71

      There was a reason for that though. The OCP71 were tested as phototransistors and it's the ones that had failed testing that were painted up and sold as OC71. You could use them as OCP71 but in some way they didn't properly meet the spec. Likely they were just a little less sensitive, but still fine for most uses.

  11. Paul Herber Silver badge

    so glad they did the test properly, shows they applied plenty of rigour ...

  12. FlossyThePig

    Pete and Dud?

    Derek and Clive had more unsuitable responses that sounded a bit like "You can't"

    1. Anonymous Coward
      Anonymous Coward

      Re: Pete and Dud?

      Laughed? We nearly errrr... sat

      1. The Mighty Spang

        Re: Pete and Dud?

        we hadn't laughed so hard since grandma died

        1. Andrew Moore

          Re: Pete and Dud?

          Or since auntie Mabel had that unfortunate accident with the mangle...

  13. Anonymous Coward
    Anonymous Coward

    Radiotelescopes also have tricky-to-find issues

    Related: Australia National Telescope Facility's decade-long hunt for an interference source feeding burst signals into their Parkes Telescope. Which was eventually found to be staff opening the microwave before the timer beeped. A tale well told at:

    http://www.swinburne.edu.au/news/latest-news/2015/05/how-we-found-the-source-of-the-mystery-signals-at-the-dish.php

    1. Terje

      Re: Radiotelescopes also have tricky-to-find issues

      There's also the case where a telescope doing VLBI would get out of position for no apparent reason, turned out to be a heater on one side of the "pillar" supporting the dish causing uneven heating and a slight bend moving the telescope a minute distance but enough to throw the VLBI off.

  14. Anonymous Coward
    Anonymous Coward

    An unexpected turn

    Given the mention of a new microprocessor, I was expecting it to be the radiation flipping bits in the silicon.

    1. Brewster's Angle Grinder Silver badge
      Mushroom

      Re: An unexpected turn

      Back in the old days, the bits were so chunky a supernova couldn't flip one.

    2. Robert Carnegie Silver badge

      Re: An unexpected turn

      Just a different kind of radiation!

  15. The Mighty Spang

    read in dud's voice

    so was following the test specifications to the letter, but this one particular instrument was behaving very strangely. Funny.... I thought. I'd not seen an instrument behave so strangly since my Aunt Dolly tried to play the trumpet after drinking half a bottle of gin. Her embouchure was so sloppy her teeth shot out, flew across the room and hit the cat.

  16. chivo243 Silver badge
    Facepalm

    Interactive Displays

    We recently had some LARGE interactive displays start going crazy. But only between 9 and 10 in the morning. Turned out there are cameras along the bezel of the screen to aid in the touchiness, and the sun would shine into this room just right for an hour or so to make it act possessed. Once the seasons changed it was no longer an issue. Plus there are now some foil screens on the window now as well

  17. Pen-y-gors Silver badge

    Dodgy electrons

    Something similar...

    Scene: University observatory, mid 1970s. Small group doing some work with a photometer plugged in to the telescope, to measure the amount of light coming from the target star. Working fine, but suddenly the count dropped. And then went back up again. Then dropped. etc. First theory (aliens) rapidly discounted.

    Eventually worked out what was happening. There was a coax cable running across the floor from the telescope to the recording kit, and Tony had been standing on it when the count dropped. This was final proof (to us) that electricity works just like water - stand on the hose and the water/electrons can't get through.

    In fact - what do we know about co-ax cable? The resistance depends on the shape - minimum when a circle and increases as it is flattened.

    1. james_smith Silver badge

      Re: Dodgy electrons

      Used to be a similar issue with the primitive network cabling at my first job. I think it's what was called "thick net" and had these "vampire taps" attached at each point you needed to connect a computer (or VT in this case). The cabling was pretty ad-hoc, just laying in the floor, and resting your foot on the tap would often cause the networking to go on the fritz as the cable would short out.

  18. IJD

    Reverse case -- the first silicon of first chip I ever designed (an echo canceller for ISDN) didn't work. Looking down a microscope at it to see if I could see anything wrong (suspect was a comparator inside an ADC) as I moved the chip around on the stage it suddenly started to work. Turned out the problem was offset voltage in the comparator, the light from the microscope injected photocarriers into the input stage which cancelled this out so long as you looked at it *just* right -- you had to within a few microns of the "sweet spot", which was in a different place for each chip...

  19. STOP_FORTH Silver badge
    WTF?

    That's nothing, you were looky.

    In the 1980s the telecomms market in the UK was being deregulated. Energis deployed a "figure of eight" network backbone that went from London to Birmingham and then Manchester. The initial fibre-optic cables were wound around National Grid cables by crawling machines that looked like lawn-mowers designed by spiders. (Manual intervention by humans was required every time they came to a pylon.)

    An SDH network was implemented over the fibre and some (or all?) of the capacity was flogged to the BBC so they could distribute uncompressed digital video to their transmitter network and regional studios. In the wee small hours the network would fall over spectacularly but eventually recover.

    When the sun hove into view it would heat up the conducters and fibre-optic cables causing the physical length of the network to increase by a small amount (but very rapidly). The clock adjustment/slew mechanisms in the network were not fast enough to keep up with the rapid changes causing everything to fall out of sync. The cables cooled down more slowly at sunset so this issue was only seen in the morning. I assume this was eventually fixed by increasing the maximum slew rate.

    Eventually, the fibres were incorporated into the cables as they were manufactured. (Cables are replaced on a regular basis, I don't know the cadence.)

    And you tell that to kids today and they don't believe you!

  20. Throgmorton Horatio III
    Coat

    This has sparked a memory

    Around the same time as the events described I worked for a big Pharma company. The company was developing one of the first monoclonal antibody-based anti-cancer therapies, and my role was to devise a test to measure biological activity for early-phase material. We had a nice little assay going using Chromium 51 (medium-hard gamma emmitter) in fresh lymphocytes, and in order to validate the test a second team in another building were also required to reproduce my results.

    All went well at first until one day, a couple of weeks in, they lost sensitivity and started having bizarre high backgrounds and unpredictable results. This came and went over the next couple of weeks, with things getting worse after a period of more intense work, then dropping back to normal levels. The more they tried to fix it, the worse it became, then would suddenly work properly again.

    The gamma counters we used at the time, made by LKB Wallac, were pretty much industry standard and usually used for counting Iodine 125, a softer gamma emmitter and were legendary for reliability. The basic design had the gamma detectors in 2 rows of 6 with lead shielding between to reduce crosstalk, samples were placed in a rack by hand which was then loaded into the instrument for counting over a fixed interval.

    After this had been going on for a couple of weeks I wandered down to talk things through, see if there was anything 'obvious' in their technique that might cause trouble. On reaching the lab, the cause became immediately apparent - the waste bag containing 'hot' waste from previous experiements was being stored under the bench, directly beneath the counter. When the team had previously used Iodine, the gamma from that had been too weak and used at too low a level to penetrate the lead shielding, but the Chromium was being used in much larger amounts, and the higher energy gamma would go straight through a centimeter or so of lead shielding.

    We moved the bag, the assay performed like normal again. The treatment eventually made it to clinical use. Lab coat of course. :-)

    1. Alan Brown Silver badge

      Re: This has sparked a memory

      " and the higher energy gamma would go straight through a centimeter or so of lead shielding."

      At what point do people start thinking about using a waste bin with a water jacket?

    2. rskurat

      Re: This has sparked a memory

      I used P32 once a week and had the usual film badge finger ring with minimal exposure readings for months. Suddenly one month it came up offscale. Turned out I had left the ring for a whole week on the shelf over the waste bin rather than in the drawer under the bench. Big oops, radiation safety was NOT happy with me for a long long time.

  21. SteveCoops

    Two stories I'm sure I've mentioned in these halls before with interference:

    1) Our CEO many moons ago had an apartment in Paris and used to moan that his WiFi (original 802.11b) was unreliable, even after purchasing a full blown outdoor antenna, complete with lightning arrestor. I told him he was talking rubbish so he shipped me out there on the EuroStar and to my astonishment I could stand at the end of the corridor looking at this massive outdoor antenna mounted 5m away at the end of his hallway I lost my signal. Turned out he lived under half a mile from the Eiffel Tower and as you stepped into the hallway where a large floor to ceiling window showed off the Tower you immediately lost all WiFi. The Eiffel Tower was spewing all sorts of EMF. The CEO later told me taxi radios didn't work round here either amongst other weirdness. Ended up putting Wifi repeaters in every room of the apartment.

    2) We moved IT into a new office and I kitted out the place network-wise and I also put up a dozen large LCD TVs for various monitoring uses. These TVs used HDMI over CAT5/6 and were SO unreliable it drove me almost mad. They used to blank out all the time at random times. It turned out after one late night working on the screen resolutions, refresh rates and so on that a cleaner walked in and the TVs that had been fine for hours suddenly all blanked out! It was the IR sensors that kept the lights on! Whenever it detected any movement, it send out a load of EMF radiation that messed up the HDMI over CAT6. Had to replace all the cabling with shielded cabling and then it was fine.

    F F S

    1. Giles C

      I’ve seen the it detector problem myself, doing a wireless survey and the signal goes weird suddenly, we found it was moving into range of the ir sensor.

      I suggested we upgrade the sensors and got told no, so the users had to live with dodgy WiFi when they moved around...

  22. LeahroyNake Silver badge

    There is a reason

    Most Copier manufacturers state in the installation requirements that the device should not be located in direct sunlight. Some even say strong light.

    The worst one I have seen was in an estate agents front window in full glare of the afternoon sun. Normally not an issue as long as the only real effect is to bleach the panels a horrible colour. Unfortunately some idiot thought drilling a few holes in the copier to mount a keyboard was a good idea. They managed to get some of the holes in the right l place but the uncovered test ones managed to let light bounce inside and onto the drum.

    2cm of black nasty tape and telling off and they were mostly happy.

  23. Anonymous Coward
    Anonymous Coward

    Real Nice

    I love stories like this.

    Have a whack of them but I'm still under the "Shut your Mouth" articles.

    Be nice too see more of the old stuff from the Wyse and dec days....

    1. The Oncoming Scorn Silver badge
      Coat

      Re: Real Nice

      Just as long as it's not Ant & Dec.

  24. a_builder

    Many moons ago I was working as a post doc in an underfunded university lab.

    We had bough a vac freeze drier at an auction when Shell closed their Sittingbourne site.

    The freeze drier was missing the stainless Christmas tree that sits on top of it but otherwise worked fine under test. The Christmas tree is very expensive to buy so we asked the workshop if they could make on - no problem came the reply - we do vac welding all the time.

    A few weeks later we get the item. Finish quality wasn’t great but who cares.

    We put it in the vac tester to check for pinholes - no vac at all? odd put flat plate on the vac tester - excellent vac.

    Very odd.

    So we try it on another rig - no vac.

    Hmme we thought what do we try now. Went home for the night - made a brew in the morning and as I was pouring it a terrible thought entered my head.

    I inverted the Christmas tree under a lab tap and filled it with water. My heart sank as water piddled out of numerous welds. The sods in the workshop hadn’t done a decent weld or even the most basic testing.

  25. Will Godfrey Silver badge
    Facepalm

    And it continues

    It wasn't so long ago that the Raspberry Pi foundation were embarrassed by Pi's crashing when being photographed. It turned out one of the chips' packaging wasn't opaque enough to block the UV emitted by some flashes.

  26. MachDiamond Silver badge

    Assumptions

    The assumption was that the light shield was doing its job. This sort of thing is one of the hardest problems to train yourself for. You can't go into troubleshooting something with the mindset that nothing works right and the whole things was built by monkeys. This also reminds me of one of the things I got spoken to about when I was working at a start up aerospace firm. I spent time documenting my work and even going as far as writing up a 'theory of operation' for some of my more clever designs. A freshly minted COO with no technical experience had words (email really) about all the time I spent not designing stuff. He failed to remember that the documentation that I did was a big contributor to a million dollar aerospace prize and also received praise from the US Air Force when we were working on getting permission to fly rockets from Cape Canaveral. (this is not SpaceX). What I'm vaguely getting to is developing the habit of looking at things from first principles. If you didn't design it yourself, a theory of operation document can go a long way to understanding a device, circuit or system. It could even be more valuable than a diagram or schematic since those can get out of date due to modifications, but the basics of what being done stays the same. Knowing how a scintillation counter works in simple terms could give insight into the sorts of things you need to check.

    1. DropBear Silver badge

      Re: Assumptions

      Well no you actually kinda can. Of course, first assumption is that things normally taken for granted are actually good, but the first time you encounter something that simply shouldn't happen or is inexplicable in any way, the first thing you do is take a mental step back and ask yourself "which ones of my default assumptions would have a chance of influencing this outcome if they were to turn out false?". And then you actually check whether they are indeed valid assumptions, of course. All it takes is a mindset of taking nothing for granted if falsifying it would have a chance of influencing the phenomenon you're out of explanations for.

  27. cutterman

    Multi-years ago working with a laser-doppler flowmeter. Instead of watching a needle jumping around, fed out put thru a DA converter and wrote a program to graph and analyse results. In Sinclair Basic. Worked like a charm. Down to the real experimental work.

    4mm laser head fitted into a little stick-on button that warned the skin ro 38deg C to stabilise output. Results sometimes good, sometimes bad - no consistency.

    After much farting about discovered that the laser head was sensitive to rotation - turn it 90deg or more and you got different results.

    Painted an indexing mark on the head and button and hey it's Christmas! The manufacturerers wre happy to find out about that and off went my long-forgotten paper.

    Ah me...

    Mac

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020