Sometimes shining a light on a nuclear problem just makes things worse Welcome to On Call, The Register's receptacle of woe for those unfortunate enough to be at the beck and call of that species we call The Customer. Today's tale concerns a duo we will name "Pete" and "Dud" because, frankly, we're of a certain age and still remember Not Only... But Also with a fondness. There were, however, no …
A component was supposed to be impervious to light, but wasn't. You can't guess that that is the problem - something has to reveal it. Had "Dud" done literally anything else, they would not have found the root of the issue and we would be reading about a product failure - or likely not reading about it at all.
To be fair to "Pete" there's no indication from the article that he was stymied by the problem, or even that he'd got very far into troubleshooting after getting the diagnostics running. There's every chance that checking the integrity of the light shields was towards the top of his check list for investigating high readings from scintillation counters - Dud's theatrics simply gave him an early clue.
I know a scientist who always liked to do "dirty" experiments. In other words, he didn't set up his experiments to measure just one thing but ran them in a somewhat uncontrolled environment. He said you discover more that way about things you don't expect.
A favourite trick used to be to 'help' a colleague trying to fix a recalcitrant umatic tape player by ensuring there was plenty of light on the job - thereby confusing all the IR sensors that tracked where the tape was loading...
And I can recall as a very young engineer building radios with detector diodes that were light sensitive, so they worked in the daytime but at night picked up an impressive 100Hz hum.
Ah, the joys of (inter alia) the OA47. You bring back memories.
A lot of those old components were light sensitive. Then Mullard realised that the OC71 was being used instead of the very expensive OCP71 and started putting a blob of opaque silicone in the OC71 cases. And then people discovered that the blob was too small, and a few minutes in the centrifuge would restore the light sensitivity.
I used to work for a company that produced motor controllers for various platforms, including fork-lift trucks. A specially modified one was used for testing, with a plate mounted on the back so that it was easy to replace the controller, attach instrumentation and replace the CPU - this was in the days of one-time programmable devices and very, very expensive, gold-flashed EPROM parts for development.
One of the engineers was driving this thing back-and-forward one day testing some new software. It was generally working fine, but every now and then the watchdog would trip, putting the system into a fail-safe state.
Much time was spent reviewing instrumentation and code, but, as the cause of the problem could not be found I decided to offer some assistance. I asked to see the problem, and, sure enough, the controller would enter the fail-safe state once in a while. Strangely, this always seemed to happen when the truck was moving backwards - which was when I noticed the shaft of sunlight that travelled up the back of the truck, over the (uncased) controller and into the window of the EPROM CPU. There was just enough UV in the sunlight to cause the CPU to glitch, showing that those little stickers you're supposed to put over the window are important (though never used during "agile" development phases)!
I discovered similar with UV Pic micro that was being used to develop an engine tuning device, we'd been called out to investigate the gas analyser and they were convinced it was faulty because it would sometimes show unusually high figures.
Turns out the guys who designed the product had no idea about the well documented light sensitivity of those and many other MCU devices with UV windows, a simple piece of black tape and a large bill were all it took to solve the prpblem
Long ago, when wireless networking was best effort (and on 2.4ghz), we had one problem area in the school, call it one problem room. When ever the students started all writing a test on the laptops, the wireless would drop out of this room. We busted our heads and asses trying to get our minds around this problem. After a few observations of the issue, the coin dropped. Every time there was test on the laptops, the instructor would position themselves squarely in the doorway(to observe for cheating etc). This instructor didn't miss any meals, and didn't know when to push away from the dinner table. The AP was directly across the hall from their door and their large carriage was blocking the signal. We asked them to move away from the door, and presto, connection to the network was restored! We couldn't bring ourselves to tell them it was their own fault...
I had the opposite problem in a previous job.
The mouse hooked up via the KVM to all our servers sat on a metal tray, with a crinkly white gloss finish. Even when left still, it would detect minor movements, so the cursor would jitter on the screen, and the screensaver, and more importantly, the screenlock, would never engage.
I solved the problem by sticking a post-it note under where the mouse was left at rest, and less reflective surface banished the phantom movements.
More fun can be had by sticking the post-it to the bottom of a colleagues mouse, but they usually work it out quite quickly.
For wired rodents, a piece of post-it note strategically placed over the centre pins of the usb plug before carefully plugging it back in the back into the computer, isolating the 2 centre data connectors, but not the power lines on the left and right of the plug. Mouse lights up (if it’s a LED and not an IR laser), but no movement data is sent. That generally takes someone a few more minutes especially when they turn the mouse upside down, no post it and a shining red light saying all is well...
I remember when a new, non-technical manager too over our department. He decided the scruffy looking metal mouse mats had to go, and over one weekend they were replaced by company branded rubbery ones. Cue howls of protest on the Monday when we programmers and admins couldn't use or mice as they needed the metal mats! Worked out OK in the end, as we ordered new rollerball Sun mice that were more convenient. The cost was deducted from the new manager's salary, since Sun branded stuff was so exorbitantly priced.
I can remember similar : on a project requiring extreme frequency stability and type approval, we installed a state-of-the-art Temperature Compensated Crystal Oscillator with a ceramic encapsulation.
We installed it on a PCB with a plastic case around it.
On test, it functioned perfectly until somebody turned on the light in the test chamber and it hopped frequency.
Turns out the plastic case and ceramic TCXO encapsulation was transparent to infrared light from an incandescent bulb.
I was servicing a recently installed system at a hospital and they mentioned it would trip if they used the rotation option as part of the treatment. It was only in a small section of the arc but nothing was obvious until I noticed a gap between two panels with a ribbon cable behind which carried various monitor signals. Popped down the local Spar shop to get some tinfoil which I wrapped round the cable and then added an earth wire to and as if by magic, all now OK. A proper sheiled cable was sent to replace my homebrew efforts but I can honestly say it was no better and looked boring in comparison.
This post has been deleted by its author
I met, and cursed, the man at Mullard who changed the black paint on the OC71 to a black filling so you could not scrape the paint off and get an OCP71 (the photosensitive version). They sold the photosensitive version for a much higher price by leaving the paint off and relabelling them. A long time ago but I think 36 shillings and sixpence with no paint and nineteen shillings and sixpence painted! Bastards!
There was a reason for that though. The OCP71 were tested as phototransistors and it's the ones that had failed testing that were painted up and sold as OC71. You could use them as OCP71 but in some way they didn't properly meet the spec. Likely they were just a little less sensitive, but still fine for most uses.
Related: Australia National Telescope Facility's decade-long hunt for an interference source feeding burst signals into their Parkes Telescope. Which was eventually found to be staff opening the microwave before the timer beeped. A tale well told at:
http://www.swinburne.edu.au/news/latest-news/2015/05/how-we-found-the-source-of-the-mystery-signals-at-the-dish.php
There's also the case where a telescope doing VLBI would get out of position for no apparent reason, turned out to be a heater on one side of the "pillar" supporting the dish causing uneven heating and a slight bend moving the telescope a minute distance but enough to throw the VLBI off.
so was following the test specifications to the letter, but this one particular instrument was behaving very strangely. Funny.... I thought. I'd not seen an instrument behave so strangly since my Aunt Dolly tried to play the trumpet after drinking half a bottle of gin. Her embouchure was so sloppy her teeth shot out, flew across the room and hit the cat.
We recently had some LARGE interactive displays start going crazy. But only between 9 and 10 in the morning. Turned out there are cameras along the bezel of the screen to aid in the touchiness, and the sun would shine into this room just right for an hour or so to make it act possessed. Once the seasons changed it was no longer an issue. Plus there are now some foil screens on the window now as well
Something similar...
Scene: University observatory, mid 1970s. Small group doing some work with a photometer plugged in to the telescope, to measure the amount of light coming from the target star. Working fine, but suddenly the count dropped. And then went back up again. Then dropped. etc. First theory (aliens) rapidly discounted.
Eventually worked out what was happening. There was a coax cable running across the floor from the telescope to the recording kit, and Tony had been standing on it when the count dropped. This was final proof (to us) that electricity works just like water - stand on the hose and the water/electrons can't get through.
In fact - what do we know about co-ax cable? The resistance depends on the shape - minimum when a circle and increases as it is flattened.
Used to be a similar issue with the primitive network cabling at my first job. I think it's what was called "thick net" and had these "vampire taps" attached at each point you needed to connect a computer (or VT in this case). The cabling was pretty ad-hoc, just laying in the floor, and resting your foot on the tap would often cause the networking to go on the fritz as the cable would short out.
Reverse case -- the first silicon of first chip I ever designed (an echo canceller for ISDN) didn't work. Looking down a microscope at it to see if I could see anything wrong (suspect was a comparator inside an ADC) as I moved the chip around on the stage it suddenly started to work. Turned out the problem was offset voltage in the comparator, the light from the microscope injected photocarriers into the input stage which cancelled this out so long as you looked at it *just* right -- you had to within a few microns of the "sweet spot", which was in a different place for each chip...
In the 1980s the telecomms market in the UK was being deregulated. Energis deployed a "figure of eight" network backbone that went from London to Birmingham and then Manchester. The initial fibre-optic cables were wound around National Grid cables by crawling machines that looked like lawn-mowers designed by spiders. (Manual intervention by humans was required every time they came to a pylon.)
An SDH network was implemented over the fibre and some (or all?) of the capacity was flogged to the BBC so they could distribute uncompressed digital video to their transmitter network and regional studios. In the wee small hours the network would fall over spectacularly but eventually recover.
When the sun hove into view it would heat up the conducters and fibre-optic cables causing the physical length of the network to increase by a small amount (but very rapidly). The clock adjustment/slew mechanisms in the network were not fast enough to keep up with the rapid changes causing everything to fall out of sync. The cables cooled down more slowly at sunset so this issue was only seen in the morning. I assume this was eventually fixed by increasing the maximum slew rate.
Eventually, the fibres were incorporated into the cables as they were manufactured. (Cables are replaced on a regular basis, I don't know the cadence.)
And you tell that to kids today and they don't believe you!
Around the same time as the events described I worked for a big Pharma company. The company was developing one of the first monoclonal antibody-based anti-cancer therapies, and my role was to devise a test to measure biological activity for early-phase material. We had a nice little assay going using Chromium 51 (medium-hard gamma emmitter) in fresh lymphocytes, and in order to validate the test a second team in another building were also required to reproduce my results.
All went well at first until one day, a couple of weeks in, they lost sensitivity and started having bizarre high backgrounds and unpredictable results. This came and went over the next couple of weeks, with things getting worse after a period of more intense work, then dropping back to normal levels. The more they tried to fix it, the worse it became, then would suddenly work properly again.
The gamma counters we used at the time, made by LKB Wallac, were pretty much industry standard and usually used for counting Iodine 125, a softer gamma emmitter and were legendary for reliability. The basic design had the gamma detectors in 2 rows of 6 with lead shielding between to reduce crosstalk, samples were placed in a rack by hand which was then loaded into the instrument for counting over a fixed interval.
After this had been going on for a couple of weeks I wandered down to talk things through, see if there was anything 'obvious' in their technique that might cause trouble. On reaching the lab, the cause became immediately apparent - the waste bag containing 'hot' waste from previous experiements was being stored under the bench, directly beneath the counter. When the team had previously used Iodine, the gamma from that had been too weak and used at too low a level to penetrate the lead shielding, but the Chromium was being used in much larger amounts, and the higher energy gamma would go straight through a centimeter or so of lead shielding.
We moved the bag, the assay performed like normal again. The treatment eventually made it to clinical use. Lab coat of course. :-)
I used P32 once a week and had the usual film badge finger ring with minimal exposure readings for months. Suddenly one month it came up offscale. Turned out I had left the ring for a whole week on the shelf over the waste bin rather than in the drawer under the bench. Big oops, radiation safety was NOT happy with me for a long long time.
Two stories I'm sure I've mentioned in these halls before with interference:
1) Our CEO many moons ago had an apartment in Paris and used to moan that his WiFi (original 802.11b) was unreliable, even after purchasing a full blown outdoor antenna, complete with lightning arrestor. I told him he was talking rubbish so he shipped me out there on the EuroStar and to my astonishment I could stand at the end of the corridor looking at this massive outdoor antenna mounted 5m away at the end of his hallway I lost my signal. Turned out he lived under half a mile from the Eiffel Tower and as you stepped into the hallway where a large floor to ceiling window showed off the Tower you immediately lost all WiFi. The Eiffel Tower was spewing all sorts of EMF. The CEO later told me taxi radios didn't work round here either amongst other weirdness. Ended up putting Wifi repeaters in every room of the apartment.
2) We moved IT into a new office and I kitted out the place network-wise and I also put up a dozen large LCD TVs for various monitoring uses. These TVs used HDMI over CAT5/6 and were SO unreliable it drove me almost mad. They used to blank out all the time at random times. It turned out after one late night working on the screen resolutions, refresh rates and so on that a cleaner walked in and the TVs that had been fine for hours suddenly all blanked out! It was the IR sensors that kept the lights on! Whenever it detected any movement, it send out a load of EMF radiation that messed up the HDMI over CAT6. Had to replace all the cabling with shielded cabling and then it was fine.
F F S
Most Copier manufacturers state in the installation requirements that the device should not be located in direct sunlight. Some even say strong light.
The worst one I have seen was in an estate agents front window in full glare of the afternoon sun. Normally not an issue as long as the only real effect is to bleach the panels a horrible colour. Unfortunately some idiot thought drilling a few holes in the copier to mount a keyboard was a good idea. They managed to get some of the holes in the right l place but the uncovered test ones managed to let light bounce inside and onto the drum.
2cm of black nasty tape and telling off and they were mostly happy.
Many moons ago I was working as a post doc in an underfunded university lab.
We had bough a vac freeze drier at an auction when Shell closed their Sittingbourne site.
The freeze drier was missing the stainless Christmas tree that sits on top of it but otherwise worked fine under test. The Christmas tree is very expensive to buy so we asked the workshop if they could make on - no problem came the reply - we do vac welding all the time.
A few weeks later we get the item. Finish quality wasn’t great but who cares.
We put it in the vac tester to check for pinholes - no vac at all? odd put flat plate on the vac tester - excellent vac.
Very odd.
So we try it on another rig - no vac.
Hmme we thought what do we try now. Went home for the night - made a brew in the morning and as I was pouring it a terrible thought entered my head.
I inverted the Christmas tree under a lab tap and filled it with water. My heart sank as water piddled out of numerous welds. The sods in the workshop hadn’t done a decent weld or even the most basic testing.
The assumption was that the light shield was doing its job. This sort of thing is one of the hardest problems to train yourself for. You can't go into troubleshooting something with the mindset that nothing works right and the whole things was built by monkeys. This also reminds me of one of the things I got spoken to about when I was working at a start up aerospace firm. I spent time documenting my work and even going as far as writing up a 'theory of operation' for some of my more clever designs. A freshly minted COO with no technical experience had words (email really) about all the time I spent not designing stuff. He failed to remember that the documentation that I did was a big contributor to a million dollar aerospace prize and also received praise from the US Air Force when we were working on getting permission to fly rockets from Cape Canaveral. (this is not SpaceX). What I'm vaguely getting to is developing the habit of looking at things from first principles. If you didn't design it yourself, a theory of operation document can go a long way to understanding a device, circuit or system. It could even be more valuable than a diagram or schematic since those can get out of date due to modifications, but the basics of what being done stays the same. Knowing how a scintillation counter works in simple terms could give insight into the sorts of things you need to check.
Well no you actually kinda can. Of course, first assumption is that things normally taken for granted are actually good, but the first time you encounter something that simply shouldn't happen or is inexplicable in any way, the first thing you do is take a mental step back and ask yourself "which ones of my default assumptions would have a chance of influencing this outcome if they were to turn out false?". And then you actually check whether they are indeed valid assumptions, of course. All it takes is a mindset of taking nothing for granted if falsifying it would have a chance of influencing the phenomenon you're out of explanations for.
Multi-years ago working with a laser-doppler flowmeter. Instead of watching a needle jumping around, fed out put thru a DA converter and wrote a program to graph and analyse results. In Sinclair Basic. Worked like a charm. Down to the real experimental work.
4mm laser head fitted into a little stick-on button that warned the skin ro 38deg C to stabilise output. Results sometimes good, sometimes bad - no consistency.
After much farting about discovered that the laser head was sensitive to rotation - turn it 90deg or more and you got different results.
Painted an indexing mark on the head and button and hey it's Christmas! The manufacturerers wre happy to find out about that and off went my long-forgotten paper.
Ah me...
Mac
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
