My “favorite” magtape story(minnie.tuhs.org)
minnie.tuhs.org
My “favorite” magtape story
https://minnie.tuhs.org/pipermail/tuhs/2023-January/027546.html
40 comments
Back in the early '90s I worked at a PC builder. They got in a batch of drives that had their PCBs closer than usual to the screw holes and installed quite a few before they noticed. They tried to RMA the drives, but were refused. In the end the got a bunch of shorter screws for those drives.
They were a fairly large local business, but they priced out big quotes for the University based on the price of parts dropping over 3 years, and then one year in the late 90s prices went up rather than down, and it suddenly put them out of business.
They were a fairly large local business, but they priced out big quotes for the University based on the price of parts dropping over 3 years, and then one year in the late 90s prices went up rather than down, and it suddenly put them out of business.
I can relate to the screws issue.
We also did have a 'The cleaner unplugged the packet network termination unit to use the socket' episode.
We also did have a 'The cleaner unplugged the packet network termination unit to use the socket' episode.
Forgot one of the best ones:
A large leisure centre/sports complex had 'ghost typing' occurring on several new Olivetti PCs.
Root cause: Due to the size of the site, the staff had 5W walkie talkies to keep in touch. The new batch of Olivetti computers had cheapened keyboards, and instead of the electronics being encased in a metal clamshell, there was just a foil-backed piece of card folded over the PCB; this did not work as an effective rf screen and the walkie talkie signal strength was hitting the circuitry and causing random keypresses - including CTRL-P to keep dumping screen images to the dot matrix printers (this was DOS world late 1980s/early 1990s).
Immediate fix: Replace new keyboards with earlier revisions.
Longer-term fix: I seem to recall it was 'a few more capacitors' for rf decoupling.
A large leisure centre/sports complex had 'ghost typing' occurring on several new Olivetti PCs.
Root cause: Due to the size of the site, the staff had 5W walkie talkies to keep in touch. The new batch of Olivetti computers had cheapened keyboards, and instead of the electronics being encased in a metal clamshell, there was just a foil-backed piece of card folded over the PCB; this did not work as an effective rf screen and the walkie talkie signal strength was hitting the circuitry and causing random keypresses - including CTRL-P to keep dumping screen images to the dot matrix printers (this was DOS world late 1980s/early 1990s).
Immediate fix: Replace new keyboards with earlier revisions.
Longer-term fix: I seem to recall it was 'a few more capacitors' for rf decoupling.
'80's. Super-tweaked Siemens b/w inkjet printers modded to print color. Siggraph in Vegas. Feverishly iterating through successive builds until the very last second, catching a red-eye out to opening day. Exploded booth printer displays were cool. Beautiful main pcb built around four 2712 eproms. Huge crowds in booth "Color!" Test print after print after print, smooth... Obsequious Canon operative, "May take picture?" "Sure!" Flash pops off - 2 of the 3 show printers immediately into Grand Mal seizures..Vomiting ink everywhere. Power cycling restores order. "May take picture again?" "Um.. ok" Same catatonic response.
Sweating.
Naked eprom windows. Photons. Masking tape saves day.
Sweating.
Naked eprom windows. Photons. Masking tape saves day.
Had random wifi dropouts in student flat = took far too long to work out we had a leaky microwave
Random bit flips = squeaky raised floor tile in data center
Also filed under threats you never thought of: I can't find it at the moment, but there was a case report in the New England Journal of Medicine of an iron worker who presented, as I recall, vomiting blood. Turns out some years ago, he had been working with a wire wheel and felt a sharp sensation in the back of his throat, but it subsided. Over the next decade or so, he had intermittent chest pain. A wire had flown out of the spinning wire brush, he had inhaled it midflight, it had landed in his trachea. Over the next decade, it slowly migrated through the trachea and penetrated one of the great vessels of the heart. Retrieving the wire and repairing both the trachea and the great vessel was a delicate matter.
Full face shields are vastly underutilized pieces of safety equipment. Highly recommended, especially if you fog up safety glasses.
Around 1993. An overseas government office, high ranking guy,light-years beyond my rank, brilliant in his political field, complains he's lost data, AGAIN, furiously summons me to make amends about the deplorable system of which I am in charge and FIX THIS SHIT.
Office has a low standing safe next to his desk. Each safe has a little magnetic sign, one side says "open" the other "closed". While the safe was open, he put the sign on top of the safe, out of the way.
It seems (who knew?) that placing your floppy disks on top of magnetic signs might corrupt your data. I accepted responsibility, I told him, because I didn't anticipate he might do this, and I should have briefed him. On one hand it was politically expedient to end the quarrel quickly. On the other, perhaps true: never assume your users or clients have a single shred of intelligence beyond their field.
Office has a low standing safe next to his desk. Each safe has a little magnetic sign, one side says "open" the other "closed". While the safe was open, he put the sign on top of the safe, out of the way.
It seems (who knew?) that placing your floppy disks on top of magnetic signs might corrupt your data. I accepted responsibility, I told him, because I didn't anticipate he might do this, and I should have briefed him. On one hand it was politically expedient to end the quarrel quickly. On the other, perhaps true: never assume your users or clients have a single shred of intelligence beyond their field.
In a similar vein we recently had an issue with one employees monitors turning off when a coworker sat down. We traced it to the cylinder of the chair apprently making some electric disturbance. Our minds were blown for a while, but I remembered reading something similar.
In dry air and right shoes I can zap devices in the office off.
One keyboard restarts even when hovering charged hand above it.
One keyboard restarts even when hovering charged hand above it.
Ok so this literally happens to me! There’s a desk behind my back. When the guy there stands up, roughly 10% of the time my Dell monitor will go blank and reset.
Seems like this is a relatively common bug. Some months ago, in the HN thread for a blog post by the Lunar app developer and weird bugs people sent him, a commenters posts how this is a known bug and, for example, it's in DisplayLink's support database, and they even suggest some fixes:
"Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue."
HN Thread: https://news.ycombinator.com/item?id=32631017
DisplayLink support: https://support.displaylink.com/knowledgebase/articles/73861...
Lunar app bugs: https://notes.alinpanaitiu.com/Weird%20monitor%20bugs
Edit: Looking around I've found this funny/interesting short paper from 1999 called "Unusual Forms of ESD and Their Effects". They refer to a Japanese paper already in 1993 pointing to chairs ESD disruption, and also to using a ziplock bag with a bunch of coins to test ESD protection...
Short and interesting! : https://www.emcesd.com/pdf/uesd99-w.pdf
Edit 2: Interesting 2006 research about one of the researchers for the 93 paper:
"ESD noise radiated from walkers was observed in the 5-GHz-band to study the influence on the quality degradation of radio communication. We determined that the radiated ESD noise in the 5-Gk-band is originated by "Collision ESD" and "Induced ESD" when people walk while wearing metal objects."
https://ieeexplore.ieee.org/document/5256785
"Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue."
HN Thread: https://news.ycombinator.com/item?id=32631017
DisplayLink support: https://support.displaylink.com/knowledgebase/articles/73861...
Lunar app bugs: https://notes.alinpanaitiu.com/Weird%20monitor%20bugs
Edit: Looking around I've found this funny/interesting short paper from 1999 called "Unusual Forms of ESD and Their Effects". They refer to a Japanese paper already in 1993 pointing to chairs ESD disruption, and also to using a ziplock bag with a bunch of coins to test ESD protection...
Short and interesting! : https://www.emcesd.com/pdf/uesd99-w.pdf
Edit 2: Interesting 2006 research about one of the researchers for the 93 paper:
"ESD noise radiated from walkers was observed in the 5-GHz-band to study the influence on the quality degradation of radio communication. We determined that the radiated ESD noise in the 5-Gk-band is originated by "Collision ESD" and "Induced ESD" when people walk while wearing metal objects."
https://ieeexplore.ieee.org/document/5256785
So, out of curiosity, how do you think the signal was taken in from the antenna in that paper?
I would like to do EM scans around my house just out of curiosity, and the graph is nice.
Is this an application for a cheap usb software defined radio, or is this the output of a several hundred dollar oscilloscope?
I would like to do EM scans around my house just out of curiosity, and the graph is nice.
Is this an application for a cheap usb software defined radio, or is this the output of a several hundred dollar oscilloscope?
My EMI knowledge is almost zero, but I think you can see a couple of examples and projects with interesting information (and cheaper options) about it here in the RTL SDR blog with the 'emi' tag: https://www.rtl-sdr.com/tag/emi/
In fact I'm going to read it later
In fact I'm going to read it later
When someone stands up from an office chair, they also create the ideal conditions for static electricity. If the air is fairly dry, and they have sat and squirmed in that chair for a while, then they will have built up a little bit of charge. However, their body and the seat of the chair act as a capacitor, as they are effectively two objects (plates, in a capacitor) separated by an insulator. So, as they stand up and the distance between the two "plates" increases, the capacitance decreases, but since the charge held remains constant, the voltage must correspondingly increase.
It can be a neat trick to stand up from an office chair and then point your finger at your co-worker's ear.
It's also one reason why I am extra-super-careful to ground myself when digging around the guts of any computer systems when I'm sitting on an office chair.
It can be a neat trick to stand up from an office chair and then point your finger at your co-worker's ear.
It's also one reason why I am extra-super-careful to ground myself when digging around the guts of any computer systems when I'm sitting on an office chair.
Haha. Our team has about 10 or so "talking buttons" with sounds of various crassness recorded on them for comedic relief. On a particularly dry day someone noticed that static electricity does not only mess with half of the monitors, but spontaneously triggers a random button.
We found that when one of our engineers sat down on a chair which triggered a button with an obscenely wet fart sound.
Needless to say, our productivity went through the roof that day.
We found that when one of our engineers sat down on a chair which triggered a button with an obscenely wet fart sound.
Needless to say, our productivity went through the roof that day.
AM radio was (still is) one of the best tools for finding issues like this sparking motor. The real trick is to find an AM radio station that you enjoy listening to every day, and when you hear interference, drop what you are doing to find the source. Engines and motors close to your server room will create disk errors and absolutely wreck any signal in unshielded wiring.
Back in the early days of large screen CRTs (19 inch color), late 80s. I was working on bringing X up on a new display board .... suddenly the whole screen sort of twisted and shifted, and the went OK again .... outside my window a really long truck with recently replaced BART (electric train) rails was going by .... essentially a large bar magnet ....
In 2005 I live next to a railway. I could tell when the trains left the station a few kilometers away, because the peak current would make my crt image vibrate.
When I worked for HP, I was visiting an HP partner’s datacenter. While there, an electrician was doing work cutting metal (conduit if I recall) with a chop saw in the server room.
I did report it … since we had systems we were loaning them in that data center.
I did report it … since we had systems we were loaning them in that data center.
My one. I walked into the office once when we had the cable monkeys in and the DLT drives and server pile was covered in plaster dust where they had been drilling above it. Both drives died within a month. Had to replace them and all the tapes. Cheap ass director bought a rack after that.
Edit: found potato cam pictures (this was nearly 20 years ago) https://imgur.com/a/gjyCOQj
Edit: found potato cam pictures (this was nearly 20 years ago) https://imgur.com/a/gjyCOQj
Is there a list somewhere of these crazy bug stories? In addition to some great ones in these comments, the one that always sticks out at me is the "500 mile email" story.[0] I'm sure someone must have a big collection of these somewhere.
0: https://web.mit.edu/jemorris/humor/500-miles
0: https://web.mit.edu/jemorris/humor/500-miles
You're in the list right now!
Reminded me of this classic about irradiated cows messing with a computer at a railway junction: https://beza1e1.tuxen.de/lore/crash_cows.html
Also "Always mount a scratch monkey"
https://edp.org/monkey.htm
https://edp.org/monkey.htm
I thought the monkeys in this story were going to be users. Well. They are, but you know what I mean.
I wonder if anybody ever fact checked this story. I expected to see something related to Mayak plant disaster upon reading Sverdlovsk.
https://en.m.wikipedia.org/wiki/Kyshtym_disaster
https://en.m.wikipedia.org/wiki/Kyshtym_disaster
We had a server that was failing intermittently at about the same time of day, around 3am. Looked like a power failure from the logs but there was no problems with other servers and this server was on a dedicated circuit that was power backed up and the server had a UPS attached.
So one of my team volunteered to stay and watch the server.
Around 3am, one of the custodians buzzes himself into the secure server room that has limited access, at the time only 3 of us on the IT team, our head of Regulatory and the CFO, were supposed to have access.
The custodian walks over to the server, says hi to Chris and proceeds to unplug the server’s UPS from the bright orange socket panel and plug in his vacuum cleaner. He then starts to vacuum the raised floor in our air filtered server room. The failure was intermittent because sometimes he finished vacuuming before the UPS ran out.
Apparently our facilities manager considered herself and her staff exempt from our corporate security policies. She also told the contract custodian to “just unplug something if they didn’t have the courtesy to leave you a free plug”.
So one of my team volunteered to stay and watch the server.
Around 3am, one of the custodians buzzes himself into the secure server room that has limited access, at the time only 3 of us on the IT team, our head of Regulatory and the CFO, were supposed to have access.
The custodian walks over to the server, says hi to Chris and proceeds to unplug the server’s UPS from the bright orange socket panel and plug in his vacuum cleaner. He then starts to vacuum the raised floor in our air filtered server room. The failure was intermittent because sometimes he finished vacuuming before the UPS ran out.
Apparently our facilities manager considered herself and her staff exempt from our corporate security policies. She also told the contract custodian to “just unplug something if they didn’t have the courtesy to leave you a free plug”.
In olden days i worked at a datacenter that had problems when work on the floor above included grinding (and creating some iron fillings all over the place). The tops of the dg computers below were not covered and the iron fillings caused all sorts of grief inside the computer itself.
On a rollout of new workstations for the local bank, which were moving into a nice, newly built, office building. On the trader floor, one of the workstations would not finish setup and would constantly reboot. And one of the (three! awesome back in the late '90s) monitors wouldn't work properly. Turned itself off and on all the time.
I said to a colleague that the power circuit was probably borked, off-spec and faulty somehow.
Since the building had just been put up, there was still a flurry of electricians and plumbers and painters and everything else inside trying to get everything ready over that weekend.
After my colleagues and others had tried figuring out the problem for a good time by trying everything else, I asked one of the passing electricians to get the circuit for that specific desk checked out.
First time I had right in the workplace by just deducing a cause from experience, as a fresh IT worker in a big company. There was indeed something wrong with the power circuit. Felt good. Never learned exactly what was wrong though. Bad cable perhaps.
I said to a colleague that the power circuit was probably borked, off-spec and faulty somehow.
Since the building had just been put up, there was still a flurry of electricians and plumbers and painters and everything else inside trying to get everything ready over that weekend.
After my colleagues and others had tried figuring out the problem for a good time by trying everything else, I asked one of the passing electricians to get the circuit for that specific desk checked out.
First time I had right in the workplace by just deducing a cause from experience, as a fresh IT worker in a big company. There was indeed something wrong with the power circuit. Felt good. Never learned exactly what was wrong though. Bad cable perhaps.
Similar stories talk about the bits of wire floor scrubbing brushes getting sucked up into multi-million dollar computers in the 1960s.
I could listen to such stories all day. Any idea of where I can find more?
the Magic / More Magic switch is a favorite
http://www.catb.org/jargon/html/magic-story.html
"In my house there's this light switch that doesn't do anything. Every so often I would flick it on and off just to check. Yesterday, I got a call from a woman in Madagascar. She said, 'Cut it out.'"
--Steven Wright
--Steven Wright
I have an unused switch on my wall. This gives me an idea
Regular increase in coax Ethernet network error rate = moon tides affecting nearby river and increased ground water during heavy rain affecting ground voltage level on side of building next to the river.
Poor or unable-to-connect modem link = nail through telephone cable on wall.
Misbehaving fibre optic token ring topology; 'data getting corrupted in transit' = EMP being injected into one electrical/optical transceiver on one network card in a PC closest to 'The largest laser in Europe' (and its power supply). This was at a UK Government defence research lab in the 1990s.
200MB hard disk in PC - would spin down when the case screws were inserted. Never did find the root cause (pressure on 'something'). This was my work PC so I just left off the screws.
Computer room overheats = Delivery of listing paper put on floor in front of self-contained circulating air cooling system in the corner of the computer room. Fix: Move the boxes.