Always Mount A Scratch Monkey!
I spent a lot of time in the late 70s and early 80s crammed into small computer rooms. It was an era when computers needed a room of their own, and if you wanted to use one, you needed to be in it as well. I found that invariably, there was also a middle-aged chainsmoking woman in there somewhere with you too, doing data processing.
I don’t know if the job attracted middle-aged chainsmoking women, or if the job slowly morphed non-smoking young women this way over time. Whatever the case though, I endured a lot of secondhand smoke while learning to program. And so did the computer systems.
Here is where my first encounter with computer Field Service happened. One day in high school while I was in the computer room trying to extract Snoopy banner printouts from our PDP-8, a DEC Field Service guy showed up, with a strange device. It was like a mini washing machine, full of some kind of solvent I hope was not too cancer-causing. He proceeded to grab all of our RK05 disk packs, and take the covers off them. One by one, he dropped the raw disk platters into this machine.
And it washed them.
I was amazed. And I still am. I asked about it, and he said that there was a lot of smoke and dust that built up, and that affected the life of the disks (he may have given a side glance to Mrs. Sutton, who was in the corner puffing away as she typed on the school’s very dated IBM 029 card punch).
The washing machine spun and sloshed, and then the clean, dried disk was removed and put back in its case. He assured me they would work just as before, and as far as I could tell, they did. (It was after all a time before disk heads flew microns above the platter, so you could get away with exposing a disk to a dusty room. )
But I was struck by the confidence this service guy had with what he was doing. I had always considered our PDP-8 to be this priceless jewel, sitting cloistered in its own temple. You handled it with kid gloves and most often, you did not handle it directly at all. But the service tech would come in and pluck this machine apart with ease, doing what seemed to be dangerous things.
On another visit, the service guy came to check a problem we had with intermittent crashes. He brought his oscilloscope, and his own diagnostics RK05 disk. I was fascinated. Seeing the wires and boards inside the machine, watching the glowing green waveforms, and by his alien software that had direct control of the machine, without any of the familiar OS environment around it.
Experiences like this have a lot to do with why I became an Engineer. Peeking at what’s behind the curtain was always of interest to me, no matter what the device. And the more complex the better. Computers were pretty much the most complex beasts I had ever come across, and I imagined myself as that technician and getting to be the one who gets to take them apart.
It took a few years for me to learn more about the possible careers you could have in electronics and computers, and that one could aim towards a job not just fixing these machines, but designing and building them. I went to Engineering school just because of that interest, with little info on how hard it would be, or how promising a path it could open. After I graduated, I ended up working for Digital building computers, the very same company whose PDP-8 computer I had used in school.
The Inside Story
Once at DEC, my immediate fascination for watching Field Service work dropped off sharply. We were awash in computers there, in offices and data centers and at every site. Clusters of them had been created, and they were now networked and managed internally and somewhat invisibly. Also, we already had our share of hardware adventures in taking things apart and putting them together just as part of our jobs.
I did however enjoy hearing stories about weird Field Service situations. This would be mid-1980s, so no internet yet to spend time on instead of working, but we did have an internal employee networked discussion board system of sorts called “DEC Notes”. DEC Notes was widely used inside the company, and had a huge range of topics to explore, including a discussion from DEC Field Service about strange service calls.
This was a favorite topic of mine to browse. There were perhaps hundreds of stories there, about all sorts of weird and unlikely places customers put their PDP-11s for instance, including inside the bore tunnel for an underground nuclear test chamber. According to the story, that poor machine’s job was to collect as much data as possible and send it out in the few milliseconds between the blast and it getting vaporized.
Another one I remember was a kind of mystery story. It concerned a problem with the network at one of the DEC sites going down at almost exactly the same time every night. This had been happening for a while, but got to be a serious problem because it disrupted simulations that were running overnight on a project.
Field Service was called to find out why, and they narrowed it down to a switching system in an equipment closet in the building that kept rebooting. They replaced the router, to no effect. Later, logging and diagnostic software was added to see why the system kept resetting. Y2K was not yet a thing, but the idea that some sort of overflow or clock problem was happening was a leading theory, given it seemed to occur right around midnight.
Finally, in an exasperated and desperate move, a field service guy actually decided to camp out in the equipment closet at night to see what was happening with the router. Right around midnight, the door suddenly opened, much to the surprise of the field service camper. A hand reached in, unplugged the router, and plugged in… a vacuum.
It seems the nightly cleaning crew was in need of a power outlet in that particular stretch of hallway, and had decided to improvise.
Don’t Like It But I Guess I’m Burnin’
It was hard to tell how many of those stories are true, and how many had been fabricated or embellished when you read through all those DECNotes. This next one is a famous (or infamous) DEC Field Service story, and I always thought it was completely made up. But some quick research on it seems to point to enough detail to make it, as the Mythbusters would say, “Plausible”.
The phrase “Always Mount a Scratch Monkey” has since entered the hacker lexicon as a commonly-known meme, but I’m sharing the story of it again in the hope that at least some readers have not heard it. (fully expecting to get comments from old DECies that have though)
This tale concerns a lab involved with research using primates at the University of Toronto, in 1980. The Department of Medicine had been conducting experiments using monkeys, with electrodes attached directly to their brains. These electrodes and associated experiments were run using the department’s new VAX 11/780, which had a superior speed to the older PDP11/05 they had been using previously, and this speed was needed due to the nature of the real-time data collection and processing being done.
The problem was though that the VAX 780 was a very new, and buggy machine in 1980. (Some of the security-related bugs were things I wrote about taking advantage of previously.) At some point, the VAX started crashing, and DEC Field Service was called. They proceeded to check the machine out by shutting down VMS and running a set of dedicated hardware diagnostics tests, that exercised things like the CPU, disk, and memory.
Oh yeah. And the I/O system. That was attached to five live monkeys.
No one had bothered to tell the Field Service guy about these monkey connections, which were normally instrumented to read analog signals from the brains of the monkeys, but were also capable of generating analog voltages as output as well. A feature that was precariously (and as it turns out, insufficiently) disabled on the system.
Diagnostics being what they are, the VAX I/O system was put through grueling cycles of both reading and writing the I/O ports, resulting in voltages far beyond what was healthy being sent to the monkey brains. The monkeys all went through violent convulsions, and three actually died. In the telling of it, the Field Service guy was pretty traumatized and even threatened to call the Humane Society on the lab.
The lab itself seems also to have been doing sketchy if not unethical experiments in general and was later shut down. The whole story is pretty macabre, but as time went by, people (as is usually the case) found some humor in the whole incident.
The correct procedure when running disk drive diagnostics is to mount an unused, “scratch disk”, so the contents of a valuable disk are not destroyed as the diagnostics are run. The Field Service guy had unfortunately not brought a “scratch monkey” with him that day. Advice for that later circulated through DECnotes and beyond, to become the famous Field Service meme.
Take Your Monkey To The Genius Bar
I was going to go on to talk about how you really don’t see Field Service anymore, but I kept remembering how it is still around. The cable guy was here last week fixing my modem/fiber connection, and we need to have someone come and look at our wonky oven control panel someday.
I find it interesting that my laptop is worth more than our oven, yet unlike the oven, it would be really tough to get someone to actually show up at the house to fix it.
I thought about it and finally arrived at this handy algorithm for figuring out if Field Service is still a thing for whatever you have that’s broken:
Unfortunately for those wanting a computer repair guy to come to the house, everything works against you in the flow as computers get cheaper, more portable, and harder to repair. But if you are willing to go for a drive, you can still get yours fixed (sometimes).
Just make sure that if there are any monkeys attached, you show up to the Genius Bar with a scratch one!
…Got a good Field Service story to share?
Explore Further
Next Time: The author shares a drink with an enemy, makes the wrong friends, and other botched moves in a desparate business jaunt through Japan. Why they keep me at home mostly and not on the road in: Internal Success Before External Success
The Mad Ned Memo has been featured on Hackaday and is frequently popular on Hacker News and other tech-loving sites. But you don’t have to count on coming across it randomly if you subscribe! Get strange and nerdy tales of computer technology and engineering delivered straight to your inbox, and never miss an issue! It is cost-free and ad-free, and you can unsubscribe at any time.
The Mad Ned Memo takes subscriber privacy seriously and does not share email or other personal information with third parties. For more information, click here.