Catastrophe of atoms versus catastrophe of bits
Software engineering is hard, but software is hard in decidedly virtual ways. Sure, sometimes you’re limited by available people or available hardware, or you don’t have enough CAT5 or the right kind of RAM module, but software problems are generally solvable with a text editor, some caffeine, and patience.
Mechanical engineering is hard, and it is often hard in decidedly physical ways. There are usually wires and pipes and gears and pumps and things. Usually when a computer crashes, there’s a lot of cursing and angst. When something mechanical crashes, there’s also a lot of cursing and angst, but sometimes there is blood. And usually there is a mess to clean up.
Today there was quite a mess to clean up.
But first, a little background: the plastic recycling process that my dad and his partners are designing is what’s called a hydrogravity separation process. This is just a fancy way of saying that some things float and sink in solutions depending on the density of the solution, and if you can make the good stuff float and the bad stuff sink (or vice versa) you’re in pretty good shape. The whole process sounds pretty simple when you hear it explained—this stuff floats, so skim it off the top, and this stuff sinks, so scrape it off the bottom—but the devil is absolutely in the details.
One of the details is that of scale—it’s one thing to have a table-top demonstration of stuff floating and sinking, but when you go from table-top models to a 20,000 square-foot production facility, a lot of things get much, much more complicated. Take water, for instance. When you want to make a lot of stuff float or sink, you need a lot of water. Or a lot of solution. Or, in the case of the plastic recycling plant, you need a lot of both. So you get some big tanks to hold a lot of water or solution. Really big tanks.
I’d never given much thought to what 5,000 gallons of water looks like, but when you see a 5,000-gallon plastic tank, it’s an impressive sight. And when you see four such tanks standing next to each other, your first thought is usually something along the lines of “that’s a damn hell lot of water.” Because it is.
The thing about the tanks, though, is that once you’ve seen them and get over the fact that they hold a damn hell lot of water, you tend to forget about them. A lot of things can go wrong when you’re dealing with motors and pumps and driers and forklifts and propane and half-ton bags of scrap plastic and mixers and such. Things that spin and whir and grind and move are somewhat threatening, but tanks are deceptively innocuous. After all, they’re pretty passive things; they just stand there and hold water. Except when they don’t.
I mentioned that there was a mess to clean up today, right? And I’ve talked a bit about the tanks, right? So how many of you have guessed it was the tanks? Good guess. But hang on—it gets better (or worse, I suppose).
One of the things you don’t want to hear during the construction of any kind of processing or manufacturing facility is “oh, crap.” Usually this means someone has done some damn fool thing that will cause them a half hour or so of undoing and redoing it right this time. But there are worse things than “oh, crap” things. There are “oh, shit” things.
Because the hydrogravity process uses lots of water and solution and stuff, drainage is a major consideration. There are drainage toughs in the concrete floor that surround the processing stations, and they are designed to be able to handle a worst-case scenario where one of the 5,000-gallon tanks bursts and dumps its entire contents onto the floor. The troughs are drained by a series of high-capacity sump pumps—more than one, of course, to ensure mechanical redundancy in case one pump fails.
All of this sounds wonderful—a tank bursts, and dumps 5,000 gallons of liquid in to the troughs, and the sumps pump it up and out. Ah, but up and out to where? The wrinkle is that because most of the liquid involved isn’t water, but a salt solution, you can’t just go pumping and dumping the stuff into the sewers. At least, you can’t without getting a nasty visit from the EPA. So what the sumps actually do is drain the troughs into another giant tank, which is periodically filtered, treated, and then disposed.
Yes, yes, yes, I hear you say—get to the part with the mess. Okay, okay. Today, during some regular construction and maintenance, one of the workers (no, not me) accidentally backed a forklift into the main discharge valve at the bottom of one of the tanks. This is the textbook definition of an “oh, shit” moment. The thing about an “oh, shit” moment is that it’s a bit like playing a game of telephone, except that there is absolutely no chance that someone is going to get the message mixed up.
The sound of 5,000 gallons of liquid gushing out of a giant plastic tank is, in a rather distressing way, actually quite a fascinating sound. The thing about it is that you don’t quite know what you’re hearing at first, because it’s not the kind of noise you’re used to hearing. This makes the whole experience that much more potent, because everyone goes though the same several seconds during which their brain finishes putting the noise into context, whereupon they invariably have the same reaction: “Oh, SHIT!”
Of course, such a catastrophe had been planned for—just fire up the sump pumps and let them do their work. Ah, but there’s the rub—the sumps drain into a storage tank. And I think you’ve probably guessed which tank it was that the forklift hit, right?
Yup. The sump pump drainage tank.
Now we have a conundrum: there’s a big water leak, something on the order of 200 gallons per minute, gushing out of a four-inch hole on the bottom. The tank is about fifteen feet tall, so there’s about ten feet of head, and the weight of all that water is pushing out a stream in an arc about six or seven feet long. You can’t start the pumps, because they’ll just recycle the water back into the tank that’s been hit with a forklift. Each second that goes by gets you wetter and sticker with solution, dumps gallons of liquid all over the place, and you’ve got to make it stop, or at least slow it down. Quick—what do you do?
You sure as hell don’t call a bug triage meeting, that’s for sure. You probably do something like what my dad and his coworkers did: you improvise, and you improvise RIGHT FUCKING NOW. Now, I wasn’t privy to exactly what was going on at the time; I felt it was best to stay out of the way and let the smart people who knew something about the system fix the problem. This sort of problem is pretty much outside of my core competencies; I can get spyware off your computer no problem, but my first reaction to a geyser of salt solution coming out of a 5,000-gallon tank would probably be to scream like a girl and run away crying.
Everybody knows the story of the little boy who stuck his finger in a dam to plug a leak, but this was no ordinary dam, and what was needed was a pretty big finger. So they improvised. The end result was that my dad and some guys jammed a wooden 2-by-3 wrapped with towels into the gushing hole in the tank to slow the rate of drainage, and a couple other guys whacked the hell out of the thing to plug the hole. Meanwhile, everyone else was busy turning some of the smaller, empty tanks into temporary drainage tanks, getting the sumps running and redirected, getting boxes and computers and stuff off of the floor, and pushing water around with brooms and giant squeegees.
The “good” news is that the broken tank had “only” about 3,000 gallons of liquid in it. So now I’ve seen what happens when 3,000 gallons of liquid dumps onto the floor, with no place to drain, and it is a hell of a thing. As it turns out, I’m pretty handy with a giant squeegee, too.
Login with LiveJournal or OpenID