Reach for the common denominator

Problem solving, how to get better at it, and how to teach it, remains a concern with me. In looking at problem solving heuristics in IT the other day, I mentioned that one important rule for me is that when several problems start up at once, odds are they are related. So one of my approaches is to investigate the most transparent of the problems, and see if that leads to a general solution. But I realized while helping a client solve a problem the other day that there is a second approach I use all the time that also stems from this rule. When several problems crop up at once, look for possible common denominators and test those possibilities. The real life scenario: At one of our YMCA clients, things seemed to be going from bad to worse over a period of several weeks. Problem one: the serial-to-ethernet converters that let all the door access-controls communicate with the server kept crashing, and would have to be reset frequently. Since this system is slated for replacement, no one worried about it too much. A few weeks later, the web site started periodically loosing its connection to the database. Some calls were placed to the hosting to service to see what was up. No real results came of this. Then a third problem started: The credit card software was timing out on a regular basis. So the users started looking into upgrading that to the latest version of that program. Three problems, three attempted solutions. What if the problems had a common cause? What might it be? Well, all of these diverse processes are driven by services on one central box - these things could all happen if the server was periodically loosing its network connection. We checked the event log on the network, and indeed found that the door-control software was reporting frequent loss of network connectivity. Our hypothesis looked even better. But here I fell victim to the bias of "availability". All of the tools my company developed are resident on that one server - so I assumed that problem was located there -- it's the box I'm most familiar with at that site. So we checked the obvious things - the cable and the port on the switch-- and the less obvious, looking for a rougue process running on the box, but everything seemed fine. It's a network problem, in my mind, became synonymous with "This server is messed up," even though we could not find anything wrong with it. To the rescue came their network guy, who took a broader view of the network. He spent some time looking at their network traffic and found the firewall was being swamped by messages from a random PC that had been possibly taken over by a virus or bot-net scam. That machine was removed from the net, and all the problems went away. So here's another rule of thumb - it really does help to get additional minds on the problem -- someone else will see past your blind spots.

Biases that block effective problem solving.

All the best problem solving techniques can be defeated by biases that get in the way of clear thinking. And the worst of it is, we can't often tell when we are in the grip of these patterns. There are three we've seen often in our work that having a major impact on technical problem solving. One is what my wife refers to as "The Hot Dog" syndrome. Michael Anissimov calls it "The widespread tendency to categorize oneself as above average ". Certainly we techies tend to think of ourselves as a clever lot; this can cause us problems when it encourages us to disregard input from others, to discount the possibility that we made a mistake, act without due dilligence, or so often to underestimate how long it will take us to do something. It also contributes to the oft-cited failing of Not-Invented-Here, where a techie will devote a week cobbling together a solution to a problem rather than buying a two-hundred dollar utility to accomplish the same task. A countervailing bias is the "Cover your Ass" syndrome [CYA]. Since in real life a large number of the problems a techie is called upon to solve are problems she may have played some role in creating, the need to defend one's reputation really is inherent in every IT crisis. It affects problem solving: I really don't want to find out that the reason no credit cards have been processed for three weeks is because I reconfigured the firewall. So I instinctively look for changes other people, other vendors, or other departments made. In organizations where blame is a big part of the culture, CYA can have a major effect on problem solving. Organizational culture can play a significant role in keeping people from thinking straight. I was talking to a client the other day who referred to the change log in our application as the "Blame" table - you go there to see which of your co-workers is to blame. Really striking. I've heard other users in their in-house training say "This is how you can find out who knows why a particular change was made". Quite a difference. Be on guard in particular for teams that need to affix the blame before attempting to solve the problem. Another important bias is one Groopman in his article called "Availability." This bias is more cognitive than emotional, and reflects our tendency to stop at the first two or three possible solutions that come to mind, that are the most available to us. It can mean that it does not occur to us to seek additional information when we should, because we immediately apply one of the first few solutions that come to mind. I'm thinking of a time when I was given an old mainframe printer. I wanted to hook it up to a computer, but it used the pins in the parallel interface differently than the PC did, so it would not work when I connected it. I'm a software guy. So the solution that occurred to me - that was available, was to write a routine that hooked into the interrupt handler for the parallel port and changed each character's pin out as it was transmitted. It worked fine, but it had to be loaded when the computer was booted, and then prevented any other printer from being used on that port. What I should have done was make a custom cable. Duh. What is the value of categorizing these harmful patterns? Giving patterns names can help us recognize them. For the negative patterns or biases, this enhanced recognition can help us avoid falling into them. The first two biases often have feelings associated with them -- if I'm sweating bullets while I work on something, I'm vulnerable to CYA. If I'm showing off to my client, I might be a Hot Dog victim. The third one is most insidious, and leave fewer traces. If we are moving very quickly to a time-consuming or expensive solution with a sense of certainty that we know what to do, or if we are feeling a sense of time pressure, we may be in the availability trap.

Rules of Thumb

Jerome Groopman is a physician who frequently writes on medical topics for the New Yorker. His stuff is always fascinating. A few issues back, they ran a piece of his called What's the Problem, where he looked at the heuristics, or rules of thumb, that physicians use when making a diagnosis. And at certain recognizable errors doctors make that lead to incorrect conclusions about a patient's ailment. Ed Batista, in a post called When Heuristics Go Bad, immediately picked up on how the same kind of analysis Groopman uses can be applied to management decision making. And I thought, this explains so many problems I see in non-profit IT teams, especially those consisting of Accidental Techies. Where's the buzz? There are three heuristics that guide me through 90% of the problem solving I need to do. The first is one I learned when I was 12. I call it the "Where's the buzz" rule. If your stereo is buzzing, how do you decide which component is buzzing: the speakers, the cd player, or the amplifier? By removing each component from the mix and seeing if the problem goes away. Try using the same speakers and cd player, but another amplifier. If the buzz goes away, the problem is in the amplifier. This same approach is a great tool to apply to IT problems like why your meetings registration web pages no longer will charge credit cards. I'm always surprised to see people who have never added this trick to their toolbox. What did you break this time? Everyone knows that half the time when you try to fix something, you break something else. When I change a light bulb, I scratch up the walls carrying the ladder back to the basement. When I refine the security settings on the firewall, ecommerce goes down on the website. This is not just an annoying fact of life - it is a valuable trick for problem solving. When a new problem emerges, the first rule to apply should be "What has changed?" Odds are the last changes made to your software, to your network, to your pc, have caused the new problem. When sorrows come, they come not single spies, but in battalions The Bard can add a third heuristic to this set. So often, several problems emerge all at once. I've found that when this happens they may well have the same cause, even if this seems really unlikely at first glance. So pick the one that looks easiest to solve, and see if this resolves the other problems as well! But as Dr Groopman points out - even with a some useful heuristics at your disposal, things can still go wrong. Tomorrow we will look at a few common patterns that defeat effective problem solving.