Tarnished Gold

I will bet that many people have some kind of polish in their house. It could be polish for their silverware, it could be polish for their shoes, or maybe even for their car. We like shiny things. Things that are shiny capture our attention, help make us proud, and are a sign that we know how to care for those things we find important. The Marines of 8th and I in Washington DC, have a ritual of shining the brass every morning. In fact it is among many great traditions and with the summer upon us, I would urge you to check out their Friday Evening parade, if you are interested in a spine tingling pageant of pride.

I don't think it is coincidence then, that we, often times, call our master images the 'Gold Image'. Record labels called them gold masters for decades before we computer geeks got into this game of storing information on a disc. The Gold Image is important because it takes us back to a time when our environment was pristine, a time where we knew every parameter by name, and while we might not have been in control, it sure felt like we were at a good state of equilibrium. In a small town, this is easy to achieve right? However, you move to a big city, or build out a bigger environment, and suddenly there are other actors in my play called 'life'. With new people in my environment, it suddenly becomes more difficult to manage everything they are doing. The next thing you know, they are inviting over their own friends, and all of a sudden I don't know the names of all of my parameters any more. Values are changed, files like chairs at the table, are moved, and the guy I thought was named Steve is now a girl named Sue(there is a garbled up song in there somewhere).

People by their very nature cause change. The challenge is how to track that change, know when it occurred and understand if their actions made things better or worse. In science they call them controlled experiments, and every variable is meticulously tracked. What if we changed the name of change control, or change management to 'controlled experiments', would that change our behavior to ensure we documented everything? Probably not. The funny difference between Scientists and Engineers (IT) is that we in IT think we know all of the answers. I am not being flippant, just a realist. If we looked at our end users, shrugged our shoulders, and said, let me conduct some controlled experiments to see if I can find the root cause of the problem, they would probably laugh behind our back, lose confidence in our ability, and ultimately look to replace us. We are almost forced to lie to them, and tell the user what they want to hear. 'Don't worry, I know exactly what the problem is', even when we have no clue. The worst thing is we then lie to each other as well, right? Have you ever looked at your DBA or your Application Admin, and said "I know this will fix the problem, trust me"? Meanwhile your fingers are crossed, your hope is diminished, and inside you feel like you just lobbed a 'Hail Mary' pass 70 yards down the field.

If we go back to my small town analogy, and consider how it grows, we know that one person cant manage all aspects of the town. We get someone who specializes in roads, another person who specializes in buildings, and so on. These specialists then make changes on what works best for them. What they are not good at, is understanding the coordinated impacts of their changes. This is very similar to what we do in IT, when a network person, or storage person or whomever makes a change thinking it should have no impact to an application running atop it, only to find out later they were wrong. We try to capture these with coordinated changes events, but because there are so many moving parts our ability to test these things ahead of time is very hampered. I would propose the better route is to make the change, know how to validate and test its coordinated impacts and stand ready to back-out if you need too. It introduces more risk, but the return for that risk I believe is material to a teams productivity. The trick of course is having eyes and ears on that change so that you can pinpoint it immediately. We claim to test changes in our lower environments all the time, that have no REAL reflection of what happens in production because the environments, regardless of what we claim, are not the same.

No one is doing this to be malicious or because they don't care about their work. Rather they care too much sometimes. We skip work outs, we skip meals, we skip family holidays sometimes, for the love of our environments and happy users. It is only human nature then to find ways to cut corners. "Oops I forgot to document this", "Ouch, I didn't get approval for that", or "Who was I suppose to coordinate with" You got to be kidding me, I have to remember all of that when I just pulled a 40 hour shift, troubleshooting a hung process, and my vision is blurry, but you want me to stop and do paperwork? I am the gosh dang hero of the moment. I'm not doing no stinking paperwork. I may be tarnished at this point, but I am still the golden boy of the hour. Crisis has been averted for another day!

We have, though, at our disposal the ability to apply polish to our Gold Image. That polish allows us to get our image shiny again, and it also lets us take a look back at how we got where we are. In the case of environments like Oracle Applications or PeopleSoft, the beauty is we have some frames of reference in terms of the pre-built environments. For custom applications, while a little more work, the fact is the principles and applied discipline are exactly the same. So if tracking change is the beauty of the proposition, then I think the sexy stuff is when we correlate performance to that change. If you modified your carburetor (I know I am dating myself for those who have only known fuel injection) and got slower RPM's, you would want to back out that change right? The two go hand in hand, but all too often in the field of IT, they aren't looked at or managed that way. I feel like at this point I should offer to start a petition and get a new ballot measure put on the California voting scheme. Short of political voting, we here at Quest are looking at exactly how to make this type of thing easier every day. It should be simple and should solve some of our biggest problems of the day.

In summary, make sure you are polishing all that is important to you. If it's the end of work week, feel free to polish off a few beers. But while the cold, crisp taste of grains are disappearing from your palate, ask yourself if you have polished your Gold Image, and is it as pristine as you once knew it to be.

Carl