Coding Horrors, Cargo Cult Programming, Ego, and Other Ghoulish Figures

Share |

Well, it seems a good time to discuss coding horrors, cargo cult programming, ego, and other ghoulish things. Let us do a few coding horrors first, just to whet your appetite for destruction.

[Note: this is an updated, edited and expanded version of a blog entry that was posted earlier, and then replaced by a highly edited version that removed the discussion of personality traits in programming.]

Here’s a gem I once encountered:

Dataset.Open;

While not dataset.eof do

Try

DoSomething(dataset);

Finally

Dataset.Next;

End;

I am at a loss as to figure out what is the point of putting the dataset.next in the finally block. The purpose of a finally block is to ensure that something gets done even if an error occurs. The explanation I was given by the coder was that he did not want it missing any records just because an exception occurred. The code obviously does not ensure any such coverage. It will not execute the DoSomething procedure on any record that comes after a record where an error occurred, because an exception would cause the program to return to the calling code, after it moved to the next record. (If you were to report this error in that calling code and wanted to show data from the record in the error message, you would be showing the wrong record’s data!)

Now, you might be saying to yourself, “What in the world?!!”, but what will blow your mind is the years of experience and the attitude of the programmer that wrote this code. He claimed to have been doing Delphi for over 15 years, and claimed his code is always solid. He also boasted that he doesn’t need to read books about Delphi or programming. In fact, when he recently saw a visiting programmer with a book on C++, he decried that as a sign that the guy was just a mere beginner. But the kicker, at least for me, is that this guy could never admit he made a mistake.

The admission of error and ignorance are vital characteristics for the proper carrying out of the trial and error iterations that comprise real world software product cycles. How does one correct or perfect one’s code and coding styles if one never admits error? It is impossible. So these types keep repeating the same year of experience over and over again until they retire or otherwise leave the field.

I didn’t make a really big deal about that snippet of code with him when I came across it (I was just a plebe anyway and there was a lot of more important stuff to do). I mentioned it to him, explained why it would not work the way he thought and then gave up when it was very obvious that the guy was not listening any more. He reminds me of a lot of programmers in that he shares that common belief that everything would be fine in the product if the programmers could just code as long as they felt was necessary. He’s not even talking design, modeling, etc. Just coding. “Just give us time to fix all the errors!” is his motto, and he means ALL the errors. In his opinion there ought to be no product releases until there are no errors at all left in the code. That type of monism bothers me worse than coding silliness, because software development is an economic process, and it exists to satisfy customer demands, not programmer demands. Unfortunately, there seem to be a LOT of programmers that think the same way he does, and they all find it convenient to blame the crappiness of their code on the short deadlines under which they are always placed by their company’s marketing departments. “If we only had the time to do it right!” Well, they all do have the time to do it right, as long as they control scope enough for “it” to be doable. But programmers are the very ones that let scope inflate uncontrollably. They do it to themselves, by perennially underestimating how long it will take them to do something, by indirectly overestimating their own abilities to create good solid code in a timely manner.

But I digress. Let’s get back to coding horrors, which is more fun. I have often said that a corporation is just a way to separate decisions from the knowledge required to make those decisions, therefore the ones that are best at separating themselves from knowledge end up making all the decisions. Here’s a case in point:

Try

For I := List.Count-1 to 0 do

Begin

DoSomething(List[i]);

End;

Except

End;

When I asked the guy who wrote this why he wrote it this way, he said he did not know at the time that Delphi had a “downto” operator. I think the silliness of the above code is obvious enough that I don’t have to belabor it. What is relevant here is that this code was written by a guy that shortly thereafter became a director of IT operations at his company. What really strikes me as interesting about that coding horror is that it was obvious that his approach to a new language was not to buy a book or two to get familiar with it, but just to go into the code without any preparation. Maybe he thinks real programmers can just get going all by themselves without any outside help. Never mind where it is that they are going TO.

Well, our next coding horror is relatively mild, but still might frighten small children. This involves the insistence that every FreeAndNil(x) in the code must be preceded by “if Assigned(x) then”. This is obviously a well-intentioned standard, and probably not harmful, but it is silly, of course. (The explanation for this that I was offered was that if x had never been assigned that FreeAndNil(x) would "cause problems". When I asked what they meant, the two of them then elaborated by saying that it would be messing up memory that only causes AV's later, not there. Or something like that.) The Free method of TObject that is called by FreeAndNil does the check for nil before doing the destroy. The discussion on the Delphi.non-tech newsgroup on this coding horror nevertheless led to some interesting learning experiences on my part. I’ve written a separate blog entry on this.

That last coding horror also brings us to the phenomenon of cargo cult programming. This is named after the interesting religious phenomenon known as the Cargo Cults. What happened was this. During WWII, many otherwise backward, primitive island tribes in the south Pacific saw that planes and other wondrous magical things brought lots of supplies and marvelous stuff to their islands. However, once the war stopped, the planes and other types of transport stopped bringing supplies and marvelous stuff to their islands. The islanders then built bamboo and wood replicas of airplanes and other types of transport in the hope that this would make the supplies and marvelous stuff return to their islands. A lot of programmers do the same thing with code and coding habits. See http://en.wikipedia.org/wiki/Cargo_cult_programming, which is quoted below:

Cargo cult programming is a style of computer programming characterized by the ritual inclusion of code or program structures that serve no real purpose. Cargo cult programmers will usually explain the redundant code as a way of working around a computer bug encountered in the past. Typically, however, they do not understand either the bug or the apparent solution (compare shotgun debugging, voodoo programming).

Cargo cult programming can also refer to the practice of (over)applying a design principle blindly without understanding the reasons behind that design principle in the first place. An example would be a novice being taught that commenting code is good, and then commenting every line with comments such as "increment i by 1"; other examples involve overly complex use of design patterns or certain forms of coding style.

The over-use of FreeAndNil, and obviously the use of “if Assigned then” before it, are examples of cargo cult programming. It is not uncommon to see a programmer use FreeAndNil to free a local variable at the very end of a method, a local variable that will immediately go out of scope and never be able to be accessed again. It makes no sense to set a variable to nil if it is about to immediately go out of scope. It can never be dereferenced. Here’s double cargo cult programming:

Procedure SomeMethod;

Var T: TSomething’

Begin

T := TSomething.Create;

…do stuff…

if Assigned(T) then FreeAndNil(T);

End;

The quote from Wikipedia about cargo cult programming also referred to shotgun debugging and voodoo programming. They also seem rather interesting. Here’s the entries on those:

Shotgun debugging is a process of making relatively undirected changes to software in the hope that a bug will be perturbed out of existence. This almost never works except in very simple programs, or when used as an attempt to work around programming language features that one may be using improperly; it usually introduces more bugs. These undirected, random changes can however cause more symptoms to occur, which assists in locating (and therefore fixing) problems.

A very common example of Shotgun Debugging can occur when people are working with complex multi-threaded applications. A programmer attempting to debug a race condition problem (such as a Deadlock), may find themselves writing debug text output code into the application (i.e. By using OutputDebugString in C++ or System.out.println in Java) to try and track down the problem. Since the output code itself will change the speed of one Thread in relation to another, this can cause the problem to disappear. Although apparently a solution to the problem, it is a fix by pure chance only, since anything else that changes the behaviour of the Threads may well cause it to resurface. For example a different computer may be used that has a faster processor, or a different scheduling system. Also any extra code that's added to the system may well negate any affect that the "fix" had.

Voodoo programming (a term derived from 'voodoo economics') is a tongue-in-cheek term for using a programming device, system or language which one does not fully understand. The implication is that the end result should not actually work, or even if it does work one does not understand why it works properly. The term can also apply to doing something which you know should not work, but actually does work, such as successfully recompiling some code which refused to compile the first time. Some voodoo programming is probably due to spurious glitches, subtle bugs (such as uninitialized data), or incorrect/misleading documentation in the compiler, APIs, or OS.

It is similar to black magic, except that black magic typically isn't documented and nobody understands it.

A person with experience of voodoo programming is sometimes called medicine man or witch doctor, such as "Java medicine man" or "C++ witch doctor" as equivalent for guru or wizard; the traditional terms imply high sophistication, study and knowledge over the matters and discipline, while voodoo programming implies getting things working but not fully understanding why.

Many coding horrors, cargo cult programming, shotgun debugging and voodoo programming are all signs of programmers that have not done due diligence to understand the tools and/or code with which they are working. There seem to be other tell-tale signs.

I think one of these is a tendency to blame the pre-existing code. A long time ago, I worked at a place where every time they came across a bug they could not figure out, they told their boss it was in the pre-existing code, and the pre-existing code needed to be re-written. One of the programmers, a primadonna who wrote the first coding horror in this blog, kept telling the management that they need to give up on the idea of fixing the existing code and let them rewrite it. He repeatedly said that the error was in “trying to build a mansion on an outhouse foundation”. Every time the management would ask how long it would take to fix a specific bug he would repeat his dictum that it made no sense trying little tweaks here and there, because the base code itself needed replacement. Finally, they had their chance to do the rewrite that was supposed to fix all the stubborn errors, including one AV that they could never figure out. When asked how long it would take, they said a week. So they were given a week without interruption to do the rewrite. But of course at the end of the week they were far from done. “Just a few more days!” they said and they were then given a few more days of closed-door uninterrupted time to complete the rewrite, just as they had requested. The three of them all huddled in one office and pounded away for ten to twelve hours a day, even on weekends. Finally, after a few weeks of this, they started giving builds to the QA department. Basic functionality was now broken, and what was even more telling was that the bugs the rewrite was supposed to fix were still there. The sad thing is that anyone that had enough software development experience and knew the code could tell that the pre-existing code was actually of higher quality than the code the soon-to-be-rewriters were layering on top of it, and that even if a rewrite was warranted, these guys were not the guys that should be doing it. In fact, these guys did not even have a correct assessment of the code they were planning to replace. (For example, the same primadonna that gave us the first coding horror above also said that the original code used a replacement memory manager that made it impossible to use third-party debugging and testing tools. However, a search for the one function call by which this replacement would occur [SetMemoryManager IIRC] revealed that, in fact, the code relied on the default memory manager and only used an alternative debug memory manager when DEBUG was defined.) Most of the times that they showed me what they considered a WTF in the pre-existing code, it turned out to be something that only advanced or experienced programmers would know, but which obviously confused them. It became such a habit for them to blame the pre-existing code that the management eventually concluded, when the rewrite proved worse than fruitless, that the whole code base should be thrown out and a replacement started immediately in the most popular Microsoft language. The collateral damage from this fiasco was huge and as usual most of the people that ended up losing their jobs in the aftermath of the missed deadlines had nothing to do with the misguided rewrite. That whole incident reminded me of something my high school orchestra teacher would say when beginners would complain that it was not their fault that the sounds coming from their violins, violas, cellos and double basses resembled the caterwauling of cats: “It is a poor musician who blames his instrument”.

There is something highly egotistical about the idea that most of the problems with a code base could be solved simply by letting us rewrite it according to our ideas of the way code ought to be written. Programmers tend to be like that though. They want to write new code and put their ideas into practice. They don’t want to have to operate within the confines of somebody else’s ideas and design, asking a programmer to do that is like asking an artist to simply touch up other artists’ work. Yet, programs usually have a lifespan that is several times longer than their gestation, implying that most of the economically useful software development is updating existing code, not creating new code. The software developer that can’t live in other software developers’ worlds is therefore of limited economic usefulness. In fact, the kind of primadonnas that insist on whining about pre-existing code and insist on rewrites are death to most projects of which they are a part. It was not simply bad luck that the project above ended in mass layoffs, it was inevitable. It simply was not possible to create solid code that did all the pre-existing code did, in just a week, or even in 4 weeks, which is roughly how long it was before the rewrite was abandoned as a failure. The rewrite was addressing code that affected nearly every part of a million-line project, simply unit testing that much code would take longer than the week they initially estimated. The risk associated with changing such a deeply influential block of code was astronomical, and way too high for anything but the very first stage of a software development life cycle. That it was being done in a project that was already several months past a promised deadline was simply unconscionable. Yet, there it was. Why? Why would apparently intelligent people engage on such a doomed mission with all the confidence of a modern army about to attack a cavalry force? I blame two factors: ego and ignorance of the economics of software development.

Ego in programming is a problem that many others have addressed sufficiently. Here’s one example, from Steve McConnell’s classic, “Code Complete”:

Great intelligence is only loosely connected to being a good programmer.

What? You don’t have to be superintelligent?

No you don’t. Nobody is really smart enough to program computers. Fully understanding an average program requires an almost limitless capacity to absorb details and an equal capacity to comprehend them all at the same time. The way you focus your intelligence is more important than how much intelligence you have.

At the 1972 Turing Award Lecture, Edsger Dijkstra delivered a paper titled “The Humble Programmer.” He argued that most of programming is an attempt to compensate for the strictly limited size of our skulls. The people who are the best at programming are the people who realize how small their brains are. They are humble. The people who are the worst at programming are the people who refuse to accept the fact that their brains aren’t equal to the task. Their egos keep them from being great programmers. The more you learn to compensate for your small brain, the better a programmer you’ll be. The more humble you are, the faster you’ll improve.

That’s good advice [in fact if you haven’t yet read Code Complete, stop reading this blog long enough to order it right now. The book is full of gems]. I think that if a programmer’s ego is big enough, he completely eliminates the possibility for growth, and so he keeps repeating the same year of experience over and over again. He might have twenty years of chronological experience but he really is only as good as a one-year programmer, because his ego keeps him from admitting, and then learning from, his mistakes. Furthermore, if his ego prevents the admission of mistakes, it surely then prevents the elimination of that error from his programming. So he will not progress past a beginning level of practical knowledge, and will make the kinds of mistakes that beginners would make (like we see at the top of this blog entry).

I read about Delphi, work with Delphi, surf the web about Delphi, for several hours every day, 365 days a year, since it came out. It is a passion of mine. But I’m still learning important and significant things about Delphi. For example, I recently discovered that if you have a nil reference to a class, that non-virtual methods will not AV until they attempt to access an instance member or a virtual method of that class, but virtual methods will AV at the point where they are called. I’ve covered this in a separate blog entry, to which I earlier referred. It is a very big world and our brains are very, very small.

There is plenty more that could be written about ego in programming, but this log needs to end some time. As for the economics of software development, that is an ongoing topic of mine, to which I’ll be returning again and again over the next few years in this space.