Saturday, June 30, 2007

A Developer's Life Stages

I ended up writing a novel as a comment on Jamie Fristrom's blog post on Speculative Generality. Read his entry. I'm reposting my comment here.

Developing The Simplest Thing That Could Possibly Work

In Forrester Research's 10 Mistakes in Software Development, #3 is overscoping a solution and #9 is jumping into development without enough research, and we've all done it. :(

A friend consulted at a large transportation company in the Kansas City area. They'd been gearing up for a massive conversion from their existing AS/400 systems to a completely Java-based system. Remarkably, in the early stages of the project, they decided to wrap the entire JDK in their own custom framework and decreed that all developers use these wrappers exclusively. As a result of this, they obtained -

  • little or no added functionality,
  • more limited abstractions than the raw JDK,
  • minimally tested code underlying everything,
  • the requirement to train all incoming Java developers on their framework and,
  • worst of all, the obligation to maintain many more thousands of lines of code.
I've observed that developers tend to progress through a number of stages in their career. These are like the stages of grief: inescapable. This is because each stage produces a set of skills and philosophies that are eventually synthesized in the seasoned software developer.

The Underengineer

The underengineer is elated at his ability to get things done quickly and be productive, disdaining others (usually the overengineer) for what they see as excessive rumination and for producing an excessive amount of code for a given task. You can count on these guys to get stuff done. Will it be done 'the right way?' Only if by accident. Underengineers are often unable to discern 'the right way,' often because they've spent little time reading and maintaining other people's code.

A developer may begin to come out of this stage when he tires of fixing or writing the same code (in different places) over and over again. Or perhaps he'll be propelled into the second stage by an enthusiastic reading of Design Patterns or a sudden grasp of UML.

The developer at this stage is fascinated by accomplishing the task at hand and it is this stage in which he learns how to code and debug.

The Overengineer

The overengineer knows that he can solve any single development task and has become fascinated by possibility. Abstraction is magical. How can any piece of code be used to solve multiple tasks? Paradoxically, this usually ends up producing more code for any one task. While the underengineer shakes his head at this, the overengineer knows that his approach will produce much less code for an entire set of similar problems. Meanwhile, the overengineer looks at the underengineer's hackery with contempt. The question the overengineer rarely asks himself, though, is will there be a large enough set of similar problems to justify this effort? And so he abstracts everything in the name of flexibility.

The overengineer is sometimes shocked to realize he's become much less productive in terms of functionality than before. He can't seem to get anything done! Still, he produces as much (if not more) code and suddenly becomes capable of building much larger systems than he could as an underengineer.

This is the stage in which the developer learns architecture and the value (and painfully, the costs) of abstraction. This is the stage of the architects of the transportation system conversion discussed above.

What kills the overengineer? Maintenance. Dependencies. Broken abstraction. Realizing that the flexiblity he built in is unused and real systems change in ways he could never have anticipated. At some point, and probably many, the overengineer will get a phone call at 2am asking him to come in and fix an issue that breaks his whole architecture. After a few of these heartbreaking moments, his world view begins to change again.

The Seasoned Developer

The seasoned developer is skilled in coding and debugging from his time as an underengineer and is a skilled architect from his time as an overengineer. What's more, he's come to understand the following:

The primary reason for abstraction is to simplify implementation for a given set of requirements.

If abstraction doesn't simplify, ditch it.

The seasoned developer also realizes that the solution domain for a given problem may extend beyond technology. People often try to use technology to solve social or organizational problems. Conversely, people frequently attempt social or organizational solutions for essentially technological problems. Recognizing these incongruities before they become a software design and implementation can help to avoid doomed projects.

The organizational problem in the transportation example above is that software architects were assigned to do a job before anyone knew what job they were to do. The proper response would be to do nothing - or better, to work on a different project until the problem domain was fully understood.

Instead they chose speculatively general busy work, incurring a much higher cost.

22 comments:

David Felstead said...

Nice post, and quite accurate :)

Unfortunately, so many folks, especially in the Java/.NET world never get past the "overengineer" stage - in fact, they receive praise for their overengineering and become those wonderful "design pattern architects".

More often than not, in my experience, the "seasoned developers" are the ones who go off on their own to start consulting, whereas the others tend to live and thrive in the corporate world.

HarryC said...

Tremendous post! Thank you...

I was thinking the same thing the other day--when I first came out of college, I thought I was hot stuff because I could get some perl scripts to compile and I knew my way around the theory of computing and programming languages, but some of the realities of production coding were hard. Copy-and-paste was no longer my friend :-)

Then I came under the tutelage of a bunch of developers-cum-architects, right after the GOF book had been published. They bought it, hook line and sinker. They engineered (and continue to engineer) architectures where applications ought to be. They see hinges everywhere, and the Java/Spring/Hibernate stack makes this too easy to do. You have thousands of lines of support code similar to the example cited--it's not code in your domain; why are you writing it?!

I'm striving to get to that next level--be the seasoned developer. Thanks for putting that thought into words.

Monkeyget said...

Nice post,
This is a spot on summary :).

sdv said...

This is quite a roadmap.

WarpedJavaGuy said...

I have also progressed through several stages in my career as a developer and currently find myself in the "do it once and do it right with no bugs" seasoned developer-ish kind of stage. I like to abstract, stub, implement, and test on a feature by feature basis. In this way I can be both productive and thorough. I can very quickly write interfaces and stubs and release them immediately to the rest of the team and keep them productive while I develop and test the real implementation. This gives me the freedom to focus more attention on writing good quality code with high unit and integration test coverage. I release it once and am fairly confident that it will work correctly.

Bob said...

I almost think you can figure out what stage someone is at by listening in on their hallway conversation topics:

Underengineer: IDEs and keyboard shortcuts, linked lists and data structures, how to use specific APIs to do stuff.

Overengineer: design patterns, academic "design principles" like interface segregation, the Law of Demeter, wrappers.

Seasoned developer: development processes like unit testing and automated builds, usability, triage "which design will be less costly", the problem domain they're writing software for.

Senzee said...

Comments on programming.reddit.com:

http://programming.reddit.com/info/22iwf/comments

From there (funny):

How the way people code "Hello World" varies depending on their age and job.

Anonymous said...

Bravo, senzee, well-said. I have a developer working for me who's definitely in the overengineer phase of his career. A bright young guy, passionate and really up on the technology we're using, but it's tough sometimes keeping him focused on our goals. When he describes some gee-whiz model-service-factory-manager that will only take him two weeks to implement, my first question is "What does this get us?" If his answer starts with (and it usually does) "well, someday, if we ever need to..." I stop him right there and nix the idea. I cut him a lot of slack, though, because he does produce some good work, and I remember when I went through that phase myself.

And although I'm not a member of the Church of XP, my two favorite mantras are borrowed from those guys: "You ain't gonna need it" and "do the simplest thing that could possibly work."

thecodewitch said...

I think we've internalized the message that decisions made early on can have a massive impact later in the project. We've probably seen lots of examples of ugly code and uglier workarounds, and decided we can do better.

Because of this, a lot of time is spent trying to get these decisions right. This leads to debilitating periods of endlessly trying to find the neatest, most concise and conceptually elegant way to represent something. You want your objects to be decoupled, atomic, simple and clean.

I think its best to internalize a different message - its better to build early and make a mistake, than not to build anything at all. You can redo the whole thing with a different approach, with a deeper appreciation of what is important and how to do it better next time, and the journey is much more enjoyable and interesting as a result as well, because we get lots of feedback along the way (feedback in the form of running the thing you're making)

(This is taken from my post to irrlicht forum a while ago - http://irrlicht.sourceforge.net/phpBB2/viewtopic.php?t=25748 )

Greg
http://createuniverses.blogspot.com/

Martijn Faassen said...

Good article. I wrote something similar about a year ago that might be interesting to readers here:

http://faassen.n--tree.net/blog/view/weblog/2007/11/01/0

Martijn Faassen said...

This time with an actually clickable link

Jessta said...

I'd imagine I'm still in the underengineer stage because I still find it difficult to work out why so many projects are so huge and complex and I wonder why our programming tools are so bug prone.
So I started a programming language design project to try and discover this because I still expect that my inexperience is why I can't yet see the explanation.

Alex said...

@WarpedJavaGuy

You release the interfaces and stubs, and then write the implementation? Has it never turned out while writing the implementation that the original interfaces had been poorly chosen?

Anonymous said...

I'm not being nitpicky, but I don't agree with most of what you're saying.

In fact, I'd add this other stage in there, between your over-engineer and the seasoned developer, and I'd call it "the developer who *thinks* he's seasoned, and can make generalizations about others".

While your description of the seasoned developer is more accurate than not, your steps before it aren't in my opinion.

In my opinion, the first stage is a grasshopper: whatever his characteristics are, they are statistically insignificant, as every person brings along with them entirely creative ways to be wrong and unproductive.

The second step is the over engineer.

The third and - again, in my opinon is the under-engineer. Who'd rather use a combination of `at` and `grep` and a simple text file than write a daemon with a framework that launches some limited trigger every 12 days. The under-engineer does the bare minimum, as you said, but has the knowledge to recognize what that bare minimum will mean with insight into the full lifecycle of the product.

And all along this, a programmer can or can not be a social communicator. A developer who can understand a problem in the context of its domain.
Only when a dev reaches the under-engineer part and is a social communicator is he great.

s├Şgemaskineoptimering said...

Liveblogging Blogher: Life Stages of Online Communities ... has been community manager for multiple developer and end-user communities...and ..

rachat de credit said...

.. I ended up writing a novel as a comment on Jamie Fristrom's blog post on Speculative Generality. Read his entry. I'm reposting my comment ...

Joshua Smith said...

Thanks a lot for talking about useful, good review. It needs to know that outsourced software development services could help in your business by installing custom development software applications.

Buy Luvox Online said...

I really enjoy reding your posts as I learn a lot from them. I also broaden my thinking as far as what I can use and do with things

Generic Claritin said...

I just found your site and wanted to say that I have really enjoyed browsing your posts.In any case I'll be subscribing to your blog and I hope you post again soon

Racing said...

Hello really such a great post admin keep posting such a interesting posts thanks for sharing this blog.
Lingerie

Rummy Card Game said...

Wow, interesting Article, i enjoyed it a lot, i am going to share this link on face book.

iPhone Developers said...

This seems great and means a lot to me, nice and informative post.