em

HackNot - Design

Part of Hacknot: Essays on Software Development

The Folly of Emergent Design1

One of the most pernicious ideas to proceed from the current focus on lightweight methodologies is that of Emergent Design. It’s difficult to find a precise description of emergent design – most discussion on the subject carefully avoids committing to any particular definition. One of the most succinct descriptions I’ve encountered is this, from the adaptionsoft web site:

“Many systems eventually require drastic changes. You cannot anticipate them all, so stop trying to anticipate any of them. Code for today, and keep your code and your team agile.”2

Proponents of Emergent Design tout the following advantages of such an approach:

  • Visible signs of progress appear more quickly .
  • The system reaches a state in which it can be evaluated by customers sooner, which is useful for verifying existing requirements and teasing out as yet undiscovered requirements.
  • The risk of “analysis paralysis” is eliminated.
  • No effort is wasted in the preparation of infrastructure to support anticipated requirements that never actually manifest.
  • An increased ability / willingness to adapt to changing requirements, as the development effort is not burdened by prior commitment to a particular solution approach.

Opponents of Emergent Design claim the following disadvantages:

  • Exploration of alternative solutions takes much longer when using code as the vehicle for exploration, rather than a more abstract medium such as UML.
  • The “code for today” approach discourages the reaping of long term savings in implementation effort by investing in supporting functionality in the short term.

Proponents will counter these by referencing the incremental nature of constant refactoring. Opponents will counter this with appeals to the benefits of a middle ground where “just enough” design is partnered with early prototyping3. Eventually, somebody makes comment on somebody else’s mother and her preference for military footwear, and all hope of rational discussion is lost.

An Example Of The Hazards Of Emergent Design

As near as I can ascertain, the project upon which I am currently working employs Emergent Design, although there has been no explicit statement to that effect. At the beginning of the year there were one or two group design sessions, which identified the major subsystems of the product and how they would collaborate to achieve one of the principal use cases. Since then, any design efforts which have occurred have been of an incremental nature, and generally done “on the back of an envelope” as individuals have struggled to implement various aspects of a subsystem’s functionality against pressing deadlines. Thus, developers have only done what was necessary to achieve the functionality need for the task at hand – which seems consistent with the philosophy of Emergent Design.

The resulting code base bears some interesting characteristics which I believe illustrate some of the difficulties inherent with the practical application of an Emergent Design approach. To illustrate, consider the following three classes from the application’s current code base, presented here in abbreviated form:

  public class YearLevel {
    public YearLevel(NormYearLevel,
                     Country, String, String);
    public getNormYearLevel() : NormYearLevel;
    public getCountry()       : Country;
    public getScanText()      : String;
    public getLabel()         : String;
  }
  public class NormYearLevel {
    public static final NormYearLevel
      NORM_YEAR_1 = new NormYearLevel(1);
    public static final NormYearLevel
      NORM_YEAR_12 = new NormYearLevcel(12);
    private NormYearLevel(int aYearLevel);
  }
  public class RawYearLevel {
    public RawYearLevel(String aScanText);
  }

The main purpose of this application is to process the responses of junior and secondary school students to multiple choice exams. A given exam may be taken by students from different countries and therefore different educational systems. The results are captured in individual and aggregate reports, which are printed and dispatched to the participating schools.

It takes as input the data files resulting from the optical scanning of the exam papers. Students indicate their “year level” as defined by the educational system in force in their country (a “year” is variously referred to as a “grade”, “form” etc). For example, a student in year 3 in Australia would indicate a “3”; a student in Grade 4 in France would indicate a “4” and so on.

What is notable about these three classes is that they represent three different aspects of the same concept, and might well have been collapsed into a single abstraction. More significant than the choice and number of abstractions used to represent the concept, is the way these disparate representations came into being. Each was created by a different developer, working in a different subsystem from the others, and employing a philosophy consistent with Emergent Design. A review of the version control history for each class traces their genesis.

First came RawYearLevel, conceived of and implemented by a developer concerned with the early stages of the data processing pipeline, as a way of representing the student’s literal indication of what year they were in.

In parallel with RawYearLevel, the YearLevel class was created by a second developer working in another subsystem, who was focusing on the opposite end of the pipeline, where the results are embodied in hard copy reports. The YearLevel class (without the NormYearLevel association) captured enough information to print on a report “This student was in Year 6” or “This student was in Grade 8”, depending on the country and the educational system it employed.

Lastly came the NormYearLevel class, created by a third developer working in a subsystem between the two mentioned above, that was responsible for calculating individual and population statistics. In the course of these calculations it becomes necessary to relate a year level in one country with its educational equivalent in another country. So the concept of a Normative Year Level was introduced, and the YearLevel abstraction was country-specific augmented to be associated with it’s normative equivalent.

Each of these classes has “emerged” from an individual developer’s immediate need to implement some portion of a subsystem’s functionality. To meet that need, they have done the simplest thing that could possibly work4. That often means writing a class from scratch. If another developer creates the same or a similar abstraction in parallel, each will be unaware of the duplication until their work is integrated. Sometimes it is considered simpler to get partial leverage from an existing abstraction. In either case, the imperative is to achieve the target functionality as quickly as possible, such is the time pressure the developers are under (a situation common to many development shops). It is by no means certain that the design issues surrounding these abstractions will ever be revisited.

Just Refactor It

The inefficiency of maintaining the above three abstractions is compounded by the amount of surrounding code that does little more than map from one type to another. Proponents of Emergent Design would suggest that the problem can be very simply overcome – just refactor the code. Of course, this is entirely possible. However there are some very real reasons why the abstractions have persisted in the application for 6 months or more, and have not been eliminated through refactoring.

  • Nobody considers the refactoring to be of high enough priority to warrant spending our limited developer resources on. The task is not immediately related to any particular operational requirement, and so it is viewed as being less important than making functional progress.
  • There is considerable psychological inertia associated with a body of code that is basically functional. Refactoring will mean losing that functionality for the duration of the refactoring task, and so superficially appears a retrograde step.
  • The classes have become part of the vocabulary of the developers, and they have come to think of them as being an intrinsic part of the system i.e. their presence is not openly questioned.

Constraining Evolution Leads To Mutants

Emergent Design is frequently likened to the process of evolution. Proponents speak of “evolving a design” , the implication being that some software equivalent of natural selection is weeding out the inferior mutants, leaving only the fittest to survive. If this is the case, why have the three classes above not evolved into a better design? Or is that evolution yet to occur? Or are these three classes actually the fittest to survive already, for some suitable definition of “fittest”?

I conject that the practical application of Emergent Design so constrains the evolution of the design elements that we cannot expect such an approach to have a reasonable chance of giving rise to a good design.

Comparing the evolution of a software design with the evolution of a species, we see the following significant differences:

  • Evolution can take its time exploring as many dead ends and genetic cul-de-sacs as it likes. There is no supervising authority standing by looking for visible signs of monotonic progress. There are no time constraints or fiscal limitations that require evolution to produce a workable result within a certain number of generations.
  • Evolution can explore many alternatives in parallel, but a development group will rarely have sufficient resources to try a large number of different design alternatives in parallel. A very limited number of resources assigned to a design task must try alternatives in series, if at all. Obviously there is a strong tendency to stick with the first one tried that appears to hold promise.
  • Evolution is objective in its evaluation of the success of each alternative. There is no attachment to a genetic alternative that is nearly good enough. However software developers often favor “pet” design approaches, or try and force non-optimal designs further than they should go because there is the promise of success just around the corner, and the attendant resolution of an uncompleted task. That is to say, it is very human to normalize deviation.
  • Evolution is not required to be predictable. No one has bet their financial future on the lesser fairy penguin evolving heat dissipation mechanisms to cope with increasing Arctic temperatures, and doing it in no more than 3 generations. But stakeholders in software development efforts will commonly invest large sums to see successful designs produced (and thereby business problems solved) within a limited contract period.

You will find any number of elegant analogies in the Emergent Design literature – but finding one that addresses the above constraints is quite another matter.

For example, there is the delightful story (probably apocryphal) of the landscaping engineer who was asked to cement pathways at a University, after the buildings had been erected. Rather than predict the correct place to put the pathways, the engineer stood back for one semester and let the students make their own way between buildings. The furrows they wore in the ground were adopted as the courses for the cement pathways.

How very Zen… really, it’s a terrific tale. I love it. But before we spin our prayer wheels and marvel at the engineers’ wisdom, let’s think of the liberties that the landscape engineer was allowed in pursuing such a solution method. Liberties which would be denied a great many University contractors in the real world:

  • The landscape engineer was allowed to take the time necessary to wait for the paths to emerge. What if the University had required completion sooner than that – say, before the semester started?
  • The landscape engineer was allowed an entire semester in which he was not required to demonstrate visible progress. What if a competitor had taken advantage of this lull and offered to complete the job using best guesses of the correct routes for the pathways.
  • The landscape engineer was free to distribute the labor and materials cost over the course of the project as he saw fit. What if the budgeting system of the University had made allowances for expenditure on landscaping in this semester, but not in the following one?
  • An entire University cohort spent a semester walking through the mud after every rainfall. They were willing to put up with this discomfort so that the engineer could let his design emerge. I wonder how the senior lecturers felt about this. More importantly, I wonder how those students in wheelchairs coped.

Emergent Design has the capacity to lead to some very elegant solutions – eventually. That design may be wonderfully efficient – if you have the financial stamina to await its arrival and the confidence that you will recognize it when it appears.

Conclusion

Does Emergent Design work? Of course - just look in the mirror. You and every other product of evolution is testament to the potential success of the approach.

Does that imply that it is a suitable model for designing software? No.

While the idea has aesthetic appeal, the practical context in which the emergence occurs makes all the difference. The requirements for timeliness and predictability in a software development project, together with the subjective nature of those who gauge the cost/benefit of a particular approach, mean that true, uninhibited evolution cannot occur. If the compromises embodied in an emergent design are consistent with our corporate priorities, then it will be by coincidence only – and that’s too important a matter to leave to chance.

The Top Ten Elements of Good Software Design5

“You know you’ve achieved perfection in design, not when you have nothing more to add, but when you have nothing more to take away.” – Antoine de Saint-Exupery

Much is spoken of “good design” in the software world. It is what we all aim for when we start a project, and what we hope we still have when we walk away from the project. But how do we assess the “goodness” of a given design? Can we agree on what constitutes a good design, and if we can neither assess nor agree on the desirable qualities of a design, what hope have we of producing such a design?

It seems that many software developers feel that they can recognize a good design when they see or produce one, but have difficulty articulating the characteristics that design will have when completed. I asked three former colleagues – Tedious Soporific, Sparky and Willa Wonga – for their “Top 10 Elements of Good Software Design”. I combined these with my own ideas, then filtered and sorted them based upon personal preference and the prevailing wind direction, to produce the list you see below. A big thanks to the guys for taking the time to write up their ideas.

Below, for your edification and discussion, is our collective notion of the Top 10 Elements of Good Software Design, from least to most significant. That is, we believe that a good software design …

10. Considers The Sophistication Of The Team That Will Implement It

Does it seem odd to consider the builder when deciding how to build? We would not challenge the notion that a developer’s skill and experience has a profound effect on their work products, so why would we fail to consider their experience with the particular technologies and concepts our design exploits? Given fixed implementation resources, a good design doesn’t place unfamiliar or unproven technologies in critical roles, where they become a likely point of failure.

Further, team size and their collocation (or otherwise) are considered. It would not be unusual for such a design’s structure to reflect the high level structure of the team or organization that will implement it.

9. Uniformly Distributes Responsibility And Intelligence

Classes containing too much intelligence become both a point of contention for version control purposes, and a bottleneck for maintenance and development efforts. They also suggest that a class is capturing more than a single data abstraction.

8. Is Expressed In A Precise Design Language

The language of a design consists of the names of the entities within it, together with the names of the operations those entities perform. It is easier to understand a design expressed in precise and specific terms, as they provide a more accurate indication of the purpose of the entities and the way they cooperate to achieve the desired functionality. Look for the following features:

  • The objective of the designed thing can be described in one or two sentences completely.
  • The interface requirements of the entities are stated precisely.
  • The contracts between an entity and its callers are stated precisely and contract adherence is enforced programmatically (Design by Contract).
  • Entities are named with accurate and concrete terms, and specified fully enough to form a suitable basis for implementation.

7. Selects Appropriate Implementation Mechanisms

Certain mechanisms are problematic and more likely to produce difficulties at implementation time. A good design minimizes the use of such mechanisms. Examples are:

  • Reflection and introspection
  • Dynamic code generation
  • Self-modifying code
  • Extensive multi-threading

Sometimes the use of such mechanisms is unavoidable, but at other times a design choice can be made to sacrifice more complex, generic mechanisms for those easier to manage cognitively.

6. Is Robustly Documented

As long as a design lies hidden in the complexities of the code, so too does our ability arrive at an understanding of the code’s structure as a whole. As the abstract structure becomes apparent to us, either through rigorous examination of the code or study of an accompanying design document, we gradually develop a course understanding of the code’s topography. A good design document is used before or during implementation as a justification and guide, and after construction as a way for those new to the code base to get an overview of it more quickly than they can through reverse engineering. Captured in abstract form, we can discuss the pros and cons of different approaches and explore design alternatives more quickly than we can if we were instead manipulating a code-level representation of the design.

But as soon as the abstract and detailed records of a design part company, discrepancy between the two becomes all but inevitable. Therefore it is essential to document designs at a level of detail that is sufficiently abstract to make the document robust to changes in the code and not unnecessarily burdensome to keep up to date. A good design document should place an emphasis upon temporal and state relationships (dynamic behavior) rather than static structure, which can be more readily obtained from automated analysis of the source code. Such a document will also explain the rationale behind the principal design decisions.

5. Eliminates Duplication

Duplication is anathema to good design. We expect different instances of the same problem to have the same solution. To do otherwise introduces the unnecessary burden of understanding two different solutions where we need only understand one. There are also attendant integrity problems with maintaining consistency between the two differing solutions. Each design problem should be solved just once, and that same solution applied in a customized way to different instances of the target problem.

4. Is Internally Consistent And Unsurprising

We often use the term “intuitive” when describing a good user interface. The same quality applies to a good design. Something is “intuitive” if the way you expect (intuit) it to be is in accord with how it actually is. In a design context, this means using well-known and idiomatic solutions to common problems, resisting the urge to employ novelty for its own sake. The philosophy is one of “same but different” – someone looking at your design will find familiar patterns and techniques, with a small amount of custom adaptation to the specific problem at hand. Additionally, we expect similar problems to be solved in similar ways in different parts of the system. A consistency of approach is achieved by employing common patterns, concepts, standards, libraries and tools.

3. Exhibits High Cohesion And Low Coupling

Our key mechanism for coping with complexity is abstraction – the reduction of detail in order to reduce the number of entities, and the number of associations between those entities, which must be simultaneously considered. In OO terms this means producing a design that decomposes a solution space into a half dozen or so discrete entities. Each entity should be readily comprehensible in isolation from the other design elements, to which end it should have a well defined and concisely stateable purpose. Each entity, be it a sub-system or class, can then be treated separately for purposes of development, testing and replacement. Localization of data and separation of concerns are principles which lead to a well decomposed design.

2. Is As Simple As Current And Foreseeable Constraints Will Allow

It is difficult to overstate the value of simplicity as a guiding design philosophy. Every undertaking regarding a design – be it implementation, modification or rationalization – begins with someone developing an understanding of that design. Both a detailed understanding of a particular focus area, and a broader understanding of the focus area’s role in the overall system design, are necessary before these tasks can commence.

It is necessary to distinguish between accidental and essential complexity6. The essential complexity of a solution is that which is an unavoidable ramification of the complexity of the problem being solved. The accidental complexity of a solution is the additional complexity (beyond the essential complexity) that a solution exhibits by virtue of a particular design’s approach to solving the problem. A good design minimizes accidental complexity, while handling essential complexity gracefully. Accidental complexity is often the result of the intellectual conceit of the designer, looking to show off their design “chops.” Sometimes a “simple” approach is misinterpreted as being “simple-minded.” On the other hand, we might make a design too simple to perform efficiently. This seems to be a rather rare occurrence in the field. As the scope of software development broadens at the enterprise level and attracts greater essential complexity, the reduction of accidental complexity becomes ever more important.

1. Provides The Necessary Functionality

The ultimate measure of a design’s worth is whether its realization will be a product that satisfies the customer’s requirements. Software development occurring in a business context must provide business value that justifies the cost of its construction. Also of significant importance is the design’s ability to accommodate the inevitable modifications and extensions that follow on from changes in the business environment in which it operates.

But it is necessary to exercise great caution when predicting future requirements. An excessive focus upon anticipatory design can easily result in wasted effort resulting from faulty predictions, and encumber a design with unnecessary complexity resulting from generic provisions which are never exploited. Terms like “product line” and “framework” may be warning signs that the design is making high-risk assumptions about the future requirements it will be subject to.

It is easy to overlook the non-functional requirements (e.g. performance and deployment) incumbent upon the design. Taking different “views” of the design, in the manner of the “4+1” architectural views in RUP7, can help provide confidence that there are no gaping holes (functional or otherwise) and that the design is complete.

  1. First published 14 Oct 2003 at http://www.hacknot.info/hacknot/action/showEntry?eid=29 ↩

  2. http://www.adaptionsoft.com/xp_practices_simple_design.html ↩

  3. Extreme Programming Refactored, M Stephens and D Rosenberg, Apress, 2003 ↩

  4. http://xp.c2.com/DoTheSimplestThingThatCouldPossiblyWork.html ↩

  5. First published 18 May 2004 at http://www.hacknot.info/hacknot/action/showEntry?eid=54 ↩

  6. The Mythical Man Month, Anniversary Edition, F. Brooks, Addison-Wesley,, 1995 ↩

  7. Rational Unified Process, P Kruchten, Addison-Wesley, 1999 ↩