Spaces, Paths, Paces (Part 1)

Objective

Development processes start with requirements and wind up in code; in between there isn’t much of a consensus among the software engineering community about how to define the scope (spaces), how to sequence the tasks (paths), and how to time deliveries (paces). On one side of the debate phased approaches hope for fixed spaces and ordered paths but often get entangled in moving lines. On the other side of the debate agile teams try to find their space by increments but risk losing the path while still on their way.

This lack of agreed upon concepts and principles entrusts personal skills and best practices as primary success factors. Conversely, that could explain the rate of failures for software projects, significantly higher than for “hard” engineering ones; given the quasi absence of physical constraints, the opposite would have been expected, which would suggest some critical intrinsic flaw.

With the “benefits” of hindsight and agile assessment of waterfall flaws, the focus has been put on fixed scope and schedule, in particular with regard to requirements and quality management:

• Fixed requirements set upfront: since there is an inverse relationship between the level of details and the reliability and stability of requirements, staking the whole project on requirements fully defined at such an early time is arguably a very hazardous policy.
• Quality as an afterthought: given that finding defects is not very gratifying when undertaken in isolation, delegating the task will offer few guarantees if not associated with rewards commensurate to findings; moreover, quality as a detached concern may easily turn into a collateral damage when set along mounting costs and scheduling constraints. Alternatively, quality checks may change into a more positive endeavor when conducted as an intrinsic part of development.

Agile answer to those failings has been to conduct specifications, development, and quality assurance into integrated iterations. As a consequence, the definition of scope becomes a byproduct of development cycles, with requirements itemized as features in order to be developed progressively. Moreover, with specifications and schedules managed dynamically, timetables become impracticable and deliveries can only be carried out by shuttles.

The agile “reformation” has open new perspectives and beget many fruitful practices, and the objective here is to see how those approaches of scope and schedule can be reformulated within the perspective of architecture layers. This part examines the congruence between the alternate flows of use cases and the backlog of users’ stories, and considers their complementarity as path-finders. The second part will focus on the role of time-boxes as pace-makers and the benefits for quality assurance.

Architectures and Projects Scope

Whatever the compass, agile or phased, projects footprint can be set across three architecture layers:

• Enterprise architectures describe business environments and objectives, resources and regulatory constraints.
• System architectures describe enterprises in terms of functional entities of human agents and physical and software assets.
• Technical architectures describe the platforms supporting functional entities.

Projects are meant to carry out changes within architectures initiated by business, engineering, or services management processes:

• Business processes are defined by business environment and objectives. Changes may have to deal with domains and activities, organization and supported operations, and quality of service as experienced by users.
• Engineering processes focus on the development of software systems supporting business processes: business domains and applications, system functionalities, platform implementations.
• Services management stand between engineering deliveries and operational concerns: location of assets, access to services, releases deployment, and systems configuration.

While development projects may (and will usually) cross architecture layers, their roots and stakes should nonetheless be clearly positioned if projects are to be planned within the respective time-spans, governed by the relevant authority, and their products accepted by the right stakeholders.

That put projects governance at crossroads between (a) business objectives set by market opportunities, (b) the deployment of features into functional architectures, and (c) the deployment of releases according to changes in technical architecture. With phased developments, scope and schedules are fixed upfront, which means that the business layer forces its time-frame over the ones of system and technical layers, which may introduce frictions regarding scope as well as quality:

• With regard to scope, frictions stem from features and schedules set fully and definitively at enterprise level, independently of any feedback from functional and technical layers.
• With regard to quality, frictions stem from tests performed at technical layer when scope and schedules can no longer be revised in case of negative outcomes.

The consequences are all too easy to observe, with business needs partially satisfied and software quality sacrificed. Hence the need of a balanced approach that would consolidate the different maps and time-frames in order to minimize frictions between layers.

Mapping Project Scope

Projects scope can be described along two dimensions, one set by business logic, the other by system functionalities:

• First, one have to circumscribe the business variants to be taken into consideration. For that purpose the project footprint, first introduced as users’ stories, will have to be documented by activity or business process diagrams.
• Then, the project scope will have to mark out the subset of business requirements to be supported by system functionalities. That will usually be done with use cases describing interactions between system and users.

That makes those descriptions both orthogonal and complementary: orthogonal because use cases cut across activity diagrams, complementary because use cases are meaningless without targeted activities.

More importantly,  they are associated with different architecture layers and governed by different concerns:

• At business level (business processes or activities), the perimeter and granularity of requirements must be congruent with the continuity and consistency constraints of business objects and operations.
• At functional level (use cases), the span and granularity of interactions between system and users must coincide with execution paths. But the rationale governing users interactions is not the same as the one governing the integrity of business processes. As a consequence, the paths considered for development may pick sequences of operations defined by business processes but should not define them anew based upon interaction constraints.
• Finally, assuming that use cases see systems as black-boxes, their footprint should not depend on decisions taken at technical level.

Those concerns can be dealt with separately if projects scope is explored iteratively, e.g using activity diagrams for business logic and use case diagrams for users interactions.

Iterative Mapping of Project Footprint

Iterative development is not just about increments but, first and foremost, about exploring development spaces. That is especially useful when projects overlap architecture layers and cannot rely on fully fledged requirements.

Such projects have to deal with two challenges:

• They must identify and manage work units according to the state of requirements and the nature of dependencies (business, organization, or technology, …).
•  They must carry on with developments based on incomplete specifications while exploring alternatives and deferring decisions until the “last responsible moment” when further delay would limit the options at hand.

Taking a leaf out of the agile book, projects should be driven by users’ value, with requirements first introduced as users’ stories. From that springboard, as informal and incomplete as could be, stories must be fleshed out and organized in order to support the reasoned exploration of project scope.

At inception stories are no more than a user, an objective, and an activity, all set at business level independently of the part played by systems. Scope exploration must therefore begin with activities backbone and be furthered with variants, aka scenarii.

Given a set of business scenarii, the candidates for system support must be ordered and mapped to system features, actual or planned. That should provide a blueprint for development paths. Unfortunately, as variants are added to plots, narratives can easily turn messy, mixing features and capabilities across architecture layers. And that’s where the benefits of use cases are to be found.

Development Paths: From Users’ Stories to Use Cases

Whatever the context, iterations are formal constructs defined by invariants, increments, and exit conditions. When applied to development spaces, iterations are defined by:

• Invariants: conditions on architectural assets supporting the scenario under consideration.
• Increments: features or variants added to scenarii.
• Exit condition: no more features or variants (empty backlog) or time-out.

Applied to architecture layers, invariants provide for reasoned iterations and backlogs:

1. Enterprise layer are the first to be considered: cycles are set for persistency and execution units and bound by domains (identification mechanisms, integrity constraints, and semantics); within cycles, increments target attributes, operations and variants.
   
2. Functional layer come second: cycles are set for interaction units (aka use cases), and bound by the continuity and consistency business objects and activities; increments target transient attributes and operations. 
3. Technical layer come last: cycles are set for platforms and bound by functional units.

But there is a catch: while users’ stories and activity (or business process) diagrams are set at enterprise level, development projects are considered at system level; because systems functionalities are not supposed to appear in users’ stories or activity diagrams, there could be a gap between business and functional requirements. As it happens, use cases provide a bridge: on one hand they focus on the interactions between users and systems, on the other hand their basic and alternate flows can be directly mapped to the paths in activity diagrams.

That provides a clear and sound basis for the definition of development paths: on one hand alternate flows can be ranked according users’ priorities; on the other hand they determine the sequence of use cases that will have to be developed.

Backlogs and Pathfinders

While the objective of users’ stories is to tie up projects in business value, the objective of use cases is to anchor them in the context of system functionalities. That perspective, and the role of models, may be ignored for standalone projects, but it is necessary when project development paths are to be governed both by business and functional dependencies, described respectively by users’ stories and use cases.

In that case the exploration of development paths should be guided by invariants set along MDA model layers: computation independent (business processes), platform independent (systems functionalities), and platform specific (technology platforms).

When projects are rooted in business activities (a), e.g the possibility of upgrading a customer, stories describe execution paths and are ranked according business priorities. Iterations will proceed with development and new cycles added for alternative paths to the basic one.

Depending on context dependencies, development projects can be directly initiated from given sequences of activities (b) or conducted in parallel with users’ stories. Use cases remain the option of choice when the features supporting users’ stories are meant to be shared (e.g checkout). In that case the development paths are governed both by users’ value and functional dependencies. When features are deemed specific (e.g upgrade), use cases can be bypassed and development paths explored simultaneously according users’ and development concerns.

Backlog organization is more complex when development paths cross the divide between functional and technical concerns. Ideally, one would expect a clear separation of concerns, with use cases defined independently of technical options, just like business logic doesn’t depend on system functionalities. But alternatives may be blurred due to the dependencies between interactions design and platform capabilities, the risk being to associate technical options with functional variants, e.g specialized use cases.

That’s the case when features can only be implemented on specific platforms. If those features are also specific the corresponding development cycle can be managed as a whole. Otherwise the relevant decisions should be factored out. The same principle applies for features supporting different business processes.

Squaring the Circles: From Epics to Releases

Iterations run within boundaries set by invariants, and with regard to projects scope, those invariants are set by architecture capabilities: enterprise on one hand, systems on the other hand.

From the enterprise perspective, development projects (b) are meant to support business objectives (a), not to define them. As a consequence, users’ stories must remain within borders set upfront. That can be achieved by introducing business projects (aka strategies, aka epics) and portfolios of development ones.

From the system perspective, a clear distinction should be maintained between projects supported by platforms capabilities (b), and projects targeting platforms capabilities (d). Eventually, those different levels of explorations will have to be consolidated as releases (e), and that is where one may find the agile answer to waterfall.

A Time for Every Purpose: Time-boxes as Pace-Makers

As Einstein famously said, “The only reason for time is so that everything doesn’t happen at once.”  In other words time is what happens between events, and the use of a single time-frame will put all events under the same rationale.

But architectures are best understood as shearing layers whose events are governed by different rationales, respectively: business opportunities, engineering constraints, or operational needs.

That is arguably the critical flaw of waterfall solutions as they force business, development, and operations under the same set of strictures. And that’s why agile’s solution to components release may be its pivotal innovation as it establishes the autonomy of those three layers and introduces time-boxes as their pace-makers.

Further Reading

• Time and Events
• Agile and Phased approaches
• Agile and Models
• Work Units
• Products, Projects, Processes
• Project Management