Thread: Agile

Preamble

All too often when agile development models are discussed, the debate turns into a religious war with Waterfall cast as the villain. But throwing out phased babies with flawed waters is to ignore the diversity of situations: agile is not a panacea and misguided exercises may hamper a broader adoption of its principles. Hence the need of reasoned guidelines:

• To begin with, agile is in the spirit of its principles, not in the letters of implementation procedures.
• As a consequence its use should be specific to projects but aligned with enterprise circumstances.
• Then, the development of users’ stories is the cornerstone of any agile project.
• Finally, the schemes used to explore the space (scope), define the paths (backlog), and time the paces (deliveries) may vary with implementations.

The Scope of Agile Principles

The Agile development model as pioneered by the eponymous Manifesto is based both on universal principles meant to be applied in any circumstances, and on more specific ones subject to some prerequisites. Sorting out the provisos may help to extend and improve the agile footprint.

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

• Scope: Specific to agile development model.
• Requisite: Iterative and incremental development.

2. Welcome changing requirements, even late in  development. Agile processes harness change for  the customer’s competitive advantage.

• Scope: Universal.
• Requisite: Requirements traceability and modular design.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

• Scope: Universal.
• Requisite: Modular design and incremental development.

4. Business people and developers must work together daily throughout the project.

• Scope: Specific to agile development model.
• Requisite: Shared ownership, no external dependencies.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

• Scope: Universal.
• Requisite: Dedicated organization and human resources policy.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

• Scope: Universal.
• Requisite: Corporate culture.

7. Working software is the primary measure of progress.

• Scope: Universal.
• Requisite: Quality management and effective assessment of returns and costs.

8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

• Scope: Universal.
• Requisite: Dedicated project management and human resources policy.

9. Continuous attention to technical excellence and good design enhances agility.

• Scope: Universal.
• Requisite: Corporate culture and human resources policy.

10. Simplicity–the art of maximizing the amount of work not done–is essential.

• Scope: Universal.
• Requisite: Quality management and corporate culture.

11. The best architectures, requirements, and designs emerge from self-organizing teams.

• Scope: Universal.
• Requisite: Shared ownership and no external dependencies.

12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

• Scope: Universal.
• Requisite: Dedicated organization and corporate culture

Perhaps surprisingly, most of agile principles extend far beyond the eponymous methods and should bring benefits to all and every type of project. Of a tally of twelve only two (#1 and #4) cannot be applied universally.

Phased vs Iterative

Compared to administrative processes, collaborative ones entail some kind of shared ownership, with teams granted full responsibility for decisions, schedules, and quality. When that cannot be achieved the different concerns, purposes or authorities, possibly but not necessary at odds, are to be managed by different organizational units, and “frozen” process states exchanged as value objects between those units.

Taking the basic Requirements/Development/Deployment activities for example, phased approaches will segment processes accordingly and freeze the respective deliverables; conversely, iterative ones  will combine activities into a single cycle repeatedly specifying, developing, qualifying, and deploying homogeneous deliverables.

That makes phased and iterative processes defined along completely different perspectives. The former with regard to the specifics of development flows:

• Tasks differentiation, depending on contexts, purposes, and responsibilities.
• Milestones,  introduced for the coordination of participants.
• Intermediate products, for consolidated outcomes exchanged between participants.

The latter with regard to the dynamics of control loops:

• Scope: the same activity is repeated until a given set of specifications (invariant) is satisfied (exit).
• Responsibility: different activities are repeated by a given organizational unit (invariant) until it realizes its objective (exit).
• Quality: the same activity is repeated until a given set of constraints and measures (invariant) is satisfied (exit).

Comparison can be further refined when projects are mapped to the OMG’s Model Driven Architecture (MDA), which introduces a distinction between business, functional, and technical concerns respectively represented by computation independent (CIM), platform independent (PIM), and platform specific (PSM) models.

Before deciding between iterative and phased, the first step should be to look for archetypes: complex and  standalone applications for the former, shared or architecture based functionalities for the latter. Then, archetypes set apart, iterative and phased development could be combined when external dependencies are to be taken into account, with models used as the glue :

• Organizational dependencies are managed across enterprise architecture layers: business requirements govern system functionalities which govern platforms implementations.
• Functional dependencies are  managed across architecture functional tiers: transient non shared components (aka boundaries) are governed by transient shared components (aka controls) which are governed by persistent shared components (aka entities).
• Development dependencies should not cross projects limits as they should be managed at domain level using inheritance or delegation.

On a broader perspective, as illustrated by the agile focus on users’ stories, the iterative vs phased alternative can be set against the business vs engineering point of view: while the former is synchronic as business objectives, constraints, and risks change continuously with environment and opportunities, the latter is diachronic as engineering must ensure the stability and consistency of shared systems’ assets and mechanisms. Challenges begin when they cannot be set apart.

Agile Falls

Phased processes are built along milestones and engage different organization units, raising the risk of cumbersome bureaucracies and significant overheads. Agile processes for their part are made of repeated cycles governed by cooperation, cutting down milestones and intermediate deliverables. But while they fix some critical flaws of phased approaches, agile solutions come with their own limitations.

Agile solutions effectively correct the two primary faults of phased solutions, namely an upfront and unyielding management of scope and commitments, and a delayed and detached approach to quality. But that is achieved by putting specific business needs in the driver seat and assuming no cross architectural or organizational dependencies. Set within a MDA framework, there will be no consolidation of computation independent (CIM), platform independent (PIM), and platform specific (PSM) models.

That may lead to redundancies or even inconsistencies between applications, and would probably curtail the scope of reuse. The real challenge for the agile approach would therefore to square its basic principles of iterative collaboration and shared responsibility within the walls raised by organizational structures or functional architectures. For that purpose agile projects should be managed in relation with reusable engineering assets: business (CIMs), functionalities (PIMs), and platforms (PSMs).

With iterations defined like programming loops (see above), the mapping could be done by defining invariants, increments, and exit conditions in relation to the relevant items in the CIMs, PIMs, or PSMs.

Agile Is All About Stories

Depending on devotees or dissenters, the Agile development model is all too often presented as dead-end or end-of-story. A consensual understanding would compare users’ stories to plants that simultaneously grow the roots, trunks, and branches of requirements into their code counterparts.

For that to be achieved, users’ stories have first to be anchored to contexts and purpose:

• Every story is supposed to be told from a specific point of view within the organization. That should be materialized by a single leading role: even if other participants are involved, the narrative should reflect the needs of the primary actor.
• If a story is to provide a one-lane path between past and future business practices, it must focus on a clearly identified activity whose details can be initially overlooked.
• Specific purposes must be set within broader enterprise objectives.

Once introduced with roles (Who) and goals (Why), activities (What) must be circumscribed with regard to actions (How), places (Where) and timing (When). For that purpose one may use Aristotle’s three unities for drama:

1. Unity of action: story units must have one main thread of action introduced at the beginning. Subplots, if any, must return to the main plot after completion.
2. Unity of place: story units must be located into a single space where all activities can be carried out without depending on the outcome of activities performed elsewhere.
3. Unity of time: story units must be governed by a single clock under which all happenings can be organized sequentially.

A story is fully documented (exit condition) when no more details can be added without breaking the three unities rule or affecting its characters (role and goal) or the semantics of features (attributes and operations).

As universally demonstrated across ages and civilizations, there is no limit whatsoever on the nature and complexity of stories told according to Aristotle’s three unities. But problems arise when stories intermingle, which is to happen when stories are set across business domains and supporting systems are taken into account.

Business Stories: Stakeholders’ Plots & Users’ Narratives

As Aristotle noted some time ago, plots are the backbone of any story as they uphold the causal sequence of events and actions: they provide the “why” of what happens, compared to narratives, which tell “how” what happened is being told.

Along that reasoning, business analysts may have to accommodate specific and short-lived opportunities identified by business units with broader and long-standing (aka scaled) objectives defined at corporate level:

• Business objectives (as plots) are meant to apply continuously and consistently to different agents, different concerns, and different contexts. As such they are best defined as rules and constraints (declarative scheme).
• Users’ stories (as narratives) are supposed to translate as soon as possible into business transactions. As such they are best defined as sequences of operations governed by users’ choices (procedural scheme).

And the benefits of the distinction between declarative (for business plots) and procedural (for users’ narratives) blueprints is not limited to business analysis but can be extended to systems architecture (for plots) and software design (for narratives). On that basis declarative schemes could be applied to business functions and architectures capabilities, and procedural ones to users’ stories (or use cases) and software design.

These objectives are more easily achieved with models

Agile & Models

Shared ownership and define/build/accept cycles are arguably at the core of agile development, which translates into actual continuity between users requirements and applications. With users and developers working together to define, build, and test programs, there is no need for models and actual developments can be organized around three kinds of artifacts:

• Features (blue): system functionalities meant to support business objectives. They may also be associated with non functional requirements.
• Stories (green): realization of system features in terms of users’ interactions. Stories usually relate to business objectives. They are used to define, build, and test system features (b).
• Releases (brown): products implementing features (from stories) and components are released at fixed intervals by the Agile Release Train (c).

Yet, as noted above, shared ownership and continuous delivery cannot always be achieved and models may be necessary when architecture assets are to be shared:

• Enterprise architecture may affect requirements through organizational dependencies.
• New applications are often supported by existing functional architectures.
• Development dependencies are defined by constraints on artifacts structure and reuse.

Taking a leaf from the Scaled Agile Framework, agile solutions can be reset within a broader scope by introducing epics at enterprise level. Managed though portfolio, epics are used to describe how business processes are meant to realize business objectives.

That set agile and model driven approaches at different levels: the former for development teams, the latter for development processes, with models managed by epics and referenced by stories.

At enterprise level the benefits of models with agile can be assessed as documents used for governance, communication, or storage.

At project level Use Cases provide a simple and robust modeling scheme.

Bypasses & Roundabouts

Development processes start with requirements and wind up in code; in between there isn’t much of a consensus about how to define the scope (spaces), how to sequence the tasks (paths), and how to time deliveries (paces).

Phased approaches hope for fixed spaces and ordered paths, agile ones explore spaces by increments, but in any case define/build/accept cycles may encroach on all architecture layers: enterprise (business requirements), systems (functional requirements), and platforms (technical requirements).

That put projects at crossroads between market circumstances and business opportunities on one hand, functional architectures shared across domains on the other hand. Such intersections can be dealt with bypasses, traffic lights, or roundabouts:

Bypasses: agile schemes implies shared ownership and continuous delivery so they are the best option for standalone projects.

Traffic lights: phased solutions successively freeze activities and, as a corollary, force business scope and time-frame over system and technical counterparts. While that may avoid collisions between activities, that may also introduce frictions for scope as well as quality:

• With regard to scope, frictions stem from features and schedules set fully and definitively at enterprise level, independently of any feedback from functional and technical layers.
• With regard to quality, frictions stem from tests performed at technical layer when scope and schedules can no longer be revised whatever the feedback.

The consequences are all too easy to observe, with business needs partially satisfied and software quality sacrificed.

Roundabouts: UML’s Use Cases are the best option when users’ stories share business functions supported by functional architectures.

Exploring Problem & Solution Spaces: Agile & Use Cases

Use cases describe what happens between business processes and supporting system. Carried out along agile principles they allow the iterative exploration of business requirements with regard to supporting systems functionalities:

• First, one have to circumscribe the business variants to be taken into consideration. For that purpose the project footprint, introduced as users’ stories, will have to be documented, e.g with activity or business process diagrams (right).
• Then, the project scope will have to mark out the subset of business requirements to be supported by system functionalities. That could be done with use cases describing interactions between system and users (left).

That makes those descriptions both orthogonal and complementary: orthogonal because use cases cut across activity diagrams, complementary because use cases are meaningless without targeted activities. Moreover, they can be associated with different architecture layers and governed by different concerns:

• At business level (processes), the perimeter and granularity of stories must be congruent with the continuity and consistency constraints of business objects and operations.
• At functional level (systems), the span and granularity of use cases must coincide with execution paths. But the rationale governing users interactions is not necessarily congruent with the one governing the integrity of business processes. As a consequence, development paths may pick sequences defined by business processes but should not define them anew based upon interaction constraints.
• Finally, assuming that use cases see systems as black-boxes, their footprint should not depend on decisions taken at technical level.

Those concerns can be dealt with separately if projects scope is explored iteratively, e.g using activity diagrams for business logic and use case diagrams for users interactions. But whatever the context, iterations must be defined by:

• Invariants: conditions on architectural assets supporting the scenario under consideration.
• Increments: features or variants added to scenarii.
• Exit condition: no more features or variants (empty backlog) or time-out.

Applied to architecture layers, invariants provide for reasoned iterations and backlogs:

1. Enterprise layer is the first to be considered: cycles are set according persistency and execution units and bound by domains (identification mechanisms, integrity constraints, and semantics); within cycles, increments target attributes, operations and variants.
2. Functional layer comes second: cycles are set for interaction units (aka use cases), and bound by the continuity and consistency of business objects and activities; increments target transient attributes and operations.
3. Technical layer come last: cycles are set for platforms and bound by functional units.

When problem space (aka scope) overlap different architecture layers Use Cases provide a level of indirection: on one hand they focus on the interactions between users and systems (functional and technical requirements), on the other hand they include the business logic (users’ stories, activity diagrams, or BPM).

Managing Development Paths & Paces: Granularity & Time Boxes

Agile projects carry out specifications, development, and quality assurance into integrated iterations, making room for a progressive exploration of problem spaces and solution paths, and consequently for informed decision-making and better risk management. Yet, at some point, scope and features will have to be committed. Since targeted features cannot be scheduled in advance, they will be delivered by shuttles set along fixed time boxes. Instead of planned schedules, project managers will have to decide on the duration of time boxes and the granularity of deliverables.

Sizing time boxes and deliverables are the two faces of the same coin: granularity can be set alternatively with regard to delivery units or time boxes.

On the lower side of the granularity scale there is a risk of exponential complexity, but on the upper side the development paths may run across layers:

1. Business requirements: alternative paths (business logic) and priorities (enterprise time-frame set by business context and opportunities).
2. Functional requirements: supported features and priorities (systems time-frame set by portfolio management and project planning).
3. Technical requirements: how to implement and deploy functionalities (platforms time-frame set by operational and technical constraints).

Dealing with such situations, especially when decisions are to be delayed and made further along development paths, may create frictions between respective time-frames. Not surprisingly, time-boxes are to provide a solution by introducing a level of indirection between time-frames: being fixed once for all, time-boxes can be used as timers set within different time-frames.

Yet, since the same cannot be assumed for granularity, time-boxes should be small enough to accommodate the size of deliverables within targeted layers. That may call for some balancing on the size of users’ stories: while shorter ones do not necessarily decrease the time to market, they may generate bigger backlogs and exponential overheads in case of dependencies across architecture layers.

Agile between Space & Time

As noted above, only two of the twelve agile principles are specific about targeted projects due to their assumptions with regard to time and space:

Continuous delivery means that software consignments are governed by their intrinsic properties and not by external events. In other words, once their objectives (but not their scope) have been set, projects are supposed to be carried out on their own pace, independently of what happens in business and organizational contexts. That’s a pretty strong assumption that should be explicitly endorsed by stakeholders and users. Compared to phased approaches with fixed requirements and upfront commitments, that would put time under two very different perspectives:

• Time capsules: (agile) projects deal with changing requirements subject to frozen business context and organization.
• Time scales: (phased) projects deal with frozen requirements subject to planned changes in business context and organization.

Direct collaboration means that once projects are started, all decisions regarding business requirements, systems functionalities, and technical implementations can be taken by the development team. Like a counterpart to time continuity that also implies some organizational equivalent that cannot be taken for granted. As with time, the proviso determines a clear divide between development policies:

• Collaborative: problems are solved and decisions taken within some unified organizational space.
• Procedural: problems are dealt with and decisions made separately within different organizational spaces, and communication between them is carried out through prearranged organizational  channels.

Those options clearly depend on organizational and technical environments, hence the benefits of classifying projects with regard to constraints on ownership and delivery:

• Business requirements with distinct stakeholders and objectives directly associated with business value (a).
• Business requirements with objectives and business value involving separate stakeholders (b).
• Functional requirements targeting features used by different applications (c).
• Non functional requirements targeting features used by different applications across different systems (d).

In any case it must be reminded that those constraints are not necessarily set once and for all, and organizations can be adapted to projects or even to development policies set globally.

Managing Deployment: Dynamic Programming

Business driven development doesn’t mean that developments are pushed by requirements but that they are pulled by deployment. Given that business, engineering and operations are governed by different concerns set along different time-frames, some buffering is needed, materialized by the distinction between releases (engineering concerns) and deployment (business and operational concerns).

That makes for multiple dynamics, first for updated backlogs, then for updated deployment targets, and finally for possible feedback through their nexus. That is to be achieved with dynamic programming.

Dynamic programming appears as a primary factor with regard to three core tenets of the agile development model:

• Business driven development doesn’t mean that developments are pushed by requirements but that they are pulled by deployment.
• Just-in-time delivery can only be achieved with the help of a buffer between development and deployment. This buffer should not be confused with an inventory as it has nothing to do with product quantities.
• On the contrary, this buffer, combined with dynamic programming, plays a critical role in the cutback of intermediate documents and models (aka development inventories).

Agile & More

Stemmed from the manifest flaws of Waterfall, the agile development model has established a convincing track record, subject to shared ownership and continuous delivery. But software engineering is not the end of the story and the agile paradigm can be extended to broader enterprise concerns:

• Engineering: how to combine agile with other development models at process and architecture level.
• Business: how to deal with the integration of IT systems with enterprise business environments.

But extending the concept of agility from processes to architectures implies some change in semantics as the former is about activities while the latter is about assets.

Engineering Processes

Whatever their apparel, development models are supposed to bring programming tails to requirements heads, which can be summarily characterized as formal, procedural, or agile.

• Procedural schemes (e.g Waterfall) put the focus on the definition of tasks. They induce significant overheads with the risks of coercing complexities into upfront definitions (artifacts) and one-fits-all jacket (tasks).
• Formal schemes (e.g relational) put the focus on the definition of artifacts. They work best for data, but less so for business logic.
• Agile schemes put the focus on collaboration. They work best for complex problems that can be dealt with without external dependencies.

As it’s safe to assume that large and structured organizations have to keep their options open, that will require:

• Open thinking about practices.
• Robust guidelines for projects’ liaison.
• Non-zero sum approaches to project management.

Engineering Architecture

As repeatedly noted, agile’s benefits are conditioned by shared ownership and continuous delivery, i.e no external dependencies. While that double condition provides a clear incentive for sound and modular project planning, it may not be met for developments set at architecture level. Hence the need of an engineering framework shared by model based and agile software developments.

At first, they can be seen as difficult bedfellows, principally (but not uniquely) because the former insists on the importance of models with some bias toward phased processes, while the latter is all for iterative processes with models mentioned as an afterthought, if at all. Yet both schemes could be made complementary on condition that models could be processed iteratively. And that could be achieved with a declarative approach to models transformation that would anchor agile processes to their organizational and technical environments:

• Downstream automated transformation will ensure just-in-time delivery without undue or redundant intermediate documents and models (aka development inventories).
• Upstream conceptual (or meta-) models will ensure semantic consistency across projects.

More generally combining agile and model based development will generalize the benefits of lean and just-in-time processes to the whole engineering architecture.

Business Agility

Traditional business governance used to be phased: analyze the market, select opportunities, build capabilities, launch operations. But nowadays competitive edge comes from continuous readiness, quicker tempo, and the ability to weave on the fly observed facts with business plans, and that cannot be achieved without real-time integration of business processes with IT systems. Hence the relevancy of the Observation, Orientation, Decision, Action (OODA) decision-making paradigm developed in the sixties by Colonel John Boyd from his experience as fighter pilot and military strategist.

If some similarities between dogfights and business competition may seem metaphorical, one critical aspect is all too real, namely the role played by supporting machines (IT systems or fighter jets) with regard to business agility:

1. Instant understanding of changes in business opportunities (Observation).
2. Simultaneous assessment of the reliability and shelf-life of pertaining information with regard to current positions and operations (Orientation).
3. Weighting of options with regard to enterprise capabilities and broader objectives (Decision).
4. Carrying out of decisions within the relevant time-span (Action).

That understanding of business agility is to be compared with its development and architecture cousins.

Agile Enterprise Architecture

Agility is all about change, and if enterprise governance is not to be thrown aside decision-making has to be supported by a comprehensive and consistent view of assets (architectures) and modus operandi (processes) organized with regard to change.

At enterprise level that is to be achieved through the versatility and plasticity of assets, the former an attribute of function, the latter of form:

• Versatility: enterprise ability to perform different activities in changing environments without having to change its architectures.
• Plasticity: enterprise ability to change its architectures without affecting its performances.

Assets under consideration here can be summarily defined as technical and organizational.

For the former, agility will depend on the ability of the engineering framework to integrate emerging factors with architectures backbone and strategic objectives.

For the latter agility will first depend first on the alignment of systems and organization, and then on the their ability to integrate corporate and social communication channels and leverage personal collaboration and social creativity.

Agility & Entropy

Competition in networked digital markets is carried out at enterprise gates, which puts the OODA loop at the nexus of territories (markets), observations (data), maps for orientation (analytics), business objectives for decisions, and supporting systems for action.

Along that understanding stakes are not limited to immediate business gains but extend to corporate knowledge and enterprise governance; translated into cybernetics parlance, a competitive edge would depend on enterprise ability to export entropy, that is to decrease confusion and disorder inside, and increase it outside.

Assuming a corporate cousin of  Maxwell’s demon with deep learning capabilities standing at the gates in its OODA loop, his job would be to analyze the flows and discover ways to decrease internal complexity (i.e enterprise representations) and increase external one (i.e competitors’ representations).

Further Reading

Principles

• The Scope of Agile Principles
• Thinking about Practices
• Agile vs Waterfall: Right vs Left Brain ?

Users’ Stories

• How to Mind a Tree Story
• From Stories to Models
• Agile Business Analysis: From Wonders to Logic
• Business Stories: Stakeholders’ Plots & Users’ Narratives

Processes

• Iterative Processes
• Agile Falls
• Agile & Models
• Agile between Space & Time
• Agile Delivery & Dynamic Programming
• Spaces, Paths, Paces (Part 1)
• Spaces, Paths, Paces (Part 2)
• Tests in Driving seats

Agile & Beyond

• Projects as non-zero sum games
• Projects Have to Liaise
• Use Cases are Agile Tools
• Models Transformation & Agile Development
• Agile Architectures: Versatility meets Plasticity
• Agile Collaboration & Social Creativity
• Business Agility & the OODA Loop
• Business Agility vs Systems Entropy
• Phased Yet Lean Processes

External Links

• “To Agile, or not to Agile: A Comparison of Software Development Methodologies” Ruslan Shaydulin, Justin Sybrandt