Agile principles to software development are essentially about project and team management, with only sparse recommendations regarding products engineering or processes design. Given this seeming oversight some have even argued that there is no room for models and milestones under the agile roof. Such misconceptions could be unfortunate, especially for project managers having to deal with organizational or architectural dependencies. But omission is not opposition as authors like Robert C. Martin or Dean Leffingwell have demonstrated respectively for products and processes.
With the benefits of hindsight, the debate about agile vs traditional project management has turned into a debate about their respective benefits and drawbacks, and misunderstandings about milestones and models have been progressively ironed out.
Regarding traditional approaches, poorly defined and managed requirements are widely recognized as a major source of projects failures. And that is especially true for waterfall ones with their upfront and rigid approach to scope, changes, and quality:
- Fixed requirements assumption: since there is an inverse relationship between the level of detail and the stability of requirements, defining all requirements upfront is arguably a very hazardous policy.
- Quality as an afterthought: as a detached concern, quality may easily turn into a collateral damage when set along mounting costs and scheduling constraints.
Agile methods, while providing organizational solutions to projects with well circumscribed requirements, do not go further into dealing with the factors that make requirements difficult to manage and acceptance tests difficult to define, namely projects external dependencies, organizational or technical.
For that purpose it is necessary to introduce intermediate outcomes (textual specifications, test cases, models, code, etc.), in order to deal with discontinuities:
- Time discontinuity: when work units cannot be performed in continuity, output/input flows must be managed during intervals.
- Team discontinuity: when work units are to be performed by different teams (whatever the reason, technical or organizational), shared development flows must be managed on their own.
- Semantic discontinuity: when abstractions are to be introduced in order to align specific business concerns (users’ stories) with broader objectives.
Defining and managing those intermediate outcomes, whatever their support, is at the root of development process design and assessment.
Requirements and External Dependencies
Projects open on requirements and close on acceptance. Waterfall ones define full requirements upfront before proceeding with developments; Agile ones start with lean requirements to be refined, consolidated, and tested en-route. While shortened time-span and continuous updates should provide for greater accuracy and stability, they cannot by themselves prevent external factors from affecting requirements. Apart from “internal” dependencies between business processes, system functionalities, and platform implementations, projects may be subject to “external” dependencies across contexts:
- Enterprise architecture may affect requirements through organizational dependencies.
- New applications are often supported by existing functional architectures.
- Development dependencies are defined by constraints on artifacts structure and reuse.
All those concerns, problems, and solutions must be managed on their own, by models or otherwise.
Running without Models
From its origin, the agile approach has been defined in opposition to traditional phased (aka waterfall) methods:
- Agile project planning merge traditional tasks (e.g requirements, analysis, design, coding) into unified work units.
- Instead of sequenced execution of specialized tasks, jack-of-all-trade work units are performed iteratively.
- Whenever possible, textual or graphical descriptions (requirements, specifications, documentation, etc) are replaced by working software components.
- Covenants between customers and providers are replaced by shared ownership and collaboration.
- Features based planning (schedules are fixed for targeted features) is replaced by time-boxing (releases are made of whatever features may have been developed during a fixed time-span).
With users and developers working together to define, build, and test products, there is no need for models and actual developments can be organized around three kinds of artifacts:
- Features: system functionalities meant to support business objectives. They may also be associated with non functional requirements (NFR).
- Releases: products implementing features (from stories) and components (from NFR) are released at fixed intervals by the Agile Release Train (ART).
Yet, this apparent simplicity is not without constraints, and a typical agile project has to fulfill three basic conditions (inspired by Aristotle’s three unities for drama):
- Unity of action: a project should have one main objective that it follows, with secondary threads, if any, set within circumscribed requirements and returning to the main thread after completion. In other words, the project team must constantly know about development dependencies.
- Unity of place: developments should be executed into a space within which all resources can be obtained directly and communication carried out without mediation.
- Unity of time: work units should be set within fixed and continuous time-frames. That is a necessity if innards and synchronization of tasks are to be managed independently of context contingencies.
Set in the broader perspective of enterprise architecture, agile projects can be characterized by:
- Portfolios, where business objectives are translated into consistent strategies (aka “epic” stories), to be eventually implemented by business processes. Business strategies may be associated with non functional requirements and schedules.
- Project teams, with users and developers taking responsibility for components or features, designed from stories or use cases, and implemented by running software (products).
- Releases, driven by fixed time-boxing and quality as controlled variables, scope becoming an adjustment variable managed according portfolio schedules and architecture constraints.
But on that basis, agile approaches may be challenged when:
- Physical constraints bear upon execution of engineering tasks.
- Non functional requirements introduce architectural dependencies between features.
- Business objectives entail cross dependencies different organizational units.
- Decisions are contingent on contexts uncertainties, resources availability, or tasks completion.
Those circumstances may call for intermediate models and milestones.
Dealing with Physical Constraints: Split up requirements
Discontinuities can be directly caused by geographical constraints or indirectly by engineering ones (e.g access to physical resources). The solution in that case should be to split the stories between teams and to consolidate features at program level:
- Assuming a single authority at portfolio level, stories can be allocated to different teams subject to the same architectural constraints.
- Each team defines, builds and tests its products as usual.
- The program manager merges the teams’ outcomes into a single Agile Release Train (ART) according the targeted features and schedules.
In that case the different teams work in parallel on parts of the same development flows and there is no need of intermediate milestones or models.
The same organization can be applied for projects too large to be developed by a single team.
Dealing with Technical Dependencies: Specialized Teams
Because non functional requirements deal with system capabilities independently of specific business applications, they are not readily expressed as users’ stories. Moreover, their realization usually calls for technical skills and expertise focused on systems architecture. For both reasons a distinction is often introduced between (1) features supporting users’ stories, and (2) components, seen as technical assets shared across features.
A simplistic organization would set features and components teams working in parallel, but that would introduce a dependency as features could not be tested independently of non functional components.
As far as technical dependencies between components and features remain local, they can be dealt with through project planning and release management without the need for models. But models will have to be introduced if components are shared by features from different projects and reused in different contexts, in other words if dependencies are architectural.
Dealing with Architecture Dependencies: Models and Processes
Sharing architecture components across projects calls for organizational solutions. One option would be to adopt component-based processes that will split the development of new features between components teams organized along architectural layers: boundaries, control, persistency, and communication. In that case models will have to be introduced in order to manage development assets and support communications between teams.
While such an organization is well adapted to large enterprises with complex architectures, it clearly contradicts a core agile tenet, namely that the whole development processes should be driven by business value.
Alternatively, feature-based processes could allocate the development of components to features teams according their technical skills. That would keep projects solidly anchored to users needs, but would also require a broader specialization of developments teams.
With the collaborative and incremental approach still maintained, models of shared architecture components may not have to be managed; yet their absence will probably raise the risks of “silo” developments.
Hence the benefits of pragmatic solutions in line with the sharing of development assets:
- Persistency and communication layers form the core of shared architecture components and are therefore less prone to change in business needs and short-term adjustments;
- Control layers implement business logic and are meant to support business processes. Some belong to specific domains, others are shared across applications. The former may change according to contexts, concerns, or opportunities, the latter provide for continuity, but both are definitively driven by users needs.
- Boundaries are specific both to business domains and technical platforms, as such they should directly adjust to users needs.
A mixed organization would combine component and feature driven processes, using models to manage shared development assets.
Theoretically, all projects could be run along core agile principles, some driven by users stories, some by system requirements (functional or not), with models used to describe shared architectural assets. Practically that would more difficult for architecture ones if their requirements are not defined at a single source. That would call for a “Disciplined Agile Architecture” as proposed by Scott Ambler
Dealing with Organizational Dependencies: Services
Whereas architecture dependencies are technical and are supposed to be managed by a single authority, organizational ones appear when objectives set at enterprise level call for the collaboration of different units, with their own responsibilities on interdependent components or features. That significantly extends the nature of models by adding a contractual dimension: models are not only needed to support collaborations, they are also required to materialize commitments.
From an agile perspective, such new dimension is critical because it puts a limit to shared ownership and replaces individual responsibilities with organizational (i.e collective) ones. Moreover, contrary to architectural dependencies which can be clearly expressed at components (i.e code) level, organizational ones usually appear well before, and cut across basic agile development cycles. Introducing models to relay such dependencies between projects will clearly contradict core agile principles.
Things would be different if organizational dependencies were matched to architectural ones, and that is precisely what services oriented architectures are meant to achieve:
- Enterprise architects are introduced as a new category of users.
- Objectives addressing business and architectures at enterprise level can be expressed as corporate stories (aka epics) and organized into portfolios.
- From the portfolio perspective business value is much more than the final users value and takes into account the full range of costs and returns.
- Services can be organized into backlogs and then developed and released according standard agile practices.
Along that perspective, there would be two types of models, a white-box one for development artifacts, a black-box one for architectures.
Agile and Model Driven Development
Shared ownership and the define/build/accept cycle are arguably at the core of agile development, which translates into actual continuity between users requirements and code. In other words, even if models are not explicitly excluded, it seems there isn’t much room for them, the reason being the nature of iterative development.
Agile iterations are not just about increments but also about test driven development and integration between requirements and design. In other words the agile cycle is based on a feedback loop between what the system is expected to do and how it may do it. One step further, and architectures may even emerge from iterations. With designs and even architectures being reconsidered all along, there is not much use for models until things are stabilized, and then it’s time for code.
Yet, shared ownership and emerging architectures are not always possible and models may be necessary when organizational units have to share architecture assets. That set agile and model driven approaches at different levels: the former for development teams, the latter for development processes, with models managed by epics and referenced by stories.
- Agile between Space & Time
- The Scope of Agile Principles
- Agile vs Waterfall: Right vs Left Brain ?
- Spaces, Paths, Paces (Part 1)
- Spaces, Paths, Paces (Part 2)
- Tests in Driving seats
- Projects as non-zero sum games
11 thoughts on “Agile & Models”
Iterations control (invariants, increments, exit condition) can be set on sound basis
AA: I have not yet seen one UNACKNOWLEDGED basic model at the heart of Agile: Negative Feedback Control System NFCS with a SET-POINT (goal).
BB: The validity and success of Agile practices (when it works) depends on the validity of SET-POINT, that is, the GOAL for each iterative incremental “Action & Correction based on deviation from SET POINT”.
CC: The set-point itself may change but there must be an OVERALL GOAL l which is stable for the control session or the expected duration of the project or control session. While the coarse or low-resolution Goal is acceptable, its scope or boundaries should not change as the details are added (when resolution of goal is increased)
DD: Although NFCS is very effective when the overall goal is well defined & stable and the system is CONTROLLABLE (as at CC), all systems (projects) do not meet these conditions. I understand that such systems are called emerging or evolving systems for which there is no effective model.
EE: IMO, Agile practices seek to apply the principles of NFCS that too without well-defined overall goal. That is why there is no predictability or repeatability of success of Agile practices. Success depends on the maturity and virtuosity of the Agile team and the nature of the project and circumstances of it.
A well researched comprehensive article on Agile but
The message is that engineers have to decide which tools they should use, with what method. Agile solutions should be the default solution except when shared ownership and continuous delivery cannot be achieved. And models can also be useful within agile development processes.
I’m at a loss as to how an Agile Project Management approach should undermine software engineering principles.
The message that seems to come through these Agile discussions is that programmers are using Agile to avoid good software engineering and architectural practice in the name of producing software quickly.
> Does lean means no models ? (Rémy Fannader)
I would prefer to define first what is the model by its own and I
tend to see that this is a formal symbolic representation how to keep “things” consistent. It does not seem any contradiction between the Model-based approach and Agile, simply because the Agile is a certain model by itself. The Agile process cannot exist without a backbone project Model that represents the actual ideas that we would like to see as workable.
All we normally have is a sliced pie of Models facilitated by some agile process if we even do not write it with the uppercase A. It is all the time around anyway.
“Shared ownership and the define/build/accept cycle are arguably at the core of agile development, which translates into actual continuity between users requirements and code.”
Yes, but there can be several layers between users requirements and code and models provide strong support in that case: they give synthesis on system of interest (operational requirements, functions, physical architecture, software architecture) and ease impact analysis. I have opportunity to be involved in such a situation and models speed both requirement analysis and design decisions.
I would tend to conclude that modelling usage will depend on system/software complexity and Agile approach will just leverage that usage because there is need for quicker responses in order to run the cycle in intended time-box iteration.
I truly agree with “I am concerned that Agile could be used as an excuse to avoid the tasks that programmers hate and I am of a mind to proceed with caution and choose an appropriate approach for each project and not just become totally Agile.”
To me , it is hard to deliver Business Intelligence / datawarehouse solutions without formal modeling as these solutions highly depend on the data models and related business processes. These are not so much SW development heavy but more “design/configuration” related projects though and this may be why I do not favor Agile in these.
In making software solutions that need to deliver benefit to the business immediately I can see the benefit of of not slowing the process by formal modelling. If that software is expected to persist for a long time after the delivery then I can see the benefit of collecting thoughts from yesterday and or reverse-engineering to produce formal documentation. If time permits to document first and develop software afterwards then I believe this is still the best approach to ensuring good communication within the team and with the business to be sure that we are delivering the correct solution. I am concerned that Agile could be used as an excuse to avoid the tasks that programmers hate and I am of a mind to proceed with caution and choose an appropriate approach for each project and not just become totally Agile.
Roger and out
If the organization does value some documentation (in the form of formal models) more than the equivalent amount of working software, they can indeed request the development team to use some of the effort that would go to creating working software to produce models instead.
However, such a decision should take into account the true cost of formal models. Unlike software, there will generally be no automated testing to identify when they are no longer correct. So, the maintenance cost of models are much higher than software developed in the Agile way. In practice, what this usually means is that over time the value of the formal models become negative as decision made based on the validity of the models become increasing poor.
What I would say is that there is little need in Agile for *persistent* models.
That does not mean that that there is little room for modeling as an activity for communication and collaboration. The models that facilitate agile modeling are neither formal nor persistent.
There is lots of room for modeling, just not much room for formal models. One could similarly say there is lots of room for planning in agile software development, just not much room for formal plans.