continuous delivery – Caminao's Ways

Synopsis

Business driven development is arguably the cornerstone of the agile development model. On one side it means business value set by users and stakeholders, on the other side it entails continuous and just-in-time delivery; what happens in-between is set by backlogs (for development) and product increments (for delivery).

A sanguine understanding of continuous releases may assume that planning is no longer relevant, and that deployment can be carried out “on-the-fly”. But that would assume that stakeholders and product owners are ready to put aside roadmaps, overlook milestones, and more generally forget that time is money. That would go against a basic objective of agile, namely that developments must be driven by business needs, and products delivered just-in-time for best value. Given the well established track record of dynamic programming for manufacturing processes, could the technique be usefully applied to agile engineering processes ?

Delivery, Deployment, & Continuity

Continuity doesn’t mean synchronization: business, engineering, and operations are governed by different concerns set along different time-frames. Some buffering is needed, materialized by the distinction between releases (engineering concerns) and deployment (business and operational concerns).

A time for every purpose: Epics & business roadmap (a), associated backlogs of users’ stories (b), released features (c), architectural capabilities (d), deployed components (e).

Such distinctions introduce both overlapping (with business time-frame) and discontinuity (between development and deployment):

Product roadmaps are set in business time-frames and determine development and deployment time-frames.
Development time is set by product roadmaps and runs clockwise from project inception to software releases (a>b>c).
Deployment time is also set relative to product roadmaps but it runs counterclockwise, from product deployment as planned by business back to released software components (c<e).

Development and deployments runs can be compared to crews tunneling through a mountain from both sides; where and when they meet leaves room to adjustments. Yet more is at stake in the meeting between development completion and deployment inception. Apart from time, adjustments may also bear on formats and contents; and given the specificity of development and deployment purposes, their adjustment may also be seen as the morphing of continuous software releases (project perspective) into discrete increments (product perspective).

Dynamic Programming

Dynamic programming (aka multistage programming) is a problem solving method that combines two principles:

Divide & conquer is a general purpose strategy that deals with the intrinsic complexity of problems by breaking them down into collections of simpler sub-problems to be solved separately depending on sequencing constraints.
Recursion deals with the lack of complete or accurate information upfront, solving the problem in stages rather than as one entity. Each stage is optimized separately on the basis of the current state reflecting decisions taken at previous stages, the optimality principle guaranteeing the optimality of the final outcome.

That incremental and iterative approach clearly befits the tenets of the agile development model.

Twofold Planning

As noted above, and whatever the technique, agile processes entwine two phases, one for development, the other for deployment, each with its own planning:

The development planning, epitomized by backlog management, deals with the definition of work units and their sequencing.
The deployment planning deals with the merging of software released into product and their incremental deployment.

Project work units are sequenced (backlog), Product increments are merged, and both are dynamically adjusted around their nexus.

That makes for multiple dynamics, first for updated backlogs, then for updated deployment targets, and finally for possible feedback through their nexus. That can only be achieved with dynamic programming.

Backlog: Multistage sequencing

Backlogs are used to manage work units targeting self-contained requirements items; they can be represented by graphs with nodes standing for work units and arcs weighted by constraints.

Basically, the problem is to optimize the development of a given set of items given users’ priorities, technical constraints, and resources availability. When all the information is available upfront, optimum solutions can be obtained with simple Shortest Path algorithms. Yet, given the iterative and exploratory nature of agile processes, backlogs are meant to be updated as the project advances, taking advantage of improved knowledge:

Users may introduce new items, remove ones, or change their priorities due to a better understanding of requirements space.
Engineers may also introduce new items (e.g for technical debt) or reconsider technical difficulties and dependencies due to a better understanding of solutions space.

Dynamic reordering of backlogs: looking forward for the optimum path to completion.

Dynamic programming is introduced in order to support step-wise decisions optimizing the whole process:

Backlog states (t1 & t2) are defined by the remaining work units, rankings, and feasibility constraints.
Each stage redefines the optimum path to completion taking into account the current state and updated information. Recursive computation being based on the summary information etched in states, it ensues that all future decisions can be selected optimally without recourse to information regarding previously made decisions.

Given a set of feasible paths (as defined by technical dependencies and time), the aim at each stage is to select the optimum path for the remaining units based on current state. Optimization functions will typically consider users’ value, learning curve and associated risks, and resources availability and costs.

As illustrated below, nodes can represent grouped items, e.g when several projects have to share resources or releases are to be regrouped.

Deployment: Dynamic merging

Given a set of released software components, the aim of deployment planning is to decide which increment to add to deployed product. Assuming that technical concerns have already been dealt with by releases, the objective at each stage is select the items maximizing the ROI of the deployed product. It must be noted that, contrary to the development algorithm looking forward for the optimization, the deployment algorithm select the optimum path by looking backward at the ROI of deployed products.

Dynamic reordering of deployments: looking backward for the optimum path to completion.

But the backward impact of deployment optimization can go further and affect backlogs.

Feedback

Shared ownership and continuous delivery are two main pillars of the agile development model, the former giving the development full authority and responsibility, the latter ensuring the users have a firm hand on the helm. Yet, as already noted, development, deployment, and business are governed by different time-frames, which could induce some frictions, e.g if business units were forced to synchronize products deployments with software releases. While severe disruptions can be avoided if releases and deployments are managed separately, development teams cannot be completely sheltered from changes in business or operational priorities. That is where the dynamic reassessment of optimum paths is to help: assuming a change in deployment planning (nq instead of op), the new priorities are fed back into development (aka backlog) rankings, and the optimum path is updated.

It must be noted that such feedback only affects ranks and must leaves contents unchanged.

Conclusion: Business driven, Just-in-time delivery, & Lean Inventories

Dynamic programming appears as a primary factor with regard to three core tenets of the agile development model:

Business driven development doesn’t mean that developments are pushed by requirements but that they are pulled by deployment.
Just-in-time delivery can only be achieved with the help of a buffer between development and deployment. This buffer should not be confused with an inventory as it has nothing to do with product quantities.
On the contrary, this buffer, combined with dynamic programming, plays a critical role in the cutback of intermediate documents and models (aka development inventories).

External Links

Continuity of delivery and locus

While the scope of agile principles extends far beyond the eponymous methods, some of them are more specific and their applicability contingent on contexts and objectives; two of them are especially important as they entail very specific assumptions with regard to time and space:

Continuous delivery of valuable software.
Direct collaboration between software users and developers all along the development process.

Material inconsistency (H. Zamora) — Continuity and collaboration (H. Zamora)

Time Capsules vs Time Scales

If time is seen as a discrete gauge introduced to sequence events, it ensues that continuity indicates that no external events are to be taken into account. In other words, once its objectives have been set, software development must be carried out according its own pace, whatever happens in business context and enterprise organization. That’s a pretty strong assumption that should be explicitly endorsed by stakeholders and users.

Interestingly, this assumption can be set against fixed requirements and upfront design associated with waterfall approaches, as if development policies were to be set between two alternative options:

Time capsules: projects deal with changing requirements subject to frozen business context and organization.
Time scales: projects deal with frozen requirements subject to planned changes in business context and organization.

A pragmatic approach should take the best of each option: put limited and self-contained objectives into capsules and organize capsules along scales.

Collaborative vs Procedural Spaces

The reason for processes is that tasks cannot be carried out simultaneously, and the reason for human contribution is that some tasks involve decision-making based on circumstances and provisos that cannot be determined upfront.

Those decisions may concern different domains of concerns with their respective objectives and roles: business requirements, systems functionalities, or technical implementations. Agile and phased approaches clearly disagree about how those decisions are to be taken and conveyed along processes and across organizational spaces, the former ruling that all the parties should deal with them jointly, the latter opting for a separation of concerns. As a counterpart to time continuity, agile recommendation implies some continuity across organizational spaces that cannot be taken for granted. Whether that proviso can be met will determine the choice of a development policy:

Collaborative: problems are solved and decisions taken through collaboration between parties within a single organizational space.
Procedural: parties deal separately with problems and decision-making within their respective organizational spaces, and communication between those spaces is carried out through prearranged channels.

Those options clearly depend on organizational and technical environments, hence the benefits of charting projects with regard to constraints on ownership and delivery.

Charting projects: Ownership and Delivery

Given that choosing the right development model is a primary factor of success, decision-making should rely on simple and robust criteria. First, if decisions are to be taken and implemented collectively, shared governance has to be secured. Second, if deliveries are to be carried out continuously, projects shouldn’t be dependent on external events.

When charted with regard to ownership and delivery, projects can be regrouped into four basic categories:

Business requirements with distinct stakeholders and objectives directly associated with business value (a).
Business requirements with objectives and business value involving separate stakeholders (b).
Functional requirements targeting features used by different applications (c).
Non functional requirements targeting features used by different applications across different systems (d).

vvv — Charting projects with regard to ownership and delivery constraints

That straightforward taxonomy provide clear guidelines for project managers: options (a) and (d) associated respectively with agile and phased approaches, and options (b) and (c) to be decided with regard to actual projects, the ability of organizations to accomodate particular development schemes, and long term objectives regarding engineering processes.

Tag: continuous delivery

Agile Delivery & Dynamic Programming