Preamble
Paraphrasing Einstein, the only reason for processes [time] is so that everything doesn’t happen at once. As a corollary, processes are meant to introduce distinctions between outcomes and activities.
Software & Systems Engineering
The objective of systems engineering is to develop software components and arrange them into systems architectures.
As far as standalone applications are concerned, there is no point to distinguish between system architecture and software design, and projects can take full responsibility for products.
That’s not the case when projects have to take into account external dependencies, business, organizational or technical. How these dependencies are taken into account is what processes are about.
Depending on organizational contexts defined by agents and responsibilities on one hand, technical contexts set by technology, resources and legacy systems on the other hand, the objective of processes is to rationalize the development of software components.
Requirements
As noted before, the outcome of engineering processes must meet different kinds of requirements: business goals, system functionalities, quality of service, and platform implementations. In a perfect Model Driven Engineering world there would be one stakeholder, one architecture, and one time-scale.
Unfortunately, those requirements are usually set by different stakeholders, governed by different rationales, and subject to changes along different time-frames. With regard to governance, the objective of engineering processes is therefore to manage those goals along time and arbitrage between overlapping or even conflicting agendas.
The reason for Processes: The Cooking Metaphor
Engineering processes must be considered when tasks cannot be performed simultaneously and have to be divided and sequenced. That’s the case when, due to contexts or dependencies, developments cannot be carried out by atomic activities but call for more complex undertakings: paraphrasing Einstein, the only reason for processes [time] is so that everything doesn’t happen at once.
The rationale of development processes may be illustrated by the cooking metaphor, with development artifacts equated to cooking ingredients. While there is an infinity of ingredients, most belong to four categories: water, proteins, carbohydrates, and fats. The ways they react to physical processing and can be combined are set by chemical laws. It is therefore possible to characterize cooking recipes and identify basic states and processing sequences independently of their specifics.
Taking as examples a few egg-based recipes:
- Poached Eggs (atomic, single flow): a single operation (put eggs into boiling water) whose outcome must be synchronized with users needs
- Omelette (composition, single flow): two sequenced (not necessarily synchronized) operations (beat eggs, fry) performed on the same contents (not necessarily by the same operator), under a single clock, with all resources at hand.
- Eggs Benedict (aggregate, joint flows) : joint and synchronized operations, executed in parallel by different operators, under a single clock, with all resources at hand.
- French Crepes (aggregate, single flow): two phases (making the crepes, adding savories), the first one made of sequenced operations, executed successively by different operators, possibly under different clocks, within different working spaces.
- Crepes Suzette (restaurant): the same as above but necessarily executed within different spatial and organizational contexts: kitchen for the making and in front of the customer for the “flambé”.
Recipes 1 and 2 don’t call for engineering processes or intermediate outcomes. Recipe 3 must take into account intrinsic constraints on flows, as described by a process; intermediate outcomes don’t have to be managed. Recipes 4 and 5 should follow predefined processes describing tasks, flows, intermediate outcomes and associated constraints.
As illustrated by these examples, discontinuity can be intrinsic (nature of products) or extrinsic (nature of participants): for simple processes (e.g poached eggs), all products are delivered to customers, for structured ones products will also include intermediate outcomes to be instantly used (e.g eggs Benedict) or kept aside for a time (e.g crepes).
Models & Processes
In contrast to cooking which is all about physical ingredients, software engineering is all about symbolic ones, which is why models are the epitome of intermediates products. Yet, as illustrated by model driven engineering the same principles apply: reasoned processing of models depending on the fundamental properties of their contents: computation independent, platform independent, platform specific.
With regard to software development artifacts, and whatever their form (textual specifications, test cases, models, code, etc.), intermediate products are introduced to support discontinuity in time or teams:
- Time discontinuity is set by the nature of development flows: when sequenced tasks can be performed asynchronously, output/input flows should be managed independently of associated producing and consuming tasks.
- Team discontinuity is set by organizational context: when sequenced tasks are to be performed by different teams (whatever the reason, technical or organizational), shared flows should be managed independently of the organizational units involved.
Obviously, time and team discontinuities are not at the same level, the former being governed by products characteristics and the latter by projects organization.
Products and Development Flows
Products involved in software engineering processes can be mapped to architectures set by the Actual/Symbolic and Persistent/Transient categories:
- Enterprise architecture deals with organizational and business contexts: locations, agents and organization, business objects and activities, events.
- Functional architecture describe systems functionalities with regard to business processes: business domains and functions, symbolic representation of persistent objects.
- Application architecture describe software with regard to business logic, collaboration, and users interfaces.
- Technical architecture: describe systems, processes, and components with regard to applications.
Taking into account those discontinuities, developments can be set along two axes:
- Operations/Applications: self-contained developments target applications that can be defined, built, and accepted under a single authority.
- Functional/Technical: shared developments target architecture components whose specifications, integration or acceptance cannot be set under a single authority.
As it happens, that distinction can be used to characterize projects according to their footprint (enterprise or system) and purpose (assets or users’ value):
- Purpose: management of shared assets (architects) vs specific requirements (analysts).
- Target: enterprise vs information systems.
Applications development can be directly driven by requirements (use cases, users’ stories, …) when these can be fully set under a shared ownership, otherwise cross dependencies will have to be consolidated by the functional architecture and managed through models.
Projects and Development Models
- Bureaucratic (aka waterfall) processes follow predefined sequences of tasks targeting standard outcomes and deliverables. Tasks and deliverables can be customized but adjustments don’t affect contents. Due to the one-fits-all philosophy such processes lack built-in mechanisms; as a consequence quality management must be performed from the outside and the effectiveness of the whole depends heavily on support and expertise. Their main advantage is with integrated projects involving different teams across organizations, domains and architectures.
- Democratic (aka agile) processes take the opposite view and let project organize work units and deliverables as befit the objectives and contexts. Milestones are set aside by direct collaboration between business and system analysts, and scheduling is replaced with iterative modelling and time-boxing. Whereas such processes still lack built-in mechanisms, quality management is performed from within, based upon responsibility and communication. Overheads are limited but expertise and best practices are essential. Agile approaches may bring substantial benefits to complex projects with well-defined perimeter and limited external dependencies.
- Automated processes (as promoted by model based approaches) are the holy grail of software engineering. Processes are organized around predefined models subject to transformation. Providing comprehensive and consistent semantics, built-in mechanisms can be used all along the processes, especially for quality management. Expertise can be concentrated on models and rules, with best practices taking a back seat.
Each approach addresses some part of the problem, and combining them within a unifying paradigm should bring cross benefits: phased solutions with milestones for integration across business units and architectures, agile when conditions on shared ownership and continuous delivery can be met, automated transformations when analysis and design decisions are set.
Procedural vs Declarative Frameworks
As notoriously illustrated by Waterfall, phased approaches are often understood as procedural, if not bureaucratic. But that’s not a necessary consequence: when development dependencies are defined on artifacts declarative scheme can be used with phased development. Since iterative development models are by nature declarative, that would bring agile and phased development models under a common conceptual roof. That could be achieved with a consolidated understanding of work units:
- Activity-based work units (waterfall): work breakdown starts with one-fits-all organization, follows top-down with predefined activities, which are then entrusted with production objectives
- Team-based work units (agile): no breakdown but iterations managed collectively.
- Flow-based work units (Model based): work breakdown starts with production steps and builds processes bottom-up, depending upon constraints.
Manufacturing processes, built from material flows and engineering constraints, have demonstrated the efficiency of flow-based approaches. Despite advances made by model based engineering frameworks, works units defined on the status of artifacts are generally limited to automated code generation from models. Upstream work units, when explicitly defined, remain customized versions of the same catch-all activities. In order to generalize the domain specific schemes used for code generation, their repository of engineering artifacts would have to be extended to models. That could be done with conceptual thesauruses.
On a broader perspective, the lack of principled associations between products and work units thwart attempts to map problems (what is to be developed) to solutions (how it should be done). That’s what processes are about.
Processes and Development Patterns
Just like software products are solutions to application problems, development processes are solutions to development problems, namely how to deal with dependencies between work units.
On that basis, standard work units can be associated to development flows themselves derived from development problems. Road-maps can then be organized along architecture tiers and model layers:
- Architecture tiers deal with functional dependencies: standalone (no dependency), shared and transient, shared and persistent, communications.
- Model layers take into account stakeholders and concerns along application life-cycle: business, functionalities, components.
When engineering is driven by architectures, processes can be devised depending on enterprise concerns and engineering contexts. While that could come with various terminologies, the partitioning principles will remain unchanged, e.g:
- Taking advantage of the absence of dependencies, agile processes will combine requirements with development and bypass analysis phases (a).
- Projects meant to be implemented by Commercial-Off-The-Shelf Software (COTS) will start with business requirements, possibly using BPM, then carry on directly to platform implementation, bypassing system analysis and design phases (b).
- Changes in enterprise architecture capabilities will be rooted in analysis of enterprise objectives, possibly but not necessarily with inputs from business and operational requirements, continue with analysis and design of systems functionalities, and implement the corresponding resources at platform level (c).
- Projects dealing with operational concerns will be conducted directly through systems design of and platform implementation (d).
Whereas selecting processes best fitted to problems is arguably the right policy, it comes with two basic requisites: open minded teams and a conceptual repository of engineering artifacts.