Category: Software Engineering

Models Transformation & Agile Development

Models transformation is generally recognized as the basic mechanism of model based systems engineering (MBSE). Yet, the actual scope of transformations is somewhat limited to design-to-code, and its sequential bias puts MBSE at odds with agile development approaches. Could a revisited understanding help to figure out this apparent discrepancy ?

andreaBurger
Iterative Transformations (Andrea Burger)

Transformation Issues

Traditional transformation paradigm involves ordered sequences of models obtained by applying rules to their immediate predecessor(s). That organizational scheme has three critical consequences, for applicability, economics of reuse, and development processes.

  • Applicability: the effectiveness of transformations is conditioned by (a) an executable language for the description of targets, and (b) a closed and compact set of unambiguous patterns. Those conditions can only be satisfied for the downstream part of the development process.
  • Reuse: given the sequencing constraints, models are to be managed and reused along tree-like structures with duplicates introduced at branching points.
  • Development processes: sequenced models brings forth phased options and leaves out agile solutions.

Assuming those issues are not conclusive, they may be overcame by revisiting the nature of transformations.

Transformation vs Inheritance & Composition

Most of the proposed taxonomies (see references below) put the focus on languages and mechanisms (e.g rules) of sequential transformation without paying enough attention to the nature and the semantics of models contents. Even when abstraction levels are taken into account, the respective contents of each level remain undefined. As it happens, that issue may be the key to a better understanding of models transformation.

To begin with, rule-based transformation has to be compared to inheritance and composition:

  • Structural inheritance can be used to refine models as to take into account business scenarii previously ignored; e.g special conditions for good customers.
  • Functional inheritance can be used to introduce new capabilities; e.g new authentication procedures.
  • Functional composition can be used to apply capabilities across different scenarii; e.g customized authentication procedures.
  • Rules-based solutions can be used by any kind of transformation.
vvvv
A broader understanding of models transformation should include inheritance and composition

That taxonomy implies a clear distinction between operations executed within the same level of abstraction and those targeting artifacts defined at different levels: contrary to rules-based transformations, inheritance and composition can only be applied to artifacts sharing common semantics.

heterogeneous Models

While that would clearly prevent their use for models organized along abstraction levels, semantic pitfalls could be mastered for models built from artifacts from different abstraction levels.

Releasing models from (still to be defined) abstraction levels would bring two critical benefits:

  • Whatever the terminology (abstract, conceptual, functional, etc.), abstraction semantics are much easier to define for artifacts than for models.
  • That would remove a chunk of restrictions on the design of transformation processes.
Heterogeneous models are not bound to abstraction layers.
Releasing models from abstraction layers.

In that case transformation rules could be turned into combination ones and sequential transformation turned into cross-breeding.

Mendel, Models, Mongrels

Taking a cue from Gregor Mendel’s use of cross-fertilization, the aim of a revisited transformation paradigm would be three-fold:

  1. To refine the granularity of reuse, from models to artifacts
  2. To substitute combination for sequential transformation whenever possible.
  3. To substitute graphs for trees, with models organized along two basic layers, final (aka mongrels) or reusable (aka blueprints).
vvv
Models combination (top) replaces transformation phases (bottom) by a distinction between blueprints (full line) and mongrels (dashed line).

As far as MBSE is concerned, the genetics metaphor helps to clarify the nature of abstraction. Conceptually, it introduces a distinction between artifacts and models:

  • With regard to artifacts, abstraction layers are defined by scope: enterprise, systems, platforms.
  • With regard to models, abstraction layers are defined by capabilities: reusable (stable traits), or final (recessive traits).

That taxonomy is corroborated by its functional counterpart: artifacts transformation is carried out with inheritance and composition, models transformation relies on combination.

More important, that understanding goes a long way solving the issues regarding scope, reuse, and development processes.

Scope: Weaving Analysis & Design Traits

Definitions and taxonomies should always be assessed with regard of their applicability. On that account there isn’t much to say for abstraction layers applied to models: they don’t fit because too many traits can be defined across different layers, e.g: business rules, authentication, encryption, etc.

That difficulty can be neatly and consistently removed by models built from artifacts defined at different levels.

Models Reuse: Blueprints vs Mongrels

Reuse is all too often seen as a contentious objective with inconclusive ROI. One one hand it requires significant overheads to manage the resources, on the other hand the outcomes can introduce regressive traits. The distinction between sound reusable models and final ones significantly reduces both the costs of the former and the risks of the latter.

Processes Organization: MBSE & Agile

Model based systems engineering and the agile development model are arguably two of the most conclusive approaches to software engineering. Unfortunately they are often seen as difficult bedfellows, principally (but not uniquely) because the former insists on the importance of models with some bias toward phased processes, while the latter is all for iterative processes with models mentioned as an afterthought, if at all. Yet, both approaches could be made complementary on condition that models could be processed iteratively. And that could be achieved if sequenced transformations of homogeneous models would be replaced by the combination of heterogeneous ones.

vvv
Iterative development mixing new business requirements with existing functionalities (a) and business rules (b).

Within such a framework an agile team could, e.g, iteratively develop new business requirements, taking into account existing functionalities (a) and business rules (b), and generate code (c).

Further readings

External Links

 

Conceptual Models & Abstraction Scales

Following the recent publication of a new standard for conceptual modeling of automation systems (Object-Process Methodology (ISO/PAS 19450:2015) it may be interesting to explore how it relates to abstraction and meta-models.

oskar-schlemmer-at-bahaus
Meta-models are drawn along lean abstraction scales (Oskar Schlemmer )

Models & Meta Models

Just like models are meant to describe sets of actual instances, meta-models are meant to do the same for sets of modeling artifacts independently of their targets. Along that reasoning, conceptual modeling of automation systems could be achieved either with a single language covering all aspects, or with a meta-language dealing with different sets of models, e.g MDA’s computation independent, platform independent, and platform specific models.

Modeling Languages covering technical, functional, and business concerns.
Two alternative options for the modeling of automation systems: unified language, or a meta language covering technical (e.g PSMs), functional (e.g PIMs), and business (e.g CIMs) scopes.

Given a model based engineering framework (e.g MDA), meta-models are generally used to support downstream models transformation targeting designs and code. But when upstream conceptual models are concerned, the challenge is to tackle the knowledge-to-systems transition. For that purpose some shared modeling roof is required for the definition of the symbolic footprint of the targeted business in the automation system under consideration.

Symbolic Footprint

Given that automation systems are meant to manage symbolic objects (aka surrogates), one should expect the distinction between actual instances and their symbolic representations to be the cornerstone of corresponding modeling languages. Along that reasoning, modeling of automation systems should start with the symbolic representation of actual business footprints, namely: the sets of objects, events, and processes, the roles played by agents (aka active objects), and the description of the associated states and rules. Containers would be added for the management of collections.

Automation systems modeling begins with the symbolic representation of actual instances
Automation systems modeling begins with the symbolic representation by systems of actual instances of business related objects and phenomena.

Next, as illustrated by the Object/Agent hierarchy, business worlds are not flat but built from sundry structures and facets to be represented by multiple levels of descriptions. That’s where abstractions are to be introduced.

Abstraction & Variants

The purpose of abstractions is to manage variants, and as such they can be used in two ways:

  • For partial descriptions of actual instances depending on targeted features. That can be achieved using composition (for structural variants) and partitions (for functional ones).
  • As hierarchies of symbolic descriptions (aka types and sub-types) subsuming variants identified at instances level.

On that basis the challenge is to find the level of detail (targeted actual instances) and abstraction (symbolic footprint) that will best describe supporting systems functionalities. Such level will have to meet two conditions:

  1. A minimal number of comprehensive and exclusive categories covering the structural variants of the sets of instances to be uniformly, consistently, and continuously identified by both enterprise and supporting systems.
  2. A consistent but adjustable set of types and sub-types anchored to the core structural categories and covering the functional variants .

Climbing up and down abstraction ladders looking for right levels is arguably the critical part of conceptual modeling, but the search will greatly benefit from the distinction between models and meta-models. Assuming meta-models are meant to ignore domain specific features altogether, they introduce a qualitative gap on abstraction scales as the respective hierarchies of models and meta-models are targeting different kind of instances. The modeling of agents and roles epitomizes the benefits of that distinction.

Abstraction & Meta Models

Taking customers for example, a naive approach would use Customer as a modeling type inheriting from a super-type, e.g Party. But then, if parties are to be uniformly identified (#), that would preclude any agent for playing multiple roles, e.g customer and supplier.

A separate description of parties and roles would clearly be a better option as it would unify the identification of the former without introducing unwarranted constraints on the latter which would then be defined and identified as the realization of a relationship played by a party.

Not surprisingly, that distinction would also be congruent with the one between models and meta-model:

  • Meta-models will describe generic aspects independently of domain-specific considerations, in particular organizational context (units and roles) and interactions with systems (a).
  • Models will define Staff, Supplier and Customer according to the semantics of the business considered (b).
Composition, partitions and specialization can be used to detail the symbolic footprint
Composition, partitions and specialization can be used along two different abstraction scales.

That distinction between abstraction scales can also be applied to the conceptual modeling of automation systems.

Abstraction Scales & Conceptual Models

To begin with definitions, conceptual representations could be used for all mental constructs, whereas symbolic representations would be used only for the subset earmarked for communication purposes. That would mean that, contrary to conceptual representations that can be detached of business and enterprise practicalities, symbolic representations are necessarily built on design, and should be assessed accordingly. In our case the aim of such representations would be to describe the exchanges between business processes and supporting systems.

That understanding neatly fits the conceptual modeling of automation systems whose purpose would be to consolidate generic and business specific abstraction scales, the former for symbolic representations of the exchanges between business and systems, the latter symbolic representation of business contents.

At this point it must be noted that the scales are not necessarily aligned in continuity (with meta-models’ being higher and models’ being lower) as their respective ontologies may overlap (Organizational Entity and Party) or cross (Function and Role).

Toward an Ontological Framework for Enterprise Architectures Modeling

Along an analytic perspective, ontologies are meant to determine the categories that can comprehensively and consistently denote the instances of a domain under consideration; applied to enterprise concerns that would entail:

  • Thesaurus: for the whole range of terms and concepts.
  • Documents: for documents with regard to topics.
  • Business: for enterprise organization and business objects and activities.
  • Engineering: for systems surrogates associated to enterprise organization and business objects and activities
Ontologies provide a common conceptual framework for  models and meta-models

That would open the door to a seamless integration of business intelligence, systems engineering, knowledge management, and decision-making.

Further Reading

External Links

Agile Business Analysis: From Wonders to Logic

Time and again new recruits will ask about the role of business analysts. Considering that such a question is seldom heard from software engineers, are BAs more curious about their job, or are they standing on more tentative grounds ? If that’s the case agility would help them to flip-flop between business quicksands to systems hard rocks.

vvv
How to make sense of business wonders (Hieronymus Bosch)

Holding the fort vs scouting outskirts

Systems architects and software engineers may have to meet esoteric business requirements, but their responsibility is first and foremost to guarantee the functional and economic sustainability of systems. On that account they are given licence to build solid walls and secure gateways, and to enforce their own languages and rules upon well vetted parties.

Business analysts don’t get such a free hand: while being straitened by software engineers constructs and constraints, their primary undertaking is to explore business wilds, reconnoitre competitors, trace new tracks, and learn the dialects of any nicknamed natives ready to trade.

No wonder the qualms of new business analysts.

Great businesses make their own rules

The best rules in business are the ones still unbeknownst, as success is most often brought by disruptive initiatives taking advantage of previously undiscovered opportunities. It ensues that at its core, BAs’ job description is to relentlessly look across the frontier for still uncharted businesses, and bring them back to the digitized world of shipshape business domains and processes.

For that purpose BAs will have to juggle with the fuzzy idiosyncrasies of new business openings until they can be aligned with the functionalities of “legacy” systems.

BA’s Agility

While usually presented as a software engineering hallmark, agility may be equally useful for business analysts as they have to balance two crossing perspectives:

  • Analysis: sorting detailed activities into business processes.
  • Synthesis: factoring out business functions and mapping them to systems capabilities.

That could be a challenging achievement if carried out sequentially: crossing back and forth between changing scope and steady capabilities could generate unsettling alternatives and unbounded complexity.

The agile development model is meant to tackle the difficulties through iterations and collaboration without being too specific about the kind of agility required from business analysts and software engineers.

Yet the apparent symmetry between the parties may be misleading: whereas software engineers don’t have (and shouldn’t even try) to second guess business analysts, business analysts shouldn’t forget that at the end of the day business expectations, however exotic or esoteric, will have to feed very conformist logical beasts.

Further Readings

Agile Collaboration & Enterprise Creativity

Open-plan offices and social networks are often seen as significant factors of collaboration and innovation, breeding and nurturing the creativity of knowledge workers, weaving their ideas into webs of truths, and molding their minds into some collective intelligence.


Trust & Communication (Juan Munoz)

Yet, as creativity comes with agility, knowledge workflows should give brains enough breathing space lest they get more pressure than pasture.

Collaboration & Thinking Flows

Collaboration is a means to an end. To be of any use exchanges have to be fed with renewed ideas and assumptions, triggering arguments and adjustments, and opening new perspectives. If not they may burn themselves out with hollow considerations blurring clues and expectations, clogging the channels, and finally stemming the thinking flows.

Taking example from lean manufacturing, the first objective should be to streamline knowledge workflows as to eliminate swirling pools of squabbles, drain stagnant puddles of stale thoughts, and gear collaboration to flowing knowledge streams. As illustrated by flood irrigation, the first step is to identify basin levels.

Dunbar Numbers & Collaboration Basins

Studying the grooming habits of social primates, psychologist Robin Dunbar came to the conclusion that the size of social circles that individuals of a living species can maintain is set by the size of brain’s neocortex. Further studies have confirmed Dunbar’s findings, with the corresponding sizes for humans set around 10 for trusted personal groups and 150 for untried social ones. As it happens, and not by chance, those numbers seem to coincide with actual observations: the former for personal and direct collaboration, the latter for social and mediated collaboration.

Based on that understanding, the objective would be to organize knowledge workflows across two primary basins:

  • On-site and face-to-face collaboration with trusted co-workers. Corresponding interactions would be driven by personal dispositions and attitudes.
  • On-line and networked collaboration with workers, trusted or otherwise. Corresponding interactions would be based on shared interests and past exchanges.

Knowledge Workflows

The aim of knowledge workflows is to process data into information and put it to use. That is to be achieved by combining different kinds of tasks, in particular:

  • Data and information management: build the symbolic descriptions of contexts, concerns, and means.
  • Objectives management: based on a set of symbolic descriptions, identify and refine opportunities together with the ways to realize them.
  • Tasks management: allocate rights and responsibilities across organizations and collaboration frames, public and shallow or personal and deep.
  • Flows management: monitor and manage actual flows, publish arguments and propositions, consolidate decisions, …

Taking into account constraints and dependencies between the tasks, the aims would be to balance creativity and automation while eliminating superfluous intermediate products (like documents or models) or activities (e.g unfocused meetings).

With regard to dependencies, KM tasks are often intertwined and cannot be carried out sequentially; moreover, as illustrated by the impact of “creative accounting” on accounted activities, their overlapping is not frozen but subject to feedback, changes and adjustments.

With regard to automation, three groups are to be considered: the first requires only raw processing power and can be fully automated; the second also involves some intelligence that may be provided by smart systems; and the third calls for decision-making that can only be done by human agents entitled by the organization.

At first sight some lessons could be drawn from lean manufacturing, yet, since knowledge processes are not subject to hardware constraints, agile approaches should provide a more informative reference.

Iterative Knowledge Processing

A simple preliminary step is to check the applicability of agile principles by replacing “software” by “knowledge”. Assuming that ground is secured, the core undertaking is to consider what would become of cycles and iterations when applied to knowledge processing:

  • Cycle invariants: tasks would be iterated on given sets of symbolic descriptions applied to the state of affairs (contexts, concerns, and means).
  • Iterations content: based on those descriptions data would be processed into information, changes would be monitored, and possibilities explored.
  • Exit condition: cycles would complete with decisions committing changes in the state of affairs that would also entail adjustments or changes in symbolic descriptions.

That scheme meets three of the basic tenets of the agile paradigm, i.e open scope (unknowns cannot be set in advance), continuity of delivery (invariants are defined and managed by knowledge workers), and users in driving seats (through exit conditions). Yet it still doesn’t deal with creativity and the benefits of collaboration for knowledge workers.

Thinking Space & Pace

The scope of creativity in processes is neatly circumscribed by the nature of flows, i.e the possibility to insert knowledge during the processing: external for material flows (e.g in manufacturing), internal for symbolic flows (e.g in software engineering and knowledge processing).

Yet, whereas both software engineering and knowledge processes come with some built-in capability to redefined their symbolic flows on-the-fly, they don’t grant the same room to creativity. Contrary to software engineering projects which have to close their perspectives on the delivery of working products, knowledge processes are meant to keep them open to new understandings and opportunities. For the former creativity is the means to an end, for the latter it’s the end in itself, with collaboration as means.

Such opposite perspectives have direct consequences for two basic agile collaboration mechanisms: backlog and time-boxing:

  • Backlogs are used to structure and manage the space under exploration. But contrary to software processes whose space is focused and structured by users’ needs, knowledge processes are supposed to play on workers’ creativity to expand and redefine the range under consideration.
  • Time-boxes are used to synchronize tasks. But with creativity entering the fray, neither space granularity or thinking pace can be set in advance and coerced into single-sized boxes. In that case individuals must remain in full control of the contents and stride of their thinking streams.

It ensues that when creativity is the primary success factor standard agile collaboration mechanisms are falling short and intelligent collaboration schemes are to be introduced.

Creativity & Collaboration Tiers

The synchronization of creative activities has to deal with conflicting objectives:

  • On one hand the mental maps of knowledge workers and the stream of their thoughts have to be dynamically aligned.
  • On the other hand unsolicited face-to-face interactions or instant communications may significantly impair the course of creative thinking.

When activities, e.g software engineering, can be streamlined towards the delivery of clearly defined outcomes, backlogs and time-boxes can be used to harness workers’ creativity. When that’s not the case more sophisticated collaboration mechanisms are needed.

Assuming that mediated collaboration has a limited impact on thinking creativity (emails don’t have to be answered, or even presented, instantly), the objective is to steer knowledge workflows across a two-tiered collaboration framework: one personal and direct between knowledge workers, the other social and mediated through enterprise or institutional networks.

On the first tier knowledge workers would manage their thinking flows (content and tempo) independently, initiating or accepting personal collaboration (either through physical contact or some kind of instant messaging) depending on their respective “state of mind”.

The second tier would be for social collaboration and would be expected to replace backlogs and time-boxing. Proceeding from the first to the second tier would be conditioned by workers’ needs and expectations, triggered on their own initiative or following prompts.

From Personal to Collective Thinking

The challenging issue is obviously to define and implement the mechanisms governing the exchanges between collaboration tiers, e.g:

  • How to keep tabs on topics and contents to be safeguarded.
  • How to mediate (i.e filter and time) the solicitations and contribution issued by the social tier.
  • How to assess the solicitations and contribution issued by individuals.
  • How to assess and manage knowledge deemed to remain proprietary.
  • How to identify and manage knowledge workers personal and social circles.

Whereas such issues are customary tackled by various AI systems (knowledge management, decision-making, multi-players games, etc), taken as a whole they bring up the question of the relationship between personal and collective thinking, and as a corollary, the role of organization in nurturing corporate innovation.

Conclusion: Collaboration Spaces vs Panopticon

As illustrated by the rising of futuristic headquarters, leading technology firms have been trying to tackle these issues by redefining internal architecture as collaboration spaces. Compared to traditional open spaces, such approaches try to fuse physical and digital spaces into overlapping layers of collaboration spaces, using artificial intelligence to harness cooperation.

Yet, lest uniform and comprehensive transparency brings the worrying shadow of a panopticon within which everyone can be unknowingly observed, working spaces have to be designed as to enhance collaboration without trespassing on privacy.

That could be achieved with a layered transparency set along the nature of collaboration:

  • Immediate and personal: working cells regrouping 5 to 10 workstations earmarked for a task and used indifferently by teams members.
  • Delayed and personal: open physical spaces accommodating working cells, with instant messaging and geo-localization; spaces are hinged on domains and focused on shared knowledge.
  • On-line and networked: digital spaces merging physical spaces and organizational structures.

That mix of physical and virtual spaces could be dynamically redefined depending on activities, projects, location, and organisation.

Further Readings

External Links

Use Cases are Agile Tools

Preamble

Use cases are often associated with the whole of UML diagrams, and consequently with cumbersome procedures, and excessive overheads. But like cats, use cases can be focused, agile, and versatile.

Use cases are agile tools with side view
Use Case: agile, simple-minded, with a side view (P. Picasso)

Simple minded, Robust, Easy going

As initially defined by Ivar Jacobson, the aim of use cases is to describe what happens between users and systems. Strictly speaking, it means a primary actor (possibly seconded) and a triggering event (possibly qualified); that may (e.g transaction) or may not (e.g batch) be followed by some exchange of messages . That header may (will) not be the full story, but it provides a clear, simple and robust basis:

  • Clear and simple: while conditions can be added and interactions detailed, they are not necessary parts of the core definition and can be left aside without impairing it.
  • Robust: the validity of the core definition is not contingent on further conditions and refinements. It ensues that simple and solid use cases can be defined and endorsed very soon, independently of their further extension.

As a side benefit, use cases come with a smooth learning curve that enable analysts to become rapidly skilled without being necessarily expert.

Open minded and Versatile

Contrary to a somewhat short-sighted perspective, use cases are not limited to users because actors (aka roles) are meant to hide the actual agents involved: people, devices, or other systems. As a consequence, the scope of UCs is not limited to dialog with users but may also includes batch (as one-step interactions) and real-time transactions.

Modular and Inter-operable

Given their simplicity and clarity of purpose, use cases can be easily processed by a wide array of modeling tools on both sides of the business/engineering divide, e.g BPM and UML. That brings significant benefits for modularity.

At functional level use cases can be used to factor out homogeneous modules to be developed by different tools according to their nature. As an example, shared business functions may have to be set apart from business specific scenarii.

Use cases at the hub of UML diagrams
Use cases can be easily combined with a wide range of modeling tools

At technical level, interoperability problems brought about by updates synchronization are to be significantly reduced simply because modules’ core specifications (set by use cases) can be managed independently.

Iterative

Given their modularity, use cases can be easily tailored to the iterative paradigm. Once a context is set by business process or user’s story, development iterations can be defined with regard to:

  • Invariants (use case): primary actor and triggering event.
  • Iterations (scenarii): alternative execution paths identified by sequences of extension points representing the choices of actors or variants within the limits set by initial conditions.
  • Backlog units: activities associated to segments of execution paths delimited by extension points.
  • Exit condition: validation of execution path.
cccc
Execution paths & Development cycles

Scalable

Last but not least, use cases provide a sound and pragmatic transition between domain specific stories and architectural features. Taking a leaf from the Scaled Agile Framework, business functions can be factored out as functional features represented by shared use cases whose actors stand for systems and not final users.

BP2SOA_4
Mapping Processes to Services (through Users’ Stories)

Depending on targeted architectures, those features may or may not be implemented by services.

Further Reading

External Links

Agile Delivery & Dynamic Programming

Synopsis

Business driven development is arguably the cornerstone of the agile development model. On one side it means business value set by users and stakeholders, on the other side it entails continuous and just-in-time delivery; what happens in-between is set by backlogs (for development) and product increments (for delivery).

greekAthl_RN3c
Continuous Delivery

A sanguine understanding of continuous releases may assume that planning is no longer relevant, and that deployment can be carried out “on-the-fly”. But that would assume that stakeholders and product owners are ready to put aside roadmaps, overlook milestones, and more generally forget that time is money. That would go against a basic objective of agile, namely that developments must be driven by business needs, and products delivered just-in-time for best value. Given the well established track record of dynamic programming for  manufacturing processes, could the technique be usefully applied to agile engineering processes ?

Delivery, Deployment, & Continuity

Continuity doesn’t mean synchronization: business, engineering, and operations are governed by different concerns set along different time-frames. Some buffering is needed, materialized by the distinction between releases (engineering concerns) and deployment (business and operational concerns).

A time for every purpose: Epics & business roadmap (a), associated backlogs of users’ stories (b), released features (c), architectural capabilities (d), deployed components (e).
A time for every purpose: Epics & business roadmap (a), associated backlogs of users’ stories (b), released features (c), architectural capabilities (d), deployed components (e).

Such distinctions introduce both overlapping (with business time-frame) and discontinuity (between development and deployment):

  • Product roadmaps are set in business time-frames and determine development and deployment time-frames.
  • Development time is set by product roadmaps and runs clockwise from project inception to software releases (a>b>c).
  • Deployment time is also set relative to product roadmaps but it runs counterclockwise, from product deployment as planned by business back to released software components (c<e).

Development and deployments runs can be compared to crews tunneling through a mountain from both sides; where and when they meet leaves room to adjustments. Yet more is at stake in the meeting between development completion and deployment inception. Apart from time, adjustments may also bear on formats and contents; and given the specificity of development and deployment purposes, their adjustment may also be seen as the morphing of continuous software releases (project perspective) into discrete increments (product perspective).

Dynamic Programming

Dynamic programming (aka multistage programming) is a problem solving method that combines two principles:

  • Divide & conquer is a general purpose strategy that deals with the intrinsic complexity of problems by breaking them down into collections of simpler sub-problems to be solved separately depending on sequencing constraints.
  • Recursion deals with the lack of complete or accurate information upfront, solving the problem in stages rather than as one entity. Each stage is optimized separately on the basis of the current state reflecting decisions taken at previous stages, the optimality principle guaranteeing the optimality of the final outcome.

That incremental and iterative approach clearly befits the tenets of the agile development model.

Twofold Planning

As noted above, and whatever the technique, agile processes entwine two phases, one for development, the other for deployment, each with its own planning:

  • The development planning, epitomized by backlog management, deals with the definition of work units and their sequencing.
  • The deployment planning deals with the merging of software released into product and their incremental deployment.

Project work units are sequenced (backlog), Product increments are merged, and both are dynamically adjusted around their nexus.
Project work units are sequenced (backlog), Product increments are merged, and both are dynamically adjusted around their nexus.

That makes for multiple dynamics, first for updated backlogs, then for updated deployment targets, and finally for possible feedback through their nexus. That can only be achieved with dynamic programming.

Backlog: Multistage sequencing

Backlogs are used to manage work units targeting self-contained requirements items; they can be represented by graphs with nodes standing for work units and arcs weighted by constraints.

Basically, the problem is to optimize the development of a given set of items given users’ priorities, technical constraints, and resources availability. When all the information is available upfront, optimum solutions can be obtained with simple Shortest Path algorithms. Yet, given the iterative and exploratory nature of agile processes, backlogs are meant to be updated as the project advances, taking advantage of improved knowledge:

  • Users may introduce new items, remove ones, or change their priorities due to a better understanding of requirements space.
  • Engineers may also introduce new items (e.g for technical debt) or reconsider technical difficulties and dependencies due to a better understanding of solutions space.

Dynamic reordering of backlogs: looking forward for the optimum path to completion.
Dynamic reordering of backlogs: looking forward for the optimum path to completion.

Dynamic programming is introduced in order to support step-wise decisions optimizing the whole process:

  • Backlog states (t1 & t2) are defined by the remaining work units, rankings, and feasibility constraints.
  • Each stage redefines the optimum path to completion taking into account the current state and updated information. Recursive computation being based on the summary information etched in states, it ensues that all future decisions can be selected optimally without recourse to information regarding previously made decisions.

Given a set of feasible paths (as defined by technical dependencies and time), the aim at each stage is to select the optimum path for the remaining units based on current state. Optimization functions will typically consider users’ value, learning curve and associated risks, and resources availability and costs.

As illustrated below, nodes can represent grouped items, e.g when several projects have to share resources or releases are to be regrouped.

Deployment: Dynamic merging

Given a set of released software components, the aim of deployment planning is to decide which increment to add to deployed product. Assuming that technical concerns have already been dealt with by releases, the objective at each stage is select the items maximizing the ROI of the deployed product. It must be noted that, contrary to the development algorithm looking forward for the optimization, the deployment algorithm select the optimum path by looking backward at the ROI of deployed products.

Dynamic reordering of deployments: looking backward for the optimum path to completion.
Dynamic reordering of deployments: looking backward for the optimum path to completion.

But the backward impact of deployment optimization can go further and affect backlogs.

Feedback

Shared ownership and continuous delivery are two main pillars of the agile development model, the former giving the development full authority and responsibility, the latter ensuring the users have a firm hand on the helm. Yet, as already noted, development, deployment, and business are governed by different time-frames, which could induce some frictions, e.g if business units were forced to synchronize products deployments with software releases. While severe disruptions can be avoided if releases and deployments are managed separately, development teams cannot be completely sheltered from changes in business or operational priorities. That is where the dynamic reassessment of optimum paths is to help: assuming a change in deployment planning (nq instead of op), the new priorities are fed back into development (aka backlog) rankings, and the optimum path is updated.

Feedback
Change in deployment priorities (nq instead of op) can be fed back to backlog planning (f before a).

It must be noted that such feedback only affects ranks and must leaves contents unchanged.

Conclusion: Business driven, Just-in-time delivery, & Lean Inventories

Dynamic programming appears as a primary factor with regard to three core tenets of the agile development model:

  • Business driven development doesn’t mean that developments are pushed by requirements but that they are pulled by deployment.
  • Just-in-time delivery can only be achieved with the help of a buffer between development and deployment. This buffer should not be confused with an inventory as it has nothing to do with product quantities.
  • On the contrary, this buffer, combined with dynamic programming, plays a critical role in the cutback of intermediate documents and models (aka development inventories).

Further Readings

External Links

Agile Architectures: Versatility meets Plasticity

Synopsis

At enterprise level agility can be understood as a mix of versatility and plasticity, the former an attribute of function, the latter of form:

  • Versatility: enterprise ability to adapt business processes to changing environments without having to change architectures.
  • Plasticity: enterprise ability to change architectures without affecting business processes.

Plasticity is for form, versatility for function
Agility: Forms & Performances (P. Pénicaud)

Combining versatility and plasticity requires a comprehensive and consistent view of assets (architectures) and modus operandi (processes) organized with regard to change. And that can be achieved with model based systems engineering (MBSE).

MBSE & Change

Agility is all about change, and if enterprise governance is not to be thrown aside decision-making has to be supported by knowledgeable descriptions of enterprise objectives, assets, and organization.

If change management is to be the primary objective, targets must be classified along two main distinctions:

  • Actual (business context and organization) or symbolic (information systems).
  • Objects (business entities or system surrogates) or activities (business processes or logic).

Entropy_muove
Comprehensive and consistent descriptions of actual and symbolic assets (architectures) and modus operandi (processes) with regard to change management.

The two axis determine four settings supporting transparency and traceability:

  • Dependencies between operational and structural elements.
  • Dependencies between actual assets and processes and their symbolic representation as systems surrogates.

Versatility and plasticity will be obtained by managing changes and alignments between settings.

Changes & Alignments

Looking for versatility, changes in users’ requirements must be rapidly taken into account by applications (changes from actual to symbolic).

Looking for plasticity, changes in business objectives are meant to be supported by enterprise capabilities (changes from operational to structural).

The challenge is to ensure that both threads can be weaved together into business functions and realized by services (assuming a service oriented architecture).

With the benefits of MBSE, that could be carried out through a threefold alignment:

  • At users level the objective is to ensure that applications are consistent with business logic and provide the expected quality of service. That is what requirements traceability is meant to achieve.
  • At system level the objective is to ensure that business functions and features can be directly mapped to systems functionalities. That is what services oriented architectures (SOA) are  meant to achieve.
  • At enterprise level the objective is to ensure that the enterprise capabilities are congruent with its business objectives, i.e that they support its business processes through an effective use of assets. That is what maturity and capability models are meant to achieve.

Alignment
Versatility comes from users’ requirements, plasticity from architectures capabilities.

That would make agility a concrete endeavor across enterprise, from business users and applications to business processes and architectures capabilities.

Further Reading

Inducing Functional Patterns from Design ones: a look in rear view mirror

Preamble

Assuming patterns are meant to chart the path from problems to solutions, and given the foundational contribution of the Gang of Four (GoF), it may help to look at functional (aka representation) patterns backward  from the perspective of the now well established solutions framework (aka design patterns).

(M.Kippenberger)
Matching Patterns (M.Kippenberger)

Patterns & Models

Patterns are handrails from stereotyped problems to well known and tested solutions, bringing the benefits of reuse for costs, quality, and traceability. Set within the Model Driven Architecture, patterns can be regrouped depending on their source and target:

  • Functional patterns deal with the representation of business objects and activities by system symbolic surrogates. They stand between computation and platform independent models. Whereas functional patterns are not to be confused with business patterns (used with CIMs), they are also known as analysis patterns when targeting PIMs.
  • Design patterns deal with the implementation of symbolic surrogates as software components.  They stand between platform independent and platform specific models.
MDA layers clearly coincide with reusable assets
Patterns and MDA layers

As it happens, there is a striking contrast between the respective achievements for design and functional (or analysis) patterns, the former, firmly set up by the pivotal work of the GoF, is nowadays widely and consistently applied to software design; the latter, frustrated by the medley of requirements capture, remain fragmented and confined to proprietary applications. Given that discrepancy, it may be worth to look in the rear-view mirror  and consider functional patterns from the perspective of design ones.

Objects, Classes, & Types

Design patterns are part and parcel of the object oriented (OO) approach and built on some of its core tenets, in particular:

  • Favor object composition over class inheritance.
  • Program to an interface, not an implementation.

Whereas the meaning of classes, implementation, and inheritance is specific to OO design, the principles can nonetheless bear upon analysis providing they are made to apply to types and composition, whose semantics have a broader scope. To begin with, if classes describe the software implementation of symbolic objects, and inheritance the relationships between them, types only pertain to their functional capabilities. As a corollary, the semantics of inheritance and composition makes no difference with regard to types, as opposed to classes. The lesson for functional patterns would therefore to favor types whenever possible, i.e when functional capabilities doesn’t depend on identities. Then, “programming to interfaces” would translate (for functional patterns) into describing communications between types. That approach being best represented by aspect oriented analysis. Not by chance, the lessons drawn above appear to be directly supported by the two criteria used by the the GoF to classify design patterns: scope and purpose. With regard to scope, class patterns deal with inheritance relationships between static descriptions fixed at compile-time, while object patterns deal also with relationships between instances which can be changed at run-time. With regard to purpose, creational patterns deal with objects instanciation, structural ones carry on with composition, and behavioral ones with algorithms and responsibilities.

Design Patterns Catalog
Design Patterns Catalog

The corresponding entries can be summarily described as:

  • Class/Creational: for new identified instances.
  • Object/Creational: for new parts of identified instances.
  • Class/Structural: for mixing interfaces using inheritance.
  • Object/Structural: for mixing interfaces using composition.
  • Class/Behavioral: for the definition of algorithms.
  • Object/Behavioral: for the collaboration between instances.

The next step would be to induce a catalog of functional patterns from those entries.

Inducing a Catalog of Functional Patterns

Leveraging the GoF’s design patterns, the objective is to spread the benefits upstream to the whole of the modeling process through a transparent mapping of functional and design patterns. And that could be achieved by inducing a functional catalog from its design equivalent. With regard to scope, the objective would be to factor out the architectural impact of requirements. Hence the distinction between architecture patterns dealing with business entities and processes consistently identified at enterprise and system levels, and aspect patterns dealing with features that can be managed separately. With regard to purpose the same semantics can be applied: creational patterns deal with objects instanciation, structural ones carry on with their composition and features, and behavioral ones with algorithms and responsibilities.

Functional Patterns Catalog: examples
Functional Patterns Catalog: examples
  • Architecture/Creational: for the mapping of business entities to their software surrogates. For instance, customers must be identified as parties (architecture/creational) before being fleshed out with specific features.
  • Aspect/Creational: for the intrinsic components of identified business entities. For instance, once created and identified as a customer, a foreign one is completed with the specific mandatory features of foreigners.
  • Architecture/Structural:  for inherited (i.e intrinsic) features of identified business entities.  For instance, foreign sales inherit specific invoice features.
  • Aspect/Structural:  for associated (intrinsic or otherwise) features of identified business entities. For instance, foreign sales delegate invoicing in foreign currencies.
  • Architecture/Behavioral: for the definition business rules attached to entities and managed by business domains. For instance invoicing rules are specialized for foreign currencies.
  • Aspect/Behavioral: for the definition of collaboration between entities independently of business domains. For instance allocating responsibilities at run-time depending on the invoice.

As can be seen with the MDA diagram above, this catalog is neutral as its entries are congruent with the categories used by the main modeling methodologies : use cases, domains, business rules, services, objects, aspects, etc.

Recommended Reading

Gamma E., Helm R., Johnson R. and Vlissides John M. , “Design Patterns: Elements of Reusable Object-Oriented Software”, Addison-Wesley (1994).

Further Reading

Feasibility & Capabilities

Synopsis

As far as systems engineering is concerned, the aim of a feasibility study is to verify that a business solution can be supported by a system architecture (requirements feasibility) subject to some agreed technical and budgetary constraints (engineering feasibility).

Feasibility is about requirements, capabilities with supporting systems (Urs Fisher)

Where to Begin

A feasibility study is based on the implicit assumption of slack architecture capabilities. But since capabilities are set with regard to several dimensions, architectures boundaries cannot be taken for granted and decisions may even entail some arbitrage between business requirements and engineering constraints.

Using the well-known distinction between roles (who), activities (how), locations (where), control (when), and contents (what), feasibility should be considered for supporting functionalities (between business processes and systems) and implementation (between functionalities and platforms):

Feasibility with regard to Systems and Platforms

Depending on priorities, feasibility considerations could look from three perspectives:

  • Focusing on system functionalities (e.g with use cases) implies that system boundaries are already identified and that the business logic will be defined along with users’ interfaces.
  • Starting with business requirements puts business domains and logic on the driving seat, making room for variations in system functionalities and boundaries .
  • Operational requirements (physical environments, events, and processes execution) put the emphasis on a mix of business processes and quality of service, thus making software functionalities a dependent variable.

In any case a distinction should be made between requirements and engineering feasibility, the former set with regard to architecture capabilities, the latter with regard to development resources and budget constraints.

Requirements Feasibility & Architecture Capabilities

Functional capabilities are defined at system boundaries and if all feasibility options are to be properly explored, architectures capabilities must be understood as a trade-off between the five intrinsic factors e.g:

  • Security (entry points) and confidentiality (domains).
  • Compliance with regulatory constraints (domains) and flexibility (activities).
  • Reliability (processes) and interoperability (locations).
Feasible options must be set against capabilities

Feasible options could then be figured out by points set within the capabilities pentagon. Given metrics on functional requirements, their feasibility under the non functional constraints could be assessed with regard to cross capabilities. And since the same five core capabilities can be consistently defined across enterprise, systems, and platforms layers, requirements feasibility could be assessed without prejudging architecture decisions.

Business Requirements & Architecture Capabilities

One step further, the feasibility of business and operational objectives (the “Why” of the Zachman framework) can be more easily assessed if set on the outer range and mapped to architecture capabilities.

Business Requirements and Architecture Capabilities

Engineering Feasibility & ROI

Finally, the feasibility of business and functional requirements under the constraints set by non functional requirements has to be translated in terms of ROI, and for that purpose the business value has to be compared to the cost of engineering the solution given the resources (people and tools), technical requirements, and budgetary constraints.

Architecture Capabilities for Processes, with deontic (basic) and alethic (dashed) dependencies.
ROI assessment mapping business value against functionalities, engineering outlays, and operational costs.

That where the transparency and traceability of capabilities across layers may be especially useful when alternatives and priorities are to be considered mixing functionalities, engineering outlays, and operational costs.

Further Reading

Business Processes & Use Cases

Summary

As can be understood from their theoretical basis (Pi-Calculus, Petri Nets, or State Machines), processes are meant to describe the concurrent execution of activities. Assuming that large enterprises have to keep a level of indirection between operations and business logic, it ensues that activities and business logic should be defined independently of the way they are executed by business processes.

Communication Semantics vs Contexts & Contents (G. Winogrand)

For that purpose two basic modeling approaches are available:  BPM (Business Process Modeling) takes the business perspective, UML (Unified Modeling Language) takes the engineering one. Yet, each falls short with regard to a pivotal conceptual distinctions: BPM lumps together process execution and business logic, and UML makes no difference between business and software process execution. One way out of the difficulty would be to single out communications between agents (humans or systems), and specify interactions independently of their contents and channels.

vvv
Business Process: Communication + Business Logic

That could be achieved if communication semantics were defined independently of domain-specific languages (for information contents) and technical architecture (for communication channels). As it happens, and not by chance, the outcome would neatly coincide with use cases.

Linguistics & Computers Parlance

Business and functional requirements (see Requirements taxonomy) can be expressed with formal or natural languages. Requirements expressed with formal languages, domain-specific or generic, can be directly translated into some executable specifications. But when natural languages are used to describe what business expects from systems, requirements often require some elicitation.

When that challenge is put into a linguistic perspective, two school of thought can be considered, computational or functional.

The former approach is epitomized by Chomsky’s Generative Grammar and its claim that all languages, natural or otherwise, share an innate universal grammar (UG) supporting syntactic processing independently of their meanings. Nowadays, and notwithstanding its initial favor, that “computer friendly” paradigm hasn’t kept much track in general linguistics.

Alternatively, the functionalist school of thought considers linguistics as a general cognitive capability deprived of any autonomy. Along that reasoning there is no way to separate domain-specific semantics from linguistic constructs, which means that requirements complexities, linguistic or business specific, have to be dealt with as a lump, with or without the help of knowledgeable machines.

In between, a third approach has emerged that considers language as a functional system uniquely dedicated to communication and the mapping of meanings to symbolic representations. On that basis it should be possible to separate the communication apparatus (functional semantics) from the complexities of business (knowledge representation).

Processes Execution & Action Languages

Assuming the primary objective of business processes is to manage the concurrent execution of activities, their modeling should be driven by events and their consequences for interactions between business and systems. Unfortunately, that approach is not explicitly supported by BPM or UML.

Contrary to the “simplex” mapping of business information into corresponding data models (e.g using Relational Theory), models of business and systems processes (e.g Petri Nets or State Machines) have to be set in a “duplex” configuration as they are meant to operate simultaneously. Neither BPM nor UML are well equipped to deal with the task:

  • The BPM perspective is governed by business logic independently of interactions with systems.
  • Executable UML approaches are centered on software processes execution, and their extensions to action semantics deal essentially on class instances, features value, and objects states.

Such shortcomings are of no serious consequences for stand-alone applications, i.e when what happens at architecture level can be ignored; but overlooking the distinction between the respective semantics of business and software processes execution may critically hamper the usefulness, and even the validity, of models pertaining to represent distributed interactions between users and systems. Communication semantics may help to deal with the difficulty by providing relevant stereotypes and patterns.

Business Process Models

While business process models can (and should) also be used to feed software engineering processes, their primary purpose is to define how concurrent business operations are to be executed. As far as systems engineering is concerned, that will tally with three basic steps:

  1. Dock process interactions (aka sessions) to their business context: references to agents, business objects and time-frames.
  2. Specify interactions: references to context, roles, events, messages, and time related constraints.
  3. Specify information: structures, features, and rules.

Communication with systems: dock to context (1), interactions (2), information (3).
Communication with systems: dock to context (1), interactions (2), information (3).

Although modeling languages and tools usually support those tasks, the distinctions remain implicit, leaving users with the choice of semantics. In the absence of explicit guidelines confusion may ensue, e.g between business rules and their use by applications (BPM), or between business and system events (UML). Hence the benefits of introducing functional primitives dedicated to the description of interactions.

Such functional semantics can be illustrated by the well known CRUD primitives for the creation, reading, updating and deletion of objects, a similar approach being also applied to the design of domain specific patterns or primitives supported by functional frameworks. While thick on the ground, most of the corresponding communication frameworks deal with specific domains or technical protocols without factoring out what pertains to communication semantics independently of information contents or technical channels.

Communication semantics should be independent of business specific contents and systems architectures.
Communication semantics should be independent of business specific contents and systems architectures.

But that distinction could be especially productive when applied to business processes as it would be possible to fully separate the semantics of communications between agents and supporting systems on one hand, and the business logic used to process business flows on the other hand.

 Communication vs Action Semantics

Languages have developed to serve two different purposes: first as a means to communicate (a capability shared between humans and many primates), and then as a means to represent and process information (a capability specific to humans). Taking a leaf from functional approaches to linguistics, it may be possible to characterize messages with regard to action semantics, more precisely associated activity and attendant changes:

  • No change: messages relating to passive (objects) or active  (performed activity) states.
  • Change: messages relating to achievement (no activity) or accomplishment (attendant on performed activity).

Basic action semantics for interactions between users (BPM) and systems (UML)

Communication semantics can then be fully rounded off by adding changes in agents’ expectations to the ones in the states of objects and activities, all neatly modeled with state machines.

This image has an empty alt attribute; its file name is etats_21_41-e1544196131304.jpg

Communication semantics: changes in expectations, objects, and activities.

It must also be noted that factoring out the modeling of agents’ expectations with regard to communications is in line with the principles of service oriented architectures (SOA).

Additionally, transitions would have to be characterized by:

  • Time-frame: single or different.
  • Address space : single or different.
  • Mode: information, request for information, or request for action.

Organizing business processes along those principles, would enable the alignment of BPMs with their UML counterpart.

Use Cases as Bridges between BPM & UML

Use cases can be seen as the default entry point for UML modeling, and as such they should provide a bridge from business process models. That can be achieved by if use cases are understood as a combination of interactions and activities, the former obtained from communications as defined above, the latter from business logic.

Use Cases are best understood as a combination of interactions and business logic
Use Cases are best understood as a combination of interactions and business logic

One step further, the distinction between communication semantics and business contents could be used to align models respectively to systems and software architectures:

Last but not least, the consolidation of models contents at architecture level would be congruent with modeling languages, BPM and UML.

Further Reading

External Links