caminao

MDA & EA: Is The Tail Wagging The Dog ?

Making Heads or Tails

OMG’s Model Driven Architecture (MDA) is a systems engineering framework set along three model layers:

Computation Independent Models (CIMs) describe business objects and activities independently of supporting systems.
Platform Independent Models (PIMs) describe systems functionalities independently of platforms technologies.
Platform Specific Models (PSMs) describe systems components as implemented by specific technologies.

Since those layers can be mapped respectively to enterprise, functional, and technical architectures, the question is how to make heads or tails of the driving: should architectures be set along model layers or should models organized according architecture levels.

(judy Kensley McKie) — A Dog Making Head or Tail (Judy Kensley McKie)

In other words, has some typo reversed the original “architecture driven modeling” (ADM) into “model driven architecture” (MDA) ?

Wrong Spelling, Right Concepts

A confusing spelling should not mask the soundness and relevance of the approach: MDA model layers effectively correspond to a clear hierarchy of problems and solutions:

Computation Independent Models describe how business processes support enterprise objectives.
Platform Independent Models describe how systems functionalities support business processes.
Platform Specific Models describe how platforms implement systems functionalities.

MDA layers correspond to a clear hierarchy of problems and solutions

That should leave no room for ambiguity: regardless of the misleading “MDA” moniker, the modeling of systems is meant to be driven by enterprise concerns and therefore to follow architecture divides.

Architectures & Assets Reuse

As it happens, the “MDA” term is doubly confusing as it also blurs the distinction between architectures and processes. And that’s unfortunate because the reuse of architectural assets by development processes is at the core of the MDA framework:

Business objects and logic (CIM) are defined independently of the functional architectures (PIM) supporting them.
Functional architectures (PIM) are defined independently of implementation platforms (PSM).
Technical architecture (PSM) are defined independently of deployment configurations.

MDA layers clearly coincide with reusable assets — MDA layers coincide with categories of reusable assets

Under that perspective the benefits of the “architecture driven” understanding (as opposed to the “model driven” one) appear clearly for both aspects of enterprise governance:

Systems governance can be explicitly and transparently aligned on enterprise organization and business objectives.
Business and development processes can be defined, assessed, and optimized with regard to the reuse of architectural assets.

With the relationship between architectures and processes straightened out and architecture reinstated as the primary factor, it’s possible to reexamine the contents of models used as hinges between them.

Languages & Model Purposes

While engineering is not driven by models but by architectures, models do describe architectures. And since models are built with languages, one should expect different options depending on the nature of artifacts being described. Broadly speaking, three basic options can be considered:

Versatile and general modeling languages like UML can be tailored to different contexts and purposes, along development cycle (requirements, analysis, design) as well as across perspectives (objects, activities, etc) and domains (banking, avionics, etc)
Non specific business modeling languages like BPM and rules-based languages are meant to be introduced upfront, even if their outcome can be used further down the development cycle.
Domain specific languages, possibly built with UML, are also meant to be introduced early as to capture domains complexity. Yet, and contrary to languages like BPM, their purpose is to provide an integrated solution covering the whole development cycle.

Languages: general purpose (blue), process or domain specific (green), or design. — Languages: general purpose (blue), process or domain specific (green), or design (brown).

As seen above for reuse and enterprise architecture, a revised MDA perspective clarifies the purpose of models and consequently the language options. With developments “driven by models”, code generation is the default option and nothing much is said about what should be shared and reused, and why. But with model contents aligned on architecture levels, purposes become explicit and modeling languages have to be selected accordingly, e.g:

Domain specific languages for integrated developments (PSM-centered).
BPM for business specifications to be implemented by software packages (CIM-centered).
UML for projects set across system functional architecture (PIM-centered).

The revised perspective and reasoned association between languages and architectures can then be used to choose development processes: projects that can be neatly fitted into single boxes can be carried out along a continuous course of action, others will require phased development models.

Enterprise Architecture & Engineering Processes

Systems engineering has to meet different kinds of requirements: business goals, system functionalities, quality of service, and platform implementations. In a perfect (model driven engineering) world there would be one stakeholder, one architecture, and one time-frame. Unfortunately, requirements are usually set by different stakeholders, governed by different rationales, and subject to changes along different time-frames. Hence the importance of setting forth the primary factors governing engineering processes:

Planning: architecture levels (business, systems, platforms) are governed by different time-frames and engineering projects must be orchestrated accordingly.
Communication: collaboration across organizational units require traceability and transparency.
Governance: decisions across architecture levels and business units cannot be made upfront and options and policies must be assessed continuously.

Those objectives are best supported when engineering processes are set along architecture levels:

Requirements: at enterprise level requirements deal with organization and business processes (CIMs). The enterprise requirements process starts with portfolio management, is carried on with systems functionalities, and completed with platforms operational requirements.
Problems Analysis: at enterprise level analysis deals with symbolic representations of enterprise environment, objectives, and activities (PIMs). The enterprise analysis process starts with the consolidation of symbolic representations for objects (power-types) and activities (scenarii), is carried on with functional architectures, and completed with platforms non-functional features. Contrary to requirements, which are meant to convey changes and bear adaptation (dashed lines), the aim of analysis at enterprise level is to consolidate symbolic representations and guarantee their consistency and continuity. As a corollary, analysis at system level must be aligned with its enterprise counterpart before functional (continuous lines) requirements are taken into account.
Solutions Design: at enterprise level design deals with operational concerns and resources deployment. The enterprise design process starts with locations and resources, is carried on with systems configurations, and completed with platforms deployments. Part of it is to be supported by systems as designed (PSMs) and implemented as platforms. Yet, as figured by dashed arrows, operational solutions designed at enterprise level bear upon the design of systems architectures and the configuration of their implementation as platforms.

When engineering is driven by architectures, processes can be devised depending on enterprise concerns and engineering contexts. While that could come with various terminologies, the partitioning principles will remain unchanged, e.g:

Agile processes will combine requirements with development and bypass analysis phases (a).
Projects meant to be implemented by Commercial-Off-The-Shelf Software (COTS) will start with business requirements, possibly using BPM, then carry on directly to platform implementation, bypassing system analysis and design phases (b).
Changes in enterprise architecture capabilities will be rooted in analysis of enterprise objectives, possibly but not necessarily with inputs from business and operational requirements, continue with analysis and design of systems functionalities, and implement the corresponding resources at platform level (c).
Projects dealing with operational concerns will be conducted directly through systems design of and platform implementation (d).

Processes should be devised according enterprise concerns and engineering contexts — Examples of process templates depending on objectives and contexts.

To conclude, when architecture is reinstated as the primary factor, the MDA paradigm becomes a pivotal component of enterprise architecture as it provides a clear understanding of architecture divides and dependencies on one hand, and their relationship with engineering processes on the second hand.

Postscript

MDA illustrates a contrario the straying agenda of the OMG: despite its soundness as a foundation of enterprise architecture as well as its complementarity with the Unified Modeling Language (UML), MDA three-tiers framework has been left unattended, to be replaced by an open-ended (more than two hundred, and counting) hotpotch of amorphous and overlapping meta-models.

Nonetheless, the paradigm is not to be forsaken and can be found behind the Zachman framework and its revamping by Caminao.

https://caminao.blog/wp-content/uploads/2019/01/symbotransfo_pagoda.jpg?w=600

MDA reincarnation as the Pagoda Blueprint

External Links

Arcadia

From Processes to Services

Objective

Even in the thick of perplexing debates, enterprise architects often agree on the meaning of processes and services, the former set from a business perspective, the latter from a system one. Considering the rarity of such a consensus, it could be used to rally the different approaches around a common understanding of some of EA’s objectives.

BOB1951003W00007/ICP912 — Process with service (Robert Capa)

A Governing Dilemma

Systems have long been of three different species that communicated but didn’t interbred: information ones were calmly processing business records, industrial ones were tensely controlling physical devices, and embedded ones lived their whole life stowed away within devices. Lastly, and contrary to the natural law of evolution, those three species have started to merge into a versatile and powerful new breed keen to colonize the whole of enterprise ecosystem.

When faced with those pervading systems, enterprises usually waver between two policies, containment or integration, the former struggling to weld and confine all systems within technology boundaries, the latter trying to fragment them and share out the pieces between whichever business units ready to take charge.

While each approach may provide acceptable compromises in some contexts, both suffer critical flaws:

Centralized solutions constrict business opportunities and innovation by putting all concerns under a single unwieldy lid of technical constraints.
Federated solutions rely on integration mechanisms whose increasing size and complexity put the whole of systems integrity and adaptability on the line.

Service oriented architectures may provide a way out of this dilemma by introducing a functional bridge between enterprise governance and systems architectures.

Separation of Concerns

Since governance is meant to be driven by concerns, one should first consider the respective rationales behind business processes and system functionalities, the former driven by contexts and opportunities, and the latter by functional requirements and platforms implementation.

While business processes usually involve various degrees of collaboration between enterprises, their primary objective is to fulfill each one’s very specific agenda, namely to beat the others and be the first to take advantage of market opportunities. That put systems at the cross of a dual perspective: from a business point of view they are designed to provide a competitive edge, but from an engineering standpoint they aim at standards and open infrastructures. Clearly, there is no reason to assume that those perspectives should coincide, one being driven by changes in competitive environments, the other by continuity and interoperability of systems platforms. That’s where Service Oriented Architectures should help: by introducing a level of indirection between business processes and system functionalities, services naturally allow for the mapping of requirements with architecture capabilities.

bp2sa_layers — Services provide a level of indirection between business and system concerns.

Along that reasoning (and the corresponding requirements taxonomy), the design of services would be assessed in terms of optimization under constraints: given enterprise organization and objectives (business requirements), the problem is to maximize the business value of supporting systems (functional requirements) within the limits set by implementation platforms (non functional requirements).

Services & Capabilities

Architectures and processes are orthogonal descriptions respectively for enterprise assets and activities. Looking for the footprint of supporting systems, the first step is to consider how business processes should refer to architecture capabilities :

From a business perspective, i.e disregarding supporting systems and platforms, processes can be defined in terms of symbolic objects, business logic, and the roles of agents, devices, and systems.
The functional perspective looks at the role of supporting systems; as such, it is governed by business objectives and subject to technical constraints.
From a technical perspective, i.e disregarding the symbolic contents of interactions between systems and contexts, operational processes are characterized by the nature of interfaces (human, devices, or other systems), locations (centralized or distributed), and synchronization constraints.

Service oriented architectures typify the functional perspective by factoring out the symbolic contents of system functionalities, introducing services as symbolic hinges between enterprise and system architectures. And when defined in terms of customers, messages, contract, policy, and endpoints, services can be directly mapped to architectures capabilities.

Services are a perfect match for capabilities

Moreover, with services defined in terms of architecture capabilities, the divide between business and operational requirements can be drawn explicitly:

Actual (external) entities and their symbolic counterpart: services only deal with symbolic objects (messages).
Actual entities and their roles: services know nothing about physical agents, only about symbolic customers.
Business logic and processes execution: contracts deal with the processing of symbolic flows, policies deal with operational concerns.
External events and system time: service transactions are ACID, i.e from customer standpoint, they appear to be timeless.

Those distinctions are used to factor out the common backbone of enterprise and system architectures, and as such they play a pivotal role in their alignment.

Anchoring Business Requirements to Supporting Systems

Business processes are meant to met enterprise objectives given contexts and resources. But if the alignment of enterprise and system architectures is to be shielded from changes in business opportunities and platforms implementation, system functionalities will have to support a wide range of shifting business goals while securing the continuity and consistency of shared resources and communication mechanisms. In order to conciliate business changes with system continuity, business processes must be anchored to objects and activities whose identity and semantics are set at enterprise level independently of the part played by supporting systems:

Persistent units (aka business objects): structured information uniquely associated to identified individuals in business context. Life cycle and integrity of symbolic representations must be managed independently of business processes execution.
Functional and execution units: structured activity triggered by an event identified in business context, and whose execution is bound to a set of business objects. State of symbolic representations must be managed in isolation for the duration of process execution.

Services can be defined according persistency and functional units (#)

The coupling between business units (persistent or transient) identified at business level and their system counterpart can be secured through services defined with regard to business processes (customers), business objects (messages), business logic (contract), and business operations (policy).

It must be noted that while services specifications for customers, messages, contracts, and policy are identified at business level and completed at functional level, that’s not the case for endpoints since services locations are set at architecture level independently of business requirements.

Filling out Functional Requirements

Functional requirements are set in two dimensions, symbolic and operational; the former deals with the contents exchanged between business processes and supporting systems with regard to objects, activities and events, or actors; the latter deals with the actual circumstances of the exchanges: locations, interfaces, execution constraints, etc.

Given that services are by nature shared and symbolic, they can only be defined between systems. As a corollary, when functionalities are slated as services, a clear distinction should be maintained between the symbolic contents exchanged between business processes and supporting systems, and the operational circumstances of actual interactions with actors.

Interactions: symbolic and local (a), non symbolic and local (b), symbolic and shared (c).

Depending on the preferred approach for requirements capture, symbolic contents can be specified at system boundaries (e.g use cases), or at business level (e.g users’ stories). Regardless, both approaches can be used to flesh out the symbolic descriptions of functional and persistency units.

From a business process standpoint, users (actors in UML parlance) should not be seen as agents but as the roles agents play in enterprise organization, possibly with constraints regarding the quality of service at entry points. That distinction between agents and roles is critical if functional architectures are to dissociate changes in business processes on one hand, platform implementation on the other hand.

Along that understanding actors triggering use cases (aka primary actors) can represent the performance of human agents as well as devices or systems. Yet, as far as symbolic flows are concerned, only human agents and systems are relevant (devices have no symbolic capabilities of their own). On the receiving end of use cases (aka secondary actors), only systems are to be considered for supporting services.

Mapping Processes to Services (through Use Cases)

Hence, when requirements are expressed through use cases, and assuming they are to be realized (fully or partially) through services:

Persistency and functional units identified by business process would be mapped to messages and contracts.
Business processes would fit service policy.
Use case containers (aka systems) would be registered as service customers.

Alternatively, when requirements are set from users’ stories instead of use cases, persistency and functional units have to be elicited through stories, traced back to business processes, and consolidated into features. Those features will be mapped into system functionalities possibly, but not necessarily, implemented as services.

Mapping Processes to Services (through Users’ Stories)

Hence, while the mapping of business objects and logic respectively to messages and contracts will be similar with use cases and users’ stories, paths will differ for customers and policies:

Given that use cases deal explicitly with interactions at system boundaries, they represent a primary source of requirements for services’ customers and policy. Yet, as services are not supposed to be directly affected by interactions at systems boundaries, those elements would have to be consolidated across use cases.
Users’ stories for their part are told from a business process perspective that may take into account boundaries and actors but are not focused on them. Depending on the standpoint, it should be possible to define customers and policies requirements for services independently of the contingencies of local interactions.

In both cases, it would be necessary to factor out the non symbolic (aka non functional) part of requirements.

Non Functional Requirements: Quality of Service and System Boundaries

Non functional requirements are meant to set apart the constraints on systems’ resources and performances that could be dealt with independently of business contents. While some may target specific business applications, and others encompass a broader range, the aim is to separate business from architecture concerns and allocate the responsibilities (specification, development, and acceptance) accordingly.

Assuming an architecture of services aligned on capabilities, the first step would be to sort operational constraints:

Customers: constraints on usability, customization, response time, availability, …
Messages: constraints on scale, confidentiality, compliance with regulations, …
Contracts: constraints on scale, confidentiality, …
Policy: availability, reliability, maintenance, …
Endpoints: costs, maintenance, security, interoperability, …

BP2SOA_QoS — Non functional constraints may cut across services and capabilities

Since constraints may cut across services and capabilities, non functional requirements are not a given but the result of explicit decisions about:

Architecture level: should the constraint be dealt with locally (interfaces), at functional level (services), or at technical level (resources).
Services: when set at functional level, should the constraint be dealt with by business services (e.g domain or activity), or architecture ones (e.g authorization or orchestration).

The alignment of services with architecture capabilities will greatly enhance the traceability and rationality of those decisions.

A Simple Example

This example is based on the Purchase Order case published with the OMG specifications: http://www.omg.org/spec/SoaML/1.0/Beta2/PDF/

A simple purchase order process analyzed in terms of service customers, messages and entities (#), contracts, and policy (aka choreography)

External References

Microservices vs Serverless Architectures

Modeling Languages: Differences Matter

“Since words are only names for things, it would be more convenient for all men to carry about them such things as were necessary to express a particular business they are to discourse on.”

Jonathan Swift, Gulliver’s Travels

Objective

Modeling languages are meant to support the description of contexts and concerns from specific standpoints. And because those different perspectives have to be mapped against shared references, language must also provide constructs for ironing out variants and factoring out constants.

Digging or Ironing out differences (Erwitt Elliott) — Ironing out differences in features and behaviors (Erwitt Elliott)

Yet, while most languages generally agree on basic definitions of objects and behaviors, many distinctions are ignored or subject to controversial understanding; such shortcomings might be critical when architecture capabilities are concerned:

Actual entities and their symbolic counterpart.
Actual entities and their roles.
Business logic and business operations
External events and system time.

As those distinctions set the backbone of functional architectures, languages should be assessed according their ability to express them unambiguously using the least possible set of constructs.

Business objects vs Symbolic Surrogates

As far as symbolic systems are concerned, the primary purpose of models is to distinguish between targets and representations. Hence the first assessment yardstick: modeling languages must support a clear and unambiguous distinction between objects and behaviors of concern on one hand, symbolic system surrogates on the other hand.

A primer on representation: objects of concerns and surrogates

Yet, if objects and behaviors are to be consistently and continuously managed, modeling languages must also provide common constructs mapping identities and structures of actual entities to their symbolic counterpart.

Agents vs Roles

Given that systems are built to support business processes, modeling languages should enable a clear distinction between physical entities able to interact with systems on one hand, roles as defined by enterprise organization on the other hand.

In that case the mapping of actual entities to systems representations is not about identities but functionalities: a level of indirection has to be introduced between enterprise organization and system technology because the same roles can be played,simultaneously or successively, by people, devices, or other systems.

Business logic vs Business operations

Just like actual objects are not to be confused with their symbolic description, modeling languages must make a clear distinction between business logic and processes execution, the former defining how to process symbolic objects and flows, the latter with the coupling between process execution and changes in actual contexts. That distinction is of a particular importance if business and organizational decisions are to be made independently of supporting systems technology.

Pull vs Push Rule Management — Business logic and operational contingencies

Language constructs must also support the consolidation of functional and operational units, the former being defined by integrity constraints on symbolic flows and roles authorizations on objects and operations, the latter taking into account synchronization constraints between the state of actual contexts and the state of their system symbolic counterpart. And for that purpose languages must support the distinction between external and internal events.

External vs Internal Events

Paraphrasing Albert Einstein, time is what happens between events. As a corollary, external time is what happens between context events, and internal time is what happens between system ones. While both can coincide for single systems and locations, no such assumption should be made for distributed systems set in different locations. In that case modeling language should support a clear distinction between external events signalling changes set in actual locations, and internal events signalling changes affecting system surrogates.

Time scales can be design on purpose — Time scales are set by events and concerns

Along that perspective synchronization is to be achieved through the consolidation of time scales. For single locations that can be done using system clocks, across distributed locations the consolidation will entail dedicated time frames and mechanisms set in reference to some initial external event.

Conclusion: How to share differences across perspectives

Somewhat paradoxically, multiple modeling languages erase differences by paring down shared descriptions to some uniform lump that can be understood by all. Conversely, agreeing on a set of distinctions that should be supported by every language could provide an antidote to the Babel syndrome.

That approach can be especially effective for the alignment of enterprise and systems architectures as the four distinctions listed above are equally meaningful in both perspectives.

Thinking about Practices

A few preliminary words

A theory (aka model) is a symbolic description of contexts and concerns. A practice is a set of activities performed in actual contexts. While the latter may be governed by the former and the former developed from the latter, each should stand on its own merits whatever its debt to the other.

Good practice has no need to show off theory to hold sway (Demetre Chiparus)

Good practices hold sway without showing off theoretical subtext (Demetre Chiparus)

With regard to Software Engineering, theory and practice are often lumped together to be marketed as snake oil, with the unfortunate consequence of ruining their respective sways.

Software Engineering: from Requirements heads to Programs tails

While computer science deals with the automated processing of symbolic representations, software engineering uses it to develop applications that will support actual business processes; that may explain why software engineering is long on methods but rather short on theory.

Yet, since there is a requirements head (for business processes) to the programming tail (for automated processing), it would help to think about some rationale in between. Schools of thought can be summarily characterized as formal or procedural.

TheoPrati_0 — How to make program tails from requirements heads

Formal approaches try to extend the scope of computing theories to functional specifications; while they should be the option of choice, their scope is curtailed by the lack of structure and formalism when requirements are expressed in natural languages.

Procedural approaches deal with the difficulty of capturing users requirements by replacing theoretical assumptions about software artifacts with guidelines and best practices for modus operandi. The fault here is that the absence of standardized artifacts makes the outcomes unyielding and difficult to reuse.

ccc — Procedural (p), formal (f), and agile (a) approaches to software development.

Pros and cons of those approaches point to what should be looked for in software engineering:

As illustrated by Relational theory and State machines, formal specifications can support development practice providing requirements can be directly aligned with computing.
As illustrated by the ill-famed Waterfall, development practices should not be coerced into one-fits-all procedures if they are to accommodate contexts and tasks diversity.

Agile answers to that conundrum have been to focus on development practices without making theoretical assumptions about specifications. That left those development models halfway, making room for theoretical complements. That situation can be clarified using Scott Ambler’s 14 best practices of AMDD:

Active Stakeholder Participation / How to define a stakeholder ?
Architecture Envisioning / What concepts should be used to describe architectures and how to differentiate architecture levels ?
Document Continuously / What kind of documents should be produced and how should they relate to life-cycle ?
Document Late / How to time the production of documents with regard to life-cycle ?
Executable Specifications / What kind of requirements taxonomy should be used ?
Iteration Modeling / What kind of modeling paradigm should be used ?
Just Barely Good Enough (JBGE) artifacts / How to assess the granularity of specifications ?
Look Ahead Modeling / How to assess requirements complexity.
Model Storming / How to decide the depth of granularity to be explored and how to take architectural constraints into account ?
Multiple Models / Even within a single modeling paradigm, how to assess model effectiveness ?
Prioritized Requirements / How to translate users’ value into functional complexity when there is no one-to-one mapping ?
Requirements Envisioning / How to reformulate a lump of requirements into structured ones ?
Single Source Information / How to deal with features shared by multiple users’ stories ?
Test-Driven Design (TDD) / How to differentiate between business-facing and technology-facing tests ?

That would bring the best of two world, with practices inducing questions about the definition of development artifacts and activities, and theoretical answers being used to refine, assess and improve the practices.

Takes Two To Tango

Debates about the respective benefits of theory and practice are meaningless because theory and practice are the two faces of engineering: on one hand the effectiveness of practices depends on development models (aka theories), on the other hand development models are pointless if not validated by actual practices. Hence the benefits of thinking about agile practices.

Along that reasoning, some theoretical considerations appear to be of particular importance for good practice:

Enterprise architecture: how to define stakes and circumscribe organizational responsibilities.
Systems architecture: how to factor out shared architecture functionalities.
Products: how to distinguish between models and code.
Metrics: how to compare users’ value with development charge.
Release: how to arbitrage between quality and timing.

Such questionings have received some scrutiny from different horizons that may eventually point to a comprehensive and consistent understanding of software engineering artifacts.

External Links

Sifting through a Web of Things

The world is the totality of facts, not of things.

Ludwig Wittgenstein

Objective

At its inception, the young internet was all about sharing knowledge. Then, business concerns came to the web and the focus was downgraded to information. Now, exponential growth turns a surfeit of information into meaningless data, with the looming risk of web contents being once again downgraded. And the danger is compounded by the inroads of physical objects bidding for full netizenship and equal rights in the so-called “internet of things”.

Wang-Qingsong_iot — How to put words on a web of things (Wang-Qingsong)

As it happens, that double perspective coincides with two basic search mechanisms, one looking for identified items and the other for information contents. While semantic web approaches are meant to deal with the latter, it may be necessary to take one step further and to bring the problem (a web of things and meanings) and the solutions (search strategies) within an integrated perspective.

Down with the System Aristocrats

The so-called “internet second revolution” can be summarized as the end of privileged netizenship: down with the aristocracy of systems with their absolute lid on internet residency, within the new web everything should be entitled to have a voice.

cccc — Before and after the revolution: everything should have a say

But then, events are moving fast, suggesting behaviors unbecoming to the things that used to be. Hence the need of a reasoned classification of netizens based on their identification and communication capabilities:

Humans have inherent identities and can exchange symbolic and non symbolic data.
Systems don’t have inherent identities and can only exchange symbolic data.
Devices don’t have inherent identities and can only exchange non symbolic data.
Animals have inherent identities and can only exchange non symbolic data.

Along that perspective, speaking about the “internet of things” can be misleading because the primary transformation goes the other way: many systems initially embedded within appliances (e.g cell phones) have made their coming out by adding symbolic user interfaces, mutating from devices into fully fledged systems.

Physical Integration: The Meaning of Things

With embedded systems colonizing every nook and cranny of the world, the supposedly innate hierarchical governance of systems over objects is challenged as the latter calls for full internet citizenship. Those new requirements can be expressed in terms of architecture capabilities :

Anywhere (Where): objects must be localized independently of systems. That’s customary for physical objects (e.g Geo-localization), but may be more challenging for digital ones on they way across the net.
Anytime (When): behaviors must be synchronized over asynchronous communication channels. Existing mechanism used for actual processes (e.g Network Time protocol) may have to be set against modal logic if it is used for their representation.
Anybody (Who): while business systems don’t like anonymity and rely on their interfaces to secure access, things of the internet are to be identified whatever their interface (e.g RFID).
Anything (What): objects must be managed independently of their nature, symbolic or otherwise (e.g 3D printed objects).
Anyhow (How): contrary to business systems, processes don’t have to follow predefined scripts and versatility and non determinism are the rules of the game.

Taking a sortie in a restaurant for example: the actual event is associated to a reservation, car(s) and phone(s) are active objects geo-localized at a fixed place and possibly linked to diners, great wines can be authenticated directly by smartphone applications, phones are used for conversations and pictures, possibly for adding to reviews, friends in the neighborhood can be automatically informed of the sortie and invited to join.

A dinner on the Net: place (restaurant), event (sortie), active objects (car, phone), passive object (wine), message (picture), business objects (reviews, reservations), and social beholders (network friends).

As this simple example illustrates, the internet of things brings together dumb objects, smart systems, and knowledgeable documents. Navigating such a tangle will require more than the Semantic Web initiative because its purpose points in the opposite direction, back to the origin of the internet, namely how to extract knowledge from data and information.

Moreover, while most of those “things” fall under the governance of the traditional internet of systems, the primary factor of change comes from the exponential growth of smart physical things with systems of their own. When those systems are “embedded”, the representations they use are encapsulated and cannot be accessed directly as symbolic ones. In other words those agents are governed by hidden agendas inaccessible to search engines. That problem is illustrated a contrario (things are not services) by services oriented architectures whose one of primary responsibility is to support services discovery.

Semantic Integration: The Actuality of Meanings

The internet of things is supposed to provide a unified perspective on physical objects and symbolic representations, with the former taken as they come and instantly donned in some symbolic skin, and the latter boiled down to their documentary avatars (as Marshall McLuhan famously said, “the medium is the message”). Unfortunately, this goal is also a challenge because if physical objects can be uniformly enlisted across the web, that’s not the case for symbolic surrogates which are specific to social entities and managed by their respective systems accordingly.

With the Internet of Systems, social entities with common endeavors agree on shared symbolic representations and exchange the corresponding surrogates as managed by their systems. The Internet of Things for its part is meant to put an additional layer of meanings supposedly independent of those managed at systems level. As far as meanings are concerned, the latter is flat, the former is hierarchized.

The internet of things is supposed to level the meaning fields, reducing knowledge to common sense.

That goal raises two questions: (1) what belongs to the part governed by the internet of things and, (2) how is its flattened governance to be related to the structured one of the internet of systems.

A World of Phenomena

Contrary to what its name may suggest, the internet of things deals less with objects than with phenomena, the reason being that things must manifest themselves, or their existence be detected, before being identified, if and when it’s possible.

Things first appear on radar when some binary footprint can be associated to a signalling event. Then, if things are to go further, some information has to be extracted from captured data:

Coded data could be recognized by a system as an identification tag pointing to recorded known features and meanings, e.g a bar code on a book.
The whole thing could be turned into its digital equivalent, and vice versa, e.g a song or a picture.
Context and meanings could only be obtained by linking the captured data to representations already identified and understood, e.g a religious symbol.

How to put things in place — From data to information: how to add identity and meaning to things

Whereas things managed by existing systems already come with net identities with associated meaning, that’s not necessarily the case for digitized ones as they may or may not have been introduced as surrogates to be used as their real counterpart: if handshakes can stand as symbolic contract endorsements, pictures thereof cannot be used as contracts surrogates. Hence the necessary distinction between two categories of formal digitization:

Applied to symbolic objects (e.g a contract), formal digitization enables the direct processing of digital instances as if performed on actual ones, i.e with their conventional (i.e business) currency. While those objects have no counterpart (they exist simultaneously in both realms), such digitized objects have to bear an identification issued by a business entity, and that put them under the governance of standard (internet of systems) rules.
Applied to binary objects (e.g a fac-simile), formal digitization applies to digital instances that can be identified and authenticated on their own, independently of any symbolic counterpart. As a corollary, they are not meant to be managed or even modified and, as illustrated by the marketing of cultural contents (e.g music, movies, books …), their actual format may be irrelevant. Providing agreed upon de facto standards, binary objects epitomize internet things.

To conclude on footprint, the Internet of Things appears as a complement more than a substitute as it abides by the apparatus of the Internet of Systems for everything already under its governance, introducing new mechanisms only for the otherwise uncharted things set loose in outer reaches. Can the same conclusion hold for meanings ?

Organizational vs Social Meanings

As epitomized by handshakes and contracts, symbolic representations are all about how social behaviors are sanctioned.

Signals are physical events with open-ended interpretations — When not circumscribed within organizational boundaries, social behaviors are open to different interpretations.

In system-based networks representations and meanings are defined and governed by clearly identified organizations, corporate or otherwise. That’s not necessarily the case for networks populated by software agents performing unsupervised tasks.

The first generations of those internet robots (aka bots) were limited to automated tasks, performed on the account of their parent systems, to which they were due to report. Such neat hierarchical governance is being called into question by bots fired and forgotten by their maker, free of reporting duties, their life wholly governed by social events. That’s the case with the internet of things, with significant consequences for searches.

As noted above, the internet of things can consistently manage both system-defined identities and the new ones it introduces for things of its own. But, given a network job description, the same consolidation cannot be even considered for meanings: networks are supposed to be kept in complete ignorance of contents, and walls between addresses and knowledge management must tower well above the clouds. As a corollary, the overlapping of meanings is bound to grow with the expanse of things, and the increase will not be linear.

Contrary to identities, meanings usually overlap when things float free from systems' governance. — Contrary to identities, meanings usually overlap when things float free from systems’ governance.

That brings some light on the so-called “virtual world”, one made of representations detached from identified roots in the actual world. And there should be no misunderstanding: “actual” doesn’t refer to physical objects but to objects and behaviors sanctioned by social entities, as opposed to virtual, which includes the ones whose meaning cannot be neatly anchored to a social authority.

That makes searches in the web of things doubly challenging as they have to deal with both overlapping and shifting semantics.

A Taxonomy of Searches

Semantic searches (forms and pattern matching should be seen as a special case) can be initiated by any kind of textual input, key words or phrase. As searches, they should first be classified with regard to their purpose: finding some specific instance or collecting information about some topic.

Searches about instances are meant to provide references to:

Locations, addresses, …
Antiques, stamps,…
Books, magazines,…
Alumni, friends,…
Concerts, games, …
Cooking recipes, administrative procedures,…
Status of shipment, health monitoring, home surveillance …

Searches about categories are meant to provide information about:

Geography, …
Products marketing , …
Scholarly topics, market researches…
Customers relationships, …
Business events, …
Business rules, …
Business processes …

That taxonomy of searches is congruent with the critical divide between things and symbolic representations.

Things and Symbolic Representations

As noted above, searches can be heeded by references to identified objects, the form of digital objects (sound, visuals, or otherwise), or associations between symbolic representations. Considering that finding referenced objects is basically a indexing problem, and that pattern matching is a discipline of its own, the focus is to be put on the third case, namely searches driven by words (as opposed to identifiers and forms). From that standpoint searches are best understood in the broader semantic context of extensions and intensions , the former being the actual set of objects and phenomena, the latter a selected set of features shared by these instances.

A search can therefore be seen as an iterative process going back and forth between descriptions and occurrences or, more formally, between intentions and extensions. Depending on the request, iterations are made of four steps:

Given a description (intension) find the corresponding set of instances (extension); e.g “restaurants” > a list of restaurants.
Given an instance (extension), find a description (intension); e.g “Alberto’s Pizza” > “pizzerias”.
Extend or generalize a description to obtain a better match to request and context; e.g “pizzerias” > “Italian restaurants”.
Trim or refine instances to obtain a better match to request and context; e.g a list of restaurants > a list of restaurants in the Village.

Iterations are repeated until the outcome is deemed to satisfy the quality parameters.

The benefit of those distinctions is to introduce explicit decision points with regard to the reference models heeding the searches. Depending on purpose and context, such models could be:

Inclusive: can be applied to any kind of search.
Domain specific: can only be applied to circumscribed domains of knowledge. That’s the aim of the semantic web initiative and the Web Ontology Language (OWL).
Institutional: can only be applied within specific institutional or business organizations. They could be available to all or through services with restricted access and use.

From Meanings to Things, and back

The stunning performances of modern search engines comes from a combination of brawn and brains, the brainy part for grammars and statistics, the brawny one for running heuristics on gigantic repositories of linguistic practices and web researches. Moreover, those performances improve “naturally” with the accumulation of data pertaining to virtual events and behaviors. Nonetheless, search engines have grey or even blind spots, and there may be a downside to the accumulation of social data, as it may increase the gap between social and corporate knowledge, and consequently the coherence of outcomes.

Meanings can be inclusive, domain specific, or institutional

That can be illustrated by a search about Amedeo Modigliani:

A inclusive search for “Modigliani” will use heuristics to identify the artist (a). An organizational search for an homonym (e.g a bank customer) would be dealt with at enterprise level, possibly through an intranet (c).
A search for “Modigliani’s friends” may look for the artist’s Facebook friends if kept at the inclusive level (a1), or switch to a semantic context better suited to the artist (a2). The same outcome would have been obtained with a semantic search (b).
Searches about auction prices may be redirected or initiated directly, possibly subject to authorization (c).

External Links

Knowledge Based Model Transformation

Objective

Even when code is directly developed from requirements, software engineering can be seen as a process of transformation from a source model (e.g requirements) to a target one (e.g program). More generally, there will be intermediate artifacts with development combining automated operations with tasks involving some decision-making. Given the importance of knowledge management in decision-making, it should also be considered as a primary factor for model transformation.

ccccc — What has been transformed, added, abstracted ? (Velazquez-Picasso)

Clearing Taxonomy Thickets

Beside purpose, models transformation is often considered with regard to automation or abstraction level. Classifying transformations with regard to automation is at best redundant, inconsistent and confusing otherwise: on one side of the argument, assuming a “manual transformation” entails some “intelligent design”, it ensues that it cannot be automated; but, on the other side of the argument, if manual operations can be automated, the distinction between manual and automated is pointless. And in any case a critical aspect has been ignored, namely what is the added value of a transformation. Classifying transformations with regard to abstraction levels is also problematic because, as famously illustrated above, there are too many understandings of what those levels may be. Considering for instance analysis and design models; they are usually meant to deal respectively with system functionalities and implementation. But it doesn’t ensue that the former is more abstract than the latter because system functionalities may describe low-level physical devices and system designs may include high level abstract constructs.

A Taxonomy of Purposes

Depending on purpose, model transformation can be set in three main categories:

Translation: source and target models describe the same contents with different languages.
Rephrasing: improvement of the initial descriptions using the same language.
Processing: source and target models describe successive steps along the development process. While they may be expressed with different languages, the objective is to transform model contents.

Basicam2_Transfos — Translation, Rephrasing, Processing.

Should those different purposes be achieved using common mechanisms, or should they be supported by specific ones ?

A Taxonomy of Knowledge

As already noted, automated transformation is taken as a working assumption and the whole process is meant to be governed by rules. And if purpose is used as primary determinant, those rules should be classified with regard of the knowledge to be considered:

Syntactic knowledge would be needed to rephrase words and constructs without any change in content or language.
Thesaurus and semantics would be employed to translate contents into another language.
Stereotypes, Patterns, and engineering practices would be used to process source models into target ones.

Basicam2_TransfosKnow — Knowledge with regard to transformation purposes

On that basis, the next step is to consider the functional architecture of transformation, in particular with regard to supporting knowledge.

Procedural vs Declarative KM

Assuming transformation processes devoid of any decision-making, all relevant knowledge can be documented either directly in rules (procedural approach) or indirectly in models (declarative approach). To begin with, all approaches have to manage knowledge about languages principles (ML) and the modeling languages used for source (MLsrc) and target (MLtrg) models. Depending on language, linguistic knowledge can combine syntactic and semantic layers, or distinguish between them.

uml_metas — Meta-models can be structured along linguistic layers or combine syntax and semantics.

Given exponential growth of complexity, procedural approaches are better suited for the processing of unstructured knowledge, while declarative ones are the option of choice when knowledge is organized along layers. When applied to model transformation, procedural solutions use tables (MLT) to document the mappings between language constructs, and programs to define and combine transformation rules. Such approaches work well enough for detailed rules targeting specific domains, but fall short when some kind of generalization is needed; moreover, in the absence of distinction at language level (see UML and users’ concerns), any contents differentiation in source models (Msrc) will be blurred in target ones (Mtrg) .

Narratives_PvsD — Transformation Policies: Procedural vs Declarative

And filtering out noise appears to be the main challenge of model processing, especially when even localized taints or ambiguities in source models may contaminate the whole of target ones; hence the benefit of setting apart layers of source models according their nature and reliability. That can be achieved with the modularity of declarative solutions, with transformations carried out separately according the nature of knowledge: syntactic, semantic, or engineering. While declarative solutions also rely on rules, those rules can be managed according to:

Purpose: rephrasing, translation, or development.
Knowledge level: rules about facts, categories, or reasoning.
Confidence level: formal, heuristic, fuzzy, modal.

That makes declarative solutions more easily assessed and therefore improved.

Assessment

Advances in model transformation call for sound assessment, and that can be done with regard to usual knowledge management capabilities:

Primitive operations: how to create, read, update, or delete (CRUD) transformations.
Transparency: how to reason about transformations.
Traceability: how to record the history of performed transformations and chart the effects of hypothetical ones.
Reliability: built-in consistency checks, testability.
Robustness: how to deal with incomplete, noisy, or inconsistent models.
Modularity, reusability: how to customize, combine, or generalize transformations.

In principle declarative approaches are generally seen as superior to procedural ones due to the benefits of a clear-cut separation between the different domains of knowledge on one hand, and the programs performing the transformations on the other hand:

The complexity of transformations increases exponentially with the number of rules, independently of the corpus of knowledge.
The ensuing spaghetti syndrome rapidly downgrades transparency, traceability, and reliability of rules which, in procedural approaches, are the only gate to knowledge itself.
Adding specific rules in order to deal with incomplete or inconsistent models significantly degrades modularity and reusability, with the leveraging impact of new rules having to be added for every variant.

In practice the flaws of procedural approaches can be partially curbed providing that:

Rules apply to a closely controlled domain.
Models are built with complete, unambiguous and consistent languages.

Both conditions can be met with domain specific languages (DSLs) and document based ones (e.g XML) targeting design and program models within well-defined business domains. Compared with the diversity and specificity of procedural solutions, the extent of declarative ones have been checked by their more theoretically demanding requirements, principally met by relational models. Yet, the Model Driven Architecture (MDA) distinction between computation independent (CIMs), platform independent (PIMs), and platform specific (PSMs) models could provide the basis for a generalization of knowledge-based model transformation.

Focus: Rules & Architecture

Preamble

Rules can be seen as the glue holding together business, organization, and systems, and that may be a challenge for enterprise architects when changes are to be managed according to different concerns and different time-scales. Hence the importance of untangling rules upfront when requirements are captured and analysed.

How to outline the architectural footprint of rules (John Devlin)

Primary Taxonomy

As far as enterprise architecture is concerned, rules can be about:

Business and regulatory environments.
Enterprise objectives and organization.
Business processes and supporting systems.

That classification can be mapped to a logical one:

Rules set in business or regulatory environments are said to be deontic as they are to be met independently of enterprise governance. They must be enforced by symbolic representations if enterprise systems are to be aligned with environments.
Rules associated with objectives, organization, processes or systems are said to be alethic (aka modal) as they refer to possible, necessary or contingent conditions as defined by enterprise governance. They are to be directly applied to symbolic representations.

Whereas both are to be supported by systems, the loci will differ: system boundaries for deontic rules (coupling between environment and systems), system components for alethic ones (continuity and consistency of symbolic representations). Given the architectural consequences, rules should be organized depending on triggering (actual or symbolic) and scope (environment or enterprise):

Actual deontic rules are triggered by actual external events that must be processed synchronously.
Symbolic deontic rules are triggered by external events that may be processed asynchronously.
Actual alethic rules are triggered by business processes and must be processed synchronously.
Symbolic alethic rules are triggered by business processes and can be processed asynchronously.

Rules should be classified upfront with regard to triggering (actual or symbolic) and scope (environment or enterprise)

Footprint

The footprint of a rule is made of the categories of facts to be considered (aka rule domain), and categories of facts possibly affected (aka rule co-domain).

As far as systems are concerned, the first thing to do is to distinguish between actual contexts and symbolic representations. A naive understanding would assume rules to belong to either actual or symbolic realms. Given that the objective of modeling is to decide how the former should be represented by the latter, some grey imprints to be expected and dealt with using three categories of rules, one for each realm and the third set across the divide:

Rules targeting actual contexts. They can be checked through sensors or applied by actuators. Since rules enforcement cannot be guaranteed on non symbolic artifacts, some rules will have to monitor infringements and proscribed configurations. Example: “Cars should be checked on return from each rental, and on transfer between branches.”
Rules targeting symbolic representations. Their enforcement is supposedly under the full control of system components. Example: “A car with accumulated mileage greater than 5000 since its last service must be scheduled for service.”
Rules defining how changes in actual contexts should impact symbolic representations: what is to be considered, where it should be observed, when it should be recorded, how it should be processed, who is to be authorized. Example: ” Customers’ requests at branches for cars of particular models should be consolidated every day.”

That analysis should be carried out as soon as possible because rules set on the divide will determine the impact of requirements on architecture capabilities.

Semantics and Syntax

Rules footprints are charted by domains (what is to be considered) and co-domains (what is to be affected). Since footprints are defined by requirements semantics the outcome shouldn’t be contingent on formats.

From an architecture perspective the critical distinction is between homogeneous and heterogeneous rules, the former with footprint on the same side of the actual/symbolic divide, the latter with a footprint set across.

RulesCapabsH — Homogeneous vs Heterogeneous footprints

Contrary to footprints, the shape given to rules (aka format, aka syntax,) is to affect their execution. Assuming homogeneous footprints, four basic blueprints are available depending on the way domains (categories of instances to be valued) and co-domains (categories of instances possibly affected) are combined:

Partitions are expressions used to classify facts of a given category.
Constraints (backward rules) are conditions to be checked on facts: [domain].
Pull rules (static forward) are expressions used to modify facts: co-domain = [domain].
Push rules (dynamic forward) are expressions used to trigger the modification of facts: [domain] > co-domain.

Anchors & Granularity

In principle, rules targeting different categories of facts are nothing more than well-formed expressions combining homogeneous ones. In practice, because they mix different kinds of artifacts, the way they are built is bound to significantly bear on architecture capabilities.

Systems are tied to environments by anchors, i.e objects and processes whose identity and consistency must be maintained during their life-cycle. Rules should therefore be attached to anchors’ facets as to obtain as fine-grained footprints as possible:

Features: domain and co-domain are limited to attributes or operations.
Object footprint: domain and co-domain are set within the limit of a uniquely identified instance (#), including composites and aggregates.
Connections: domain and co-domain are set by the connections between instances identified independently.
Collections: domain and co-domain are set between sets of instances and individuals ones, including subsets defined by partitions.
Containers: domain and co-domain are set for whole systems.

While minimizing the scope of simple (homogeneous) rules is arguably a straightforward routine, alternative options may have to be considered for the transformation of joint (heterogeneous) statements, e.g when rules about functional dependencies may be attached either to (1) persistent representation of objects and associations or, (2) business applications.

Heterogeneous (joint) Footprints

Footprints set across different categories will usually leave room for alternative modeling options affecting the way rules will be executed, and therefore bearing differently on architecture capabilities.

Basic alternatives can be defined according to requirements taxonomy:

Business requirements: rules set at enterprise level that can be managed independently of the architecture.
System functionalities: rules set at system level whose support depends on architecture capabilities.
Quality of service: rules set at system level whose support depends on functional and technical architectures.
Operational constraints: rules set at platform level whose support depends on technical capabilities.

Rules do not necessarily fit into clear requirements taxonomy

While that classification may work fine for homogeneous rules (a), it may fall short for mixed ones, functional (b) or not (c). For instance:

“Gold Customers with requests for cars of particular models should be given an immediate answer.”
“Technical problems affecting security on checked cars must be notified immediately.”

As requirements go, rules interweaving business, functional, and non functional requirements are routine and their transformation should reflect how priorities are to be sorted out.

Moreover, if rule refactoring is to be carried out, there will be more than syntax and semantics to consider because almost every requirement can be expressed as a rule, often with alternative options. As a corollary, the modeling policies governing the making of rules should be set explicitly.

Sorting Out Mixed Rules

Taking into account that functional requirements describe how systems are meant to support business processes, some rules are bound to mix functional and business concerns. When that’s the case, preferences will have to be set with regard to:

Events vs Data: should system behavior be driven by changes in business context (as signaled by events from users, devices, or other systems), or by changes in symbolic representations.
Activities vs Data: should system behavior be governed by planned activities, or by the states of business objects.
Activities vs Events: should system behavior be governed by planned activities, or driven by changes in business context.

Taking the Gold Customer example, a logical rule (right) is not meant to affect the architecture, but expressed at control level (left) it points to communication capabilities.

RulesBuFu1 — How to express Gold customers’ rule may or may not point to communication capabilities.

The same questions arise for rules mixing functional requirements, quality of service, and operational constraints, e.g:

How to apportion response time constraints between endpoints, communication architecture, and applications.
How to apportion reliability constraints between application software and resources at location .
How to apportion confidentiality constraints between entry points, communication architecture, and locations.

Those questions often arise with non functional requirements and entail broader architectural issues and the divide between enterprise wide and domain specific capabilities.

External Links

Modernization & The Archaeology of Software

The past is not dead. In fact, it’s not even past. –

William Faulkner

Objective

Retrieving legacy code has something to do with archaeology as both try to retrieve undocumented artifacts and understand their initial context and purpose. The fact that legacy code is still well alive and kicking may help to chart their structures and behaviors, but it may also confuse the rationale of initial designs.

ccc — Legacy Artifact: Was the bowl intended for turkeys ? at thanksgiving ? Why the worm ?

Hence the importance of traceability and the benefits of a knowledge based approach to modernization organized along architecture layers (enterprise, systems, platforms), and processes (business, engineering, supporting services).

Model Driven Modernization

Assuming that legacy artifacts under consideration are still operational (otherwise re-engineering would be pointless), modernization will have to:

Pinpoint the deployed components under consideration (a).
Identify the application context of their execution (b).
Chart their footprint in business processes (c).
Define the operational objectives of their modernization (d).
Sketch the conditions of their (re)engineering (e) and the possible integration in the existing functional architecture (f).
Plan the re-engineering project (g).

RefactorModern_Roadmap — Modernization Road Map

Those objectives will usually not be achieved in a big bang, wholly and instantly, but progressively by combining increments from all perspectives. Since the different outcomes will have to be managed across organizational units along multiple engineering processes, modernization would clearly benefit from a model based approach, as illustrated by MDA modeling layers:

Platform specific models (PSMs) should be used for collecting legacy artifacts and mapping them to their re-engineered counterparts.
Since platform independent models (PIMs) are meant to describe system functionalities independently of implementations, they should be used to consolidate the functionalities of legacy and re-engineered artifacts.
Since computation independent models (CIMs) are meant to describe business processes independently of supporting systems, they should be used to reinstate, document, and validate re-engineered artifacts within their business context.

RefactorModern_Layers — Model Driven Modernization

Corresponding phases can be expressed using the archaeology metaphor: field survey and collection (>PSMs), analysis (PSMs/PIMs), and reconstruction (CIMs/PIMs).

Field Survey

The objective of a field survey is to circumscribe the footprint of the modernization and collect artifacts under consideration:

Given targeted business objects or activities, the first step is to collect information about locations, distribution and execution dependencies.
Sites can then be searched and executable files translated into source ones whose structure and dependencies can be documented.
The role of legacy software can then be defined with regard to the application landscape .

RefactorModern_Field — Analysis (with regard to presentation, control, persistency, and services)

It must be noted that field survey and collection deal with the identification and restoration of legacy objects without analyzing their contents.

Analysis

The aim of analysis is to characterize legacy components, first with regard to their architectural features, then with regard to functionalities. Basic architectural features take into account components’ sharing and life-cycle.

The analysis of functionalities can be achieved locally or at architecture level:

Local analysis (a) directly map re-factored applications to specific business requirements, by-passing functional architecture. That’s the case when targeted applications can be isolated, e.g by wrapping legacy code.
Global analysis (b) consolidate newly supported applications with existing ones within functional architecture, possibly with new functionalities.

RefactorModern_Analysis — Analysis (with regard to presentation, control, persistency, and services)

It must be noted that the analysis of legacy components, even when carried out at functional architecture level, takes business processes as they are.

Reconstruction

The aim of reconstruction is to set legacy refactoring within the context of enterprise architecture. That should be done from operational and business perspectives:

As the primary rationale of modernization is to deal with operational problems or bottlenecks, its benefits should be fully capitalized at enterprise level.
Re-factored applications usually make room for improvements of users’ experience; that may bring about further changes in organization and business processes.

RefactorModern_Reconstruct — Reconstruction

Hence, modernization is not complete until potential benefits of re-factored applications are considered, for business processes as well as for functional architecture.

From Workshops to Workflow

As noted above, modernization can seldom be achieved in a big bang and should be planned as a model based engineering process. Taking a leaf from the MDA book, such a process would be organized across four workshops:

Technical architecture (deployment models): that’s where legacy components are collected, sorted, and documented.
Software architecture (platform specific models): where legacy components are put in local context.
Functional architecture (platform independent models): where legacy components are put in shared context.
Enterprise architecture (computation independent models): where legacy components are put into organizational context.

RefactorModern_Workshops — Modernization & MBSE Workshops

Those workshops would be used to manage the outcomes of the modernization workflow:

Collect and organize legacy code; translate into source files.
Document legacy components.
Build PSMs according basic architecture functional patterns.
Map to PIMs of system functional architecture.
Consolidate enterprise architecture.

RefactorModern_Engineer — Modernization Workflow

With the relevant workflows defined in terms of model-based systems engineering, modernization can be integrated with enterprise architecture.

Enterprise Governance & Knowledge

Knowledgeable Processes

While turf wars may play a part, the debate about Enterprise and Systems governance is rooted in a more serious argument, namely, how the divide between enterprise and systems architectures may affect decision-making.

Informed Decision_making (Eleanor Antin)

The answer to that question can be boiled down to four guidelines respectively for capabilities, functionalities, visibility, and uncertainty.

Architecture Capabilities

From an architecture perspective, enterprises are made of human agents, devices, and symbolic (aka information) systems. From a business perspective, processes combine three kinds of tasks:

Authority: deciding how to perform processes and make commitments in the name of the enterprise. That can only be done by human agents, individually or collectively.
Execution: processing physical or symbolic flows between the enterprise and its context. Any of those can be done by human agents, individually or collectively, or devices and software systems subject to compatibility qualifications.
Control: recording and checking the actual effects of processes execution. Any of those can be done by human agents, individually or collectively, some by software systems subject to qualifications, and none by devices.

Hence, and whatever the solutions, the divide between enterprise and systems will have to be aligned on those constraints:

Platforms and technology affects operational concerns, i.e physical accesses to systems and the where and when of processes execution.
Enterprise organization determines the way business is conducted: who is authorized to what (business objects), and how (business logic).
System functionalities sets the part played by systems in support of business processes.

That gives the first guideline of systems governance:

Guideline #1 (capabilities): Objectives and roles must be set at enterprise level, technical constraints about deployment and access must be defined at platform level, and functional architecture must be designed as to get the maximum of the former subject to the latter’s constraints.

Informed Decisions: The Will to Know

At its core, enterprise governance is about decision-making and on that basis the purpose of systems is to feed processes with the relevant information so that agents can be put it to use as knowledge.

Those flows can be neatly described by crossing the origin of data (enterprise, systems, platforms) with the processes using the information (business, software engineering, services management):

Information processing begins with data, which is no more than registered facts: texts, numbers, sounds, visuals, etc. Those facts are collected by systems through the execution of business, engineering, and servicing processes; they reflect the state of business contexts, enterprise, and platforms.
Data becomes information when comprehensively and consistently anchored to identified constituents (objects, activities, events,…) of contexts, organization, and resources.
Information becomes knowledge when put to use by agents with regard to their purpose: business, engineering, services.

ccc — Information processing: from data to knowledge and back

Along that perspective, capabilities can be further refined with regard to decision-making.

Starting with business logic one should factor out decision points and associated information. That will determine the structure of symbolic representations and functional units.
Then, one may derive decision-making roles, together with implicit authorizations and access constraints. That will determine the structure of I/O flows and the logic of interactions.
Finally, the functional architecture will have to take into account synchronization and deployment constraints on events notification to and from processes.

Who should know What and When — Who should know What, Where, and When

That can be condensed into the second guideline of system governance:

Guidelines #2 (functionalities): With regard to enterprise governance, the role of systems is to collect data and process it into information organized along enterprise concerns and objectives, enabling decision makers to select and pull relevant information and translate it into knowledge.

Qualified Information: The Veils of Ignorance

Ideally, decision-making should be neatly organized with regard to contexts and concerns:

Contexts are set by architecture layers: enterprise organization, system functionalities, platforms technology.
Concerns are best identified through processes: business, engineering, or supporting services.

Qualified Information Flows across Architectures and Processes

Actually, decisions scopes overlap and their outcomes are interwoven.

While distinctions with regard to contexts are supposedly built-in at the source (enterprise, systems, platforms), that’s not the case for concerns whose distinction usually calls for reasoned choices supported by different layers of governance:

Assets: shared decisions whose outcome bears upon several business domains and cycles. Those decisions may affect all architecture layers: enterprise (organization), systems (services), or platforms (purchased software packages).
Users’ Value: streamlined decisions governed by well identified business units providing for straight dependencies from enterprise (business requirements), to systems (functional requirements) and platforms (users’ entry points).
Non functional: shared decisions about scale and performances affecting users’ experience (organization), engineering (technical requirements), or resources (operational requirements).

Separation of Concerns and Requirements Taxonomy — Qualified Information and Decision Making

As epitomized by non functional requirements, those layers of governance don’t necessarily coincide with the distinction between business, engineering, and servicing concerns. Yet, one should expect the nature of decisions be set prior the actual decision-making, and decision makers be presented with only the relevant information; for instance:

Functional requirements should be decided given business requirements and services architecture.
Scalability (operational requirements) should be decided with regard to enterprise’s objectives and organization.

Hence the third guideline of system governance:

Guideline #3 (visibility): Systems must feed processes with qualified information according to contexts (business, organization, platforms) and governance level (assets, user’s value, operations) of decision makers.

Risks & Opportunities: Mining Beyond Visibility

Long secure behind organizational and technical fences, enterprises must now navigate through open digitized business environments and markets. For business processes it means a seamless integration with supporting applications; for corporate governance it means keeping track of risks and opportunities in changing business contexts while assessing the capability of organizations and systems to cope, adjust, and improve.

On one hand risks and opportunities take root beyond the horizon and are not supposed to square with established information models; but on the other hand deep learning technologies is revolutionizing data analytics. purpose ontologies could be used to bring architectures and environments modeling into a common paradigm.

Hence the fourth guideline of system governance:

Guideline #4 (uncertainty): assuming that business edge is built on unqualified information about risks or opportunities, explicit and implicit knowledge should be processed within shared conceptual frames; that can be achieved with ontologies.

Tests in Driving Seats

Objective

Contrary to its manufacturing cousin, a long time devotee of preventive policies, software engineering is still ambivalent regarding the benefits of integrating quality management with development itself. That certainly should raise some questions, as one would expect the quality of symbolic artifacts to be much easier to manage than the one of their physical counterparts, if for no other reason than the former has to check symbolic outcomes against symbolic specifications while the latter must also to overcome the contingencies of non symbolic artifacts.

Thanks to agile approaches, lessons from manufacturing are progressively learned, with lean and just-in-time principles making tentative inroads into software engineering. Taking advantage of the homogeneity of symbolic development flows, agile methods have forsaken phased processes in favor of iterative ones, making a priority of continuous and value driven deliveries to business users. Instead of predefined sequences of dedicated tasks, products are developed through iterations regrouping definition, building, and acceptance into the same cycles. That push differentiated documentation and models on back seats and may also introduce a new paradigm by putting tests on driving ones.

From Phased to Iterative Tests Management

Traditional (aka phased) processes follow a corrective strategy: tests are performed according a Last In First Out (LIFO) framework, for components (unit tests), system (integration), and business (acceptance). As a consequence, faults in functional architecture risk being identified after components completion, and flaws in organization and business processes may not emerge before the integration of system functionalities. In other words, the faults with the more wide-ranging consequences may be the last to be detected.

Phased and Iterative approaches to tests

Iterative approaches follow a preemptive strategy: the sooner artifacts are tested, the better. The downside is that without differentiated and phased objectives, there is a question mark on the kind of specifications against which software products are to be tested; likewise, the question is how results are to be managed across iteration cycles, especially if changing requirements are to be taken into account.

Looking for answers, one should first consider how requirements taxonomy can support tests management.

Requirements Taxonomy and Tests Management

Whatever the methods or forms (users’ stories, use case, functional specifications, etc), requirements are meant to describe what is expected from systems, and as such they have two main purposes: (1) to serve as a reference for architects and engineers in software design and (2) to serve as a reference for tests and acceptance.

With regard to those purposes, phased development models have been providing clearly defined steps (e.g requirements, analysis, design, implementation) and corresponding responsibilities. But when iterative cycles are applied to progressively refined requirements, those “facilities” are no longer available. Nonetheless, since tests and acceptance are still to be performed, a requirements taxonomy may replace phased steps as a testing framework.

Taxonomies being built on purpose, one supporting iterative tests should consider two criteria, one driven by targeted contents, the other by modus operandi:

With regard to contents, requirements must be classified depending on who’s to decide: business and functional requirements are driven by users’ value and directly contribute to business experience; non functional requirements are driven by technical considerations. Overlapping concerns are usually regrouped as quality of service.

ccc — Requirements with regard to Acceptance.

That distinction between business and architecture driven requirements is at the root of portfolio management: projects with specific business stakeholders are best developed with the agile development model, architecture driven projects set across business domains may call for phased schemes.

That requirements taxonomy can be directly used to build its testing counterpart. As developed by D. Leffingwell (see selected readings), tests should also be classified with regard to their modus operandi, the distinction being between those that can be performed continuously along development iterations and those that are only relevant once products are set within their technical or business contexts. As it happens, those requirements and tests classifications are congruent:

Units and component tests (Q1) cover technical requirements and can be performed on development artifacts independently of their functionalities.
Functional tests (Q2) deal with system functionalities as expressed by users (e.g with stories or use cases), independently of operational or technical considerations.
System acceptance tests (Q3) verify that those functionalities, when performed at enterprise level, effectively support business processes.
System qualities tests (Q4) verify that those functionalities, when performed at enterprise level, are supported by architecture capabilities.

Tests Matrix for target and MO (adapted from D. Leffingwell).

Besides the specific use of each criterion in deciding who’s to handle tests, and when, combining criteria brings additional answers regarding automation: product acceptance should be performed manually at business level, preferably by tools at system level; tests performed along development iterations can be fully automated for units and components (black-box), but only partially for functionalities (white-box).

That tests classification can be used to distinguish between phased and iterative tests: the organization of tests targeting products and systems from business (Q3) or technology (Q4) perspectives is clearly not supposed to be affected by development models, phased or iterative, even if resources used during development may be reused. That’s not the case for the organization of the tests targeting functionalities (Q2) or components (Q1).

Iterative Tests

Contrary to tests aiming at products and systems (Q3 and Q4), those performed on development artifacts cannot be set on fixed and well-defined specifications: being managed within iteration cycles they must deal with moving targets.

Unit and components tests (Q1) are white-box operations meant to verify the implementation of functionalities; as a consequence:

They can be performed iteratively on software increments.
They must take into account technical requirements.
They must be aligned on the implementation of tested functionalities.

Hence, if unit and component tests are to be performed iteratively, (1) they must be set against features and, (2) functional tests must be properly documented and available for reuse.

Functional tests (Q2) are black-box operations meant to validate system behavior with regard to users’ expectations; as a consequence:

They can be performed iteratively on software increments.
They don’t have to take into account technical requirements.
They must be aligned on business requirements (e.g users’ stories or use cases).

Assuming (see previous post) a set of stories (a,b,c,d) identified by alternative paths built from features (f1…5), functional tests (Q2) are to be defined and performed for each story, and then reused to test the implementation of associated features (Q1).

At that point two questions must be answered:

Given that stories can be changed, expanded or refined along development iterations, how to manage the association between requirements and functional tests.
Given that backlogs can be rearranged along development cycles according to changing priorities, how to update tests, manage traceability, and prevent regression.

With model-driven approaches no longer available, one should consider a mirror alternative, namely test-driven development.

Tests Driven Development

Test driven development can be seen as a mirror image of model driven development, a somewhat logical consequence considering the limited role of models in agile approaches.

The core of agile principles is to put the definition, building and acceptance of software products under shared ownership, direct collaboration, and collective responsibility:

Shared ownership: a project team groups users and developers and its first objective is to consolidate their respective concerns.
Direct collaboration: decisions are taken by team members, without any organizational mediation or external interference.
Collective responsibility: decisions about stories, priorities and refinements are negotiated between team members from both sides of the business/system (or users/developers) divide.

Assuming those principles are effectively put to work, there seems to be little room for organized and persistent documentation, as users’ stories are meant to be developed, and products released, in continuity, and changes introduced as new stories.

With such lean and just-in-time processes, documentation, if any, is by nature transient, falling short as a support of test plans and results, even when problems and corrections are formulated as stories and managed through backlogs. In such circumstances, without specifications or models available as development handrails, could that be achieved by tests ?

ccc — Given the ephemeral nature of users’ stories, functional tests should take the lead.

To begin with, users’ stories have to be reconsidered. The distinction between functional tests on one hand, unit and component tests on the other hand, reflects the divide between business and technical concerns. While those concerns may be mixed in users’ stories, they are progressively set apart along iteration cycles. It means that users’ stories are, by nature, transitory, and as a consequence cannot be used to support tests management.

The case for features is different. While they cannot be fully defined up-front, features are not transient: being shared by different stories and bound to system functionalities they are supposed to provide some continuity. Likewise, notwithstanding their changing contents, users’ stories should be soundly identified by solution paths across problems space.

Paths and Features can be identified consistently along iteration cycles.

That can provide a stable framework supporting the management of development tests:

Unit tests are specified from crosses between solution paths (described by stories or scenarii) and features.
Functional tests are defined by solution paths and built from unit tests associated to the corresponding features.
Component tests are defined by features and built by the consolidation of unit tests defined for each targeted feature according to technical constraints.

The margins support continuous and consistent identification of functional and component tests whose contents can be extended or updated through changes made to unit tests.

One step further, and tests can even be used to drive iteration cycles: once features and solution paths soundly identified, there is no need to swell backlogs with detailed stories whose shelf life will be limited. Instead, development processes would get leaner if extensions and refinements could be directly expressed as unit tests.

System Quality and Acceptance Tests

Contrary to development tests which are applied iteratively to programs, system tests are applied to released products and must take into account requirements that cannot be directly or uniquely attached to users stories, either because they cannot be expressed from a business perspective or because they are shared concerns and best described as features. Tests for those requirements will be consolidated with development ones into system quality and acceptance tests:

System Quality Tests deal with performances and resources from the system management perspective. As such they will combine component and functional tests in operational configurations without taking into account their business contents.
System Acceptance Tests deal with the quality of service from the business process perspective. As such they will perform functional tests in operational configurations taking into account business contents and users’ experience.

Development Tests are to be consolidated into Product and System Acceptance Tests.

Requirements set too early and quality checks performed too late are at the root of phased processes predicaments, and that can be fixed with a two-pronged policy: a preemptive policy based upon a requirements taxonomy organizing problem spaces according concerns business value, system functionalities, components designs, platforms configuration; a corrective policy driven by the exploration of solution paths, with developments and releases driven by quality concerns.

Tests & Framework

Insofar as large and complex enterprise architectures are concerned, it’s safe to assume that different development models (agile or phased) and tests policies (unit, system, acceptance, …) will have to be cohabit, and that would not be possible without an architecture framework:

Development or unit tests are defined at platform level and applied to software components.
Integration or system tests are defined at system level and built from tested components.
Acceptance tests are defined at enterprise level and built from tested functionalities.

Pentaheds-tests — Tests should be aligned on architecture layers

On a broader perspective such a framework is to provide the foundation of enterprise architecture workflows.

Making Heads or Tails

Wrong Spelling, Right Concepts

Architectures & Assets Reuse

Languages & Model Purposes

Enterprise Architecture & Engineering Processes

Postscript

Further Reading

External Links

Objective

A Governing Dilemma

Separation of Concerns

Services & Capabilities

Anchoring Business Requirements to Supporting Systems

Filling out Functional Requirements

Non Functional Requirements: Quality of Service and System Boundaries

A Simple Example

Further Reading

External References

Objective

Business objects vs Symbolic Surrogates

Agents vs Roles

Business logic vs Business operations

External vs Internal Events

Conclusion: How to share differences across perspectives

Further reading

A few preliminary words

Software Engineering: from Requirements heads to Programs tails

Takes Two To Tango

Further Reading

External Links

Objective

Down with the System Aristocrats

Physical Integration: The Meaning of Things

Semantic Integration: The Actuality of Meanings

A World of Phenomena

Organizational vs Social Meanings

A Taxonomy of Searches

Things and Symbolic Representations

From Meanings to Things, and back

Further Reading

External Links

Objective

Clearing Taxonomy Thickets

A Taxonomy of Purposes

A Taxonomy of Knowledge

Procedural vs Declarative KM

Assessment

Further Readings

Kaleidoscope Series

Other Caminao References

External references

Preamble

Primary Taxonomy

Footprint

Semantics and Syntax

Anchors & Granularity

Heterogeneous (joint) Footprints

Sorting Out Mixed Rules

Further Reading

External Links

The past is not dead. In fact, it’s not even past. –

William Faulkner

Objective

Model Driven Modernization

Field Survey

Analysis

Reconstruction

From Workshops to Workflow

Further Reading

Knowledgeable Processes

Architecture Capabilities

Informed Decisions: The Will to Know

Qualified Information: The Veils of Ignorance

Risks & Opportunities: Mining Beyond Visibility

Further Reading

Objective

From Phased to Iterative Tests Management

Requirements Taxonomy and Tests Management

Iterative Tests

Tests Driven Development