Time and again new recruits will ask about the role of business analysts. Considering that such a question is seldom heard from software engineers, are BAs more curious about their job, or are they standing on more tentative grounds ? If that’s the case agility would help them to flip-flop between business quicksands to systems hard rocks.
How to make sense of business wonders (Hieronymus Bosch)
Holding the fort vs scouting outskirts
Systems architects and software engineers may have to meet esoteric business requirements, but their responsibility is first and foremost to guarantee the functional and economic sustainability of systems. On that account they are given licence to build solid walls and secure gateways, and to enforce their own languages and rules upon well vetted parties.
Business analysts don’t get such a free hand: while being straitened by software engineers constructs and constraints, their primary undertaking is to explore business wilds, reconnoitre competitors, trace new tracks, and learn the dialects of any nicknamed natives ready to trade.
No wonder the qualms of new business analysts.
Great businesses make their own rules
The best rules in business are the ones still unbeknownst, as success is most often brought by disruptive initiatives taking advantage of previously undiscovered opportunities. It ensues that at its core, BAs’ job description is to relentlessly look across the frontier for still uncharted businesses, and bring them back to the digitized world of shipshape business domains and processes.
For that purpose BAs will have to juggle with the fuzzy idiosyncrasies of new business openings until they can be aligned with the functionalities of “legacy” systems.
BA’s Agility
While usually presented as a software engineering hallmark, agility may be equally useful for business analysts as they have to balance two crossing perspectives:
Analysis: sorting detailed activities into business processes.
Synthesis: factoring out business functions and mapping them to systems capabilities.
That could be a challenging achievement if carried out sequentially: crossing back and forth between changing scope and steady capabilities could generate unsettling alternatives and unbounded complexity.
The agile development model is meant to tackle the difficulties through iterations and collaboration without being too specific about the kind of agility required from business analysts and software engineers.
Yet the apparent symmetry between the parties may be misleading: whereas software engineers don’t have (and shouldn’t even try) to second guess business analysts, business analysts shouldn’t forget that at the end of the day business expectations, however exotic or esoteric, will have to feed very conformist logical beasts.
Open-plan offices and social networks are often seen as significant factors of collaboration and innovation, breeding and nurturing the creativity of knowledge workers, weaving their ideas into webs of truths, and molding their minds into some collective intelligence.
Trust & Communication (Juan Munoz)
Yet, as creativity comes with agility, knowledge workflows should give brains enough breathing space lest they get more pressure than pasture.
Collaboration & Thinking Flows
Collaboration is a means to an end. To be of any use exchanges have to be fed with renewed ideas and assumptions, triggering arguments and adjustments, and opening new perspectives. If not they may burn themselves out with hollow considerations blurring clues and expectations, clogging the channels, and finally stemming the thinking flows.
Taking example from lean manufacturing, the first objective should be to streamline knowledge workflows as to eliminate swirling pools of squabbles, drain stagnant puddles of stale thoughts, and gear collaboration to flowing knowledge streams. As illustrated by flood irrigation, the first step is to identify basin levels.
Dunbar Numbers & Collaboration Basins
Studying the grooming habits of social primates, psychologist Robin Dunbar came to the conclusion that the size of social circles that individuals of a living species can maintain is set by the size of brain’s neocortex. Further studies have confirmed Dunbar’s findings, with the corresponding sizes for humans set around 10 for trusted personal groups and 150 for untried social ones. As it happens, and not by chance, those numbers seem to coincide with actual observations: the former for personal and direct collaboration, the latter for social and mediated collaboration.
Based on that understanding, the objective would be to organize knowledge workflows across two primary basins:
On-site and face-to-face collaboration with trusted co-workers. Corresponding interactions would be driven by personal dispositions and attitudes.
On-line and networked collaboration with workers, trusted or otherwise. Corresponding interactions would be based on shared interests and past exchanges.
Knowledge Workflows
The aim of knowledge workflows is to process data into information and put it to use. That is to be achieved by combining different kinds of tasks, in particular:
Data and information management: build the symbolic descriptions of contexts, concerns, and means.
Objectives management: based on a set of symbolic descriptions, identify and refine opportunities together with the ways to realize them.
Tasks management: allocate rights and responsibilities across organizations and collaboration frames, public and shallow or personal and deep.
Flows management: monitor and manage actual flows, publish arguments and propositions, consolidate decisions, …
Taking into account constraints and dependencies between the tasks, the aims would be to balance creativity and automation while eliminating superfluous intermediate products (like documents or models) or activities (e.g unfocused meetings).
With regard to dependencies, KM tasks are often intertwined and cannot be carried out sequentially; moreover, as illustrated by the impact of “creative accounting” on accounted activities, their overlapping is not frozen but subject to feedback, changes and adjustments.
With regard to automation, three groups are to be considered: the first requires only raw processing power and can be fully automated; the second also involves some intelligence that may be provided by smart systems; and the third calls for decision-making that can only be done by human agents entitled by the organization.
At first sight some lessons could be drawn from lean manufacturing, yet, since knowledge processes are not subject to hardware constraints, agile approaches should provide a more informative reference.
Iterative Knowledge Processing
A simple preliminary step is to check the applicability of agile principles by replacing “software” by “knowledge”. Assuming that ground is secured, the core undertaking is to consider what would become of cycles and iterations when applied to knowledge processing:
Cycle invariants: tasks would be iterated on given sets of symbolic descriptions applied to the state of affairs (contexts, concerns, and means).
Iterations content: based on those descriptions data would be processed into information, changes would be monitored, and possibilities explored.
Exit condition: cycles would complete with decisions committing changes in the state of affairs that would also entail adjustments or changes in symbolic descriptions.
That scheme meets three of the basic tenets of the agile paradigm, i.e open scope (unknowns cannot be set in advance), continuity of delivery (invariants are defined and managed by knowledge workers), and users in driving seats (through exit conditions). Yet it still doesn’t deal with creativity and the benefits of collaboration for knowledge workers.
Thinking Space & Pace
The scope of creativity in processes is neatly circumscribed by the nature of flows, i.e the possibility to insert knowledge during the processing: external for material flows (e.g in manufacturing), internal for symbolic flows (e.g in software engineering and knowledge processing).
Yet, whereas both software engineering and knowledge processes come with some built-in capability to redefined their symbolic flows on-the-fly, they don’t grant the same room to creativity. Contrary to software engineering projects which have to close their perspectives on the delivery of working products, knowledge processes are meant to keep them open to new understandings and opportunities. For the former creativity is the means to an end, for the latter it’s the end in itself, with collaboration as means.
Such opposite perspectives have direct consequences for two basic agile collaboration mechanisms: backlog and time-boxing:
Backlogs are used to structure and manage the space under exploration. But contrary to software processes whose space is focused and structured by users’ needs, knowledge processes are supposed to play on workers’ creativity to expand and redefine the range under consideration.
Time-boxes are used to synchronize tasks. But with creativity entering the fray, neither space granularity or thinking pace can be set in advance and coerced into single-sized boxes. In that case individuals must remain in full control of the contents and stride of their thinking streams.
It ensues that when creativity is the primary success factor standard agile collaboration mechanisms are falling short and intelligent collaboration schemes are to be introduced.
Creativity & Collaboration Tiers
The synchronization of creative activities has to deal with conflicting objectives:
On one hand the mental maps of knowledge workers and the stream of their thoughts have to be dynamically aligned.
On the other hand unsolicited face-to-face interactions or instant communications may significantly impair the course of creative thinking.
When activities, e.g software engineering, can be streamlined towards the delivery of clearly defined outcomes, backlogs and time-boxes can be used to harness workers’ creativity. When that’s not the case more sophisticated collaboration mechanisms are needed.
Assuming that mediated collaboration has a limited impact on thinking creativity (emails don’t have to be answered, or even presented, instantly), the objective is to steer knowledge workflows across a two-tiered collaboration framework: one personal and direct between knowledge workers, the other social and mediated through enterprise or institutional networks.
On the first tier knowledge workers would manage their thinking flows (content and tempo) independently, initiating or accepting personal collaboration (either through physical contact or some kind of instant messaging) depending on their respective “state of mind”.
The second tier would be for social collaboration and would be expected to replace backlogs and time-boxing. Proceeding from the first to the second tier would be conditioned by workers’ needs and expectations, triggered on their own initiative or following prompts.
From Personal to Collective Thinking
The challenging issue is obviously to define and implement the mechanisms governing the exchanges between collaboration tiers, e.g:
How to keep tabs on topics and contents to be safeguarded.
How to mediate (i.e filter and time) the solicitations and contribution issued by the social tier.
How to assess the solicitations and contribution issued by individuals.
How to assess and manage knowledge deemed to remain proprietary.
How to identify and manage knowledge workers personal and social circles.
Whereas such issues are customary tackled by various AI systems (knowledge management, decision-making, multi-players games, etc), taken as a whole they bring up the question of the relationship between personal and collective thinking, and as a corollary, the role of organization in nurturing corporate innovation.
Conclusion: Collaboration Spaces vs Panopticon
As illustrated by the rising of futuristic headquarters, leading technology firms have been trying to tackle these issues by redefining internal architecture as collaboration spaces. Compared to traditional open spaces, such approaches try to fuse physical and digital spaces into overlapping layers of collaboration spaces, using artificial intelligence to harness cooperation.
Yet, lest uniform and comprehensive transparency brings the worrying shadow of a panopticon within which everyone can be unknowingly observed, working spaces have to be designed as to enhance collaboration without trespassing on privacy.
That could be achieved with a layered transparency set along the nature of collaboration:
Immediate and personal: working cells regrouping 5 to 10 workstations earmarked for a task and used indifferently by teams members.
Delayed and personal: open physical spaces accommodating working cells, with instant messaging and geo-localization; spaces are hinged on domains and focused on shared knowledge.
On-line and networked: digital spaces merging physical spaces and organizational structures.
That mix of physical and virtual spaces could be dynamically redefined depending on activities, projects, location, and organisation.
Functions map inputs with outputs as expected from a performing agent whatever its nature (concrete or abstract) or means (physical, logical, or magical).
They are not to be confused with objectives (which don’t necessarily specify performing agents or detail inputs) nor with activities (which purport to describe concrete execution paths).
Functions and Processes are set along orthogonal dimensions (Simon Fujiwara)
Functions are complete (contrary to objectives) and abstract (contrary to activities) descriptions of what organizations (represented by actors), system architectures (represented by services), or objects (through operations) can do. As such they are akin to interfaces or types, and cannot be instanciated on their own. Processes on the contrary describe how activities are executed, i.e instanciated (#).
Business processes describe sets of execution instances (#). Functions describe what can be expected from enterprise or functional architectures. Business logic describe how the flows are to be processed.
That understanding provides for a modular approach to business processes:
Business processes can be defined with regard to business functions independently of the way they are supported.
Business rules can be managed independently of the way they are applied, by people or systems.
Business logic can be factored out in functions (business or systems) or set within specific processes.
Yet that would not be possible without some modeling across enterprise architecture layers.
Functions & Models
Functions are meant to facilitate reuse across enterprise architectures, which entails descriptions that are clearly and easily accessible: context, modus operandi, expected outcome. Whatever the modeling method(s) in use, it’s safe to assume that different stakeholders across enterprise architectures will pursue different objectives, to be defined with different concepts. If they are to communicate they will need some explicit and unambiguous semantics for the links between processes, functions, and activities:
Functional flows are used between processes and functions (a) or actors (d), or between actors and functions (e).
Composition or aggregates are used to specify where the business logic is to be employed, by functions (b) or by processes (c).
Documentation references (f) are used between unspecified actors and business logic, in case it would performed by people.
Semantics of connectors: functional flows (a,d,e), aggregates (b) and composition (c), and documentation (f).
Finally, the semantics of connectors used between functions will have to be consistent with the one used to connect them to processes and activities.
Combining Functions
Considering that functions are neatly set within the systems modeling realm, one would assume that inheritance and structure connectors can be used to detail and combine them. Yet, since functions cannot be instantiated, some paring down can be applied to their semantics:
Traditional structure connectors are set with regard to identification: bound to structure for composition, set independently otherwise. Since functions have no instances that criterion is irrelevant and the same reasoning goes for composition.
Likewise, since functions have no states to be considered, inheritance of functions can be represented by aggregates.
Functions can be combined at will using only aggregates
As far as functions are concerned, structures as well as inheritance connectors can be fully and soundly replaced by aggregate ones, which could significantly improve the mapping of business processes, activities, and supporting functions.
According to a leading tools provider operational intelligence (OI) is the ability to “discover and analyze relationships between business events and corresponding IT events”.
From a marketing perspective, the moniker suggests some kind of cross-breeding between operational research, artificial intelligence, and real-time analytics. Yet, behind vendor dressing, problems and policies remain the ones traditionally dealt with by decision-making and knowledge management, and as far as marketing is concerned, pitches will hardly affect the assessment of field professionals.
Nevertheless, functional pitches may have a deeper influence if they try to outline the aims of operational intelligence to the people directly involved, affecting the way problems are understood and dealt with. That may be the case if business and system events are seemed to be put on a par: overlooking the directed dependency between actual events and their systems counterparts can critically hamper the very capabilities of systems decision-making.
Facts, Data, & Information
The new connected world of human brains and smart things have scaled down space and time by orders of magnitude, up to the point that events seem to come out as soon as they happen, wherever that may be. Facts and updates, that once were incoming as discrete and manageable batches of information, are now bursting continuously and massively as seamless streams of data that have to be processed on-the-fly into information lest they be cannibalized by ambient noise. That new configuration blurs the distinction between operational data (pushed, shallow, transient) and underlying information (pulled, deep, persistent), making it unworkable, if not meaningless altogether.
Taking inventories decisions as an example, traditional schemes rely on periodic readings of actual inventories and sales crossed with market foresight. Now, with on-line sales and the internet of things, real-time data can be used to build on-the-fly indicators whose biases and inaccuracy would be dynamically readjusted on the basis of information built on hindsight. At any given time (t), decision-makers will be presented with actual observations (a), initial estimations of previous observations (b1, b2), and revised estimations of previous observations (c).
At any given time (t), decision-makers are presented with actual observations (a), initial estimations of previous observations (b1, b2), and revised estimations of previous observations (c).
Set along this framework, the debate about big data can be misleading as it puts the focus on the quantity of data feeding the processes, overlooking the process itself and the distinction between data, information, and knowledge.
Information, Knowledge, & Decision-making
Generally speaking, the distinction between data and information can be set with reference to time and context, data being instant and standalone, and information associated to a shelf life and domain. With regard to decision-making, it would mean that data can be directly used within the context of the current activities and circumstances; e.g, whereas on-line sales data may (or may not) be directly (i.e despite inaccuracies and biases) used to allocate inventories across depots, it has to be “mined” into consolidated information before being used in the broader perspective of inventories planning.
Compared to the transition between data and information, which is carried out by adding time and context, the one between information and knowledge is best understood in terms of decision-making.
Information is obtained by anchoring data to time-frames and contexts, knowledge is acquired by putting information to use.
Decisions are best defined as commitments set against some unknown circumstances: somebody, somewhere, or sometime. First, it ensues that decision-making calls for specific and timed information that has to be maintained up-to-date until decisions are taken. Then, taking decisions introduces some irreversible change in the state of affairs or expectations, making potentially obsolete all relevant information. So it may be argued that decisions is what transform information into knowledge.
Operational Intelligence: Objectives & Tools
Assuming decisions mark the nexus between information and knowledge, operational intelligence could be defined as the ability to put information to use, that ability being supported by the analysis of the relationships between business events and corresponding IT events.
Far from being academic, that distinction is essentially pragmatic as it marks the boundary between OI objectives and tools capabilities:
The aim of OI is to make sense (and profit) from the dynamic relationship between business (aka external) events on one hand, business objectives and enterprise capabilities on the other hand.
The role of supporting tools is to define and manage IT (aka internal) events used to reflect external ones and analyze them.
Whereas business events (red) represent change in the state of affairs, IT events (blue) only represent changes in associated information.
Since IT events are artifacts built on purpose there isn’t much to discover or analyze about them; not to mention the fact that confusing business events and their IT shadows is bound to undermine the whole decision-making process. So what is at stake for OI is how to design IT events as to timely and accurately trail the relevant business events.
Operational Intelligence & Actual Knowledge
As already noted, operational intelligence (OI) is about decision-making, which entails changing the state of objects, processes, or expectations. Compared to knowledge management (KM) which may or may not be time-related, OI is inherently bound to the actual state of affairs: on one hand it relies on specific and timed information, on the other hand it renders that information obsolete when it triggers decisions.
At the risk of oversimplification, operational intelligence can first be understood as a combination of traditional disciplines:
Data-mining is to filter facts and events, capture data, and analyze it into information.
Knowledge management chart information with regard to business objectives and enterprise capabilities.
Decision-making manage time-stamps and plan commitments subject to accuracy and likelihood.
But the specificity of operational intelligence is to be found in the way these functions are intertwined and cross-fed by operational concerns.
To begin with, data mining can be dynamically adjusted depending on what is needed for decision-making, and when. As a corollary, with the benefits of data so cooked in advance, some decisions can be taken directly, bypassing the mediation (and delays) of information processing. From a cognitive point of view that would be the equivalent of non symbolic (aka implicit) knowledge to be processed by neuronal networks.
Decision-making and differentiated knowledge management
Conversely, information processing could benefit from operational feedback so that knowledge management would be driven by business value, and the supporting information weighted by timing and shelf-life considerations. Whereas part of it could be done through implicit connections, it would be more comprehensively and explicitly achieved through symbolic representations.
Operational Intelligence: Signals vs Symbols
Assuming that intelligence is the ability to figure out situations and solve problems, one may conclude that it is inherently operational. Along the same reasoning, if knowledge is information put to use, it may be implicit as well as explicit.
Nonetheless, the merit of operational intelligence is to bring to a single functional roof symbolic and non symbolic knowledge, the former explicit, using mediation of semantic constructs and used to weight information and support managed decisions, the latter implicit, using direct associations between actual objects or phenomena, and supporting automated decisions.
Whereas financial results are being established on an annual basis, events nowadays unfold within a seamless space/time dimension:
Social networks put long-planned strategies at the mercy of consumers weekly whims often unconcerned by borders or products intents.
Mining of big data may curtail innovative head-starts to a matter of weeks if not days.
On-line (not to mention high frequency) trading put markets sanction at investors fingers tips, and enterprises’ stocks at predators’ claws.
So why should enterprises bother with yearly schedules ?
Evolutionary Arms Race Doesn’t Wait for Saint Sylvester
Business is governed by the same rule as nature, namely the survival of the fittest, and as with biological ecosystems, enterprises’ individual fitness depends on their relationships with others. Hence, as suggested by Lewis Carroll’s Red Queen, the survival of enterprises in their evolutionary arms race doesn’t depend on their absolute speed set against some time-frame, but on the relative one set with regard to competitors. In that case Saint Sylvester could be ignored. Or should it ? Because seasons do play their part in biological races.
From Race to Game
As it happens, business races are becoming more complicated as extensive and ubiquitous information and communication systems redefine the traditional predator-prey casting. With time and space cut down to symbolic dimensions, collaboration and competition can no longer be safely allocated to time-spans and market segments, which entails that the roles of predator and prey can be upended on thespur of a moment and the turn of a switch.
Introducing this kind of option transforms tracks into playgrounds and races into games because there is no point in running without knowing who to run against and who to run with. Adding to the challenge, these playgrounds are moving ones, and so the rules of the game: one week a non-zero sum game, the next a winner-gets-all. That is when calendars make a come-back.
Games come with Boards and Time Scales
Contrary to races, games have comprehensive and detailed rules, or regulations in business parlance. As soon as enterprises start to play with roles they have to meet conditions, follow procedures, and take into account specific constraints; all defined with regard to institutional spaces and calendar time, e.g:
Customers’ behavior is often seasonal.
Regulatory bodies rule within geographical borders and their decisions stand for calendar periods.
M&A have to align with stock markets schedules
So, assuming that institutional time-frames cannot be avoided when strategies are set, Saint Sylvester may provide a practical meter for yearly moves.
When to Board a Shuttle
Enterprises have therefore to set their decision-making with regard to the accelerating pulse of business events on one hand, institutional time-boxes on the other hand. That corresponds to a typical shuttle scheme, with taking a decision being associated with boarding a shuttle.
Along that understanding, and assuming that taking a decision means closing alternative options, boarding a shuttle excludes some of the next one(s), and missing it may be beneficial as well as detrimental. That’s the reasoning behind the Last Responsible Moment principle which, when put to use for periodic decision-making, suggests that there can be as much risk at boarding a shuttle than at missing it.
Generally speaking, quality may refer to intrinsic properties, functional characteristics, or some external yardstick. With regard to software engineering it would mean code, users experience, and operations, each with its own specific stakeholders and criteria.
Circling Quality Metrics (Martine Franck)
On one side, traditional phased approaches to QA are meant to deal with those different aspects, yet they fall short when those facets are weaved together across enterprise architectures and business environments. On the other side agile quality solutions may also fail to cope with transverse business functions shared across architectures. Hence the need of a bird’s-eye view putting quality into a broader enterprise perspective.
Who Cares for Quality
Whatever the attributes considered, quality should clearly encompass actual products as well as their uses. For that purpose quality has to be assessed with regard to the requirements as expressed by business stakeholders, users, or systems engineers and administrators. Given the constraints and specificity of changing environments, objective yardsticks are of limited use and quality is often to be assessed for the lack thereof:
Business requirements: the product doesn’t meet expectations with regard to business contents (objects and logic).
Functional requirements: while the product meets business requirements, the part played by supporting systems doesn’t meet users’ expectations.
Quality of service: while the product meets business and functional requirements, users’ experience doesn’t meet expectations.
Technical requirements: while the product meets users’ expectations (business, functional, and ease of use), there are problems with deployment, maintenance, or operations.
Quality is best defined with regard to requirements and checked with regard to architectures
Crossing those concerns, quality assessment has to deal with two primary challenges:
Since assessment at each level can be conditioned by lower levels, outcomes must be described and traced accordingly. That is to be the role of quality management.
Since assessment has to cover both products and their use during their shelf life, uncertainty will have to be taken into account. That is to be the role of quality assurance.
A third aspect can be added for externalities, i.e factors whose impact cannot be clearly or uniquely attributed: external risks are not under control, ergonomy cannot be accurately measured, and the assessment of ROI for processes improvement remains a matter of insight.
Quality Management & Documentation
The primary objective of quality management is to identify, define, and track the targeted outcomes and the factors deemed to affect their characteristics: contracts, products traceability, models reuse, tests, etc.
Depending on target and development model, management footprint can be defined at three levels of detail:
With regard to the use of products in their operational context, the focus is to be on deployed systems compared to textual specifications (a).
With regard to the intrinsic properties of deliverables, the focus is to be extended to software components (b).
When products are to be deployed in different environments, or to be maintained or modified along time, additional documentation will be necessary to trace changes to functional (c) and enterprise (d) architectures.
Assessment at each level may be conditioned by lower levels
In any case (i.e with or without intermediate documentation,) traceability is to be a corner-stone of quality management:
Business processes with regard to business objectives, e.g how to assess insurance premiums or compute missile trajectory.
Code with regard to textual requirements.
System functionalities with regard to business processes. Use cases are widely used to describe how systems are to support business processes, and system functionalities are combined to realize use cases.
System components as technical implementations of functionalities targeted to different users, locations, and configurations.
And another dimension of traceability is required when quality assurance has to deal with uncertainty, risks, and decision-making.
From Management to Assurance
The objective of quality assurance is to define, carry on, and monitor operations in order to improve the characteristics concerned and reduce the probability that something will go amiss during the planned shelf life of products.
For that purpose assurance footprint and granularity must be aligned with the layers defined by quality management:
Integration and acceptance tests are carried out in reference to requirements on the assumption that software components have been validated.
Code checking and unit tests are carried out in reference to business and functional requirements on the assumption that their consistency has been checked.
External consistency is checked with regard to business requirements independently of functional or technical ones.
Internal consistency is checked with regard to functional requirements on the assumption that the business requirements (external) consistency has been checked.
Footprint & granularity of management and assurance must be congruent
Those operations, meant to deal with the quality of each layers, have to be combined with schemes of secure transformations between layers, e.g reuse, patterns or code generation. That would put quality on a sound basis were it not for externalities.
Quality Assurance & Risk Management
As already noted, QA has to take into account uncertainties and risks both external (business or technical environments) and internal (development processes). Assuming quality assurance has to include risk assessment, policies should be driven by risk acceptance levels:
No risk: quality assurance can be designed as to eliminate some uncertainties (e.g reuse and code generation).
No risk taken: whereas business and technology options are not sure bets some must be carried out regardless of what happens in the environment (e.g unexpected regulatory change or delay in critical technology). In that case QA must provide fallback solutions.
Managed risks: some defaults or delays can be priced and weighted by likelihood. In that case QA should monitor the risks and balance their cost (e.g resources consumption, late delivery) against the cost of preventive (e.g more systematic checks on consistency, additional staff) or corrective (e.g tests or maintenance) measures.
Quality management should be set at the nexus between risks management and quality assurance.
Data mining explores business opportunities and competitive advantage, requirements analysis considers supporting applications. Both use models, the former’s are predictive and ephemeral, the latter’s descriptive (or prescriptive) and perennial.
Data mining: sorting business wheat from world chaff (Andreas Gursky)
As the generalization of digitized environment calls for more integration of business and software engineering processes, understanding the relationship between data mining and requirements analysis could significantly improve processes maturity and agility.
Data vs Requirements Analysis
Nowadays the success of a wide range of enterprises critically depends on two achievements:
Mapping business models to changing environments by sorting through facts, capturing the relevant data, and processing the whole into meaningful and up to date information. That can be achieved through analysis models mapping business expectations to supporting systems.
Putting that information into effective use through business processes and supporting systems. That is done through systems architecture and design models meant to prescribe how to build software artifacts.
Those challenges are converging: under the pressure of markets forces and technological advances most of traditional fences between business channels and IT systems are crumbling, putting the focus on the functional integration between data mining and production systems. That’s where predictive models can help by anchoring descriptive models to moving markets and by cross-feeding analysis and operations. How that can be achieved has been the bread and butter of good corporate governance for some time, but there has been less interest for the third branch, namely how data analysis (predictive models) could “inform” business requirements (descriptive models).
Facts are not given but must be captured through a symbolic description of actual observations. That entails some observer set on task using a mix of conceptual and technical apparatus. Data mining and requirements analysis are practical realizations of that process:
Data mining relies on analytic tools to extract revealing information that could be used to chart opportunities along business models.
Requirements analysis relies on business processes and users’ practice to extract symbolic descriptions that will be used to build models of supporting applications.
If both walk the path from data to information, their objectives are different: the former’s is to improve business decisions by making sense of actual observations; the latter’s is to build system surrogates from the symbolic descriptions of actual business objects and activities.
Anchors & Structures: Plasticity of Business Entities
Perhaps paradoxically, business agility calls for terra firma because nimble trades must be rooted in corporate identity and business continuity. As a consequence, the first step of requirements analysis should be to associate individuals business objects or activities with stable and consistent identification mechanisms, and to group them with regard to that mechanism:
External entities with natural (person) or designed identity (car).
Symbolic entities for roles (customer) or commitments (maintenance contract).
Actual activities (promotion campaign) and events (sale) or business logic (promotion).
Anchors
Conversely, as the aim of data analysis is to explore every business angle, individual observations are supposed to be moved across groups; yet, since the units identified by data analysis will have to be aligned with the ones described by requirements analysis, moves must also keep track of identities. That dilemma between continuity of identified structures on one side, plasticity of functional aspects on the other side, can be illustrated by banks which, in response to marketing requirements, had to shift from account (internal identification) to customer (external identification) based systems.
It’s easier to market insurance from customer centered systems (right) than from account centered ones (left)
That challenge can be overcome by linking the identification of symbolic entities to external anchors.
Profiles & Features: Versatility of Business Opportunities
As noted above, requirements and data analysis are set on the same road but driven by different forces: the former tries to group individuals with regard to identification mechanisms before fleshing them out with relevant features; the latter tries to group individuals with given identities according to features and opportunity profiles. Yet, what could appear as collision courses may become a meeting of minds if both courses are charted with regard to variants analysis.
From the requirements perspective the primary concern is to distinguish between structural and functional variants:
Structural variants are bound to identities, i.e set up-front for the respective life-cycle of individual business objects or transactions. As a consequence they cannot be changed without undermining business continuity. Moreover, being part and parcel of descriptors (e.g types and use cases) their change will affect engineering processes.
Functional variants may vary during the respective life-cycle of individual business objects or transactions. As a consequence they can be changed without undermining business continuity, and changes in descriptors (e.g partitions and scenarii) can be managed without affecting engineering processes.
From the data mining perspective the objective is to improve the benefits of information systems for decision-making processes:
Static: how to classify individuals as to reduce the uncertainty of predictions
Dynamic: how to classify business options as to reduce the uncertainty of decisions.
Since those objectives are set for individuals, constraints on continuity and consistency can be dealt with independently of the description of symbolic surrogates.
Identified individuals with profiles for customers (a), their behaviors (b), and promotional gestures (c)
It ensues that perspectives can be adjusted by factoring out the constraints of continuity and consistency for business objects (e.g cars), agents (e.g customer) and processes (e.g repair). Profiles for agents (a), behaviors (b), and business options (c) could then be freely explored and tailored with regard to changes in business environment and objectives.
Applying Data Analysis to Requirements
Not surprisingly data analysis techniques can be used to adjust perspectives. For that purpose a sample of individuals (business objects and operations) representing the population targeted by requirements would have to be submitted to basic mining routines. Borrowing a catalog from F. Provost & T. Fawcett:
Classification: estimates the probability for each individual (objects or operations) to belong to a set of classes; can be used to assess the closeness of the variants (respectively power-types or execution paths) identified by requirements analysis.
Regression: reverse classification; estimates how much of individual features valuations can be explained by the proposed classifications.
Similarity: a shallow version of classification; can be used to assess the distance between variants and consolidate the proposed classifications.
Clustering: a deep version of classification; can be used to distinguish between shallow and natural classifications.
Co-occurrence: deals with behavioral variants; can be used to distinguish between functional and structural classifications.
Profiling: reverse of co-occurrence; can be used to consolidate functional and structural classifications.
Links prediction: can be used to define relationships.
Data reduction: eliminate redundant individuals; can be used to consolidate requirements and refine tests scenarii.
Causal modeling: brings together business logic (events and rules) and users decisions; should provide the backbone of tests scenarii.
Besides the direct benefits for requirements, such procedures may help to bridge the span between data and requirements analysis and significantly improve processes’ capability and maturity level.
Data mining being first and foremost about competitive edge, it relies on a timely and effective coupling between enterprises capabilities and business opportunities. But the dilemma between continuity and plasticity described above for business objects and processes reappears at enterprise level: how to conciliate architecture, by nature perennial, with the agility needed to make the best of changing and competitive environments ?
As architectural big bang is arguably a last resort option, answers to that question must be progressive and local: if changes are to be swift and pertinent they must be both circumscribed and leveraged to the relevant parts of architecture. Taking an (amended) leaf of the Zachman framework, its sixth column (“Why” ) could be reset as a line for business and operational objectives that would cross the original five columns instead of the architecture layers. Using a pentagonal representation of enterprise architecture, that line would be set as circling the outer range.
Enterprise Architecture and the loci of change
It is worth to note that setting objectives on a line crossing the columns of capabilities instead of a column crossing the lines of layers means that objectives are set at enterprise level and their cascading impact traced and managed through layers.
Conceptual Models & Business Contexts
But even that updated framework doesn’t take into account the fundamental changes in business environments. Once secure behind organizational and technical fences, enterprises must now navigate through open digitized business environments and markets. For business processes it means a seamless integration with supporting applications; for corporate governance it means keeping track of heterogeneous and changing business contexts and concerns while assessing the capability of organizations and systems to cope, adjust, and improve.
As long as environments were a hotchpotch of actual and symbolic artifacts the pros and cons of integration could be balanced. But the generalization of digital flows and transactions has upended the balance: there is no more room or time for latency and enterprises must bring all symbolic representations (business, organization, and systems) under a common conceptual roof:
Conceptual models as bridges between business processes, and systems.
A canonical approach would be to introduce a conceptual indexing scheme open to extensions but with its footprint defined by business processes and systems functionalities. That would ensure a better integration of processes and supporting engineering but will do nothing for the modeling gap between enterprise architecture and external contexts. That could be achieved with ontologies.
Conceptual Loop: Ontologies & Business Intelligence
As far as data mining is concerned, three kinds of operations are to be considered:
Data understanding gives form and semantics to raw material.
Business understanding charts business contexts and concerns in terms of objects and processes descriptions.
Modeling consolidate data and business understanding into descriptive, predictive, and operational models.
The aim of data mining is to refine raw data into meaningful information
While traditional approaches fall short when tasked with iterative modeling of unstructured data, ontologies may fare better because their explicit aim is only to describe what could exist in a domain of discourse:
They are made of categories of things, beings, or phenomena; as such they may range from simple catalogs to philosophical doctrines.
They are driven by cognitive (i.e non empirical) purposes, namely the validity and consistency of symbolic representations.
They are meant to be directed at specific domains of concerns, whatever they can be: politics, religion, business, astrology, etc.
With regard to models, only the second point puts ontologies apart: contrary to models, ontologies are about understanding and are not supposed to be driven by empirical purposes. It ensues that ontologies can be understood as conceptual (aka canonical) models, used as such for business analysis and extended with purposes for systems analysis and design.
In addition to the integration with enterprise architectures, ontologies benefits for business intelligence would be twofold.
On one side, and whatever their use, ontologies could be aligned with the nature of contexts and their impact on business and enterprise governance e.g:
Institutional: mandatory semantics sanctioned by regulatory authority, steady, changes subject to established procedures.
Professional: agreed upon semantics between parties, steady, changes subject to established procedures.
Corporate: enterprise defined semantics, changes subject to internal decision-making.
Social: definedpragmatic semantics, no authority, volatile, continuous and informal changes.
Personal: customary semantics defined by named individuals.
Ontologies, capabilities (Who,What,How, Where, When), and architectures (enterprise, systems, platforms).
On the other side ontologies could be defined according to the nature of targeted items, namely terms, documents, symbolic representations, or actual objects and phenomena. That would outline four basic concerns that may or may not be combined:
Thesaurus: ontologies covering terms and concepts.
Document Management: ontologies covering documents with regard to topics.
Organization and Business: ontologies pertaining to enterprise organization, objects and activities.
Engineering: ontologies pertaining to the symbolic representation of products and services.
Ontologies: Purposes & Targets
On a broader perspective, ontologies could be influential in spanning the gap between explicit and implicit knowledge. In a few years’ time practically unlimited access to raw data and the exponential growth in computing power have opened the door to massive sources of unexplored knowledge which is paradoxically both directly relevant yet devoid of immediate meaning:
Relevance: mined raw data is supposed to reflect the geology and dynamics of targeted markets.
Meaning: the main value of that knowledge rests on its implicit nature; applying existing semantics would add little to existing knowledge.
Assuming that deep learning can transmute raw base metals into knowledge gold, ontologies would be decisive in framing the understanding, assessment, and improvement of the processes.
Operational Loop: Business Intelligence & Decision-making
Once carried out separately and periodically, decision-making is to be carried out iteratively at operational, tactical, and strategic level; while each level is to be set along its own time-frames, all are to rely on data-mining, with cycles following the same pattern:
Observation: understanding of changes in business opportunities.
Orientation: assessment of the reliability and shelf-life of pertaining information with regard to current positions and operations.
Decision: weighting of options with regard to enterprise capabilities and broader objectives.
Action: carrying out of decisions within the relevant time-frame.
Given business new playground, decision-making processes have to weave together material and digitized flows, actual contexts (aka territories) and symbolic descriptions (maps), and overlapping time-frames (operational tactical, strategic). That operational loop could then be coupled with the broader one of business intelligence:
Integration of operational analytics, business intelligence, and decision-making.
Frameworks are meant to abet the design and governance of enterprises’ organization and systems, not to add any methodological layer of complexity. If that entry level is to be attained preconditions are to be checked out for comprehensiveness, modularity, clarity of principle, and consistency.
Modularity & Clarity (Cildo Meireles)
Meeting core EA framework requirements will in turn greatly facilitate declarative and iterative approaches to enterprise architecture.
Comprehensiveness
The primary objective of an enterprise framework is to bring under a common management roof different contexts and concerns (business, technical, organizational), and synchronize their respective time-frames. That can only be achieved through an all-inclusive and unified conceptual perspective.
Suggested check: Variants for core concepts like agents or events must be clearly defined at enterprise and system levels; e.g people (agents with identity and organizational status), roles (organization and access to systems), and bots (software agents without identity and organizational status).
Modularity
On one hand enterprise frameworks must deal with strategic issues without being sidetracked by enterprises idiosyncrasies. On the other hand swift and specific adaptations to changing environments should not be hampered by cumbersome procedures or steep learning curve. That can only be achieved by lean and versatile frameworks built from a clear and compact set of architecture artifacts, to be readily extended, specialized or implemented through the enactment of dedicated processes.
Suggested check: How a framework is to further the development of a new business, facilitate the merging of organizations, or support the transition to a new architecture (e.g SOA).
Clarity of Principle
Comprehensiveness and modularity are pointless without a principled backbone supporting incremental changes and a smooth learning curve. For that purpose a clear separation should be maintained between the semantics of the core patterns used to describe architectures and the processes to be carried out for their evolution.
Suggested check: The meaning of primary terms (event, role, activity, etc) should be uniquely and unambiguously defined based on the core framework principles, independently of the processes using them.
Consistency
EA frameworks should be more compass than textbook, drawing clear lines of action before providing details of implementation. Lest architects been lost in compilations of ambiguous or overlapping definitions and rules, core meanings must remain unaffected when put to use across the framework.
Suggested check: Carry out a comprehensive search for a sample of primary terms (e.g event, role, activity, etc.), list and compare the different (if any) definitions, and verify that they can be boiled down to a unique and unambiguous one.
EA & Model Based Systems Engineering
These basic requirements get their full meaning when set in the broader context of EA evolution. Contrary to their IT component, enterprise architectures cannot be reduced to planned designs but grow from a mix of organization, culture, business environments, technology constraints, and strategic planning. As EA evolution is by nature incremental, supporting frameworks should provide for iterative development based on declarative knowledge of their organizational or technical constituents. That could be achieved by combining EA with model based systems engineering.
Business driven development is arguably the cornerstone of the agile development model. On one side it means business value set by users and stakeholders, on the other side it entails continuous and just-in-time delivery; what happens in-between is set by backlogs (for development) and product increments (for delivery).
Continuous Delivery
A sanguine understanding of continuous releases may assume that planning is no longer relevant, and that deployment can be carried out “on-the-fly”. But that would assume that stakeholders and product owners are ready to put aside roadmaps, overlook milestones, and more generally forget that time is money. That would go against a basic objective of agile, namely that developments must be driven by business needs, and products delivered just-in-time for best value. Given the well established track record of dynamic programming for manufacturing processes, could the technique be usefully applied to agile engineering processes ?
Delivery, Deployment, & Continuity
Continuity doesn’t mean synchronization: business, engineering, and operations are governed by different concerns set along different time-frames. Some buffering is needed, materialized by the distinction between releases (engineering concerns) and deployment (business and operational concerns).
A time for every purpose: Epics & business roadmap (a), associated backlogs of users’ stories (b), released features (c), architectural capabilities (d), deployed components (e).
Such distinctions introduce both overlapping (with business time-frame) and discontinuity (between development and deployment):
Product roadmaps are set in business time-frames and determine development and deployment time-frames.
Development time is set by product roadmaps and runs clockwise from project inception to software releases (a>b>c).
Deployment time is also set relative to product roadmaps but it runs counterclockwise, from product deployment as planned by business back to released software components (c<e).
Development and deployments runs can be compared to crews tunneling through a mountain from both sides; where and when they meet leaves room to adjustments. Yet more is at stake in the meeting between development completion and deployment inception. Apart from time, adjustments may also bear on formats and contents; and given the specificity of development and deployment purposes, their adjustment may also be seen as the morphing of continuous software releases (project perspective) into discrete increments (product perspective).
Dynamic Programming
Dynamic programming (aka multistage programming) is a problem solving method that combines two principles:
Divide & conquer is a general purpose strategy that deals with the intrinsic complexity of problems by breaking them down into collections of simpler sub-problems to be solved separately depending on sequencing constraints.
Recursion deals with the lack of complete or accurate information upfront, solving the problem in stages rather than as one entity. Each stage is optimized separately on the basis of the current state reflecting decisions taken at previous stages, the optimality principle guaranteeing the optimality of the final outcome.
That incremental and iterative approach clearly befits the tenets of the agile development model.
Twofold Planning
As noted above, and whatever the technique, agile processes entwine two phases, one for development, the other for deployment, each with its own planning:
The development planning, epitomized by backlog management, deals with the definition of work units and their sequencing.
The deployment planning deals with the merging of software released into product and their incremental deployment.
Project work units are sequenced (backlog), Product increments are merged, and both are dynamically adjusted around their nexus.
That makes for multiple dynamics, first for updated backlogs, then for updated deployment targets, and finally for possible feedback through their nexus. That can only be achieved with dynamic programming.
Backlog: Multistage sequencing
Backlogs are used to manage work units targeting self-contained requirements items; they can be represented by graphs with nodes standing for work units and arcs weighted by constraints.
Basically, the problem is to optimize the development of a given set of items given users’ priorities, technical constraints, and resources availability. When all the information is available upfront, optimum solutions can be obtained with simple Shortest Path algorithms. Yet, given the iterative and exploratory nature of agile processes, backlogs are meant to be updated as the project advances, taking advantage of improved knowledge:
Users may introduce new items, remove ones, or change their priorities due to a better understanding of requirements space.
Engineers may also introduce new items (e.g for technical debt) or reconsider technical difficulties and dependencies due to a better understanding of solutions space.
Dynamic reordering of backlogs: looking forward for the optimum path to completion.
Dynamic programming is introduced in order to support step-wise decisions optimizing the whole process:
Backlog states (t1 & t2) are defined by the remaining work units, rankings, and feasibility constraints.
Each stage redefines the optimum path to completion taking into account the current state and updated information. Recursive computation being based on the summary information etched in states, it ensues that all future decisions can be selected optimally without recourse to information regarding previously made decisions.
Given a set of feasible paths (as defined by technical dependencies and time), the aim at each stage is to select the optimum path for the remaining units based on current state. Optimization functions will typically consider users’ value, learning curve and associated risks, and resources availability and costs.
As illustrated below, nodes can represent grouped items, e.g when several projects have to share resources or releases are to be regrouped.
Deployment: Dynamic merging
Given a set of released software components, the aim of deployment planning is to decide which increment to add to deployed product. Assuming that technical concerns have already been dealt with by releases, the objective at each stage is select the items maximizing the ROI of the deployed product. It must be noted that, contrary to the development algorithm looking forward for the optimization, the deployment algorithm select the optimum path by looking backward at the ROI of deployed products.
Dynamic reordering of deployments: looking backward for the optimum path to completion.
But the backward impact of deployment optimization can go further and affect backlogs.
Feedback
Shared ownership and continuous delivery are two main pillars of the agile development model, the former giving the development full authority and responsibility, the latter ensuring the users have a firm hand on the helm. Yet, as already noted, development, deployment, and business are governed by different time-frames, which could induce some frictions, e.g if business units were forced to synchronize products deployments with software releases. While severe disruptions can be avoided if releases and deployments are managed separately, development teams cannot be completely sheltered from changes in business or operational priorities. That is where the dynamic reassessment of optimum paths is to help: assuming a change in deployment planning (nq instead of op), the new priorities are fed back into development (aka backlog) rankings, and the optimum path is updated.
Change in deployment priorities (nq instead of op) can be fed back to backlog planning (f before a).
It must be noted that such feedback only affects ranks and must leaves contents unchanged.
Conclusion: Business driven, Just-in-time delivery, & Lean Inventories
Dynamic programming appears as a primary factor with regard to three core tenets of the agile development model:
Business driven development doesn’t mean that developments are pushed by requirements but that they are pulled by deployment.
Just-in-time delivery can only be achieved with the help of a buffer between development and deployment. This buffer should not be confused with an inventory as it has nothing to do with product quantities.
On the contrary, this buffer, combined with dynamic programming, plays a critical role in the cutback of intermediate documents and models (aka development inventories).
Given the ubiquity of information and communication technologies on one hand, the falling apart of technical fences between systems, enterprises, and business environments on the other hand, applying the operating system (OS) paradigm to enterprise architectures seems a logical move.
Users and access to services (Queuing at a Post Office in French West Indies)
Borrowing the blueprint of computers operating systems, enterprise operating systems (EOS) would be organized around a kernel managing shared resources (people, hardware and software) and providing services to business, engineering or operational processes.
Gerrymandering & Layers
When IT was neatly fenced behind computer screens managers could keep a clear view on organization, roles, and responsibilities. But with physical hedges replaced by clouded walls, the risk is that IT may appear as the primary constituent of enterprise architecture. Given the lack of formal arguments against what may be a misguided understanding, enterprise architects have to rely on pragmatic answers. Yet, they could prop their arguments by upending the very principles of IT operating systems and restore the right governance footprint.
To begin with, turfs must be reclaimed, and that can be done if the whole of assets and services are layered according to the nature of problems and solutions: business processes (enterprise), supporting functionalities (systems), and technologies (platforms).
EA must separate and federate concerns along architecture layers
Then, reclaiming must also include governance, and for that purpose EOS are to rely on a comprehensive and consistent understanding of assets, people and mechanisms across layers:
Physical assets, including hardware.
Non physical assets, including software.
Agents (identified people with organizational responsibilities) and roles.
Events (changes in the state of objects, processes, or expectations) and activities.
Mimicking traditional OS, that could be achieved with a small and compact conceptual kernel of formal concepts bearing out the definitions of primitives and services for the whole of enterprise processes.
EOS’s Kernel: 12 concepts
A wealth of definitions may be the main barrier to enterprise architecture as a discipline because such profusion necessarily comes with overlaps, ambiguities, and inconsistencies. Hence the benefit of relying on a small set of concepts covering the whole of enterprise systems:
Six for individuals actual (objects, events, processes) and symbolic (surrogates objects, activities, roles) elements.
One for actual (locations) or symbolic (package) containers.
One for the partitioning of behaviors (branch) or surrogates (power type).
Four for actual (channels and synchronization) and symbolic (references and flows) connectors.
Governance calls for comprehensive and consistent semantics
Considering that nowadays business entities (enterprise), services (systems), and software components (technology) share the same distributed world, these concepts have to keep some semantic consistency across layers whatever their lexical avatars. To mention two critical examples, actors (aka roles) and events must be consistently understood by business and system analysts.
Those concepts are used to describe enterprise systems building blocks which can be combined with a small set of well known syntactic operators:
Two types of connectors depending on target: instances (associations) or types (inheritance).
Three types connections for nondescript, aggregation, and composition.
Syntactic operators are meant to be applied independently of targets semantics
Again, Occam’s razor should be the rule: just like semantics are consistently defined across architecture layers, the same syntactic operators are to be uniformly applied to artifacts independently of their semantics.
Kernel’s Functions
Continuing with the kernel analogy, based on a comprehensive and consistent description of resources, the traditional OS functions can be reinterpreted with regard to architecture capabilities implemented across layers:
What: memory of business objects and operations (enterprise), data base logical entities (systems), data base physical records (platforms).
When: business events and processes (enterprise), transaction management (systems), and middleware (platforms).
Where: sites (enterprise), logical processing units (systems), network architecture (platforms).
How: business processes (enterprise), applications (systems), and programs (platforms).
Traceability of Capabilities across architecture layers
That fits with the raison d’être of a kernel which is to combine core functions in order to support the services called by processes.
Services
Still milking the OS analogy, a primary goal of an enterprise kernel is to support a seamless integration of services:
Business driven: the definition of services must be directly and unambiguously associated to business ends and means across enterprise layers.
Traceability: they must ensure the transparency of the tie-ups between organization and processes on one hand, information systems on the other hand.
Plasticity: they must facilitate the alignment of changes in business objectives, organization and supporting systems.
A reasoned way to achieve these objectives is to classify services with regard to the purpose of calling processes:
Business processes deal with the transactions between the enterprise and its environment.
Engineering processes deal with the development of enterprise resources independently of their use.
Operational processes deal with the management of enterprise resources when directly or indirectly used by business processes.
Enterprise Operating System: Layers & Services
That classification can then be crossed with architecture levels:
At enterprise level services are bound to assets to be used by business, engineering, or operational processes.
At systems level services are bound to functions supporting business, engineering, or operational processes.
At platform level services are bound to resources used by business, engineering, or operational processes.
As services will usually rely on different functions across layers, the complexity will be dealt with by kernel primitives and masked behind interfaces.
Services called by processes can combine different functions directly (basic lines) or across layers (dashed lines).
Finally, that organization of services along architecture layers may be aligned with governance levels: strategic for enterprise assets, tactical for systems functionalities, and operational for platforms and resources.
