Reporting and sharing data with other systems

Introduction

Many of our customers want data to be made available for reporting. This design guide is about reporting, but in fact it is more general: the design guide is about sharing data with other systems. Since Blueriq itself does not provide reports, but merely makes data available for other systems to create reports based on this data, the title reporting is perhaps a bit expectant.

Intermezzo

Making data available for reporting is not the same as storing or tracing data, nor is it the same as using web services. Furthermore, reporting is not an equivalent of business activity monitoring or process mining. To make a clear distinction between these types of data interaction and data interpretation, see the image below.

In this design guide

Persistency

Persistency is all about storing and preserving data and sharing that data among other operational applications. Persistency is typically done in the operational environment and deals with live/actual/realtime data. This data is usually stored a long time. Typical example are: Wedding insurance request data / Person data / Risk profile data. In Blueriq, persistency is achieved by aggregate definitions in the domain, combined with CRUD and search services and a list container.

For more information, see the Persistency Management guide.

Business activity monitoring

Business Activity Monitoring also deals with live/actual/realtime data, but this data does not have to be stored or preserved at all. BAM typically concerns data about the production/operational environment at this very moment, mostly about (the number of) cases/tasks/users etc. Since the data shown in BAM is volatile and constantly changing, the process database is queried at runtime (also the aggregate database since aggregates can be used to mimic cases, as discussed in Designing cases using aggregates). Typical example are: The number of open/closed/rejected cases of type Wedding Insurance Request at this moment. In Blueriq, containers are available that query the process database, the aggregate database and create visualizations of those queries.

For more information, see Container type: AQ_Statistic_Aggregate, Container type: AQ_Statistic_Process, Container type: AQ_Statistic_Visualization.

Audit/Trace

Traceability is mostly about storing who did what when. The main goal is to make sure that after a while it is still possible to find out what happened. While tracing, the entire profile (in process-tracing) or the aggregate profile (in aggregate-tracing) or the entire runtime decision graph (in decision-tracing) is stored. Since the amount of data that is stored might grow rapidly - and also not to unnecessarily burden the production environment - trace data is stored separately from the production environment. Tracing is done automatically for processes and aggregates (and can be switched off) and can be configured for decisions. A typical example is: Aggregate Person is updated by user Frank at 2015-07-01 11:19:35 and the aggregate content is <...>.

For more information, see Traceability.

Web services

Web services are used to communicate with other systems via messages. A typical examples is a social security number sent to a web service, and the current marital status of that person is retrieved. Web services definitions are available in Blueriq by means of schema sets (XSD) or domain schemas. Web service calls can be created using services and are available in XML and JSON format.

For more information, see Web Services guide.

Process mining

Process mining typically concerns the afterward evaluation of a process and all its instances. In order to do this, the process trace database is interpreted. A typical example is: When viewing all instances of the Wedding insurance request case process, in how many instances was a risk profile needed? Process mining is not (yet) standard Blueriq functionality.

Reporting

Reporting concerns data in the (near) past and is in most cases done using data marts that originate in a datawarehouse. In order to feed the datawarehouse with proper data, a reporting service can be used. This service is called when necessary and generates an XML file with live/actual/realtime data. The XML is stored in a database, along with some metadata. Since the amount of data that is sent to other systems might grow rapidly - and also not to unnecessarily burden the production environment - reporting data will be stored separately from the production environment. There is a lot of overlap between data that is persisted and data that is sent for reporting, apart from the fact that reporting data also contains system set (derived) data and persisted data mostly does not.

The procedure of reporting is fairly simple and boils down to the use of a single service: Service call type: AQ_Report. What exactly is exported by this service, is the subject of the rest of this design guide.

Decoupling

Although reporting is all about facilitating accurate data at the right time, it does not mean that the reporting model must reflect the application model exactly all the time. Some sort of decoupling between the application model and the model intended for reporting is necessary, to avoid dependencies and make agile application development in Blueriq possible. This is illustrated below.

As shown above, the application module (sometimes referred to as domain module) is not the source for reporting. A mapping between an application module and a reporting module acts as this source. By doing this, the application module can be adjusted, without bothering the reporting module and thus the external system. This is illustrated by the numbers in the elements in the application and the reporting module:

Initially in iteration 1 (depicted by 1's in the elements), a reporting module was created that contained a subset of the elements of the application module.
A datamapping was created that mapped the application module elements onto corresponding reporting module elements.
Development of the application model and the application took place, at iterations 2 and 3.
It was decided that a certain element created in iteration 1 should be replaced by a corresponding but different element created in iteration 2.
The datamapping was adjusted accordingly, the reporting domain remains completely unaffected.

This form of decoupling between an application and an external system, i.e. decoupling the domain definition and the reporting definition, is helpful in agile application development as the external system probably does not use the same release cycle as the Blueriq application.

Reporting module

At a certain moment in the application development process, a reporting module is created, along with a mapping from the application module onto the reporting module. This moment in time has to be chosen wisely and will probably be late in this process, when most of the domain is defined and created. When creating such a reporting module, beware to adhere to the following rules:

The reporting module must be an interaction module.

To create a reporting module, the business engineer can copy the application module. See How to copy modules on where and how to make a copy of a module. Most likely, the copied module will be a flattened (or "rolled up") copy, where all elements of base modules are part of the copy.

Only the following elements should be present in the reporting module:

Entities
Attributes
Relations
Value lists

Other elements are useless in a reporting module, since the reporting service completely ignores these elements.

After creating a reporting module, which could be made by hand or be a flattened copy of the application module, the business engineer should delete all elements from this copy, except all entities, attributes, relations and value lists. After that, entities, attributes, relations and/or value lists that are not part of the reporting domain can be deleted.

When the reporting module is constructed as desired a datamapping must be created. See Generate an auto-datamapping to find out how this can be done very quickly.

The datamapping from an interaction module to a reporting module should be uni-directional, meaning only from the domain module to the reporting module.
Mapping the other way around is useless, since the reporting service completely ignores this datamapping.

Strategy

One of the benefits of decoupling the domain model and the reporting model by means of modules and a datamapping is that it is possible to create multiple reporting modules and multiple datamappings, all based on the same domain. Furthermore, it is possible to have a single datamapping to be used to export to different connections. This is illustrated below.

Default strategy

The default strategy regards a simple export; within an application module, a service is called that uses a datamapping (that maps the application module to a reporting module) to export and places this export in a single connection.

Based on this strategy, the following exporting strategies can be used.

Strategy A

Within strategy A the application module consists of service calls that export using two mappings to different reporting modules. Each export is configured on another connection as well.

One use case immediately comes to mind here: the two exports, the two connections, the two reporting modules and the two datamappings split the entire application module in two and are completely disjunctive. For instance, one is about a health insurance and the other about a mortgage application. However, good practice would probably not have allowed to model this in a single application module.

Strategy A fits better in the following use case: Different parties want different parts of the entire application domain.

The first export could very well be the entire export, as described in the default strategy, whereas the second export could be a subset with for instance only data about persons and not their permit requests. It is most likely that another party is interested in this subset (hence the different connection) and because of the fact that it is not 100% sure that it will always remain a subset, it is wise to create another reporting module in this particular case.

Strategy B

Within strategy B, a single connection will receive all exports. From the point of view of the receiving party however, there are two types of export, therefore there are two reporting modules. This is almost the same as strategy A, with the exception that both exports are picked up by the same external party on the same connection. From the application's point of view, there are two types of exports: one could be containing all health insurance requests and one could consist of dental insurance requests. If the underlying domains for health and dental are very much the same, strategy C should be used rather than this strategy.

Strategy C

Within strategy C, a single connection will receive all exports. From the point of view of the receiving party, there is only one export, since there is only one reporting domain, opposed to strategy B. Within this single reporting domain, two distinct flavors can be distinguished. From the application's point of view, there are two types of exports: one could be containing all health insurance requests and one could consist of dental insurance requests. If these domains differ a lot, strategy B should be used rather than this strategy.

Strategy D

Strategy D resembles the default strategy very much. Within an application module, a service is called that creates an export using a datamapping (that maps the application module to a reporting module). Instead of placing it in a single connection, it places the export in multiple connections. However, since the connection can only be configured once, two export services are created, both of them point to the same datamapping but use different connections.