Why a new version management system?

Over the last years, Blueriq Studio has been used in more demanding environments where larger models are built with more people that collaborate on a project. The SDF-based version management system has shown to be reaching its limits under those circumstances, prompting to periodically trim the revision history to avoid performance problems. Additionally, the versioning concept was so deeply integrated into Studio that operations that are independent of historic data still had to deal with the versioning aspects, which resulted in complexity and scalability constraints.

In order to overcome these constraints, we set out to separate out the version management aspect into two distinct concepts: historic data and a "workspace" concept in which pending changes are collected before they are committed as historic data. This separation allows for most operations to be done without taking historic data into account, thereby avoiding the performance penalty that would become worse over time as more revisions were being registered.

About the new version management system

The new version management system contains many improvements over the current (SDF-file based) system. Some of these improvements have been described below.

Improved performance

You can enjoy improved speed when performing operations with the version management system: among others, performance when storing revisions, creating branches and merging them has been greatly improved. There are two major changes in how the version management works that allow for this improved speed:

• Data is now grouped by revision instead of element type once it is committed, allowing for increased performance when retrieving revisions.
• This new storage method also allows for a reduction in the amount of blocking operations (i.e. operations that cannot be performed in parallel, locking out all other users of Studio when such an action is performed). These operations, such as merging and deleting branches, now only need to access the data it directly affects: unrelated data can still be accessed.

Compatibility with Blueriq products

Storage in the new version management system is content agnostic, allowing it to store the elements used in new and upcoming products. One of these is the Blueriq Model Mapper, which introduces data mapping unit tests.

Furthermore, the history server component that is part of this new version management system does not need to be updated to support new elements; resulting in smaller updates in the future.

Compatibility with Linux

The newly introduced components that store historical data (i.e. the History server and PostgreSQL) are Linux compatible, allowing these components to be hosted on recent distributions of Linux. Support for Microsoft Windows is of course still present.

What changes for Studio Users

For users of Blueriq Studio few things will change. The most notable changes are the speed when performing tasks related to versioning and schema migrations are now done the moment a project is opened.

Performance increase

With regards to performance, users may notice a performance increase in many operations. This is a result of the new version management system, which should avoid the performance penalty for large sets of revisions; allowing for more revisions to be committed. Furthermore, it performs better with concurrent usage: performing a large merge operation or deleting a branch will not delay the actions of other users. For example, when one user deletes an unused branch, other users can keep working in other branches without delays.

Datamigrations visible

Automatic upgrades to model data are now done when a workspace is opened for the first time after upgrading to a new version of Studio. Technically, when Studio is upgraded to a newer version, the data structure of elements in a workspace may need to be migrated (i.e. store project data in a different way to make it compatible with a new version of Blueriq Studio). This migration used to take place during the installation of the new version of Studio, affecting all revisions across all repositories in the SDF database. In contrast, this migration is now done when opening a workspace that was last opened in an older version of Studio, without affecting the committed revision at all. Instead, a migration will store any updates that have to be made as a set of pending changes, so that it becomes visible which data has been affected by a migration. These changes will not interfere with any pending changes that existed before the upgrade, but are required to be included in the earliest commit. If necessary, migrated elements can be updated as desired before committing them.

Revisions no longer have a branch name

With the new versioning system, a revision no longer has the name of the branch on which it was committed.

Tags have to be unique

Previously, each revision could have a set of tags associated with it, allowing for tags to be reused across multiple revisions. This has changed with the new versioning system, in which tags uniquely refer to a single revision. As a result, tag names have to be unique. If you were using tags with the same name, we recommend that you introduce a numeric suffix to avoid duplicate tags.

What to know before migrating

There are several things to keep in mind before migrating to the new version management system, these are listed below:

One-way migration

This migration is a one-way procedure: once you have migrated your data from the SDF-file based solution to the new versioning system, it cannot be migrated in the opposite direction. It is highly recommended to keep a backup of the SDF so that it may still be reverted to, but keep in mind that any changes that were applied to the model in the new version management system will be lost.

Backups will change

Data that needs to be backed up will change, requiring changes in any backup scripts. See the section After the migration for more details.

Or see Studio#Creatingbackups for the manual.

Upgrade timely

Migration is available from Blueriq 12.9 onwards and can be scheduled at your own convenience. Note, however, that support for the current version management system will be dropped with Blueriq 13.0.

Prepare your migration

While the migration is in progress, projects can not be edited through the Blueriq Studio server. Additionally, there is some downtime. Prepare for this accordingly. See Preparing your migration below for more details.

Tags are reformatted, unique and unordered

With the new version management system there have been a couple of changes to how tags work:

1. Duplicate tags are not allowed anymore
2. Tags can only consist of characters, numbers, dots, underscores and hyphens.
3. Tags are no longer sortable.

The migration tool will deduplicate and reformat the tags for you. Duplicate tags are suffixed with an underscore and a follow up number, for example `tag, tag_1`. Tags which are invalid by the new formatting standard will be reformatted. Tags can only be composed out of characters, numbers, dots, underscores and hyphens. Each part of the tags that violates the format will be replaced with an underscore, for example `tag 1` will become `tag_1`

Data is secure

The migration tool provides a verification step, making sure that all revisions for all branches are migrated properly. Tags are also checked. This will ensure you that once the migration is complete, all data is copied to the history server correctly. See Steps performed under The Studio SDF Migrator for more information.

Allocate enough time for your migration

The time required to perform this migration will depend on both your server hardware and the size of your repositories (both in the number and size of the projects therein and the number of revisions that are stored.) Therefore, it is not possible to predict how long the migration process will take. However, a typical server should be able to process an average of 1.000 revisions per hour.

Note that this process should not be interrupted. If it is, the migration cannot be resumed and will have to start over from the beginning.

Preparing your migration

Upgrade to Blueriq 12.9 or later

The new version management system is available in Blueriq versions 12.9 or later, and a requirement for both performing the migration and using it. Please refer to the upgrade instructions below for more details on upgrading.

It is not possible to upgrade directly to Blueriq 13 and only then perform the migration, as that version is unable to read the SDF database. The migration tool only accepts SDF versions that have at least been opened in Blueriq 12.9 or higher to ensure that all model migrations have been applied, so one should first migrate to the latest Blueriq 12 release to run any final model migrations in the SDF database, before being able to perform the migration.

It is however possible to run the migration tool on an SDF database and targeting the history service that was installed as part of Blueriq 13.

Downtime during migration

As the history store for the Blueriq Studio environment will be replaced during this migration, no changes to any of the projects in the repositories may be made during this process. Furthermore, the Studio environment will need to be taken offline to change its configuration.

Therefore, it is recommended to select a timeslot to perform this migration beforehand and communicate to all users of the Studio environment that no changes are to be made during this process to prevent data loss. Any changes that are made to projects in Studio after the process has started may not be migrated to the new history server. If there are changes made during this process that do need to be copied into the history server, the migration will need to be run again.

Resolving pending changes

In contrast with upgrading Blueriq, this migration also requires that there are no pending changes on projects, as these cannot be migrated. Users of Blueriq Studio will need to either commit or revert all their pending changes. The migration tool has a built-in check for this as well: if there are any pending changes left on one or more branches, the migration cannot be performed.

The Studio SDF Migrator

This migration is aided by the Studio SDF Migrator (hereafter referred to as the Migrator), a specialized Windows-compatible tool that will perform the actual data migration for you. Download and extract it to a folder on the server that you want to migrate.

Steps performed

The migrator performs its work in two phases per repository: migration and verification. During the first phase, all repositories and branches (and the data therein) are read from the configured SDF-file, transformed, and transferred to the provided history server. After this process is complete, the verification phase makes sure that all transferred revisions contain a complete set of changes. Furthermore, the full revision content of branches and tags is verified.

This two-phase approach allows you to be sure that all data contained in the SDF-file has been transferred to the history server correctly.

Update backup process

As the data is no longer stored in a single file on disk, a backup process in which this file was simply copied into a backup location will no longer be applicable. For more information on the required changes, please refer to the After the migration section.

Performing the migration

Configure the migration tool

Configuration of this tool is done through a configuration file named StudioSDFMigrator.config, which can be found in the directory the Migrator was extracted to. In this file, two properties need to be changed: under the <migration>-element:

• <datasource>: the location-attribute will need to refer to the SDF-file containing the repositories you want to migrate.
• <versioningServer>: the url­-attribute needs to point towards the endpoint containing the history server. This endpoint is provided when the History server component is selected when installing Blueriq 12.9 or later. By default, this value is: http://localhost:12095.

Your configuration file may now look like this:

<migration>
    <datasource location="C:\Program Files\Blueriq 12\Studio\Repository\Repository.sdf"/>
    <versioningServer url="http://localhost:12095"/>
</migration>

Configure the history server

During the SDF migration, the history server has to be configured to have authorization checks disabled. To accomplish this, navigate into the history server installation directory in [Blueriq installation]/Services/history-server/BlueriqHistoryService.xml and add an additional argument:

<argument>--enableSecurity=false</argument>

Then restart the "Blueriq History Service" so that the change is taken into account.

HINT: 'Task manager' (Ctrl + Shift + Esc) > 'Services'-tab > Right click on 'Blueriq 12 History Service' > Restart

HINT 2: Run the tool on the command line to see the output.

HINT 3: If you don't configure the history server, or don't restart the Blueriq 12 history server after changing the history server configuration, you will probably see this in the output (truncated):

"StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:"
AND "{"errors":[{"status":"401","title":"Not authenticated","detail":"No access token found"}]}"

Ensure no activity is performed in Studio

As stated under Preparing your migration, changes made to models while the migration is running might not propagate to the history server. Ensure that there are no users of Blueriq Studio at this moment. The administration page provides a feature to broadcast a message to all logged in users and log them out by hand.

Check that required services are running

This migration requires only the History server to be running. Start this service if this is not the case, allowing some time to properly start up and receive incoming connections. Note that the Blueriq Studio Service should not be running, as the SDF to migrate should not in use at the time of the migration.

Backup configuration and repositories

This migration requires the SDF-file containing repositories and projects to be read, and the Blueriq configuration file of your installation to be edited. Back up both to a secure location, to allow reverting back to a pre-migration state.

Take the following steps to backup:

• Repositories and projects: create an export using the Blueriq Studio administration page.
• Blueriq Studio configuration file: copy it to a secure location; by default, it can be found at: C:\Program Files\Blueriq 12\Studio\Services\StudioService.exe.config.

Run the Migrator

At this point all preparations have taken place, and the actual migration can be initiated. Within the directory where the Migrator tool has been extracted to, execute the run.bat script. If the SDF to migrate is contained within the Program Files directory, the script must be run as Administrator to be able to read the SDF file. As the tool takes all its input from the corresponding configuration file no further input is required during this process.

As stated in Preparing your migration, the Migrator can process revisions at a rate of about 1.000 revisions per hour on typical hardware. Output is shown on the screen while the migration takes place, indicating the repository that is currently being processed as well as the phase (migration or verification). The tool will indicate when the migration has been completed successfully. Be sure not to interrupt the process while it is running, as you will need to redo the complete migration.

Restore history server configuration

After the migration has successfully completed, the history server configuration has to be reverted back to re-enable authorization checks. As such, open [Blueriq installation]/Services/history-server/BlueriqHistoryService.xml and remove the <argument>--enableSecurity=false</argument> that was added before the migration. Once this has been done, the History service should be restarted for the change to be applied.

Configure Blueriq Studio to utilize the History server

At this point, all repositories stored in the configured SDF file have been migrated to the history server. In this final step, Blueriq Studio will be configured to use the History server to retrieve and store revisions instead of the SDF-file. This is done by editing its configuration file.

Stop the Blueriq Studio service and open the configuration file named StudioService.exe.config, which can be found in the Studio installation directory (by default, this is: C:\Program Files\Blueriq 12\Studio\Services\).

Open the configuration of the Studio service in [Blueriq installation]\Studio\Services\StudioService.exe.config and adjust the configuration according to the steps described in switching versioning system. Once this has been done, the Blueriq Studio service can be started again and the migrated repositories should be visible in the Studio client.

From this point on, the SDF-file is not used anymore. It can be kept on the server to quickly revert to a state just before the migration.

After the migration

Once the Blueriq Studio service has been started again, the migration is complete. However, backups will need to be reconfigured, as the way data is stored has been changed.

In previous versions of Blueriq, all Studio model data was stored inside of Repository.sdf. This was convenient with respect to backups, as a single file copy was sufficient to backup all data. With the transitioning to a dedicated storage environment for history data, however, the situation is no longer that simple. Data related to history is now stored separately from the working revision data, such as pending changes. To ensure that data is backed up consistently and safely, an integrated backup strategy is offered to achieve single file backups once again.

The Management Service of Blueriq Studio has operations to backup and restore all Blueriq model data into a single archive. This operation will request a backup for each Blueriq repository from the history server and combines it with any workspace data, such as pending changes and open conflicts. The result of executing this operation is a single file that can be backed up in the same way as was possible with the Repository.sdf file. Restoring of such backup files also need to happen through the Management Service, which will then ensure the data is restored into the appropriate location.

As backups are typically scheduled and executed using simple batch/bash scripts, having to interact with a SOAP service is not ideal. To alleviate this, a CLI application is provided that performs the necessary Management Service operations to write the backup file into a dedicated location, so that it can easily be backed up as desired. More information on this tool can be found in the Studio installation guide. Or go directly Studio#Creatingbackups

Management Service changes

The Management Service continues to offer the operations related to the versioning system, however there are some minor discrepancies in its behavior.

Revisions

Most of the changes can be found in the Revision type that is returned from GetRevision, GetRevisionDescriptor, GetRevisions and GetRevisionsWithLeveledMergeRevisions. The following difference can be observed:

• Branch will always be empty, as revisions no longer have a branch name associated with them
• ParentRevisionIds will always be empty. Instead, the primary parent revision id can be found in Parent, whereas a merge parent is provided in MergeParent.
• MergeRevisions will always be empty. Previously, fetching revisions would return a whole tree of merged revisions and the specialized GetRevisionsWithLeveledMergeRevisions could be used to limit the number of levels to include in the response. This has proven to be problematic for large revision graphs, due to extensive amounts of data and nesting within the response. This also means that GetRevisionsWithLeveledMergeRevisions now behaves identically to just GetRevisions.

Branch status

The operations to fetch details about a branch, GetStatus and GetStatusMaxDepth, are slightly altered:

• WorkingRevision will always be empty, as there is no longer a working revision. Instead, we refer to the place where pending changes accumulate as the workspace, where a single branch always corresponds with a single workspace. This change means that it is no longer possible to refer to the current workspace using its working revision id. Additionally, the revision id is not known until the pending changes in the workspace have been committed.
• MergeRevisions is mostly the same as before, except for the changes as listed under "Revisions" above. This list will now only be a linear list of revisions that are being merged into the branch. Previously, these revisions would all refer to their merge revisions, forming a tree of revisions. This is why the operation GetStatusMaxDepth was present, in order to reduce the number of nested merge revisions. As a result, the GetStatusMaxDepth operation behaves identically to GetStatus. 

Disclaimer

The history service internally uses the Git version control system to manage its revisions and data. This, however, should be considered an implementation detail that must not be relied upon. It is not supported to manually interact with the Git objects, as doing so can result in data that the Studio is unable to process.

Furthermore, the REST API offered by the history server should not be considered as public API at this moment. This means that endpoints and request/response formats may change at any time.