Digital Identity and Profile Migration Guide

Intellectual Property

Protected by patents in the U.S. and Europe:

US10430037, EP2684143, US8365188, EP1997041, US8880710, EP1979840, US8898309, EP1979839

Compares the Identity Graph and Profile Builder to the functionality provided by legacy Profile Server, and presents strategies for migration.

1. Introduction

Celebrus 9.4 introduced the Identity Graph and Profile Builder, technologies which supersede the legacy Profile Server. These new technologies provide significant flexibility in identity recognition, reconciliation and building of profiles.

This guide compares these technologies, the advantages the new generation of technology brings and how to go about migration.

The legacy Profile Server remains supported in Celebrus 8.x and Celebrus 9.x as per the Celebrus Support Policy.

2. Comparison to Profile Server

Table 1. Identity Graph vs Profile Server
ASPECT	IDENTITY GRAPH	PROFILE SERVER
What is the out of the box behaviour?	Does nothing. No assumptions are made as to how the system should behave. For example, some users never wish to use anonymous / assumed identity based on device identifier.	Creates device profiles, based on the device identifier.
Support for organisations with differing user identities per brand/channel?	Yes	No, has a single user identity, known as a `ProfileUiid`.
Supports real-time processing	Yes	Yes
Supported databases	Apache Derby, MariaDB, Microsoft SQL Server, MySQL, Oracle, Oracle OCI, PostgreSQL, Teradata	MySQL, MariaDB, Oracle, Oracle OCI
Redundancy	Yes, each Real-Time Server provides Identity Graph processing.	No, all processing performed by single Profile Server.
Supports sophisticated identity reconciliation with priority ordering?	Yes	No

Table 2. Profile Builder vs Profile Server
FEATURE	PROFILE BUILDER	PROFILE SERVER
Allows profile to be tailored to your requirements?	Yes	No, profile has a fixed structure.
Supports real-time	Yes, real-time processing with identity updates written immediately, and profile updates written to database at end of session (eventual consistency).	Yes
Event	`ProfileFeatures`. One Profile Builder configuration can be nominated to produce `HistoricEvent`, rather than Profile Server. If no Profile Builder configurations are nominated the Profile Server generates `HistoricEvent`.	`HistoricEvent`
Supported databases	Apache Derby, MariaDB, Microsoft SQL Server, MySQL, Oracle, Oracle OCI, PostgreSQL, Teradata	MySQL, MariaDB, Oracle, Oracle OCI
Redundancy	Yes, each Real-Time Server provides Profile Building processing.	No, all processing performed by single Profile Server.
Supports session level data?	Yes, profile consists of sessions and their attributes (features).	No, profile is a summary.
Supports personalisation use cases targeting bespoke time frames?	Yes	No
Supports persistent (sticky) profile attributes (features)?	Yes, Celebrus 9.5 introduces Sticky Sessions which allows features such as Name which are unlikely to ever change, to be kept forever.	Yes

3. Prerequisites

Existing Profile Server database.
Installed / upgraded v9.6 or later Profile Server
Existing Identity Graph database with schema and users in place
Existing Profile Builder database and schema and users in place
JDBC driver jars for the source and target database types
Copy of the current EventPlugin.xml file from the resources folder of a Real-Time Server

4. Migration process

The migration process is not like upgrading from one Celebrus version to another, where the Semantic configuration is upgraded and you carry on much as before, albeit with increased functionality. Identity Graph and Profile Builder are a different paradigm to Profile Server. You need to understand the data you are currently capturing to Profile Server and you need a good grasp of your identity requirements and the data you want to store in Profile Builder.

The migration tool can migrate an existing Profile Server database to Identity Graph only, or Identity Graph and Profile Builder. It is recommended that the Identity Graph schema and Profile Builder schema are separate schema, but the can be held in the same database instance. Any database type supported by Identity Graph or Profile Builder can be used to hold the Identity Graph and Profile Builder schemas. The Profile Sever database will be in MySQL / MariaDB or Oracle.

The migration tool makes a best effort attempt to migrate the data in the Profile Server database to Identity Graph and Profile Builder databases. It does not attempt to fix any inconsistencies it finds in the data. Nor does it offer any facilities to purge data during the migration.

The migration tool can be run in parallel with Profile Server and the Identity graph and Profile Builder data enrichments.

The migration tool may take many hours to complete when run on large Profile Server databases. It should be executed in an environment where the session will not be terminated.

The migration tool produces a date and timestamped parameters file each time it is run, this would enable a run to be restarted if necessary.

4.1. Strategies

4.1.1. Parallel running with Profile Server

Pros: With this strategy, the Identity Graph and Profile Builder Data Enrichments are enabled and running in parallel with Profile Server. If there is a problem with the migration, Profile Server is still collecting data. Faster migration as the tool is not competing with Profile Server for database resources. New Identity and Profile data goes directly to the Data Enrichments.

Cons: While the Profile Server is active in the cluster, the Real-Time Server will make historic profile requests to Profile Server in preference to Profile Builder.

4.1.2. Decommission Profile Server before migration

Pros: New Identity and Profile data goes directly to the Data Enrichments. Historic requests for profile data are satisfied by Profile Builder. Performance of the migration will be greater when Profile Server is not running.

Cons: Profile Server must be decommissioned and removed from the cluster. If there is a problem with the migration process, then Profile Server is not up-to-date.

5. Performing a Migration

5.1. Planning the migration

5.1.1. Understand the current Profile Server data

Before running the Migration tool, you need a through understanding of the data captured in Profile Server for each individual.

What data is captured as the UIID?
Do you capture name information?
Do you capture address information?
Do you capture telephone numbers?
Do you capture email addresses?
Do you capture gender?
Do you capture date of birth?

5.1.2. Plan the Identity Graph migration

You need to understand the data items that you want to use as identity elements in Identity Graph.

It makes sense that the Profile Server UIID value is used as a defined user identity element and the Profile Server cookie id is used as a advisory device identity element. However you can use this opportunity to move to a different identity model, while still migrating the existing identity data to advisory identity data.

If you capture email addresses or telephone numbers, do you want those to be used as defined user identities or as advisory identity elements?

Do you want to migrate all Profile Server cookie id’s for an individual to advisory user device identities in Identity Graph, or only the most recent cookie id?

See the Digital Identity User Guide for a detailed description of identity and how to configure the Identity Graph enrichment.

The information you gather here will be used to form your ProfileToIdentityMap.properties file.

5.1.3. Plan the Profile Builder migration

Looking at the personal information for an individual already captured in Profile Server, how do you want to map this to Profile Builder features?

Some information has an obvious translation, for example, title, first name, last name, company name and date of birth information. For other information, like addresses, are they home addresses, delivery addresses or work addresses? The same question for telephone numbers and email addresses. Do you want to migrate information held in Profile Server generic attributes, or CSV loaded attributes to Profile Builder features?

Do you need to create any custom Profile Builder features to hold existing data, or is the default Profile template sufficient?

See the Digital Profile User Guide for a detailed description of Profile features and how to configure the Profile Builder enrichment.

The information you gather here will be used to form your ProfileToProfileBuilderMap.properties file.

5.2. Setup the option and mapping files

Gather together the source and target database details and JDBC driver jars.

Put a copy of the source and target JDBC driver jars in the <install_dir>/ProfileServer/Migrationtool/ folder.

Edit the template IdgProfileMigrationTool*.properties file for your source database type (Oracle or MySQL / MariaDB) and enter the required information.

Add the details for the target Identity graph and Profile Builder database instances.

Add any other options that you require to the end of the file, for example --processAllCookieId=false. Refer to section Command line options for details of all available options and their defaults.

Use the information gathered in Plan the Profile Builder migration to construct your mapping files.

5.3. Running the migration tool

The Migration tool can be run from a Linux or Windows command shell. The migration may take several hours, or even days running in parallel with Identity Graph and Profile Builder.

The Migration tool should be run in an environment where the tool will not be interrupted.

If there are any command line options you do not want to add to the properties file, such as database passwords, they can be added to the command line after the options filename. They should also be removed from the options file before running. By default the migration tool will not add passwords to the generated options file.

Run the migration tool from the command line using the following commands.

Windows:

D:\Installs\Celebrus9.6\ProfileServer\MigrationTool> IdgProfileMigrationTool.bat @IdgProfileMigrationToolMySQL.properties [any properties you do not want to put in the options file]

Linux:

/Installs/Celebrus9.6/ProfileServer/MigrationTool$ IdgProfileMigrationTool.sh @IdgProfileMigrationToolMySQL.properties [any properties you do not want to put in the options file]

By default the migration tool logs warnings and errors to the console and a log file. Normal informational level logging is sent to the log file by default. The amount of logging information can be customised using the log4j2.xml configuration file.

6. Migration tool reference

6.1. Configuration

The Windows batch and Linux shell script files to run the migration contain the following definition that only applies when the source or target databases are Oracle. If your Oracle system is still configured to use a database SID rather than a service name, change the value to false before running the migration.

 -Dcelebrus.use.oracle.serviceName=true

By default the source and target database driver jar files are loaded from the <install_folder>/ProfileServer/MigrationTool folder. The classpath can be changed in the script if required, but the default is to add all jars in the folder to the classpath.

The Migration tool is installed with four default properties files. Two example input property files:

IdgProfileMigrationToolMySQL.properties
IdgProfileMigrationToolOracle.properties

that can be modified to suit your environment.

Along with two example mapping files:

ProfileToIdentityMap.properties
ProfileToProfileBuilderMap.properties

that are detailed in the following sections.

6.1.1. Profile Server to Identity Graph mapping file

This file specifies the mapping from Identity Graph identity entry to Profile Server attribute. The Identity Graph identity entry is validated against the provided eventPlugin.xml file to ensure it exists. The Profile Server attribute is validated against the standard, fixed, set of Profile Server attributes.

# Identity Graph identity=Profile Server attribute
uiid=uiid
email=emailAddress1
phone=telephoneNumber1
device=cookieId

6.1.2. Profile Server to Profile Builder mapping file

This file specifies the mapping from Profile Builder features to Profile Server attributes. The Profile Builder Feature is validated against the provided eventPlugin.xml file to ensure it exists. The Profile Server attribute is validated against the standard, fixed, set of Profile Server attributes.

# Profile Server attribute=Profile Builder feature
Behaviour=BEHAVIOURS.Behaviour
Name.Title=PROFILE.ProfileTitle
Name.FirstName=PROFILE.ProfileFirstName
Name.LastName=PROFILE.ProfileLastName
DateOfBirth.Day=PROFILE.ProfileDayOfBirth
DateOfBirth.Month=PROFILE.ProfileMonthOfBirth
DateOfBirth.Year=PROFILE.ProfileYearOfBirth
Address1.Line1=HOME.HomeFirstLineOfAddress
Address1.City=HOME.HomeCity
Address1.Postcode=HOME.HomePostCode
EmailAddress1.EmailAddress=HOME.HomeEmailAddress
TelephoneNumber1.TelephoneNumber=HOME.HomeTelephoneNumber
GenericAttribute1.Description=GOALS.Goal
GenericAttribute1.Value=GOALS.Goal
CsvAttribute1.Description=GOALS.Goal
CsvAttribute1.Value=GOALS.Goal

The standard, fixed, set of Profile Server attributes includes the following:

Behaviour

MultiUser

Gender

AccountNumber

Name.Title
Name.JobTitle
Name.FirstName
Name.MiddleName
Name.LastName

DateOfBirth.Day
DateOfBirth.Month
DateOfBirth.Year

Address1.Description
Address1.Company
Address1.Line1
Address1.Line2
Address1.Line3
Address1.Line4
Address1.City
Address1.Region
Address1.Postcode

Address2.Description
Address2.Company
Address2.Line1
Address2.Line2
Address2.Line3
Address2.Line4
Address2.City
Address2.Region
Address2.Postcode

Address3.Description
Address3.Company
Address3.Line1
Address3.Line2
Address3.Line3
Address3.Line4
Address3.City
Address3.Region
Address3.Postcode

EmailAddress1.Description
EmailAddress1.EmailAddress
EmailAddress2.Description
EmailAddress2.EmailAddress
EmailAddress3.Description
EmailAddress3.EmailAddress

TelephoneNumber1.Description
TelephoneNumber1.TelephoneNumber
TelephoneNumber2.Description
TelephoneNumber2.TelephoneNumber
TelephoneNumber3.Description
TelephoneNumber3.TelephoneNumber

GenericAttribute1.Description
GenericAttribute1.Value

to

GenericAttribute10.Description
GenericAttribute10.Value

CsvAttribute1.Description
CsvAttribute1.Value

to

CsvAttribute50.Description
CsvAttribute50.Value

6.2. Command line options

Using the command line options -h, --help, or running the tool with no arguments will provide the following output:

Usage: IdgProfileBuilderMigrationTool
                    [-hV]
                    --sourceDatabaseType=<sourceDatabaseType>
                    --sourceDatabaseHostname=<sourceDatabaseHostname>
                    --sourceDatabasePort=<sourceDatabasePort>
                    --sourceDatabaseName=<sourceDatabaseName>
                    --sourceDatabaseUser=<sourceDatabaseUser>
                    --sourceDatabasePassword=<sourceDatabasePassword>
                    --identityDatabaseType=<identityDatabaseType>
                    --identityDatabaseHostname=<identityDatabaseHostname>
                    --identityDatabasePort=<identityDatabasePort>
                    --identityDatabaseName=<identityDatabaseName>
                    --identityDatabaseUser=<identityDatabaseUser>
                    --identityDatabasePassword=<identityDatabasePassword>
                    --profileDatabaseType=<profileDatabaseType>
                    --profileDatabaseHostname=<profileDatabaseHostname>
                    --profileDatabasePort=<profileDatabasePort>
                    --profileDatabaseName=<profileDatabaseName>
                    --profileDatabaseUser=<profileDatabaseUser>
                    --profileDatabasePassword=<profileDatabasePassword>
                    [--startRecencyTimestamp=<startRecencyTimestamp>]
                    [--endRecencyTimestamp=<endRecencyTimestamp>]
                    [--batchSize=<batchSIze>]
                    [--startProfileId=<startProfileId>]
                    [--endProfileId=<endProfileId>]
                    [--processAllCookieId]
                    [--migrateToProfileBuilder]
                    [--eventPluginConfigurationLocation=<eventPluginConfigurationLocation>]
                    [--identityMappingFile=<identityMappingFile>]
                    [--profileBuilderMappingFile=<profileBuilderMappingFile>]
                    [--includePasswordInAtFile] [@<filename>...]

[@<filename>...]       One or more argument files containing options.

-h, --help                 Show this help message and exit.

-V, --version              Print version information and exit.

--sourceDatabaseType=<sourceDatabaseType>
                       Source Profile Server database type: MariaDB, MySQL, Oracle, OracleOCI

--sourceDatabaseHostname=<sourceDatabaseHostname>
                       Source Profile Server database hostname

--sourceDatabasePort=<sourceDatabasePort>
                       Source Profile Server database port

--sourceDatabaseName=<sourceDatabaseName>
                       Source Profile Server database name

--sourceDatabaseUser=<sourceDatabaseUser>
                       Source Profile Server database user

--sourceDatabasePassword=<sourceDatabasePassword>
                       Source Profile Server database password

--identityDatabaseType=<identityDatabaseType>
                       Identity Graph database type: Derby, MariaDB, MySQL, Oracle, OracleOCI, PostgreSQL, SQL Server, Teradata

--identityDatabaseHostname=<identityDatabaseHostname>
                       Identity Graph database hostname

--identityDatabasePort=<identityDatabasePort>
                       Identity Graph database port

--identityDatabaseName=<identityDatabaseName>
                       Identity Graph database name

--identityDatabaseUser=<identityDatabaseUser>
                       Identity Graph database user

--identityDatabasePassword=<identityDatabasePassword>
                       Identity Graph database password

--profileDatabaseType=<profileDatabaseType>
                       Profile Profile Builder database type: Derby, MariaDB, MySQL, Oracle, OracleOCI, PostgreSQL, SQL Server, Teradata

--profileDatabaseHostname=<profileDatabaseHostname>
                       Profile Profile Builder database hostname

--profileDatabasePort=<profileDatabasePort>
                       Profile Profile Builder database port

--profileDatabaseName=<profileDatabaseName>
                       Profile Profile Builder database name

--profileDatabaseUser=<profileDatabaseUser>
                       Profile Profile Builder database user

--profileDatabasePassword=<profileDatabasePassword>
                       Profile Profile Builder database password

--startRecencyTimestamp=<startRecencyTimestamp>
                       Only migrate profiles after this recency timestamp. String format "yyyy-MM-dd HH:mm:ss"

--endRecencyTimestamp=<endRecencyTimestamp>
                       Do not migrate profiles after this recency timestamp. String format "yyyy-MM-dd HH:mm:ss"

--batchSize=<batchSIze>
                       Number of source profiles to process in each batch, Default 5000 profiles.

--startProfileId=<startProfileId>
                       Start migrating records from this profile id. Default 1

--endProfileId=<endProfileId>
                       End migrating records at this profile id. Default -1, process all profiles.

--processAllCookieId   If true process all cookie id records for an individual. Default true

--migrateToProfileBuilder
                       If true migrate data to Profile Builder as well as Identity Graph. Default true

--eventPluginConfigurationLocation=<eventPluginConfigurationLocation>
                       Event Plugin configuration file location

--identityMappingFile=<identityMappingFile>
                       Mappings between Profile Server fields and defined identities

--profileBuilderMappingFile=<profileBuilderMappingFile>
                       Mappings between Profile Server fields and Profile Builder attributes

--includePasswordInAtFile
                       If true source and target passwords are included in the generated @file output. Default false

7. Frequently asked questions

Q. Can I run both Identity Graph and Profile Builder alongside the Profile Server?: Yes, see Strategies for further information.
Q. When running both Profile Builder alongside Profile Server, which controls HistoricEvent?: A Profile Builder can be set to manage system-wide audit requests and historic data events. This is configured within the Advanced settings of the Profile Builder configuration.
Q. Profile Builder writes data when a session expires. What happens if another session occurs in the meantime?: If the second session is processed on the same Real-Time server, the in-memory state is an accumulation of both sessions. If the second session is processed on a different Real-Time server, the in-memory state on each server represents a single session.
Q. Profile Builder has settings to control frequency of updates, with the default being five seconds. How does this relate to the end of session writing?: This setting controls only the production of the ProfileFeatures event within the event stream. It has no impact on the writing of profiles.
Q. I have an existing RTIM integration based on the Profile Server. How can this be migrated to Identity Graph and Profile Builder?: Your nominated Profile Builder configuration can populate the state sent to RTIM. Your existing ProfileUiid scenarios containing the user identity remains in use, allowing RTIM to lookup further data based on this.
Q. Should I be using unauthenticated, guessable identifiers as part of my Identity Graph’s User Identities, or should I be using them within IdentityGraph scenarios only to merely relate to identities?: Using guessable, non-authenticated identifiers in any part of identity graph is not recommended as this would allow users to impersonate others.
Q. How do I migrate to Identity Graph from the Profile Server, where I’ve been using UserIdentification events?: Configure Profile scenarios which match on SystemIdentity events, setting the UIID to the required identity from the UserIdentities table..
Q. Is it possible to determine a visitor’s identity based purely on a biometric signature and nothing else, not even a device identifier?: No. Identity, be it a device cookie or explicit login can be verified using biometric identity to determine a level of confidence in identity, but it can’t be used as the primary identification mechanism.
Q. If a user on one device identifies with email address, then on a different device identifies themselves first using a username, then on an email address, two system identities exist, one with email, the other with username. Why doesn’t username get snapped in to the existing system identity?: Identity Graph can only relate identities if there is a common identity in play. In the example above only if the second identification on the second device contained username in addition to email, would Identity Graph be able to determine a relationship, otherwise it sees username and email as belonging to separate individuals, hence two system identities.