Find duplicates step - Advanced configuration

Data Studio provides default blocking keys and rulesets that you can use to run the Find duplicates step. However, you can create your own custom rules and blocking keys that are tailored to your needs. 

Blocking keys and rulesets are designed and specified using elements - a representation of your input data after it has been mapped in the Find duplicates step and gone through the initial standardization process. Find out about elements. 

The standardization process also creates additional versions of these elements to assist further - known as modifiers. Modifiers can correct, enhance or derive many known terms that appear in the input. For example, a DERIVED modifier may be created when the element was not contained in the input but the standardization process was able to determine the value (e.g. COUNTY can sometimes be derived from the LOCALITY and POSTCODE input). Find out about modifiers.

Elements can also be put into specific groups to separate them from other elements of the same type. This is especially important when you want to create blocking keys or rules that treat them as separate entities, such as cross-field matching. 

These elements, modifiers and groups (along with blocking key algorithms and rule comparators) can then be combined to make your own blocking keys and rules specific to your design and data.

The table below covers the available elements that can be used when designing blocking keys and rules, together with the possible modifiers and rule comparators that can be used with them. Note that blocking key algorithms are not listed here because they apply to all elements.

The table below covers the available element modifiers that can be used. See the elements table to find out which modifiers can be used with which elements.

Any element can be used multiple times in an input to represent separate or unique sets of information. For example, the input may include multiple:

• street numbers, street names and postcodes (e.g. a delivery and a billing address) 
• forenames and surnames (e.g. a primary account holder and a spouse)
• generic strings (e.g. an account and a customer reference)
• dates (e.g. a date of birth and a registration date)
• phone numbers (e.g. a cell phone number and a landline number)
• email addresses (e.g. a personal and a work email) 

Using groups in blocking keys

You may want to create different blocking key specifications for different groups of the same element type, or use the same blocking key specification for multiple groups of the same element type. To do this, you need to add elementGroups to your element specification within your blocking key design. Find out more about elementGroups.

For example, if you have a billing and a delivery address and you want to have two different blocking key specifications for them that have different designs, you would add the following to each address element of the group within your blocking key designs:

"elementGroups":["BillingAddress"]

"elementGroups":["DeliveryAddress"]

Alternatively, if you wanted to use the same blocking key specification for both groups, you would add the following to each address element of the group within that single blocking key design:

"elementGroups":[
    "BillingAddress",
    "DeliveryAddress"
]

The blocking key specification examples below are taken from the default blocking keys available in Data Studio, and have been amended to show how they would be used for the two cases above.

Two differently designed blocking key specifications for two different groups of the same element type:

{
    "description": "SurnameMinorStreetNumberLocality",
    "countryCode": "GBR",
    "elementSpecifications": [
      {
        "elementType": "SURNAME",
        "algorithm": {
          "name": "INITIAL"
        },
        "includeFromNChars": 1
      },
      {
        "elementType": "MINORSTREET_NUMBER",
        "elementGroups": ["BillingAddress"],
        "includeFromNChars": 1,
        "truncateToNChars": 5
      },
      {
        "elementType": "LOCALITY",
        "elementGroups": ["BillingAddress"],
        "elementModifiers": [
          "STANDARDSPELLING",
          "DERIVED"
        ],
        "includeFromNChars": 2,
        "truncateToNChars": 30
      }
   ]
}

{   
    "description": "POBoxNumberLocality",
    "countryCode": "GBR",
    "elementSpecifications": [
      {
        "elementType": "POBOX_NUMBER",
        "elementGroups": ["DeliveryAddress"],
        "includeFromNChars": 1
      },
      {
        "elementType": "LOCALITY",
        "elementGroups": ["DeliveryAddress"],
        "elementModifiers": [
          "STANDARDSPELLING",
          "DERIVED"
        ],
        "includeFromNChars": 2,
        "truncateToNChars": 30
      }
   ]
}

The same blocking key specification for two different groups of the same element type:

{
    "description": "FullPostcode",
    "countryCode": "GBR",
    "elementSpecifications": [
      {
        "elementType": "POSTCODE",
        "elementGroups": ["BillingAddress","DeliveryAddress"],
        "elementModifiers": [
          "STANDARDSPELLING"
        ],
        "includeFromNChars": 5,
        "truncateToNChars": 7
      }
   ]
}

Using groups in rules

You may want to handle the items listed above separately during rule evaluation. To do this, you need to prefix the rule with a hash symbol, followed by the group name in square brackets, followed by a period, followed by the rest of the rule.

For example, if you have a billing address and a delivery address and you want to evaluate them separately, you could write rules such as the ones below.

Match.L0 = {Name.L0 & #[Billing].Address.L0 & #[Delivery].Address.L0}
Match.L1 = {Name.L1 & (#[Delivery].Address.L1 | #[Billing].Address.L1)}

By default, this will mean that the two different groups will use the same underlying rules (in the case above, the same address rules). To use different rules for two groups that are of the same element type, you can use theme rules that are then utilized at the top level.

Match.L0={#[Billing].LooseAddressRule.L0 & #[Delivery].StrictAddressRule.L0}

LooseAddressRule.L0={Postcode.L0}
StrictAddressRule.L0={MinorStreet_Number & Postcode.L0}

MinorStreet_Number.L0={PremiseCompare[ExactMatch]}
Postcode.L0={PostcodeCompare[Part1Match] & PostcodeCompare[Part2Match]}

In the above example, two different theme rules have been used to evaluate an address (one being stricter than the other since it requires both the premises number AND postcode to be compared), but each is only used with one group as defined at the match level.

You can also specify more than one group in any rule, such as the example below. This will perform cross-field matching. 

Match.L0 = {Name.L0 & #[Billing,Delivery].Address.L0}

The Find duplicates step uses blocking keys to create blocks of similar records to assist with the generation of suitable candidate record pairs for scoring via a ruleset.

Blocks are created from records that have the same blocking key values. Blocking keys are created for each input record from combinations of the record’s elements that have been keyed. Keying is the process of encoding individual elements to the same representation so that they can be matched despite minor differences in spelling. To be effective, blocking keys should represent a range of contact data sub-element combinations.

Data Studio provides default blocking keys tuned for name and address matching for the Find duplicates step. However, you can modify these or create your own to suit your needs.

All blocking keys have to be built in the structure below.

"elementSpecifications":[
   { "elementType":"FORENAMES" },
   { "elementType":"SURNAME" },
   { "elementType":"MINORSTREET_NUMBER" }
]

Each element within elementSpecifications has to then be built in the structure below:

"elementGroups":[
   "HomeAddress",
   "WorkAddress"
]

"elementModifiers":[
   "STANDARDSPELLING",
   "DERIVED"
]

"algorithm": {
   "name" : "MIDDLE_SUBSTRING",
   "properties": {
      "START" : "2",
      "END" : "6"
   }
 }

The algorithms below are available for use in blocking key specifications, and apply to all non-numeric elements. If no algorithm is specified, the SIMPLIFIED_STRING algorithm is used.

CONSONANT and SOUNDEX support the following character sets:

• Basic Latin (ASCII)
• Latin-1 Supplement
• Latin Extended-A
• Latin Extended Additional

All other key types support the following Latin character sets:

• Basic Latin (ASCII)
• Latin-1 Supplement
• Latin Extended-A
• Latin Extended-B
• Latin Extended-C
• Latin Extended-D
• Latin Extended-E
• Latin Extended Additional
• IPA Extensions
• Phonetic Extensions
• Phonetic Extensions Supplement

Let’s use the default blocking keys in Data Studio for more advanced examples of blocking key design:

{
   "description": "FullZipCode",
   "countryCode": "USA",
   "elementSpecifications": [
     {
       "elementType": "POSTCODE",
       "elementModifiers": [
         "STANDARDSPELLING"
       ],
       "includeFromNChars": 9
     }
   ]
}

This first blocking key specification is comprised of just the POSTCODE element, using the STANDARDSPELLING element modifier, and restricting the keys created from it in that they must be at least 9 characters long.

This will result in the generation of candidate score pairs for every record in the data with the same zip and +4 code.

The sample input USA zip codes below would therefore create the following keys with this blocking key specification:

• 90210-8560   ⇒    902108560    - complete example, with the correct number of characters
• 90210-08560 ⇒    902100856    - truncated example, with the final 0 cut off
• 90210             ⇒     <null>              - no blocking key generated due to min. length of 9 being unsatisfied

This second blocking key specification below is a little more advanced - it combines a few elements together and uses various modifiers and algorithms, as well as different character number restrictions.

{
    "description": "ForenameSurnameMinorStreetNumber",
    "countryCode": "GBR",
    "elementSpecifications": [
      {
        "elementType": "FORENAMES",
        "elementModifiers": [
          "ROOTNAME"
        ],
        "algorithm": {
          "name": "DOUBLE_METAPHONE_FIRST_WORD"
        },
        "includeFromNChars": 1,
        "truncateToNChars": 10
      },
      {
        "elementType": "SURNAME",
        "algorithm": {
          "name": "DOUBLE_METAPHONE"
        },
        "includeFromNChars": 1,
        "truncateToNChars": 10
      },
      {
        "elementType": "MINORSTREET_NUMBER",
        "includeFromNChars": 1,
        "truncateToNChars": 5
      }
   ]
}

This will result in the generation of candidate score pairs for every record in the data with the same forename root name double metaphone value, surname double metaphone value, and premises number.

• The first element defined is FORENAMES and uses:
  • the ROOTNAME element modifier
  • the DOUBLE METAPHONE_FIRST_WORD algorithm on that root name
  • key restrictions where the forename must be at least 1 character long, and then stop including characters in the key after the 10^th
    
    
• The second element defined is SURNAME without any element modifier, then uses:
  • the DOUBLE METAPHONE algorithm on the surname
  • key restrictions where the surname must be at least 1 character long, and then stop including characters in the key after the 10^th
    
    
• The third (and final) element defined is MINORSTREET_NUMBER, and uses:
  • no element modifiers or element algorithms
  • key restrictions where the number must be at least 1 digit long, and then stop including characters in the key after the 5^th

The sample name and addresses below show what those blocking keys would look like:

MRS VAL JONES, 45 MAIN ROAD, LONDON, E1 2AS

• FORENAMES = VAL
  • Modifier: ROOTNAME of VAL = VALERIE
  • Algorithm: DOUBLE_METAPHONE_FIRST_WORD of VALERIE = FLR
  • Length restriction: Length of FLR = 3, and 3 is greater than 1 and less than 10 so satisfies restriction

⇒ FLR

• SURNAME = JONES
  • Modifier: <none>
  • Algorithm: DOUBLE_METAPHONE of JONES = JNS
  • Length restriction: Length of JNS = 3, and 3 is greater than 1 and less than 10 so satisfies restriction

⇒JNS

• MINORSTREET_NUMBER = 45
  • Modifier: <none>
  • Algorithm: <none>
  • Length restriction: Length of 45 = 2, and 2 is greater than 1 and less than 5 so satisfies restriction

⇒ 45

Therefore, the final key generated for this address is: FLRJNS45.

MR JOHNNY ANDERSON-THOMPSON, 123456 HIGH STREET, BRIGHTON, BN1 3SX

• FORENAMES = JOHNNY
  • Modifier: ROOTNAME of JOHNNY = JOHNATHON
  • Algorithm: DOUBLE_METAPHONE_FIRST_WORD of JOHNATHON = JN0N
  • Length restriction: Length of JN0N = 4, and 4 is greater than 1 and less than 10 so satisfies restriction

⇒ JN0N

• SURNAME = ANDERSON-THOMPSON
  • Modifier: <none>
  • Algorithm: DOUBLE_METAPHONE of ANDERSON-THOMPSON = ANTR
  • Length restriction: Length of ANTR = 4, and 4 is greater than 1 and less than 10 so satisfies restriction

⇒ ANTR

• MINORSTREET_NUMBER = 123456
  • Modifier: <none>
  • Algorithm: <none>
  • Length restriction: Length of 123456 = 6, and 6 is greater than 1 but more than 5 so truncate at the 5^th character

⇒ 12345

Therefore, the final key generated for this address is: JN0NANTR12345.

PAUL SMITH, HANCOCK BUILDING, GRACE AVENUE, NOTTINGHAM

• FORENAMES = PAUL
  • Modifier: ROOTNAME of PAUL = PAUL
  • Algorithm: DOUBLE_METAPHONE_FIRST_WORD of PAUL = PL
  • Length restriction: Length of PL = 2, and 2 is greater than 1 and less than 10 so satisfies restriction

⇒ PL

• SURNAME = SMITH
  • Modifier: <none>
  • Algorithm: DOUBLE_METAPHONE of SMITH = SM0
  • Length restriction: Length of SM0 = 3, and 3 is greater than 1 and less than 10 so satisfies restriction

⇒ SM0

• MINORSTREET_NUMBER = null
  • Modifier: <none>
  • Algorithm: <none>
  • Length restriction: Length of null = 0, which is NOT greater than 1 so key restriction NOT satisfied

⇒ <null>

Because not all the blocking key specification has been satisfied in this case (due to the missing premises number in the input), no blocking key is generated for this record from this blocking key specification.

Whilst blocking keys are used to generate suitable candidate pairs of input records, rules are then used to score those pairs together and cluster together any that are considered a good enough match.

You can modify the default rulesets or even create your own. Before you do though, we recommend that you review the concepts below:

• Rules take the following form: <rule reference>=<expression>
• A rule reference consists of a rule name followed  by a "." and followed by a match level (e.g. MyRule.L0)
• An expression may take multiple forms:
  • Low-level expression - operates on the elements within a record.
  • Higher-level expression - composed of references to other rules.
• A ruleset consists of a combination of three rule types which increase in specificity from match, to theme, to element, as illustrated in the diagram below.
  • Match and theme rules are comprised of references to rules from the level below.
  • Element rules are comprised of rules that are set on specific data elements, such as a postcode or a building number. These element rules can use special comparators depending on the element type.

All rules have a rule reference on the left-hand side.

A rule reference takes the following format: <rule name>.<match level>

The rule name may be any of the following:

• "Match" (match rule).
• A custom identifier (theme or multi-element rule). Must begin with an alphabetical character and may contain alphanumeric characters, underscores, and hyphens. 
• Element (single element rule).

The match level can be: L0, L1, L2, or L3. Note that you can override these values by using aliases.

The right-hand side of a rule:

• Is always surrounded by curly brackets: { }
• May contain logical operators: & and |

Expressions may be nested and logical operators combined (parentheses are required), e.g. MyRule.L2 = {((RuleA.L3 & RuleB.L2) | (RuleA.L2 & RuleB.L3)) | RuleC.L0}

Element rules include the element and the allowed result set (enclosed in [ ] and comma-separated) and may also include an optional element modifier and/or comparator.

Any theme or element rule may also optionally include a group from the input mappings, which is defined by using a hash symbol before the group name. For example:
 #MyFieldGroup.PostcodeTheme.L0 = {Postcode[ExactMatch]}.

Find out about groups.

You can also specify a default country in the rules file. This lets the Find duplicates step know what country to use when processing and standardizing data if:

• you're not using a country tag when mapping your data columns or
• the country field for a record is blank.

To specify a default country, add @default.country=<countryISO> to the top of the rules file (where <countryISO> is the ISO 3166-1 alpha-3 country code). For example: to set Australia as the default country, add: @default.country=AUS.

You can use the default rules provided with the Find duplicates step as the basis for customization since they already contain the correct default country value.

There are 4 match levels that can be used within each rule specification:

Note that you can override these level names with your own custom names. To do so, you have to use the expression define <Alias> as <Match level> at the top of the rules file, e.g. 

define High as L0.
Match.High={Name.High & Address.High}

When working with match levels, note that:

• They will be evaluated in order from L0 through to L3, stopping at the first level that passes.
• A rule level will evaluate as true if the same rule has evaluated as true for a higher level. For example, if MyRule.L1 is true, then MyRule.L2 and MyRule.L3 will also be true. 
• Records will be considered a match and they will be clustered together, provided that any of the defined top level overall rule match levels evaluate to true.
• Every rule must include a match level as part of the rule reference.

Rules are evaluated as follows:

1. Match rules first (Match.<Match Level>).
2. Left to right.
3. Lazily.

Three rule types can be defined:

• Match rules
• Theme rules
• Element rules

This is the highest rule level, defining an overall match between two records. A match rule is made up of references to other rules. 

At least one match rule must be defined for a successful matching job.

• Example: Match.L0={Name.L0 & Address.L0} (Name.L0 and Address.L0 have been defined separately).

You can combine rule references into compound logical expressions. This way, you have complete control over the logic used to determine matches.

• Example: Match.L1={(Name.L0 & Address.L0) | (Name.L0 & Email.L0 & Phone.L0)}

Theme rules represent the next level down, after match rules.

Similar to a match rule, a theme rule is made up of references to other rules. The theme rule name must begin with an alphabetical character and may contain alphanumeric characters, underscores and hyphens. The rule name cannot contain "Match" or the set of reserved elements.

• Example: Address.L0={Premise.L0 & Street.L0 & Locality.L0 & Postcode.L0}

The rule references within the expression can either be other theme rules, or low-level element rules.

Element rules are the most granular of rules. They can be used to specify how to compare individual elements within a record. Elements are basic units of data that comprise an overall theme. For example, postcode and premise could be elements of an address theme.

Rules are designed to evaluate and compare elements using special comparators. The table below covers the available comparators you can use.

If you want to know which comparators are available for which elements, see the elements table.

These are element rule examples that focus on how different elements, modifiers and comparators can be used when designing the rules:

Initial vs full name

Name.L2 = {Forenames.ForenameCompare[InitialVsFullName] & Surname[ExactMatch]}

Minor street number

StreetAddress.L1 = {MinorStreet_Number.PremiseCompare[NumberMatchWithTrailingAlpha] & MinorStreet_Description[ExactMatch] & MinorStreet_Type.StandardAbbreviation[ExactMatch]}

Poscode

Address.L2 = {Building_Number[ExactMatch] & MinorStreet_Description[ExactMatch] & Locality[ExactMatch] & Postcode.PostcodeCompare[Part1Match]}

Cross-field matching allows you to match across multiple fields of the same type to find potential duplicates.

For example, if your data consists of three phone number fields (e.g. home, work and mobile number), you can configure the Find duplicates step to find potential phone number matches across all three of them. 

To configure the Find duplicates step to perform cross-field matching, you have to set up your blocking keys and rulesets in such a way that your groups of similar data will be blocked and scored together.

Example

You have multiple address fields (home and billing) and want to identify potential duplicates where the billing address of one record matches the delivery address of another record.

Using the default rules and blocking keys for GBR and modifying them to consider these two addresses in separate groups (HomeAddress and BillingAddress), the two records will not be matched together since the home address for the two records will not block together or evaluate as a match in the rules. Similarly, the two records would still not match together based on the billing address group since the billing addresses are also different.

The desired behavior is for the following cross-field operations to be performed (with the operation in bold causing a match using the above example):

• HomeAddress 1 vs. HomeAddress 2
• BillingAddress 1 vs. BillingAddress 2
• HomeAddress 1 vs. BillingAddress 2
• BillingAddress 1 vs. HomeAddress 2

To achieve this, add: "elementGroups": ["HomeAddress","BillingAddress"] to the blocking keys for each address component of each blocking key definition and #[HomeAddress,BillingAddress] to the address rules of the final match level rules.

Blocking keys

The blocking key definition below is from the default GBR individual blocking keys, combining the surname initial, street number and locality into a blocking key. By default, this will create keys for a single address. However, it can be modified to create keys for multiple addresses using the items in italics below:

{
   "description": "SurnameMinorStreetNumberLocality",
   "countryCode": "GBR",
   "elementSpecifications": [
     {
       "elementType": "SURNAME",
       "algorithm": {
         "name": "INITIAL"
       },
       "includeFromNChars": 1
     },
     {
       "elementType": "MINORSTREET_NUMBER",
       "elementGroups": ["HomeAddress","BillingAddress"],
       "includeFromNChars": 1,
       "truncateToNChars": 5
     },
     {
       "elementType": "LOCALITY",
       "elementGroups": ["HomeAddress","BillingAddress"],
       "elementModifiers": [
         "STANDARDSPELLING",
         "DERIVED"
       ],
       "includeFromNChars": 2,
       "truncateToNChars": 30
     }
   ]
 }

This will now cause two blocking keys to be generated for every record:

• SURNAME_Initial + HomeAddress_MINORSTREET_NUMBER + HomeAddress_LOCALITY
• SURNAME_Initial + BillingAddress_MINORSTREET_NUMBER + BillingAddress_LOCALITY

Using the sample data, the following blocking keys will be created:

• Record 1
  • S1LONDON
  • S48BRIGHTON
• Record 2
  • S12LONDON
  • S1LONDON

Since the first and last keys are identical, records 1 and 2 will now be blocked together and identified as a candidate pair for scoring in the rules. 

Rules

The rule snippet below is from the default GBR individual ruleset that has been amended to include cross-field support as an example (with the change in italics):

Match.Exact={Name.Exact & #[HomeAddress,BillingAddress].Address.Exact}

This will now cause the following rule evaluations to be performed:

The third case will be true since the home address of record 1 matches the billing address of record 2 as an exact match.