Unique Masks

Unique masks are masks that will ensure every value is replaced with a unique value for that column. For other masks see all mask functions.

• From Unique (from_unique)
  Generates random but unique strings or numbers, from a format string (databases only)
• From Unique Imitate (from_unique_imitate)
  Transforms strings or numbers to be random, retaining format and uniqueness

From unique (from_unique)

A mask that generates string values that are guaranteed to be unique within the target column.

Note: To automatically cascade primary and unique key values to foreign keys or mask composite keys, consider using mask_unique_key instead.

Parameters

• format (required): The format that will be used to generate values. See format string syntax for details on how to construct a format string.

version: '1.0'
tasks:
  - type: mask_table
    table: drivers
    key: id
    rules:
      - column: licence_plate
        masks:
          - type: from_unique
            format: "{[A-Z],3}{[0-9],3}"

Note

• All values produced by from_unique will be strings. To convert values to other data types, you can chain your from_unique mask with a typecast mask.
• Unlike from_format_string, from_unique ensures generated values are unique. This is achieved using the same underlying unique value generation procedure as mask_unique_key.
• The specified format must allow for a sufficient number of unique values to cover the full number of rows in the target table or file (rounded up to the nearest multiple of the batch_size for mask_table tasks), otherwise an error will be reported when executing the run.
• Using the from_unique mask type differs from the mask_unique_key task type in the following ways:
  • from_unique cannot perform any cascading of values to related columns, such as foreign keys.
  • from_unique cannot be used to guarantee joint uniqueness across the columns in a composite key, unless guaranteeing uniqueness within a single column within the composite key is sufficient for the target use case.
  • from_unique can be used to update unique columns at the same time as other columns in a mask_table task, without the need for a separate database update performed by a mask_unique_key task.
  • from_unique can be used to generate unique values in mask_tabular_file tasks or in mask_table tasks for databases that do not support mask_unique_key (such as Amazon DynamoDB).
• Using from_unique in a mask_table task where worker_count > 1 is only supported for Oracle and Amazon DynamoDB connections.
• from_unique cannot be used in the rules of a mask_file task.
• Like other mask functions, from_unique cannot be used to update the key columns of a mask_table task (except for databases that allow mask_table to update key columns, such as Amazon DynamoDB).
• All from_unique masks within a given task will be produced from the same sequence of unique values, such that two from_unique masks with identical format will produce the same value for the same row. This can be useful to ensure values produced by identical from_unique masks in different if branches are jointly unique.

From Unique Imitate (from_unique_imitate)

Note: When used in a mask_table task, this mask type will be skipped in a dry run as it requires writing to the database.

The from_unique_imitate mask is very similar to the imitate mask:

• alphabetical characters a-z and A-Z are replaced by other alphabetical characters of the same case
• digits are replaced by other digits
• all other characters (whitespace, symbols such as %, and so on) are left as-is.

However, there are the following differences:

• Most notably, from_unique_imitate can also be used on primary keys and columns that have a UNIQUE constraint. Indeed, it is designed specifically for data that must take unique values per row.
• The masked values are guaranteed to be consistent for the same given input, and different for different inputs. (The specific values generated will vary across runs unless you control the seeding.)
  • For example, if a column has a UNIQUE constraint and the existing data satisfies that constraint, the resulting masked values will all be unique.
  • If a column contains the same value in every row, the content of the column after masking will also have the same value in every row.

The from_unique_imitate mask will always create a different result to its input, as if it has an implicit force_change: true. The mask may occasionally preserve individual letters and digits in the same position, but it is always guaranteed that at least one (and very likely almost all) of the alphanumeric characters will be different. For example, ABC-123 may be masked to BYC-457.

This mask can only be used on columns of integer or string (char / varchar) type.

null values will always be left as null.

Important! from_unique_imitate does not support IDENTITY columns in Oracle or Microsoft SQL Server.

You can apply a from_unique_imitate mask to a primary key column or a column that is used as a foreign key in another table. References will be updated automatically. Composite primary keys are supported.

Parameters

• skip_letters (optional): A boolean to enable or disable the skipping alphabetical characters from being masked. Defaults to false (alphabetical characters will be replaced).
• skip_digits (optional): A boolean to enable or disable the skipping of digits from being masked. Defaults to false (digits will be replaced).
• checksum (optional): A string to specify an algorithm to use to generate unique valid replacements. Options: brazilian_cpf, credit_card, luhn or icp.
• on_invalid (optional): A string to specify the action to take if the value fails checksum validation. One of:
  • mask (default): If the input length and format is valid (ie, the input value is only invalid because of an invalid checksum), the input value will be masked with a new uniquely assigned invalid checksum. Otherwise, the input value will be masked without consideration of a checksum.
  • skip: Skip to the next value, the value remains unchanged.
  • error: Raise an error and stop masking.
• retain_prefix_length (optional): The number of characters of the input value to retain. See Retaining Prefixes below. By default, no prefix is retained (i.e. the entire input value is masked).
• include_leading_zeros (optional): Whether to mask leading zeros in input values and allow leading zeros in masked values. See the table below for details and examples of how this option works. One of:
  • true: Leading zeros will be masked.
  • false: Leading zeros will be preserved.
  • warn (default): Like false, but also issues a warning in the run log if a string value with all zero digit(s) is encountered, reminding the user that the digits (or perhaps the entire value) will not be masked.
• min_digits (optional): Only applicable when include_leading_zeros is true. Pads integer-type values to the given number of digits before masking. Valid values for min_digits are between 1 and 18.
• on_too_long (optional): Only applicable when include_leading_zeros is true and min_digits is specified. Determines what action to take when an integer value with more than min_digits digits is to be masked. (The leading - sign in a negative number does not count as a digit.) Valid values are:
  • error (default): Raise an error and stop masking.
  • mask_suffix: The last min_digits digits of the value will be masked; earlier digits will be left as-is.
  • mask_all: The entire value will be masked.
• disable_warning_on_skipped_characters (optional): A boolean which, when set to true, will prevent warnings about characters being skipped being logged during masking. Defaults to false.

The include_leading_zeros option has no effect for integer data unless min_digits is also specified. By default, integer values are always masked to values with the same number of digits as the input.

Because it can create duplicate values, masking negative integers with min_digits is not supported. If this option is set and a negative integer value is encountered, DataMasque raises an error and stops masking.

version: "1.0"
tasks:
  - type: mask_table
    table: '"my_table"'
    key: '"id"'
    rules:
      - column: '"value_as_string"'
        masks:
          - type: from_unique_imitate
            include_leading_zeros: true
      - column: '"value_as_integer"'
        masks:
          - type: from_unique_imitate
            include_leading_zeros: true
            min_digits: 6

Invalid Parameter Combinations

• Setting both skip_digits and skip_letters to true is prohibited as no masking would take place.
• Using the checksum option requires masking digits, so if any checksum is specified then you cannot set skip_digits to true. Further, the icp checksum is an alphanumeric checksum, so you cannot set skip_digits nor skip_letters to true when using this checksum.
• Because it would have no effect, include_leading_zeros cannot be specified if skip_digits is set to true.
• The min_digits option is not compatible with checksum, nor with retain_prefix_length.

Values Requiring Checksums

When the output values must satisfy a checksum, specify the name of the checksum as the checksum parameter. Unique values will be generated that satisfy that checksum algorithm. The available options for checksum are:

• brazilian_cpf
• credit_card
• luhn
• icp

For each checksum, the input value must contain a certain number of digits 0-9 and no letters A-Z or a-z. Other non-letter characters that are used for formatting are retained in the output. The replacement value will conform to the checksum algorithm, even if the input did not, provided it is of the correct length.

To handle masking of values that may not match the checksum, or that may contain letters, specify the on_invalid parameter. Please refer to Using on_invalid for a detailed explanation of the behavior of each on_invalid option.

brazilian_cpf

Use this checksum type to generate values that satisfy the Brazilian CPF (Cadastro de Pessoas Físicas) number checksum.

For valid CPFs to be generated, the input value must contain 11 digits (and may contain spaces or punctuation). For handling of invalid input values (for example, incorrect length, bad checksum or the presence of letters), please refer to Using on_invalid.

The table below shows example input and output data, based on the default parameters.

credit_card and luhn

The credit_card and luhn checksums both generate values that satisfy the Luhn checksum algorithm. The difference is how they each validate the length of the number:

• luhn may be applied to any number containing two or more digits.
• credit_card is only valid for numbers of length 12 to 19, inclusive.

For values of length 12-19 characters, the behaviour of both checksums is identical.

If you are masking only credit cards, then credit_card should be preferred, as it will also validate the length of existing values. luhn should be used when generating values of other lengths, for example, mobile phone IMEI numbers.

For handling of invalid input values (for example, incorrect length, bad checksum or the presence or letters), please refer to Using on_invalid.

icp

Use this checksum type to generate values that satisfy the New Zealand Installation Control Point (ICP) checksum.

For valid ICPs to be generated, the input value must contain 15 digits and letters (and may contain spaces or punctuation). For handling of invalid input values (for example, incorrect length, or bad checksum), please refer to Using on_invalid.

The table below shows example input and output data, based on the default parameters.

Example

This example will apply from_unique_imitate masks to the vehicle_id, license_plate and validation_code columns.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules:
      - column: vehicle_id
        masks:
          - type: from_unique_imitate
      - column: license_plate
        masks:
          - type: from_unique_imitate
      - column: validation_code
        masks:
          - type: from_unique_imitate

Using on_invalid

The on_invalid parameter can be used to control how invalid values are handled by from_unique_imitate. It can only be used in combination with the checksum parameter, since without specifying a checksum there is no way of considering a value "invalid".

A value is considered invalid if:

• It is too short for the checksum, or,
• The checkdigit(s) are not valid for the checksum, or,
• It contains letters – this can be controlled with the skip_letters parameter. See Determining and Handling Invalid Values for more detail.

Note that null is a special case and is not considered invalid. null input is masked to null output, regardless of settings.

The following table illustrates the behaviour of from_unique_imitate based on different on_invalid parameters.

Determining and Handling Invalid Values

There two are main ways that a value can be invalid:

• It contains letters.
• It has the wrong length or checksum.

Note that special characters, punctuation, and spaces, do not affect the validity of values. For example, from_unique_imitate considers the values 123 456, 123-456 and 123456 the same.

Determining the validity based on the presence of letters can be controlled with the skip_letters parameter. When set to true, a value will not be invalid if it contains letters. However, if the digits themselves do not satisfy the given checksum, then the value would be considered invalid.

The following table shows the validity of some example values for the brazilian_cpf algorithm, with and without the use of skip_letters: true. The rules are applicable to any checksum.

Once a value is determined to be invalid:

• If using on_invalid: error then the masking task will stop with an error.
• If using on_invalid: skip then the value will be returned unmasked.
• If using on_invalid: mask then masking will continue, and is described in more detail below.

For invalid values, the behaviour of the mask will change based on the reason for it being invalid.

If masking a value that is invalid only because of a bad check checksum, the output will mask uniquely to a new value with a uniquely masked, invalid checksum. If the value is invalid for another other reason, the string will be masked uniquely.

For example, the brazilian_cpf algorithm requires 11-digit values. An 11-digit value that is not a valid CPF number will be masked to a invalid CPF number (an 11-digit number that does not have a valid CPF number checksum).

However, a number that is not 11-digits will not be masked to a valid CPF number, as the number of digits is not correct for that algorithm, but the numbers will be masked.

The following table shows the minimum and maximum value length to which the checksum applies.

If the value is invalid because it contains letters:

• The letters will be masked if skip_digits is false.
• The letters remain unchanged if skip_digits is true.

The following table gives examples of masked outputs for invalid inputs, for different skip_letters options. on_invalid is set to mask, otherwise no masking would occur. The example uses the brazilian_cpf algorithm but the rules are applicable to any checksum.

Warning: When using on_invalid: mask with the brazilian_cpf checksum, 10 of the possible 1,000,000,000 digit combinations are not maskable with this algorithm.

Where the data to mask has 11 digits, and the first nine digits are all the same digit (eg, 111.111.111-xx regardless of the checksum), the number itself is considered to be invalid.

Where this data is required to be masked, it is recommended that the from_unique_imitate rule is wrapped by an if rule that detects this data, and directs the engine to apply an alternative algorithm.

When the masking process is configured to uniquely mask a CPF number that is invalid due to repeated digits, the run log will note a warning, and these entries will not be masked. This does not affect other checksum masking as this situation is specific to Brazilian CPF unique masking.

Example

This example will apply from_unique_imitate masks to the cpf_number and apply a redaction to known invalid numbers by leveraging an if conditional masking rule.

version: '1.0'
tasks:
  - type: mask_table
    table: employees
    key: id
    rules: 
      - if:
        - column: cpfnumber
          matches: '(?:.*?(\d)\1\1){3}.*'
        rules:
          - column: cpfnumber
            masks:
              - type: from_fixed
                value: 'redacted'
        else_rules:
          - column: cpfnumber
            masks:
              - type: from_unique_imitate
                checksum: brazilian_cpf
                on_invalid: mask

Retaining Prefixes

When masking values with from_unique_imitate, the retain_prefix_length option be used to specify the number of prefix characters of the input to retain in the output.

The length takes into account only characters that would be masked based on the parameters of the mask. Since from_unique_imitate doesn't mask punctuation or spaces, then these are not counted towards the prefix length. Similarly, if using skip_letters/skip_digits, then letters or digits (respectively) won't be counted in the prefix.

If the retain_prefix_length is equal to or longer than the values to be masked, then an error will be raised during masking.

The following table shows the retained prefix for example values, based on different parameters.

The retain_prefix_length parameter can be combined with the checksum parameter. The checksum will be generated after combining the prefix with the masked values.

For example, using the luhn checksum, with retain_prefix_length of 5. The value to be masked is 211287932175 (which is valid for the Luhn checksum).

The order of masking is:

• Extract 5 characters as a prefix: 21128
• Apply unique masking to all digits but the checksum digit 793217, giving (for example) 123456
• This value is combined with the prefix, giving 21128123456
• The checksum digit is calculated from this prefix and masked value, which is 0
• The checksum digit is appended to the masked value, giving final output value of 211281234560, which is also valid for the Luhn algorithm.

When retaining the prefix and using a checksum, the number of digits must be less than the number of digits in the original value, minus the length of the check digits. For example, to mask a Brazilian CPF, a maximum of 8 digits can be retained, since the value is made up of 11 digits with the final 2 being checkdigits.

Checksum validity is checked when retaining the prefix, and validation is on the original value (i.e. including the prefix). To control how to handle invalid values, refer to Determining and Handling Invalid Values.

The following example ruleset shows masking a column using the credit_card checksum, retaining the first 4 digits.

version: "1.0"
tasks:
  - type: mask_table
    table: customers
    key: customer_id
    rules:
      - column: credit_card_number
        masks:
          - type: from_unique_imitate
            retain_prefix_length: 4
            skip_letters: true
            checksum: credit_card

RedShift Limitations

Due to the current method in which we mask with RedShift, please note the current limitations:

• PRIMARY KEY constrained columns will be transformed into columns which are UNIQUE constrained instead of PRIMARY KEY constrained.
• NOT NULL constrained columns will lose their NOT NULL constraint. However, this is not just limited to from_unique_imitate masking.