Sensitive Data Discovery

• Overview
• Methodology
• JSON Sensitive Data Discovery
• In-Data Discovery
  • Configuring Options for In-Data Discovery
• Reporting and notifications
  • Per-run data discovery report
  • Notification of new sensitive data
• Keywords configuration
  • Global Custom Data Classification Keywords
  • Global Ignored keywords
• Built-in data discovery keywords
• Appendix: In-data discovery rules

Overview

DataMasque can be configured to automatically discover sensitive data in your databases during masking. When a masking ruleset contains the special purpose run_data_discovery task type, DataMasque will inspect the database metadata and generate a discovery report for each masking run. Additionally, each user will receive email notifications of any new, unmasked sensitive data that has been discovered by DataMasque, providing ongoing protection against new sensitive data being added to your schemas over time.

Note: The schema discovery feature does not currently support Amazon DynamoDB or Microsoft SQL Server (Linked Server) databases.

Methodology

To perform sensitive data discovery, DataMasque uses regular expressions (regex) to scan the metadata of the target database. When a database column is identified as sensitive, DataMasque will compare it with the masking rules specified in the ruleset to determine the masking coverage for the column.

DataMasque comes with over 90 built-in keywords to help discover various types of sensitive data (account numbers, addresses, etc.) in your database. These built-in keywords are global keywords, used for sensitive data discovery by default and for schema discovery when enabled. Each pattern is classified into one or more categories. The included data classification categories are:

• Personally Identifiable Information (PII)
• Personal Health Information (PHI)
• Payment Card Information (PCI)

JSON Sensitive Data Discovery

Sensitive information contained within JSON data can be discovered through the JSON Sensitive Data Discovery feature available through the YAML Ruleset Editor

Here you can enter JSON data into the text field and run the discovery feature to identify the lowest level sensitive keys which could contain sensitive information. The sensitive items are evaluated using the built-in data discovery keywords.

Once finished it will return an automatically generated rule to mask the sensitive key's values contained within the specified JSON data, this rule can then be copied or inserted into the ruleset on the YAML Editor.

Example

This example will show the benefit of generating the rule through the JSON Mask Generator. Suppose a column contains the following JSON data:

{
  "customers": [
    {
      "primary": {
        "name": "Foo",
        "credit card": 123456789
      },
      "secondary":
        {
          "name": "Bar",
          "credit card":987654321
        }
    }
  ]
 }

This data can be entered into the JSON Data field of the JSON Mask Generator and a rule will be created to mask the sensitive keys (name and credit card) as shown below. This can then be copied into the ruleset under the relevant task.

type: json
transforms:
  - path:
      - customers
      - '*'
      - primary
      - name
    masks:
      - type: from_fixed
        value: redacted
    on_null: skip
    on_missing: skip
    force_consistency: false
  - path:
      - customers
      - '*'
      - primary
      - credit card
    masks:
      - type: chain
        masks:
          - type: credit_card
            validate_luhn: true
            pan_format: false
            preserve_prefix: false
          - type: take_substring
            start_index: 0
            end_index: 9
    on_null: skip
    on_missing: skip
    force_consistency: false
  - path:
      - customers
      - '*'
      - secondary
      - name
    masks:
      - type: from_fixed
        value: redacted
    on_null: skip
    on_missing: skip
    force_consistency: false
  - path:
      - customers
      - '*'
      - secondary
      - credit card
    masks:
      - type: chain
        masks:
          - type: credit_card
            validate_luhn: true
            pan_format: false
            preserve_prefix: false
          - type: take_substring
            start_index: 0
            end_index: 9
    on_null: skip
    on_missing: skip
    force_consistency: false

In-Data Discovery

Note: This feature is in preview.

In-data discovery is an optional extra part of sensitive data discovery. It examines the contents of database tables and matches the data against a list of built-in rules and Custom rules to identify columns that likely contain sensitive data, even if the column's name doesn't match any of the built-in or configured sensitive data keywords. For example, it can find credit card numbers stored in a column named pcd (for payment card details) or column19.

In-data discovery is currently only available as part of schema discovery. You can enable it through the ruleset generator page. It is supported on all database platforms that support schema discovery (Postgres, Oracle, MySQL, MariaDB, Microsoft SQL Server and Amazon Redshift).

In-data discovery samples 1,000 random rows (or the entire table, if fewer than 1,000 rows) and attempts to match the content of those rows against its matching rules. To avoid false positives, a certain minimum percentage of the data within a column has to match a rule for the column to be flagged as sensitive data. If a column's data matches multiple rules, only the first match will be returned in the sensitive data discovery or schema discovery report. The list below gives the rules in priority order, highest priority first.

Configuring Options for In-Data Discovery

Configuring In-Data Discovery options can be done from the ruleset generator page by clicking the cog icon next to In-data discovery, that will open the following modal.

The options that can be configured are below:

• Rows to sample: This option determines the number of rows used for profiling.
• Custom Rules: These are user-provided rules used for profiling.
  • Name: The name of the rule that, if data potentially matches, will be displayed in the Reason for Flag column.
  • Rule: A regex pattern for the data you want to match against.

If at least 10% of the sampled rows from the column match the rule, the column will be flagged with the name of the rule and Custom in the Data Classification column. When a starting a discovery run, if a rule is invalid, the modal will re-open with an error message underneath the invalid pattern.

An imitate mask will be generated from columns flagged by custom rules, this can be changed later in the Ruleset YAML editor as needed.

Reporting and notifications

Per-run data discovery report

After a run_data_discovery task has completed, the corresponding data discovery report can be downloaded alongside the run logs of the masking run:

The sensitive data discovery report will be downloaded in CSV format and may be opened in a text editor or spreadsheet viewer such as Microsoft Excel. The report contains information to assist you in discovering and masking the sensitive data in your database. Every column that has been identified as potentially containing sensitive data is included in the report, along with a classification of the data and an indication of whether the masking ruleset contains a rule targeting the matched column.

The CSV report includes the following columns:

Table schema	The schema of the table containing a sensitive data match.
Table name	The name of the table containing a sensitive data match.
Column name	The name of the column which has matched against a commonly used sensitive data identifier.
Reason for flag	Description of pattern which caused the column to be flagged for sensitive data.
Data classifications	A comma-separated list of classifications for the flagged sensitive data. Possible classifications include PII (Personally Identifiable Information), PHI (Personal Health Information), and PCI (Payment Card Information).
Covered by ruleset	A boolean value (True/False) indicating whether the masking ruleset contains a rule to target the identified column.

Note for Oracle: Sensitive data discovery reports will only cover the tables owned by the user or schema as defined in Connection. Schema will take precedence over user.

Note for Microsoft SQL Server: Sensitive data discovery reports will only cover the tables owned by the user's default schema.

Note for PostgreSQL: Sensitive data discovery reports will only cover the visible tables in the current user search path.

A table is said to be visible if its containing schema is in the search path and no table of the same name appears earlier in the search path.

Notification of new sensitive data

DataMasque users will receive email notifications¹ of newly detected, unmasked sensitive data in your databases. This feature provides ongoing protection against new sensitive data making its way into your databases over time.

Disabling notifications

Each user can opt out of receiving email updates by navigating to the My Account page and disabling the Notify me when sensitive data is found option of the Edit Account form:

When notifications are sent

Each user that has enables this option will receive daily notification emails when new, unmasked sensitive data is detected on any Connection that has been masked in the previous 24 hours. If there is nothing to report, no notifications will be sent.

Notifications will be sent, given that in the previous 24 hours a data discovery task has been run:

• On a new Connection that contains unmasked sensitive data.
• As part of a ruleset in which a masking rule that was previously protecting sensitive data has been removed.
• On an existing Connection which has had one or more columns containing sensitive data added to the database.

Notes:

• New sensitive data is only detected during masking runs that include a run_data_discovery task. Include this task in all masking runs to receive ongoing protection.
• This feature requires that SMTP has been configured, allowing DataMasque to send outbound email.

Keywords configuration

DataMasque can be configured with additional custom keywords and ignored keywords to facilitate sensitive data discovery. Both custom keywords and ignored keywords are case-insensitive. These can be configured from the Settings page.

Global Custom Data Classification Keywords

Column names are matched to Global Custom Data Classification keywords in addition to the built-in data discovery keywords. Two formats are supported for custom keywords:

• Keyword format:
  • The matching behaviour of the keyword will depend on the number of period-separated segments:
  • If no period is present, the keyword will be compared to the column name.
  • If a single period is present, the keyword will be compared to the schema and table name: schema.table.
  • If two periods are present, the keyword will be compared to the schema, table, and column name: schema.table.column.
  • Literal periods (.) and backslashes (\) in names can be escaped with a preceding backslash.
  • The keyword will be matched case-insensitively against the data-dictionary representations of schema/table/column names.
  • The keyword will still match if the name contains additional characters preceding/following a substring that matches the corresponding segment of the keyword.
  • Spaces in a keyword will match space, underscore, and hyphen delimiter characters, and will also match in the absence of such a delimiter character. For example, the space-separated keyword credit card number would match columns such as credit card number, creditcardnumber, creditcard_number, and credit-card number.
  • The * wildcard is also supported, for example you can discover all columns in a specific table by specifying schema_name.table_name.*
  • Only alphanumeric characters, spaces, underscores, hyphens, periods, asterisk wildcards, and escaping backslashes are allowed.
• Regex format:
  • A keyword prefixed with regex: will be treated as a regular expression over a full schema_name.table_name.column_name string.
  • The regex will be matched case-insensitively against the data-dictionary representations of schema/table/column names.
  • Any backslashes or periods in schema/table/column data-dictionary names will be prefixed by a backslash in the string to be matched by the regex.
  • For more details on regular expressions, see: Common regular expression patterns.

Column names that match the Global Custom Data Classification keywords will be reported and tagged with the data classification "Custom" in Sensitive Data Discovery reports.

Global Ignored keywords

Global Ignored keywords will only ignore exact matches of a column name, which allows you to exclude specific column names from Sensitive Data Discovery reports. Two formats are supported:

• Keyword format:
  • The matching behaviour of the keyword will depend on the number of period-separated segments:
  • If no period is present, the keyword will be compared to the column name.
  • If a single period is present, the keyword will be compared to the schema and table name: schema.table.
  • If two periods are present, the keyword will be compared to the schema, table, and column name: schema.table.column.
  • Literal periods (.) and backslashes (\) in names can be escaped with a preceding backslash.
  • The keyword will be matched case-insensitively against the data-dictionary representations of schema/table/column names.
  • The * wildcard is also supported, for example you can discover all columns in a specific table by specifying schema_name.table_name.*
  • Only alphanumeric characters, spaces, underscores, hyphens, periods, asterisk wildcards, and escaping backslashes are allowed.
• Regular expression format:
  • A keyword prefixed with regex: will be treated as a regular expression over a full schema_name.table_name.column_name string.
  • The regex will be matched case-insensitively against the data-dictionary representations of schema/table/column names.
  • Any backslashes or periods in schema/table/column data-dictionary names will be prefixed by a backslash in the string to be matched by the regex.
  • For more details on regular expressions, see: Common regular expression patterns.

For example, with the ignored keyword p_id, columns named p_id will be ignored, and will be no longer identified as sensitive data.

Built-in data discovery keywords

DataMasque uses regular expressions to search for columns which may contain the following sensitive information:

• Category PII

  • Name / first name / middle name / last name / surname
  • Mail / email
  • Date of birth / DoB
  • SSN / Social Security Number
  • Address
  • Post code
  • Country / state / city / zip code
  • Phone number
  • Fax number
  • Insurance number
  • Passport number
  • Driver license number
  • Gender
  • Age
  • Vehicle identification number / VIN
  • Login
  • Job position / role / title
  • Workspace / company
  • NRIC / Identity Card Number
  • IC number
  • ID number
  • IRD number / Inland Revenue Department number
  • NINO
  • Unique taxpayer reference / UTR
  • Identity / identification / ID
  • Tax number
  • Internet protocol (IP) address
  • Media access control (MAC) address
  • Licence plate
  • Licence number
  • Certificate number
  • Identifiers / serial number

• Category PCI

  • Credit / payment/ debit card
  • Credit / payment / debit number
  • Account number
  • Security code
  • Expiry date
  • Name / first name / middle name / last name / surname
  • PIN / Personal identification numbers
  • CVV / Card Verification Value
  • Address
  • Post code
  • Country / state / city / zip code
  • Phone number
  • Fax number

• Category PHI

  • PHI number
  • NHI number
  • Medical record number
  • Insurance number
  • Internet protocol (IP) address
  • Name / first name / middle name / last name / surname
  • Health plan beneficiary number
  • Identifiers / ID
  • Identifying number / code
  • Serial number
  • Licence plate
  • Licence number
  • Certificate number
  • Address
  • Post code
  • Country / state / city / zip code
  • Phone number
  • Fax number

Appendix: In-data discovery rules

The list of rules is built-in and currently cannot be changed or customized.

DataMasque can detect the following data in columns.

Credit Card Numbers

• How it matches: Uses the Luhn checksum algorithm.
• Threshold: 12% of sampled data should match this check.
• Minimum Column Length: 12 characters.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: credit_card
  pan_format: false
  pan_character: '#'
  generate_luhn: true
  apply_weighting: false
  generate_card_number: true
  on_null: skip
  on_invalid: mask
  output_format_choice: retained

Partial Credit Card Numbers

• How it matches: Uses a regular expression.
• Pattern to match: ^(\d{6}(?:X{6}|X{5}\d|X{4}\d{2}|X{3}\d{3}|X{2}\d{4}|X{1}\d{5}|#{6}|#{5}\d|#{4}\d{2}|#{3}\d{3}|#{2}\d{4}|#{1}\d{5}))$
• Threshold: 10% of sampled data should match this check.
• Minimum Column Length: 15 characters.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: imitate

Brazilian CPF Numbers

• How it matches: Uses the Brazilian CPF checksum algorithm.
• Mask for a match: Full Brazilian CPF numbers are typically preserved.
• Threshold: 10% of sampled data should match this check.
• Minimum Column Length: 11 characters.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: brazilian_cpf
  on_null: skip
  on_invalid: mask
  output_format_choice: retained

IP Addresses

• How it matches: Uses a regular expression.
• Pattern to match: \b(?:\d{1,3}\.){3}\d{1,3}\b
• Threshold: 10% of sampled data should match this check.
• Minimum Column Length: 15 characters.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: concat
  masks:
    - type: from_random_number
      max: 255
      min: 0
    - type: from_random_number
      max: 255
      min: 0
    - type: from_random_number
      max: 255
      min: 0
    - type: from_random_number
      max: 255
      min: 0
  glue: .

MAC Addresses

• How it matches: Uses a regular expression.
• Pattern to match: ^[0-9A-Fa-f]{2}[:-]?[0-9A-Fa-f]{2}[:-]?[0-9A-Fa-f]{2}[:-]?[0-9A-Fa-f]{2}[:-]?[0-9A-Fa-f]{2}[:-]?[0-9A-Fa-f]{2}$
• Threshold: 10% of sampled data should match this check.
• Minimum Column Length: 12 characters.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: chain
  masks:
    - type: imitate
      force_change: false
      uppercase: false
      lowercase: false
      digits: true
    - type: replace_regex
      regex: '[a-f]'
      masks:
        - type: from_format_string
          value_format: '{[a-f],1}'
      preserve_length: true
    - type: replace_regex
      regex: '[A-F]'
      masks:
        - type: from_format_string
          value_format: '{[A-F],1}'
      preserve_length: true

Email Addresses

• How it matches: Uses a regular expression.
• Pattern to match: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,24}$
• Threshold: 10% of sampled data should match this check.
• Minimum Column Length: No minimum length.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: concat
  masks:
    - type: from_file
      seed_file: DataMasque_firstNames_mixed.csv
      seed_column: firstname-mixed
    - type: from_file
      seed_file: DataMasque_lastNames.csv
      seed_column: lastnames
    - type: from_file
      seed_file: DataMasque_fake_email_suffixes.csv
      seed_column: email-suff

First Names

• How it matches: Matches column data against the built-in DataMasque_firstNames_mixed.csv seed file.
• Threshold: 40% of sampled data should match this check.
• Minimum Column Length: No minimum length.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: from_file
  seed_file: DataMasque_firstNames_mixed.csv
  seed_column: firstname-mixed

Last Names

• How it matches: Matches column data against the built-in DataMasque_lastNames.csv seed file.
• Threshold: 25% of sampled data should match this check.
• Minimum Column Length: No minimum length.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: from_file
  seed_file: DataMasque_lastNames.csv
  seed_column: lastnames

Full Names

• How it matches: Matches column data against the built-in DataMasque_firstNames_mixed.csv and DataMasque_lastNames.csv seed files.
• Threshold: 60% of sampled data should match this check.
• Minimum Column Length: No minimum length.
• Maximum Column Length: No maximum length.

Generated Mask Definition

- type: concat
  masks:
    - type: from_file
      seed_file: DataMasque_firstNames_mixed.csv
      seed_column: firstname-mixed
    - type: from_file
      seed_file: DataMasque_lastNames.csv
      seed_column: lastnames
  glue: ' '