Masking runs

Overview
Create database masking run
- Database run options
Create file masking run
- File run options
Preview and confirm run
Run logs
Run History table
Simultaneous runs

Overview

A masking run is the application of a masking Ruleset to a database Connection. Runs are created from the Database Masking Dashboard, and can also be triggered via the API (see Best Practices).

Create database masking run

A new database masking run can be configured with the following steps:

Navigate to the Database Masking dashboard
Select a connection from the list of available connections.
Select a ruleset from the list of available rulesets.
Set additional run options (detailed below)
Click PREVIEW RUN or PREVIEW DRY RUN*
After configuring the run, you will be taken to the Preview and confirm run screen.

*Dry Run allows you to test your rulesets without modifying the database. When a dry run is executed, DataMasque performs every operation as usual except:

the final UPDATE operation of mask_table tasks, which would otherwise write the masked value to the database

the value generation and subsequent UPDATE operation for mask_unique_key tasks

the truncate_table operation

the run_sql operation

New run options

Database run options

Run options are displayed on the Run Options section of the dashboard page. The following options are available:

Option	Default	Description
Batch Size¹	`50,000`	The maximum number of rows that will be fetched, masked, and updated in a single operation by DataMasque. Larger batch sizes will reduce database operation overhead, but using a batch size value that is too large may result in DataMasque memory exhaustion. This value does not affect the total number of rows masked. The maximum allowed batch size is 50,000. Note: The Batch Size parameter is not applicable for Amazon Redshift and Amazon DynamoDB masking runs. Only for database masking runs.
Max rows³⁴	`unset`	The maximum number of rows that will be masked by each `mask_table` task³. May be used for speeding up test iterations when developing rulesets. Warning: In the case that a table contains more rows than the value specified here, the remaining rows will contain unmasked data. Note: The Max rows parameter is not applicable for Amazon Redshift masking runs. Only for database masking runs.
Run secret	`unset`	The run secret is used in the random generation of masked values. A run secret must consist of at least 20 characters. Providing a consistent run secret will ensure that repeated runs on the same DataMasque instance will produce the same results. Note: If a `random_seed` value is provided in addition to the `run_secret` value, the `random_seed` will take precedence and the `run_secret` will be ignored.
Continue on failure	`false`	If there is a task failure, and this option is false, DataMasque will skip all remaining non-started tasks. If this option is true, DataMasque will continue performing other tasks even if there is a task failure. It can be useful to set this option to true when testing/debugging your masking ruleset to identify as many failures as possible in each run.
Notify me when this run completes	`false`	Email the current user when the job completes. The DataMasque instance must have SMTP configured.
Disable instance secret	`false`	If this option is set to `true`, DataMasque will exclude its instance-specific secret and generate masked values based solely on the run secret. You may wish to disable the instance secret in order to achieve consistent masking across DataMasque instances. However, by disabling the instance secret, any DataMasque instance using the same `run_secret` could replicate your data masking.
Override connection's Server Side Encryption settings	`false`	Only valid for Amazon DynamoDB connections. If checked, table SSE key settings may be overridden on a per-table basis. Click the gear icon to view the settings. See the Specifying SSE Options documentation for DynamoDB for more information about these settings
Max file size²	See description	The max file size is only applicable to Amazon Redshift and Amazon DynamoDB databases. For Redshift it sets the MAXFILESIZE of the UNLOAD command. For DynamoDB it determines the maximum pre-compression size of batch files split from the original files exported by DynamoDB to S3. In both cases, this determines the maximum file size in MB of records that will be masked at once by a single DataMasque worker. The max file size can be set to any integer value between 5 MB and 1000 MB, and defaults to 10 MB for Redshift and 100MB for DynamoDB (Redshift files are stored in the compact Parquet format, which expands to a greater extent when loaded into DataMasque's memory). Only for database masking runs.
Enable Diagnostic Logging	`false`	With diagnostic logging enabled, extra information (Columns, constraints, foreign keys and indexes) is captured to assist with diagnosing problems with masking runs. Because of the extra information collected, masking runs may take longer to execute with this option enabled. Memory information of the main process will be output to the run log in the following format `Memory: T:31.19GB / F:0.26GB / A:10.60GB` where `T`, `F`, and `A` stand for Total, Free and Available memory respectively. Memory of workers will also be captured in the following format `Memory for PID(58): 355.17MB`.

Notes:

¹Batch Size applies to database types other than Amazon Redshift and Amazon DynamoDB.

²Max file size only applies to Amazon Redshift and Amazon DynamoDB databases.

³Max rows does not apply to mask_unique_key tasks.

⁴ Use of the Max rows run option for Amazon Redshift is not yet supported in DataMasque. This is on our roadmap and will be added in a future release.

Create file masking run

A new file masking run can be configured with the following steps:

Navigate to the File Masking dashboard
Select a source connection from the list of available connections (files will be read from here).
Select a ruleset from the list of available rulesets.
Select a destination connection from the list of available connections (files will be written to here).
Set additional run options (detailed below)
Click PREVIEW RUN or PREVIEW DRY RUN*
After configuring the run, you will be taken to the Preview and confirm run screen.

*Dry Run allows you to test your rulesets without modifying the database. When a dry run is executed, DataMasque download files from the source, applies include/skip rules and masks them, but does not upload the masked data to the source. This allows you to see which files would get masked, and if there are any errors in your ruleset.

New run options

File run options

Run options are displayed on the Run Options section of the dashboard page. The following options are available:

Option	Default	Description
Run secret	`unset`	The run secret is used in the random generation of masked values. Providing a consistent run secret will ensure that repeated runs on the same DataMasque instance will produce the same results. Note: If a `random_seed` value is provided in addition to the `run_secret` value, the `random_seed` will take precedence and the `run_secret` will be ignored.
Continue on failure	`false`	If there is a task failure, and this option is false, DataMasque will skip all remaining non-started tasks. If this option is true, DataMasque will continue performing other tasks even if there is a task failure. It can be useful to set this option to true when testing/debugging your masking ruleset to identify as many failures as possible in each run.
Notify me when this run completes	`false`	Email the current user when the job completes. The DataMasque instance must have SMTP configured.
Disable instance secret	`false`	If this option is set to `true`, DataMasque will exclude its instance-specific secret and generate masked values based solely on the run secret. You may wish to disable the instance secret in order to achieve consistent masking across DataMasque instances. However, by disabling the instance secret, any DataMasque instance using the same `run_secret` could replicate your data masking.
Enable Diagnostic Logging	`false`	With diagnostic logging enabled, extra memory information is captured to assist with diagnosing problems with masking runs. Because of the extra information collected, masking runs may take longer to execute with this option enabled. Memory information of the main process will be output to the run log in the following format `Memory: T:31.19GB / F:0.26GB / A:10.60GB` where `T`, `F`, and `A` stand for Total, Free and Available memory respectively. Memory of workers will also be captured in the following format `Memory for PID(58): 355.17MB`.

Preview and confirm run

This screen shows the preview of the configured run. Check the run parameters here before proceeding with execution via the START RUN button. After the run has been started, you will be redirected to the Run logs page where you can monitor the run output and progress.

You can view a curl command for starting an equivalently configured run using the DataMasque API by clicking the VIEW RUN COMMAND button. For more information, see the Best Practices guide and API Reference

Preview run

Run logs

The Run Logs screen displays a log of all historic runs, their statuses, and their individual log outputs. To access the Run Logs screen, choose the Run logs item from the main menu.

Run details

When a run is selected in the Run Logs panel, its details and log history are displayed in the Masking Run panel. While a run is still being executed, its log output will be streamed for continuous feedback on the run progress.

The run options used by this run can be found on the first log line.

Run logs

Database masking job summary

On completion of a database masking run, the run log will display the following information:

Masking run status: The final status of the run on completion. This will indicate whether the run completed successfully or failed.
Started at: The time at which the masking run was started.
Finished at: The time at which the run completed, failed or was cancelled.
Total time: The total time taken for the run.
Total tables masked: The total number of tables that were successfully masked.
Total columns masked: The total number of columns that were successfully masked.
Total rows masked: The total number of rows that were successfully masked.

An example of a successful database masking run log can be seen below:

Successful database run log

When a run fails, if continue_on_failure was not enabled, the total tables, columns and rows masked will be displayed as 0. While some rows or column may be masked, it cannot be determined which particular rows or columns are masked on a failed run, so all tables should be considered unmasked.

Failed database run log

If continue_on_failure was enabled for the run, the tables, columns and rows will instead reflect the total numbers of tables, columns and rows masked in successful mask_table and mask_unique_key tasks.

File masking job summary

On completion of a file masking run, the run log will display the following information:

Masking run status: The final status of the run on completion. This will indicate whether the run was completed successfully or failed.
Started at: The time at which the masking run was started.
Finished at: The time at which the run completed, failed or was cancelled.
Total time: The total time taken for the run.
Total files masked: The total number of files that were successfully masked. On failed runs, this will reflect the number of files that were masked by successful file masking tasks.

An example of a successful file masking run log can be seen below:

Successful database run log

An example of a failed file masking run log can be seen below:

Successful database run log

Connection and ruleset snapshots

Snapshots of the connection and ruleset are kept for every masking run, maintaining an historical record of the exact configuration that was used for each run. Connection and ruleset snapshots can be viewed for the selected masking run by clicking on the connection or ruleset name displayed on the run detail panel.

Run detail snapshot

A modal window will open to display the snapshots, as captured at the time of masking run creation. The snapshot status indicates whether the current connection or ruleset configuration has been changed since this snapshot was taken. Clicking the edit link will allow you to edit the current connection or ruleset corresponding to the displayed snapshot.

Run log connection snapshot Run log ruleset snapshot

There are three different statuses that may be displayed for a snapshot:

current: The details have not been changed since the run was performed.
modified: The details have been changed since the run was performed. The details shown no longer reflect the current state of the connection or ruleset.
deleted: The connection or ruleset no longer exists. If this is the case, there will be no option to edit.

Downloading a run log

While hovering over a run in the Run Logs panel, a download button will be shown. Clicking on this button will start a download containing the logs for this masking run.

Downloading a sensitive data discovery report

When a run_data_discovery task is included in the masking ruleset, the resulting report for each run can be downloaded by clicking either the shield icon on the run row in the Run Logs list, or the Discovery Report chip on the Masking run detail panel. The report will be downloaded in CSV format and may be opened in a text editor or spreadsheet viewer such as Microsoft Excel. See Sensitive Data Discovery for more details.

Cancelling a run

If you wish to cancel a run, you may do so with the following steps:

Select the run you wish to cancel from the list in the 'Run Logs' panel.
Click the CANCEL RUN button at the bottom of the screen.
After clicking YES on the confirmation dialog, the run status will be updated to cancelling and DataMasque will proceed to stop the run's in-progress masking tasks.
Once all the run's tasks have been stopped, the run status will be updated to cancelled.

Note: During task cancellation, DataMasque will send an explicit request to the database to cancel any running queries. In most cases (if the query is in an interruptable phase), the database will catch this and stop the query immediately. If the query is in a non-interruptable phase, the database will still complete that phase before the query is terminated.

Run History table

When a masking run completes, regardless of success or failure, DataMasque will attempt to record the result of a masking run in the database being masked.

The table DATAMASQUE_RUN_HISTORY is created, if it does not exist on the default schema for the user performing the masking. This occurs on database masking runs, and only masking runs that would (or did) actually mask something. For example dry runs or schema/data discovery runs will not be recorded in the history table.

This table acts as a record of masking runs, and to help identify that a masking run did actually occur. It does not guarantee that any amount of data has actually been masked.

The format of the table is as follows:

Column Name	Data Type	Description	Example Value
`completion_time`	`DATETIME` or `TIMESTAMP`	Time the run completed, in UTC	`2024-01-15 15:23:11`
`dm_version`	`VARCHAR(80)`	The DataMasque version	2.18.0
`run_status`	`VARCHAR(15)`	The result of the masking run	`finished`, `failed` or `cancelled`
`run_id`	`NUMBER` or `INT`	The ID of the run (as used in API or run logs)	156
`ruleset_uuid`	`VARCHAR(36)`	A unique identifier for each ruleset	`516aef12-98ce-5192-a9c8-154a8c9b8d12`
`ruleset_content_sha256`	`VARCHAR(64)`	A hash of the contents of the ruleset used for masking	64-character SHA256 Hash
`connection_snapshot`	`VARCHAR(150)`	Summary of the databases connection, including type, username and host. Does not include the password.	`MySQL - mysql_user@mysql-host:3306`
`metadata`	See note below	Extra information specific to each masking run	Currently unused

Note: the metadata column is stored as a CLOB for Oracle, JSONB for Postgres, JSON for MySQL/MariaDB, and NVARCHAR(MAX) for MSSQL

A new row is added for every masking run, assuming the ruleset is valid and a run finishes.

Insertion and creation of the DATAMASQUE_RUN_HISTORY table is automatic, and will not cause problems if any part of it fails.

For example, if the user for the database connection of the masking run does not have the access required to create a table, this operation will silently fail, and masking will not be affected.

As the ruleset content is not captured in the table, manual validation is required to compare it against the ruleset_content_sha256.

To ensure consistency, the ruleset's content needs to be hashed using SHA256, resulting in the same hash stored in the DATAMASQUE_RUN_HISTORY table. Ensure that the ruleset content is encoded in UTF-8 format and includes any leading or trailing spaces for accurate hash matching.

Note: This run result table only exists for database masking. Only MariaDB, MySQL, MSSQL, PostgreSQL, and Oracle databases support the mask history table.

Simultaneous runs

To the same database

Simultaneous runs to the same database are not currently supported in DataMasque, as simultaneous masking can result in data being incorrectly masked. When there is a masking run in the status of queued, running or cancelling, subsequent masking runs to the same database connection cannot be scheduled.

If you wish to mask multiple tables in the same database simultaneously, it is recommended to use parallel tasks in a single ruleset.

To different databases

Simultaneous runs to different databases are supported in DataMasque.

For file connections

Simultaneous runs to the same source or destination file connection are not supported.