Masking runs
Overview
A masking run is the application of a masking Ruleset to a database Connection. Runs are created from the Database Masking Dashboard, and can also be triggered via the API (see Best Practices).
Create run
A new masking run can be configured with the following steps:
- Navigate to the Dashboard
- Select a connection from the list of available connections.
- Select a ruleset from the list of available rulesets.
- Set additional run options (detailed below)
- Click PREVIEW RUN or PREVIEW DRY RUN*
- After configuring the run, you will be taken to the Preview and confirm run screen.
*Dry Run allows you to test your rulesets without modifying the database. When a dry run is executed, DataMasque performs every operation as usual except:
- the final
UPDATE
operation ofmask_table
tasks, which would otherwise write the masked value to the database- the value generation and subsequent
UPDATE
operation formask_unique_key
tasks- the truncate_table operation
- the run_sql operation
Run options
Run options are displayed on the Run Options section of dashboard page. The following options are available:
Option | Default | Description |
---|---|---|
Batch Size1 | 50,000 |
The maximum number of rows that will be fetched, masked, and updated in a single operation by DataMasque. Larger batch sizes will reduce database operation overhead, but using a batch size value that is too large may result in DataMasque memory exhaustion. This value does not affect the total number of rows masked. The maximum allowed batch size is 50,000. Note: The Batch Size parameter is not applicable for Amazon Redshift masking runs. Only for database masking runs. |
MAXFILESIZE2 | 10 |
The MAXFILESIZE parameter is applicable to Amazon Redshift databases and is only available as a parameter when using using DataMasque APIs currently. The maximum file size in MB of records that will be unloaded (as MAXFILESIZE of Amazon Redshift UNLOAD command), masked, and loaded in a single operation by DataMasque. Defaults to 10 MB, but can be set to any integer value between 5 MB and 1000 MB. Only for database masking runs. |
Max rows34 | unset |
The maximum number of rows that will be masked by each mask_table task3. May be used for speeding up test iterations when developing rulesets. Warning: In the case that a table contains more rows than the value specified here, the remaining rows will contain unmasked data. Note: The Max rows parameter is not applicable for Amazon Redshift masking runs. Only for database masking runs. |
Continue on failure | false |
If there is a task failure, and this option is false, DataMasque will skip all remaining unstarted tasks. If this option is true, DataMasque will continue performing other tasks even if there is a task failure. It can be useful to set this option to true when testing/debugging your masking ruleset to identify as many failures as possible in each run. |
Run secret | unset |
The run secret is used in the random generation of masked values. Providing a consistent run secret will ensure that repeated runs on the same DataMasque instance will produce the same results. Note: If a random_seed value is provided in addition to the run_secret value, the random_seed will take precedence and the run_secret will be ignored. |
Disable instance secret | false |
If this option is set to true , DataMasque will exclude its instance-specific secret and generate masked values based solely on the run secret. You may wish to disable the instance secret in order to achieve consistent masking across DataMasque instances. However, by disabling the instance secret, any DataMasque instance using the same run_secret could replicate your data masking. |
Notes:
1
Batch Size
applies to database types other than Amazon Redshift.2
MAXFILESIZE
only applies to Amazon Redshift databases.3
Max rows
does not apply tomask_unique_key
tasks.4 Use of the
Max rows
run option for Amazon Redshift is not yet supported in DataMasque. This is on our roadmap and will be included in future releases.Preview and confirm run
This screen shows the preview of the configured run. Check the run parameters here before proceeding with execution via the START RUN button. After the run has been started, you will be redirected to the Run logs page where you can monitor the run output and progress.
You can view a curl
command for starting an equivalently configured run using the DataMasque API by clicking the VIEW RUN COMMAND button. For more information, see the Best Practices guide and API Reference
Run logs
The Run Logs screen displays a log of all historic runs, their statuses, and their individual log outputs. To access the Run Logs screen, choose the Run logs item from the main menu.
Run details
When a run is selected in the Run Logs panel, its details and log history are displayed in the Masking Run panel. While a run is still being executed, its log output will be streamed for continuous feedback on the run progress.
The run options used by this run can be found on the first log line.
Connection and ruleset snapshots
Snapshots of the connection and ruleset are kept for every masking run, maintaining an historical record of the exact configuration that was used for each run. Connection and ruleset snapshots can be viewed for the selected masking run by clicking on the connection or ruleset name displayed on the run detail panel.
A modal window will open to display the snapshots, as captured at the time of masking run creation. The snapshot status indicates whether the current connection or ruleset configuration has been changed since this snapshot was taken. Clicking the edit link will allow you to edit the current connection or ruleset corresponding to the displayed snapshot.
There are three different statuses that may be displayed for a snapshot:
- current: The details have not been changed since the run was performed.
- modified: The details have been changed since the run was performed. The details shown no longer reflect the current state of the connection or ruleset.
- deleted: The connection or ruleset no longer exists. If this is the case, there will be no option to edit.
Downloading a run log
While hovering over a run in the Run Logs panel, a download button will be shown. Clicking on this button will start a download containing the logs for this masking run.
Downloading a sensitive data discovery report
When a run_data_discovery
task is included in the
masking ruleset, the resulting report for each run can be downloaded by clicking either the shield
icon on the run row in the Run Logs list, or the Discovery Report chip on the
Masking run detail panel. The report will be downloaded in CSV format and may be opened in a
text editor or spreadsheet viewer such as Microsoft Excel. See Sensitive Data Discovery
for more details.
Cancelling a run
If you wish to cancel a run, you may do so with the following steps:
- Select the run you wish to cancel from the list in the 'Run Logs' panel.
- Click the CANCEL RUN button at the bottom of the screen.
- After clicking YES on the confirmation dialog, the run status will be updated to
cancelling
and DataMasque will proceed to stop the run's in-progress masking tasks. - Once all of the run's tasks have been stopped, the run status will be updated to
cancelled
.
Note: During task cancellation, DataMasque will send an explicit request to the database to cancel any running queries. In most cases (if the query is in an interruptable phase), the database will catch this and stop the query immediately. If the query is in a non-interruptable phase, the database will still complete that phase before the query is terminated.
Simultaneous runs
To the same database
Simultaneous runs to the same database are not currently supported in DataMasque,
as simultaneous masking can result in data being incorrectly masked.
When there is a masking run in the status of queued
, running
or cancelling
,
subsequent masking runs to the same database connection cannot be scheduled.
If you wish to mask multiple tables in the same database simultaneously, it is recommended to utilise parallel tasks in a single ruleset.
To different databases
Simultaneous runs to different databases are supported in DataMasque.