Performance Optimisation

In this document, you'll discover guidelines focused on performance optimisation. It provides guidance on utilising multiple processes, allowing for parallel execution and, consequently, enhanced performance.

• Masking using parallelism and workers
• Masking a table with multiple workers
• Performing tasks in parallel
• Worker and parallel task count selection
• If performance does not increase
• Masking large text data

Masking using parallelism and workers

The total number of worker processes should not exceed twice the number of CPUs available to the DataMasque instance. For example, if your virtual machine has four CPUs, the total number of worker processes should not exceed eight.

The total number of worker processes is equal to the number of parallel tasks times the number of workers per task.

For example:

1 task in parallel * 2 workers = 2 worker processes

or

2 tasks in parallel * 2 workers = 4 worker processes

DataMasque can run a maximum of 10 parallel tasks simultaneously, however each task may have multiple workers (thus allowing more than 10 worker processes).

Masking a table with multiple workers

To improve masking performance on a single table, you can enable parallelism which allows multiple processes to work together simultaneously to mask a single table. This can be achieved simply by specifying a number of workers greater than 1 for a task.

It is also recommended to increase the batch size in addition to increasing the number of workers to achieve optimal performance. Increasing the batch size increases the number of rows that are fetched, masked, and updated in a single operation. This will reduce the database operation overhead at the cost of increasing the memory usage of DataMasque. More details on batch size can be found under the Database Run Options guide.

When using multiple workers (workers > 1), each worker process operates on a separate batch of rows and these worker processes will run simultaneously. This can result in reduced masking run time, and hence improved performance as more rows of the table are masked at once.

Note: Increasing the number of workers will increase the amount of memory used (as well as CPU consumption). It is recommended to monitor resource usage when using parallelism.

In the ruleset specification below workers: 4 is specified, therefore four worker processes will be used to mask the users table simultaneously.

version: "1.0"
tasks:
  - type: mask_table
    table: users
    workers: 4
    key: id
    rules:
      - column: last_name
        masks:
        - type: from_fixed
          value: 'redacted last name'

The following diagram describes how multiple worker processes work in the example ruleset above.

Notes:

• Number of rows in each buffer is set by the batch size parameter/run option.
• As each worker finishes masking a batch of rows, it will move on to the next unmasked batch of rows.

Performing tasks in parallel

When using the parallel task type, DataMasque performs masking using multiple processes which allows masking to run in parallel across multiple tables at once. Parallel tasks can reduce the time needed to mask a database when compared to performing masking on individual tables sequentially.

Below is an example ruleset of how mask_table table tasks can be set up in to run in parallel. Three tables are masked simultaneously in each parallel task block. Once the first three tables are masked, the next parallel task block is executed, until finally all three parallel task blocks are complete and all tables in the ruleset are masked.

version: "1.0"
tasks:
  - type: parallel
    tasks:
      - type: mask_table
        table: table_1
        ...
      - type: mask_table
        table: table_2
        ...
      - type: mask_table
        table: table_3
        ...
  - type: parallel
    tasks:
      - type: mask_table
        table: table_4
        ...
      - type: mask_table
        table: table_5
        ...
      - type: mask_table
        table: table_6
        ...
  - type: parallel
    tasks:
      - type: mask_table
        table: table_7
        ...
      - type: mask_table
        table: table_8
        ...
      - type: mask_table
        table: table_9
        ...

Note: mask_unique_key tasks are not allowed to be run in parallel.

The following diagram describes how parallel execution works in the example ruleset shown above.

Worker and parallel task count selection

As a general guideline, DataMasque should be configured to execute two workers or parallel tasks per CPU (or vCPU).

For example, a virtual machine with two vCPUs could run:

• A single mask_table task with four workers; or,
• Two mask_table tasks in parallel, each with two workers; or,
• Four mask_table tasks in parallel, each with one worker, etc.

Increasing the worker or parallel count will also increase the amount of memory that DataMasque consumes. Therefore, it is crucial to monitor memory usage carefully when adjusting these settings.

We recommend experimenting with different worker counts to find the optimal configuration for your environment, while monitoring memory usage if increasing worker counts.

Memory consumption will increase if any of the following increase:

• Worker count.
• The number and types of columns being masked (database masking).
• Batch size (database masking).
• The size of individual files being masked (file masking).

Memory consumption does not increase based on the number of rows in a table, or the number of files being masked.

Once you have a stable number of workers for a particular table or file structure, the worker count should not need to be adjusted as the number of rows in the table grows or number of files to mask increases.

If performance does not increase

If you find that the performance of DataMasque does not improve as you add more workers, then you will need to identify the bottleneck.

First, make sure the DataMasque instance itself has enough free resources. Some masking configurations or certain tables/files may use more CPU and memory than others. Even if you are under the recommended number of workers per vCPU, check that the DataMasque instance has not exhausted its resources.

If the DataMasque instance has CPU, memory and IOPS capacity, it may indicate that the bottleneck lies elsewhere.

Consider the following possibilities:

• Database Server Limitations: Your database server might need additional resources, such as more CPU power, higher IOPS, or increased memory, to handle the increased load efficiently.
• Network Speed: For file masking tasks, particularly when dealing with cloud storage, performance is heavily dependent on your network speed. To a lesser extent, network speed can also affect database masking tasks, especially if there is high latency. Check for network saturation, and try to host your database(s) and DataMasque instance in the same region and availability zone if possible.
• Disk I/O: The speed of the mounted file share can also impact file masking tasks. If you are working with on-premise or mounted storage, consider upgrading to faster disks or optimizing the storage configuration.

In these cases, adjusting the resources allocated to your database server or network infrastructure may yield better results. We recommend continuing to monitor resource usage closely and experimenting with these settings to find the optimal configuration for your environment.

Masking large text data

To mask data, DataMasque loads data into memory in batches. The memory consumed depends on the size of each row and the number of rows processed at a time.

For smaller data types, such as integers, floats, dates, and small VARCHAR columns, memory usage is typically minimal. Since these data types result in small row sizes, usually only a few kilobytes, they generally do not cause memory issues.

However, when masking larger data types, memory usage becomes more significant. Large text fields, for instance, can use up to 4 times their size in memory when loaded into DataMasque. For example, a single TEXT row of 100 MB would consume 400 MB of memory. If you are masking 20 rows, each 100 MB in size, the memory requirement becomes: 20 rows x 100 MB x 4 = 8000 MB. This would exceed the available memory in an 8 GB DataMasque instance.

To avoid memory exhaustion, you can reduce the batch_size to a lower value based on the size of the rows being processed. For example, masking 5 rows instead of 20 would significantly reduce memory usage.

Additionally, certain mask types, such as xml, json, and imitate, require extra memory during masking operations. These types can consume up to 20 times the size of the data being masked. When working with large text data, be cautious to avoid memory overload.