DataMasque Portal

General Troubleshooting and Maintenance

Installation and Maintenance

Installation with HSTS enabled

On browsers with HSTS enforced on the domain used to access DataMasque, the browser will block access to DataMasque if the default self-signed SSL certificate is in use. In this case, we recommend installing your own SSL credentials which are valid and trusted on your network.

To access DataMasque prior to installing your own trusted SSL certificate, you may replace the instance hostname with the server's IP address in the browser's address bar. For example, if your instance hostname is my-datamasque-server.internal.org.domain, and HSTS is enforced your organisation's internal.org.domain, you would replace https://my-datamasque-server.internal.org.domain with https://<datamasque-server-ip> where <datamasque-server-ip> is the external IP address of your server.

Restarting DataMasque

To restart the DataMasque Docker containers, run the following command (if your user does not belong to the docker and datamasque groups, you will need to escalate to root privileges by prefixing the command with sudo):

Docker:

docker compose -f <path-to-datamasque-installation>docker-compose.yml restart

Podman:

docker-compose -f <path-to-datamasque-installation>docker-compose.yml restart

Replacing the path with your own installation path, defaults to: /usr/local/etc/datamasque/.

You can verify that all five DataMasque containers have successfully restarted by running the following command:

docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"

Checking the status of DataMasque containers

To verify the status of the DataMasque containers, run the following command.

docker ps

For installations with Podman:

sudo podman ps

Displayed below is an example output for the command.

CONTAINER ID   IMAGE                                         COMMAND                  CREATED        STATUS          PORTS                                                                                    NAMES
72cb9569262a   datamasque/admin-frontend:2-10-1-final-9539   "/entrypoint.sh"         28 hours ago   Up 53 minutes   80/tcp, 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp   datamasque_admin-frontend_1
d53939f9c9d6   datamasque/admin-server:2-10-1-final-9539     "/entrypoint.sh"         28 hours ago   Up 53 minutes                                                                                            datamasque_admin-server_1
583fc8ae5164   datamasque/agent:2-10-1-final-9539            "/entrypoint.sh"         28 hours ago   Up 53 minutes                                                                                            datamasque_agent-worker_1
9b293366ebd5   datamasque/admin-db:2-10-1-final-9539         "docker-entrypoint.s…"   28 hours ago   Up 53 minutes   5432/tcp                                                                                 datamasque_admin-db_1
e384b3101326   datamasque/agent-queue:2-10-1-final-9539      "docker-entrypoint.s…"   28 hours ago   Up 53 minutes   6379/tcp                                                                                 datamasque_agent-queue_1

You should be able to see containers with the following names:

datamasque_admin-frontend_1
datamasque_admin-server_1
datamasque_agent-worker_1
datamasque_admin-db_1
datamasque_agent-queue_1

These containers should also be marked with a container ID, an image name, as well as the status. The status for these containers should be up with a time stamp of how long they have been up for.

Force Downgrade or Reinstall

It is possible to downgrade to a prior version of DataMasque using the --force flag in conjunction with --upgrade. If you are downgrading from version v2.24.0 or higher to a version prior to v2.24.0, you must first stop DataMasque by running docker compose down:

# Become root, as the DataMasque installation directory is owned by root.
sudo -s

# Change to the directory where you installed DataMasque;
# /usr/local/etc/datamasque is the default.
cd /usr/local/etc/datamasque

# Stop DataMasque.
# If using Docker Compose standalone, use `/path/to/docker-compose down` instead,
# specifying the path to the Docker Compose standalone executable.
docker compose down

# Log out of the root shell.
exit

Warning: Ensure all the containers display as Stopped and that the docker compose down command does not display any errors. You can verify all the containers are stopped with docker ps (or podman ps), which should display no running containers.

If any containers fail to stop or any errors are displayed, do not proceed with the downgrade and contact DataMasque support for further troubleshooting.

If the current version and prior version are both v2.23.1 or earlier, or both at least v2.24.0, then the above step is not required but is safe to perform anyway.

Next, proceed with the downgrade, by performing an "upgrade" to the prior version:

tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh --upgrade --force

For installations with Podman, replace the last command with:

sudo ./install.sh --podman --upgrade --force

This can also be used to reinstall the same version of DataMasque again.

Warning: Use of the --force flag may cause inconsistencies in your DataMasque storage database (not target databases) if the DataMasque schema has changed between versions. This may require the complete removal of DataMasque's Docker volumes and the loss of all DataMasque related data.

For this reason, it is not recommended to use --force unless you have a backup of all DataMasque data.

Note that the earliest version of DataMasque that supports --force is 2.11.0 so downgrading below 2.11.0 is not possible without a full removal and reinstall (see the next section Fixing installation if DataMasque images are deleted).

Fixing installation if DataMasque images are deleted

If one or more of DataMasque's Docker images are removed from the Docker engine, you can use the docker load command to manually reload the images from the DataMasque Docker Compose package:

tar -xvzf DataMasque-<version>.pkg
cd DataMasque-<version>/
for image_name in ./images/*; do
  docker load -i "$image_name"
done

For installations with Podman:

for image_name in ./images/*; do
  sudo podman load -i "$image_name";
done

Once you have reloaded the images, recreate the Docker containers and restart DataMasque by following the process for Restarting DataMasque.

Viewing container logs

For DataMasque 2.17 and newer

DataMasque logs can be downloaded through the web UI, by selecting Logs in the sidebar, then clicking Application Logs…. If the web UI is not accessible, then follow the instructions for DataMasque older than 2.17.0 below.

For DataMasque older than 2.17.0

Important DataMasque logs can be extracted from the Docker containers with the following commands.

Create a directory to store the logs:

  mkdir -p <path to a log directory>

Then, follow the instructions below, which vary according to DataMasque version. Choose the section corresponding to the version of DataMasque that generated the logs you want (normally the current version - but if you have recently upgraded, you may be looking for logs from the previous version).

Old log files are not deleted on upgrade, so if you have upgraded to 2.16.1 or newer from a previous version, you can follow both sections to retrieve all log files.

For DataMasque 2.16.1 and newer

DataMasque 2.16.1 introduced log rotation. There can now be up to 10 of each log file, hence the entire log directory must be copied from the container. The most recent log file has the extension .log while the others have the extension .log.1, .log.2, and so on.

To copy the DataMasque logs to a <log directory>:

sudo docker cp datamasque_admin-server_1:/files/logs/ <log directory>

DataMasque records four types of logs:

  • Application runner logs are in the files starting with masque_requests.log.
  • Web application logs are in the files starting with masque_admin_server.log.
  • Masking agent logs are in the files starting with masque_agent.log.
  • In-flight Server logs are in the files starting with masque_in_flight_server.log.

For DataMasque 2.16.0 and earlier

  • To copy the DataMasque web application runner logs to a <log directory>:
  sudo docker cp datamasque_admin-server_1:/files/logs/uwsgi.log <log directory>
  • To copy the DataMasque web application logs to a <log directory>:
  sudo docker cp datamasque_admin-server_1:/files/logs/django.log <log directory>
  • To copy the DataMasque masking agent logs to a <log directory>:
  sudo docker cp datamasque_agent-worker_1:/files/logs/celery.log <log directory>
  • To obtain the STDOUT from a container, run docker logs <NAME OF CONTAINER>.
  • For example, to obtain the STDOUT logs from the admin-db container, run the following command:
  docker logs datamasque_admin-db_1

General issues

Elapsed run duration is incorrect

Problem

During a masking run, the run duration timer on the 'Run Logs' page shows an incorrect duration.

Solution

The time on your DataMasque server is incorrect. Ensure your server is configured with the correct time or is synchronised with a valid NTP server.

Masking run stalls

Problem

Masking appears to stop midway through a run with no new logs appearing.

Solution

2.1 Ensure there are no active DataMasque processes still running or DataMasque processes are not blocked by other database transactions. If there are, the reason could be due to the slowness of the masking run. If additional optimisation is needed, please refer to our Performance Optimisation Guide. If there are no DataMasque processes still running, please follow the rest of the steps.

2.2 For Microsoft SQL Server - Ensure that all key columns (specified in mask_table tasks) are indexed. Un-indexed key columns may result in significant performance degradation and can also lead to deadlocks on Microsoft SQL Server

2.3 Check network performance between DataMasque and the target database. Slow networks with numerous hops between the database server and DataMasque can degrade performance. Co-locating DataMasque and target databases on the same network is recommended where possible.

User is unable to sign in

Problem

A user is unable to sign in. They receive an error message:

Unable to sign in with provided credentials.

Solutions

User sign-in will be denied in the following cases:

Disabled account

    Problem: The user account has been disabled from the Users management page (see user management).

    Solution: Enable the user account to allow user authentication.

SSO authentication is required

    Problem: The user account should be authenticated with SSO, but the user is attempting to sign in to DataMasque using local credentials.

    Solution: Check the Users management page to view the user account type (SSO or Local).

Local logins have been disabled

    Problem: The Disable local logins switch has been turned on in the application Settings.

    Solution: Disable this switch to allow local accounts to sign-in.

Incorrect password

    Problem: The user has entered an incorrect password.

    Solution: The user may reset their password if necessary using the "Forgot password?" link on the sign-in page1.

1Requires SMTP to be configured. See SMTP email settings.

Dangling connections

Problem

A warning may be displayed on the "Preview run" page if there is a possibility of a dangling connection. This is a reminder for you to clear dangling connections before starting the next masking run against that connection.

Dangling connections can occur in the following scenarios:

  • When the DataMasque instance loses connectivity with a masking worker (e.g. due to a server or network outage), the worker process may still be operating.
  • When a masking run is cancelled, any queries or commands already submitted may still be running (even once the run has reached the cancelled status).

Solutions

In order to avoid incorrect masking behaviour resulting from simultaneous masking of a database or file storage, it is recommended to clear any "dangling connections" that may still be operating on the target before starting a new masking run.

For example, this might involve killing any running DataMasque initiated queries or processes, and for dangling database connections, checking any blocking locks present on the database server. For issues with dangling database connections, it is also recommended to consider restoring the database to the pre-masking state before starting another masking run, in case the previous masking run has left the data partially masked.

DataMasque fails to open with a 'Disallowed host' error

Problem

You can no longer access DataMasque using the designated IP address or host name, due to it not being listed in the Hostnames settings.

Solution

In order to reset the allowed hosts, you will need to SSH into the target Linux server (or EC2 instance, VPS, etc.) on which DataMasque is installed, then run a few commands.

For DataMasque on Docker or Podman

  1. SSH in to EC2 on which DataMasque is running.

  2. For Docker, Run the following command to reset the allowed hosts:

    sudo docker exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh reset_allowed_hosts'

    Or, for Podman, run:

     sudo podman exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh reset_allowed_hosts'

  3. Run the exit command to disconnect from the SSH session.

  4. Visit the target Linux server or AWS EC2 instance IP address to be able to log in to DataMasque, navigate to the settings page and change the allowed hosts to the current IP Address.

For DataMasque on EKS

See the EKS troubleshooting guide.

Operation is not permitted error when installing or upgrading DataMasque

Problem

For RHEL: encountering the following error message when installing or upgrading DataMasque (sudo ./install.sh or sudo ./install.sh --upgrade)

Error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted

The issue is that /tmp doesn't have permission to execute.

Solution

  1. Run the following command to mount /tmp with exec privilege.

    sudo mount /tmp -o remount,exec

  2. Retry the failed install or upgrade DataMasque command again.

Upgrading to a newer version of DataMasque fails

Problem

When upgrading from an AWS Marketplace instance of DataMasque, the following error is experienced:

ERROR the command executing at the time of the error was: docker compose down >> "$LOG_PATH" 2>&1

Upon viewing the log file, the error that is shown is Read timed out. Error from docker compose.

Solution

  1. SSH in to the instance performing the upgrade if not already there.

  2. Change user to the root user: sudo su

  3. Change directory to the installation path (defaults to: /usr/local/etc/datamasque).

  4. Get the previous version tag:
    4.1: Run docker ps to see all the containers still running. 4.2: Copy everything to the right of the : from the image of one container: datamasque/admin-frontend:2-7-0-final-7906

  5. Stop and remove the containers:

    docker compose down

  6. Run the following command to clean up the images of the previous version:

    docker images | grep datamasque/ | grep "<previoustag>" | awk '{ print $1":"$2 }' | xargs --no-run-if-empty docker rmi

    Swapping <previoustag> to the tag copied from step 4.2. e.g.

    docker images | grep datamasque/ | grep "2-7-0-final-7906" | awk '{ print $1":"$2 }' | xargs --no-run-if-empty docker rmi

  7. Spin up the new images:

    docker compose up -d

The new version of DataMasque should now be installed.

To potentially avoid this error in the future, run export COMPOSE_HTTP_TIMEOUT=120 on the EC2 instance to set a longer timeout for docker compose as it could be due to a slow network.

Resetting a user's login attempts or changing a user's password

Problem

A user's account has been locked due to exceeding the max login attempts (5).

The error message Account locked: too many login attempts. Please try again later. appears when the user attempts to log-in.

Solution

  1. SSH into the instance on which DataMasque is running.

  2. Run the following commands (replacing with a valid username):

    • To reset the number of login attempts, run:
     sudo docker exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh axes_reset_username <username>'
    • To change the password for a user run:
     sudo docker exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh changepassword <username>'
    • Follow the prompts to change the password.

    OR if Podman is being used instead of Docker:

    • To reset the number of login attempts, run:
     sudo podman exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh axes_reset_username <username>'
    • To change the password for a user run:
     sudo podman exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh changepassword <username>'
    • Follow the prompts to change the password.

  3. Run the exit command to disconnect from the SSH session.

The user's password will now be reset, and they can attempt to log in with the new password.

DataMasque containers have stopped

Problem

The DataMasque containers have stopped running.

Solution

  1. SSH into the instance on which DataMasque should be running.

  2. Start the containers:

    • For installations with Docker:
      • Start the containers with docker compose:
    sudo docker compose -f <path-to-datamasque-installation>docker-compose.yml up -d

    Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

    • For installations with Podman:
      • Start the system services related to Podman:
    sudo systemctl start podman.socket
sudo systemctl start datamasque_podman.service

Changes to docker-compose.yml are not picked up

Problem

Changes that have been made to the docker-compose.yml file and Docker Compose is not them picking up.

Solution

  1. SSH into the instance on which DataMasque should be running.

  2. Stop the containers:

      sudo docker compose -f <path-to-datamasque-installation>docker-compose.yml down

    Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

  3. Start the containers:

    • For installations with Docker:
      • Start the containers with docker compose:
    sudo docker compose -f <path-to-datamasque-installation>docker-compose.yml up -d
    • For installations with Podman:
      • Start the system services related to Podman:
    sudo systemctl start podman.socket
sudo systemctl start datamasque_podman.service

Docker Compose is not reading environment variables

Problem

If Docker Compose is not reading the environment variables from the .env file, created during installation, the following error will appear when running the following command:

admin-server_1    | django.db.utils.OperationalError: fe_sendauth: no password supplied
admin-server_1    |
admin-server_1    | Database initialization failed. Retrying in 10 seconds (1/15)

To see the logs, run:

docker compose -f <path-to-datamasque-installation>docker-compose.yml logs

Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

Solution

  1. SSH into the instance.

  2. Source the environment needed variables, located in the installation path (defaults to: /usr/local/etc/datamasque/). If access is denied change to root user with the sudo su command.

    source <path-to-datamasque-installation>/.env

    Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

  3. Run Docker Compose with the environment variables, make sure to include any variables that were added manually. The only environment variable added during installation is MASQUE_ADMIN_DB_PASSWORD.

    MASQUE_ADMIN_DB_PASSWORD=$MASQUE_ADMIN_DB_PASSWORD docker compose -f <path-to-datamasque-installation>docker-compose.yml up -d

  4. Confirm DataMasque is running as expected by loading it in a web browser.

    OR

    Check the Docker Compose logs that no errors are being reported.

    docker compose -f <path-to-datamasque-installation>docker-compose.yml logs

Masking errors

Conflicting data types

When masking, DataMasque checks that all masked data in a column is of the same data type. For example, a ruleset that uses conditional masking to mask some rows to integers and other rows to strings will fail with an error similar to this:

Mask for column `<column>` returned two mask types: "str" and "int". Please use a typecast if your mask could potentially return multiple datatypes.

Solution

To fix this, use the typecast mask to change the data type of the masked data. Preferably the masked data type should be of the correct type for the column (so string type for varchar, and so on), but at the very least, it must be consistent for all rows. This includes any rows that have not been masked due to use of if blocks, which retain their original column value and data type.

The following two masks have common pitfalls:

  • The from_random_number mask returns a number formatted as a string. Use a typecast mask with typecast_as set to float, decimal, or integer to get a numeric value.
  • The from_random_boolean mask returns the integer 0 or 1 rather than the Boolean values false and true. Use a typecast mask with typecast_as: boolean to get a Boolean value.

Deadlock errors on Microsoft SQL Server

When using multiple workers to mask a table on Microsoft SQL Server, you may see warning messages similar to the following:

Encountered error while fetching batch from database. Retrying the query (attempt 1 of 3).
    Database error: [40001] Transaction (Process ID 105) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. (1205) (SQLFetch)

This is caused by a deadlock occurring within the SQL Server database when one worker is reading a batch of rows to be masked (a SELECT query), and another worker is writing masked rows back to the database (an UPDATE query). The database rejects the SELECT query so that the UPDATE query can proceed.

Note: You may see this message many times, each with attempt 1 of 3. This is not unexpected; it simply indicates several workers are retrying their SELECT query. Each worker makes three attempts for each of the batches it is assigned to mask. The number of attempts is reset after finishing a batch.

Solution

When DataMasque detects this situation, it automatically retries the SELECT query up to two further times. Almost all the time one of these retries will succeed and masking can then proceed as normal. In this case, the warning message can be considered benign.

In the event that the third retry also fails, the task (and the masking run, if not using the Continue on failure option) will fail. The following suggestions may mitigate the problem:

  • Use a batch size of fewer than 4,000 rows. SQL Server has a threshold of 5,000 row and page locks at which it starts employing lock escalation. By keeping the number of queried rows and pages below this limit, the deadlock is avoided.
  • If the affected task is in a parallel block, move it out of the parallel block to reduce database contention.
  • Improve the performance of the database, for example by using a larger Amazon RDS instance size. This allows large queries to complete faster, thus reducing database contention.
  • Use fewer workers for the task.

Licensing issues

Unable to check out contract product license

Problem

If using a Contract Product license, an error is displayed when selecting the license type or when starting a masking run.

For example:

User: <arn> is not authorized to perform: license-manager:CheckoutLicense
because no identity-based policy allows the license-manager:CheckoutLicense action

or:

No Entitlements Allowed.

Solution

For not authorized errors, make sure your EC2 instance has been configured correctly.

  1. The Instance Metadata Service (IMDS) must be accessible. Without this configured correctly, DataMasque will not be able to retrieve information about your IAM Role, product ID or instance ID. Refer to IMDS Version guide for more information.

  2. An IAM role with license-manager:CheckoutLicense and license-manager:CheckInLicense permissions must be attached to the EC2 instance. Refer to the Contract Product guide for information on IAM policy setup.

For No Entitlements Allowed errors:

  1. Make sure the DataMasque license you have purchased through your AWS account (Business or Enterprise) is the same as the type of license you have set in your DataMasque installation. You can check and change the license type on the My Account screen in DataMasque.

  2. Check the entitlements that are granted for the license in the License Manager in the AWS Console. If it shows that entitlements have been checked out, make sure you are not running more DataMasque instances than you have entitlements for.

  3. If you have recently terminated a DataMasque instance, and created a new one, the license the old instance was using may not have been checked back in. Licenses are checked out for an hour at a time; please wait an hour for the license to be automatically checked back in, and then try checking out the license again on the new instance. License check out is performed when switching the license type or starting a masking run.