DataMasque Portal

General Troubleshooting

General issues

1. Elapsed run duration is incorrect

Problem

During a masking run, the run duration timer on the 'Run Logs' page shows an incorrect duration.

Solution

The time on your DataMasque server is incorrect. Ensure your server is configured with the correct time or is synchronised with a valid NTP server.

2. Masking run stalls

Problem

Masking appears to stop midway through a run with no new logs appearing.

Solution

2.1 Ensure there are no active DataMasque processes still running or DataMasque processes are not blocked by other database transactions. If there are, the reason could be due to slowness of the masking run. If additional optimisation is needed please refer to our Performance Optimisation Guide. If there are no DataMasque processes still running please follow the rest of the steps.

2.2 For Microsoft SQL Server - Ensure that all key columns (specified in mask_table tasks) are indexed. Un-indexed key columns may result in significant performance degradation and can also lead to deadlocks on Microsoft SQL Server

2.3 Check network performance between DataMasque and the target database. Slow networks with numerous hops between the database server and DataMasque can degrade performance. Co-locating DataMasque and target databases on the same network is recommended where possible.

3. User is unable to sign-in

Problem

A user is unable to sign-in. They receive an error message:

Unable to sign in with provided credentials.

Solutions

User sign-in will be denied in the following cases:

3.1 Disabled account

    Problem: The user account has been disabled from the Users management page (see user management).

    Solution: Enable the user account to allow user authentication.

3.2 SSO authentication is required

    Problem: The user account should be authenticated with SSO, but the user is attempting to sign in to DataMasque using local credentials.

    Solution: Check the Users management page to view the user account type (SSO or Local).

3.3 Local logins have been disabled

    Problem: The Disable local logins switch has been turned on in the application Settings.

    Solution: Disable this switch to allow local accounts to sign-in.

3.4 Incorrect password

    Problem: The user has entered an incorrect password.

    Solution: The user may reset their password if necessary using the "Forgot password?" link on the sign-in page1.

1Requires SMTP to be configured. See SMTP email settings.

4. Dangling connections

Problem

A warning may be displayed on the "Preview run" page if there is a possibility of a dangling connection. This a reminder for you to clear dangling connections before starting the next masking run against that connection.

Dangling connections can occur in the following scenarios:

  • When the DataMasque instance loses connectivity with a masking worker (e.g. due to a server or network outage), the worker process may still be operating.
  • When a masking run is cancelled, any queries or commands already submitted may still be running (even once the run has reached the cancelled status).

Solutions

In order to avoid incorrect masking behaviour resulting from simultaneous masking of a database or file storage, it is recommended to clear any "dangling connections" that may still be operating on the target before starting a new masking run.

For example, this might involve killing any running DataMasque initiated queries or processes, and for dangling database connections, checking any blocking locks present on the database server. For issues with dangling database connections, it is also recommended to consider restoring the database to the pre-masking state before starting another masking run, in case the previous masking run has left the data partially masked.

5. DataMasque fails to open with a 'Disallowed host' error

Problem

You can no longer access DataMasque using the designated IP address or host name, due to it not being listed in the Hostnames settings.

Solution

In order to reset the allowed hosts you will need to SSH into the target Linux server (or EC2 instance, VPS, etc.) on which DataMasque is installed, then run a few commands.

For DataMasque on Docker or Podman

  1. SSH in to EC2 on which DataMasque is running.

  2. For Docker, Run the following command to reset the allowed hosts:

    sudo docker exec -ti datamasque_admin-server_1 bash -c 'python3 reset_allowed_hosts.py'

    Or, for Podman, run:

    sudo podman exec -ti datamasque_admin-server_1 bash -c 'python3 reset_allowed_hosts.py'

  3. Run the exit command to disconnect from the SSH session.

  4. Visit the target Linux server or AWS EC2 instance IP address to be able to log in to DataMasque, navigate to the settings page and change the allowed hosts to the current IP Address.

For DataMasque on EKS

See the EKS troubleshooting guide.

6. Operation not permitted error when installing or upgrading DataMasque

Problem

For RHEL: encountered the following error message when installing or upgrading DataMasque (./install.sh or ./install.sh --upgrade) "Error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted".
The issue is that /tmp doesn't have permission to execute.

Solution

  1. Run the following command to mount /tmp with exec privilege.
    sudo mount /tmp -o remount,exec
  2. Retry the failed install or upgrade DataMasque command again.

7. Upgrading to a newer version of DataMasque fails

Problem

When upgrading from an AWS Marketplace instance of DataMasque the following error is experienced:

ERROR the command executing at the time of the error was: docker-compose down >> "$LOG_PATH" 2>&1

Upon viewing the log file, the error that is shown is Read timed out. Error from docker-compose.

Solution

  1. SSH in to the instance performing the upgrade if not already there.
  2. Change user to the root user: sudo su
  3. Change directory to the installation path (defaults to: /usr/local/etc/datamasque)
  4. Get the previous version tag:
    4.1: Run docker ps to see all the containers still running.
    4.2: Copy everything to the right of the : from the image of one container: datamasque/admin-frontend:2-7-0-final-7906
  5. Stop and remove the containers: docker-compose down
  6. Run the following command to clean up the images of the previous version: docker images | grep datamasque/ | grep "<previoustag>" | awk '{ print $1":"$2 }' | xargs --no-run-if-empty docker rmi
    Swapping <previoustag> to the tag copied from step 4.2.
    e.g. docker images | grep datamasque/ | grep "2-7-0-final-7906" | awk '{ print $1":"$2 }' | xargs --no-run-if-empty docker rmi
  7. Spin up the new images: docker-compose up -d

The new version of DataMasque should now be installed.

To potentially avoid this error in the future, run export COMPOSE_HTTP_TIMEOUT=120 on the EC2 instance to set a longer timeout for docker-compose as it could be due to a slow network.

8. Resetting a User's login attempts or changing a users password

Problem

A user's account has been locked due to exceeding the max login attempts (5).

The error message Account locked: too many login attempts. Please try again later. appears when the user attempts to log-in.

Solution

  1. SSH into the instance on which DataMasque is running.

  2. Run the following commands (replacing with a valid username):

    • To reset the number of login attempts run:
      sudo docker exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh axes_reset_username <username>'

    • To change the password for a user run:
      sudo docker exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh changepassword <username>' Follow the prompts to change the password.

    OR if Podman is being used instead of Docker

    • To reset the number of login attempts run:
      sudo podman exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh axes_reset_username <username>'

    • To change the password for a user run:
      sudo podman exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh changepassword <username>'
      Follow the prompts to change the password.

  3. Run the exit command to disconnect from the SSH session.

The user's password will now be reset, and they can attempt to log in with the new password.