General Troubleshooting and Maintenance

• Installation and Maintenance
  • Installation with HSTS enabled
  • Restarting DataMasque
  • Checking the status of DataMasque containers
  • Force Downgrade or Reinstall
  • Fixing installation if DataMasque images are deleted
  • Viewing container logs
• General issues
  • Elapsed run duration is incorrect
  • Masking run stalls
  • User is unable to sign in
  • Dangling connections
  • DataMasque fails to open with a 'Disallowed host' error
  • Operation is not permitted error when installing or upgrading DataMasque
  • Upgrading to a newer version of DataMasque fails
  • Resetting a User's login attempts or changing a user's password
  • DataMasque containers have stopped
  • Changes to docker-compose.yml are not picked up
  • Docker-Compose is not reading environment variables

Installation and Maintenance

Installation with HSTS enabled

On browsers with HSTS enforced on the domain used to access DataMasque, the browser will block access to DataMasque if the default self-signed SSL certificate is in use. In this case, we recommend installing your own SSL credentials which are valid and trusted on your network.

To access DataMasque prior to installing your own trusted SSL certificate, you may replace the instance hostname with the server's IP address in the browser's address bar. For example, if your instance hostname is my-datamasque-server.internal.org.domain, and HSTS is enforced your organisation's internal.org.domain, you would replace https://my-datamasque-server.internal.org.domain with https://<datamasque-server-ip> where <datamasque-server-ip> is the external IP address of your server.

Restarting DataMasque

To restart the DataMasque Docker containers, run the following command (if your user does not belong to the docker and datamasque groups, you will need to escalate to root privileges by prefixing the command with sudo):

docker-compose -f <path-to-datamasque-installation>docker-compose.yml restart

Replacing the path with your own installation path, defaults to: /usr/local/etc/datamasque/).

You can verify that all five DataMasque containers have successfully restarted by running the following command:

docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"

Checking the status of DataMasque containers

To verify the status of the DataMasque containers, run the following command.

docker ps

For installations with Podman:

sudo podman ps

Displayed below is an example output for the command.

CONTAINER ID   IMAGE                                         COMMAND                  CREATED        STATUS          PORTS                                                                                    NAMES
72cb9569262a   datamasque/admin-frontend:2-10-1-final-9539   "/entrypoint.sh"         28 hours ago   Up 53 minutes   80/tcp, 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp   datamasque_admin-frontend_1
d53939f9c9d6   datamasque/admin-server:2-10-1-final-9539     "/entrypoint.sh"         28 hours ago   Up 53 minutes                                                                                            datamasque_admin-server_1
583fc8ae5164   datamasque/agent:2-10-1-final-9539            "/entrypoint.sh"         28 hours ago   Up 53 minutes                                                                                            datamasque_agent-worker_1
9b293366ebd5   datamasque/admin-db:2-10-1-final-9539         "docker-entrypoint.s…"   28 hours ago   Up 53 minutes   5432/tcp                                                                                 datamasque_admin-db_1
e384b3101326   datamasque/agent-queue:2-10-1-final-9539      "docker-entrypoint.s…"   28 hours ago   Up 53 minutes   6379/tcp                                                                                 datamasque_agent-queue_1

You should be able to see containers with the following names:

datamasque_admin-frontend_1
datamasque_admin-server_1
datamasque_agent-worker_1
datamasque_admin-db_1
datamasque_agent-queue_1

These containers should also be marked with a container ID, an image name, as well as the status. The status for these containers should be up with a time stamp of how long they have been up for.

Force Downgrade or Reinstall

It is possible to downgrade to a prior version of DataMasque using the --force flag in conjunction with --upgrade, for example:

tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh --upgrade --force

For installations with Podman:

sudo ./install.sh --podman --upgrade --force

This can also be used to reinstall the same version of DataMasque again.

Warning: Use of the --force flag may cause inconsistencies in your DataMasque storage database (not target databases) if the DataMasque schema has changed between versions. This may require the complete removal of DataMasque's Docker volumes and the loss of all DataMasque related data.

For this reason, it is not recommended to use --force unless you have a backup of all DataMasque data.

Note that the earliest version of DataMasque that supports --force is 2.11.0 so downgrading below 2.11.0 is not possible without a full removal and reinstall (see the next section How to fix installation if DataMasque images are deleted).

Fixing installation if DataMasque images are deleted

If one or more of DataMasque's Docker images are removed from the Docker engine, you can use the docker load command to manually reload the images from the DataMasque Docker Compose package:

tar -xvzf DataMasque-<version>.pkg
cd DataMasque-<version>/
for image_name in ./images/*; do
  docker load -i "$image_name"
done

For installations with Podman:

for image_name in ./images/*; do
  sudo podman load -i "$image_name";
done

Once you have reloaded the images, you can recreate the Docker containers and restart DataMasque by running:

docker-compose -f <path-to-datamasque-installation>docker-compose.yml up -d

Replacing the path with your own installation path, defaults to: /usr/local/etc/datamasque/).

Note that reloading the Docker images and recreating Docker containers with docker-compose will not delete DataMasque's application data, as long as DataMasque's Docker volumes are not removed.

Viewing container logs

For DataMasque 2.17 and newer

DataMasque logs can be downloaded through the web UI, by selecting Logs in the sidebar, then clicking Application Logs…. If the web UI is not accessible, then follow the instructions for DataMasque older than 2.17.0 below.

For DataMasque older than 2.17.0

Important DataMasque logs can be extracted from the Docker containers with the following commands.

Create a directory to store the logs:

  mkdir -p <path to a log directory>

Then, follow the instructions below, which vary according to DataMasque version. Choose the section corresponding to the version of DataMasque that generated the logs you want (normally the current version - but if you have recently upgraded, you may be looking for logs from the previous version).

Old log files are not deleted on upgrade, so if you have upgraded to 2.16.1 or newer from a previous version, you can follow both sections to retrieve all log files.

For DataMasque 2.16.1 and newer

DataMasque 2.16.1 introduced log rotation. There can now be up to 10 of each log file, hence the entire log directory must be copied from the container. The most recent log file has the extension .log while the others have the extension .log.1, .log.2, and so on.

To copy the DataMasque logs to a <log directory>:

sudo docker cp datamasque_admin-server_1:/files/logs/ <log directory>

DataMasque records three types of logs:

• Application runner logs are in the files starting with masque_requests.log.
• Web application logs are in the files starting with masque_admin_server.log.
• Masking agent logs are in the files starting with masque_agent.log.

For DataMasque 2.16.0 and earlier

• To copy the DataMasque web application runner logs to a <log directory>:

  sudo docker cp datamasque_admin-server_1:/files/logs/uwsgi.log <log directory>

• To copy the DataMasque web application logs to a <log directory>:

  sudo docker cp datamasque_admin-server_1:/files/logs/django.log <log directory>

• To copy the DataMasque masking agent logs to a <log directory>:

  sudo docker cp datamasque_agent-worker_1:/files/logs/celery.log <log directory>

• To obtain the STDOUT from a container, run docker logs <NAME OF CONTAINER>.
• For example, to obtain the STDOUT logs from the admin-db container, run the following command:

  docker logs datamasque_admin-db_1

General issues

Elapsed run duration is incorrect

Problem

During a masking run, the run duration timer on the 'Run Logs' page shows an incorrect duration.

Solution

The time on your DataMasque server is incorrect. Ensure your server is configured with the correct time or is synchronised with a valid NTP server.

---

Masking run stalls

Problem

Masking appears to stop midway through a run with no new logs appearing.

Solution

2.1 Ensure there are no active DataMasque processes still running or DataMasque processes are not blocked by other database transactions. If there are, the reason could be due to the slowness of the masking run. If additional optimisation is needed, please refer to our Performance Optimisation Guide. If there are no DataMasque processes still running, please follow the rest of the steps.

2.2 For Microsoft SQL Server - Ensure that all key columns (specified in mask_table tasks) are indexed. Un-indexed key columns may result in significant performance degradation and can also lead to deadlocks on Microsoft SQL Server

2.3 Check network performance between DataMasque and the target database. Slow networks with numerous hops between the database server and DataMasque can degrade performance. Co-locating DataMasque and target databases on the same network is recommended where possible.

---

User is unable to sign in

Problem

A user is unable to sign in. They receive an error message:

Unable to sign in with provided credentials.

Solutions

User sign-in will be denied in the following cases:

1 Disabled account

    Problem: The user account has been disabled from the Users management page (see user management).

    Solution: Enable the user account to allow user authentication.

SSO authentication is required

    Problem: The user account should be authenticated with SSO, but the user is attempting to sign in to DataMasque using local credentials.

    Solution: Check the Users management page to view the user account type (SSO or Local).

Local logins have been disabled

    Problem: The Disable local logins switch has been turned on in the application Settings.

    Solution: Disable this switch to allow local accounts to sign-in.

Incorrect password

    Problem: The user has entered an incorrect password.

    Solution: The user may reset their password if necessary using the "Forgot password?" link on the sign-in page¹.

¹Requires SMTP to be configured. See SMTP email settings.

---

Dangling connections

Problem

A warning may be displayed on the "Preview run" page if there is a possibility of a dangling connection. This is a reminder for you to clear dangling connections before starting the next masking run against that connection.

Dangling connections can occur in the following scenarios:

• When the DataMasque instance loses connectivity with a masking worker (e.g. due to a server or network outage), the worker process may still be operating.
• When a masking run is cancelled, any queries or commands already submitted may still be running (even once the run has reached the cancelled status).

Solutions

In order to avoid incorrect masking behaviour resulting from simultaneous masking of a database or file storage, it is recommended to clear any "dangling connections" that may still be operating on the target before starting a new masking run.

For example, this might involve killing any running DataMasque initiated queries or processes, and for dangling database connections, checking any blocking locks present on the database server. For issues with dangling database connections, it is also recommended to consider restoring the database to the pre-masking state before starting another masking run, in case the previous masking run has left the data partially masked.

---

DataMasque fails to open with a 'Disallowed host' error

Problem

You can no longer access DataMasque using the designated IP address or host name, due to it not being listed in the Hostnames settings.

Solution

In order to reset the allowed hosts, you will need to SSH into the target Linux server (or EC2 instance, VPS, etc.) on which DataMasque is installed, then run a few commands.

For DataMasque on Docker or Podman

1. SSH in to EC2 on which DataMasque is running.

2. For Docker, Run the following command to reset the allowed hosts:

   sudo docker exec -ti datamasque_admin-server_1 bash -c 'python3 reset_allowed_hosts.py'
   

   Or, for Podman, run:

   sudo podman exec -ti datamasque_admin-server_1 bash -c 'python3 reset_allowed_hosts.py'
   

3. Run the exit command to disconnect from the SSH session.

4. Visit the target Linux server or AWS EC2 instance IP address to be able to log in to DataMasque, navigate to the settings page and change the allowed hosts to the current IP Address.

For DataMasque on EKS

See the EKS troubleshooting guide.

---

Operation is not permitted error when installing or upgrading DataMasque

Problem

For RHEL: encountering the following error message when installing or upgrading DataMasque (sudo ./install.sh or sudo ./install.sh --upgrade)

Error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted

The issue is that /tmp doesn't have permission to execute.

Solution

1. Run the following command to mount /tmp with exec privilege.

   sudo mount /tmp -o remount,exec
   

2. Retry the failed install or upgrade DataMasque command again.

---

Upgrading to a newer version of DataMasque fails

Problem

When upgrading from an AWS Marketplace instance of DataMasque, the following error is experienced:

ERROR the command executing at the time of the error was: docker-compose down >> "$LOG_PATH" 2>&1

Upon viewing the log file, the error that is shown is Read timed out. Error from docker-compose.

Solution

1. SSH in to the instance performing the upgrade if not already there.

2. Change user to the root user: sudo su

3. Change directory to the installation path (defaults to: /usr/local/etc/datamasque).

4. Get the previous version tag:
   4.1: Run docker ps to see all the containers still running. 4.2: Copy everything to the right of the : from the image of one container: datamasque/admin-frontend:2-7-0-final-7906

5. Stop and remove the containers:

   docker-compose down
   

6. Run the following command to clean up the images of the previous version:

   docker images | grep datamasque/ | grep "<previoustag>" | awk '{ print $1":"$2 }' | xargs --no-run-if-empty docker rmi
   

   Swapping <previoustag> to the tag copied from step 4.2. e.g.

   docker images | grep datamasque/ | grep "2-7-0-final-7906" | awk '{ print $1":"$2 }' | xargs --no-run-if-empty docker rmi
   

7. Spin up the new images:

   docker-compose up -d
   

The new version of DataMasque should now be installed.

To potentially avoid this error in the future, run export COMPOSE_HTTP_TIMEOUT=120 on the EC2 instance to set a longer timeout for docker-compose as it could be due to a slow network.

---

Resetting a user's login attempts or changing a user's password

Problem

A user's account has been locked due to exceeding the max login attempts (5).

The error message Account locked: too many login attempts. Please try again later. appears when the user attempts to log-in.

Solution

1. SSH into the instance on which DataMasque is running.

2. Run the following commands (replacing with a valid username):

   • To reset the number of login attempts, run:

    sudo docker exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh axes_reset_username <username>'
   

   • To change the password for a user run:

    sudo docker exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh changepassword <username>'
   

   • Follow the prompts to change the password.
     

   OR if Podman is being used instead of Docker:

   • To reset the number of login attempts, run:

    sudo podman exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh axes_reset_username <username>'
   

   • To change the password for a user run:

    sudo podman exec -ti datamasque_admin-server_1 bash -c 'bash django_manage.sh changepassword <username>'
   

   • Follow the prompts to change the password.

3. Run the exit command to disconnect from the SSH session.

The user's password will now be reset, and they can attempt to log in with the new password.

---

DataMasque containers have stopped

Problem

The DataMasque containers have stopped running.

Solution

1. SSH into the instance on which DataMasque should be running.

2. Start the containers:

   • For installations with Docker:
     • Start the containers with docker-compose:

   sudo docker-compose -f <path-to-datamasque-installation>docker-compose.yml up -d
   

   Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

   • For installations with Podman:
     • Start the system services related to Podman:

   sudo systemctl start podman.socket
   sudo systemctl start datamasque_podman.service
   

---

Changes to docker-compose.yml are not picked up

Problem

Changes that have been made to the docker-compose.yml file and Docker-Compose is not them picking up.

Solution

1. SSH into the instance on which DataMasque should be running.

2. Stop the containers:

     sudo docker-compose -f <path-to-datamasque-installation>docker-compose.yml down
   

   Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

3. Start the containers:

   • For installations with Docker:
     • Start the containers with docker-compose:

   sudo docker-compose -f <path-to-datamasque-installation>docker-compose.yml up -d
   

   • For installations with Podman:
     • Start the system services related to Podman:

   sudo systemctl start podman.socket
   sudo systemctl start datamasque_podman.service
   

---

Docker-Compose is not reading environment variables

Problem

If Docker-Compose is not reading the environment variables from the .env file, created during installation, the following error will appear when running the following command:

admin-server_1    | django.db.utils.OperationalError: fe_sendauth: no password supplied
admin-server_1    |
admin-server_1    | Database initialization failed. Retrying in 10 seconds (1/15)

To see the logs, run:

docker-compose -f <path-to-datamasque-installation>docker-compose.yml logs

Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

Solution

1. SSH into the instance.

2. Source the environment needed variables, located in the installation path (defaults to: /usr/local/etc/datamasque/). If access is denied change to root user with the sudo su command.

   source <path-to-datamasque-installation>/.env
   

   Replacing the path with your own installation path (defaults to: /usr/local/etc/datamasque/).

3. Run Docker-Compose with the environment variables, make sure to include any variables that were added manually. The only environment variable added during installation is MASQUE_ADMIN_DB_PASSWORD.

   MASQUE_ADMIN_DB_PASSWORD=$MASQUE_ADMIN_DB_PASSWORD docker-compose -f <path-to-datamasque-installation>docker-compose.yml up -d
   

4. Confirm DataMasque is running as expected by loading it in a web browser.

   OR

   Check the Docker-Compose logs that no errors are being reported.

   docker-compose -f <path-to-datamasque-installation>docker-compose.yml logs