DataMasque Portal

DataMasque Installation with Docker

This guide describes the process for deploying DataMasque with Docker Compose on a Linux server.

System requirements

Minimum requirements

  • Supported Operating Systems:

    • Ubuntu 18.04 or 20.04 LTS, x86 64-bit
    • RHEL 7 or 8, x86 64-bit

  • 4 vCPUs

  • 8 GiB RAM

  • 40 GiB available disk space

  • A static external IP or hostname

  • Ingress requests allowed on port 443 (and optionally port 80)

    • These ports are used to serve DataMasque's web interface.
    • If HTTP traffic is allowed on port 80, it will be automatically upgraded to HTTPS on port 443.

  • For admin access (SSH), ingress requests allowed on port 22.

DataMasque Ingress requests

  • Client web browser: Google Chrome 85 or above

  • Docker 19.03 to 20.10 with Docker Compose 1.17 to 1.29 (installation instructions - for RHEL, follow the CentOS installation instructions)

    • Docker should be installed from packages, not a convenience script.
    • You should ensure the user account that will be performing the DataMasque installation and administrative actions is added to the docker group. If you do not complete this step, all docker and docker-compose commands will require elevation to root privileges via sudo.

Note: The docker service starts automatically on Debian based distributions. On RPM based distributions, such as CentOS or RHEL, you need to start it manually using the appropriate systemctl command: sudo systemctl start docker or refer to Configure Docker to start on boot guide

docker-compose Installation

DataMasque requires the standalone docker-compose command, not the docker compose that is integrated into Docker. Follow these instructions to install docker-compose: Follow these instructions to install the docker-compose standalone package.

  1. Download docker-compose using curl:

For Ubuntu:

$ sudo curl -SL https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose

For RedHat:

$ sudo curl -SL https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64 -o /usr/bin/docker-compose

 

  1. Make docker-compose executable with chmod +x:

For Ubuntu:

$ sudo chmod +x /usr/local/bin/docker-compose

For RedHat:

$ sudo chmod +x /usr/bin/docker-compose

 

  1. Test that docker-compose is installed correctly by printing out its version with the docker-compose --version command:
$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c

Recommendations

  • Scaling: If you intend to run masking tasks in parallel or to perform multiple masking runs simultaneously, you should allocate 1 vCPU and at least 1 GiB of RAM per concurrent masking task.
  • Network and I/O: Refer to Network and I/O best practices for server networking recommendations.
  • Host security: It is recommended to deploy DataMasque on a dedicated VM/server with appropriate access control and enable host filesystem encryption.
  • Free space: DataMasque requires at least 6GB of free space inside the Docker root directory before install or upgrade. By default, the Docker root directory is /var/lib/docker.

Installing DataMasque with Docker Compose

Extract the provided DataMasque Docker Compose package on the server, and run the included installation script:

tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh

During installation, a new datamasque user and group will be created and assigned ownership of the DataMasque configuration files under /usr/local/etc/datamasque/. You can manually select the uid and gid for the datamasque user and group by providing --uid and --gid options to install.sh:

sudo ./install.sh --uid 1010 --gid 1010

Once the installation script has run, you can confirm the DataMasque Docker containers are running with the following command:

docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"

If installation was successful, you should see five DataMasque containers listed:

CONTAINER ID        STATUS              NAMES
60114d0c370d        Up 3 minutes        datamasque_admin-frontend_1
da8bacfbbe48        Up 3 minutes        datamasque_admin-server_1
c9555bf98017        Up 3 minutes        datamasque_agent-worker_1
d46296fd5b3b        Up 3 minutes        datamasque_agent-queue_1
1d5516575e1c        Up 3 minutes        datamasque_admin-db_1

You can now access your DataMasque instance from a client web browser at: https://<instance-ip-or-hostname>. Follow the Initial Setup guide to complete the installation of DataMasque.

Note: The first time you visit DataMasque, your browser will display a security warning because the self-signed certificate generated by DataMasque will be untrusted by your browser. You may proceed past this warning for now, and install your own trusted certificate after installation is complete (refer to Installing SSL Credentials). If you are unable to proceed past the warning due to HSTS being enabled for your domain, refer to Installation with HSTS enabled.

Upgrading DataMasque with Docker Compose

It is recommended that you take backups of all rulesets, connections and uploaded files before upgrading your DataMasque instance.

To upgrade your DataMasque instance, extract the new DataMasque Docker Compose package, and run the included installation script with the --upgrade option:

tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh --upgrade

Note: If your user belongs to the docker and datamasque groups, root privilege escalation via sudo is not necessary.

Troubleshooting and maintenance

Installation with HSTS enabled

On browsers with HSTS enforced on the domain used to access DataMasque, the browser will block access to DataMasque if the default self-signed SSL certificate is in use. In this case it is recommended to install your own SSL credentials which are valid and trusted on your network.

To access DataMasque prior to installing your own trusted SSL certificate, you may replace the instance hostname with the server's IP address in the browser's address bar.

For example, if your instance hostname is my-datamasque-server.internal.org.domain, and HSTS is enforced your organisation's internal.org.domain, you would replace https://my-datamasque-server.internal.org.domain with https://<datamasque-server-ip> where <datamasque-server-ip> is the external IP address of your server.

Data protection

It is recommended to take regular backups of your DataMasque EC2 instance. It is also recommended to periodically save copies of your Run Logs, as well as Ruleset and Connection configurations.

Restarting DataMasque

To restart the DataMasque Docker containers, run the following command (if your user does not belong to the docker and datamasque groups, you will need to escalate to root privileges by prefixing the command with sudo):

docker-compose -f /usr/local/etc/datamasque/docker-compose.yml restart

You can verify that all five DataMasque containers have successfully restarted by running the following command:

docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"

Checking status of DataMasque containers

To verify the status of the DataMasque containers, run the following command.

docker ps

Displayed below is an example output for the command.

CONTAINER ID   IMAGE                                         COMMAND                  CREATED        STATUS          PORTS                                                                                    NAMES
72cb9569262a   datamasque/admin-frontend:2-10-1-final-9539   "/entrypoint.sh"         28 hours ago   Up 53 minutes   80/tcp, 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp   datamasque_admin-frontend_1
d53939f9c9d6   datamasque/admin-server:2-10-1-final-9539     "/entrypoint.sh"         28 hours ago   Up 53 minutes                                                                                            datamasque_admin-server_1
583fc8ae5164   datamasque/agent:2-10-1-final-9539            "/entrypoint.sh"         28 hours ago   Up 53 minutes                                                                                            datamasque_agent-worker_1
9b293366ebd5   datamasque/admin-db:2-10-1-final-9539         "docker-entrypoint.s…"   28 hours ago   Up 53 minutes   5432/tcp                                                                                 datamasque_admin-db_1
e384b3101326   datamasque/agent-queue:2-10-1-final-9539      "docker-entrypoint.s…"   28 hours ago   Up 53 minutes   6379/tcp                                                                                 datamasque_agent-queue_1

You should be able to see containers with the following names:

datamasque_admin-frontend_1
datamasque_admin-server_1
datamasque_agent-worker_1
datamasque_admin-db_1
datamasque_agent-queue_1

These containers should also be marked with a container ID, an image name, as well as the status. The status for these containers should be up with a time stamp of how long they have been up for.

DataMasque logs

Important DataMasque logs can be extracted from the Docker containers with the following commands:

  • To copy the DataMasque web application runner logs to a <target-file-path>:
  docker cp datamasque_admin-server_1:/files/logs/uwsgi.log <target-file-path>
  • To copy the DataMasque web application logs to a <target-file-path>:
  docker cp datamasque_admin-server_1:/files/logs/django.log <target-file-path>
  • To copy the DataMasque masking agent logs to a <target-file-path>:
  docker cp datamasque_agent-worker_1:/files/logs/celery.log <target-file-path>
  • To obtain the STDOUT from a container, run docker logs <NAME OF CONTAINER>. For example, to obtain the STDOUT logs from the admin-db container, run the following command:
    docker logs datamasque_admin-db_1

DataMasque start on boot

DataMasque's containers are configured with Docker Compose to always restart, so DataMasque will start automatically after system boot if the Docker daemon is configured to start on boot (which is the default configuration for a standard installation of Docker). For more information on configuring Docker to start on boot, refer to: https://docs.docker.com/engine/install/linux-postinstall/#configure-docker-to-start-on-boot.

Force Downgrade or Reinstall

It is possible to downgrade to a prior version of DataMasque using the --force flag in conjunction with --upgrade, for example:

tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh --upgrade --force

This can also be used to reinstall the same version of DataMasque again.

Warning: Use of the --force flag may cause inconsistencies in your DataMasque storage database (not target databases) if the DataMasque schema has changed between versions. This may require the complete removal of DataMasque's Docker volumes and the loss of all DataMasque related data.

For this reason, it is not recommended to use --force unless you have a backup of all DataMasque data.

Note that the earliest version of DataMasque that supports --force is 2.11.0 so downgrading below 2.11.0 is not possible without a full removal and reinstall (see the next section How to fix installation if DataMasque images are deleted).

How to fix installation if DataMasque images are deleted

If one or more of DataMasque's Docker images are removed from the Docker engine, you can use the docker load command to manually reload the images from the DataMasque Docker Compose package:

tar -xvzf DataMasque-<version>.pkg
cd DataMasque-<version>/
for image_name in ./images/*; do
  docker load -i "$image_name"
done

Once you have reloaded the images, you can recreate the Docker containers and restart DataMasque by running:

docker-compose -f /usr/local/etc/datamasque/docker-compose.yml up -d

Note that reloading the Docker images and recreating Docker containers with docker-compose will not delete DataMasque's application data, as long as DataMasque's Docker volumes are not removed.