DataMasque Installation with Docker
This guide describes the process for deploying DataMasque with Docker Compose on a Linux server.
- System requirements
- Installing DataMasque with Docker Compose
- Upgrading DataMasque with Docker Compose
- Troubleshooting and maintenance
System requirements
Minimum requirements
Supported Operating Systems:
- Ubuntu 18.04 or 20.04 LTS, x86 64-bit
- RHEL 7 or 8, x86 64-bit
4 vCPUs
8 GiB RAM
40 GiB available disk space
A static external IP or hostname
Ingress requests allowed on port
443
(and optionally port80
)- These ports are used to serve DataMasque's web interface.
- If HTTP traffic is allowed on port
80
, it will be automatically upgraded to HTTPS on port443
.
For admin access (SSH), ingress requests allowed on port
22
.
Client web browser: Google Chrome 85 or above
Docker 19.03 to 20.10 with Docker Compose 1.17 to 1.29 (installation instructions - for RHEL, follow the CentOS installation instructions)
- Docker should be installed from packages, not a convenience script.
- You should ensure the user account that will be performing the
DataMasque installation and administrative actions is added to
the
docker
group. If you do not complete this step, alldocker
anddocker-compose
commands will require elevation to root privileges viasudo
.
Note: The
docker
service starts automatically on Debian based distributions. On RPM based distributions, such as CentOS or RHEL, you need to start it manually using the appropriatesystemctl
command:sudo systemctl start docker
or refer to Configure Docker to start on boot guide
docker-compose
Installation
DataMasque requires the standalone docker-compose
command, not the docker compose
that is integrated into Docker.
Follow these instructions to install docker-compose
:
Follow these instructions to install the docker-compose
standalone package.
- Download
docker-compose
usingcurl
:
For Ubuntu:
$ sudo curl -SL https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose
For RedHat:
$ sudo curl -SL https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64 -o /usr/bin/docker-compose
- Make
docker-compose
executable withchmod +x
:
For Ubuntu:
$ sudo chmod +x /usr/local/bin/docker-compose
For RedHat:
$ sudo chmod +x /usr/bin/docker-compose
- Test that
docker-compose
is installed correctly by printing out its version with thedocker-compose --version
command:
$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c
Recommendations
- Scaling: If you intend to run masking tasks in parallel or to perform multiple masking runs simultaneously, you should allocate 1 vCPU and at least 1 GiB of RAM per concurrent masking task.
- Network and I/O: Refer to Network and I/O best practices for server networking recommendations.
- Host security: It is recommended to deploy DataMasque on a dedicated VM/server with appropriate access control and enable host filesystem encryption.
- Free space: DataMasque requires at least 6GB of free space inside the Docker root directory before install or
upgrade. By default, the Docker root directory is
/var/lib/docker
.
Installing DataMasque with Docker Compose
Extract the provided DataMasque Docker Compose package on the server, and run the included installation script:
tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh
During installation, a new
datamasque
user and group will be created and assigned ownership of the DataMasque configuration files under/usr/local/etc/datamasque/
. You can manually select the uid and gid for thedatamasque
user and group by providing--uid
and--gid
options toinstall.sh
:sudo ./install.sh --uid 1010 --gid 1010
Once the installation script has run, you can confirm the DataMasque Docker containers are running with the following command:
docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"
If installation was successful, you should see five DataMasque containers listed:
CONTAINER ID STATUS NAMES
60114d0c370d Up 3 minutes datamasque_admin-frontend_1
da8bacfbbe48 Up 3 minutes datamasque_admin-server_1
c9555bf98017 Up 3 minutes datamasque_agent-worker_1
d46296fd5b3b Up 3 minutes datamasque_agent-queue_1
1d5516575e1c Up 3 minutes datamasque_admin-db_1
You can now access your DataMasque instance from a client web browser
at: https://<instance-ip-or-hostname>
. Follow the Initial
Setup guide to complete the installation of
DataMasque.
Note: The first time you visit DataMasque, your browser will display a security warning because the self-signed certificate generated by DataMasque will be untrusted by your browser. You may proceed past this warning for now, and install your own trusted certificate after installation is complete (refer to Installing SSL Credentials). If you are unable to proceed past the warning due to HSTS being enabled for your domain, refer to Installation with HSTS enabled.
Upgrading DataMasque with Docker Compose
It is recommended that you take backups of all rulesets, connections and uploaded files before upgrading your DataMasque instance.
To upgrade your DataMasque instance, extract the new DataMasque Docker
Compose package, and run the included installation script with the
--upgrade
option:
tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh --upgrade
Note: If your user belongs to the
docker
anddatamasque
groups, root privilege escalation viasudo
is not necessary.
Troubleshooting and maintenance
Installation with HSTS enabled
On browsers with HSTS enforced on the domain used to access DataMasque, the browser will block access to DataMasque if the default self-signed SSL certificate is in use. In this case it is recommended to install your own SSL credentials which are valid and trusted on your network.
To access DataMasque prior to installing your own trusted SSL certificate, you may replace the instance hostname with the server's IP address in the browser's address bar.
For example, if your instance hostname is my-datamasque-server.internal.org.domain
,
and HSTS is enforced your organisation's internal.org.domain
,
you would replace https://my-datamasque-server.internal.org.domain
with https://<datamasque-server-ip>
where <datamasque-server-ip>
is the external IP address of your server.
Data protection
It is recommended to take regular backups of your DataMasque EC2 instance. It is also recommended to periodically save copies of your Run Logs, as well as Ruleset and Connection configurations.
Restarting DataMasque
To restart the DataMasque Docker containers, run the following command
(if your user does not belong to the docker
and datamasque
groups,
you will need to escalate to root privileges by prefixing the command
with sudo
):
docker-compose -f /usr/local/etc/datamasque/docker-compose.yml restart
You can verify that all five DataMasque containers have successfully restarted by running the following command:
docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"
Checking status of DataMasque containers
To verify the status of the DataMasque containers, run the following command.
docker ps
Displayed below is an example output for the command.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
72cb9569262a datamasque/admin-frontend:2-10-1-final-9539 "/entrypoint.sh" 28 hours ago Up 53 minutes 80/tcp, 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp datamasque_admin-frontend_1
d53939f9c9d6 datamasque/admin-server:2-10-1-final-9539 "/entrypoint.sh" 28 hours ago Up 53 minutes datamasque_admin-server_1
583fc8ae5164 datamasque/agent:2-10-1-final-9539 "/entrypoint.sh" 28 hours ago Up 53 minutes datamasque_agent-worker_1
9b293366ebd5 datamasque/admin-db:2-10-1-final-9539 "docker-entrypoint.s…" 28 hours ago Up 53 minutes 5432/tcp datamasque_admin-db_1
e384b3101326 datamasque/agent-queue:2-10-1-final-9539 "docker-entrypoint.s…" 28 hours ago Up 53 minutes 6379/tcp datamasque_agent-queue_1
You should be able to see containers with the following names:
datamasque_admin-frontend_1
datamasque_admin-server_1
datamasque_agent-worker_1
datamasque_admin-db_1
datamasque_agent-queue_1
These containers should also be marked with a container ID, an image name, as well as the status. The status for
these containers should be up
with a time stamp of how long they have been up for.
DataMasque logs
Important DataMasque logs can be extracted from the Docker containers with the following commands:
- To copy the DataMasque web application logs to a
<target-file-path>
:
docker cp datamasque_admin-server_1:/files/logs/uwsgi.log <target-file-path>
- To copy the DataMasque masking agent logs to a
<target-file-path>
:
docker cp datamasque_agent-worker_1:/files/logs/celery.log <target-file-path>
- To obtain the STDOUT from a container, run
docker logs <NAME OF CONTAINER>
. For example, to obtain the STDOUT logs from the admin-db container, run the following command:
docker logs datamasque_admin-db_1
DataMasque start on boot
DataMasque's containers are configured with Docker Compose to always restart, so DataMasque will start automatically after system boot if the Docker daemon is configured to start on boot (which is the default configuration for a standard installation of Docker). For more information on configuring Docker to start on boot, refer to: https://docs.docker.com/engine/install/linux-postinstall/#configure-docker-to-start-on-boot.
Force Downgrade or Reinstall
It is possible to downgrade to a prior version of DataMasque using the --force
flag in conjunction with --upgrade
,
for example:
tar -xvzf datamasque-docker-v<version>.pkg
cd datamasque/<version>/
sudo ./install.sh --upgrade --force
This can also be used to reinstall the same version of DataMasque again.
Warning: Use of the
--force
flag may cause inconsistencies in your DataMasque storage database (not target databases) if the DataMasque schema has changed between versions. This may require the complete removal of DataMasque's Docker volumes and the loss of all DataMasque related data.For this reason, it is not recommended to use
--force
unless you have a backup of all DataMasque data.
Note that the earliest version of DataMasque that supports --force
is 2.11.0 so downgrading below 2.11.0 is not
possible without a full removal and reinstall (see the next section How to fix installation if DataMasque images are
deleted).
How to fix installation if DataMasque images are deleted
If one or more of DataMasque's Docker images are removed from the
Docker engine, you can use the docker load
command to manually
reload the images from the DataMasque Docker Compose package:
tar -xvzf DataMasque-<version>.pkg
cd DataMasque-<version>/
for image_name in ./images/*; do
docker load -i "$image_name"
done
Once you have reloaded the images, you can recreate the Docker containers and restart DataMasque by running:
docker-compose -f /usr/local/etc/datamasque/docker-compose.yml up -d
Note that reloading the Docker images and recreating Docker containers
with docker-compose
will not delete DataMasque's application data,
as long as DataMasque's Docker volumes are not removed.