In-Flight Server Configuration and Logs

This document explains how to configure the DataMasque in-flight server, view logs, and adjust settings for different deployment scenarios. You'll learn how to modify environment variables, handle common configuration scenarios, and understand the available configuration options.

• In-Flight Server Logs
• Changing In-Flight Settings
  • Updating Environment Variables for Docker/Podman
    • Restarting Docker
    • Restarting Podman
  • Updating Settings for EKS
• Common Scenarios
  • Handling More Requests
  • Disabling In-Flight
  • Rate Limiting
• Environment Variables Reference
  • IN_FLIGHT_MASKING_RATE_LIMIT_ENABLED
  • IN_FLIGHT_MASKING_RATE_LIMIT_MINUTE
  • IN_FLIGHT_MASKING_RATE_LIMIT_SECOND
  • IN_FLIGHT_WORKERS

In-Flight Server Logs

The logs for in-flight server requests can be downloaded from the DataMasque dashboard. Refer to the guide on viewing container logs.

There are two log files available:

• masque_in_flight_server.log: Downloadable by selecting In-flight application logs from the Application Logs menu. This contains general messages for the in-flight server, including any unhandled errors or exceptions.
• masque_in_flight_server_requests.log: Downloadable by selecting In-flight request logs from the Application Logs menu. This file contains the source IP, user, HTTP method, path and response code for all in-flight requests, for auditing purposes. Refer to common status codes for an explanation of what the status codes mean.

Example request log lines are shown below:

2024-12-10 22:49:30 UTC - 192.168.86.118 (Unknown User) - "POST /ifm/ruleset_plans/" 401
2024-12-10 22:49:35 UTC - 192.168.86.119 (alice) - "GET /ifm/ruleset_plans/" 200

Changing In-Flight Settings

The DataMasque in-flight server is configurable either by using environment variables (for Docker/Podman), or with Helm chart settings (for EKS). The method for updating these variables depends on your deployment type:

• Docker/Podman: Settings are stored in a .env file.
• EKS: Settings are defined in the values.yaml file for the Helm Chart.
• AWS/Azure Marketplace: Follow Docker instructions (these are Docker-based deployments).

The following sections explain how to update settings for each deployment type.

Updating Environment Variables for Docker/Podman

First, locate the .env file. It is inside the DataMasque installation directory, which is /usr/local/etc/datamasque/ by default.

During DataMasque installation, the installation directory can be set to a different path. If your DataMasque install directory has been customised, you will find the .env file in this path instead. The directory will also contain a docker-compose.yml file.

Edit the .env file and change the necessary variable.

For example, to change the worker count from 4 to 8, change the line:

IN_FLIGHT_WORKERS=4

to

IN_FLIGHT_WORKERS=6

then save the file.

Next, restart your container platform.

Important: Restarting Docker or Podman will cause any masking jobs in progress to end. Please make sure no jobs are running before restarting.

Restarting Docker

Restart Docker with systemctl:

$ sudo systemctl restart docker

Restarting Podman

Restart the DataMasque Podman service with systemctl:

$   sudo systemctl restart datamasque_podman.service

Updating Settings for EKS

These settings are available in the Helm chart's values.yaml file, but use different names than their environment variable counterparts:

Environment Variable	Helm Chart Setting
IN_FLIGHT_MASKING_RATE_LIMIT_ENABLED	ifmRateLimitEnabled
IN_FLIGHT_MASKING_RATE_LIMIT_MINUTE	ifmRateLimitMinute
IN_FLIGHT_MASKING_RATE_LIMIT_SECOND	ifmRateLimitSecond
IN_FLIGHT_WORKERS	ifmWorkers

Note: This document uses environment variable names in section headers, with the corresponding Helm chart setting name listed within each section.

These settings are defined under an ifm section in the values.yaml file. For example:

# The rest of the values.yaml has been omitted for brevity
ifm:
  ifmWorkers: 4
  ifmRateLimitEnabled: true
  ifmRateLimitMinute: "6000/minute"
  ifmRateLimitSecond: "150/second"

After editing them, use helm upgrade to update the cluster with the new variables.

$ helm upgrade datamasque ./datamasque-helmchart -f values.yaml

Note that while helm upgrade normally does not cause downtime, it should be executed while other masking jobs are not running, as any configuration errors could cause them to stop.

Common Scenarios

These are some common reasons for editing settings.

Handling More Requests

Edit IN_FLIGHT_WORKERS and specify a larger value (defaults to 4) to dedicate more worker processes to service in-flight requests.

As a general guideline, configure no more than 2 in-flight workers per CPU core available to DataMasque. Remember that both in-flight and static masking tasks use workers from this total pool. For example, if you have 4 CPU cores, you should have no more than 8 workers total across all tasks.

For detailed guidance on selecting worker count and balancing workers between in-flight and static masking tasks, see the Performance Guide.

Disabling In-Flight

To disable the in-flight server, set IN_FLIGHT_WORKERS to 0. Note that the in-flight-server container will still be active, but will not run any processes.

Rate Limiting

Server-side rate limiting of worker processes is controlled by the IN_FLIGHT_MASKING_RATE_LIMIT_MINUTE and IN_FLIGHT_MASKING_RATE_LIMIT_SECOND environment variables. Using both allows you to limit requests per second, (to prevent traffic spikes), and limit per minute (to limit sustained request traffic). Exceeding either rate limit means response will be returned with a 429 status code (i.e. only one limit needs to be exceeded to be rate-limited).

Rate limits are applied per worker process.

For example, with 4 workers and a limit of 150 requests/second per worker:

• Individual worker capacity: 150 requests/second
• Total system capacity: 150/second * 4 workers = 600 requests/second

In general this behavior is OK, as having more workers means more requests can be serviced.

The format of limits is in <num_requests>/<unit>. For example, 6000/minute or 150/second.

Environment Variables Reference

IN_FLIGHT_MASKING_RATE_LIMIT_ENABLED

Default: true Helm Chart Name: ifmRateLimitEnabled

Set to false to disable rate limiting. It is not recommended to disable rate limiting.

IN_FLIGHT_MASKING_RATE_LIMIT_MINUTE

Default: 6000/minute Helm Chart Name: ifmRateLimitMinute

Controls the maximum number of requests allowed per minute. Any requests beyond this limit will receive a response with a 429 status code.

IN_FLIGHT_MASKING_RATE_LIMIT_SECOND

Default: 150/second Helm Chart Name: ifmRateLimitSecond

Controls the maximum number of requests allowed per second. Any requests beyond this limit will receive a response with a 429 status code.

IN_FLIGHT_WORKERS

Default: 4 Helm Chart Name: ifmWorkers

Controls the number of in-flight worker processes to start. More workers allows more requests to be processed in parallel, at the expense of higher memory usage.