Security Best Practices

Here you find a list of all security best practices that should be considered when running logprep in a production environment.

To compare your production environment against these best practices we provide a Best Practice Check List for your use.

HttpGetter

Security Best Practice - HttpGetter

If recourses are loaded via HttpGetters it is recommended to

• use a credential file to securely manage authentication

• use preferably the MTLSCredentials or OAuth2PasswordFlowCredentials (with client-auth)

• use always HTTPS connections as HTTPS is not enforced by logprep

• consider that the HttpGetter does not support pagination. If the resource is provided by an endpoint with pagination it could lead to a loss of data.

Reference to original description

Configuration - Combining multiple configuration files

Security Best Practice - Configuration - Combining multiple configuration files

Consider when using multiple configuration files logprep will reject all configuration files if one can not be retrieved or is not valid. If using multiple files ensure that all can be loaded safely and that all endpoints (if using http resources) are accessible.

Reference to original description

Configuration - Authenticity and Integrity

Security Best Practice - Configuration - Authenticity and Integrity

Ensure that all configuration files are retrieved from trusted sources and have not been tampered with. Use tls to encrypt the transmission of configuration files and use authentication described in Authentication for HTTP Getters to ensure confidentiality and integrity.

Reference to original description

Configuration - Environment Variables

Security Best Practice - Configuration - Environment Variables

As it is possible to replace all configuration options with environment variables it is recommended to use these especially for sensitive information like usernames, password, secrets or hash salts. Examples where this could be useful would be the key for the hmac calculation (see input > preprocessing) or the user/secret for the opensearch connectors.

Reference to original description

Configuration - Refresh Interval

Security Best Practice - Configuration - Refresh Interval

The refresh interval for the configuration shouldn't be set too high in production environments. It is suggested to not set a value higher than 300 (5 min). That way configuration updates are propagated fairly quickly instead of once a day.

It should also be noted that a new configuration file will be read as long as it is a valid config. There is no further check to ensure credibility.

In case a new configuration could not be retrieved successfully and the config_refresh_interval is already reduced automatically to 5 seconds it should be noted that this could lead to a blocking behavior or a significant reduction in performance as logprep is often retrying to reload the configuration. Because of that ensure that the configuration endpoint is always available.

Reference to original description

Configuration - Restart Counter

Security Best Practice - Configuration - Restart Counter

The restart counter should be set to a value greater than 0 to ensure that logprep exits gracefully in case of repeated failures. This ensures that resources are released properly and any necessary cleanup is performed. Additionally the process will exit with an exit code unequal 0 to indicate that an error occurred. This is especially useful if you use an external orchestrator like k8s or systemd to manage the logprep process to get notified about failures via their respective monitoring and alerting systems.

Reference to original description

Configuration - Log-Level

Security Best Practice - Configuration - Log-Level

The log level of the root logger should be set to INFO or higher in production environments to avoid exposing sensitive information in the logs.

Reference to original description

Configuration - Metrics Configuration

Security Best Practice - Configuration - Metrics Configuration

Additionally to the below it is recommended to configure ssl on the metrics server endpoint

metrics:
  enabled: true
  port: 9000
  uvicorn_config:
    access_log: true
    server_header: false
    date_header: false
    workers: 1

Reference to original description

Configuration - Error Backlog Size

Security Best Practice - Configuration - Error Backlog Size

Depending on your environment ensure that this value adheres to your overall system resource limits. This can lead to OOM (Out Of Memory) errors if the backlog grows too large in failure situations. You have to reserve memory for this backlog to avoid DOS (Denial of Service) attacks by sending failing logs.

Reference to original description

Input Connectors

Security Best Practice - Input Connectors

It is advised to only use the ConfluentKafkaInput, HttpConnector or FileInput as input connectors in production environments. The connectors DummyInput, JsonInput and JsonlInput are mainly designed for debugging purposes.

Furthermore, it is suggested to enable the HMAC preprocessor to ensure no tempering of processed events.

hmac:
   target: <RAW_MSG>
   key: <SECRET>
   output_field: HMAC

Reference to original description

Kafka Input Consumer Authentication and Encryption

Security Best Practice - Kafka Input Consumer Authentication and Encryption

Kafka authentication is a critical aspect of securing your data pipeline. Ensure that you have the following configurations in place:

• Use SSL/mTLS encryption for data in transit.

• Configure SASL or mTLS authentication for your Kafka clients.

• Regularly rotate your Kafka credentials and secrets.

Reference to original description

Http Input Connector - Authentication

Security Best Practice - Http Input Connector - Authentication

When using basic auth with the http input connector the following points should be taken into account:

• basic auth must only be used with strong passwords

• basic auth must only be used with TLS encryption

• avoid to reveal your plaintext secrets in public repositories

Reference to original description

Uvicorn Webserver Configuration

Security Best Practice - Uvicorn Webserver Configuration

Additionally to the below it is recommended to configure ssl on the metrics server endpoint <https://www.uvicorn.org/settings/#https>`_

uvicorn_config:
    access_log: true
    server_header: false
    date_header: false
    workers: 2

Reference to original description

Input Connector - HttpConnector

Security Best Practice - Input Connector - HttpConnector

It is suggested to enable the collection of meta data (collect_meta: True) to ensure transparency of the incoming events.

Reference to original description

Output Connectors

Security Best Practice - Output Connectors

Similar to the input connectors there is a list of available output connectors of which some are only meant for debugging, namely: ConsoleOutput and JsonlOutput. It is advised to not use these in production environments.

When configuring multiple outputs it is also recommend to only use one default output and to define other outputs only for storing custom extra data. Otherwise it cannot be guaranteed that all events are safely stored.

Reference to original description

Kafka Output Producer Authentication and Encryption

Security Best Practice - Kafka Output Producer Authentication and Encryption

Kafka authentication is a critical aspect of securing your data pipeline. Ensure that you have the following configurations in place:

• Use SSL/mTLS encryption for data in transit.

• Configure SASL or mTLS authentication for your Kafka clients.

• Regularly rotate your Kafka credentials and secrets.

Reference to original description

Kafka Output Producer Authentication and Encryption

Security Best Practice - Kafka Output Producer Authentication and Encryption

Kafka authentication is a critical aspect of securing your data pipeline. Ensure that you have the following configurations in place:

• Use SSL/mTLS encryption for data in transit.

• Configure SASL or mTLS authentication for your Kafka clients.

• Regularly rotate your Kafka credentials and secrets.

Reference to original description

Output Connectors - OpensearchOutput

Security Best Practice - Output Connectors - OpensearchOutput

It is suggested to enable a secure message transfer by setting user, secret and a valid ca_cert.

Reference to original description

Output Connectors - S3Output

Security Best Practice - Output Connectors - S3Output

It is suggested to activate SSL for a secure connection. In order to do that set use_ssl and the corresponding ca_cert.

Reference to original description

Http Output Connector - Usage

Security Best Practice - Http Output Connector - Usage

This Connector is currently only used in the log generator and does not have a stable interface. Do not use this in production.

Reference to original description

Http Output Connector - SSL

Security Best Practice - Http Output Connector - SSL

This connector does not verify the SSL Context, which could lead to exposing sensitive data.

Reference to original description

Processor - Amides Model

Security Best Practice - Processor - Amides Model

Ensure that you only use models from trusted sources, as it can be used to inject python code into the runtime.

Reference to original description

Processor - Domain Resolver Timeout

Security Best Practice - Processor - Domain Resolver Timeout

Ensure to set this to a reasonable value to avoid DOS attacks by malicious domains in your logs. The default is set to 0.5 seconds.

Reference to original description

Processor - Domain Resolver Max Cached Domains

Security Best Practice - Processor - Domain Resolver Max Cached Domains

Ensure to set this to a reasonable value to avoid excessive memory usage and OOM situations by the domain resolver cache.

Reference to original description

Processor - Generic Adder Add From File Memory Consumption

Security Best Practice - Processor - Generic Adder Add From File Memory Consumption

Be aware that all values of the remote file were loaded into memory. Consider to avoid dynamic increasing lists without setting limits for Memory consumption. Additionally avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - Generic Adder Authenticity and Integrity

Security Best Practice - Processor - Generic Adder Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - Generic Resolver Max Cached Entries

Security Best Practice - Processor - Generic Resolver Max Cached Entries

Ensure to set this to a reasonable value to avoid excessive memory usage and OOM situations caused by the generic resolver cache.

Reference to original description

Processor - Generic Resolver Resolve From File Memory Consumption

Security Best Practice - Processor - Generic Resolver Resolve From File Memory Consumption

Be aware that all values of the remote file were loaded into memory. Consider to avoid dynamic increasing lists without setting limits for Memory consumption. Additionally avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - Generic Resolver Authenticity and Integrity

Security Best Practice - Processor - Generic Resolver Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - GeoIP Enricher Database Memory Consumption

Security Best Practice - Processor - GeoIP Enricher Database Memory Consumption

Be aware that all values of the remote file were loaded into memory. Avoid loading a large database via http to avoid exceeding http body limits.

Reference to original description

Processor - GeoIP Enricher Authenticity and Integrity

Security Best Practice - Processor - GeoIP Enricher Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded database.

Reference to original description

Processor - Grokker Custom Patterns Directory Memory Consumption

Security Best Practice - Processor - Grokker Custom Patterns Directory Memory Consumption

Be aware that all values of the remote zip were loaded into memory. Reserve memory for this and avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - Grokker Authenticity and Integrity

Security Best Practice - Processor - Grokker Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - Grokker DOS (Denial of Service) via Backreferences

Security Best Practice - Processor - Grokker DOS (Denial of Service) via Backreferences

Avoid using backreferences in grok patterns, as they can lead to excessive memory consumption and potential denial of service attacks.

Reference to original description

Processor - Labeler Schema File Memory Consumption

Security Best Practice - Processor - Labeler Schema File Memory Consumption

Be aware that all values of the remote file were loaded into memory. Consider to avoid dynamic increasing lists without setting limits for Memory consumption. Additionally avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - Labeler Schema File Authenticity and Integrity

Security Best Practice - Processor - Labeler Schema File Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - List Comparison list file paths Memory Consumption

Security Best Practice - Processor - List Comparison list file paths Memory Consumption

Be aware that all values of the remote files were loaded into memory. Consider to avoid dynamic increasing lists without setting limits for Memory consumption. Additionally avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - List Comparison list file paths Authenticity and Integrity

Security Best Practice - Processor - List Comparison list file paths Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - PreDetector alert_ip_list_path Memory Consumption

Security Best Practice - Processor - PreDetector alert_ip_list_path Memory Consumption

Be aware that all values of the remote file were loaded into memory. Consider to avoid dynamic increasing lists without setting limits for Memory consumption. Additionally avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - PreDetector alert_ip_list_path Authenticity and Integrity

Security Best Practice - Processor - PreDetector alert_ip_list_path Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - Pseudonymizer pubkey analyst Authenticity and Integrity

Security Best Practice - Processor - Pseudonymizer pubkey analyst Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - Pseudonymizer pubkey depseudo Authenticity and Integrity

Security Best Practice - Processor - Pseudonymizer pubkey depseudo Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - Pseudonymizer regex mapping Memory Consumption

Security Best Practice - Processor - Pseudonymizer regex mapping Memory Consumption

Be aware that all values of the remote file were loaded into memory. Consider to avoid dynamic increasing lists without setting limits for Memory consumption. Additionally avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - Pseudonymizer regex mapping Authenticity and Integrity

Security Best Practice - Processor - Pseudonymizer regex mapping Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description

Processor - Pseudonymizer max_cached_pseudonyms

Security Best Practice - Processor - Pseudonymizer max_cached_pseudonyms

Ensure to set this to a reasonable value to avoid excessive memory usage and OOM situations by the domain resolver cache.

Reference to original description

Processor - Pseudonymizer max_cached_pseudonymized_urls

Security Best Practice - Processor - Pseudonymizer max_cached_pseudonymized_urls

Ensure to set this to a reasonable value to avoid excessive memory usage and OOM situations by the domain resolver cache.

Reference to original description

Processor - Pseudonymizer

Security Best Practice - Processor - Pseudonymizer

The pseudonymizer works with two public keys for different roles. It is suggested to ensure that two different keys are being used such that the separation of the roles can be maintained. It is suggested to use the GCM mode for encryption as it decouples the key length of the depseudo and analyst keys. This leads to additional 152 bytes of overhead for the encryption compared to the CTR mode encrypter.

Reference to original description

Processor - Requester

Security Best Practice - Processor - Requester

As the requester can execute arbitrary http requests it is advised to execute requests only against known and trusted endpoints and that the communication is protected with a valid SSL-Certificate. Do so by setting a certificate path with the option cert. To ensure that the communication is trusted it is also recommended to set either an Authorization-Header or a corresponding authentication with a username and password, via auth.

Reference to original description

Processor - TemplateReplacer template Memory Consumption

Security Best Practice - Processor - TemplateReplacer template Memory Consumption

Be aware that all values of the remote file were loaded into memory. Consider to avoid dynamic increasing lists without setting limits for Memory consumption. Additionally avoid loading large files all at once to avoid exceeding http body limits.

Reference to original description

Processor - TemplateReplacer template Authenticity and Integrity

Security Best Practice - Processor - TemplateReplacer template Authenticity and Integrity

Consider to use TLS protocol with authentication via mTLS or Oauth to ensure authenticity and integrity of the loaded values.

Reference to original description