Getters

class logprep.util.getter.FileGetter

Get files (and only files) from a filesystem.

Matching string examples:

/yourpath/yourfile.extension
file://yourpath/yourfile.extension

class logprep.util.getter.HttpGetter

Get files from an api or simple web server.

Matching string examples:

Simple http target: http://your.target/file.yml

Simple https target: https://your.target/file.json

Security Best Practice - HttpGetter

If recourses are loaded via HttpGetters it is recommended to

use a credential file to securely manage authentication
use preferably the MTLSCredentials or OAuth2PasswordFlowCredentials (with client-auth)
use always HTTPS connections as HTTPS is not enforced by logprep
consider that the HttpGetter does not support pagination. If the resource is provided by an endpoint with pagination it could lead to a loss of data.

Authentication for HTTP Getters

In order for Logprep to choose the correct authentication method the LOGPREP_CREDENTIALS_FILE environment variable has to be set. This file should provide the credentials that are needed and can either be in yaml or in json format. To use the authentication, the given credentials file has to be filled with the correct values that correspond to the method you want to use.

Example for credentials file

getter:
    "http://target.url":
        # example for token given directly via file
        token_file: <path/to/token/file> # won't be refreshed if expired
    "http://target.url":
        # example for token given directly inline
        token: <token> # won't be refreshed if expired
    "http://target.url":
        # example for OAuth2 Client Credentials Grant
        endpoint: <endpoint>
        client_id: <id>
        client_secret_file: <path/to/secret/file>
    "http://target.url":
        # example for OAuth2 Client Credentials Grant with inline secret
        endpoint: <endpoint>
        client_id: <id>
        client_secret: <secret>
    "http://target.url":
        # example for OAuth2 Resource Owner Password Credentials Grant with
        # authentication for a confidential client
        endpoint: <endpoint>
        username: <username>
        password_file: <path/to/password/file>
        client_id: <client_id> # optional if required
        client_secret_file: <path/to/secret/file> # optional if require
    "http://target.url":
        # example for OAuth2 Resource Owner Password Credentials Grant for a
        # public not confidential client
        endpoint: <endpoint>
        username: <username>
        password_file: <path/to/password/file>
    "http://target.url":
        # example for OAuth2 Resource Owner Password Credentials Grant for a
        # public not confidential client with inline password
        endpoint: <endpoint>
        username: <username>
        password: <password>
    "http://target.url":
        # example for Basic Authentication
        username: <username>
        password_file: <path/to/password/file>
    "http://target.url":
        # example for Basic Authentication with inline password
        username: <username>
        password: <plaintext password> # will be overwritten if 'password_file' is given
    "http://target.url":
        # example for mTLS authentication
        client_key: <path/to/client/key/file>
        cert: <path/to/certificate/file>
    "http://target.url":
        # example for mTLS authentication with ca cert given
        client_key: <path/to/client/key/file>
        cert: <path/to/certificate/file>
        ca_cert: <path/to/ca/cert>
input:
  endpoints:
    /firstendpoint:
      username: <username>
      password_file: <path/to/password/file>
    /second*:
      username: <username>
      password: <password>

Options for the credentials file are:

class BasicAuthCredentials

Basic Authentication Credentials This is used for authenticating with Basic Authentication

BasicAuthCredentials.username: str: The username for the basic authentication.

BasicAuthCredentials.password: str: The password for the basic authentication.

class OAuth2ClientFlowCredentials

OAuth2 Client Credentials Flow Implementation as described in https://datatracker.ietf.org/doc/html/rfc6749#section-1.3.4

OAuth2ClientFlowCredentials.endpoint: str: The token endpoint for the OAuth2 server. This is used to request the token.

OAuth2ClientFlowCredentials.client_id: str: The client id for the token request. This is used to identify the client.

OAuth2ClientFlowCredentials.client_secret: str: The client secret for the token request. This is used to authenticate the client.

class OAuth2PasswordFlowCredentials

OAuth2 Resource Owner Password Credentials Grant as described in https://datatracker.ietf.org/doc/html/rfc6749#section-4.3

Token refresh is implemented as described in https://datatracker.ietf.org/doc/html/rfc6749#section-6

OAuth2PasswordFlowCredentials.endpoint: str: The token endpoint for the OAuth2 server. This is used to request the token.

OAuth2PasswordFlowCredentials.password: str: the password for the token request

OAuth2PasswordFlowCredentials.username: str: the username for the token request

OAuth2PasswordFlowCredentials.client_id: str: The client id for the token request. This is used to identify the client. (Optional)

OAuth2PasswordFlowCredentials.client_secret: str: The client secret for the token request. This is used to authenticate the client. (Optional)

class MTLSCredentials

class for mTLS authentication

MTLSCredentials.client_key: str: path to the client key

MTLSCredentials.cert: str: path to the client certificate

MTLSCredentials.ca_cert: str: path to a certification authority certificate

Authentication Process: