FieldManager

This presentations goal it to introduce the features of the FieldManager and how to configure it.

The challenges

I want to move or rename a field.
I want to copy a field.
I want to merge field values to a list.
I want to merge lists from different fields to one list in a new or existing field

given preprocessed log entry:

[31]:

document = {
    "client": {"ip": ["127.0.0.1", "fe89::", "192.168.5.1"], "nat": {"ip": "223.2.3.2"}},
    "destination": {"ip": "8.8.8.8"},
    "host": {"_hostname": "customer2", "ip": ["192.168.5.1", "180.22.66.3"]},
    "observer": {"ip": "10.10.2.33"},
    "server": {"ip": "10.10.2.33", "nat": {"ip": "180.22.66.1"}},
    "source": {"ip": "10.10.2.33"},
    "preexisting": "I exists already",
    "parent": {"child1": {"child2": {"child3": "I am a child"}}, "child1a": "the other child"},
}

Create rules and processor

create the rules:

create the processor config:

[32]:

processor_config = {
    "the_field_manager": {
        "type": "field_manager",
        "rules": [
            {
                "filter": "host._hostname",
                "field_manager": {
                    "source_fields": ["client.nat.ip", "source.ip"],
                    "target_field": "related.ip",
                    "overwrite_target": True,
                    "delete_source_fields": True,
                    "merge_with_target": True,
                },
            },
            {
                "filter": "parent",
                "field_manager": {
                    "mapping": {
                        "parent.child1": "newparent.child1",
                    }
                },
            },
        ],
    }
}

create the processor with the factory:

[33]:

import logging
import sys

from logprep.factory import Factory

# Configure logging
logging.basicConfig(
    level=logging.DEBUG,
    stream=sys.stdout
)

processor = Factory.create(processor_config)
processor

DEBUG:Processor:FieldManager (the_field_manager) loaded 2 rules

[33]:

field_manager

Process event

[34]:

from copy import deepcopy

mydocument = deepcopy(document)
processor.process(mydocument)

DEBUG:Processor:FieldManager (the_field_manager) processing event {'client': {'ip': ['127.0.0.1', 'fe89::', '192.168.5.1'], 'nat': {'ip': '223.2.3.2'}}, 'destination': {'ip': '8.8.8.8'}, 'host': {'_hostname': 'customer2', 'ip': ['192.168.5.1', '180.22.66.3']}, 'observer': {'ip': '10.10.2.33'}, 'server': {'ip': '10.10.2.33', 'nat': {'ip': '180.22.66.1'}}, 'source': {'ip': '10.10.2.33'}, 'preexisting': 'I exists already', 'parent': {'child1': {'child2': {'child3': 'I am a child'}}, 'child1a': 'the other child'}}

[34]:

ProcessorResult(data=[], errors=[], warnings=[], event={'client': {'ip': ['127.0.0.1', 'fe89::', '192.168.5.1']}, 'destination': {'ip': '8.8.8.8'}, 'host': {'_hostname': 'customer2', 'ip': ['192.168.5.1', '180.22.66.3']}, 'observer': {'ip': '10.10.2.33'}, 'server': {'ip': '10.10.2.33', 'nat': {'ip': '180.22.66.1'}}, 'preexisting': 'I exists already', 'parent': {'child1': {'child2': {'child3': 'I am a child'}}, 'child1a': 'the other child'}, 'related': {'ip': ['223.2.3.2', '10.10.2.33']}, 'newparent': {'child1': {'child2': {'child3': 'I am a child'}}}}, processor_name='the_field_manager')

Check Results

[35]:

document

[35]:

{'client': {'ip': ['127.0.0.1', 'fe89::', '192.168.5.1'],
  'nat': {'ip': '223.2.3.2'}},
 'destination': {'ip': '8.8.8.8'},
 'host': {'_hostname': 'customer2', 'ip': ['192.168.5.1', '180.22.66.3']},
 'observer': {'ip': '10.10.2.33'},
 'server': {'ip': '10.10.2.33', 'nat': {'ip': '180.22.66.1'}},
 'source': {'ip': '10.10.2.33'},
 'preexisting': 'I exists already',
 'parent': {'child1': {'child2': {'child3': 'I am a child'}},
  'child1a': 'the other child'}}

[36]:

mydocument

[36]:

{'client': {'ip': ['127.0.0.1', 'fe89::', '192.168.5.1']},
 'destination': {'ip': '8.8.8.8'},
 'host': {'_hostname': 'customer2', 'ip': ['192.168.5.1', '180.22.66.3']},
 'observer': {'ip': '10.10.2.33'},
 'server': {'ip': '10.10.2.33', 'nat': {'ip': '180.22.66.1'}},
 'preexisting': 'I exists already',
 'parent': {'child1': {'child2': {'child3': 'I am a child'}},
  'child1a': 'the other child'},
 'related': {'ip': ['223.2.3.2', '10.10.2.33']},
 'newparent': {'child1': {'child2': {'child3': 'I am a child'}}}}