TimestampDiffer

This presentations goal it to introduce the features of the timestamp_differ and how to configure it.

The challenge

I want calculate the time delta between two timestamps with different timestamp formats.

from this:

document = {
    "times": {
        "ingest": "06-12-2022T10:00:00",
        "processed": "2022-12-06 10:00:05",
    },
    "more": "event data"
}

to this:

expected = {
    "times": {
        "ingest": "06-12-2022T10:00:00",
        "processed": "2022-12-06 10:00:05",
        "processing_time": "5000.0",
    },
    "more": "event data"
}

Create rule and processor

create the rule:

import sys
sys.path.append("../../../../../")
import tempfile
from pathlib import Path

rule_yaml = """---
filter: 'times.ingest AND times.processed'
timestamp_differ:
  diff: ${times.processed:%Y-%m-%d %H:%M:%S} - ${times.ingest:%d-%m-%YT%H:%M:%S}
  target_field: times.processing_time
  output_format: milliseconds
description: '...'
"""

rule_path = Path(tempfile.gettempdir()) / "timestamp_differ"
rule_path.mkdir(exist_ok=True)
rule_file = rule_path / "timestamp_differ.yml"
rule_file.write_text(rule_yaml)

create the processor config:

processor_config = {
    "my_timestampdiffer":{
        "type": "timestamp_differ",
        "rules": [str(rule_path), "/dev"],
        }
    }

create the processor with the factory:

from unittest import mock
from logprep.factory import Factory

mock_logger = mock.MagicMock()
processor = Factory.create(processor_config)
processor

Process event

from copy import deepcopy
mydocument = deepcopy(document)


print(f"before: {mydocument}")
processor.process(mydocument)
print(f"after: {mydocument}")
print(mydocument == expected)