Enriching Data

After you have raw events in JSON form, the next step is enrichment: stamping metadata, deriving new values, and joining with reference tables. This page walks through the built-in actions that cover those cases.

Tag events with deployment context

Use the add action to attach static metadata, pulling values from job context or the runtime helpers ({{job}}, {{worker}}). add defaults to the unless-exists merge strategy, so it will not overwrite fields that are already present.

- add:
    output-fields:
      site: '{{job}}'
      worker_id: '{{worker}}'

Set overwrite: true if you need to replace an existing value.

Generate runtime fields

You can enrich events without leaving the pipeline:

- time:
    output-field: '@timestamp'
- script:
    let:
      - seq: count()
      - event_uuid: uuid()
      - normalized_status: md5(status)

The Lua environment that backs the script action includes helpers such as count() counters, md5() hashing, uuid() generation, and the cond() ternary helper. It also exposes encryption and decryption helpers (encrypt(), decrypt()) when you need reversible protection.

Conditional fields

Choose between literal values with script let and cond, or guard an entire block with condition.

- script:
    let:
      - quality: cond(a > 1, 'good', 'bad')

Because scripts run in place, they can both inspect and update the same event.

Table lookups with enrich

The enrich action loads lookup tables from CSV or Sqlite and matches them against event fields using typed comparisons (str, num, ip, cidr, num-range, num-list, str-list). When the underlying file changes, the runtime notices the new modification timestamp and reloads it automatically.

Include lookup assets under the job’s files: so workers download them alongside the job definition.

name: crm-enrich
files:
  - lookups/names.csv
input:
  echo:
    json: true
    event: '{ "id": 12 }'
actions:
  - enrich:
      lookup: lookups/names.csv
      match:
        - type: num
          event-field: id
          lookup-field: user_id
      add:
        event-field: full_name
        lookup-field: name
      event-fields:
        nickname: ''
        department: 'Unknown'

event-fields provides a shorthand for adding several lookup columns at once. Each entry uses the lookup column name as the event field and the YAML value as the fallback when no match is found.

Example CSV

user_id,name,nickname,department
12,Alice,ac,Finance
99,Bob,b2,Sales

If you need richer joins, set lookup to a Sqlite database, or chain the enrich action with a script to post-process the lookup results.

Sqlite lookups

When a flat file is not enough, point lookup at a Sqlite database. Workers ship the database alongside the job files and the enrich action queries the specified table.

files:
  - lookups/reference.db
input:
  echo:
    json: true
    event: '{ "email": "[email protected]" }'
actions:
  - enrich:
      lookup:
        sqlite:
          path: lookups/reference.db
          table: users
      match:
        - type: str
          event-field: email
          lookup-field: email
      event-fields:
        role: 'guest'

In this pattern, event-fields still provides defaults for missing rows, and the runtime reloads the database file when it changes.