Executing Commands

Lyft Data exposes the exec input so you can reuse shell scripts, operating system tools, and CLIs inside a job. The worker launches the command through the host shell and turns its output into events for downstream actions.

When to use exec

Reuse system utilities or legacy scripts without building a custom connector
Gather diagnostics (uptime, disk usage) alongside streaming telemetry
Fan results into tools that expect newline-delimited JSON or plain text
Prototype quick integrations before you invest in a dedicated input or output

Exec input

Exec inputs run commands on a schedule or once at startup and treat the output as incoming events. Lyft Data runs the command with /bin/sh on Unix platforms or PowerShell/cmd on Windows, so multi-line command blocks and shell features are available.

Key capabilities:

Preserve multi-line command strings with no-strip-linefeeds
Control output framing with json (treat each line as JSON) and ignore-line-breaks (emit the entire run as one event)
Schedule recurring runs via trigger.interval.duration, and bound execution with timeout.value
Provide environment variables via env.file (path) or env.values (multiline KEY=value pairs)

input:
  exec:
    command: |
      ./bin/collect-metrics \
        --tenant retail-eu
    trigger:
      interval:
        duration: 2m
    ignore-line-breaks: true
    result:
      status-field: exit_status
      stdout-field: metrics_stdout
      stderr-field: metrics_stderr
    env:
      values: |
        API_TOKEN={{secrets.api_token}}
        TENANT=retail-eu
    timeout:
      value: 30s
    retry:
      timeout: 30s
      retries: 5

When json: true, each line is parsed as JSON instead of being wrapped in the _raw field. Enable ignore-line-breaks to combine multi-line output (for example, certificate dumps) into a single event for downstream parsing.

Operational considerations

Commands run inside the worker runtime, so ensure the binary or script is present on every worker host
Treat exec jobs like any other external dependency: capture exit codes and stderr into fields and alert on changes
Prefer idempotent commands and explicit timeouts; use retry to bound repeated failures
Store secrets in context variables (for example {{secrets.api_token}}) and avoid embedding credentials directly in the command string