Variable Expansion

LyftData jobs ship with two families of placeholders. Double curly braces ({{ }}) resolve before the job ever runs, while dollar curly braces (${ }}) resolve at runtime for each event. Using the right style in the right place keeps pipelines portable across environments.

Context placeholders ({{ }})

Context values render when you stage or deploy a job. Effective precedence (highest to lowest) is:

1. Job overrides from the control plane (/api/contexts/update-job-map/{job}).
2. Worker-level overrides (worker context).
3. Global defaults.
4. Job definition defaults in the job’s own context: block.

Reference nested maps with dot notation ({{credentials.api_key}}). Missing keys cause staging to fail, so provide defaults in context files rather than relying on silent fallbacks. Use {{ }} for anything that should stay constant throughout the run: API keys, dataset names, feature toggles, or environment-specific endpoints.

Runtime expansions (${ }})

Runtime expressions evaluate inside the worker after an event arrives. Many expansions support an optional || fallback suffix (for example ${field||default} or ${dyn|NAME||default}). Secret references (${secret|scope/name}) do not support defaults.

Syntax	Source	Example
${field}	Current event	`${client.ip
`${msg	path}`	Message payload
`${dyn	NAME}`	Variables
`${secret	scope/name}`	Secrets
`${cred	id
`${stat	_BATCH_NUMBER}`	Output metadata
`${time	…}`	Scheduler clocks

Event data

${field} reads values from the current event using dot-paths (${client.ip}) or slash-paths (${client/ip}). For strict JSON Pointer selectors, start with / (for example ${/client/ip}). Supply fallbacks for optional keys to keep streaming jobs healthy:

- add:
    output-fields:
      hostname: "${server||unknown-host}"
      disk_usage_pc: "${metrics.disk_usage_pc||0}"

Message payloads

Jobs that run on a message trigger can read the triggering message’s public payload with ${msg|...} (see Advanced scheduling). The available fields depend on the message type.

For user-generated messages emitted by the message output, the original event is available under job_event:

- add:
    output-fields:
      upstream_order_id: "${msg|job_event.order_id||unknown}"
      upstream_tag: "${msg|tag||}"

Dynamic variables

${dyn|NAME} resolves values from the Variables service. If you reference a variable without a default, LyftData expects it to exist. Staging and transient runs reject missing required variables early; long-lived deployed jobs can stall until the variable is provided. Use || defaults for optional settings.

Batch metadata

Outputs that batch events expose two counters:

• ${stat|_BATCH_NUMBER} - zero-padded batch index.
• ${stat|_SCHEMA_NUMBER} - schema revision when schema tracking is enabled.

Use these to build stable object keys:

output:
  s3:
    bucket-name: logs-{{environment}}
    object-name:
      name: "runs/${time|now_time_fmt %Y/%m/%d}/run-${stat|_BATCH_NUMBER}.json"

Time macros and offsets

${time|...} expressions read from the scheduler window. Common variants include:

• ${time|now_time_secs} - current wall clock in epoch seconds.
• ${time|start_time_iso} / ${time|end_time_iso} - window bounds in ISO-8601.
• ${time|now_time_fmt %Y-%m-%d} - custom strftime formatting.

Append + 5m, - 1h, or any humantime duration to apply offsets. Invalid offsets fail the run, so stick to supported units (ms, s, m, h, d).

Putting it together

A typical production job mixes both placeholder families:

context:
  dataset: security
  bucket: logs-{{environment}}

input:
  http-poll:
    url: "https://api.example.com/{{dataset}}/events?since=${time|start_time_iso - 5m}"

actions:
  - add:
      output-fields:
        request_id: "${dyn|request_id}"
        source_host: "${msg|message_content.hostname||unknown}"

output:
  s3:
    bucket-name: "{{bucket}}"
    object-name:
      name: "${stat|_BATCH_NUMBER}/${time|now_time_fmt %Y/%m/%d}/events.json"

Tips for reliable templates

• Use || fallbacks on ${ } expressions fed by optional fields or third-party payloads.
• Keep context files in source control so reviewers can verify how {{ }} placeholders resolve across environments.
• For long-lived deployed jobs, ${dyn|...} without a default can stall processing when a required variable is absent; monitor worker logs for “waiting for variable” warnings.
• Batch counters reset when an output restarts. Treat them as per-run identifiers rather than global sequences.

For additional operators (math helpers, string casing, metadata keys), search this docs site for ${ examples and cross-check with the in-product DSL documentation (Docs → Job DSL).