Variable Expansion

Lyft Data jobs ship with two families of placeholders. Double curly braces ({{ }}) resolve before the job ever runs, while dollar curly braces (${ }}) resolve at runtime for each event. Using the right style in the right place keeps pipelines portable across environments.

Context placeholders ({{ }})

Context values render when you stage or deploy a job. The server merges three layers, with later entries taking precedence:

1. Server defaults supplied by the platform team.
2. Worker-level overrides (the worker context).
3. Job-level context defined in the job itself.

Reference nested maps with dot notation ({{credentials.api_key}}). Missing keys cause staging to fail, so provide defaults in context files rather than relying on silent fallbacks. Use {{ }} for anything that should stay constant throughout the run: API keys, dataset names, feature toggles, or environment-specific endpoints.

Runtime expansions (${ }})

Runtime expressions evaluate inside the worker after an event arrives. Every expression supports an optional || fallback suffix. The table below lists the most common prefixes.

Syntax	Source	Example
${field}	Current event	`${client.ip
`${msg	path}`	Message payload
`${dyn	NAME}`	Dynamic variables service
`${stat	_BATCH_NUMBER}`	Output metadata
`${time	…}`	Scheduler clocks

Before the ${ } pass runs, the runtime applies an @{key} helper that injects metadata such as the job name or worker ID. Combine both styles: @{job}-batch-${stat|_BATCH_NUMBER}.json.

Event data

${field} and ${field.subfield} map to JSON Pointer lookups. Supply fallbacks for optional keys to keep streaming jobs healthy:

- add:
    output-fields:
      hostname: "${server||unknown-host}"
      disk_usage_pc: "${metrics.disk_usage_pc||0}"

Message payloads

Jobs triggered by internal messages can read the payload with ${msg|...}. A job emitting message outputs might send:

- message:
    tag: orders
    message-content: |
      {
        "order_id": "${event.id}",
        "customer": "${event.customer.email}"
      }

A downstream job listening on internal-messages can then pull ${msg|message_content.order_id}.

Dynamic variables

${dyn|NAME} resolves values from the variables service. The worker pauses the run until the value becomes available, emitting warnings if the wait exceeds the backoff window. Always provide fallbacks for non-critical settings and monitor worker logs for stalled lookups.

Batch metadata and @{} helpers

Outputs that batch events expose two counters:

• ${stat|_BATCH_NUMBER} - zero-padded batch index.
• ${stat|_SCHEMA_NUMBER} - schema revision when schema tracking is enabled.

Mix these with metadata helpers to build object keys:

outputs:
  - s3:
      bucket: logs-{{environment}}
      key: "@{job}/run-${stat|_BATCH_NUMBER}/${time|now_time_fmt %Y/%m/%d}.json"

Time macros and offsets

${time|...} expressions read from the scheduler window. Common variants include:

• ${time|now_time_secs} - current wall clock in epoch seconds.
• ${time|start_time_iso} / ${time|end_time_iso} - window bounds in ISO-8601.
• ${time|now_time_fmt %Y-%m-%d} - custom strftime formatting.

Append + 5m, - 1h, or any humantime duration to apply offsets. Invalid offsets fail the run, so stick to supported units (ms, s, m, h, d).

Putting it together

A typical production job mixes both placeholder families:

context:
  dataset: security
  bucket: logs-{{environment}}

input:
  http-poll:
    url: "https://api.example.com/{{dataset}}/events?since=${time|start_time_iso - 5m}"

actions:
  - add:
      output-fields:
        request_id: "${dyn|request_id}"
        source_host: "${msg|message_content.hostname||unknown}"

outputs:
  - s3:
      bucket: "{{bucket}}"
      key: "${stat|_BATCH_NUMBER}/${time|now_time_fmt %Y/%m/%d}/events.json"

Tips for reliable templates

• Use || fallbacks on ${ } expressions fed by optional fields or third-party payloads.
• Keep context files in source control so reviewers can verify how {{ }} placeholders resolve across environments.
• Remember that ${dyn|...} waits indefinitely for required variables; monitor worker logs for “waiting for variable” warnings.
• Batch counters reset when an output restarts. Treat them as per-run identifiers rather than global sequences.

For additional operators (math helpers, string casing, metadata keys), check the field expansion reference.