Dealing With Time

Time appears in almost every pipeline. Jobs need to stamp run metadata, convert vendor timestamps, or decide whether an event falls into a window. This guide collects the canonical tools for dealing with time in Lyft Data so you can wire them together with confidence.

Stamp run context with field expansions

Use ${time|...} macros when you need a timestamp derived from the job’s current run window. The macros resolve before actions execute and do not require a separate step.

add:
  output-fields:
    exported_at: "${time|now_time_iso}"
    start_bucket: "${time|start_time_fmt %Y-%m-%dT%H:00:00Z}"

Key points:

• now_time_*, start_time_*, and end_time_* expose the window’s now, start, and end timestamps.
• Append fmt to supply a strftime pattern: start_time_fmt %Y/%m/%d.
• Add offsets with + or - and a humantime duration: now_time_fmt %Y-%m-%d - 5m.
• The syntax is literal ${...}—templating helpers such as {{ date(...) }} are not parsed by the runtime.

You can combine macros inside object keys—for example to produce partitioned paths:

output:
  s3:
    bucket: lyftdata-exports
    path: analytics/${time|now_time_fmt %Y/%m/%d}/events.jsonl

Convert timestamps with the time action

When events carry timestamps that need normalising, reach for the time action. It parses strings or numbers from one or more fields and writes the result in the format you choose.

- time:
    input-field: vendor_ts
    input-formats:
      - default_iso
      - '%s'
    output-format: default_iso
    output-field: '@timestamp'

Highlights:

• input-field selects the event field; fall back to the run now when omitted.
• input-format accepts chrono/strftime specifiers or shortcuts such as default_iso, epoch_secs, and epoch_msecs. Use input-formats to provide a list.
• input-timezone lets you parse ambiguous local times explicitly (America/Los_Angeles, for example).
• output-field (or output-fields) writes the converted value. The default format is ISO-8601 UTC with millisecond precision.

The action also emits metadata about the run window so you can reuse it in later steps without re-parsing.

Classify by time ranges

time can tag events that fall within specific intervals using when. Supply ranges using the day:start-end syntax and choose where to store the result.

- time:
    input-field: '@timestamp'
    when:
      - 'mon-fri:09:00-17:00'
      - 'sat:10:00-13:00'
    output-fields:
      business_hours: true
      range_start_secs: '${time|start_time_secs}'

For dynamic schedules, provide a comma-delimited string via when: or populate the input from an earlier enrichment. To expose the derived range bounds, set time-range-start-field and time-range-length-field—both must be provided together.

Handle timezones explicitly

By default, the time action assumes:

• Input strings without offsets represent local time.
• Output values are UTC.

Override this behaviour with input-timezone and output-timezone, using IANA names (Europe/Paris, America/New_York). Pair these options with utc-input: true when upstream timestamps already include an offset but you want to treat them as UTC.

For jobs that run in containers, remember that the TZ environment variable controls the runtime’s notion of “local”. Set it in your worker context if you need a different default.

Build full timestamps from partial data

Some sources expose only a time-of-day. You can synthesise a full timestamp by stitching the current date to the captured time and sending it back through the time action.

- time:
    output-field: current_date
    output-format: '%Y-%m-%d'
- add:
    output-fields:
      combined_ts: '${current_date} ${event_time}'
- time:
    input-field: combined_ts
    input-format: '%Y-%m-%d %H:%M:%S'
    output-field: '@timestamp'

This pattern keeps everything inside the DSL—no external scripting required.

Use schedule-aware offsets

Schedulers drive run windows, so pairing them with ${time|...} offsets works well for backfills or catch-up scenarios. For example, subtract five minutes when polling an API that lags slightly behind real time:

- add:
    output-fields:
      fetch_from: "${time|now_time_iso - 5m}"

Offsets respect any random jitter or windowing configured on the trigger, ensuring downstream systems receive consistent, deduplicated slices.