Advanced Scheduling

Most inputs in Lyft Data run continuously, but several (HTTP polling, exec, object stores, echo, and more) expose a trigger block so you can control cadence explicitly. This guide explains each trigger type, how random offsets and run windows work, and how to test schedules safely.

Trigger variants

Cron

trigger:
  cron: "0 0 * * * *"

• Uses six fields: seconds, minutes, hours, day-of-month, month, day-of-week. 0 0 0 * * * runs daily at midnight UTC.
• Supports random_offset to jitter the schedule per deployment and avoid thundering herds across large fleets.
• Optional window block lets you bound the run to a rolling time range (size, offset, start). The scheduler persists the watermark so retries or restarts pick up where the last successful window ended.

Interval

trigger:
  interval: 5m

• Accepts human-friendly durations parsed via humantime (30s, 5m, 2h, etc.).
• Shares the same random_offset and window options as cron, so you can stagger runs and limit processing to a moving window.

Message

trigger:
  message: {}

• Fires when the server publishes an internal message that matches the filters configured on the internal-messages input (kind, source, job, worker, type, or tag).
• Ideal for reactive pipelines: downstream jobs ingest the control-plane event and run immediately without waiting for a timer.

Note: Earlier documentation referenced condition: blocks inside the trigger. The current schema does not accept trigger conditions; gate execution through context variables or message triggers instead.

Context-driven cadence

Job context files provide a convenient knob for operators. Reference context keys inside the trigger and update them without redeploying the job:

trigger:
  interval: "{{dynamic_interval}}"
context:
  dynamic_interval: "5m"

Changing dynamic_interval in the context file takes effect on the very next run.

Time macros and offsets

Runtime expansions expose the scheduler window to actions and outputs:

• ${time|now_time_secs}, ${time|start_time_iso}, ${time|end_time_fmt %Y-%m-%d}
• Add offsets with + or - plus a duration: ${time|now_time_iso - 5m} backs off five minutes; ${time|start_time_fmt %H:%M + 1h} jumps one hour ahead.
• Invalid offsets raise runtime errors, so stick to supported units (ms, s, m, h, d).

Use these macros to label batches, compute backfill windows, or pass the current schedule window to downstream APIs.

Testing schedules safely

1. Configure the trigger and save your job.
2. Use Run & Trace in the editor to validate the pipeline without waiting for the schedule. Transient runs ignore the trigger and execute immediately.
3. Stage the job and deploy it to a non-production worker.
4. Monitor Operate > Job status and worker logs to confirm the trigger fires on the expected cadence. Cron entries log the next fire time; interval entries log the delay after random_offset is applied.
5. Once validated, deploy to production workers and keep the trigger under watch using Operate monitoring.

Operational tips

• Keep cron expressions in UTC unless you explicitly manage timezones with context or variable expansion.
• Use random_offset on interval jobs that rely on shared APIs to spread traffic evenly.
• When two jobs must never overlap, pair the trigger with the window block so each run processes a distinct, bounded slice.
• Combine message triggers with worker-channel outputs to build reactive pipelines without timers.

These scheduling primitives cover the production feature set. If your cadence requirements extend beyond cron, interval, or message triggers, contact the platform team to discuss dedicated scheduler support.