Context Management

Context management

Context variables let you reuse the same job definition across environments without copying and editing YAML for every deployment. Lyft Data resolves context values before a job runs, so inputs, actions, and outputs can pull in credentials, destinations, or feature flags with {{variable}} placeholders.

What is context?

A context is a key-value map that the control plane merges into a job definition at deploy time. Any field that supports interpolation can reference a context value with {{name}}. Context can contain strings, numbers, booleans, arrays, or JSON objects.

Why use context?

• Parameterise jobs for staging, production, or specific customers without duplicating pipelines.
• Centralise shared configuration (API tokens, bucket names, feature toggles).
• Override values for a single worker or job when you need host-specific tweaks.
• Roll out changes safely by editing context instead of redeploying code.

Context scopes and precedence

Lyft Data layers multiple scopes when building the runtime configuration. The most specific scope wins when the same key appears in more than one place:

1. Job overrides (control plane) – Values saved for a job inside the UI or via /api/contexts/update-job-map/{job}. Highest precedence.
2. Worker overrides – Values attached to a worker ID through the worker detail page or /api/contexts/update-worker-map/{worker}. Override global defaults and job definition values.
3. Global defaults – Environment-wide settings managed on the Context workspace or with /api/contexts/update-global. Used when no job or worker override is present.
4. Job definition defaults – The context: block stored with the job YAML. Acts as the final fallback.

This merge order mirrors the staged context logic (full_context_for) documented in the knowledge base.

Manage context in the UI

Global defaults

1. Sign in to the Lyft Data UI with an administrator account.
2. Open Context from the navigation sidebar.
3. Add, edit, or remove keys. Save changes to publish them immediately to all jobs.
4. Use the export and import controls on the page to back up or restore global context JSON.

Job-specific overrides

1. Open Jobs and select the job you want to adjust.
2. Switch to the Context tab in the job editor.
3. Add overrides for keys such as destination_path or schedule_cron.
4. Save the context, then stage or redeploy the job so workers pick up the new values.

Worker-specific overrides

1. Navigate to Workers, then open the worker you want to customise.
2. Use the Worker context card to set host-specific values (for example region or local_path).
3. Save the changes; the worker receives the updated map immediately.

Manage context via the API

Automation and scripting workflows can call the REST endpoints directly. Authenticate with an admin JWT or session cookie.

# Update a global setting
curl -s -X POST "https://lyftdata.example.com/api/contexts/update-global" \
  -H "Authorization: Bearer ${ADMIN_JWT}" \
  -H "Content-Type: application/json" \
  -d '{"s3_export_bucket": "analytics-prod"}'

# Override a value for a single job
curl -s -X POST "https://lyftdata.example.com/api/contexts/update-job-map/daily-sync" \
  -H "Authorization: Bearer ${ADMIN_JWT}" \
  -H "Content-Type: application/json" \
  -d '{"schedule_interval": "15m"}'

# Attach host-specific settings to worker-01
curl -s -X POST "https://lyftdata.example.com/api/contexts/update-worker-map/worker-01" \
  -H "Authorization: Bearer ${ADMIN_JWT}" \
  -H "Content-Type: application/json" \
  -d '{"local_cache_dir": "/var/cache/lyftdata"}'

Use update-worker-nomerge if you need to replace a worker context wholesale instead of merging keys.

Inline job defaults

Jobs can ship default values alongside the definition. Add a context block to the YAML and reference those keys throughout the pipeline:

name: ping
context:
  addr: https://status.example.com/api
  interval: 5m
input:
  http-poll:
    url: "{{addr}}"
    trigger:
      interval: "{{interval}}"
    json: true
output:
  print: {}

Operators can later override addr or interval globally, per job, or per worker without editing the job file.

Built-in context variables

The control plane injects helper keys that are always available when a job is rendered:

• {{job}} – The job name being deployed.
• {{worker}} – The identifier of the worker executing the job.

Best practices

• Treat secrets like API keys as masked entries in the UI; restrict who can reveal values.
• Prefer descriptive key names (analytics_export_bucket) over abbreviations.
• Update context through staging or change management so overrides are reviewed like code.
• Document critical keys in your team runbooks so on-call engineers know how to adjust them safely.

Debugging context

• Review the Context workspace for a live view of global variables.
• Download the merged context for a job by staging it and viewing the YAML tab, or fetch it through the API:

  curl -s "https://lyftdata.example.com/api/contexts/global" \
    -H "Authorization: Bearer ${ADMIN_JWT}" | jq
  curl -s "https://lyftdata.example.com/api/contexts/worker/worker-01" \
    -H "Authorization: Bearer ${ADMIN_JWT}" | jq

• If a value is not taking effect, confirm there is not a higher-precedence override (job or worker) masking your change.

Example: multi-level override

1. Define context.trigger_interval: 10m inside the job definition to supply a safe default.
2. Set a global override trigger_interval=5m for production in the Context workspace.
3. Pin trigger_interval=30m on a single worker that talks to a rate-limited API.

The deployed job resolves to 30m on that worker, 5m on every other worker, and falls back to 10m in environments where no overrides exist.