Skip to content
Lyft Data Docs (Beta)

Logs and Issues

Start with the Log viewer for quick queries, move to the Issues panel for job-level context, and tap into the live Messages stream when you need second-by-second insight. This page walks through those surfaces in order, then explains how the runtime feeds them.

Log viewer

Open Logs from the Operate sidebar (or visit /logs). The page is designed for fast iteration:

  • Quick counters – the toolbar shows the total rows returned plus Info/Warning/Error badges so you can see the mix immediately. The All / Warning / Error buttons toggle the log_level filter without retyping anything.
  • Time range – start and end pickers default to the last seven days. Adjust them to widen the window; each change reloads the table and resets the page size to 100 so you never accidentally request thousands of rows.
  • Worker and job facets – dropdowns are populated from the REST API (GET /api/workers/logs/workers and /jobs). Filters persist in the URL, which makes sharing views easy (/logs?worker=<id>&job=<name>). Deep links from Workers or Jobs append the same parameters automatically.
  • Record controls – pick 100, 200, or 500 rows per fetch. Refresh reruns the query with current filters, while Clear removes all filters and restores the default time range.
  • Table detail – timestamps render in your local timezone, levels show DaisyUI badges, and long messages get truncated with tooltips. If the table says No logs, double-check advanced filters or clear everything with one click.

The viewer is backed by GET /api/workers/logs, so anything you can do in the UI can also be scripted:

Terminal window
curl -s "http://<server>:3000/api/workers/logs?log_level=Warning&num_records=200" | jq '.[].msg'

Issues panel

  • The Issues badge in the header counts unresolved problems through the Svelte problemsStore. On load it fetches the latest 50 warning and error records so the badge reflects reality immediately.
  • A WebSocket subscription (subscribeToErrorsAndWarnings) streams new events as the runtime emits them. The store deduplicates by timestamp, job, worker, and severity so repeated messages do not flood the list.
  • Dismissals persist in browser storage (up to 500 signatures) so reviewed warnings stay hidden across page reloads.
  • Each issue card links outward: job and worker badges jump to the relevant pages, and View Logs pivots into the Log viewer with filters pre-filled.

Messages stream

The same events power the live Messages view under Operate -> Messages. You can subscribe to Job Status, Worker Status, Job Messages, Errors & Warnings, or enable the debug firehose. Use it to:

  • Watch job deployments and shutdowns in real time.
  • Track worker heartbeats alongside lag or backoff messages.
  • Capture structured job logs without tailing server files.
  • Pause, filter, or export the buffer as JSON for audits.

Messages remembers channel selections in your browser, so the feed comes back exactly how you left it.

How the runtime feeds these surfaces

  1. Runtime captures the issue – when an action hits a problem it records a structured warning or error on the event and keeps the event flowing. Job runtime_options such as disable_warnings, plus per-run flags like inline_warnings, control whether the warning is emitted or shown inline.
  2. Messages stream distributes it – the runtime publishes structured log messages onto the Messages stream. Worker shutdowns, job retries, and warnings all travel through the same channel you can monitor in the UI.
  3. Server keeps a history – the platform stores every message and exposes it through /api/workers/logs (plus the worker/job facet endpoints) so you can query historical data and drive dashboards or scripts.
  4. UI surfaces the data – Logs provides search and filtering, Issues summarises outstanding problems per job, and Messages lets you observe the feed live.

Runtime emission rules

  • Pass-through with warnings – actions do not drop events on configuration errors; they attach warnings instead. This contract is documented and tested in docs/runtime/runtime-test-plan.md.
  • Visibility toggles – disable_warnings suppresses warning generation for a job, while inline_warnings controls whether transient runs (Run & Trace) show warnings in the response.
  • Structured context – emitters include the step name, counts, and relevant arguments. Files input also records which file and checkpoint failed (docs/runtime/files-recovery.md).

Troubleshooting the pipeline

  • No warnings recorded – ensure the job metadata does not set disable_warnings=true. Use Run & Trace with inline_warnings=true to confirm the warning is produced locally.
  • Warnings missing from the UI but present in logs –
    1. Check Operate -> Messages to confirm new events are flowing.
    2. Query /api/workers/logs directly. If results appear, inspect the browser console for WebSocket errors.
    3. Verify the browser allows local storage; dismiss persistence relies on it.
  • High volume of messages – narrow by worker or job, increase page size as needed, and export with curl for deeper analysis. Messages health cards highlight when the stream stays above recommended thresholds (docs/aggregator/performance-tasks.md).
  • Messages – full guide to the live stream.
  • Monitoring – dashboards and API checks that complement Logs & Issues.
  • Troubleshooting reference – retention settings and log collection tips.
  • docs/runtime/runtime-test-plan.md – engineering detail behind the pass-through-with-warnings contract.