Grafana JSON API Datasource

Overview

The simpod Grafana JSON API datasource can call Lyft Data’s explorer APIs to build dashboards without a bespoke connector. Once configured, Grafana can surface job throughput, licensing volumes, and any future explorer datasets exposed by the server.

Prerequisites

• Lyft Data server 1.9 or later with the metrics explorer service enabled (default).
• Administrative access to the Lyft Data server to start it with additional CLI flags or environment variables.
• A Grafana instance where you can install plugins and add datasources.
• Network connectivity from Grafana to the Lyft Data server over HTTPS (recommended) or HTTP.

Step 1 — configure explorer authentication

The explorer routes (/api/db/explorer/*) require either an admin JWT or a fixed token. Grafana works best with the fixed token so dashboards can authenticate without user credentials.

Option A: supply the token via CLI flag

Start the server with a value that includes the literal header string you want Grafana to send. Including the Bearer prefix keeps it compatible with standard auth conventions.

cargo run --bin lyftdata -- run server \
  --lyftdata-db-explorer-fixed-token "Bearer grafana-explorer-RANDOMHEX"

Replace RANDOMHEX with a random string (for example the output of openssl rand -hex 16).

Option B: use an environment variable

Set LYFTDATA_DB_EXPLORER_FIXED_TOKEN before launching the server (or in your service manager):

export LYFTDATA_DB_EXPLORER_FIXED_TOKEN="Bearer grafana-explorer-token"
cargo run --bin lyftdata -- run server

The value is stored in the settings database once the server boots. You can rotate it by restarting with a new value.

Step 2 — install the Grafana plugin

Install the datasource on your Grafana instance. For Grafana OSS or Grafana Enterprise:

grafana-cli plugins install simpod-json-datasource
systemctl restart grafana-server

Grafana Cloud users can enable the plugin from the cloud portal instead of using grafana-cli.

Step 3 — add the datasource in Grafana

1. Navigate to Connections → Data sources → Add data source.
2. Choose JSON API.
3. Fill in the basic configuration:
   • Name: for example Lyft Data explorer.
   • URL: https://your-server-host/api/db/explorer (include /api/db/explorer as the base path).
   • Access: Server (so Grafana adds the custom header).
4. Under HTTP settings → Custom HTTP Headers, add:
   • Header: Authorization
   • Value: the exact token string configured above (for example Bearer grafana-explorer-token).
5. Save & test. Grafana performs a GET /api/db/explorer/ request; the server replies with 200 OK, confirming connectivity.

If you prefer to use admin JWTs instead, generate a short-lived token and place Authorization: Bearer YOUR_JWT in the header list. Dashboards will stop working when the JWT expires, so fixed tokens are recommended for shared panels.

Step 4 — build panels with the explorer APIs

Switch to a panel and add a query using the JSON datasource. The plugin’s Builder mode aligns with Lyft Data’s explorer schema.

Metrics catalog

When the panel loads the first time, Grafana calls POST /metrics. Lyft Data returns:

[
  {
    "label": "Job Metrics",
    "value": "Jobs",
    "payloads": [
      { "name": "selected_jobs", "type": "multi-select" },
      { "name": "selected_job_versions", "type": "multi-select" },
      { "name": "selected_workers", "type": "multi-select" },
      {
        "name": "selected_series",
        "type": "multi-select",
        "options": [
          { "label": "Events In", "value": "events_in" },
          { "label": "Rated Input Bytes", "value": "rated_input_bytes" }
        ]
      }
    ]
  },
  {
    "label": "License Volumes (30 Days)",
    "value": "Volumes",
    "payloads": []
  }
]

Pick Job Metrics to unlock the payload selectors.

Dynamic payload options

For multi-select fields without inline options, Grafana issues POST /metric-payload-options with the current payload. The server resolves job names, workers, and available versions from the metrics repository. The request body looks like:

{
  "metric": "Jobs",
  "name": "selected_jobs",
  "payload": {
    "selected_workers": []
  }
}

Query execution

When you click Run query, Grafana sends POST /query. Example request:

{
  "targets": [
    {
      "refId": "A",
      "metric": "Jobs",
      "payload": {
        "selected_jobs": ["sample-job"],
        "selected_series": ["events_in", "rated_input_bytes"],
        "selected_workers": [],
        "selected_job_versions": []
      }
    }
  ],
  "range": {
    "from": "2025-01-01T00:00:00Z",
    "to": "2025-01-01T12:00:00Z"
  }
}

The server responds with Grafana data frames (time series or tables) that the panel renders immediately.

Endpoint summary

Endpoint	Method	Purpose
/api/db/explorer/	GET	Used by Grafana’s Test button; returns 200 OK and an empty body.
/api/db/explorer/metrics	POST	Returns the metrics catalog shown in Builder mode.
/api/db/explorer/metric-payload-options	POST	Resolves dependent select lists (jobs, workers, versions).
/api/db/explorer/query	POST	Executes the explorer query and returns Grafana-formatted frames.

All routes require the Authorization header and are available when the server starts with metrics services enabled.

Troubleshooting

• 401 Unauthorized / 403 Forbidden: confirm the Authorization header in Grafana exactly matches the configured fixed token. Restart the server with a new token if needed.
• Plugin errors about empty metrics: ensure the metrics subscriber service is running and that historical metrics exist; otherwise the catalog may be empty.
• Timeouts: reduce the time window or limit selected jobs; Grafana waits for Lyft Data to finish assembling the response before drawing the panel.
• Dashboard uses JWTs: rotate tokens before they expire or switch to the fixed token method to avoid interruptions.

With the datasource in place, you can mix explorer charts alongside other observability data in Grafana dashboards.