Skip to content
Lyft Data Docs (Beta)

Web Services And Client

Web Services and Client

Lyft Data ships HTTP-focused connectors so a pipeline can accept inbound webhooks, poll remote APIs, and push results to downstream services—all without custom glue. This page highlights the core pieces and when to use them.

Receive webhooks with http-server

Use the http-server input when external systems need to post data into Lyft Data. The runtime spins up a listener on the address (and optional path) you configure, captures the HTTP method, URL, decoded query parameters, headers, and body into a JSON event, and can enrich the event with the caller IP.

Key capabilities:

  • Serve plain HTTP or enable TLS by supplying tls.cert and tls.key.
  • Shape responses with custom-response and content-type, or narrow the capture to a single query parameter via query.
  • Switch to body-only mode (only-body) when you only care about the payload.
input:
  http-server:
    address: 0.0.0.0:8443
    path: /webhook/ingest
    tls:
      cert: /etc/lyftdata/certs/server.crt
      key: /etc/lyftdata/certs/server.key
    source-ip-field: client_ip
    custom-response: '{"status":"accepted"}'
    content-type: application/json

Every request produces an event containing the method, URL, decoded query map, headers, body, and (if configured) client_ip field so downstream actions can fan out or persist the exact webhook call.

Poll APIs with http-poll

http-poll is the scheduled HTTP client. It supports interval or cron triggers, arbitrary methods (GET/POST/PUT/DELETE and more), headers, query parameters, basic auth, request bodies, and form URL-encoded payloads. The response section lets you opt in to status codes, response headers, or rename the body field, while json, ignore-line-breaks, document-mode, and events-field control how the payload is emitted.

input:
  http-poll:
    url: https://status.example.com/api/incidents
    trigger:
      interval:
        duration: 60s
    method: get
    headers:
      Authorization: Bearer {{secrets.status_token}}
      Accept: application/json
    response:
      status-field: http_status
      headers-field: response_headers
      response-field: incident_body
    json: true
    ignore-line-breaks: true

When the endpoint answers, Lyft Data records the status and headers (if requested) alongside the parsed JSON body, so later actions can branch on http_status or pick fields out of incident_body without losing context.

Send events with http-post or http-get

Use the http-post output to deliver processed events to remote services. URLs support ${} field expansions, and body controls how the request payload is built (entire event, a specific field, or a literal/template string). You can also add headers, retries, batching, and method overrides—including turning the output into a GET or DELETE call. Lyft Data uses the same engine for the http-get DSL shortcut.

output:
  http-post:
    url: https://collector.example.com/events/acme-retail
    method: post
    headers:
      Content-Type: application/json
      X-Request-ID: "{{context.request_id}}"
    body:
      field:
        field: payload
    retry:
      timeout: 30s
      retries: 5
    batch:
      mode: fixed
      fixed-size: 100
      timeout: 250ms
      wrap-as-json: true

Each batch is posted with the selected HTTP verb; setting method: delete or choosing the http-get variant causes Lyft Data to reuse the same client with the alternate verb. The optional insecure: true flag allows self-signed endpoints during development (use with care).

Feature notes and compatibility

  • HTTP server output: Hotrod exposed an HTTP server output that returned the last event; Lyft Data currently omits a runtime implementation. To expose live results over HTTP, connect the job to a downstream service (for example with http-post) or build a small façade that reads from the destination store.
  • Authentication helpers: http-poll supports basic auth directly; for other schemes inject headers or tokens via context/secrets.
  • TLS: Inbound TLS termination happens inside the http-server input. Outbound calls honor system root certificates and can skip verification only when insecure is set explicitly.