Skip to content
Lyft Data Docs (Beta)

Automating Deployments (CI/CD)

This guide is for platform teams who want to ship Lyft Data jobs like code. It shows how to stage, test, and promote jobs via the CLI and integrate with a standard CI pipeline.

Prerequisites

  • Lyft Data CLI installed locally and in your CI runners.
  • API keys for staging and production servers.
  • Jobs stored in Git with environment-specific context files.
  • Familiarity with the Day 1 production pipeline guide.

1. Structure jobs in Git

Organise each job under jobs/<job-name>/:

  • job.yaml - canonical definition (input, actions, output, triggers).
  • context.dev.yaml, context.prod.yaml - overrides for staging and production.
  • README.md - owners, SLAs, rollback instructions.
  1. Maintain branches such as main (production) and develop or staging for pre-production.
  2. Require pull requests so reviewers can check context changes and scaling impact.

2. Validate before committing

  • Use your preferred linting/tests plus lyftdata jobs import --dry-run --file jobs/<job>/job.yaml to catch syntax issues before trying to publish changes.
  • Use Run & Trace in the UI with sample data to sanity-check transformations end-to-end.
  • Keep validation in CI so every pull request exercises it automatically.

3. Stage via CI

Typical pipeline snippet:

Terminal window
# Review what would change on the staging server
LYFTDATA_URL="$STAGING_SERVER_URL" \
  lyftdata jobs import \
    --file jobs/my-job/job.yaml \
    --dry-run


# Push the update once checks pass
LYFTDATA_URL="$STAGING_SERVER_URL" \
  lyftdata jobs import \
    --file jobs/my-job/job.yaml \
    --update

After publishing, trigger a smoke test run through the UI (Run & Trace) or your preferred API checks before merging the change.

4. Gate production promotion

  • Require a manual approval or release tag once staging checks pass.
  • Import the same job definition into production after approvals:
Terminal window
LYFTDATA_URL="$PROD_SERVER_URL" \
  lyftdata jobs import \
    --file jobs/my-job/job.yaml \
    --update

5. Manage secrets and context

  • Store secrets in your CI manager (GitHub Secrets, Vault, etc.).
  • Keep sensitive values out of YAML; reference them via context placeholders ({{ }} and ${ }).
  • Document merge precedence (server -> worker -> job) and test with dry runs before the first production rollout.

6. Monitor and roll back

  • Watch Monitoring dashboards immediately after promotion.
  • Export the job definition (lyftdata jobs export --dir backups/jobs --dry-run first, then run without --dry-run) so you can restore quickly if needed.
  • Log deployment outcomes in your change management system.

CI/CD checklist

  • Job definitions and context files stored in Git
  • Automated dry-run lyftdata jobs import in CI
  • Staging deployment with smoke test hook
  • Manual approval or release process before production deploy
  • Secrets injected via CI, not committed to the repo
  • Monitoring and rollback plan documented