Skip to main content

Push run and worker metrics via OTLP

Feature Requests

Metrics today come from the pull-based Prometheus exporter, which still needs an Admin key (a read-only role is Planned), or from the Datadog-only integration. Teams that are standardizing on OpenTelemetry collectors want Spacelift to push run, queue, and worker metrics over OTLP to any backend, without running and securing an exporter per cluster.

This is related to, but separate from, Richer native Prometheus metrics: full exporter coverage, run timing, worker/queue metrics. That post is about making the pull exporter complete. This one is about a push path over OTLP.

Also related: Expose usage metrics through Datadog.

Customers also want better queued-run and queue-time monitoring than they get from the current run count metrics.

What this should include:

  • The same endpoint config as the OTLP trace export request, so one destination can take traces and metrics.
  • Metrics for run count, outcome, and duration by phase, queue time, drift runs as a separate series, worker pool busy/idle/capacity, and usage minutes. Labels for stack, space, worker pool, and run type.
  • Keep the Prometheus exporter as the pull option. OTLP is the push option, not a replacement.
Workaround
Problem
Status: ⬆️ Gathering votes1 comment

Log in to comment and vote

Comments1

  • Jonah Kowall

    Team•

    Nov 11

    Thanks. We're treating this as a follow-on rather than a separate build right now. The gaps listed here (queue time, duration by phase, drift as its own series, worker pool busy/idle/capacity) are the same data the Prometheus exporter work is adding, and a read-only role will remove the Admin key requirement. We'd rather fix the data once and expose it everywhere.

    If you've standardized on the OpenTelemetry Collector, you can get these metrics to any OTLP backend today. Run the Prometheus exporter, scrape it with the Collector's prometheus receiver, and export over OTLP. Once run trace export ships, the Collector's span metrics connector can also derive run counts, outcomes and phase durations from the spans.

    Gauge data like worker pool capacity and queue depth is where a native push would add the most. If running the exporter is the blocker, tell us why (number of accounts, network policy, credential handling) and which backend you use. That's the evidence that would move this up.