How to Build an OpenTelemetry Gateway That Controls Cardinality Before Backend Ingestion

Observability cost problems usually begin upstream of the invoice. A backend can only store, index, aggregate, and retain the telemetry that reaches it. If application teams can emit arbitrary dimensions, duplicate streams, debug logs, and low-value spans directly into a premium backend, cost control becomes a billing exercise instead of an architecture decision. An OpenTelemetry Collector gateway gives infrastructure teams a place to make that decision before ingestion.

The useful design pattern is not ‘put a Collector in front of everything.’ It is to create an explicit telemetry boundary with measurable ownership: sources send to an OTLP endpoint, the gateway classifies and transforms signals, routing policy decides which backend receives which data, and the gateway itself exposes health and drop metrics. The control plane should make rejected, sampled, transformed, and failed telemetry visible rather than silently deleting data.

Start with a telemetry contract, not a processor list

Before configuring processors, define what the organization expects each signal to contain. For metrics, identify the metric families that are operationally important, the dimensions that are required for incident response, and the dimensions that are unbounded. For logs, separate security and application events from repetitive health checks, debug output, and payload-heavy records. For traces, define which services require complete visibility, where sampling is acceptable, and which attributes are needed for routing or correlation.

This contract becomes the denominator for validation. A gateway that drops 40 percent of incoming telemetry is not necessarily efficient, and a gateway that drops nothing is not necessarily safe. The question is whether the retained telemetry matches an explicit operational requirement. Store the policy revision, owner, affected services, and intended destination for every filtering or routing rule so later cost changes can be tied back to a controlled decision.

Use the gateway as a policy boundary

OpenTelemetry describes the gateway pattern as applications or other Collectors sending telemetry to a central OTLP endpoint that forwards data to one or more backends. The same architecture centralizes credentials, filtering, and sampling policy. That is useful for cost control because backend destinations stop being embedded independently in every workload. A change to routing policy can be reviewed and rolled out at the gateway layer instead of changing dozens of application configurations.

A practical topology is one gateway tier per cluster, data center, or Region when failure domains and network paths require separation. The gateway should not become a cross-Region single point of failure just to centralize policy. Keep the ownership model explicit: platform teams own gateway availability and policy mechanics, while service owners remain responsible for the semantics of the telemetry they emit.

Cardinality is a resource problem before it is a pricing problem

Prometheus documents the core risk clearly: every unique label set creates another time series, and unbounded dimensions can create a cardinality explosion. OpenTelemetry uses attributes in the same role when metrics are translated into backend-specific series. Values such as user IDs, request IDs, email addresses, pod UIDs, dynamic URLs, and ephemeral object identifiers can multiply series counts dramatically without improving infrastructure-level decisions.

Treat cardinality as a governed budget. For each metric family, identify stable dimensions, bounded dimensions, and prohibited dimensions. Stable dimensions might include service, environment, Region, or status class. Prohibited dimensions are values whose possible population is effectively unbounded. The gateway can remove or normalize these attributes before export, but the preferred long-term fix is still to correct the instrumentation source so high-cardinality data is not generated unnecessarily.

• Track unique series or attribute combinations per high-value metric family.
• Flag dimensions whose value population grows with users, requests, sessions, objects, or ephemeral infrastructure.
• Define a maximum acceptable cardinality class for each destination rather than one global threshold.
• Preserve dimensions required for alerting and incident isolation before removing labels for cost reasons.
• Record every attribute-removal or normalization rule as a versioned telemetry policy.

Filter before expensive processing and export

The OpenTelemetry Collector can filter logs, metrics, and spans before they reach exporters. Official guidance explicitly calls out filtering unwanted telemetry as a way to reduce load on both the Collector and the backend. That makes filtering the first economic control after the telemetry contract is defined. Drop records that have no operational consumer instead of paying to ingest them and hoping a retention setting will solve the problem later.

Filtering should be deterministic and testable. A rule that drops health-check logs by route and status should have fixtures showing which records are retained and which are rejected. A rule that removes low-value metrics should include the dashboards and alerts checked before deployment. If a filter depends on application naming conventions, validate those conventions continuously. Otherwise a harmless service rename can turn a narrow filter into an accidental data-loss policy.

Transform attributes only when the semantic loss is understood

The Collector transform processor can rename, remove, or alter telemetry attributes. This is useful when multiple teams emit equivalent dimensions under different names, when a backend does not need raw high-cardinality values, or when sensitive metadata should not leave the environment. Transformation can also normalize noisy dimensions into bounded categories, such as mapping raw HTTP routes to stable route templates.

Do not hide transformation behind a generic ‘cleanup’ stage. Every destructive transformation should answer three questions: which downstream queries change, which alerts depend on the original field, and whether the raw value is available anywhere else if needed for an investigation. The most dangerous cost optimization is one that appears successful until the first incident requires a dimension that was silently removed.

Protect the gateway itself with memory and queue controls

A gateway that protects the backend but collapses under traffic spikes is not an observability control. OpenTelemetry guidance recommends memory limiting and batching, and its configuration guidance also calls out queue sizing, retries, timeouts, and compression as resource controls. These settings are not just performance tuning. They define how the gateway behaves when exporters slow down, destinations fail, or incoming traffic exceeds normal rates.

Measure the gateway as a production dependency. Track accepted records, exported records, dropped records by reason, retry volume, queue utilization, export failures, memory pressure, and restart count. Use separate metrics for intentional policy drops and involuntary loss. If both appear as one ‘dropped telemetry’ counter, operators cannot distinguish cost policy from an outage.

Avoid accidental coupling between pipelines

OpenTelemetry’s architecture documentation notes that when the same receiver feeds multiple pipelines, fan-out is synchronous. If one downstream processor blocks, the other pipelines attached to that receiver can also be delayed. This matters when one pipeline sends critical security telemetry while another performs heavier transformation or exports to a slower destination. A simple configuration can therefore create hidden coupling between signals that have different reliability requirements.

Separate pipelines or Collector tiers when service-level objectives differ materially. Critical security logs may need a short, conservative processing chain and a durable destination, while bulk infrastructure metrics can tolerate aggressive filtering and a different backend. The routing architecture should reflect business priority instead of assuming every signal deserves the same latency, retention, or delivery path.

Route by value, not by tool ownership

A cost-controlled design does not require replacing a premium observability platform. It requires using each backend for the workloads it handles best. Application traces, SLO signals, high-value incidents, and service analytics may justify a premium backend. Stable host metrics, bulk infrastructure telemetry, verbose logs, or long-retention records may be better suited to Prometheus-compatible storage, Zabbix, Loki, OpenSearch, or object storage.

The gateway should express this as routing policy rather than organizational habit. For every signal class, record the destination, retention expectation, reason for routing, and fallback behavior. If a destination is unavailable, define whether the gateway queues, retries, drops, or fails over. An undefined failure mode becomes an accidental architecture decision during an incident.

Validate changes with dual-run measurements

Do not deploy a new filter or cardinality rule and judge success only from a lower bill. Use a bounded validation window. Before the change, capture incoming record rates, unique series counts, backend ingestion volume, critical dashboard queries, alert evaluation, and gateway resource usage. During the change, compare retained telemetry to the baseline and verify that required queries still answer correctly.

When feasible, duplicate a small sample to a validation destination or temporarily retain pre-filter counts so the effect of the rule is measurable. The proof of a safe optimization is not that less data was exported. It is that less low-value data was exported while required operational coverage stayed intact.

Failure modes to test deliberately

• A high-cardinality attribute is reintroduced by a new service version and series growth is not detected.
• A filter rule matches a broader set of services after a naming change.
• One slow exporter creates backpressure that affects a second pipeline sharing the same receiver.
• Queue capacity is sized larger than available memory and the Collector becomes unstable under an outage.
• Intentional policy drops and accidental exporter failures are counted together.
• A routing rule sends security-relevant logs to a lower-retention destination without review.
• A cost optimization removes a dimension used by an alert or incident query.
• A Collector restart loses in-memory queued telemetry and no metric exposes the gap.

A practical deployment sequence

1. Inventory current telemetry sources, destinations, and direct-to-vendor exporters.
2. Define signal ownership and the minimum dimensions required for operations, security, and incident response.
3. Deploy the gateway in one bounded environment without changing filtering policy.
4. Measure incoming rates, unique metric series, gateway resource use, queue behavior, and backend ingestion.
5. Add one deterministic filter or normalization rule with explicit validation criteria.
6. Compare dashboards, alerts, traces, and incident queries before and after the rule.
7. Separate pipelines when reliability or processing requirements differ.
8. Expand routing by signal value, not simply by application or team ownership.
9. Retain policy revisions and gateway metrics so future cost changes remain explainable.

The gateway becomes evidence for telemetry ownership

A well-operated OpenTelemetry gateway produces more than cheaper ingestion. It creates an auditable record of what telemetry enters the platform, what is intentionally removed or transformed, where each signal is routed, and whether delivery is healthy. That is useful for infrastructure governance because the telemetry architecture stops being an undocumented collection of agents and vendor defaults.

The engineering boundary is straightforward: source -> gateway collection -> policy processing -> routing -> backend retention -> validation. The strongest implementation keeps those stages observable and versioned. Cost reduction then becomes a consequence of telemetry ownership rather than a series of emergency exclusions made after the bill arrives.

Related resources

• Datadog Cost Reduction with Zabbix and Grafana
• How to Audit a Datadog Bill Before Changing the Platform
• How to Reduce Datadog Log Ingestion Cost

Sources

• Architecture | OpenTelemetry
• Gateway deployment pattern | OpenTelemetry
• Transforming telemetry | OpenTelemetry
• Collector configuration best practices | OpenTelemetry
• Metric and label naming | Prometheus