What happened
Deepgram announced two observability additions for its speech-to-text and text-to-speech models deployed as Amazon SageMaker AI real-time endpoints. Enhanced Metrics publish usage and billing data to the customer’s CloudWatch account, while Prometheus and OpenTelemetry integrations expose engine, host and per-GPU measurements through SageMaker AI detailed observability.
Deepgram’s announcement concerns speech-to-text and text-to-speech models running as AWS Marketplace model packages on Amazon SageMaker AI real-time endpoints in a customer’s AWS account. The company says audio and transcripts remain inside that account while SageMaker provides deployment, scaling and monitoring controls. The post frames the new capabilities as closing a visibility gap: standard endpoint monitoring can show availability and request handling, but may not reveal which speech features are used, what units drive marketplace charges or how inference uses each GPU.
The first addition, called Deepgram Enhanced Metrics, sends billing and usage information to Amazon CloudWatch through CloudWatch Embedded Metric Format records written to the container’s standard output. SageMaker forwards that output to the endpoint’s CloudWatch log group, where CloudWatch extracts the records into ordinary metrics. According to the source, the process requires no separate agent, sidecar or additional IAM permission and works with AWS Marketplace network isolation because it uses the existing SageMaker-to-CloudWatch logging path. The billing namespace is Deepgram/SageMakerInference. Its ConsumedUnits metric represents billable inference units for completed streaming, prerecorded and text-to-speech requests, while AudioDurationSeconds and CharCount describe processed audio and synthesized characters. Deepgram says these values are the same ones used for AWS Marketplace metered billing.
The second usage stream, Deepgram/SelfHosted, is emitted by the Deepgram API server and shows how traffic uses the service rather than directly reconciling a bill. The source lists metrics for streaming and prerecorded volume, model tiers such as nova-3 and flux, and features including diarization, smart formatting, redaction and keyterm prompting. These metrics aggregate across Deepgram endpoints in an AWS account and Region. The post says they use low-cardinality dimensions without transcripts, text-to-speech input or per-request identifiers, but do not include endpoint name or instance ID. The usage stream can be disabled with an environment-variable override, while the billing stream cannot be disabled because Deepgram describes it as part of the metering pipeline.
For lower-level operations, Deepgram says its containers expose Prometheus metrics that SageMaker AI detailed observability can collect through an AWS-managed OpenTelemetry Collector running on each endpoint instance. The resulting data includes DCGM exporter metrics for individual GPUs, node-exporter metrics for host CPU and memory, and Deepgram engine metrics such as active streaming requests and estimated stream capacity. The source says each series carries SageMaker resource labels, including endpoint, variant and instance identifiers. Customers can query a particular endpoint, instance or GPU using PromQL through CloudWatch, Grafana or another compatible tool. Detailed observability is enabled by default for newly created endpoints, with a 60-second publishing frequency, while older endpoints require an endpoint-configuration update using a blue/green deployment.
Read the source: aws.amazon.com ↗
Why it matters
The changes target a practical problem in self-hosted AI: customers can monitor whether an endpoint is operating, but may lack visibility into the model features driving usage, the units behind marketplace billing and the capacity of individual accelerators. The source describes a way to connect those operational and financial questions without opening outbound network access from the model container.
The immediate importance is operational visibility. Speech inference is not a uniform workload: streaming sessions, prerecorded audio and text-to-speech requests can place different demands on a deployment, and optional features can change processing volume or resource use. Deepgram’s account-level usage metrics are intended to show those differences in terms operators and finance teams can use. A customer could examine how much traffic used diarization or compare model-tier consumption over time. That may help organizations identify unexpected usage patterns, design internal chargeback systems or decide which workloads should be routed to different deployments. The source presents these as uses of the metrics, not demonstrated savings or performance improvements.
The billing connection could make marketplace-based AI procurement easier to audit. Deepgram says the ConsumedUnits metric carries the same billable-unit values used for AWS Marketplace metering, allowing customers to compare CloudWatch totals with their AWS bill. That is more specific than a request count, because a request may represent a streaming session, a quantity of audio or a volume of synthesized characters. The source says ordinary CloudWatch features—including dashboards, alarms and metric math—can be applied to the data. It does not provide an example reconciliation result, error rate, accounting control or independent confirmation that the metrics always match an invoice. Organizations would still need their own financial and compliance checks.
The per-GPU and engine-level metrics address a different problem: capacity planning in a scaled deployment. A fleet-wide average can obscure an overloaded accelerator, while request counts alone may not show concurrent-stream headroom. Deepgram says its engine’s estimated stream capacity can be compared with active requests to provide an engine-reported signal for scaling decisions, and that GPU metrics can be inspected separately on multi-GPU instances. This could help operators investigate uneven utilization, tune autoscaling thresholds or determine when an instance type is becoming a bottleneck. The wording matters: the capacity figure is the engine’s estimate, not an independently validated guarantee of how many streams an instance will sustain under every audio format, model, feature combination or workload pattern.
The security and governance angle is also concrete. The source says AWS Marketplace containers operate with network isolation and cannot make outbound connections, a design chosen by some customers for security and compliance reasons. Deepgram’s proposed telemetry path keeps measurements inside the customer’s CloudWatch environment without requiring the container to open a network route. The source also says the listed dimensions contain no personally identifiable information and do not include transcripts or request identifiers. Those statements describe the announced architecture, not a complete privacy assessment: customers remain responsible for their AWS configuration, retention settings, access controls and regulatory obligations under the shared responsibility model.
What to watch next
The source does not provide independent measurements of monitoring overhead, cost savings, accuracy of the capacity estimates or customer adoption. Teams considering the deployment will need to test whether account-level billing metrics are sufficiently granular, whether engine estimates match observed traffic and what additional CloudWatch, logging and GPU-hosting costs result in their own environments.
The first unknown is real-world performance. The announcement does not report the CPU, memory, storage or latency overhead of writing embedded metrics, scraping Prometheus endpoints or publishing data every 60 seconds. It also does not quantify CloudWatch charges for logs and metrics, collector costs or any effect on autoscaling behavior. AWS and Deepgram note that endpoint hosting, GPU instances, CloudWatch logs and metrics, networking and related infrastructure can all incur charges, even during the models’ stated 14-day Deepgram trial. Prospective users will need workload-specific measurements rather than assuming improved observability is cost-neutral.
A second issue is metric granularity. Deepgram’s billing and usage streams aggregate across endpoints in an account and Region and do not identify an endpoint or instance. That may be sufficient for broad financial reporting, but cannot by itself answer which deployment generated a particular volume. Per-endpoint and per-GPU analysis depends on the separate Prometheus and OpenTelemetry path and on detailed observability being enabled. The source says new endpoints have that setting enabled by default and older endpoints can be updated, but does not describe migration edge cases, retention periods, dashboard templates or how operators should handle missing, delayed or conflicting series.
The capacity signal also deserves validation. Deepgram describes engine_estimated_stream_capacity as the engine’s own estimate of sustainable concurrent streams. The source does not show how that estimate is calculated, how it changes with model tier or enabled features, or how it performs during traffic spikes and degraded conditions. Operators should compare it with observed latency, errors, queueing and completed sessions before using it as an automatic scaling trigger. Standard SageMaker metrics such as ConcurrentRequestsPerModel, FirstChunkLatency and Invocation5XXErrors remain relevant because the new streams complement rather than replace them.
Finally, the announcement establishes availability for Deepgram SageMaker AI deployments, but not broader ecosystem impact. It does not state how many customers have adopted the metrics, whether other speech-AI vendors offer equivalent visibility, or whether AWS will extend the same integration to additional model packages. The practical follow-up is whether users can reconcile invoices reliably, isolate resource hotspots and manage feature-level usage without adding new infrastructure. Evidence from independent deployments, documented metric behavior over longer periods and customer experience would clarify whether the feature materially changes day-to-day governance of self-hosted speech AI.
