Data race
Description
A data race exists in the OpenTelemetry metadata handling path of vmagent where metadata slices produced by the OTLP/OpenTelemetry stream parser are assigned directly to the WriteRequest.Metadata field. The parser reuses internal buffers, and the remote write path may still read the queued write request while the parser reuses its backing array, leading to concurrent mutations of the same underlying memory. The fix copies each metadata entry into a PushCtx-owned buffer (mmsDst) before assigning to WriteRequest.Metadata, eliminating the shared backing array and thus the race. This is a correctness/concurrency bug with potential memory-safety implications (crashes or inconsistent state) under concurrent OTLP streaming workloads. Affected code paths involve inserting rows for OpenTelemetry metadata and the remote write path; the fix brings them in line with how the Prometheus remote write path already handles metadata by copying to an independent buffer.
Proof of Concept
package main
import (
"fmt"
"sync"
"time"
)
type MetricMetadata struct {
MetricFamilyName string
Help string
Type int
Unit string
AccountID int
ProjectID int
}
type WriteRequest struct {
Metadata []MetricMetadata
}
func main() {
// Simulated parser backing buffer that would be reused
parserBuf := make([]MetricMetadata, 4)
for i := 0; i < len(parserBuf); i++ {
parserBuf[i] = MetricMetadata{MetricFamilyName: fmt.Sprintf("m%d", i)}
}
// Vulnerable path: consumer holds a reference to the parser-backed slice
mms := parserBuf
// Context WriteRequest shares the same metadata slice (as before the fix)
ctx := &WriteRequest{Metadata: mms}
var wg sync.WaitGroup
wg.Add(2)
// Goroutine 1: parser reuses/overwrites the backing array concurrently
go func() {
defer wg.Done()
for i := range parserBuf {
parserBuf[i].MetricFamilyName = "eviled" // mutate underlying memory
time.Sleep(1 * time.Millisecond)
}
}()
// Goroutine 2: reader consumes the WriteRequest.Metadata concurrently
go func() {
defer wg.Done()
for i := 0; i < len(ctx.Metadata); i++ {
_ = ctx.Metadata[i]
time.Sleep(500 * time.Microsecond)
}
}()
wg.Wait()
fmt.Println("Done (race intentionally demonstrated under -race)")
}
Commit Details
Author: Evgeny Martyn
Date: 2026-07-10 13:43 UTC
Message:
app/vmagent: fix opentelemetry metadata race (#11238)
Fixes a possible data race in `vmagent` when processing OpenTelemetry
metric metadata.
The OpenTelemetry stream parser passes metadata to `insertRows`:
[streamparser.go#L87](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/lib/protoparser/opentelemetry/stream/streamparser.go#L87)
Before this commit, `insertRows` assigned that slice directly to
`ctx.WriteRequest.Metadata`:
[request_handler.go#L85](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/opentelemetry/request_handler.go#L85)
As a result, the write request could keep a reference to the
parser-owned metadata backing array. The parser later reuses and resets
its internal buffers, while the remote write path may still read the
queued write request:
- `PushCtx` / `WriteRequest` reset path:
[prompb.go#L21](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/lib/prompb/prompb.go#L21)
- OpenTelemetry `writeRequestContext` reset path:
[streamparser.go#L145](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/lib/protoparser/opentelemetry/stream/streamparser.go#L145)
This may put the same metadata backing array under two independent
pools:
- `PushCtx` pool:
[push_ctx.go#L52](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/common/push_ctx.go#L52)
- OpenTelemetry parser context pool:
[streamparser.go#L253](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/lib/protoparser/opentelemetry/stream/streamparser.go#L253)
This commit fixes it by making a copy of Metadata entries into the `PushCtx`-owned
`WriteRequest.Metadata` buffer before calling `remotewrite.TryPush`.
The same approach is already used by the prom remote write handler:
[promremotewrite/request_handler.go#L78](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/promremotewrite/request_handler.go#L78)
```
for i := range mms {
mm := &mms[i]
mmsDst = append(mmsDst, prompb.MetricMetadata{
MetricFamilyName: mm.MetricFamilyName,
Help: mm.Help,
Type: mm.Type,
Unit: mm.Unit,
AccountID: mm.AccountID,
ProjectID: mm.ProjectID,
})
}
ctx.WriteRequest.Metadata = mmsDst
metadataTotal = len(mms)
```
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11238
Triage Assessment
Vulnerability Type: Race condition
Confidence: HIGH
Reasoning:
The commit resolves a data race in the OpenTelemetry metadata handling by copying metadata into a dedicated buffer before pushing, preventing concurrent modification of parser-owned backing arrays. While primarily a correctness/concurrency bug, data races can lead to crashes, inconsistent state, or potential exploitable conditions in some scenarios, which constitutes a security-relevant memory safety fix.
Verification Assessment
Vulnerability Type: Data race
Confidence: HIGH
Affected Versions: <=1.139.0
Code Diff
diff --git a/app/vmagent/opentelemetry/request_handler.go b/app/vmagent/opentelemetry/request_handler.go
index 63db6af7249e4..90b95b987668e 100644
--- a/app/vmagent/opentelemetry/request_handler.go
+++ b/app/vmagent/opentelemetry/request_handler.go
@@ -63,6 +63,7 @@ func insertRows(at *auth.Token, tss []prompb.TimeSeries, mms []prompb.MetricMeta
rowsTotal := 0
tssDst := ctx.WriteRequest.Timeseries[:0]
+ mmsDst := ctx.WriteRequest.Metadata[:0]
labels := ctx.Labels[:0]
samples := ctx.Samples[:0]
for i := range tss {
@@ -82,7 +83,19 @@ func insertRows(at *auth.Token, tss []prompb.TimeSeries, mms []prompb.MetricMeta
var metadataTotal int
if prommetadata.IsEnabled() {
- ctx.WriteRequest.Metadata = mms
+ for i := range mms {
+ mm := &mms[i]
+ mmsDst = append(mmsDst, prompb.MetricMetadata{
+ MetricFamilyName: mm.MetricFamilyName,
+ Help: mm.Help,
+ Type: mm.Type,
+ Unit: mm.Unit,
+
+ AccountID: mm.AccountID,
+ ProjectID: mm.ProjectID,
+ })
+ }
+ ctx.WriteRequest.Metadata = mmsDst
metadataTotal = len(mms)
}