Concurrency leak / race condition leading to potential deadlock in write concurrency limiter
Description
The commit fixes a race/resource-leak in the write concurrency limiter. Previously, Read() could decrease the available concurrency tokens (via DecConcurrency()) and, if IncConcurrency() failed, the token would not be re-acquired, potentially leaking a slot permanently. This could drain the concurrency limiter channel over time, eventually causing ingestion to deadlock when no slots remain. The patch adds per-reader tracking of whether a token has been obtained (increasedConcurrency) and ensures tokens are released correctly when the reader is returned.
Proof of Concept
PoC not guaranteed to be executable in all environments due to internal package APIs. Below is a conceptual reproduction outline to validate the race/leak prior to the fix. Adapt to the repository's actual test harness and API surface (GetReader, PutReader, DecConcurrency, IncConcurrency, and the underlying limiter implementation).
1) Setup a limiter with maxConcurrentInserts = 1 (or a small value) and a blocking underlying reader.
2) Obtain a Reader via GetReader(underlyingReader).
3) In a separate goroutine, trigger Read() on the Reader. This will call DecConcurrency() before I/O and IncConcurrency() after I/O.
4) Force IncConcurrency() to fail (e.g., by closing the concurrency-limiter channel or by injecting a condition that makes IncConcurrency() return an error).
5) In the main thread, ensure PutReader(reader) is called after Read() returns with the error from IncConcurrency().
6) Observe that the reduced concurrency slot from the initial DecConcurrency() is leaked (the channel becomes drained). Subsequent ingestion attempts block as no slots are available.
7) Apply the patch and repeat steps 2–6. With the fix, the Reader tracks tokens via increasedConcurrency and ensures that a token is released appropriately even if IncConcurrency() fails, preventing permanent leakage and deadlock.
Note: Since the exact types and function signatures depend on the current codebase, you may need to adapt the snippet to your test harness. The key exploit path is the leakage of a concurrency slot due to IncConcurrency() failing after a DecConcurrency() call, and verifying that the patch prevents such leakage by tracking token ownership per Reader.
Commit Details
Author: Alexander Frolov
Date: 2026-04-10 17:35 UTC
Message:
lib/writeconcurrencylimiter: prevent deadlock at IncConcurrency
Previously (*writeconcurrencylimiter.Reader).Read() could permanently leak concurrency tokens from the -maxConcurrentInserts semaphore.
Consider the following example:
* GetReader() acquires a token, then PutReader() unconditionally releases it.
* Read() calls DecConcurrency() before the underlying I/O and IncConcurrency() after it. If IncConcurrency() returns an error, Read() returns without holding a token.
* Each such failure permanently removes one slot from the concurrencyLimitCh semaphore. Slots leak one by one until the channel is fully drained, at which point DecConcurrency() blocks forever, deadlocking ingestion on vmstorage.
This commit adds tracking for obtained tokens to the reader. Which prevents possible tokens leakage.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10784
Triage Assessment
Vulnerability Type: Race condition
Confidence: MEDIUM
Reasoning:
The change fixes a race condition that could leak tokens from the concurrency limiter, leading to deadlocks and potential denial of service in ingestion. While not a traditional vulnerability like RCE/XSS, it addresses a security-relevant reliability issue in concurrency handling.
Verification Assessment
Vulnerability Type: Concurrency leak / race condition leading to potential deadlock in write concurrency limiter
Confidence: MEDIUM
Affected Versions: < 1.139.0 (versions prior to the fix in lib/writeconcurrencylimiter/concurrencylimiter.go)
Code Diff
diff --git a/lib/writeconcurrencylimiter/concurrencylimiter.go b/lib/writeconcurrencylimiter/concurrencylimiter.go
index 050b3cbe8288b..a5b2c9a677a89 100644
--- a/lib/writeconcurrencylimiter/concurrencylimiter.go
+++ b/lib/writeconcurrencylimiter/concurrencylimiter.go
@@ -32,7 +32,8 @@ var (
//
// The Reader must be obtained via GetReader() call.
type Reader struct {
- r io.Reader
+ r io.Reader
+ increasedConcurrency bool
}
// GetReader returns the Reader for r.
@@ -49,6 +50,7 @@ func GetReader(r io.Reader) (*Reader, error) {
}
rr := v.(*Reader)
rr.r = r
+ rr.increasedConcurrency = true
return rr, nil
}
@@ -58,9 +60,11 @@ func GetReader(r io.Reader) (*Reader, error) {
// It decreases the concurrency.
func PutReader(r *Reader) {
r.r = nil
+ if r.increasedConcurrency {
+ DecConcurrency()
+ r.increasedConcurrency = false
+ }
readerPool.Put(r)
-
- DecConcurrency()
}
var readerPool sync.Pool
@@ -68,12 +72,14 @@ var readerPool sync.Pool
// Read implements io.Reader.
func (r *Reader) Read(p []byte) (int, error) {
DecConcurrency()
+ r.increasedConcurrency = false
n, err := r.r.Read(p)
if errC := IncConcurrency(); errC != nil {
return n, errC
}
+ r.increasedConcurrency = true
if errors.Is(err, io.ErrUnexpectedEOF) {
// See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8704