Node tuning and monitoring

Although Nitro is a Go application, it can use significantly more memory than Go's runtime reports. Nitro relies on multiple allocators: the Go garbage-collected heap, CGO (Go's mechanism for calling C code) allocations via calloc, and direct mmap system calls, each with its own accounting. Understanding where memory lives, which configuration knobs control it, how to tune validators, and how to monitor node health is essential for stable operation.

Memory allocators in Nitro

Nitro's total resident memory (RSS) is the sum of four distinct categories:

Allocator	What uses it	Visible in Go memstats?
Go heap	State trie (dirty), transaction processing, goroutine stacks, general application data	Yes
calloc	Pebble block cache, Pebble memtables, Stylus WASM cache	No
mmap	fastcache (trie-clean and snapshot caches)	No
glibc malloc arenas	Per-thread arena overhead for CGO allocations	No

Only the Go heap is subject to Go's garbage collector and GOMEMLIMIT. The CGO and mmap allocations are invisible to Go's runtime. They don't appear in runtime.MemStats or standard Go memory profiles, but they still consume container memory and count toward your memory limit.

Go heap

The Go runtime manages its own heap for all pure-Go allocations. Key consumers include:

• Contract code cache: An LRU cache of contract bytecode, hardcoded at 256 MB. Isn't configurable.
• Stylus LRU cache: Compiled Stylus WASM modules cached on the Go heap. Default capacity is 256 MB, configurable via --execution.caching.stylus-lru-cache-capacity.
• fastcache index maps: Although fastcache stores its data via mmap, each instance maintains a Go-side index (bucket maps of uint64 to uint64). These index maps can consume hundreds of MB on the Go heap.
• Goroutine stacks, block/receipt caches, and GC overhead: Goroutine stacks, recently accessed blocks/receipts, and Go's own GC metadata collectively add further pressure.

Go reports its total memory usage via runtime.MemStats.Sys, which includes the heap, stack space, and GC metadata. This is the portion of memory that GOMEMLIMIT governs.

CGO allocations (Pebble and Stylus)

Nitro's on-disk database, Pebble, allocates its block cache and memtables through CGO calloc() calls (see pebble/internal/manual/manual.go in the source). These allocations go through the C memory allocator and are out of scope for Go's memory tracking.

Pebble block cache is the largest CGO consumer. It caches frequently read database blocks in memory to avoid disk I/O.

Pebble memtables buffer recent writes before they are flushed to disk.

Stylus WASM cache stores compiled WebAssembly modules for Stylus smart contracts. Rust allocates this cache (invoked through CGO).

Raw mmap allocations (fastcache)

Two caches use fastcache, a library that allocates memory via direct mmap system calls, bypassing both Go's allocator and CGO:

• Trie-clean cache: Caches unchanged state trie nodes.
• Snapshot cache: Caches state snapshot data for fast reads.

Because fastcache uses raw mmap, this memory doesn't appear in Go's memstats or standard profiling tools. You can only see it by inspecting /proc/<pid>/smaps at the OS level. Each fastcache instance allocates memory in 64 MB chunks, making these regions identifiable when analyzing process memory maps.

glibc malloc arenas

When Nitro makes CGO calls (for Pebble, Stylus, etc.), the resulting C-side allocations go through the system's default C memory allocator: glibc malloc. Unlike Go's garbage-collected heap, malloc manages memory by requesting large regions from the OS and subdividing them to satisfy individual allocation requests. Freed memory is returned to the allocator's internal free lists rather than immediately back to the OS, so the process's RSS can remain elevated even after allocations are freed.

To handle concurrent allocations efficiently, glibc malloc uses arenas, which are independent memory pools, each with its own lock. When a thread allocates memory, it picks an arena, reducing contention compared to a single global lock. By default, glibc creates up to 8 x CPU_count arenas, each reserving a 64 MB region. The worst-case overhead for arenas is:

Arena overhead = 8 x CPU_count x 64 MB

In containerized environments, glibc detects the underlying host CPU count (not the container's CPU requests), which often results in far more arenas than needed. As the process runs and more threads make CGO calls, glibc creates and retains new arenas, causing RSS to drift upward over days or weeks even though no individual allocation is leaking.

This can be controlled with the MALLOC_ARENA_MAX environment variable:

MALLOC_ARENA_MAX=2

Setting MALLOC_ARENA_MAX=2 caps glibc to two arenas, reducing worst-case arena overhead from gigabytes to ~128 MB. In testing, this eliminated the slow memory growth with no measurable performance impact on RPC throughput.

Without MALLOC_ARENA_MAX, a Nitro node on a large host can accumulate gigabytes of arena overhead that appears as a "memory leak" because RSS grows steadily while Go reports stable usage. This is the most common cause of unexplained memory growth in long-running Nitro nodes.

Thread stacks

Nitro spawns native threads for CGO operations (Pebble, compression libraries) and Stylus execution.

Calculating GOMEMLIMIT

GOMEMLIMIT is an environment variable that sets a soft memory limit for the Go runtime. When set, Go's garbage collector (GC) runs more aggressively as heap usage approaches the limit, helping to keep total Go memory usage below the target. Without it, the GC relies solely on the GOGC environment variable (which defaults to 100, meaning the GC triggers when the heap doubles in size since the last collection) and has no awareness of an absolute memory ceiling.

For GOMEMLIMIT to work correctly in a containerized environment, you must reserve enough headroom for all the non-Go memory that competes for the container's memory limit.

Non-Go memory budget

Sum all memory that lives outside the Go heap:

Non-Go Memory =
    Pebble block cache + memtables  (CGO)
  + fastcache (trie-clean, snapshot) (mmap)
  + Stylus WASM cache                (Rust)
  + malloc arena overhead            (glibc arenas)
  + ~300 MB                          (Thread stacks, varies by workload)

With MALLOC_ARENA_MAX=2, arena overhead is ~128 MB. Without it, arena overhead can grow to several gigabytes depending on host CPU count. See glibc malloc arenas above.

Formula

GOMEMLIMIT = Container_Memory_Limit - Non_Go_Memory - Safety_Margin

You should use a safety margin of 300-500 MB to account for allocator overhead, transient allocations, and kernel page cache.

If GOMEMLIMIT is set too high (not accounting for non-Go memory), the Go garbage collector defers collection, expecting more room than actually exists. The OS then OOM-kills the process when total RSS (Go heap plus all non-Go allocations) exceeds the container limit.

Helm chart shortcut

The community Helm chart calculates GOMEMLIMIT automatically using a multiplier of 0.9 against the container memory limit. It subtracts a non-Go memory allowance (covering Pebble caches, fastcache, Stylus WASM cache, and malloc arenas) before applying the multiplier. If you use the Helm chart with default values, GOMEMLIMIT is set for you.

Validator tuning

Validators perform CPU-intensive WASM execution to verify blocks. Their memory and CPU profiles differ from the main Nitro node.

GOMEMLIMIT for validators

Validators use a lower GOMEMLIMIT multiplier of 0.75 (compared to 0.9 for the main node) because WASM execution creates more transient Go heap allocations that need GC headroom. For a 16 GB validator container:

GOMEMLIMIT ≈ 16,384 MB × 0.75 = ~12,288 MB ≈ 12 GB

Memory free limits

Two multipliers control how much free memory Nitro reserves before throttling operations:

• resourceMgmtMemFreeLimit: Multiplier 0.05 (5% of container memory). When free memory drops below this threshold, the node throttles incoming RPC requests to prevent OOM conditions.
• blockValidatorMemFreeLimit: Multiplier 0.05 (5% of container memory). When free memory drops below this threshold, the validator pauses block validation until memory is reclaimed.

GOMAXPROCS

GOMAXPROCS controls how many OS threads Go uses for goroutine scheduling. For validators, the Helm chart defaults to a multiplier of 2 against the container CPU limit (for example, 4 CPUs yields GOMAXPROCS=8). This higher-than-usual setting helps because validators frequently block on CGO calls during WASM execution, and additional Go threads keep non-CGO goroutines progressing.

For the main Nitro node, the Helm chart does not override GOMAXPROCS, letting Go's runtime auto-detect from the container CPU limit.

Metrics and monitoring

Nitro exposes Prometheus-compatible metrics through a dedicated metrics server.

Enable the metrics server

Pass the --metrics flag when starting the node. Configure the server with these flags:

Flag	Default	Description
--metrics-server.addr	127.0.0.1	Listen address for the metrics server
--metrics-server.port	6070	Listen port for the metrics server
--metrics-server.update-interval	3s	How often internal metrics are refreshed

Example:

nitro --metrics \
  --metrics-server.addr 0.0.0.0 \
  --metrics-server.port 6070

Prometheus scrape endpoint

Once metrics are enabled, Nitro exposes a Prometheus-compatible endpoint at:

http://<metrics-server.addr>:<metrics-server.port>/debug/metrics/prometheus

The same server also exposes pprof profiling endpoints at /debug/pprof/ for CPU and memory profiling.

Key metrics to monitor

Metric	What it tells you
container_memory_rss	Actual RSS of the container. Compare against Go-reported heap to understand non-Go memory usage.
arb_feed_backlog_messages	Number of messages in the feed backlog. A growing backlog indicates the node is falling behind the sequencer feed.
Batch poster backlog	For batch poster nodes, tracks how many batches are pending submission to the parent chain.

Tip

Always alert on container_memory_rss, not Go heap metrics. As described in Memory allocators in Nitro, most of Nitro's memory is invisible to Go's runtime.

Kubernetes ServiceMonitor

If you run Nitro on Kubernetes with the Prometheus Operator, configure a ServiceMonitor to scrape the metrics endpoint:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nitro
spec:
  selector:
    matchLabels:
      app: nitro
  endpoints:
    - port: metrics
      path: /debug/metrics/prometheus
      interval: 5s

Health check patterns

Nitro components expose health information through different mechanisms depending on the binary.

Main Nitro node

The main nitro binary does not expose a dedicated /health endpoint. Node health is inferred from RPC availability: if the HTTP RPC port (default 8547) responds with HTTP 200 to a valid JSON-RPC request, the node is healthy.

Example liveness check:

curl -sf -X POST http://localhost:8547 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'

Auxiliary components

• Transaction filterer: Exposes a /liveness endpoint for health checks.
• DA server: Exposes a /health endpoint.

Kubernetes probe strategy

Initial sync can take days or weeks depending on chain history. The community Helm chart uses an aggressive startup probe to allow time for initial sync without marking the pod as failed:

startupProbe:
  httpGet:
    path: /
    port: 8547
  failureThreshold: 2419200
  periodSeconds: 1

This configuration allows up to 28 days (2,419,200 seconds) for the node to become responsive. Once the startup probe succeeds, standard liveness and readiness probes take over.

Tuning recommendations

General rules

1. Set MALLOC_ARENA_MAX=2: This is the single most impactful change for containerized nodes. Without it, glibc can waste gigabytes on arena overhead, causing RSS to drift upward over days. Set this environment variable on every Nitro container.

2. Start from the formula: Calculate GOMEMLIMIT using the formula above. Do not set it to the container memory limit.

3. Monitor RSS, not just Go heap: Set container memory alerts based on actual RSS (container_memory_rss in Prometheus / cAdvisor), not Go-reported memory.

4. Non-Go memory is bounded: With MALLOC_ARENA_MAX set, if RSS is stable and predictable, the node is behaving correctly. The memory is simply allocated outside Go's visibility.

Example configurations

Full node

MALLOC_ARENA_MAX=2
GOMEMLIMIT=<computed from formula>

nitro \
  --metrics \
  --metrics-server.addr 0.0.0.0

Validator

Validators need more CPU headroom. Use the lower GOMEMLIMIT multiplier (0.75):

# For a 16 GB container:
# GOMEMLIMIT = 16,384 × 0.75 ≈ 12,288 MB

MALLOC_ARENA_MAX=2
GOMEMLIMIT=12288MiB
GOMAXPROCS=4

nitro \
  --metrics \
  --metrics-server.addr 0.0.0.0