Log storms¶
A log storm is a burst of near-identical events from one logger: an error path stuck in a loop, a retry that never stops. Storm detection notices them, groups them, and keeps one of the events in full for you to read. In 1.0 it only reports: nothing is suppressed.
WildFly only in 1.0
Storm detection needs java.util.logging or JBoss LogManager.
The output on this page is from WildFly 26.1.3.
Turn it on¶
Storm detection starts off, because it costs a little on every log call. Turn it on for a while:
logctl enable storms for 30m
Storm detection is now enabled (was disabled) until 2026-10-10T03:11:21.486477057Z, then disabled. Measuring from 2026-10-10T02:41:21.486477057Z.
Without a tier it stays on until the JVM stops; sticky keeps it on across restarts. To have it on
from startup, start the agent with --storm-detection=on (see
Agent options). logctl disable storms turns it off.
See the storms¶
Above: a shop's console is scrolling too fast to read. logctl enable storms turns detection on,
logctl storms names the message that is bursting, and a drop rule keeps it out of the log.
logctl storms
During a storm:
Storm detection is enabled until 2026-10-10T03:11:21.486477057Z, then disabled.
[ONGOING] io.undertow.request (no exception) "Matched default handler path %s"
started 2026-10-10T02:41:22.667701316Z (4s ago) — 1,500 events — ~21,598/min
sample (event #1,000, when the storm was detected):
2026-10-10T02:41:24.990134470Z DEBUG [io.undertow.request] Matched default handler path /x997
1 storm tracked — 1 ongoing, 0 ended. measured since 2026-10-10T02:41:21.497247862Z.
and after it, once a minute has passed quietly:
[ENDED] io.undertow.request (no exception) "Matched default handler path %s"
2026-10-10T02:41:22.667701316Z–2026-10-10T02:41:26.038228375Z (3s) — 1,500 events — ended 1m9s ago
Each storm shows its logger, its exception type if there is one, and the message with the parts
that vary replaced by a placeholder, so 1,500 requests for different paths count as one storm. A
message whose logger gave a format string, like the one above, keeps its own placeholder (%s);
in a plain message, digits show as <n>, as in the film above. While it is
going on, you also see its rate and a sample: the event that made it a storm, in full, with its
message as logged and its stack trace if it has one. The sample is not the storm's first event; the
event number says how far into the storm it came (the 1,000th here, the threshold).
By default a storm is 1,000 near-identical events within 10 seconds. --limit shows only the worst
few.
Then¶
- Find the cause with the sample, and fix it.
- Meanwhile, keep the storm out of the log with a
droprule matching its message, or shorten its traces with atrimrule. See Silence noise. logctl topshows how much the storm wrote. See Find where the disk goes.