Now Showing¶
Short films of LogAperture fixing real problems, recorded with asciinema.
Each film runs against a real WildFly server with a small demo web shop deployed, and takes less
than 90 seconds. The server's console is on top, and logctl runs below. Pause a film to copy a
command straight out of it. After each film, every command in it is taken apart, with links to
its reference and to the how-to that covers it.
The programme
- Why don't I see my TRACE?: a logger is raised, and nothing appears.
- Debug one package for a while: a change that cleans itself up.
- Something is flooding the log: find the source, and silence one exception.
- Stack traces are burying the log: keep every warning, but not its trace.
- What is bursting?: find the one message behind a scrolling console.
- Is my logging config going to hurt me?: what
doctorreports, and the one fixlogctlcan make. - Ship what you tuned: turn sticky changes into the file your product ships.
Why don't I see my TRACE?¶
A customer reports failed payments, so you turn on TRACE for the payment code, and the console
shows nothing new. The logger is now at TRACE, but the console's own handler is still at
INFO, and a handler drops anything below its own level. Reach for this whenever you raise a
logger and the output you expected doesn't appear: logctl tells you which handler is in the way,
and one more command lets it through.
Commands in this film¶
logctl set logger com.example.shop.payment TRACE for 15m: raise the payment code's level.
com.example.shop.payment: the logger. A package name covers every logger under it, present and future.TRACE: the new level.for 15m: the change reverts by itself after 15 minutes. Without a duration it lastsfor 4h.
logctl warns that the CONSOLE handler is at INFO and will drop the TRACE records.
Reference: set logger · How-to:
The console follows
logctl set handler CONSOLE AUTO: let the console handler follow the logger.
CONSOLE: the handler, by the name WildFly gives it.logctl list handlerslists the names.AUTO: instead of a fixed level, the handler follows the lowestDEBUGorTRACElogger override, and goes back to its own level once none are left.
Reference: set handler · How-to:
The console follows
logctl status: show everything LogAperture has changed, and when each change reverts.
It changes nothing.
Reference: status · How-to:
See what is changed
Debug one package for a while¶
You need DEBUG from the checkout code for one investigation. Turning it on is easy; the risk is
forgetting to turn it off, and filling the disk with debug output for days. Reach for this
whenever a change should only last as long as the investigation: give it a duration, and it puts
itself back. The film uses 30 seconds so you can watch it happen; in real life it would be
for 1h or so.
Commands in this film¶
logctl set logger com.example.shop.checkout DEBUG for 30s: raise the checkout code's level
for a while.
com.example.shop.checkout: the logger, and every logger under it.DEBUG: the new level.for 30s: the change reverts by itself after 30 seconds. Durations takes,m,handd.
Reference: set logger · How-to:
A logger and everything under it
logctl set handler CONSOLE AUTO: let the console handler follow the logger, as in the
first film. It goes back to its own level when the DEBUG override
ends, so nothing is left to clean up.
CONSOLE: the handler.AUTO: follow the lowestDEBUGorTRACElogger override.
Reference: set handler · How-to:
The console follows
logctl list loggers com.example: show the loggers under com.example that have an override,
with their levels and when each reverts.
com.example: a filter, the start of the logger names to show. Add--show-allto see every logger, not just the changed ones.
Reference: list loggers · How-to:
From the start of the name
logctl status: after the 30 seconds, it shows nothing changed: the logger and the handler are
both back where they started.
Reference: status · How-to:
How a timed change ends
Something is flooding the log¶
Something is writing the same failure many times a second, and the console scrolls too fast to read. First you need to know which logger it is; then you want that one exception gone, without a restart, and without losing everything else the component logs. Reach for this when one noisy message drowns out the rest, or fills the disk.
Commands in this film¶
logctl top --limit 5: rank loggers by how much they have written, worst first, with each
one's rate and how much of it is stack traces. It changes nothing.
--limit 5: show the five worst. The default is 10.
Reference: top · How-to:
Which loggers write the most
logctl add rule drop com.example.shop.inventory.StockSync --throwable java.net.ConnectException --sample-full 1m:
discard that exception's events, and only those.
drop: discard matching events. Every one is counted.com.example.shop.inventory.StockSync: the logger the rule is attached to.--throwable java.net.ConnectException: match only events carrying exactly this exception. Everything elseStockSynclogs is untouched.--sample-full 1m: let one full event through every minute, so the log never goes completely quiet about it.
The rule lasts for 4h, since no duration was given.
Reference: add rule · How-to:
Drop a message
logctl list rules: show the rules, with their ids and when they expire. HITS counts the
events each rule has dropped.
Reference: list rules · How-to:
See and change rules
Stack traces are burying the log¶
The shop's primary payment endpoint is down because of an environment problem, and every order
logs a warning with the same ConnectException and a 28-line stack trace. The warning matters:
you want to know each time it happens. The trace doesn't: it is the same root cause every time,
and it buries everything else on the console. Reach for this when a known failure keeps printing
a trace that tells you nothing new.
Commands in this film¶
logctl add rule trim com.example.shop.payment.PaymentGateway --throwable java.net.ConnectException --any-cause:
keep every matching warning, but print its exception without a stack trace.
trim: keep the event, and shorten its stack trace. Each trimmed exception ends with a marker saying how many frames were cut.com.example.shop.payment.PaymentGateway: the logger the rule is attached to.--throwable java.net.ConnectException: match only events carrying this exception.--any-cause: match it anywhere in the cause chain. Here theConnectExceptionis the cause of the shop's ownPaymentException, which is how client libraries usually wrap it.
Without --frames, each exception is printed as one line: 2 lines per warning instead of 28. The
rule lasts for 4h, since no duration was given.
Reference: add rule · How-to:
Shorten a stack trace,
Matching on the exception
logctl list rules: show the rules, with their ids and when they expire. HITS counts the
events each rule has trimmed.
Reference: list rules · How-to:
See and change rules
What is bursting?¶
The shop's console is scrolling too fast to read, and nothing in it says why. A retry loop is logging the same message hundreds of times a second, with only an attempt number changing, so no single line stands out. Storm detection groups events like that into one storm and keeps one in full. Reach for it when the log is too fast to read and you don't know which message is doing it.
Commands in this film¶
logctl enable storms: switch storm detection on. It starts off, because it costs a little on
every log call; without a duration it stays on until the JVM stops.
Reference: storms · How-to: Turn it on
logctl storms: list the storms detected so far. Each shows its logger, its exception if it
has one, and the message with the changing parts replaced by <n>, so thousands of events with
different attempt numbers count as one storm. An ongoing storm also shows its rate and a sample: the
event that made it a storm, in full.
Reference: storms · How-to: See the storms
logctl add rule drop com.example.shop.payment.RetryQueue --message-contains Retrying: keep
the storm out of the log while the cause is fixed.
drop: discard the matching events, letting the first through and then one every 5 minutes, so the message is never far away.com.example.shop.payment.RetryQueue: the logger the storm named.--message-contains Retrying: match events whose message contains the text.
The rule lasts for 4h, since no duration was given.
Reference: add rule · How-to:
Drop a message
Is my logging config going to hurt me?¶
Nothing has failed yet, but a logging configuration can quietly cost you a disk or a noisy log: a
file with no size cap, a chatty framework logger left at DEBUG, a handler that flushes on every
record. doctor reads the configuration and says which of those apply. It only reports, and where
logctl can fix a finding it prints the exact command. Reach for it before an incident, or when
you take over a server you didn't set up.
Commands in this film¶
logctl doctor: run the checks and print one line per finding, with a detail line and a
suggested fix where there is one. It never changes anything.
- The
org.hibernatefinding is the onelogctlcan fix, and it prints the command. Here theDEBUGwas set at runtime, whichresetundoes; a level the application's own configuration sets is changed there, or turned down withlogctl set logger. - The
FILEandCONSOLEfindings are the server's own handler settings (no size cap, autoflush).doctornames them and leaves them to you: LogAperture doesn't change handler settings other than the level.
Reference: doctor, Doctor checks ·
How-to: What in the configuration will hurt
logctl reset logger org.hibernate: put the logger back to the level the application
configured. It prints the level it went back to, with (baseline).
Reference: reset · How-to: Reset
Ship what you tuned¶
You ship a product on a JVM, and you have tuned its logging live in a test environment until it is
right. Your customers shouldn't have to repeat that. export vendor-defaults writes every sticky
change, on top of the file the JVM started with, as a vendor defaults file you ship with the
product. Reach for it when the settings are right and you want customers to start from them.
Commands in this film¶
logctl set logger com.example.shop.payment WARN sticky: change a logger's level and keep it
across restarts. Only sticky changes are exported; session and for changes are left out.
Reference: set logger · How-to:
Raise a log level
logctl add rule drop com.example.shop.inventory.StockSync --message-contains synced --below WARN sticky:
drop a repeating message below WARN, kept across restarts.
drop: discard the matching events, letting the first through and then one every 5 minutes.--message-contains synced: match events whose message contains the text.--below WARN: only events belowWARN, so a warning or an error from the logger is never dropped.sticky: keep it across restarts, which is what puts it in the export.
Reference: add rule · How-to:
Drop a message
logctl export vendor-defaults: print the vendor defaults file: the file the JVM started with,
plus every sticky change. Each entry's stateId ties it to the change it came from, so when the
agent starts with the new file, the file takes those changes over and each setting is active once.
Add --out <file> to write it to a file. Ship the file with the product and start the agent with
--vendor-defaults=<file>.
Reference: export vendor-defaults · How-to:
Making the file: tune, then export