Skip to content

Now Showing

Short films of LogAperture fixing real problems, recorded with asciinema.

Each film runs against a real WildFly server with a small demo web shop deployed, and takes less than 90 seconds. The server's console is on top, and logctl runs below. Pause a film to copy a command straight out of it. After each film, every command in it is taken apart, with links to its reference and to the how-to that covers it.

The programme

  1. Why don't I see my TRACE?: a logger is raised, and nothing appears.
  2. Debug one package for a while: a change that cleans itself up.
  3. Something is flooding the log: find the source, and silence one exception.
  4. Stack traces are burying the log: keep every warning, but not its trace.
  5. What is bursting?: find the one message behind a scrolling console.
  6. Is my logging config going to hurt me?: what doctor reports, and the one fix logctl can make.
  7. Ship what you tuned: turn sticky changes into the file your product ships.

Why don't I see my TRACE?

A customer reports failed payments, so you turn on TRACE for the payment code, and the console shows nothing new. The logger is now at TRACE, but the console's own handler is still at INFO, and a handler drops anything below its own level. Reach for this whenever you raise a logger and the output you expected doesn't appear: logctl tells you which handler is in the way, and one more command lets it through.

Commands in this film

logctl set logger com.example.shop.payment TRACE for 15m: raise the payment code's level.

  • com.example.shop.payment: the logger. A package name covers every logger under it, present and future.
  • TRACE: the new level.
  • for 15m: the change reverts by itself after 15 minutes. Without a duration it lasts for 4h.

logctl warns that the CONSOLE handler is at INFO and will drop the TRACE records.

Reference: set logger · How-to: The console follows

logctl set handler CONSOLE AUTO: let the console handler follow the logger.

  • CONSOLE: the handler, by the name WildFly gives it. logctl list handlers lists the names.
  • AUTO: instead of a fixed level, the handler follows the lowest DEBUG or TRACE logger override, and goes back to its own level once none are left.

Reference: set handler · How-to: The console follows

logctl status: show everything LogAperture has changed, and when each change reverts. It changes nothing.

Reference: status · How-to: See what is changed

Debug one package for a while

You need DEBUG from the checkout code for one investigation. Turning it on is easy; the risk is forgetting to turn it off, and filling the disk with debug output for days. Reach for this whenever a change should only last as long as the investigation: give it a duration, and it puts itself back. The film uses 30 seconds so you can watch it happen; in real life it would be for 1h or so.

Commands in this film

logctl set logger com.example.shop.checkout DEBUG for 30s: raise the checkout code's level for a while.

  • com.example.shop.checkout: the logger, and every logger under it.
  • DEBUG: the new level.
  • for 30s: the change reverts by itself after 30 seconds. Durations take s, m, h and d.

Reference: set logger · How-to: A logger and everything under it

logctl set handler CONSOLE AUTO: let the console handler follow the logger, as in the first film. It goes back to its own level when the DEBUG override ends, so nothing is left to clean up.

  • CONSOLE: the handler.
  • AUTO: follow the lowest DEBUG or TRACE logger override.

Reference: set handler · How-to: The console follows

logctl list loggers com.example: show the loggers under com.example that have an override, with their levels and when each reverts.

  • com.example: a filter, the start of the logger names to show. Add --show-all to see every logger, not just the changed ones.

Reference: list loggers · How-to: From the start of the name

logctl status: after the 30 seconds, it shows nothing changed: the logger and the handler are both back where they started.

Reference: status · How-to: How a timed change ends

Something is flooding the log

Something is writing the same failure many times a second, and the console scrolls too fast to read. First you need to know which logger it is; then you want that one exception gone, without a restart, and without losing everything else the component logs. Reach for this when one noisy message drowns out the rest, or fills the disk.

Commands in this film

logctl top --limit 5: rank loggers by how much they have written, worst first, with each one's rate and how much of it is stack traces. It changes nothing.

  • --limit 5: show the five worst. The default is 10.

Reference: top · How-to: Which loggers write the most

logctl add rule drop com.example.shop.inventory.StockSync --throwable java.net.ConnectException --sample-full 1m: discard that exception's events, and only those.

  • drop: discard matching events. Every one is counted.
  • com.example.shop.inventory.StockSync: the logger the rule is attached to.
  • --throwable java.net.ConnectException: match only events carrying exactly this exception. Everything else StockSync logs is untouched.
  • --sample-full 1m: let one full event through every minute, so the log never goes completely quiet about it.

The rule lasts for 4h, since no duration was given.

Reference: add rule · How-to: Drop a message

logctl list rules: show the rules, with their ids and when they expire. HITS counts the events each rule has dropped.

Reference: list rules · How-to: See and change rules

Stack traces are burying the log

The shop's primary payment endpoint is down because of an environment problem, and every order logs a warning with the same ConnectException and a 28-line stack trace. The warning matters: you want to know each time it happens. The trace doesn't: it is the same root cause every time, and it buries everything else on the console. Reach for this when a known failure keeps printing a trace that tells you nothing new.

Commands in this film

logctl add rule trim com.example.shop.payment.PaymentGateway --throwable java.net.ConnectException --any-cause: keep every matching warning, but print its exception without a stack trace.

  • trim: keep the event, and shorten its stack trace. Each trimmed exception ends with a marker saying how many frames were cut.
  • com.example.shop.payment.PaymentGateway: the logger the rule is attached to.
  • --throwable java.net.ConnectException: match only events carrying this exception.
  • --any-cause: match it anywhere in the cause chain. Here the ConnectException is the cause of the shop's own PaymentException, which is how client libraries usually wrap it.

Without --frames, each exception is printed as one line: 2 lines per warning instead of 28. The rule lasts for 4h, since no duration was given.

Reference: add rule · How-to: Shorten a stack trace, Matching on the exception

logctl list rules: show the rules, with their ids and when they expire. HITS counts the events each rule has trimmed.

Reference: list rules · How-to: See and change rules

What is bursting?

The shop's console is scrolling too fast to read, and nothing in it says why. A retry loop is logging the same message hundreds of times a second, with only an attempt number changing, so no single line stands out. Storm detection groups events like that into one storm and keeps one in full. Reach for it when the log is too fast to read and you don't know which message is doing it.

Commands in this film

logctl enable storms: switch storm detection on. It starts off, because it costs a little on every log call; without a duration it stays on until the JVM stops.

Reference: storms · How-to: Turn it on

logctl storms: list the storms detected so far. Each shows its logger, its exception if it has one, and the message with the changing parts replaced by <n>, so thousands of events with different attempt numbers count as one storm. An ongoing storm also shows its rate and a sample: the event that made it a storm, in full.

Reference: storms · How-to: See the storms

logctl add rule drop com.example.shop.payment.RetryQueue --message-contains Retrying: keep the storm out of the log while the cause is fixed.

  • drop: discard the matching events, letting the first through and then one every 5 minutes, so the message is never far away.
  • com.example.shop.payment.RetryQueue: the logger the storm named.
  • --message-contains Retrying: match events whose message contains the text.

The rule lasts for 4h, since no duration was given.

Reference: add rule · How-to: Drop a message

Is my logging config going to hurt me?

Nothing has failed yet, but a logging configuration can quietly cost you a disk or a noisy log: a file with no size cap, a chatty framework logger left at DEBUG, a handler that flushes on every record. doctor reads the configuration and says which of those apply. It only reports, and where logctl can fix a finding it prints the exact command. Reach for it before an incident, or when you take over a server you didn't set up.

Commands in this film

logctl doctor: run the checks and print one line per finding, with a detail line and a suggested fix where there is one. It never changes anything.

  • The org.hibernate finding is the one logctl can fix, and it prints the command. Here the DEBUG was set at runtime, which reset undoes; a level the application's own configuration sets is changed there, or turned down with logctl set logger.
  • The FILE and CONSOLE findings are the server's own handler settings (no size cap, autoflush). doctor names them and leaves them to you: LogAperture doesn't change handler settings other than the level.

Reference: doctor, Doctor checks · How-to: What in the configuration will hurt

logctl reset logger org.hibernate: put the logger back to the level the application configured. It prints the level it went back to, with (baseline).

Reference: reset · How-to: Reset

Ship what you tuned

You ship a product on a JVM, and you have tuned its logging live in a test environment until it is right. Your customers shouldn't have to repeat that. export vendor-defaults writes every sticky change, on top of the file the JVM started with, as a vendor defaults file you ship with the product. Reach for it when the settings are right and you want customers to start from them.

Commands in this film

logctl set logger com.example.shop.payment WARN sticky: change a logger's level and keep it across restarts. Only sticky changes are exported; session and for changes are left out.

Reference: set logger · How-to: Raise a log level

logctl add rule drop com.example.shop.inventory.StockSync --message-contains synced --below WARN sticky: drop a repeating message below WARN, kept across restarts.

  • drop: discard the matching events, letting the first through and then one every 5 minutes.
  • --message-contains synced: match events whose message contains the text.
  • --below WARN: only events below WARN, so a warning or an error from the logger is never dropped.
  • sticky: keep it across restarts, which is what puts it in the export.

Reference: add rule · How-to: Drop a message

logctl export vendor-defaults: print the vendor defaults file: the file the JVM started with, plus every sticky change. Each entry's stateId ties it to the change it came from, so when the agent starts with the new file, the file takes those changes over and each setting is active once. Add --out <file> to write it to a file. Ship the file with the product and start the agent with --vendor-defaults=<file>.

Reference: export vendor-defaults · How-to: Making the file: tune, then export