How to Use DTrace on FreeBSD for Live Kernel and Application Tracing

DTrace can answer questions about a running FreeBSD kernel and its processes without rebuilding the application under investigation. That does not make every probe free or safe. A useful production workflow begins with a precise question, confirms that the required provider exists on this host, limits the target, aggregates in the kernel, and stops as soon as enough evidence has been collected.

FreeBSD’s implementation differs from illumos and other DTrace systems. Provider names and argument layouts are not automatically portable, and the FreeBSD Handbook describes userland DTrace as experimental. Test a script against the same FreeBSD release and architecture before treating it as a routine production diagnostic.

Establish a baseline and load support

Record the exact environment before tracing:

freebsd-version -kru
uname -m
kldstat | grep -i dtrace

Modern GENERIC kernels include DTrace support and running dtrace can load the needed modules automatically. The explicit, documented method remains:

kldload dtraceall

If this fails on a custom kernel, compare its configuration with the options documented in the Handbook rather than blindly rebuilding it. KDTRACE_HOOKS, CTF information, and architecture-specific support affect which probes work. A module loading successfully does not guarantee that a copied script’s providers exist.

Only root may use DTrace on FreeBSD because system-wide tracing can expose other users’ data and kernel activity. Do not loosen /dev/dtrace permissions as a shortcut. Run reviewed scripts through a controlled administrative path and protect their output like other diagnostic evidence.

Inventory probes before writing the script

List providers and narrow the inventory instead of reading tens of thousands of rows:

dtrace -l | awk 'NR > 1 {print $2}' | sort -u
dtrace -l -P syscall | head
dtrace -l -P profile | head

A DTrace probe description is provider:module:function:name; an empty field is a wildcard. syscall:::entry therefore selects every syscall entry probe, while a fully named FBT probe is much narrower. Check the local output of dtrace -l whenever a script says “probe description does not match any probes.”

Prefer stable, semantic providers such as syscall, profile, or documented SDT providers. FBT exposes kernel function boundaries, whose names and arguments are implementation details that can change between releases. It is powerful for a one-off investigation, but a saved FBT script needs revalidation after every kernel update.

Count activity before printing events

Start with an aggregation that answers which executables are generating syscalls:

dtrace -q -n 'syscall:::entry { @[execname] = count(); }'

Let it run for a bounded interval, then press Control-C. count() updates an in-kernel aggregation, so the terminal receives a summary rather than one line per event. Names in execname are short process names and are not unique identities; use the result to choose the next target, not to attribute activity conclusively.

For a known PID, attach it as the DTrace target and aggregate syscall names:

dtrace -q -p 1234 -n 'syscall:::entry /pid == $target/ { @[probefunc] = count(); }'

Replace 1234 with the verified PID. $target is supplied by -p, avoiding a broad execname predicate that might include unrelated workers with the same name. A syscall entry shows an attempt, not its result. To investigate failures, pair matching entry and return probes carefully and consult the local provider’s argument definitions.

Profile CPU with portable stack aggregation

The profile provider samples the code running at a specified frequency. A portable first pass for one process aggregates user stacks rather than reading an amd64-only register such as R_RIP:

dtrace -q -p 1234 -n 'profile-97 /pid == $target/ { @[ustack()] = count(); }'

Ninety-seven samples per second is a sensible starting point, not a universal setting. Increase frequency only when the shorter sampling interval is necessary and measured overhead is acceptable. Stack symbols may be incomplete for stripped binaries, JIT runtimes, or code without usable unwind information; unresolved addresses do not prove that DTrace failed.

Sampling tells where CPU time was observed. It does not explain off-CPU waits. If a process is idle because it is blocked on disk, a lock, or the network, use syscall, scheduler, VFS, or application-specific probes that match that hypothesis instead of interpreting an empty CPU profile as health.

Use FBT and raw output sparingly

Confirm an exact kernel probe exists before enabling it:

dtrace -l -n 'fbt::vfs_mount:entry'
dtrace -q -n 'fbt::vfs_mount:entry { @[stack()] = count(); }'

Kernel stacks provide more context than printing “mount attempt” repeatedly. Still, FBT can fire extremely often, and wildcarding a hot subsystem can create significant overhead. Never begin with fbt:::entry on a production server. Avoid DTrace destructive actions unless a dedicated test system and an explicit objective require them.

Printing every event also increases output cost and can reveal pathnames, arguments, or other sensitive material. If individual events are essential, filter by PID, user ID, function, or another strong predicate, write to a protected file, establish a short capture window, and document how the output will be removed.

Bound the experiment and watch the observer

Define a duration and exit condition before starting. For recurring investigations, save a reviewed .d script and use D language BEGIN and timed profile probes to print context and stop predictably instead of relying on an operator to remember Control-C. Keep the script under change control because a small predicate change can turn a process-specific capture into a system-wide one.

Observe system health while DTrace runs: CPU usage, scheduler latency, storage pressure from output, service error rate, and DTrace warnings. Compare a short baseline interval without tracing to the capture interval. Absence of an obvious slowdown does not prove zero observer effect, while a high event count does not by itself prove the traced activity is the bottleneck.

Speculative tracing is useful when entry data must be retained only if a later condition occurs, but it adds state and complexity. Begin with aggregations and exact predicates. If a script uses speculation, destructive actions, or broad kernel wildcards, require peer review and a staging run. Stop immediately when output volume, dropped records, or application behavior exceeds the predeclared limit.

Validate conclusions and leave no tracer behind

Reproduce a known action while tracing: run one test request, open one controlled file, or invoke one expected syscall. Confirm the positive event appears, then perform an unrelated action and confirm the predicate excludes it. Compare DTrace’s observation with service logs, procstat, sockstat, or another independent source before changing production configuration.

Record the command or saved script, start and stop times, host, kernel version, target PID, workload, and any dropped-data warning. A count without its observation interval is not a rate, and a stack distribution collected during an unrepresentative workload is not a general performance profile.

Terminate the DTrace process cleanly, verify it is gone, and remove sensitive captures according to policy:

pgrep -laf dtrace

DTrace is most valuable when it turns a narrow hypothesis into evidence with controlled overhead. It is not a permanent “trace everything” mode and should not replace application metrics, structured logs, or repeatable benchmarks.

Related:

Sources: