Fixing Slow Boot Times Caused by Excessive rc.d Service Dependencies

A long pause between kernel startup and login can come from one blocking rc script, an initial filesystem check, DHCP or DNS timeout, unavailable remote mount, entropy wait, or an incorrect dependency chain. rcorder(8) tells FreeBSD which script comes before another; it does not prove the earlier daemon is enabled, ready, or healthy. Measure elapsed time first, then inspect the graph around the measured script.

Capture a boot timeline with boottrace

FreeBSD 14.0 and later include boottrace(4), with annotations in kernel boot, init, and rc scripts. It is compiled in but disabled by default. Enable it for a controlled diagnostic boot through /boot/loader.conf:

kern.boottrace.enabled="1"

After reboot, preserve its table:

sysctl kern.boottrace.enabled
sysctl kern.boottrace.log > /var/tmp/boottrace.log
less /var/tmp/boottrace.log

The log contains start/done events, elapsed deltas, CPU time, and block I/O. A large wall-time delta with little CPU time often indicates a timeout or I/O wait; high CPU time indicates real computation. The sample in boottrace(4) shows how a netif script can be isolated to a multi-second interval.

Remove the loader setting after the investigation if continuous tracing is unnecessary. On older FreeBSD releases, capture a serial console or timestamped boot messages and use service-specific logs. boot_verbose="YES" and boot -v increase kernel/device diagnostics but are not substitutes for per-script elapsed timing.

Inspect the dependency graph without executing scripts

Generate the normal ordering and a Graphviz representation:

rcorder -s nostart /etc/rc.d/* /usr/local/etc/rc.d/*
rcorder -g -s nostart /etc/rc.d/* /usr/local/etc/rc.d/* \
  > /var/tmp/rcorder.dot

Current rcorder can also use -p to place scripts that could execute simultaneously on the same output line. That is a graph-analysis format; it does not establish that /etc/rc actually started them in parallel. FreeBSD’s documented generic boot remains ordered rc-script execution, while particular subsystems such as jails have their own parallel-start option.

PROVIDE, REQUIRE, and BEFORE name ordering conditions, not runtime health dependencies. REQUIRE: NETWORKING puts a script after the provider of that milestone; it does not guarantee DNS works, a remote API answers, or every interface has a lease. A service performing an unbounded network check inside its start action can still hold boot.

Look for missing providers, circular dependencies, and a third-party script placed much later than its actual prerequisites justify. Compare with the matching rcorder(8) and rc.subr(8) contracts before changing anything.

Reproduce the slow script without replaying the entire boot

Inspect the script and its enable variables:

service -e
service slowservice rcvar
service -d slowservice status

Do not rerun /etc/rc on a live machine. For a daemon safe to restart in a maintenance window, time its actual action:

time service slowservice restart

Check the service’s logs simultaneously. A restart may differ from a cold start because filesystems, DNS, routes, and caches are already available, so a fast live restart does not disprove a boot-order race.

Common measured causes include DHCP waiting for an absent server, DNS lookups before resolvers are reachable, NFS initial mount retries, a daemon waiting synchronously for a remote database, and fsck after an unclean UFS shutdown. Fix the underlying dependency or timeout policy. Skipping fsck to make boot faster can mount a damaged filesystem and is not a performance optimization.

Use rc_debug="YES" temporarily for copious rc-script diagnostics and rc_startmsgs="YES" to show start messages during faststart. Both belong in rc.conf(5), not loader.conf. Restore normal verbosity after collecting evidence.

Observe what the slow script is waiting for

During a controlled reproduction, find the script or daemon PID from another console and inspect its kernel stack and open resources:

procstat -kk PID
fstat -p PID
sockstat -c
ps axd

Replace PID. A sleep in name resolution, socket connect, NFS, disk I/O, or wait4 identifies a different dependency. wait4 often means the rc script is waiting for a child command; use the process tree to find that child rather than blaming rc.subr. A socket in SYN_SENT points toward network reachability, while a vnode on an unavailable mount points toward storage.

Avoid attaching invasive tracing to PID 1 or production daemons without understanding its impact. Start with procstat, process trees, service logs, and boottrace. If deeper syscall tracing is needed, reproduce the service in staging with the same boot-time environment.

The environment matters because service(8) and /etc/rc use a restricted PATH and HOME=/. A script that succeeds interactively may search a different executable, read an administrator’s home configuration, or inherit proxy variables. Use absolute paths and explicit configuration instead of exporting a login environment into boot.

Fix timeouts where the wait originates. If a daemon’s foreground start command waits until it is ready, changing the rc header does not shorten that work. If the script polls a PID file at the wrong path, correct pidfile or procname. If it invokes a remote health check, give that check a documented bounded timeout and meaningful failure state.

Compare enabled services and recent package changes

Boot regressions often follow a newly enabled service or package update rather than a base rc change. Preserve ordered and enabled inventories:

service -e
service -rv
pkg info -a

Compare them with the last known-good snapshot or configuration-management record. service -e lists scripts with enabled rc variables; service -rv shows the ordered inventory. A script can remain in rcorder while disabled, so the lists answer different questions.

Identify ownership before editing a local script:

pkg which /usr/local/etc/rc.d/slowservice
pkg info -D package-name

Package messages may describe required migration after an upgrade. Check /usr/local/etc/rc.conf.d, /etc/rc.conf.d, and service-specific files for old hostnames, mounts, or credentials. Do not dump secrets into a public boottrace report.

Temporarily disabling one noncritical service can confirm causality, but use its supported rc variable, record the previous value, and test application dependencies. Removing or renaming the script hides it from ordering and shutdown logic and is a poor diagnostic method.

Repair dependency declarations conservatively

A port-installed script may contain an unnecessarily broad header:

# PROVIDE: example
# REQUIRE: DAEMON NETWORKING LOGIN

Removing LOGIN is valid only if the daemon truly does not require any milestone it represents and its start/stop ordering remains safe. Test boot, shutdown, service restart, and package upgrade. A dependency that appears unnecessary during one successful boot may prevent a race on another storage or network configuration.

Edits under /usr/local/etc/rc.d can be overwritten by package updates. Prepare a patch for the port maintainer with the before/after rcorder graph, boottrace evidence, and reasoning. Do not edit base scripts under /etc/rc.d as a local performance tweak; use supported configuration or submit an upstream correction.

If the script needs a remote resource rather than an rc milestone, broadening REQUIRE may not help. Give the daemon bounded retry/backoff behavior, start it asynchronously if its documented model permits, or move readiness enforcement to the dependent application. An arbitrary sleep 10 converts a race into a fixed delay and still fails on a slower day.

Separate boot dependencies from filesystem and hardware delays

Boottrace includes kernel and userspace events, so confirm the delay is actually inside rc. A long gap before /etc/rc starting belongs to firmware, loader, kernel device probing, root mount, or filesystem recovery. A gap after /etc/rc finished belongs to a display manager, login path, or later application startup.

Storage timeouts often emit controller resets in dmesg; remote mounts report server waits; DHCP and DNS leave their own messages. Preserve all clocks in one timeline. Reducing dependencies cannot repair a disk that waits 30 seconds to time out.

Verify the whole lifecycle

After the change, repeat the same boottrace measurement rather than relying on perception. Confirm total time and the targeted script interval improved, no service started before required local resources, and every enabled service is running and reachable. Then test orderly shutdown, because rc dependency order is also used in reverse for scripts marked for shutdown.

Reboot again after a package update or cold power cycle if the original issue was intermittent. A successful warm reboot with cached data is weak evidence. The durable outcome is a documented timing reduction without missing services, startup races, or unclean shutdown behavior.

Related:

Sources: