Fixing 'Panic: Solaris Assert' Style ZFS Kernel Panics on FreeBSD

OpenZFS retains assertion macros and source naming inherited from its Solaris ancestry, so a FreeBSD panic may say solaris assert and name an expression, file, function, and line. The assertion proves that running code observed a state its developers declared impossible or unsafe. It does not by itself prove whether the cause was on-disk corruption, bad RAM, controller errors, a race, a stale kernel module, or a software defect.

Preserve the exact panic and binary provenance

Capture the entire console message, not a paraphrase. The expression and source location distinguish unrelated panics that share the words “Solaris assert.” After reboot, preserve the crash artifacts:

ls -lah /var/crash
readlink /var/crash/vmcore.last
sed -n '1,240p' /var/crash/core.txt.last
freebsd-version -kru
uname -a
zpool version

savecore(8) extracts a configured kernel dump and crashinfo(8) creates a summary. Raw vmcores contain sensitive memory and should remain root-only. Copy the dump, matching crashed kernel, debug symbols, loaded module versions, logs, and pool status to protected storage before updating or rotating files.

The symbols must match the exact crashed kernel and ZFS modules. A later kernel.debug from the same nominal release can resolve addresses incorrectly. With matching artifacts, use the Developers’ Handbook workflow:

kgdb /path/to/matching/kernel.debug /var/crash/vmcore.last

Collect the panic thread and all-thread backtraces. The assertion frame identifies where the invariant was detected, not necessarily where state was first corrupted. An ARC assertion can result from a software race or memory corruption; a block-pointer assertion can result from disk metadata, RAM, or code. Source filename alone is not root cause.

If dumps were not enabled, fix dumpdev, /var/crash capacity, and symbol retention before closing the incident. Future recurrence without evidence wastes the strongest diagnostic opportunity.

Prevent automatic writes and repeat panics

Do not repeatedly reboot a host that panics while importing the same pool. If it is the root pool, single-user mode still requires importing and mounting root and may reproduce the panic before a shell appears. Boot compatible FreeBSD rescue media instead. For a data pool, prevent automatic import through the system’s documented configuration rather than deleting cache files blindly.

Inventory devices and errors before importing:

camcontrol devlist
geom disk list
gpart show -p
dmesg | grep -i -E 'error|timeout|reset|nvme|zfs'
zpool import

The final command scans labels without importing. If drives reset, disappear, or show read errors, image or clone them before a full pool scan. A panic plus failing storage is a data-recovery incident, not a reason to stress every block with a scrub.

Prove no other cluster node or host can access the pool. Do not use zpool import -f until the old owner is fenced; -f overrides an ownership safeguard and can enable simultaneous writers.

Attempt read-only inspection before repair

If the pool is visible and hardware is stable, import without mounting datasets and with writes disabled:

mkdir -p /mnt/recovery
zpool import -N -o readonly=on -R /mnt/recovery tank
zpool status -v tank
zpool events -v

Replace tank. -N prevents mounts, readonly=on blocks pool changes, and -R supplies an alternate root and avoids populating the normal cachefile. If this path triggers the same assertion, preserve the new dump and stop repeating it. Work from cloned media with a FreeBSD/OpenZFS developer or recovery specialist.

If the pool imports read-only, copy irreplaceable datasets to independent storage before experimenting. Mount only selected datasets below the alternate root and load encryption keys through a controlled process. Verify restored files or application backups; “copy completed” does not prove application consistency.

Do not run zpool upgrade, labelclear, clear, rewind flags, or a writable checkpoint rollback while classifying the panic. Those operations can erase compatibility or evidence. zpool import -F discards recent transactions when actually applied, and -X enables extreme recovery; neither is a generic assertion fix.

Correlate the trigger with pool, memory, and hardware evidence

Determine what operation consistently precedes the panic: pool import, mounting one dataset, reading one object, deleting a snapshot, receiving a stream, scrub, heavy ARC pressure, or device removal. A minimal, repeatable trigger dramatically narrows the defect.

For storage evidence, preserve zpool status -v, ZFS event logs, controller messages, device serials, firmware versions, and enclosure history. ZFS checksum errors identify data that failed verification but do not alone name whether disk, cable, controller, RAM, or an earlier writer corrupted it.

For memory, check ECC and machine-check logs, management-controller events, and run appropriate offline memory diagnostics after data is safe. Non-ECC RAM can alter in-memory metadata before ZFS computes or verifies a checksum. A clean later memory test cannot prove no transient error occurred; repeated corrected ECC events are stronger evidence.

Check whether kernel and module versions match. A partially completed base upgrade or stale out-of-tree module can create incompatible interfaces. Record freebsd-version -kru, boot environment, package state, and module hashes before reinstalling anything.

Search for a fixed software defect with the exact signature

Assertions are also how real bugs surface. Search the FreeBSD Bugzilla database, release errata, Security Advisories, and source history using the expression, function, file, and top non-generic stack frames. Compare the exact FreeBSD patch level and OpenZFS version. Similar wording at a different line or commit may be unrelated.

If a supported later release contains a matching fix, preserve the original evidence and rehearse the upgrade on a clone. Updating rescue media can let it understand newer pool features, but never run zpool upgrade as part of incident recovery. Keep older compatible boot media until verified backups exist.

A useful bug report includes the exact panic, symbolized backtrace, all-thread trace where relevant, FreeBSD/source revision, ZFS version, pool topology without sensitive names, hardware, trigger, recent changes, and whether the issue reproduces read-only. Do not attach a raw vmcore publicly; arrange a secure transfer only if a trusted developer needs it.

Scrub only after data is protected and hardware stable

A scrub reads every allocated block and verifies checksums. On a redundant writable pool it can repair from a valid copy, but it also creates sustained I/O and may write repairs. It cannot run as a meaningful repair on a read-only recovery import. Therefore, copy data first, replace or stabilize suspect hardware, return to a controlled writable state, and then scrub under monitoring.

zpool scrub tank
zpool status -v tank

A zero-error scrub is useful but does not exonerate RAM or software: the assertion may have involved transient in-memory state not represented on disk. Conversely, checksum errors confirm affected blocks but still require hardware and software correlation.

Define recovery by evidence, not absence of another reboot

Before restoring service, require verified backups, stable hardware, a supported patched FreeBSD build, no recurrence under the known trigger, and documented pool status. Monitor ZFS events, device errors, ECC, and future dumps. Preserve the incident’s command timeline and all destructive decisions.

The safe response to a Solaris-style assertion is neither “ZFS corruption” nor “kernel bug” by default. It is to freeze writes, preserve a symbolized crash, test the pool read-only, protect data, and use the exact invariant plus hardware evidence to decide which explanation survives.

Related:

Sources: