HAST and Ctl: FreeBSD's Built-In Storage Replication and iSCSI Target

FreeBSD ships two storage components that solve adjacent but distinct problems. HAST replicates a block provider between two machines, while CTL—the CAM Target Layer—implements a kernel-based SCSI target that ctld(8) can expose through iSCSI. HAST protects a copy of blocks; CTL presents blocks to initiators. Neither component decides complete cluster policy, and combining them without fencing can make storage less safe rather than more available.

HAST: block-level replication, not a filesystem

HAST (Highly Available STorage) works below the filesystem. Each node has a local provider, such as an unused GPT partition, and hastd(8) exchanges block updates over TCP. The primary exposes /dev/hast/<resource> for I/O; the secondary receives replication and must not be mounted or written independently. Reads normally come from the primary’s local disk, while writes, deletes, and flushes are sent to both sides according to the selected replication mode.

HAST maintains metadata and a dirty-region bitmap so that a returning node can synchronize changed extents rather than recopy the entire resource after every interruption. Initial synchronization is still substantial, and a resource must not be advertised as protected until hastctl status shows the expected state.

The original description “synchronous by default” hides an important durability distinction. The hast.conf(5) manual defines three modes:

memsync, the default, completes after the local write and after the secondary acknowledges receipt in memory, before the secondary necessarily commits it to disk.
fullsync completes only after both local and remote writes complete.
async completes after the local write and sends the remote update asynchronously.

memsync lowers latency while retaining strong protection against a single-node loss, but a narrow loss window exists if the secondary acknowledges data and then loses that uncommitted data at the same time the primary becomes unavailable. fullsync closes that specific window at the cost of remote storage latency on every write. async trades a larger recovery-point window for throughput. The mode must follow an explicit recovery-point objective; “HAST enabled” is not a durability specification.

A minimal two-node resource

The same /etc/hast.conf should be installed on both nodes. Hostnames in on blocks must match each node’s hostname. This laboratory example uses dedicated replication addresses and disposable providers:

replication memsync

resource shared {
    on node-a {
        local /dev/gpt/hast-shared
        remote tcp4://10.20.0.2
    }
    on node-b {
        local /dev/gpt/hast-shared
        remote tcp4://10.20.0.1
    }
}

Initialize metadata only after confirming that both providers are the intended empty devices. Then start hastd on both nodes and assign roles:

sysrc hastd_enable=YES
hastctl create shared
service hastd start

# node-b
hastctl role secondary shared

# node-a
hastctl role primary shared
hastctl status shared

hastctl create writes HAST metadata and belongs on both nodes. Role commands are node-specific. The first filesystem is created only through /dev/hast/shared on the primary, never directly on the underlying provider. Device names, addresses, role order, and exact status output must be validated against the installed hastctl(8) and hast.conf(5) pages.

HAST’s role: replication, not clustering

HAST does not elect a leader, fence a failed host, mount a filesystem, move a service IP, or decide whether an application is healthy. CARP can move an IP address, but an IP transition is not proof that the old primary can no longer write storage. A network partition can leave both machines alive, each unable to observe the other. Promoting the secondary without first fencing the old primary creates split brain.

A safe failover sequence is site-specific but has invariant goals: stop or fence the old writer; confirm replication and role state; promote exactly one secondary; run filesystem recovery when required; mount the provider; start the application; then move or advertise the service endpoint. The reverse sequence cleanly stops the application and unmounts storage before demotion.

The hastctl(8) example explicitly warns that UFS may require fsck after the primary dies. Force-mounting because the replica is “synchronous” confuses replicated blocks with a clean filesystem transaction boundary. Application write caches, filesystem state, and replication acknowledgements are separate layers.

Where HAST sits relative to ZFS

ZFS mirrors and RAIDZ provide local redundancy inside one pool. zfs send and zfs receive transfer snapshot streams to another pool and can preserve dataset structure and incremental history. They do not provide HAST’s per-write remote acknowledgement, so their recovery point is the last replicated snapshot or stream state. Conversely, HAST does not provide ZFS snapshots, checksumming semantics, dataset delegation, or independent historical recovery points.

HAST remains useful for UFS, swap-like or application-defined block formats, and designs whose recovery-point objective requires fullsync or memsync. Placing a whole ZFS vdev on HAST is technically possible but changes ZFS’s view of physical redundancy and failure. It should not be a default architecture copied from a diagram. Decide whether the unit of protection is a block device, a filesystem transaction, a ZFS dataset, or an application-level replica.

Ctl: FreeBSD’s SCSI/iSCSI target subsystem

CTL is FreeBSD’s kernel target subsystem. ctld handles iSCSI configuration and connections, while CTL presents LUNs and SCSI behavior. An initiator sees each exported LUN as a raw disk and is responsible for partitioning and filesystems. Ordinary filesystems such as UFS are not safe for simultaneous mounting by unrelated initiators; shared block access requires a cluster-aware filesystem or a single-writer policy.

The supported persistent entry point is /etc/ctl.conf plus ctld. A restrained target listens on a dedicated storage address and authenticates initiators rather than binding unauthenticated iSCSI to every interface:

auth-group ag0 {
    chap storage-client replace-with-a-generated-secret
}

portal-group pg0 {
    discovery-auth-group ag0
    listen 10.30.0.10
}

target iqn.2026-07.com.example:target0 {
    auth-group ag0
    portal-group pg0

    lun 0 {
        path /dev/zvol/tank/iscsi-lun0
    }
}

Replace the example CHAP secret before use and protect /etc/ctl.conf permissions. CHAP authenticates the session; it does not encrypt iSCSI payloads. Put storage traffic on a controlled network and use network-layer encryption when the threat model requires confidentiality.

Validate configuration before reloading it, then inspect the kernel-visible LUNs:

ctld -t
sysrc ctld_enable=YES
service ctld start
service ctld reload
ctladm devlist
ctlstat -t

The -t option validates the configuration and exits. ctladm devlist shows configured LUNs, while ctlstat provides I/O statistics. A successful daemon reload does not prove that an initiator can discover, authenticate, and perform durable I/O; test those steps from an actual initiator and verify logs on both sides.

A ZFS zvol is a common backing path because the pool can provide checksums, redundancy, snapshots, and scrubs beneath the LUN. Snapshotting a zvol while an initiator is writing usually captures crash-consistent block state, not necessarily an application-consistent database. Coordinate guest quiescence or application backups when consistency requires more than a power-loss recovery path.

Combining HAST and CTL without inventing a cluster

A CTL LUN can be backed by /dev/hast/shared, but that only composes two block layers. During failover, initiators also need a reachable portal address, sessions must recover, CTL must be stopped on the fenced node, and the promoted node must not export a stale or synchronizing provider. Persistent reservations, multipath behavior, and application retry semantics add further requirements.

The correct order is therefore more important than the ability to express the stack. On shutdown, stop applications or disconnect initiators, stop exporting the LUN, drain and unmount local consumers, and then demote HAST. On promotion, fence first, verify HAST state, promote, perform required recovery, expose CTL, then advertise the portal. Automation should stop on ambiguous state instead of choosing availability over consistency silently.

Monitor each boundary independently: HAST role and replication status, underlying disk health, CTL LUN state, iSCSI sessions, network latency, and application errors. A green iSCSI session can still point at a degraded replica, and a synchronized HAST resource can still be hidden behind a failed portal.

Two tools, two layers

HAST answers “how are these blocks copied to a peer, and when is a write acknowledged?” CTL answers “how is this local backing store presented as SCSI LUNs?” Fencing answers “which node is allowed to write?” The filesystem or application answers “what constitutes a consistent transaction?” A reliable design documents all four answers.

FreeBSD provides the building blocks and transparent status tools, not an automatic guarantee that a two-node system cannot split. The production acceptance test must include loss of the replication link, loss of the primary, reboot of the former primary, simultaneous power loss, resynchronization, and initiator reconnection. Failover that was tested only with service hastd restart is not a tested storage cluster.

Related:

Sources: