How to Set Up High Availability with CARP on FreeBSD

CARP lets multiple FreeBSD hosts advertise one virtual IP and VHID while one is MASTER and the others remain BACKUP. It handles address ownership, not application data, service readiness, routing symmetry, or firewall rules. Build and test those layers separately. The examples use documentation-only addresses; replace them with an approved subnet and run interface changes from console access.

Plan Layer 2, identities, and service state

Use two unique management addresses and one virtual service address on the same broadcast domain:

node-a management: 192.0.2.10/24
node-b management: 192.0.2.11/24
CARP VIP:          192.0.2.100/32
VHID:              10

The VHID must be unique for that CARP group on the broadcast domain. Both nodes use the same VHID, VIP, and authentication passphrase. Store the secret through configuration management with root-only permissions; CARP traffic is not an encrypted tunnel.

Confirm the switch permits CARP IP protocol 112, multicast destination 224.0.0.18 when multicast mode is used, virtual MAC movement, and gratuitous ARP/neighbor advertisements. CARP does not require a special load balancer, but the network must carry its advertisements.

Decide application replication before failover. Both hosts need equivalent service configuration and a health policy. Database-native replication, HAST, or application storage replication must have its own consistency and fencing model. Never mount one non-clustered writable filesystem on two nodes merely because CARP moves an IP.

Load CARP and configure the primary

Load the module now:

kldload carp
kldstat -m carp

For persistent boot loading, add carp_load="YES" to /boot/loader.conf as the Handbook documents, or compile device carp into a custom kernel. Do not create carp0 on modern FreeBSD. Since FreeBSD 10, VHIDs and virtual addresses attach directly to the physical interface.

First ensure node A has its management address. Its persistent /etc/rc.conf entries are:

hostname="node-a.example"
ifconfig_em0="inet 192.0.2.10/24"
ifconfig_em0_alias0="inet vhid 10 advskew 10 pass CHANGE_ME alias 192.0.2.100/32"

Apply in a maintenance window with service netif restart em0 only if the remote-access consequences are understood; reboot is safer when console access is available. Inspect:

ifconfig em0
sysctl net.inet.carp

The output should show VHID 10 and, with no competitor, MASTER.

Configure the backup with a larger advskew

Node B uses a unique management address and the same virtual identity:

hostname="node-b.example"
ifconfig_em0="inet 192.0.2.11/24"
ifconfig_em0_alias0="inet vhid 10 advskew 100 pass CHANGE_ME alias 192.0.2.100/32"

Lower advertisement skew wins under equal health, so node A is preferred. advskew is measured in 1/256-second increments added to advbase; it is not a priority integer where larger means better. Avoid using identical skew when deterministic ownership matters.

After both nodes are configured:

ifconfig em0 | grep -A2 carp
tcpdump -ni em0 proto 112

Expect one MASTER and one BACKUP. If both are master, advertisements are not crossing the network or VHID/passphrase/address-family settings differ. Do not put the service into production with split-brain CARP state.

Decide whether the preferred master should preempt

By default, the returning preferred node does not necessarily take the VIP back automatically. To allow it, set on participating nodes:

sysctl net.inet.carp.preempt=1

Persist the choice by adding net.inet.carp.preempt=1 to /etc/sysctl.conf; sysrc is intended for rc.conf-style files and should not be used to rewrite this sysctl configuration.

Preemption restores the desired primary but causes a second transition after recovery. Applications with expensive role changes may prefer to remain on the healthy backup until an operator moves service deliberately. Document the policy rather than enabling preemption automatically.

FreeBSD also uses CARP demotion to lower a host’s eligibility. Monitor net.inet.carp.demotion and interface state. A nonzero demotion may explain why the node with lower advskew is not master.

Configure services around role, not just address

Run the service on both nodes and make it bind the VIP or wildcard address as appropriate. A daemon that binds only the management IP will not become reachable merely because CARP changes state. Conversely, a daemon accepting writes on the backup’s management address can bypass the HA role.

Use devd(8) CARP events or a cluster manager to trigger role-specific actions, but scripts must be idempotent and handle rapid transitions. CARP sees interface/advertisement health, not whether a database is writable or an HTTP worker is stuck. Add service health monitoring that can demote the node safely.

Routing and NAT must be symmetric. For gateway HA, both nodes need identical PF rules, routing tables, interface/VLAN names, and upstream/downstream reachability. Configuration synchronization is separate from CARP.

Add pfsync for stateful firewalls

pfsync(4) replicates PF state so established flows can survive firewall failover. Use a dedicated protected link when possible:

ifconfig pfsync create
ifconfig pfsync0 syncdev igb1 syncpeer 198.51.100.11 up

On the peer, reverse syncpeer to its partner. Persist the cloned interface and pfsync settings through the documented rc.conf syntax for the release.

pfsync data is not encrypted or authenticated. Do not expose it to an untrusted segment. Firewall rules, tables, NAT configuration, routing, and application state are not synchronized by pfsync; deploy them identically through configuration management.

Monitor pfsync0, PF state counts, and sync errors. During failover, some protocols or asymmetric paths may still reset, so test real long-lived traffic rather than only ping.

Test controlled failure and recovery

Before testing, maintain console access and continuous probes from a third host: ICMP, TCP connection establishment, and one established application session. Record CARP and switch logs on both nodes.

Trigger a planned failure by shutting down the primary cleanly or taking the CARP-bearing interface down from console. Confirm:

backup changes to MASTER;
the virtual MAC/IP moves through the switch;
new service connections succeed;
replicated data is current and writable only where intended;
with pfsync, eligible established sessions survive.

Restore node A and observe the documented preemption behavior. Then test service failure—not only host failure—using the health/demotion mechanism. CARP alone will keep a host master when its application dies.

Finally reboot each node separately, verify persistent configuration, and test split-brain prevention by blocking advertisements in staging. Alert on both-master, no-master, state transitions, demotion, pfsync lag, replication health, and service-level failures.

CARP answers “which host owns this address?” A complete HA system also answers “which host may write authoritative state?”, “is the service healthy?”, and “can the network route both directions?” Production readiness requires all three.

Include the switching layer in the acceptance test. Both CARP peers must share the intended layer-2 segment, and the network must permit protocol 112 advertisements plus movement of the virtual MAC between ports. Port-security limits, multicast filtering, or a stale forwarding entry can make host configuration look correct while clients still reach the former master. Verify behavior from another switch port and across every routed client path that matters.

Capture ARP or IPv6 neighbor-discovery behavior during the transition as well as CARP advertisements. A prompt master election is not sufficient if client or router neighbor caches continue sending traffic to the wrong port. Measure interruption time at the application layer and record the switch, router, and host timers that influence the observed recovery.

Related:

Sources: