How to Configure Multi-Path TCP and Advanced Routing Tables on FreeBSD

FreeBSD supports multiple forwarding information bases, or FIBs, and scalable multipath routes inside a FIB. Neither feature is Multipath TCP. FIBs select a routing view for sockets; route multipath provides multiple next hops for a destination; MPTCP is a transport protocol that lets one negotiated TCP connection use multiple subflows.

For the common requirement “send selected services through different uplinks,” multiple FIBs or PF policy routing are the relevant tools. Build the design from console access: a wrong default route or asymmetric PF state can disconnect the administrator immediately.

Inventory interfaces, routes, and forwarding state

Record the current state before changing it:

freebsd-version -kru
ifconfig -a
netstat -rn
sysctl net.fibs net.my_fibnum net.route.multipath
sysctl net.inet.ip.forwarding

Use real interface names, addresses, prefixes, and gateways from the provider. The addresses in this article are documentation ranges. Confirm that each gateway is directly reachable on its intended link and that return traffic for public addresses is routed back by the provider.

If the host will forward traffic between networks, enable routing through the documented gateway_enable setting rather than changing a runtime sysctl alone:

sysrc gateway_enable=YES
service routing restart

Restarting routing can interrupt connectivity. On a remote system, stage persistent configuration and use an out-of-band console or a timed rollback plan.

Configure the required number of FIBs

The earlier claim that net.fibs is boot-time-only is obsolete. The current setfib(2) manual states that the number can be increased after boot but cannot be reduced. It can be set in /boot/loader.conf or /etc/sysctl.conf; memory is allocated for every FIB, so do not create thousands without a measured need.

For four routing tables on future boots, add this exact setting to the appropriate configuration file:

net.fibs="4"

To test an increase on a current system:

sysctl net.fibs=4
sysctl net.fibs

FIBs are numbered from zero. FIB 0 remains the normal process default unless a process or socket selects another. Increasing the count does not populate the new tables with working routes. Connected-route behavior is also influenced by system configuration, so inspect every table instead of assuming all interface networks were copied.

Populate and verify each routing table

Use the -fib modifier of route(8) to make the target explicit:

route add default 203.0.113.1 -fib 1
route add default 198.51.100.1 -fib 2
route -n get 1.1.1.1 -fib 1
route -n get 1.1.1.1 -fib 2

These gateways are examples. A default route is unusable unless that FIB also has a route to the gateway through the correct interface. Inspect complete tables:

setfib -F 1 netstat -rn
setfib -F 2 netstat -rn

Add required connected or internal routes explicitly when they are absent, using the syntax verified in the local route(8) manual. Avoid flushing a FIB during remote maintenance. Persist the final route set with a reviewed rc configuration, routing daemon, or dedicated rc script appropriate to the network; a one-time route add disappears at reboot.

FreeBSD 15.1 enables scalable multipath routing by default through net.route.multipath. Multiple equal-cost next hops within one FIB are a separate design from placing different defaults in separate FIBs. Validate next-hop selection and failure behavior on the deployed release before using it for resilience; a route existing does not prove that upstream NAT, state, or health detection will fail over correctly.

Run a process in a selected FIB

setfib changes the default network view for sockets created by a command and its descendants:

setfib 1 route -n get 1.1.1.1
setfib 1 fetch -qo /dev/null https://www.freebsd.org/

Test with an address rather than a hostname first so DNS reachability does not hide a routing problem. Then test resolution through the resolver path the service actually uses. setfib does not bind a source address, configure NAT, or guarantee that replies return through the same ISP.

For a persistent daemon, check whether its FreeBSD rc script exposes a FIB setting or whether the application can use the SO_SETFIB socket option. Do not edit a packaged rc script in place; package upgrades can overwrite it. A wrapper or local rc service must preserve normal privilege dropping, PID handling, dependencies, logging, and shutdown semantics.

Confirm the running process’s actual behavior with packet capture and public-source verification, not only its parent command line:

tcpdump -ni em1 host 198.51.100.1
setfib 1 traceroute -n 1.1.1.1

Assign a non-VNET jail correctly

The valid jail.conf parameter is exec.fib, not the invented vnet.set_fib=1 form. For a non-VNET jail, a configuration can include:

webjail {
    path = "/jails/webjail";
    host.hostname = "webjail.example.net";
    ip4.addr = "10.0.0.5";
    exec.fib = 1;
    persist;
}

exec.fib selects the FIB for commands run inside the jail. It is routing policy, not a security barrier and not a substitute for firewall rules. A VNET jail has its own network stack and needs its own routing design; do not assume this host-FIB example applies unchanged.

After startup, verify the jail from inside and observe real traffic:

jexec webjail sysctl net.my_fibnum
jexec webjail netstat -rn

Use PF when policy depends on packet attributes

FIB selection is convenient when policy follows a process or jail. When the decision depends on source address, destination, port, or ingress interface, PF’s route-to and reply-to can be clearer. A conceptual dual-uplink fragment is:

lan_if = "igb0"
wan2_if = "igb2"
wan2_gw = "198.51.100.1"

pass in on $lan_if from 10.0.0.5 route-to ($wan2_if $wan2_gw) keep state

Treat this as a fragment, not a complete firewall. Ordering, NAT, anti-spoofing, reply direction, IPv6, and existing state policy determine whether it is correct. Validate the complete ruleset before loading it:

pfctl -nf /etc/pf.conf
pfctl -sr
pfctl -ss

reply-to is often needed to keep responses to inbound connections on the interface where they arrived, but copying it without understanding PF state can create loops or asymmetry. Test new and established TCP sessions, UDP, DNS, failover, and recovery from external clients on both links.

Do not label this configuration MPTCP

MPTCP, standardized in RFC 8684, requires endpoint support and negotiation. It can form multiple subflows for one logical connection. Multiple FIBs merely choose routes for sockets, and PF route-to steers packets; neither converts an ordinary TCP session into MPTCP.

The FreeBSD 15.1 operator documentation exposes FIB, route multipath, PF, and socket-FIB controls but no supported MPTCP deployment interface. Old FreeBSD status reports described experimental MPTCP projects, not a current production feature. If true MPTCP is mandatory, require a supported implementation at both endpoints and validate it with protocol-level evidence rather than inferring it from traffic on two NICs.

Finish with a reboot test, full route-table capture, external path tests, firewall state review, and a deliberate single-uplink failure. A multi-uplink design is complete only when outbound selection, inbound routing, NAT, DNS, application state, monitoring, and recovery agree.

Related:

Sources: