Diagnosing FreeBSD Ethernet Link Flapping at the Driver Level

An Ethernet flap is a loss and return of physical carrier, not merely a failed ping or TCP timeout. FreeBSD may report both as “the network went down” to an application, but the evidence differs. Establish whether carrier actually changes, correlate both ends of the link, and preserve counter deltas before replacing cables or applying driver tunables.

Prove that physical link state changes

Identify the interface, driver, PCI device, and current media state:

ifconfig -v em0
ifconfig -m em0
pciconf -lv
dmesg | grep -i -E 'em0|link|watchdog'

Substitute the real interface. ifconfig -v reports active media and status; ifconfig -m lists supported media. A log sequence such as “link state changed to DOWN” followed by “UP” establishes a carrier transition. A stable status: active while DNS or a remote host fails points instead toward addressing, routing, firewall, congestion, or the upstream network.

Search both current and rotated logs around the incident, not only one grep result:

grep -i 'em0.*link state' /var/log/messages
xzgrep -i 'em0.*link state' /var/log/messages.*.xz

Log rotation and compression depend on local configuration. Record exact timestamps and timezone. A flap every few seconds during idle, one coinciding with switch maintenance, and a single outage under heavy traffic are different fault signatures.

Measure counter deltas rather than absolute totals

netstat(1) provides generic packet and error counters:

netstat -I em0 -d -b
netstat -I em0 -w 1
systat -ifstat

Capture a baseline and compare it after a flap. Input errors, output errors, collisions, or drops increasing during the event are evidence; a large lifetime total on a server with years of uptime is not. Generic interface counters do not necessarily identify CRC versus alignment versus missed-buffer errors.

Some drivers expose hardware-specific statistics under dev.<driver>.<unit>, but names and availability vary:

sysctl -a | grep '^dev\.em\.0\.'

Do not assume another driver’s mac_stats tree exists. Read the matching section-4 driver manual and label counters exactly as the driver does. CRC or symbol errors that rise on both the NIC and switch port strongly implicate cable, connector, optic, interference, or a failing port. Output queue drops without loss of carrier more often indicate congestion or software pressure, not a bad cable.

Check negotiation without creating a new mismatch

Compare FreeBSD’s active media with the switch port’s speed, duplex, flow control, and Energy Efficient Ethernet state. For twisted-pair gigabit and faster Ethernet, autonegotiation is normally expected; forcing 1000baseT full-duplex on one side is not a universal repair and can create a mismatch. Set both ends according to their standards and vendor documentation.

A useful isolation sequence changes one component at a time:

Reseat both cable ends or the optic and record the time.
Replace the patch cable with a known-good, correctly rated cable.
Move to a known-good switch port while preserving configuration.
For fiber, verify supported optic type, wavelength, and diagnostics on both ends.
Return both sides to compatible autonegotiation and observe.

If each change is simultaneous, a temporary recovery proves nothing. Preserve the suspect component for retest rather than declaring it bad from one disappearance of symptoms.

SFP and SFP+ links add vendor compatibility, optical power, polarity, and dirty connector surfaces. The operating system may only see carrier disappear; the switch or transceiver diagnostics may reveal low receive power or unsupported-module alarms. Copper diagnostics, if the switch offers them, can estimate open or short faults without changing FreeBSD.

Correlate switch, topology, and power events

Inspect the switch port log over the same timestamps. Determine whether the switch also saw physical link down, administratively disabled the port, detected an authentication failure, or placed it in an error-disabled state. Spanning Tree state transitions can interrupt forwarding without necessarily dropping physical carrier, so do not call every STP event a link flap.

Also check for LACP or lagg(4) membership. A failing member can move traffic between links while the logical lagg interface remains up. Compare physical-member and aggregate counters. Duplicate MAC alarms, port-security violations, 802.1X reauthentication, and switch power events belong to the upstream investigation even if FreeBSD logs a secondary state change.

Correlated power or thermal events matter. A NIC, switch, media converter, or transceiver can reset when temperature or power delivery crosses a threshold. Check management-controller logs and switch environmental sensors rather than focusing only on packets.

Escalate to driver and firmware only with preserved evidence

Record the exact FreeBSD kernel, driver attachment, and PCI IDs:

freebsd-version -kru
uname -a
pciconf -lv em0@pci0:1:0:0
sysctl dev.em.0

The pciconf selector above is an example; use the address shown on the host. Search the hardware notes, release errata, and FreeBSD bug database for the exact device revision and panic/watchdog text. A generic search for “Intel NIC flap” combines unrelated controllers and drivers.

Energy Efficient Ethernet can cause interoperability problems on particular NIC/switch pairs. ifconfig(8) supports eee and -eee only when the driver and media implement it. Before disabling it, confirm current support with ifconfig -m, record the state on both ends, and make the change temporarily:

ifconfig em0 -eee

If flapping stops through a sufficiently long observation window, document the NIC, switch, firmware, and test duration. A short quiet period is correlation, not proof. Persist the setting only through the release-appropriate rc.conf syntax after validating reboot behavior.

Driver watchdog or reset messages without physical errors can indicate exhausted descriptors, firmware faults, PCIe errors, or a driver bug. Capture dmesg, counters, traffic/load conditions, and a reproduction before updating. Prefer a supported FreeBSD errata or release update and vendor-approved NIC firmware over undocumented sysctls.

Avoid fixes that cannot repair carrier loss

Checksum offload, large receive offload, MTU, DNS, DHCP, and TCP congestion control can affect traffic without changing the electrical carrier. Disabling rxcsum, changing an IP address, or lowering MTU may hide an application symptom but does not repair a genuine PHY down/up cycle. Keep these variables unchanged until logs establish that the interface stayed physically active.

Packet captures also have limits. Most NICs discard bad frames before tcpdump(1) sees them, and they commonly strip the Ethernet frame-check sequence. A clean capture therefore does not rule out CRC errors; use driver and switch hardware counters. Conversely, malformed TCP checksums in a host-side capture may reflect transmit checksum offload—the NIC fills them later—and are not automatically evidence of wire corruption.

When a driver resets the adapter after a watchdog timeout, link-down messages are a consequence of the reset even with perfect cabling. Preserve the watchdog line that precedes the flap and correlate PCIe or firmware events. The ordering of messages distinguishes “carrier failed, then driver noticed” from “driver reset, then carrier cycled.”

Close the incident with a controlled test

After the suspected repair, monitor longer than the previous maximum interval between failures. Confirm that carrier events stop, error-counter deltas remain normal, the switch agrees, and traffic survives the workload that previously triggered the issue. Test redundancy if the interface is part of a bridge or lagg.

The reliable decision tree is: no carrier event means investigate Layer 3 and above; carrier plus physical errors means isolate media and ports; simultaneous switch policy events mean investigate the switch; and host resets with clean physical telemetry justify driver, firmware, PCIe, and hardware escalation. FreeBSD exposes enough evidence to make that distinction without guessing—provided the counters and clocks are captured before the next change.

Related:

Sources: