				       README Notes
				Broadcom bnxt FreeBSD Driver

	                             Broadcom Inc.
				15101 Alton Parkway,
				   Irvine, CA 92618

			  Copyright (c) 2024 Broadcom Limited
				   All rights reserved


Table of Contents
=================
  Introduction
  Driver Dependencies
  Driver Download and Compilation
  Unloading and Loading Driver
  Driver Settings
  Port Speeds
  Interface Driver Statistics
  Error recovery
  Known Issues

Introduction
============
This file describes the bnxt FreeBSD driver for the Broadcom NetXtreme-C
and NetXtreme-E BCM573xx/BCM574xx/BCM575xx 10/20/25/40/50/100/200 Gbps
Ethernet Network Controllers.


Driver Dependencies
===================
The driver has no dependencies on user-space packages and necessary
firmware must be programmed in NVRAM.


Driver Download and Compilation
===============================
1. Download and unpack the driver source files.
2. Navigate to the bnxt_en directory.
   ## cd freebsd-bnxt-227.0.24.X/bnxt_en
3. Compile the driver.
   ## make clean
   ## make
      Note: The Makefile's .PATH variable points to /usr/src/sys/dev/bnxt/bnxt_en/
            for the source files, but if there are any source files in the current
	    working directory (pwd), they will take precedence over the ones
	    specified in .PATH variable.
4. The driver binary will be generated in the current directory.
   ## ls ./if_bnxt.ko

Unloading and Loading Driver
=============================
1. To unload:
   ## kldunload if_bnxt

2. To load:
   A. Navigate to the directory containing the driver binary
      ## cd /usr/obj/usr/src/amd64.amd64/sys/modules/bnxt_en/
      (or)
      ## cd freebsd-bnxt-227.0.24.X/bnxt_en
   B. Load the driver
      ## kldload ./if_bnxt.ko

Driver Settings
===============
Commands to assign IP, MTU, down/up, speed change, etc..
   ## ifconfig
   ## ifconfig -a
   ## ifconfig bnxt0 12.0.0.10/24
   ## ifconfig bnxt0 mtu 9000
   ## ifconfig -m
   ## ifconfig -m bnxt0
   ## ifconfig bnxt0 media 25GBase-CR

Port Speeds
===========
On dual-port devices, the port speed of each port must be compatible with
the port speed of the other port.  10Gbps and 40Gbps are one set of
compatible speeds.  25Gbps and 50Gbps are the other set of compatible
speeds.  The 2 sets are not compatible with each other and will result
in link loss if one port uses a speed that is incompatible with that of
the other port.


Interface Driver Statistics
============================
$sysctl dev.bnxt.0

Error recovery
==============
Firmware error recovery is supported for only Thor adapters.
Hardware and Firmware can run into two types of FAULT/error- 
1. Firmware can detect the error and signal driver to take further action. 
2. Error results into firmware in unresponsive(dead/hung) state.

Driver supports only type 1 type of errors/faults recovery on a best-effort basis.
This type of error recovery can be called as “Firmware initiated error recovery”.
e.g. Mbuf leak, losing pci credits, re_flush, firmware unhandled exception etc.

To trigger error recovery, user can write as below to sysctl interface:
	sysctl dev.bnxt.<x>.reset_ctrl="crash null_ptr"    (where 'x' = 0,1,2..) 

Limitations:
1. For firmware error recovery to function correctly, all the interfaces of its
   PFs must be in Up state (ifconfig <interface> up). Failing to do so may lead
   to unpredictable behavior, necessitating a reboot to restore normal operation.
2. Driver attempts error recovery on best-effort basis but few a times driver may abort firmware recovery.
3. In intial driver releases, few debug prints are added in firmware error recovery part for debug purpose.
   These prints can be tuned once this feature is hardened.


Known Issues:
=============
1. TCP / UDP performance may vary between 30% to 95% depending on the link speed and NIC.
2. There is a limit on Max ping packetsize that can work for various
   MTU's and this issue is seen with other vendor NICs as well.
   ========================================
	MTU	Max ping packetsize that
		works (-s option of ping)
   ========================================
	1500		25152
	2000		33584
	3000		50584
   ========================================
3. The current driver supports only MSI-x interrupts but not legacy interrupts (INT-x / INT-a).
   Disabling MSI-X may cause unpredictable behavior, including crash.
   #kenv -qv bnxt.0.iflib.disable_msix=1
4. The current driver doesn't support suspend/resume operations in FreeBSD,
   support will be added in future releases. Trying those operations may cause
   unpredictable behavior, including crashes.
5. Operations like 'driver unload' and 'vlan destroy' may be blocked when there is UDP Rx flood.
   The workaround is to stop the UDP flood so that those operations will continue.
6. It may be necessary to perform an ifconfig down/up on the interface after changing media
   speed with the 12.2-RELEASE kernel. This issue is fixed in the latest 12.2-STABLE kernel.
7. Asymmetric Tx and Rx queues configuration is not supported. configuring may result in
   undefined behavior.
8. devctl bus reset is not supported. post this command driver functionalities
   may not work until driver gets reloaded.
   # devctl reset bnxt<x> (where x=0,1..N)
9. devctl device reset is supported.
   # devctl reset -d bnxt<x> (where x=0,1..N)
10. On BCM574xx chips, when interface (bnxt0, bnxt1..) is in admin down state, 'ifconfig bnxt<x>' may
    not reflect correct link speed (may show unknown) upon link speed change operation from peer.
    To get correct link information before doing link speed change, please make sure to bring up
    the interface through command- 'ifconfig bnxt<x> up' (where x=0,1..N)
11. Driver compilation on FreeBSD12.4 OS throws a lot of set but not used warnings for variable 'e'
    And that is the kernel issue. It will be probably fixed in the upcoming freebsd OS releases.
    
    Below is the warning snippet:
    
    /usr/src/sys/dev/pci/pcivar.h:420:1: warning: variable 'e' set but not used [-Wunused-but-set-variable]
    PCIB_ACCESSOR(bus,              BUS,            uint32_t)
12. Force link speed change using ifconfig will be reflected within the angle bracket of media section in
    ifconfig only after link comes up. Below is how it looks in media section of ifconfig output:
    media: Ethernet autoselect (10GBase-CR1 <full-duplex>)
13. BCM574xx chips support LRO(Large Receive Offload) but it's disabled by default. 
    Hardware LRO can be enabled through kenv/loader.conf during driver load.
	dev.bnxt.<x>.hw_lro.enable	(where x=0,1..N)
14. ifconfig option to modify LRO settings is not effective.
15. The QSFP28 module information appears as unknown in FreeBSD 13.2, and the
    verbose information displayed by the ifconfig command may contain incorrect
    data for the module. When running the ifconfig command with the -v flag and
    specifying the interface, the output will show that the module is plugged in,
    but the type is identified as "QSFP28 Unknown".
16. The current driver does not support the 802.1ad VLAN protocol.
17. VLAN IPv6 will not support jumbo packets if jumbo MTU is inherited from base interface during VLAN creation.
    If user configures the jumbo MTU on VLAN interface after VLAN creation, then VLAN IPv6 jumbo works as expected.
    This is the OS limitation.
18. There is known disparity between Linux and FreeBSD ring counters exposed through sysfs/sysctl. This will be
    addressed on best-effort basis in future FreeBSD driver releases.
19. Ongoing multicast traffic continues without loss, when the multicast group is dynamically removed from the server.
20. In FreeBSD, driver does not set default DCBX parameters start of the day. User has to do desired DCBX programming
    through sysctl hooks. Also, it's not guaranteed DCBX settings done through systcl will be restored across system reboots/
    driver reloads.
