			Installation Notes
		Broadcom FreeBSD User Space RoCE Library

		 	Broadcom
		 5300 California Avenue
	    	Irvine, California  92617

            Copyright (c) 2024 Broadcom Inc.
		    All Rights Reserved


Table of Contents
=================

  Introduction
  Driver Dependencies
  Building User space library
  Unloading and Removing RoCE lib
  Setup Verification

Introduction
============

This file describes the libbnxtre FreeBSD RoCE user space
driver for the Broadcom NetXtreme-C and NetXtreme-E BCM575xx
10/20/25/40/50/100/200 Gbps Ethernet Network Controllers.

Driver Dependencies
===================

The user space RoCE driver depends on following kernel modules:

1. Ethernet driver for NetXtreme devices (if_bnxt.ko)
2. RoCE driver for NetXtreme devices (bnxt_re.ko)
3. uVerbs device shared lib, i.e. /lib/libibverbs.so.1

Building user space library
============================
1. Download and unpack the driver source files.
2. Navigate to the libbnxtre directory.
   ## cd freebsd-bnxt-227.0.24.X/libbnxtre
3. Compile the driver.
   ## make clean
   ## make
4. Copy the generated shared object file into /lib
   ## cp libbnxtre.so.1 /lib/


Unloading and Removing RoCE LIB
===============================
To uninstall libbnxtre, delete the shared object file from /lib
   ## rm -rf /lib/libbnxtre.so.1 

Setup Verification
==================

This section list the basic commands to verify the user space RoCE driver
configuration on the target host

List RoCE devices
-----------------

#ibv_devices
    device                 node GUID
    ------              ----------------
    bnxt_re1            be97e1fffedc99f1
    bnxt_re0            be97e1fffedc99f0


# ibv_devinfo

hca_id: bnxt_re1
        transport:                      InfiniBand (0)
        fw_ver:                         226.0.51.0
        node_guid:                      be97:e1ff:fedc:99f1
        sys_image_guid:                 be97:e1ff:fedc:99f1
        vendor_id:                      0x14e4
        vendor_part_id:                 5969
        hw_ver:                         0x4250
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: bnxt_re0
        transport:                      InfiniBand (0)
        fw_ver:                         226.0.51.0
        node_guid:                      be97:e1ff:fedc:99f0
        sys_image_guid:                 be97:e1ff:fedc:99f0
        vendor_id:                      0x14e4
        vendor_part_id:                 5969
        hw_ver:                         0x4250
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet


Traffic Test
------------

Server: rping -s -d -v -a <ip of server bnxt interface>
For example
rping -s -a 192.172.1.1 -Vv -C 3
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst

Client: rping -c -d -v -a <ip of server bnxt interface>
For example
rping -c -a 192.172.1.1 -C 3 -vV
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst

