meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
hw:server:10gbe [2022/07/13 12:04] – created niziakhw:server:10gbe [2022/10/27 21:12] (current) – niziak
Line 1: Line 1:
-====== Emulex ======+====== Emulex OCe11102 ======
  
 +2-PORT IBM/EMULEX 10GB VIRTUAL FABRIC ADAPTER 49Y7952 OCE11102 HIGH 
 +  - Low Profile
 +  - Pci Express 2.0 X8
 +  - IBM 49Y7952
 +  - ~50% faster in IOPS comparing to Intel X520
 +  - Converged Network Adapter (FCoE offloading, iSCSI offload, VIRT offloading)
 +  - Broadcom Emulex OneConnect OCe11102 ( Chipset: BladeEngine 3 be3)
 +
 +
 +
 +<code>
 +04:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
 +04:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
 +</code>
 +
 +===== Issues =====
 +
 +==== AMD-Vi ====
 +
 +On AMD Ryzen with X570 chipset:
 <code bash> <code bash>
 iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=04:00.0 address=0x1001e0b00] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=04:00.0 address=0x1001e0b00]
Line 7: Line 27:
 AMD-Vi: Completion-Wait loop timed out AMD-Vi: Completion-Wait loop timed out
 </code> </code>
 +
 +**Workaround:**
 +  - add to kernel cmdline: ''iommu=soft''
 +  - [[https://forum.proxmox.com/threads/pci-passthrough-amd-vi-completion-wait-loop-timed-out.75832/|PCI Passthrough: AMD-Vi: Completion-Wait loop timed out]]
 +
 +==== bond and bridge ====
 +
 +There is issue where Emulex is a bond0 slave and bond0 is part of a bridge.
 +
 +More here: [[vm:proxmox:issues:no_dhcp_offer]]
 +
 +**Solution:** upgrade to latest firmware.
 +
 +
 +===== Firmware =====
 +
 +==== IBM ====
 +
 +Latest published (from 2011): [[https://www.ibm.com/support/pages/support-cd-release-emulex-based-10gbe-v5142-ibm-system-x|
 +Support CD Release for Emulex-based 10GbE v5.1.42 - IBM System x
 +]]
 +
 +
 +==== Broadcom ====
 +
 +Please read user guide: 
 +  - [[https://docs.broadcom.com/doc/12398083|Emulex® Drivers for Linux for OneConnect®
 +Adapters - User Guide v 12.6]]
 +  - https://docs.broadcom.com/doc/12378876|Emulex® Drivers for Linux for OneConnect®
 +Adapters - User Guide v 11.2]]
 +
 +
 +Download firmware upgrade from:
 +  - [[https://www.broadcom.com/|Broadcom]] page. Look for ''OneConnect Flash ISO Image''
 +
 +
 +https://www.dell.com/support/home/pl-pl/drivers/driversdetails?driverid=1dhk6
 +
 +Issues
 +
 +[[https://community.hpe.com/t5/networking/frame-error-with-driver-be2net-amp-emulex-oneconnect-10gb/td-p/4792712|Frame error with driver be2net & Emulex OneConnect 10Gb]]
 +
 +=== fw versions ===
 +  - 4.1.455.10 - on old i7 machine - bond and bridge works OK
 +  - 4.6.281.21 - on old i5 machine - bond and bridge works OK
 +  - 4.6.281.8 - on AMD Ryzne - bond and bridge broadcast issue present
 +  - 11.4.1186.4 - latest firmware - works OK
 +
 +=== fw upgrade ===
 +
 +<code bash>
 +ethtool -i enp37s0f0
 +
 +driver: be2net
 +version: 5.15.30-2-pve
 +firmware-version: 4.6.281.8
 +expansion-rom-version: 
 +bus-info: 0000:25:00.0
 +supports-statistics: yes
 +supports-test: yes
 +supports-eeprom-access: yes
 +supports-register-dump: no
 +supports-priv-flags: yes
 +
 +</code>
 +
 +Flashin firmware with ethtool. Note - process takes long time~
 +
 +<code bash>
 +cp oc11-11.4.1186.4.ufi /lib/firmware/
 +ethtool --flash enp37s0f0 oc11-11.4.1186.4.ufi
 +ethtool --flash enp37s0f1 oc11-11.4.1186.4.ufi
 +</code>
 +
 +<code bash>
 +# ethtool -i enp37s0f0 
 +driver: be2net
 +version: 5.15.30-2-pve
 +firmware-version: 4.6.281.8 [11.4.1186.4]
 +expansion-rom-version: 
 +bus-info: 0000:25:00.0
 +supports-statistics: yes
 +supports-test: yes
 +supports-eeprom-access: yes
 +supports-register-dump: no
 +supports-priv-flags: yes
 +</code>
 +
 +Apply new firmware - reboot the system.
 +
 +
 +