Configure and setup Mellanox network adapter

Guestbook

A Mellanox ConnectX-4 MCX455A-ECAT PCIe x16 3.0 100GBe VPI EDR IB network adapter. Enable VT-d(Intel Virtualization Technology for Directed I/O), SR-IOV (Single Root IO Virtualization), and the number of Virtual Functions in Chipset and Network in BIOS.

The following steps are made in a Proxmox Virtual Environment (PVE).

Configuration and Setup

Go to NVIDIA Firmware Tools (MFT) https://network.nvidia.com/products/adapter-software/firmware-tools/ and download the MFT; go to Updating Firmware for ConnectX®-4 VPI PCI Express Adapter Cards (InfiniBand, Ethernet, VPI) https://network.nvidia.com/support/firmware/connectx4ib/ and download the updated firmware e.g. mft-4.26.1-6-x86_64-deb.tgz.

After installation, start up Mellanox Software Tools service:

1
2
3
4
5
6
root@pve:~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success

Check status:

1
2
3
4
5
6
7
8
9
10
11
root@pve:~# mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4115_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:06:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00

Query Mellanox network adapter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
root@pve:~# flint -d /dev/mst/mt4115_pciconf0 query
Image type:            FS3
FW Version:            12.28.2006
FW Release Date:       15.9.2020
Product Version:       12.28.2006
Rom Info:              type=UEFI version=14.21.17 cpu=AMD64
                       type=PXE version=3.6.102 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             ec0d9a030076eae2        4
Base MAC:              ec0d9a76eae2            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  LNV2180110032
Security Attributes:   N/A

Check Mellanox network adapter configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
root@pve:~# mlxconfig -d /dev/mst/mt4115_pciconf0 query
Device #1:
----------

Device type:        ConnectX4
Name:               00KH925_Ax
Description:        Mellanox ConnectX-4 EDR IB VPI Single-port x16 PCIe 3.0 HCA
Device:             /dev/mst/mt4115_pciconf0

Configurations:
        LINK_TYPE_P1                                ETH(2)
        SRIOV_EN                                    True(1)
        NUM_OF_VFS                                  8
...

Verify network adapter configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
terrence@pve:~# ethtool enp1s0np0
Settings for enp1s0np0:
	Supported ports: [ Backplane ]
	Supported link modes:   1000baseKX/Full
	                        10000baseKR/Full
	                        40000baseKR4/Full
	                        40000baseCR4/Full
	                        40000baseSR4/Full
	                        40000baseLR4/Full
	                        56000baseKR4/Full
	                        25000baseCR/Full
	                        25000baseKR/Full
	                        25000baseSR/Full
	                        50000baseCR2/Full
	                        50000baseKR2/Full
	                        100000baseKR4/Full
	                        100000baseSR4/Full
	                        100000baseCR4/Full
	                        100000baseLR4_ER4/Full
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: None	 RS	 BASER
	Advertised link modes:  1000baseKX/Full
	                        10000baseKR/Full
	                        40000baseKR4/Full
	                        40000baseCR4/Full
	                        40000baseSR4/Full
	                        40000baseLR4/Full
	                        56000baseKR4/Full
	                        25000baseCR/Full
	                        25000baseKR/Full
	                        25000baseSR/Full
	                        50000baseCR2/Full
	                        50000baseKR2/Full
	                        100000baseKR4/Full
	                        100000baseSR4/Full
	                        100000baseCR4/Full
	                        100000baseLR4_ER4/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: RS
	Speed: 100000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Direct Attach Copper
	PHYAD: 0
	Transceiver: internal
	Link detected: yes

Virtualization

1
2
3
4
5
6
7
8
root@pve:~# lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

root@pve:~# ip link show
...
4: enp1s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
    link/ether ec:0d:9a:76:eb:2a brd ff:ff:ff:ff:ff:ff
...

Write to the sysfs file the number of Virtual Functions:

1
root@pve:~# echo 8 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs

Verify that the Virtual Functions were created:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@pve:~# lspci | grep Mellanox
01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
01:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.4 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.5 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.6 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.7 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:01.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

root@pve:~# ip link show
...
4: enp1s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
    link/ether ec:0d:9a:76:eb:2a brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
...

Persistence

Print out device information:

1
2
3
4
5
6
7
8
root@pve:~# udevadm info /sys/class/infiniband/mlx5_0
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/infiniband/mlx5_0
M: mlx5_0
R: 0
U: infiniband
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/infiniband/mlx5_0
E: SUBSYSTEM=infiniband
E: NAME=mlx5_0

Make the network adapter Virtual Functions persistent after reboot:

1
2
root@pve:~# cat /etc/udev/rules.d/mlx.rules
ACTION=="add", SUBSYSTEM=="infiniband", ENV{NAME}=="mlx5_0", ATTR{device/sriov_numvfs}="8"

Network in Proxmox

Add virtualised network adapter in Resource Mappings in Proxmox:

Mellanox - Resource Mappings

List PCI Devices in Resource Mappings in Proxmox which support Virtual Functions:

Mellanox - Resource Mappings PCI Devices

Add PCI device in Resource Mappings in Proxmox in VM:

Mellanox - Network PCI Device

References

Fix network object name already existed issue in Windows

Guestbook

When rename a network adapter in Windows:

1
PS C:\> Rename-NetAdapter -Name Ethernet -NewName Mellanox

an error Rename-NetAdapter : {Object Exists} An attempt was made to create an object and the object name already existed thrown.

Work around solution is:

  1. Open Device Manager in Windows Control Panel
  2. Under menu View enable Show hidden devices
  3. Uninstall the old network adapter with the old name
  4. Then rename the network adapter again

Rename Network in Windows

References

How to enable SMB Direct client/server in Windows 11 Pro for Workstations

Guestbook

In Windows 11 Pro Station, a Mellanox ConnectX-4 MCX455A-ECAT PCIe x16 3.0 100GBe VPI EDR IB network adatper, goes to support SMB Direct, client and server side SMB Multichannel and RDMA (Remote Direct Memory Access):

Windows 11 Pro for Workstations

Open Windows Terminal as Administrator.

Enable SMB Direct:

1
2
3
4
PS C:\> Enable-WindowsOptionalFeature -Online -FeatureName SMBDirect
Path          :
Online        : True
RestartNeeded : False

Enable SMB Multichannel on the client-side:

1
2
3
4
5
6
7
8
9
10
PS C:\> Set-SmbClientConfiguration -EnableMultiChannel $true
Confirm
Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Client Configuration'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

PS C:\> Get-SmbClientConfiguration
...
EnableMultiChannel                    : True
...

Enable SMB Multichannel on the server-side:

1
2
3
4
5
6
7
8
9
10
PS C:\> Set-SmbServerConfiguration -EnableMultiChannel $true
Confirm
Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Server Configuration'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

PS C:\> Get-SmbServerConfiguration
...
EnableMultiChannel                     : True
...

Enable RDMA for a specific interface:

1
PS C:\> Enable-NetAdapterRDMA Mellanox

Verify which state of operability SMB Direct is currently configured to:

1
2
3
4
5
6
7
PS C:\> Get-WindowsOptionalFeature -Online -FeatureName SMBDirect
FeatureName      : SmbDirect
DisplayName      : SMB Direct
Description      : Remote Direct Memory Access (RDMA) support for the SMB 3.x file sharing protocol
RestartRequired  : Possible
State            : Enabled
CustomProperties :
1
2
3
4
5
6
7
8
9
10
PS C:\> Get-SmbClientNetworkInterface
Interface Index RSS Capable RDMA Capable Speed       IpAddresses                                 Friendly Name
--------------- ----------- ------------ -----       -----------                                 -------------
22              True        True         100 Gbps    {fe80::708:c529:1bcb:2432, 192.168.68.67}   Mellanox

PS C:\> Get-SmbServerNetworkInterface
Scope Name Interface Index RSS Capable RDMA Capable Speed    IpAddress
---------- --------------- ----------- ------------ -----    ---------
*          22              True        True         100 Gbps fe80::708:c529:1bcb:2432
*          22              True        True         100 Gbps 192.168.68.67

Have a look TrueNAS disk speed benchmark, over a 100Gbps ethernet network, from Windows 11 Pro for Workstations with SMB Direct, client/server SMB Multichannel and RDMA enabled:

TrueNAS disk speed benchmark

In Windows Server 2022, with SMB shared folder in Storage Spaces:

Windows Server 2022

Run Windows Powershell as Administrator user , which RDMA Capable are all True for both SMB client/server:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\Users\Administrator> Get-SmbServerNetworkInterface
Scope Name Interface Index RSS Capable RDMA Capable Speed    IpAddress
---------- --------------- ----------- ------------ -----    ---------
*          5               True        True         100 Gbps fe80::9ee0:7f4c:5128:863b
*          5               True        True         100 Gbps 192.168.68.66

PS C:\Users\Administrator> Get-SmbClientNetworkInterface
Interface Index RSS Capable RDMA Capable Speed    IpAddresses                                Friendly Name
--------------- ----------- ------------ -----    -----------                                -------------
5               True        True         100 Gbps {fe80::9ee0:7f4c:5128:863b, 192.168.68.66} Mellanox

Have a look Windows Server 2022 disk speed benchmark, over a 100Gbps ethernet network, from Windows 11 Pro for Workstations with SMB Direct, client/server SMB Multichannel and RDMA enabled:

Windows Server 2022 disk speed benchmark

References

Fixing TrueNAS SMB IP binding

Guestbook

After changing network settings, TrueNAS IP address has been updated. If modify SMB configuration, error like:

1
smb_update.bindip.0: IP address [192.168.0.51] is not a configured address for this server

thrown.

To reset and clean up already bind IP, login TrueNAS Console and run:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
root@TrueNAS[~]# midclt call smb.update '{"bindip": []}'
{
    "id": 1,
    "netbiosname": "TRUENAS",
    "netbiosalias": [],
    "workgroup": "IGLOO STUDIO",
    "description": "TrueNAS Server",
    "unixcharset": "UTF-8",
    "loglevel": "MINIMUM",
    "syslog": false,
    "aapl_extensions": true,
    "Localmaster": true,
    "guest": "nobody",
    "filemask"': "",
    "dirmask"': "",
    "smb_options": "",
    "bindip": [],
    "cifs_SID": "S-1-5-21-2487580926-3122677641-100607549",
    "ntImv1_auth": false,
    "enable_smb1": false,
    "admin_group": null,
    "next_rid": 0,
    "multichannel": true,
    "netbiosname_local": "TRUENAS"
}

then can modify SMB configuration successfully.

References

Running MacOS on Proxmox

Guestbook

Following the instructions to run the latest MacOS, Sonoma 14.6.1, on Proxmox - The Definitive Guide to Running MacOS in Proxmox.

Proxmox - MacOS

A few more put on notes:

  • You can login with Apple ID
  • Modify config.plist file under EFI folder -> OC folder, change the default Screen Resolution from 1080P 1920x1080@32 to 2K 2560x1440@32. Look more information in Configuration.pdf file in KVM-Opencore release

Connect with Apple Remote Desktop, so can use exact same Apple Keyboard mapping:

Proxmox - Apple Remote Desktop

Current KVM-Opencore hasn’t setup support for Audio device and GPU accelerator.

Also try to setup Parsec in MacOS on Proxmox, without success.

Proxmox - Parsec

Setup MikroTik CRS504-4XQ-IN and run a speed test

Guestbook

MikroTik CRS504-4XQ-IN, the Cloud Switch can handle FOUR QSFP28 100Gbps ports, equal to 16 x 25Gbps bandwidth.

MikroTik - Interfaces

Setup single link mode, only the first QSFP28 sub-interface needs to be configured, while the remaining three sub-interfaces should remain enabled. For example, connect Mellanox MCX455A-ECAT ConnectX-4 InfiniBand/Ethernet adapter card (EDR IB 100Gbps and 100GbE, single-port QSFP28, PCIe 3.0x16) using ONTi DAC QSFP28 100Gbps cable to the switch.

Change FEC Mode to fec91.

Ethernet Forward Error Correction (FEC) is a technique used to improve the reliability of data transmission over Ethernet networks by detecting and correcting errors in data packets. The two most common types of FEC used in Ethernet networks are CL74 and CL91.

CL74 and CL91 refer to two different types of FEC codes, each with its own characteristics and performance. Here’s a brief comparison between the two:

Code Rate:

CL91 has a higher code rate of 91.6%, which means that only 8.4% of the data transmitted is used for error correction.

In addition, setup the swith port connected to ONTi QSFP28 40Gbps TO 4SFP+ breakout cable:

1
2
3
4
5
6
$ ssh -l admin MikroTik.local

[admin@MikroTik] > /interface ethernet set qsfp28-1-1 auto-negotiation=no speed=10G-baseCR
[admin@MikroTik] > /interface ethernet set qsfp28-1-2 auto-negotiation=no speed=10G-baseCR
[admin@MikroTik] > /interface ethernet set qsfp28-1-3 auto-negotiation=no speed=10G-baseCR
[admin@MikroTik] > /interface ethernet set qsfp28-1-4 auto-negotiation=no speed=10G-baseCR

Speed test

In iperf3 server, run listens to 4 ports to manage connections in parallel:

1
$ iperf3 -s -p 5201 & iperf3 -s -p 5202 & iperf3 -s -p 5203 & iperf3 -s -p 5204 &

In a MacBook Pro with WiFi-6 connection, run:

1
$ iperf3 -c MikroTik.local -p 5201 -P 4 -t 1000

In a Mac Studio with 10Gbps Ethernet connection, run:

1
$ iperf3 -c MikroTik.local -p 5202 -P 8 -t 1000 -B 192.168.0.104

In a Windows 11 PC with 100Gbps Ethernet connection, run:

1
$ iperf3 -c MikroTik.local -p 5203 -P 2 -t 1000

Check the speed on switch console:

MikroTik - Speed

and in graph:

MikroTik - Performance

References

Setup network bond in Synology NAS

Guestbook

Synology NAS DS920+ with two build-in 1Gbps ethernet adapters, and two 2.5Gbps USB 3.0 Ethernet Adapters. Now setup network bond / link aggregation on them.

  • Enable Network Link Aggregation Mode

Synology - Network Link Aggregation Mode

  • Pickup network devices into bond

Synology - Physical devices

  • Setup network

Synology - Physical devices network setup

  • Accept network interface change after network bond

Synology - Bond warning message

  • Network bonded

Synology - Network bonded

  • Network bond service order

Synology - Network bond service order

Setup TrueNAS Network and Link Aggregation

Guestbook

Two network adapters have been setup with DHCP allocated addresses:

  1. ens18 192.168.0.246
  2. ens19 192.168.0.105
  • Create a Network Link Aggregation Interface

TrueNAS - Link Aggregation

After Link Aggregation Interface bond1 created, original two network adapters ens18 and ens19 IP addresses are gone. bond1 with the ONLY ONE network interface address for TrueNAS.

TrueNAS - Network

Samba setup and share in TrueNAS

Guestbook

TrueNAS installed and a few steps to setup to let users access shared Samba directories.

  • Enable SMB Samba service in TrueNAS

TrueNAS - Enable SMB service

TrueNAS - SMB Settings

  • Create an user account with home directory and all access permisssions

TrueNAS - Create user

  • Share TrueNAS pool created

TrueNAS - Sharing

Then can access shared directory from TrunNAS in both Windows and Mac:

TrueNAS - Access

Make sure there is NO a lock icon on the folder (can unlock the folder in MacOS Finder)