Homelab Router OPNsense and IPv6

Current IPv6 setting on TP-Link Router, connecting to Buddy Telco / Aussie Broadband ISP.

OPNsense - TP-Link IPv6

Setup OPNsense VM

Create a new VM for OPNsense in Proxmox:

OPNsense - VM General

OPNsense - VM OS

OPNsense - VM System

OPNsense - VM Disks

OPNsense - VM CPU

OPNsense - VM Memory

OPNsense - VM Network WAN

OPNsense - VM Network LAN

OPNsense - VM Network OPT1

OPNsense - VM In Proxmox

OPNsense - Proxmox Network

Setup IPv6

Step by step to enable IPv6 with Buddy Telco / Aussie Broadband ISP.

OPNsense - Interfaces Settings

OPNsense - WAN

  • Enable DHCP/DHCPv6 for IPv4/IPv6 Configuration Type
  • Set ISP’s Prefix delegation size to 48
  • Enable Send prefix hint

OPNsense - Router Advertisements

  • Set Router Advertisements to Unmanaged
  • Set Router Priority to Normal

Setup OPNsense Optional Port

For example, setup OPT1 interface after OPNsense installed in Proxmox.

  • Enable OPT1 Interface
  • Select Static IPv4 for IPv4 Configuration Type
  • Select Track Interface for IPv6 Configuration Type
  • IPv4 address set to 192.168.2.1/24
  • IPv6 parent interface set to WAN
  • IPv6 prefix ID set to 0x1. 0x0 has been assigned to LAN
  • Enable Allow manual adjustment of DHCPv6 and Router Advertisements

OPNsense - OPT1 Interface

  • Enable DHCP server on the OPT1 interface
  • Set IP range from 192.168.2.10 to 192.168.2.245

OPNsense - OPT1 DHCP

  • Add Firewall Rule
  • Set TCP/IP version
  • Select OPT1 net from dropdown menu as Source

OPNsense - OPT1 Firewall Rule

  • Setup Firewall Rule for IPv4 on OPT1 interface, then
  • Setup Firewall Rule for IPv6 on OPT1 interface

OPNsense - OPT1 Firewall

VLAN

Setup Trusted and Untrusted VLAN.

OPNsense - VLAN

OPNsense - VLAN Interfaces

OPNsense - Trusted VLAN

OPNsense - Untrusted VLAN

Then,

  • Set Router Advertisements to Unmanaged
  • Set Router Priority to Normal

for both VLANs.

Running

After all the change made above, reboot OPNsense instance.

OPNsense - Run In Proxmox

OPNsense - Interfaces Overview

OPNsense - Dashboard

IPv6 Test

OPNsense - Test IPv6

OPNsense - IPv6 Test

References

OPNsense - IPv6 Chart

OPNsense - IPv4 CIDR Chart

Turn On and Turn Off Recall in Windows 11

Turn On Recall feature in Windows 11 24H2, by running Widnows Power Shell as Administrator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
PS C:\> DISM /Online /Get-FeatureInfo /FeatureName:Recall

Deployment Image Servicing and Management tool
Version: 10.0.26100.1150

Image Version: 10.0.26100.2033

Feature Information:

Feature Name : Recall
Display Name : Recall
Description : Recall application.
Restart Required : Possible
State : Enabled

Custom Properties:

(No custom properties found)

The operation completed successfully.

Turn Off Recall in Windows 11 24H2:

1
2
3
4
5
6
7
8
9
10
PS C:\> DISM /Online /Disable-Feature /FeatureName:Recall

Deployment Image Servicing and Management tool
Version: 10.0.26100.1150

Image Version: 10.0.26100.2033

Disabling feature(s)
[==========================100.0%==========================]
The operation completed successfully.

Running latest MacOS Sequoia in VMware on Windows 11

This is the step by step guide, the fastest and easiest way install and run latest MacOS Sequoia 15.0.1 in VMware Workstation Pro on Windows 11.

Due to MacOS Sequoia has added detection whether OS is running in Virtual Machine, so it’s better install MacOS Sonoma at first, then upgrade to latest version MacOS Sequoia.

VMware - Apple ID

  • Build a bootable MacOS ISO image

Do it on a Mac.

Clone gibmasOS repo https://github.com/corpnewt/gibMacOS and run:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
$ ./gibMacOS.command
#######################################################
# gibMacOS #
#######################################################

Available Products:

1. macOS Sequoia 15.0.1 (24A348)
- 072-01382 - Added 2024-10-03 21:26:40 - 14.48 GB
2. macOS Ventura 13.7 (22H123)
- 062-78643 - Added 2024-09-16 17:44:05 - 12.22 GB
3. macOS Sonoma 14.7 (23H124)
- 062-78824 - Added 2024-09-16 17:42:25 - 13.68 GB
4. macOS Sequoia 15.0 (24A335)
- 062-78429 - Added 2024-09-16 17:30:21 - 14.48 GB
5. macOS Monterey 12.7.6 (21H1320)
- 062-40406 - Added 2024-08-14 20:45:56 - 12.42 GB

...

24. macOS High Sierra 10.13.6 (17G66)
- 041-91758 - Added 2019-10-19 18:19:55 - 5.71 GB
25. macOS Mojave 10.14.6 (18G103)
- 061-26589 - Added 2019-10-14 20:51:08 - 6.52 GB
26. macOS Mojave 10.14.5 (18F2059)
- 061-26578 - Added 2019-10-14 20:38:26 - 6.52 GB

M. Change Max-OS Version (Currently 12)
C. Change Catalog (Currently publicrelease)
I. Only Print URLs (Currently Off)
S. Set Current Catalog to SoftwareUpdate Catalog
L. Clear SoftwareUpdate Catalog
R. Toggle Recovery-Only (Currently Off)
U. Show Catalog URL
Q. Quit

Please select an option: 3

Downloading InstallAssistant.pkg for 062-78824 - 14.7 macOS Sonoma (23H124)...

1.35 GB/14.48 GB | = 9.34% | 101.7 MB/s | 00:02:10 left

Succeeded:
InstallAssistant.pkg
MajorOSInfo.pkg
com_apple_MobileAsset_MacSoftwareUpdate.plist
InstallInfo.plist
UpdateBrain.zip

Failed:
None

Files saved to:
/Users/terrence/Projects/gibMacOS/macOS Downloads/publicrelease/062-78824 - 14.7 macOS Sonoma (23H124)

Run InstallAssistant.pkg from above MacOS download directory. It will be using to build ISO image.

Create a disk image with size 16GB:

1
2
$ hdiutil create -o /tmp/MacOS -size 16000m -volname MacOS -layout SPUD -fs HFS+J
created: /tmp/MacOS.dmg

Mount the disk image created above:

1
2
3
4
 $ hdiutil attach /tmp/MacOS.dmg -noverify -mountpoint /Volumes/MacOSISO
/dev/disk6 Apple_partition_scheme
/dev/disk6s1 Apple_partition_map
/dev/disk6s2 Apple_HFS /Volumes/MacOSISO

Create ISO image from Install masOS app:

1
2
3
4
5
6
7
$ sudo /Applications/Install\ macOS\ Sonoma.app/Contents/Resources/createinstallmedia --volume /Volumes/MacOSISO --nointeraction
Erasing disk: 0%... 10%... 20%... 30%... 100%
Copying essential files...
Copying the macOS RecoveryOS...
Making disk bootable...
Copying to disk: 0%... 10%... 20%... 30%... 40%... 50%... 60%... 100%
Install media now available at "/Volumes/Install macOS Sonoma"

Unmount disk image and convert to an ISO image:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ hdiutil detach -force /Volumes/Install\ macOS\ Sonoma
"disk6" ejected.

$ ls -al /tmp/MacOS.dmg
-rw-r--r--@ 1 terrence wheel 16777216000 15 Oct 21:54 /tmp/MacOS.dmg

$ hdiutil convert /tmp/MacOS.dmg -format UDTO -o /tmp/MacOS-Sonoma-14.7.cdr
Reading Driver Descriptor Map (DDM : 0)…
Reading Apple (Apple_partition_map : 1)…
Reading (Apple_Free : 2)…
Reading disk image (Apple_HFS : 3)…
........................................
Elapsed Time: 19.139s
Speed: 835.9MB/s
Savings: 0.0%
created: /tmp/MacOS-Sonoma-14.7.cdr

$ mv /tmp/MacOS-Sonoma-14.7.cdr /tmp/MacOS-Sonoma-14.7.iso

$ rm /tmp/MacOS.dmg
  • Install VMware Workstation Pro

Download VMware Workstation Pro from e.g. https://softwareupdate.vmware.com/cds/vmw-desktop/ws/17.6.1/24319023/windows/core/VMware-workstation-17.6.1-24319023.exe.tar

  • Patch VMware Workstation Pro

Clone unlocker https://github.com/paolo-projects/unlocker repo on Windows and enable Apple macOS option in VMware Workstation Pro:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
PS C:\Projects\unlocker> .\win-install.cmd

Unlocker 3.0.2 for VMware Workstation
=====================================
(c) Dave Parsons 2011-18

Set encoding parameters...
Active code page: 850

VMware is installed at: C:\Program Files (x86)\VMware\VMware Workstation\
VMware product version: 17.6.1.24319023

Stopping VMware services...

...

Starting VMware services...

Finished!

VMware - Apple macOS

  • Add VMware Tools

Copy darwin.iso and darwinPre15.iso files extracted from VMware Fusion e.g. _https://softwareupdate.vmware.com/cds/vmw-desktop/fusion/12.2.5/20904517/x86/core/com.vmware.fusion.zip.tar _ into VMware Workstation Pro directory:

1
2
C:\Projects\unlocker\tools\darwin.iso -> C:\Program Files (x86)\VMware\VMware Workstation\darwin.iso
C:\Projects\unlocker\tools\darwinPre15.iso -> C:\Program Files (x86)\VMware\VMware Workstation\darwinPre15.iso
  • Create Virtual Machine for MacOS 14 Sonoma and update the settings

Add:

1
smc.version = "0"

into MacOS Sonoma.vmx file.

Clone GenSMBIOS repo https://github.com/corpnewt/GenSMBIOS and generate serial number on Windows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ ./GenSMBIOS.bat
#######################################################
# GenSMBIOS #
#######################################################

MacSerial not found!
Remote Version v2.1.8
Current plist: None
Plist type: Unknown

1. Install/Update MacSerial
2. Select config.plist
3. Generate SMBIOS
4. Generate UUID
5. Generate ROM
6. List Current SMBIOS
7. Generate ROM With SMBIOS (Currently Enabled)

Q. Quit

Please select an option: 3

Please type the SMBIOS to gen and the number
of times to generate [max 20] (i.e. iMac18,3 5): MacBookPro16,4

Type: MacBookPro16,4
Serial: C..........T
Board Serial: C0.............FB
SmUUID: A0D50403-F256-4E17-A2EC-29964D889A1D
Apple ROM: 6..........7

Copy Serial, Board Serial and Apple ROM number, apply to:

1
2
3
4
5
6
7
8
9
10
board-id = "Mac-A61BADE1FDAD7B05"
hw.model.reflectHost = "FALSE"
hw.model = "MacBookPro16,4"
serialNumber.reflectHost = "FALSE"
serialNumber = "C..........T"
smbios.reflectHost = "FALSE"
efi.nvram.var.ROM.reflectHost = "FALSE"
efi.nvram.var.MLB.reflectHost = "FALSE"
efi.nvram.var.ROM = "6..........7"
efi.nvram.var.MLB = "C0.............FB"

then add above block into MacOS Sonoma.vmx file.

Based on Apple Ethernet MAC Address range https://hwaddress.com/company/apple-inc/, change and add network settings from:

1
ethernet0.addressType = "generated"

to:

1
2
3
ethernet0.addressType = "static"
ethernet0.address = "00:21:E9:c0:92:76"
ethernet0.checkMacAddress = "FALSE"

in MacOS Sonoma.vmx file.

VMware - MacOS Sonoma

  • Mount MacOS ISO image and install

VMware - MacOS Sonoma settings

DON’T enable Location Service during the installation! Otherwise, you can’t setup Time Zone, Date Time based on your area. You can login with your Apple ID during the installation.

VMware - Apple ID login

  • Install VMware Tools

After MacOS installed and VM restarted, mount darwin.iso and install VMware Tools. Then Display Memory in MacOS becomes 128 MB, and support the Full Screen mode.

  • Upgrade MacOS

After MacOS Sonoma installed, make sure everything is OK, then copy the whole MacOS Sonoma directory to a new directory MacOS Sequoia (Upgraded). Open the new directory in VMware, and select I copied it, then you can upgrade MacOS to latest verion in System Settings -> Software Update.

  • Run MacOS in VMware Player:

VMware - Player

VMware - In Action

References

How to access Ollama runs on localhost from internet

By default, Ollama is only accessible from localhost. CORS error and HTTP 403 returned if try to invoke from other hosts.

To enable Ollama can be visited from hosts on intranet, run:

1
$ launchctl setenv OLLAMA_HOST "0.0.0.0"

in MacOS. Then restart Ollama. Then invoke it from hosts on intranet.

To enable Ollama can be visited from internet, with the help from ngrok, run:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ngrok http 11434

ngrok (Ctrl+C to quit)

Session Status online
Account Terrence Miao (Plan: Free)
Version 3.17.0
Region Australia (au)
Web Interface http://127.0.0.1:4040
Forwarding https://294b-2403-5802-1c44-0-341c-aae8-a501-8b73.ngrok-free.app -> http://localhost:11434

Connections ttl opn rt1 rt5 p50 p90
0 0 0.00 0.00 0.00 0.00

Then can visit Ollama on localhost from internet. 11434 is default port of Ollama.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ curl --location 'https://294b-2403-5802-1c44-0-341c-aae8-a501-8b73.ngrok-free.app/v1/models'

{
"object": "list",
"data": [
{
"id": "codellama:latest",
"object": "model",
"created": 1728736149,
"owned_by": "library"
},
{
"id": "deepseek-coder-v2:latest",
"object": "model",
"created": 1728735883,
"owned_by": "library"
},
{
"id": "mxbai-embed-large:latest",
"object": "model",
"created": 1713401203,
"owned_by": "library"
},
{
"id": "nomic-embed-text:latest",
"object": "model",
"created": 1708780687,
"owned_by": "library"
}
]
}

To write some code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
$ curl --location 'https://294b-2403-5802-1c44-0-341c-aae8-a501-8b73.ngrok-free.app/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
"model": "deepseek-coder-v2",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Write a function that outputs the fibonacci sequence in TypeScript."
}
]
}'

{
"id": "chatcmpl-174",
"object": "chat.completion",
"created": 1728738926,
"model": "deepseek-coder-v2",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "
Certainly! Below is a TypeScript function that outputs the Fibonacci sequence up to a specified number of terms:

This function starts with the initial terms of the Fibonacci sequence `[0, 1]` and then iteratively calculates each subsequent term
by summing the two preceding terms. The function returns an array containing the first `n` terms of the Fibonacci sequence.
"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 29,
"completion_tokens": 243,
"total_tokens": 272
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
function fibonacciSequence(n: number): number[] {
if (n <= 0) return [];
if (n === 1) return [0];

const result: number[] = [0, 1];

for (let i = 2; i < n; i++) {
result.push(result[i - 1] + result[i - 2]);
}

return result;
}

// Example usage: console.log(fibonacciSequence(10));
// Output: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Check Ollama log:

1
$ tail -f ~/.ollama/logs/server.log

References

Running a Mac app from Unidentified Developer on MacOS Sequoia

Latest MacOS Sequoia has made another security enforcement. To open and run a Mac application, e.g. VLC nightly build at https://nightlies.videolan.org/, Unidentified Developer , the following steps can take.

  • Enable “Allow Applications from Anywhere”
1
2
3
4
base ~
sudo spctl --master-disable
Password:
Globally disabling the assessment system needs to be confirmed in System Settings.

Go to System Settings -> Privacy & Security, select “Allow Applications from Anywhere”:

MacOS - Allow Applications from Anywhere

1
2
3
4
base ~
sudo spctl --master-enable
Password:
This operation is no longer supported. Please see the man page for more information.
  • Disable MacOS Quarantine on the application

In this case application VLC:

1
2
base ~
xattr -dr com.apple.quarantine /Applications/VLC.app

Configure and setup Mellanox network adapter

A Mellanox ConnectX-4 MCX455A-ECAT PCIe x16 3.0 100GBe VPI EDR IB network adapter. Enable VT-d(Intel Virtualization Technology for Directed I/O), SR-IOV (Single Root IO Virtualization), and the number of Virtual Functions in Chipset and Network in BIOS.

The following steps are made in a Proxmox Virtual Environment (PVE).

Configuration and Setup

Go to NVIDIA Firmware Tools (MFT) https://network.nvidia.com/products/adapter-software/firmware-tools/ and download the MFT; go to Updating Firmware for ConnectX®-4 VPI PCI Express Adapter Cards (InfiniBand, Ethernet, VPI) https://network.nvidia.com/support/firmware/connectx4ib/ and download the updated firmware e.g. mft-4.26.1-6-x86_64-deb.tgz.

After installation, start up Mellanox Software Tools service:

1
2
3
4
5
6
root@pve:~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success

Check status:

1
2
3
4
5
6
7
8
9
10
11
root@pve:~# mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4115_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:06:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 00

Query Mellanox network adapter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
root@pve:~# flint -d /dev/mst/mt4115_pciconf0 query
Image type: FS3
FW Version: 12.28.2006
FW Release Date: 15.9.2020
Product Version: 12.28.2006
Rom Info: type=UEFI version=14.21.17 cpu=AMD64
type=PXE version=3.6.102 cpu=AMD64
Description: UID GuidsNumber
Base GUID: ec0d9a030076eae2 4
Base MAC: ec0d9a76eae2 4
Image VSD: N/A
Device VSD: N/A
PSID: LNV2180110032
Security Attributes: N/A

Check Mellanox network adapter configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
root@pve:~# mlxconfig -d /dev/mst/mt4115_pciconf0 query
Device #1:
----------

Device type: ConnectX4
Name: 00KH925_Ax
Description: Mellanox ConnectX-4 EDR IB VPI Single-port x16 PCIe 3.0 HCA
Device: /dev/mst/mt4115_pciconf0

Configurations:
LINK_TYPE_P1 ETH(2)
SRIOV_EN True(1)
NUM_OF_VFS 8
...

Verify network adapter configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
terrence@pve:~# ethtool enp1s0np0
Settings for enp1s0np0:
Supported ports: [ Backplane ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
56000baseKR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None RS BASER
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
56000baseKR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: RS
Speed: 100000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Link detected: yes

Virtualization

1
2
3
4
5
6
7
8
root@pve:~# lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

root@pve:~# ip link show
...
4: enp1s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
link/ether ec:0d:9a:76:eb:2a brd ff:ff:ff:ff:ff:ff
...

Write to the sysfs file the number of Virtual Functions:

1
root@pve:~# echo 8 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs

Verify that the Virtual Functions were created:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@pve:~# lspci | grep Mellanox
01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
01:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.4 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.5 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.6 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.7 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:01.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

root@pve:~# ip link show
...
4: enp1s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
link/ether ec:0d:9a:76:eb:2a brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 6 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 7 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
...

Persistence

Print out device information:

1
2
3
4
5
6
7
8
root@pve:~# udevadm info /sys/class/infiniband/mlx5_0
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/infiniband/mlx5_0
M: mlx5_0
R: 0
U: infiniband
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/infiniband/mlx5_0
E: SUBSYSTEM=infiniband
E: NAME=mlx5_0

Make the network adapter Virtual Functions persistent after reboot:

1
2
root@pve:~# cat /etc/udev/rules.d/mlx.rules
ACTION=="add", SUBSYSTEM=="infiniband", ENV{NAME}=="mlx5_0", ATTR{device/sriov_numvfs}="8"

Network in Proxmox

Add virtualised network adapter in Resource Mappings in Proxmox:

Mellanox - Resource Mappings

List PCI Devices in Resource Mappings in Proxmox which support Virtual Functions:

Mellanox - Resource Mappings PCI Devices

Add PCI device in Resource Mappings in Proxmox in VM:

Mellanox - Network PCI Device

References

Fix network object name already existed issue in Windows

When rename a network adapter in Windows:

1
PS C:\> Rename-NetAdapter -Name Ethernet -NewName Mellanox

an error Rename-NetAdapter : {Object Exists} An attempt was made to create an object and the object name already existed thrown.

Work around solution is:

  1. Open Device Manager in Windows Control Panel
  2. Under menu View enable Show hidden devices
  3. Uninstall the old network adapter with the old name
  4. Then rename the network adapter again

Rename Network in Windows

References

How to enable SMB Direct client/server in Windows 11 Pro for Workstations

In Windows 11 Pro Station, a Mellanox ConnectX-4 MCX455A-ECAT PCIe x16 3.0 100GBe VPI EDR IB network adatper, goes to support SMB Direct, client and server side SMB Multichannel and RDMA (Remote Direct Memory Access):

Windows 11 Pro for Workstations

Open Windows Terminal as Administrator.

Enable SMB Direct:

1
2
3
4
PS C:\> Enable-WindowsOptionalFeature -Online -FeatureName SMBDirect
Path :
Online : True
RestartNeeded : False

Enable SMB Multichannel on the client-side:

1
2
3
4
5
6
7
8
9
10
PS C:\> Set-SmbClientConfiguration -EnableMultiChannel $true
Confirm
Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Client Configuration'.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"):

PS C:\> Get-SmbClientConfiguration
...
EnableMultiChannel : True
...

Enable SMB Multichannel on the server-side:

1
2
3
4
5
6
7
8
9
10
PS C:\> Set-SmbServerConfiguration -EnableMultiChannel $true
Confirm
Are you sure you want to perform this action?
Performing operation 'Modify' on Target 'SMB Server Configuration'.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"):

PS C:\> Get-SmbServerConfiguration
...
EnableMultiChannel : True
...

Enable RDMA for a specific interface:

1
PS C:\> Enable-NetAdapterRDMA Mellanox

Verify which state of operability SMB Direct is currently configured to:

1
2
3
4
5
6
7
PS C:\> Get-WindowsOptionalFeature -Online -FeatureName SMBDirect
FeatureName : SmbDirect
DisplayName : SMB Direct
Description : Remote Direct Memory Access (RDMA) support for the SMB 3.x file sharing protocol
RestartRequired : Possible
State : Enabled
CustomProperties :
1
2
3
4
5
6
7
8
9
10
PS C:\> Get-SmbClientNetworkInterface
Interface Index RSS Capable RDMA Capable Speed IpAddresses Friendly Name
--------------- ----------- ------------ ----- ----------- -------------
22 True True 100 Gbps {fe80::708:c529:1bcb:2432, 192.168.68.67} Mellanox

PS C:\> Get-SmbServerNetworkInterface
Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 22 True True 100 Gbps fe80::708:c529:1bcb:2432
* 22 True True 100 Gbps 192.168.68.67

Have a look TrueNAS disk speed benchmark, over a 100Gbps ethernet network, from Windows 11 Pro for Workstations with SMB Direct, client/server SMB Multichannel and RDMA enabled:

TrueNAS disk speed benchmark

In Windows Server 2022, with SMB shared folder in Storage Spaces:

Windows Server 2022

Run Windows Powershell as Administrator user , which RDMA Capable are all True for both SMB client/server:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\Users\Administrator> Get-SmbServerNetworkInterface
Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 5 True True 100 Gbps fe80::9ee0:7f4c:5128:863b
* 5 True True 100 Gbps 192.168.68.66

PS C:\Users\Administrator> Get-SmbClientNetworkInterface
Interface Index RSS Capable RDMA Capable Speed IpAddresses Friendly Name
--------------- ----------- ------------ ----- ----------- -------------
5 True True 100 Gbps {fe80::9ee0:7f4c:5128:863b, 192.168.68.66} Mellanox

Have a look Windows Server 2022 disk speed benchmark, over a 100Gbps ethernet network, from Windows 11 Pro for Workstations with SMB Direct, client/server SMB Multichannel and RDMA enabled:

Windows Server 2022 disk speed benchmark

References