Setting up DNS Server in OPNsense

Guestbook

Want to access Homelab network switches’ web based administrator interface in a user friendly way, i.e., access via http://TL-SX3016F.local/, http://TL-SX3008F.local/ and http://MikroTik.local/.

In TP-Link Deco, which is the main Homelab network backbone, also provides DHCP service, firstly is to reserve IP addresses for these switches:

DNS - TP-Link Deco Reservation

Then setup entries in OPNsense Unbound DNS server:

DNS - Host Overrides

Then lookup a host on local network:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ cat /etc/resolv.conf
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
search local
nameserver 2403:5802:8c44:3:be24:11ff:fe21:3ce0
nameserver 192.168.68.1

$ nslookup MikroTik.local
Server:		2403:5802:8c44:3:be24:11ff:fe21:3ce0
Address:	2403:5802:8c44:3:be24:11ff:fe21:3ce0#53

Name:	MikroTik.local
Address: 192.168.68.54

Enable SR-IOV in Intel 82599ES 10Gbps Ethernet Controller

Guestbook

Intel® 82599ES 10Gbps Ethernet Controller https://ark.intel.com/content/www/us/en/ark/products/41282/intel-82599es-10-gigabit-ethernet-controller.html supports SR-IOV, which is Intel® Virtualization Technology for Connectivity (VT-c) solution.

NOTE: Intel VT Virtualization Features

  • VT-x, Intel Virtualization Technology for IA-32 and Intel 64 Processors
  • VT-d, Intel Virtualization Technology for Directed I/O
  • VT-c, Intel Virtualization Technology for Connectivity

The following step by step instructions is made in Proxmox.

Intel i3 N305 Mini PC https://www.aliexpress.com/item/1005007278560105.html

Intel Network - Mini PC

equips with 2x10G SFP+ (Intel 82599ES) and 2x2.5G (Intel i226-V) network cards:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
root@sense:~# lspci -v 
...
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 12
        Memory at 80a20000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at 3020 [disabled] [size=32]
        Memory at 80a44000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number a8-b8-e0-ff-ff-05-96-4d
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 13
        Memory at 80a00000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at 3000 [disabled] [size=32]
        Memory at 80a40000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number a8-b8-e0-ff-ff-05-96-4d
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
        Flags: bus master, fast devsel, latency 0, IOMMU group 18
        Memory at 4017000000 (64-bit, prefetchable) [virtual] [size=16K]
        Memory at 4017100000 (64-bit, prefetchable) [virtual] [size=16K]
        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: vfio-pci
        Kernel modules: ixgbevf

...

02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
        Subsystem: Intel Corporation Ethernet Controller I226-V
        Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 14
        Memory at 80600000 (32-bit, non-prefetchable) [size=1M]
        Memory at 80700000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number a8-b8-e0-ff-ff-05-96-4f
        Capabilities: [1c0] Latency Tolerance Reporting
        Capabilities: [1f0] Precision Time Measurement
        Capabilities: [1e0] L1 PM Substates
        Kernel driver in use: igc
        Kernel modules: igc

03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
        Subsystem: Intel Corporation Ethernet Controller I226-V
        Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 15
        Memory at 80400000 (32-bit, non-prefetchable) [size=1M]
        Memory at 80500000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number a8-b8-e0-ff-ff-05-96-50
        Capabilities: [1c0] Latency Tolerance Reporting
        Capabilities: [1f0] Precision Time Measurement
        Capabilities: [1e0] L1 PM Substates
        Kernel driver in use: igc
        Kernel modules: igc

04:00.0 Network controller: Intel Corporation Wi-Fi 7(802.11be) AX1775*/AX1790*/BE20*/BE401/BE1750* 2x2 (rev 1a)
        Subsystem: Intel Corporation Wi-Fi 7(802.11be) AX1775*/AX1790*/BE20*/BE401/BE1750* 2x2 (BE200 320MHz [Gale Peak])
        Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 16
        Memory at 80900000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Secondary PCI Express
        Capabilities: [158] Physical Layer 16.0 GT/s <?>
        Capabilities: [17c] Lane Margining at the Receiver <?>
        Capabilities: [188] Latency Tolerance Reporting
        Capabilities: [190] L1 PM Substates
        Capabilities: [1a0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Capabilities: [2a0] Data Link Feature <?>
        Capabilities: [2ac] Precision Time Measurement
        Capabilities: [2b8] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>
        Capabilities: [500] Vendor Specific Information: ID=0023 Rev=1 Len=010 <?>
        Kernel driver in use: iwlwifi
        Kernel modules: iwlwifi
...

root@sense:~# ip a
...
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP group default qlen 1000
    link/ether a8:b8:e0:05:96:4f brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr3 state UP group default qlen 1000
    link/ether a8:b8:e0:05:96:50 brd ff:ff:ff:ff:ff:ff
4: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:b8:e0:05:96:4d brd ff:ff:ff:ff:ff:ff
5: enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:96:4e brd ff:ff:ff:ff:ff:ff
7: enp1s0f0v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7a:de:19:c7:c2:ce brd ff:ff:ff:ff:ff:ff    
...
22: wlp4s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e0:8f:4c:b2:58:95 brd ff:ff:ff:ff:ff:ff

Verify Intel 82599ES network card status:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
root@sense:~# ethtool enp1s0f0
Settings for enp1s0f0:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

root@sense:~# ethtool -i enp1s0f0
driver: ixgbe
version: 6.8.12-2-pve
firmware-version: 0x800003de
expansion-rom-version: 
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Enable IOMMU in Proxmox:

1
2
root@sense:~# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

IOMMU PT mode improves the performance of other PCIe devices in the system when passthrough is being used.

Update GRUB:

1
2
3
4
5
6
7
8
9
root@sense:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.8.12-2-pve
Found initrd image: /boot/initrd.img-6.8.12-2-pve
Found linux image: /boot/vmlinuz-6.8.4-2-pve
Found initrd image: /boot/initrd.img-6.8.4-2-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Adding boot menu entry for UEFI Firmware Settings ...
done

Reboot Proxmox and verify IOMMU is enabled:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
root@sense:~# dmesg | grep -i IOMMU
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
[    0.053988] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
[    0.054043] DMAR: IOMMU enabled
[    0.145258] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.403235] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.440538] iommu: Default domain type: Translated
[    0.440538] iommu: DMA domain TLB invalidation policy: lazy mode
[    0.483908] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.483909] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.483910] DMAR: IOMMU feature nwfs inconsistent
[    0.483911] DMAR: IOMMU feature dit inconsistent
[    0.483912] DMAR: IOMMU feature sc_support inconsistent
[    0.483912] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.493686] pci 0000:00:02.0: Adding to iommu group 0
[    0.493726] pci 0000:00:00.0: Adding to iommu group 1
[    0.493739] pci 0000:00:0d.0: Adding to iommu group 2
[    0.493764] pci 0000:00:14.0: Adding to iommu group 3
[    0.493774] pci 0000:00:14.2: Adding to iommu group 3
[    0.493786] pci 0000:00:16.0: Adding to iommu group 4
[    0.493793] pci 0000:00:17.0: Adding to iommu group 5
[    0.493806] pci 0000:00:1c.0: Adding to iommu group 6
[    0.493820] pci 0000:00:1c.6: Adding to iommu group 7
[    0.493831] pci 0000:00:1d.0: Adding to iommu group 8
[    0.493842] pci 0000:00:1d.1: Adding to iommu group 9
[    0.493859] pci 0000:00:1d.3: Adding to iommu group 10
[    0.493878] pci 0000:00:1f.0: Adding to iommu group 11
[    0.493886] pci 0000:00:1f.3: Adding to iommu group 11
[    0.493895] pci 0000:00:1f.4: Adding to iommu group 11
[    0.493903] pci 0000:00:1f.5: Adding to iommu group 11
[    0.493918] pci 0000:01:00.0: Adding to iommu group 12
[    0.493934] pci 0000:01:00.1: Adding to iommu group 13
[    0.493945] pci 0000:02:00.0: Adding to iommu group 14
[    0.493956] pci 0000:03:00.0: Adding to iommu group 15
[    0.493982] pci 0000:04:00.0: Adding to iommu group 16
[    0.493993] pci 0000:05:00.0: Adding to iommu group 17
[    3.684070] pci 0000:01:10.1: Adding to iommu group 18
[    3.684303] pci 0000:01:10.3: Adding to iommu group 19
[    3.684501] pci 0000:01:10.5: Adding to iommu group 20
[    3.684685] pci 0000:01:10.7: Adding to iommu group 21
[    3.684885] pci 0000:01:11.1: Adding to iommu group 22
[    3.685096] pci 0000:01:11.3: Adding to iommu group 23
[    3.685296] pci 0000:01:11.5: Adding to iommu group 24
[    3.685473] pci 0000:01:11.7: Adding to iommu group 25
[    3.755233] pci 0000:01:10.0: Adding to iommu group 26
[    3.755462] pci 0000:01:10.2: Adding to iommu group 27
[    3.755761] pci 0000:01:10.4: Adding to iommu group 28
[    3.755939] pci 0000:01:10.6: Adding to iommu group 29
[    3.756107] pci 0000:01:11.0: Adding to iommu group 30
[    3.756276] pci 0000:01:11.2: Adding to iommu group 31
[    3.756444] pci 0000:01:11.4: Adding to iommu group 32
[    3.756628] pci 0000:01:11.6: Adding to iommu group 33

ATTENTION: IOMMU Group not order by PCI BUS IDs

This is an ISSUE CAN”T setup PCI Device Resource Mappings, because of errors, e.g., “Configuration for iommugroup not correct (‘21’ != ‘’27)” thrown after Proxmox reboot!

Filter out network card udev information, e.g., enp1s0f1:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
root@sense:~# udevadm info -e 
...
P: /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/net/enp1s0f1
M: enp1s0f1
R: 1
U: net
I: 5
E: DEVPATH=/devices/pci0000:00/0000:00:1c.0/0000:01:00.1/net/enp1s0f1
E: SUBSYSTEM=net
E: INTERFACE=enp1s0f1
E: IFINDEX=5
E: USEC_INITIALIZED=2364228
E: ID_NET_NAMING_SCHEME=v252
E: ID_NET_NAME_MAC=enxa8b8e005964e
E: ID_OUI_FROM_DATABASE=Changwang Technology inc.
E: ID_NET_NAME_PATH=enp1s0f1
E: ID_BUS=pci
E: ID_VENDOR_ID=0x8086
E: ID_MODEL_ID=0x10fb
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_MODEL_FROM_DATABASE=82599ES 10-Gigabit SFI/SFP+ Network Connection
E: ID_PATH=pci-0000:01:00.1
E: ID_PATH_TAG=pci-0000_01_00_1
E: ID_NET_DRIVER=ixgbe
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=enp1s0f1
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enp1s0f1
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

P: /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/net/enp1s0f1/ixgbe-mdio-0000:01:00.1
M: ixgbe-mdio-0000:01:00.1
R: 1
U: mdio_bus
E: DEVPATH=/devices/pci0000:00/0000:00:1c.0/0000:01:00.1/net/enp1s0f1/ixgbe-mdio-0000:01:00.1
E: SUBSYSTEM=mdio_bus
...

Important information like: SUBSYSTEM=net, INTERFACE=enp1s0f1, ID_NET_DRIVER=ixgbe, ID_NET_NAME=enp1s0f1 …

Assign 8 Virtual Network Cards:

1
2
root@sense:~# echo 8 > /sys/class/net/enp1s0f0/device/sriov_numvfs 
root@sense:~# echo 8 > /sys/class/net/enp1s0f1/device/sriov_numvfs

Verify network cards again:

1
2
3
4
5
6
7
8
9
10
11
root@sense:~# lspci | grep Ethernet
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
...
01:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
01:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
...
01:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)

Persist number of SR_IOV Virtual Functions after Proxmox reboot:

1
2
3
root@sense:~# cat /etc/udev/rules.d/enp.rules 
ACTION=="add", SUBSYSTEM=="net", ENV{INTERFACE}=="enp1s0f0", ATTR{device/sriov_numvfs}="8"
ACTION=="add", SUBSYSTEM=="net", ENV{INTERFACE}=="enp1s0f1", ATTR{device/sriov_numvfs}="8"

Add Virtual Network Adapters into Resource Mappings in Proxmox:

Intel Network - Resource Mappings

Add Mapped PCI Devices into Virtual Machine in Proxmox:

Intel Network - Mapped PCI Devices

Virtual Network Cards can be set up as Network Cards, with hardware direct access, from a Network Cards pool (shared by other Virtual Machines), when the first Network Card is available.

Intel Network - Virtual Network Cards

Virtual Machine’s Proxmox configuration file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@sense:/etc/pve/qemu-server# cat 101.conf
agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 8
cpu: x86-64-v2-AES,flags=+aes
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,size=4M
hostpci0: mapping=enp0_vf
hostpci1: mapping=enp1_vf
ide2: local:iso/OPNsense-24.7-dvd-amd64.iso,media=cdrom,size=2131548K
machine: q35
memory: 8192
meta: creation-qemu=9.0.2,ctime=1731054484
name: LINKsense
net0: virtio=BC:24:11:0C:52:4B,bridge=vmbr2,firewall=1,queues=4
net1: virtio=BC:24:11:21:3C:E0,bridge=vmbr3,firewall=1,queues=4
numa: 0
ostype: l26
scsi0: local-lvm:vm-101-disk-1,discard=on,iothread=1,size=128G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=3e913d90-c284-47d7-881a-599013fa21a4
sockets: 1
vmgenid: aab228d8-bccc-46c1-b627-927271ad20b6

References

Homelab Router OPNsense and IPv6

Guestbook

Current IPv6 setting on TP-Link Router, connecting to Buddy Telco / Aussie Broadband ISP.

OPNsense - TP-Link IPv6

Setup OPNsense Optional Port

For example, setup OPT1 interface after OPNsense installed in Proxmox.

  • Enable OPT1 Interface
  • Select Static IPv4 for IPv4 Configuration Type
  • Select Track Interface for IPv6 Configuration Type
  • IPv4 address set to 192.168.2.1/24
  • IPv6 parent interface set to WAN
  • IPv6 prefix ID set to 0x1. 0x0 has been assigned to LAN
  • Enable Allow manual adjustment of DHCPv6 and Router Advertisements

OPNsense - OPT1 Interface

  • Enable DHCP server on the OPT1 interface
  • Set IP range from 192.168.2.10 to 192.168.2.245

OPNsense - OPT1 DHCP

  • Add Firewall Rule
  • Set TCP/IP version
  • Select OPT1 net from dropdown menu as Source

OPNsense - OPT1 Firewall Rule

  • Setup Firewall Rule for IPv4 on OPT1 interface, then
  • Setup Firewall Rule for IPv6 on OPT1 interface

OPNsense - OPT1 Firewall

Setup IPv6

Step by step to enable IPv6 with Buddy Telco / Aussie Broadband ISP.

OPNsense - Interfaces Settings

OPNsense - WAN

  • Enable DHCP/DHCPv6 for IPv4/IPv6 Configuration Type
  • Set ISP’s Prefix delegation size to 48
  • Enable Send prefix hint

OPNsense - Router Advertisements

  • Set Router Advertisements to Unmanaged
  • Set Router Priority to Normal

VLAN

Setup Trusted and Untrusted VLAN.

OPNsense - VLAN

OPNsense - VLAN Interfaces

OPNsense - Trusted VLAN

OPNsense - Untrusted VLAN

Then,

  • Set Router Advertisements to Unmanaged
  • Set Router Priority to Normal

for both VLANs.

Running

After all the change made above, reboot OPNsense instance.

OPNsense - Proxmox Network

OPNsense - Setting In Proxmox

OPNsense - Run In Proxmox

OPNsense - Interfaces Overview

OPNsense - Dashboard

IPv6 Test

OPNsense - Test IPv6

OPNsense - IPv6 Test

References

OPNsense - IPv6 Chart

OPNsense - IPv4 CIDR Chart

Turn On and Turn Off Recall in Windows 11

Guestbook

Turn On Recall feature in Windows 11 24H2, by running Widnows Power Shell as Administrator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
PS C:\> DISM /Online /Get-FeatureInfo /FeatureName:Recall

Deployment Image Servicing and Management tool
Version: 10.0.26100.1150

Image Version: 10.0.26100.2033

Feature Information:

Feature Name : Recall
Display Name : Recall
Description : Recall application.
Restart Required : Possible
State : Enabled

Custom Properties:

(No custom properties found)

The operation completed successfully.

Turn Off Recall in Windows 11 24H2:

1
2
3
4
5
6
7
8
9
10
PS C:\> DISM /Online /Disable-Feature /FeatureName:Recall

Deployment Image Servicing and Management tool
Version: 10.0.26100.1150

Image Version: 10.0.26100.2033

Disabling feature(s)
[==========================100.0%==========================]
The operation completed successfully.

Running latest MacOS Sequoia in VMware on Windows 11

Guestbook

This is the step by step guide, the fastest and easiest way install and run latest MacOS Sequoia 15.0.1 in VMware Workstation Pro on Windows 11.

Due to MacOS Sequoia has added detection whether OS is running in Virtual Machine, so it’s better install MacOS Sonoma at first, then upgrade to latest version MacOS Sequoia.

VMware - Apple ID

  • Build a bootable MacOS ISO image

Do it on a Mac.

Clone gibmasOS repo https://github.com/corpnewt/gibMacOS and run:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
$ ./gibMacOS.command
#######################################################
#                      gibMacOS                       #
#######################################################

Available Products:

1. macOS Sequoia 15.0.1 (24A348)
   - 072-01382 - Added 2024-10-03 21:26:40 - 14.48 GB
2. macOS Ventura 13.7 (22H123)
   - 062-78643 - Added 2024-09-16 17:44:05 - 12.22 GB
3. macOS Sonoma 14.7 (23H124)
   - 062-78824 - Added 2024-09-16 17:42:25 - 13.68 GB
4. macOS Sequoia 15.0 (24A335)
   - 062-78429 - Added 2024-09-16 17:30:21 - 14.48 GB
5. macOS Monterey 12.7.6 (21H1320)
   - 062-40406 - Added 2024-08-14 20:45:56 - 12.42 GB

...

24. macOS High Sierra 10.13.6 (17G66)
   - 041-91758 - Added 2019-10-19 18:19:55 - 5.71 GB
25. macOS Mojave 10.14.6 (18G103)
   - 061-26589 - Added 2019-10-14 20:51:08 - 6.52 GB
26. macOS Mojave 10.14.5 (18F2059)
   - 061-26578 - Added 2019-10-14 20:38:26 - 6.52 GB

M. Change Max-OS Version (Currently 12)
C. Change Catalog (Currently publicrelease)
I. Only Print URLs (Currently Off)
S. Set Current Catalog to SoftwareUpdate Catalog
L. Clear SoftwareUpdate Catalog
R. Toggle Recovery-Only (Currently Off)
U. Show Catalog URL
Q. Quit

Please select an option: 3

Downloading InstallAssistant.pkg for 062-78824 - 14.7 macOS Sonoma (23H124)...

1.35 GB/14.48 GB | =                    9.34% | 101.7 MB/s | 00:02:10 left

Succeeded:
  InstallAssistant.pkg
  MajorOSInfo.pkg
  com_apple_MobileAsset_MacSoftwareUpdate.plist
  InstallInfo.plist
  UpdateBrain.zip

Failed:
  None

Files saved to:
  /Users/terrence/Projects/gibMacOS/macOS Downloads/publicrelease/062-78824 - 14.7 macOS Sonoma (23H124)

Run InstallAssistant.pkg from above MacOS download directory. It will be using to build ISO image.

Create a disk image with size 16GB:

1
2
$ hdiutil create -o /tmp/MacOS -size 16000m -volname MacOS -layout SPUD -fs HFS+J
created: /tmp/MacOS.dmg

Mount the disk image created above:

1
2
3
4
 $ hdiutil attach /tmp/MacOS.dmg -noverify -mountpoint /Volumes/MacOSISO
/dev/disk6          	Apple_partition_scheme         	
/dev/disk6s1        	Apple_partition_map            	
/dev/disk6s2        	Apple_HFS                      	/Volumes/MacOSISO

Create ISO image from Install masOS app:

1
2
3
4
5
6
7
$ sudo /Applications/Install\ macOS\ Sonoma.app/Contents/Resources/createinstallmedia --volume /Volumes/MacOSISO --nointeraction
Erasing disk: 0%... 10%... 20%... 30%... 100%
Copying essential files...
Copying the macOS RecoveryOS...
Making disk bootable...
Copying to disk: 0%... 10%... 20%... 30%... 40%... 50%... 60%... 100%
Install media now available at "/Volumes/Install macOS Sonoma"

Unmount disk image and convert to an ISO image:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ hdiutil detach -force /Volumes/Install\ macOS\ Sonoma
"disk6" ejected.

$ ls -al /tmp/MacOS.dmg
-rw-r--r--@ 1 terrence  wheel  16777216000 15 Oct 21:54 /tmp/MacOS.dmg

$ hdiutil convert /tmp/MacOS.dmg -format UDTO -o /tmp/MacOS-Sonoma-14.7.cdr
Reading Driver Descriptor Map (DDM : 0)…
Reading Apple (Apple_partition_map : 1)…
Reading  (Apple_Free : 2)…
Reading disk image (Apple_HFS : 3)…
........................................
Elapsed Time: 19.139s
Speed: 835.9MB/s
Savings: 0.0%
created: /tmp/MacOS-Sonoma-14.7.cdr

$ mv /tmp/MacOS-Sonoma-14.7.cdr /tmp/MacOS-Sonoma-14.7.iso

$ rm /tmp/MacOS.dmg
  • Install VMware Workstation Pro

Download VMware Workstation Pro from e.g. https://softwareupdate.vmware.com/cds/vmw-desktop/ws/17.6.1/24319023/windows/core/VMware-workstation-17.6.1-24319023.exe.tar

  • Patch VMware Workstation Pro

Clone unlocker https://github.com/paolo-projects/unlocker repo on Windows and enable Apple macOS option in VMware Workstation Pro:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
PS C:\Projects\unlocker> .\win-install.cmd

Unlocker 3.0.2 for VMware Workstation
=====================================
(c) Dave Parsons 2011-18

Set encoding parameters...
Active code page: 850

VMware is installed at: C:\Program Files (x86)\VMware\VMware Workstation\
VMware product version: 17.6.1.24319023

Stopping VMware services...

...

Starting VMware services...

Finished!

VMware - Apple macOS

  • Add VMware Tools

Copy darwin.iso and darwinPre15.iso files extracted from VMware Fusion e.g. _https://softwareupdate.vmware.com/cds/vmw-desktop/fusion/12.2.5/20904517/x86/core/com.vmware.fusion.zip.tar _ into VMware Workstation Pro directory:

1
2
C:\Projects\unlocker\tools\darwin.iso -> C:\Program Files (x86)\VMware\VMware Workstation\darwin.iso
C:\Projects\unlocker\tools\darwinPre15.iso -> C:\Program Files (x86)\VMware\VMware Workstation\darwinPre15.iso
  • Create Virtual Machine for MacOS 14 Sonoma and update the settings

Add: 

1
smc.version = "0"

into MacOS Sonoma.vmx file.

Clone GenSMBIOS repo https://github.com/corpnewt/GenSMBIOS and generate serial number on Windows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ ./GenSMBIOS.bat
#######################################################
#                     GenSMBIOS                       #
#######################################################

MacSerial not found!
Remote Version v2.1.8
Current plist: None
Plist type:    Unknown

1. Install/Update MacSerial
2. Select config.plist
3. Generate SMBIOS
4. Generate UUID
5. Generate ROM
6. List Current SMBIOS
7. Generate ROM With SMBIOS (Currently Enabled)

Q. Quit

Please select an option:  3

Please type the SMBIOS to gen and the number
of times to generate [max 20] (i.e. iMac18,3 5): MacBookPro16,4

Type:         MacBookPro16,4
Serial:       C..........T
Board Serial: C0.............FB
SmUUID:       A0D50403-F256-4E17-A2EC-29964D889A1D
Apple ROM:    6..........7

Copy Serial, Board Serial and Apple ROM number, apply to:

1
2
3
4
5
6
7
8
9
10
board-id = "Mac-A61BADE1FDAD7B05"
hw.model.reflectHost = "FALSE"
hw.model = "MacBookPro16,4"
serialNumber.reflectHost = "FALSE"
serialNumber = "C..........T"
smbios.reflectHost = "FALSE"
efi.nvram.var.ROM.reflectHost = "FALSE"
efi.nvram.var.MLB.reflectHost = "FALSE"
efi.nvram.var.ROM = "6..........7"
efi.nvram.var.MLB = "C0.............FB"

then add above block into MacOS Sonoma.vmx file.

Based on Apple Ethernet MAC Address range https://hwaddress.com/company/apple-inc/, change and add network settings from:

1
ethernet0.addressType = "generated"

to:

1
2
3
ethernet0.addressType = "static"
ethernet0.address = "00:21:E9:c0:92:76"
ethernet0.checkMacAddress = "FALSE"

in MacOS Sonoma.vmx file.

VMware - MacOS Sonoma

  • Mount MacOS ISO image and install

VMware - MacOS Sonoma settings

DON’T enable Location Service during the installation! Otherwise, you can’t setup Time Zone, Date Time based on your area. You can login with your Apple ID during the installation.

VMware - Apple ID login

  • Install VMware Tools

After MacOS installed and VM restarted, mount darwin.iso and install VMware Tools. Then Display Memory in MacOS becomes 128 MB, and support the Full Screen mode.

  • Upgrade MacOS

After MacOS Sonoma installed, make sure everything is OK, then copy the whole MacOS Sonoma directory to a new directory MacOS Sequoia (Upgraded). Open the new directory in VMware, and select I copied it, then you can upgrade MacOS to latest verion in System Settings -> Software Update.

  • Run MacOS in VMware Player:

VMware - Player

VMware - In Action

References

How to access Ollama runs on localhost from internet

Guestbook

By default, Ollama is only accessible from localhost. CORS error and HTTP 403 returned if try to invoke from other hosts.

To enable Ollama can be visited from hosts on intranet, run:

1
$ launchctl setenv OLLAMA_HOST "0.0.0.0"

in MacOS. Then restart Ollama. Then invoke it from hosts on intranet.

To enable Ollama can be visited from internet, with the help from ngrok, run:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ngrok http 11434

ngrok                                                     (Ctrl+C to quit)

Session Status  online
Account         Terrence Miao (Plan: Free)
Version         3.17.0
Region          Australia (au)
Web Interface   http://127.0.0.1:4040
Forwarding      https://294b-2403-5802-1c44-0-341c-aae8-a501-8b73.ngrok-free.app -> http://localhost:11434

Connections     ttl     opn     rt1     rt5     p50     p90
                0       0       0.00    0.00    0.00    0.00

Then can visit Ollama on localhost from internet. 11434 is default port of Ollama.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ curl --location 'https://294b-2403-5802-1c44-0-341c-aae8-a501-8b73.ngrok-free.app/v1/models'

{
    "object": "list",
    "data": [
        {
            "id": "codellama:latest",
            "object": "model",
            "created": 1728736149,
            "owned_by": "library"
        },
        {
            "id": "deepseek-coder-v2:latest",
            "object": "model",
            "created": 1728735883,
            "owned_by": "library"
        },
        {
            "id": "mxbai-embed-large:latest",
            "object": "model",
            "created": 1713401203,
            "owned_by": "library"
        },
        {
            "id": "nomic-embed-text:latest",
            "object": "model",
            "created": 1708780687,
            "owned_by": "library"
        }
    ]
}

To write some code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
$ curl --location 'https://294b-2403-5802-1c44-0-341c-aae8-a501-8b73.ngrok-free.app/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
    "model": "deepseek-coder-v2",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Write a function that outputs the fibonacci sequence in TypeScript."
        }
    ]
}'

{
  "id": "chatcmpl-174",
  "object": "chat.completion",
  "created": 1728738926,
  "model": "deepseek-coder-v2",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "
        Certainly! Below is a TypeScript function that outputs the Fibonacci sequence up to a specified number of terms:

        This function starts with the initial terms of the Fibonacci sequence `[0, 1]` and then iteratively calculates each subsequent term
        by summing the two preceding terms. The function returns an array containing the first `n` terms of the Fibonacci sequence.
        "
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 29,
    "completion_tokens": 243,
    "total_tokens": 272
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
function fibonacciSequence(n: number): number[] {
  if (n <= 0) return [];
  if (n === 1) return [0];

  const result: number[] = [0, 1];

  for (let i = 2; i < n; i++) {
    result.push(result[i - 1] + result[i - 2]);
  }

  return result;
}

// Example usage: console.log(fibonacciSequence(10));
// Output: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Check Ollama log:

1
$ tail -f ~/.ollama/logs/server.log

References

Running a Mac app from Unidentified Developer on MacOS Sequoia

Guestbook

Latest MacOS Sequoia has made another security enforcement. To open and run a Mac application, e.g. VLC nightly build at https://nightlies.videolan.org/, Unidentified Developer , the following steps can take.

  • Enable “Allow Applications from Anywhere”
1
2
3
4
base ~
sudo spctl --master-disable
Password:
Globally disabling the assessment system needs to be confirmed in System Settings.

Go to System Settings -> Privacy & Security, select “Allow Applications from Anywhere”:

MacOS - Allow Applications from Anywhere

1
2
3
4
base ~
sudo spctl --master-enable
Password:
This operation is no longer supported. Please see the man page for more information.
  • Disable MacOS Quarantine on the application

In this case application VLC:

1
2
base ~
xattr -dr com.apple.quarantine /Applications/VLC.app

Configure and setup Mellanox network adapter

Guestbook

A Mellanox ConnectX-4 MCX455A-ECAT PCIe x16 3.0 100GBe VPI EDR IB network adapter. Enable VT-d(Intel Virtualization Technology for Directed I/O), SR-IOV (Single Root IO Virtualization), and the number of Virtual Functions in Chipset and Network in BIOS.

The following steps are made in a Proxmox Virtual Environment (PVE).

Configuration and Setup

Go to NVIDIA Firmware Tools (MFT) https://network.nvidia.com/products/adapter-software/firmware-tools/ and download the MFT; go to Updating Firmware for ConnectX®-4 VPI PCI Express Adapter Cards (InfiniBand, Ethernet, VPI) https://network.nvidia.com/support/firmware/connectx4ib/ and download the updated firmware e.g. mft-4.26.1-6-x86_64-deb.tgz.

After installation, start up Mellanox Software Tools service:

1
2
3
4
5
6
root@pve:~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success

Check status:

1
2
3
4
5
6
7
8
9
10
11
root@pve:~# mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4115_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:06:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00

Query Mellanox network adapter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
root@pve:~# flint -d /dev/mst/mt4115_pciconf0 query
Image type:            FS3
FW Version:            12.28.2006
FW Release Date:       15.9.2020
Product Version:       12.28.2006
Rom Info:              type=UEFI version=14.21.17 cpu=AMD64
                       type=PXE version=3.6.102 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             ec0d9a030076eae2        4
Base MAC:              ec0d9a76eae2            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  LNV2180110032
Security Attributes:   N/A

Check Mellanox network adapter configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
root@pve:~# mlxconfig -d /dev/mst/mt4115_pciconf0 query
Device #1:
----------

Device type:        ConnectX4
Name:               00KH925_Ax
Description:        Mellanox ConnectX-4 EDR IB VPI Single-port x16 PCIe 3.0 HCA
Device:             /dev/mst/mt4115_pciconf0

Configurations:
        LINK_TYPE_P1                                ETH(2)
        SRIOV_EN                                    True(1)
        NUM_OF_VFS                                  8
...

Verify network adapter configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
terrence@pve:~# ethtool enp1s0np0
Settings for enp1s0np0:
	Supported ports: [ Backplane ]
	Supported link modes:   1000baseKX/Full
	                        10000baseKR/Full
	                        40000baseKR4/Full
	                        40000baseCR4/Full
	                        40000baseSR4/Full
	                        40000baseLR4/Full
	                        56000baseKR4/Full
	                        25000baseCR/Full
	                        25000baseKR/Full
	                        25000baseSR/Full
	                        50000baseCR2/Full
	                        50000baseKR2/Full
	                        100000baseKR4/Full
	                        100000baseSR4/Full
	                        100000baseCR4/Full
	                        100000baseLR4_ER4/Full
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: None	 RS	 BASER
	Advertised link modes:  1000baseKX/Full
	                        10000baseKR/Full
	                        40000baseKR4/Full
	                        40000baseCR4/Full
	                        40000baseSR4/Full
	                        40000baseLR4/Full
	                        56000baseKR4/Full
	                        25000baseCR/Full
	                        25000baseKR/Full
	                        25000baseSR/Full
	                        50000baseCR2/Full
	                        50000baseKR2/Full
	                        100000baseKR4/Full
	                        100000baseSR4/Full
	                        100000baseCR4/Full
	                        100000baseLR4_ER4/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: RS
	Speed: 100000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Direct Attach Copper
	PHYAD: 0
	Transceiver: internal
	Link detected: yes

Virtualization

1
2
3
4
5
6
7
8
root@pve:~# lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

root@pve:~# ip link show
...
4: enp1s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
    link/ether ec:0d:9a:76:eb:2a brd ff:ff:ff:ff:ff:ff
...

Write to the sysfs file the number of Virtual Functions:

1
root@pve:~# echo 8 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs

Verify that the Virtual Functions were created:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@pve:~# lspci | grep Mellanox
01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
01:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.4 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.5 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.6 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:00.7 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]
01:01.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

root@pve:~# ip link show
...
4: enp1s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
    link/ether ec:0d:9a:76:eb:2a brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
...

Persistence

Print out device information:

1
2
3
4
5
6
7
8
root@pve:~# udevadm info /sys/class/infiniband/mlx5_0
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/infiniband/mlx5_0
M: mlx5_0
R: 0
U: infiniband
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/infiniband/mlx5_0
E: SUBSYSTEM=infiniband
E: NAME=mlx5_0

Make the network adapter Virtual Functions persistent after reboot:

1
2
root@pve:~# cat /etc/udev/rules.d/mlx.rules
ACTION=="add", SUBSYSTEM=="infiniband", ENV{NAME}=="mlx5_0", ATTR{device/sriov_numvfs}="8"

Network in Proxmox

Add virtualised network adapter in Resource Mappings in Proxmox:

Mellanox - Resource Mappings

List PCI Devices in Resource Mappings in Proxmox which support Virtual Functions:

Mellanox - Resource Mappings PCI Devices

Add PCI device in Resource Mappings in Proxmox in VM:

Mellanox - Network PCI Device

References