QA (public) » openQA Project (public) » openQA Infrastructure (public)

Meh, this leads to telegraf trying to read a non-existent /dev/nvme0c0n1 (because there is /sys/block/nvme0c0n1 detected). But this is some sort of NVMe multipath thingy which is not supposed to have a corresponding dev node. So this looks like telegraf diskio plugin cannot handle that nicely.

I was able to find some similar problems in other software[^3].

It looks like we do have that feature enabled [^2] (and it probably is relevant only for some, usually the larger ones, disks), it probably is also enabled by default [1]:

#  cat /sys/module/nvme_core/parameters/multipath
Y

Afaik we do not really use it so one path out of it could be just disabling it? According to [^1] it should just be a kernel parameter.

[^1]: https://documentation.suse.com/sles/15-SP6/html/SLES-all/cha-nvmeof.html#sec-nvmeof-host-configuration-multipathing
[^2]: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_device_mapper_multipath/enabling-multipathing-on-nvme-devices_configuring-device-mapper-multipath#proc_enabling-native-nvme-multipathing_enabling-multipathing-on-nvme-devices
[^3]: https://github.com/google/cadvisor/issues/3340

Maybe it is worth noting, that inputs.diskio.tagdrop is useless in this case as the attempt to get info from the /dev was already made at that point. Maybe specifying the devices instead the implicit devices=["*"] might work, but there might be the risk of not collecting everything we want/need.

I've taken worker33 as example because you mention it as one of the machines causing problems and indeed I found your mentioned device-node:

worker33:~ # ls -lah /sys/class/block/nvme?c*
lrwxrwxrwx 1 root root 0 Feb 23 03:34 /sys/class/block/nvme1c1n1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/nvme/nvme1/nvme1c1n1

unfortunately I couldn't figure out what exactly in udev causes this node to be created. I checked what we have on worker33 according to the kernel docs to see, if we loose any functionality. All of the 3 NVMes in worker33 have just one namespace so there can't be any multi-pathing here (The NVMe multipath feature in Linux integrates namespaces with the same identifier into a single block device.). The currently used policy is "NUMA" which indicates shorter paths in multi CPU machines (The NUMA policy selects the path closest to the NUMA node of the current CPU) which at least worker33 is not. I think this is why we can currently go ahead with the proposed change but can imagine several options to look into:

1. Check out if the nvme-tool has a way to disable this feature on the disk itself so udev does not create this node
2. Dig into udev if there is a way to avoid creating these nodes by e.g. setting a flag/env-variable (/usr/lib/udev/rules.d/56-multipath.rules could be interesting)
3. Understand why telegraf considers them with devices = ["*"] - what does this *-wildcard mean? Can it be influenced?

nicksinger wrote in #note-10:

1. Check out if the nvme-tool has a way to disable this feature on the disk itself so udev does not create this node

I even wasn't able to find out a way to list those, but I didn't read the documentation much.

1. Dig into udev if there is a way to avoid creating these nodes by e.g. setting a flag/env-variable (/usr/lib/udev/rules.d/56-multipath.rules could be interesting)

Good idea. That would be probably more elegant than disabling it directly in the kernel.

1. Understand why telegraf considers them with devices = ["*"] - what does this *-wildcard mean? Can it be influenced?

I can answer that right away, telegraf scans /sys/block (in a newer version it will read /sys/class/block) for devices according to the mask. All entries there are considered as a block device and used (unfortunately as we can see, not every device there has a corresponding /dev entry and we see the result). To me it looks like a bug in telegraph, but I am no expert in this area to decide that.