QA (public) » openQA Project (public) » openQA Infrastructure (public)

disk.mean { fstype: efivarfs, device: efivarfs }
16.0%   16.0%   16.0%
disk.mean { device: nvme0n1p1, fstype: vfat }
1.15%   1.15%   1.15%
disk.mean { device: nvme0n1p2, fstype: btrfs }
81.4%   86.4%   81.4%
disk.mean { device: sda2, fstype: xfs }
0.106%  0.106%  0.106%

Not sure if anyone deleted files. For now numbers are going down. The alert is not currently active.

I asked in Slack yesterday if people could delete virtual machines. That apparently helped.

I will take a look and see if I can follow the instructions. Asking our users directly in Slack also seems like a good idea.

I'm able to connect ssh -A -L 8443:ada-ipmi:443 -NT jumpy@qe-jumpy.prg2.suse.org which gets me a web interface. However I don't know what password to use 🤔

I can connect to ada.qe.suse.de via SSH. This is where I'm probably running into the same problem as VMM says authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage' when trying to connect.

So I'm asking for advice on Slack.

livdywan wrote in #note-10:

I will take a look and see if I can follow the instructions. Asking our users directly in Slack also seems like a good idea.

I'm able to connect ssh -A -L 8443:ada-ipmi:443 -NT jumpy@qe-jumpy.prg2.suse.org which gets me a web interface. However I don't know what password to use 🤔

I can connect to ada.qe.suse.de via SSH. This is where I'm probably running into the same problem as VMM says authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage' when trying to connect.

So I'm asking for advice on Slack.

The credentials for the BMC can be found here: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls?ref_type=heads#L2664

I was able to connect with virt-manager and got a list of all machines. The polkit problems you both encountered seem related to your local configuration although it is indeed strange that connections to e.g. osiris work. I checked my machine and found that /usr/share/polkit-1/rules.d/50-libvirt.rules should provide the requested permission if the user is in the libvirt group. The rules-file comes from the libvirt-daemon-package. Could you check that you have this installed?

I was able to connect with virt-manager and got a list of all machines. The polkit problems you both encountered seem related to your local configuration although it is indeed strange that connections to e.g. osiris work. I checked my machine and found that /usr/share/polkit-1/rules.d/50-libvirt.rules should provide the requested permission if the user is in the libvirt group. The rules-file comes from the libvirt-daemon-package. Could you check that you have this installed?

Turns out my user on ada wasn't in the group and that's why the connection failed. I'm updating the instructions accordingly.

sudo du -h /var/lib/libvirt/images /mnt/SSD_extension /extra_storage
2,5T    /var/lib/libvirt/images
0       /mnt/SSD_extension
0       /extra_storage

Nobody is following the recommendation to use SSD_extension written in red front. Which is on the same disk anyway:

sudo df -h /var/lib/libvirt/images /mnt/SSD_extension /extra_storage/
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  3,5T  3,0T  584G  84% /var
/dev/nvme0n1p2  3,5T  3,0T  584G  84% /
/dev/sda2       464G  506M  464G   1% /extra_storage

So I removed SDD_extension from the instructions. Instead I'm recommending extra_storage. The GUI actually shows the amount of available space, too.

I removed two images from /extra_storage/liv-openqa-* to my machine and I moved them on my machine.
I also delete some snapshots and deleted files with sudo lsof |grep deleted | grep qemu | awk '{print $2}' | xargs sudo kill -9
but df doesnt show reduced size.

only /extra_storage
```iob@ada:~> sudo du -h /var/lib/libvirt/images /mnt/SSD_extension /extra_storage
2.6T /var/lib/libvirt/images
0 /mnt/SSD_extension
0 /extra_storage

I dont know if it helps but I run also sudo btrfs balance -d -m /. I expect to run for a while. currently is 27% done

TODOs:

1. Disable copy-on-write for the libvirt images, e.g. something like chattr +C /var/lib/libvirt/images or however that command was
2. Use /dev/sda2 as btrfs filesystem extension
3. Ensure proper btrfs scrub,trim,balance
4. Ask machine and image owners which images can be removed/moved/shrunk
5. Consider ordering a bigger SSD to put into a storage extension slot and use that for /var/lib/libvirt/images

I used for dom in $(virsh list --all --name); do virsh dumpxml --domain "$dom" --xpath '//devices/disk/source/@file'; done | sort -u | cut -f2 -d= | tr -d \" to identify all used images and cross-checked it with the output of virsh vol-list default. That helped me to identify some old, unused qcow files and make some more space.