QA » openQA Project » openQA Infrastructure

Tried some deployments on local x86_64 machines with different configurations.
Wanted to see the inside of malbec and then deploy bcache according to its hardware and os envirenment.

Currently blocked by 'dead' malbec and waiting for it up.
Filed a infra ticket (107529) for this issue:
https://infra.nue.suse.com/Ticket/Display.html?id=107529

Don't hack this into malbec - hack this into the salt receipes.

coolo wrote:

Don't hack this into malbec - hack this into the salt receipes.

Got it, will look into deploy by salt

malbec issue has no any update since 2nd of March:

2018-03-01 10:44 The RT System itself - Status changed from 'new' to 'open'
2018-03-01 10:44 lrupp (Lars Vogdt) - Given to mcaj (Martin Caj)
2018-03-01 10:44 lrupp (Lars Vogdt) - Queue changed from BuildOPS to infra
2018-03-01 14:30 mcaj (Martin Caj) - Given to mmaher (Maximilian Maher)
2018-03-02 11:25 fghariani (Fatma Ghariani) - Stolen from mmaher (Maximilian Maher)
2018-03-02 11:25 fghariani (Fatma Ghariani) - Given to ihno (Ihno Krumreich)

Also setup a salt master and minon on local to compose sls file which is for future bcache deployment

Done a sls file to deploy backing device in a looped file and cache device on a x86_64 test machine.
Looked into alive power8 worker to get disk configurations, adapting sls to match the real worker.

An update, https://fsp1-malbec.arch.suse.de/cgi-bin/cgi is now acessible again but doesn't allow ipmi or any other type of interaction apparently.

However the machine seems to have booted in baremetal mode OR there might be network issues that don't allow the ip to be set... who knows.

I am adapting bcache sls for Power8-4 worker, and some questions were pop-up:

1. I can't find any ssd device on the machine, so where the caching device should create?
2. /var/lib/opneqa is mounted on a partition, is it ok to format the partition as backing device and mount to /var/lib/openqa? so it would clean up all existing workers and re-install them on bcache device. or use looped file as backing device for newly added workers?

if we had ssd in these workers, we wouldn't run them on tmpfs. And we don't want to cache onto ssds, but use the backing device only for overcommiting the memory.

Don't use the full /var/lib/openqa because we don't want to store /var/lib/openqa/cache in memory I guess. But you can shrink it to gain space, but possibly we even other disks.

wip pr submitted:
https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/37

malbec is back, last update of https://infra.nue.suse.com/Ticket/Display.html?id=107529:
2018-03-21 11:47 jloeser (Jan Loeser) - Status changed from 'open' to 'resolved'

Did I misunderstand that we don't need make cache on malbec if no ssd device?
As I talked with Coly, only using backing device is not quite different with not using bcache.

Also i don't get the idea about to reduce memory overcommiting by using bcache, since bcache is mainly to use fast device act as a cache for some slower storage device, and the memory usage would be getting lager along with ssd size increase.

And if a cache is needed and no ssd device, cache on hdd is not meaningful, if cache on tmpfs which is still using memory as cache but also limited by memory size and memory is twice used.

btw, since malbec is alive, further work should be specific on malbec.

I'm guessing that the solution that coolo is refering to, and that alex pointed him to, was to use the bcache as backing device that would extend the memory of the system via zram... this would extend the ammount of memory available in the pool, so that the actual data that is commonly used is kept in the faster ram while the other is moved to the zram-bcache frankenstein... but my memory is foggy here.

@mitiao, the idea with using rotating disks, is that since we have some ram to use, just to make sure that there' s enough extra space to store big files. Speed is second to some extent here.

Gitlab PR looks fine so far... but well, let malbec be the play ground unless coolo says otherwise :)

This is not about zram, no. We have enough memory - in general. But we need to have a strategy where to put the blocks in case the RAM is overcommited. If bcache can't be used with tmpfs, then we need to find a different solution to have RAM as backing store to the pools.

coolo wrote:

This is not about zram, no. We have enough memory - in general. But we need to have a strategy where to put the blocks in case the RAM is overcommited. If bcache can't be used with tmpfs, then we need to find a different solution to have RAM as backing store to the pools.

You can just use a genuine ramdisk rather than tmpfs as backing store for bcache. That gives you the same effect as the ZRAM variant, but without RAM overhead reduction. The big benefit of bcache that we found was that it nicely continuously streams dirty data onto the backing storage, so it actually saturates the links rather than generate big latency spikes when flushing from time to time.

It was agreed during the sprint planning to move forward with this.

@mitiao are you going to keep working on this?

algraf wrote:

You can just use a genuine ramdisk rather than tmpfs as backing store for bcache. That gives you the same effect as the ZRAM variant, but without RAM overhead reduction.

Do you mean ramfs rather then tmpfs or something like mke2fs -m 0 /dev/ram0 ? Wouldn't in both cases the system memory be prone to run full and cause more havoc than a size-limited tmpfs?

agreed, thx for the update