QA » openQA Project

Well, I would think that the basic behaviour must be the same whatever the dbus daemon is, it's just a question of how the daemon handles it. dbus-broker apparently uses a hard limit; perhaps dbus-daemon is willing to just let a client keep piling up a backlog of messages (or whatever is going on) until system RAM is exhausted and it gets OOM killed, or something. But the underlying behaviour should be there and hence debuggable either way.

I will try and find someone who can help me debug this if no-one else gets to it first, but it's not #1 on my priority list as we're in final freeze for F34 right now.

The log shows 14 processes with significant OutgoingBytes buffers in dbus-broker. This means there is data leaving dbus-broker but not being dequeued by the client (of course, there is always the possibility of this being a bug in the broker).

The 14 processes have between 12 and 60 megabytes of data queued, so this is a sign of something being wrong. All 14 processes are instances of /usr/bin/isotovideo: backend. No other process in this log shows this behavior. Everything else looks normal, on first glance. (Search for OutgoingBytes in the logs, and look at the entries which have multiple megabytes of data listed)

The logs also show that all these clients have a single match rule installed. So my suspicion is that these processes do not dispatch the dbus file-descriptor, while someone keeps sending them messages.

Maybe someone with knowledge of "isotovideo: backend" can have a quick look? I might not be able to look further into this before next Monday.

Thanks a lot for the info!

So in the openQA system, isotovideo is the process that ultimately does the heavy work of actually running tests. It launches an instance of qemu and interacts with it, recording a video and several screenshots along the way.

An openQA "worker host" will have several long-running "worker" services; each time a test job is scheduled, one of these "worker" instances launches isotovideo to run the actual test. The isotovideo process only lasts as long as it takes to run the test job. I believe the worker process and isotovideo signal to each other via DBus, and that's probably where the problem lies.

Hmm, I might have butchered that (I was going from memory). Poking through the code, it looks more like the worker service and isotovideo have their own system for communicating. DBus is used by isotovideo to signal to os-autoinst-openvswitch, which is a helper process for interacting with openvswitch (we use openvswitch for some advanced network configurations in certain openQA tests). Broadly it looks like os-autoinst-openvswitch provides the org.opensuse.os_autoinst.switch interface, and exports the methods "set_vlan", "unset_vlan", and "show"; isotovideo is only meant, AFAICT, to send those signals, it's not meant to receive anything.

Key files to look at are https://github.com/os-autoinst/os-autoinst/blob/master/backend/qemu.pm (which has the only dbus-related code in the isotovideo backend that I can find) and https://github.com/os-autoinst/os-autoinst/blob/master/os-autoinst-openvswitch (the os-autoinst-openvswitch process which does act as an exporter).

Check out this line:

https://github.com/os-autoinst/os-autoinst/blob/965960f534c93ef12f0978014d589b8b2be6e6d2/backend/qemu.pm#L138

This calls into Net::Dbus->system and thus into Net::Dbus::_new(). This function installs a D-Bus match for NameOwnerChanged. Hence, this connection will continuously accumulate signals for every name-change on the system bus. If you do not dequeue those signals, they will slowly pile up. Note that every connect/disconnect to the bus will trigger such signals. Given that your previous logs showed connection-ids like :1.230716, this means there have been at least about half a million NameOwnerChanged signals. These signals will pile up inside of dbus-broker, because D-Bus is not a lossy protocol.

This is just a guess. My perl knowledge is rather small. Looking at Net::Dbus code, it seems to integrate with the Net/DBus/Reactor.pm-mainloop, but only if you import the module beforehand. I would assume there is nothing dispatching your d-bus messages otherwise.

I would recommend passing "private"=true to the constructor and then making sure to destroy the connection when you no longer need it. But this really needs input of the respective developers. This is beyond what I can safely judge.

But this really needs input of the respective developers.

The D-Bus service has been introduce by https://github.com/os-autoinst/os-autoinst/commit/315074ecb8b12817771fde11ad80291cdd4cf534. He's only contributing occasionally. I don't think anybody in the current team has much experience with Perl's Net::DBus::Object module to know anything beyond what @dvdhrm has already found out.

I suppose the first step would be to come up with a test which reproduces the problem. The real fix might be rather small.

Alternatively we could also just ditch D-Bus and simply make it an HTTP server. That would be a bigger change but also an easy one which would definitely solve the issue.

The perl-Net-Dbus author (dpberrange) is actually a Fedora/Red Hat person, I believe. I'll ask them if they can take a look and see what they think/suggest.

Sent https://github.com/os-autoinst/os-autoinst/pull/1641 as a possible way to deal with this, based on discussion with danpb. It at least doesn't seem to break anything. Will have to run with it for a while to see if the bug is fixed.

Yeah, I have it running on both our deployments now. If we haven't had any more incompletions by next week I'll try and remember to close this with a note.

I was gonna confirm tomorrow, but yeah, it looks like it's resolved indeed.