openQA Project (public)

It's not about the test modules but the individual results (screenshots, serial output).

To reproduce https://bugzilla.suse.com/show_bug.cgi?id=1210075, I asked for runs of

    while (1) {
        clear_console();
        assert_script_run("true");
    }

(FTR, that was a success: The jobs eventually failed)

Sorry, I meant test steps, not modules.

a small idea for os-autoinst: https://github.com/os-autoinst/os-autoinst/pull/2313

Most effective would be a limit enforced by the web UI controller. This way nobody can upload too many results using a patched version of os-autoinst or the worker. The most straight forward way to implement this is adding a result_count column (in addition to the existing result_size column), increment it via UPDATE … WHERE … result_count < $limit and return an error if the row could not be updated. The only annoyance is the additional column itself. Alternatively one could just use a text file in the results directory to store the counter but making this work race-free is a bit more difficult. We could also just do a directory listing and count the relevant files but distinguishing relevant and irrelevant files in the listing is also not that nice.

I guess the approach from https://github.com/os-autoinst/os-autoinst/pull/2313 could be followed-up upon as well. Of course it is only a client-side check and thus not really secure. As it has been implemented so far it would also only affect the number of steps within a single job module (but likely we want just a global limit).

I think just limiting in os-autoinst is ok as there is no other backend. And if an instance suffers from old os-autoinst then obviously the solution is to upgrade os-autoinst :) and if an admin identifies a malicious worker they can simply remove the API key

I was more thinking about malicious users trying to break things intentionally. However, considering https://github.com/os-autoinst/os-autoinst/pull/2322#issuecomment-1580378712 and the effort a server-side limit would be it is likely ok to just limit it client-side.

So what's now still missing is to mark the job as incomplete with a meaningful reason.

The PR now covers marking the job as incomplete with a meaningful reason together with https://github.com/os-autoinst/openQA/pull/5193.