QA (public)

What i have seen so far (after have restarted the dashboard-amqp-watcher.service)

a. amqp messages are populated to the queue once a job is restarted.

[x] suse.openqa.job.restart:{"ARCH":"x86_64","BUILD":":32885:kernel-ec2", "FLAVOR":"Server-DVD-Incidents-Kernel", ents-Kernel@64bit-with-ltp.qcow2,"MACHINE":"64bit","TEST":"ltp_controllers","auto":0, "bugref":"bsc#1220264","bugurl":"https:\/\/bugzilla.suse.com\/show_bug.cgi?id=1220264", "group_id":281,"id":"13782223","remaining":1,"result":{"13782223":13791364}}

b. another job is restarted and journal also logs once a job is restarted

qam2 dashboard[30577]: [30577] [i] restart 13793528 -> 13793577

c. https://dashboard.qam.suse.de/blocked on the Network tab shows constant blocked request in regular frequency

observation are after a couple of hours (~2h)
I will keep an eye on the queues for a while to see any setback

I don't understand. Your previous point "c." reads like a confirmation of the original issue, is it? And in the next comment you stated "after a whole day the dashboard and amqp still properly". I assume you wanted to say "still work properly"? But I have not seen any statement of applied changes. So what are your planned steps?

restart of the service has no impact

now dashboard works again for some reason and i cant reproduce the problem. no change from my side

ybonatakis wrote in #note-9:

now dashboard works again

What exactly you do you mean by that?

okurz wrote in #note-10:

ybonatakis wrote in #note-9:

now dashboard works again

What exactly you do you mean by that?

The underline problem is not reproducible. Restarted jobs on openqa side, and checking the dashboard to verify that the state is changing as expected. When jobs fails again the dashboard shows the failure.

Liv and I tried to add debug logs but we couldnt make it work. More specifically, we edit the /etc/systemd/system/dashboard-amqp-watcher.service added Environment="MOJO_LOG_LEVEL=debug" and restart the service.
Another attempt was to set that in the environment MOJO_LOG_LEVEL=debug and start the service on this terminal. but no debug logs appeared.

personally i am running out of ideas.
I have confirmed that the openqa messages appear in the queue even when the dashboard misfunction.

I hardcode the log level in https://github.com/openSUSE/qem-dashboard/blob/main/lib/Dashboard.pm#L41 and i will leave it running, hoping that it is gonna give some useful data in case it starts misfunction again

In general there is not problem at all with amqp. All the data are provided in the requests.
I ended believing that the problem comes down to the UI and particular the https://github.com/openSUSE/qem-dashboard/blob/main/assets/vue/components/PageBlocked.vue. I think it might be something wrong with the script that match the accidents and updates the UI. this is the next step.

Apparently the patch to the log level causes the pipeline to fail https://gitlab.suse.de/qa-maintenance/qamops/-/jobs/2400856

I start looking to the scripts to make sense. At that point I am completely lost.
the data between /blocked request and openqa and dashboard are not synced. And they appear sync after long time in some cases. But it is hard to follow among all the data.

Sometimes /blocked+openqa looks same but dashboard
others /blocked+dashboard but openqa not

And sometimes i cant find entries in /blocked data

dashboard-amqp-watcher.service logs stopped since Mar 20 00:47:02 (today morning)
dashboard.service keeps doing request but cleanup reports (in cleanup) Can't remove file "/tmp/prefork.pid": Operation not permitted at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Server/Prefork.pm line 20. not sure if it is related but i thought to mention it.

I think this issue should have higher priority

livdywan wrote in #note-15:

Apparently the patch to the log level causes the pipeline to fail https://gitlab.suse.de/qa-maintenance/qamops/-/jobs/2400856

I reverted it and restart the service. And i also retriggered https://gitlab.suse.de/qa-maintenance/qamops/-/pipelines/1046526

I restarted ltp_syscalls from https://openqa.suse.de/tests/overview?build=%3A32948%3Akernel-ec2&groupid=193

journalctl -u dashboard-amqp-watcher.service | grep 13814830
Mar 18 21:07:13 qam2 dashboard[16674]:     "13778984" => 13814830
Mar 18 21:07:13 qam2 dashboard[16674]: [16674] [i] restart 13778984 -> 13814830
Mar 18 21:40:14 qam2 dashboard[16674]:   "id" => 13814830,
Mar 18 21:40:14 qam2 dashboard[16674]: [16674] [i] 13814830: failed (failed)
Mar 20 10:42:54 qam2 dashboard[27458]: [27458] [i] restart 13814830 -> 13827412```

but /blocked data doesnt contain anything after 13 requests. How i check: on dev panel>select /blocked XHR> find (select regex and switch to raw print) > type `:kernel-debug\S+12-SP3-TERADATA Kernel`
Also checking the queues of messages, i dont see anything for this job

Maybe is easier to debug the functionality using the https://dashboard.qam.suse.de/incident/ and checking the responses where
something like

"incident_summary": {
            "failed": 1,
            "passed": 49,
            "waiting": 1
        },

shows the inconsistent data

ybonatakis wrote in #note-19:

Maybe is easier to debug the functionality using the https://dashboard.qam.suse.de/incident/ and checking the responses where
something like

"incident_summary": {
            "failed": 1,
            "passed": 49,
            "waiting": 1
        },

shows the inconsistent data

My mistake. because data looks correct. but due to https://github.com/openSUSE/qem-dashboard/blob/61972b1b63f5f29c9ab17237cc933b0e243b9d2c/assets/vue/components/ResultSummary.vue#L14 the red box is shown as it has priority over the waiting kobs

So what happens is:

1. Jobs are running. The box on the blocked page is blue as expected.
2. One job fails (e.g. https://openqa.suse.de/tests/13826903). The box is red as expected.
3. A new job is created (e.g. openQA restarts the job) and this new/scheduled job takes its place in openQA's test result overview. The box is still red, that's probably not expected. It would probably make more sense if the box would go back to blue (and the job was considered "waiting" instead of "failed").

Maybe it is already intended that the box goes back blue and that's just not working because:

• openQA doesn't create the expected event for the newly scheduled job
• the event isn't evaluated correctly by the dashboard's AMQP bot

So I guess we need to check those two points.

I just found out a weird AMQP message:

[2024-06-20 13:38:12.19083] [22997] [debug] HANDLE suse.openqa.job.restart {
  "ARCH" => "ppc64le",
  "BUILD" => "144.8",
  "FLAVOR" => "Online-QR",
  "ISO" => "SLE-15-SP5-Online-ppc64le-Build144.8-Media1.iso",
  "MACHINE" => "ppc64le-hmc-single-disk",
  "TEST" => "create_hdd_textmode_hmc",
  "auto" => 0,
  "bugref" => undef,
  "group_id" => 517,
  "id" => 14674537,
  "remaining" => 14,
  "result" => {
    "14674537" => 14692218,
    "14674538" => undef,
    "14674539" => 14692219,
    "14674540" => 14692220,
    "14674541" => 14692221,
    "14674542" => 14692222,
    "14674543" => 14692223,
    "14674544" => 14692224,
    "14674545" => 14692225,
    "14674546" => 14692226,
    "14674547" => 14692227,
    "14674548" => 14692228,
    "14674549" => 14692229,
    "14674550" => 14692230,
    "14674551" => 14692231
  }
}

Note the undef.

jbaier_cz wrote in #note-31:

I would guess this could count as a validation https://suse.slack.com/archives/C02CANHLANP/p1719403335705509

As proprietary solutions have a tendency to go away and be unaccessible eventually let me quote the complete conversation behind that URL:

(Felix Niederwanger) Hey QA-Tools team! I love the new feature where http://dashboard.qam.suse.de/ respons very quickly to restarted jobs! How is this done internally? RabbitMQ?
(Jan Baier) yes, dashboard app is listening to AMQP events emitted from openQA
(Felix Niederwanger) :wow_amaze: This is very cool! Kudos :thumbs-up: