QA

from logs everything looks ok , but in database:

dashboard_db=# select count(*) from update_openqa_settings where product = 'SLES15SP3';
 count 
-------
   803
(1 row)

dashboard_db=# select count(*) from update_openqa_settings where product = 'SLES15SP4';
 count 
-------
     0
(1 row)

--> https://github.com/openSUSE/qem-bot/pull/54 - for logging result of PUT request..

manually pushed data to database ( using python3> requests.put(url, headers=token,json=data["qem"]) ) with data parsed from gitlab log, all went OK so probably update of dashboard database during run went into hidden problems

--> so we need add retry based on status to post_qem method

https://github.com/openSUSE/qem-bot/pull/58

osukup wrote:

https://github.com/openSUSE/qem-bot/pull/58

This was a very fast PR and review. You guys are awesome 😁️

follow up https://github.com/openSUSE/qem-bot/pull/59

after I looked how is backoff_factor used in urllib3

Hello, I see the ticket is on 'feedback' state but there are still several update requests with only an incident green box and not being auto approved. http://dashboard.qam.suse.de/blocked

Few examples:
http://dashboard.qam.suse.de/incident/24723
http://dashboard.qam.suse.de/incident/24743
http://dashboard.qam.suse.de/incident/24762

Aggregates disappeared again, few minutes ago. Before that they were showing up correctly.
They disappeared just after every single aggregate got green :(
Hope that this was not the cause, since it does not happens often :P

I've made a small change to the dashboard so the journal is not flooded by HTTP request data anymore. Should make it easier to keep track of what data the dashboard cleans up (and rule out any possible regressions there). https://github.com/openSUSE/qem-dashboard/commit/1e6321f02b0c082f5659b24ef97898b24f248fcb

Discussed briefly in the in the Unblock. It would be great if others take a look and review the current code for flaws i.e. pretend it's a new pull request and we can then see if that helps us find some gaps or ideas where to improve logging. A the latest tomorrow in the mob session we can discuss it, or comment here earlier

kraih wrote:

I've made a small change to the dashboard so the journal is not flooded by HTTP request data anymore. Should make it easier to keep track of what data the dashboard cleans up (and rule out any possible regressions there). https://github.com/openSUSE/qem-dashboard/commit/1e6321f02b0c082f5659b24ef97898b24f248fcb

We've investigated this issue during the mob session today. A regression in the cleanup code has been ruled out, there are no log entries for the very recent incident 25413. We have tracked it down to a call of this code from the qem-bot. The bot thinks the update_settings have been added to the database, but they do not appear to have ever been added. The next step will be to find out if the problem is on the bot or dashboard side.

We added logging of post ID to qem-bot , and next we need another occasion of problem to analyze what is going on

needs coop with openqa-qam reviewers ( @mgrifalconi ?)

We do not have a good way to find "incidents with missing aggregate tests" programmatically so we are relying on users to tell us about further suspicious cases. If anybody finds cases where you assume there are missing aggregate tests please tell us and we can take a look into it.

Hello, some tests with No data yet, and more than 1 day in smelt queue:

http://dashboard.qam.suse.de/blocked

http://dashboard.qam.suse.de/incident/24979
http://dashboard.qam.suse.de/incident/25004
http://dashboard.qam.suse.de/incident/25265
http://dashboard.qam.suse.de/incident/25213
http://dashboard.qam.suse.de/incident/25423

Could you please verify? Thank you

I have a new suspicion where the problem could be, but to confirm that i need one concrete example of an incident with missing aggregate jobs from the past few days.

osukup wrote:

We added logging of post ID to qem-bot , and next we need another occasion of problem to analyze what is going on

https://github.com/openSUSE/qem-bot/pull/64/files

Since there's not been much progress recently, i'll look into a possible solution later this week.

My suspicion turned out to be a definitive bug. Aggregate jobs could be deleted by accident if update_settings are being used for longer than 90 days. And the solution i had in mind looks promising in preliminary tests. So i'll try to add some unit tests too and then deploy it to production.

Committed a possible fix. Now we'll have to keep an eye on it again. https://github.com/openSUSE/qem-dashboard/commit/9cf6e655007fad2a366c9a9b4bf6f0f353de69fd

okurz wrote:

Regarding the ACs I tried select * from job_settings where key ~ 'TEST_REPOS' and value ~ '25171' limit 10; but have not found any results. I wonder, can we re-add results so that they show up where they should?

That data would have to come from the bot, not sure if it is possible. Probably not.

I've simplified the ACs, since bringing back old data would be very hard, if it is possible at all.

I went through the incident data currently in the dashboard and it mostly looks fine. There are a few suspicious ones in the blocked list, one was new enough to have been created after the fix was deployed. Incident 26186 (Smelt) had no data at all yet. From the bot logs in the pipelines it looked like jobs were being scheduled though:

INFO: not scheduling: Flavor: Leap-DVD-Incidents, version: 15.3 incident: 26186 , arch: x86_64  - exists in openQA

 {'approved': False,
  'channels': ['SUSE:Updates:SLE-Manager-Tools:15:x86_64',
               'SUSE:Updates:SLE-Manager-Tools:15:s390x',
               'SUSE:Updates:SLE-Manager-Tools:15:ppc64le',
               'SUSE:Updates:SLE-Manager-Tools:15:aarch64',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:aarch64',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:ppc64le',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:s390x',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:x86_64',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.3:x86_64',
               'SUSE:Updates:openSUSE-SLE:15.4',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.3:s390x',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.3:ppc64le',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.3:aarch64',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.2:x86_64',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.2:s390x',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.2:ppc64le',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.2:aarch64',
               'SUSE:SLE-15-SP1:Update',
               'SUSE:Updates:Storage:6:x86_64',
               'SUSE:Updates:Storage:6:aarch64',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.1:x86_64',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.1:s390x',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.1:ppc64le',
               'SUSE:Updates:SLE-Module-SUSE-Manager-Proxy:4.1:aarch64',
               'SUSE:Updates:openSUSE-SLE:15.3',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:aarch64',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:ppc64le',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:s390x',
               'SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:x86_64'],
  'emu': False,
  'inReview': True,
  'inReviewQAM': True,
  'isActive': True,
  'number': 26186,
  'packages': ['golang-github-prometheus-alertmanager'],
  'project': 'SUSE:Maintenance:26186',
  'rr_number': 281776}

INFO: Inc 26186 does not have any aggregates settings
...
INFO: Inc 26186 has failed job in incidents

But after speaking with Ondrej i found out that the missing results in the dashboard are intentional. Jobs are scheduled for the products openSUSE-SLE:15.3/15.4, but both are still in the development folder, and therefore all openQA test results are ignored by the bot and never reach the dashboard. This condition should probably be highlighted better, because the reviewers were also not aware of it. But so far it looks like we don't have any missing aggregate job results since the fix has been deployed.

I will be disabling the cleanup feature completely now.

And all removed update settings will be logged. https://github.com/openSUSE/qem-dashboard/commit/4a4ea13506ba4ae8be1266667da80ca284db53a8

The next wave of aggregate jobs has arrived on the dashboard and it looks good again. Now we have to wait and see again. If something goes wrong the new logging will help us locate the problem.

So far the data in the dashboard is looking good. If it stays that way i'll permanently remove the cleanup feature and call the issue resolved.

And we've had another case today, pretty much all aggregate jobs vanished form the dashboard at once. Reviewing the logs and database now.

Quite interesting results in the logs. The data must have vanished between 11:00 and 15:00. With the cleanup feature disabled now only the incident reuse feature can delete update settings. And there was only one case today:

Oct 17 12:14:18 qam2 dashboard[26091]: [26091] [i] Cleaning up old jobs for incident 26179, rr_number change: 282072 -> 282453
Oct 17 12:14:18 qam2 dashboard[26091]: [26091] [i] Update settings cleaned up for incident 26179: 49611, 49593, 49495, 49703, 49471, 49479, 49492, 49689, 49708, 49853, 49712, 49706, 49806, 49489, 49488, 49630,
49582, 49613, 49698, 49815, 49756, 49800, 49744, 49602, 49587, 49646, 49502, 49486, 49757, 49581, 49583,
49580, 49817, 49812, 49604, 49588, 49485, 49854, 49645, 49754, 49704, 49612, 49422, 49600, 49594, 49743, 49524, 49821, 49589, 49476, 49595, 49537, 49522, 49523,
49831, 49824, 49692, 49813, 49820, 49427, 49591, 49536, 49825, 49707, 49851, 49482, 49597, 49690, 49713, 49532, 49490, 49710, 49585, 49818, 49810, 49474, 49424,
49809, 49752, 49701, 49501, 49481, 49816, 49642, 49814, 49525, 49426, 49421, 49702, 49801, 49478, 49472, 49586, 49477, 49832, 49491, 49521, 49751, 49811,
49695, 49822, 49740, 49592, 49469, 49644, 49855, 49590, 49850, 49470, 49802, 49601, 49535, 49694, 49711, 49709, 49531, 49741, 49714, 49631, 49584, 49803, 49723,
49635, 49697, 49520, 49693, 49691, 49596, 49641, 49473, 49425, 49699, 49634, 49603, 49715, 49534, 49487, 49805, 49696, 49480, 49833, 49633, 49599, 49742, 49598,
49722, 49705, 49721, 49494, 49605, 49755, 49799, 49503, 49852, 49745, 49493, 49804, 49647, 49700, 49632, 49579, 49483, 49475, 49823, 49807, 49808, 49484,
49819

So the only remaining explanation is that this incident reuse caused a cascading delete of update settings and their jobs that was large enough to clear the entire blocked list. That is certainly unexpected, but opens up a new option for a possible fix. Instead of deleting all the associated data, we could also just sever the link the between incident number and update settings in the incident_in_update table.

I've deployed a new fix that only removes the link between incident number and update settings, but does not remove the settings/jobs anymore. https://github.com/openSUSE/qem-dashboard/commit/e6c1c8914458a3e92055d09be148bf65e70c8793

Feedback from test reviewers has been good so far. I'm not going to call this resolved yet, since i don't want to jinx it again. :)

cdywan wrote:

I guess we can wait til the end of the week

Or someone else than me sets it to resolved. ;)

All data vanishing today was not related to this ticket, it was just a SMELT outage.