Project

General

Profile

Actions
Copy link

action #120973

closed

[qem-dashboard] 500 internal server errors reported by qem-bot

Added by kraih about 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
kraih
Category:
-
Target version:
openQA Project (public) - Ready
Start date:
2022-11-25
Due date:
% Done:

0%

Estimated time:
Tags:
reactive work

Description

Observation

The qem-bot pipeline https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1256680 started reporting 500 errors today:

2022-11-25 13:47:50 ERROR    Smelt Incidents were not synced to dashboard: error 500
2022-11-25 13:47:50 ERROR    Smelt Incidents were not synced to dashboard: error 500
2022-11-25 13:47:50 ERROR    Smelt Incidents were not synced to dashboard: error 500

A look at the qem-dashboard logs showed that it appears to be responsible for the errors:

Nov 25 12:13:56 qam2 dashboard[17595]: [17595] [e] [Fv4nYiFO8Hb9] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:13:56 qam2 dashboard[17595]: [17595] [e] [qt4QwGntMM0u] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:13:56 qam2 dashboard[12869]: [12869] [e] [YdRxgD1Of09s] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:15:26 qam2 dashboard[26872]: [26872] [e] [y-dXDfJS4mg8] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:15:26 qam2 dashboard[17595]: [17595] [e] [5PdDzxiNwfUM] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:15:26 qam2 dashboard[14450]: [14450] [e] [4K-Md0ngkjF9] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:17:57 qam2 dashboard[17595]: [17595] [e] [-4Bjs_dgJRr1] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261
...

Acceptance criteria

  • AC1: No more 500 errors

Suggestions

  • Look at dashboard input validation
  • Check for recent changes to the qem-bot and possibly do a rollback
Actions
Copy link
 #1

Updated by kraih about 2 years ago

In the dashboard logs the problem was first seen at Nov 25 12:13:56. And the SQL query in question is an update that should only be getting data validated with a JSON schema:

  $db->query(
    'UPDATE incidents SET packages = ?, rr_number = ?, review = ?, review_qam = ?, approved = ?, emu = ?, active = ?
       WHERE id = ?', $incident->{packages}, $incident->{rr_number}, $incident->{inReview}, $incident->{inReviewQAM},
    $incident->{approved}, $incident->{emu}, $incident->{isActive}, $id
  );
Actions
Copy link
 #2

Updated by kraih about 2 years ago

The dashboard is at least partially at fault here because invalid data gets past the validation, but i suspect that the root cause is a regression in the qem-bot from a patch that has been applied today.

Actions
Copy link
 #3

Updated by kraih about 2 years ago

It appears some of the qem-bot refactorings from today made certain methods not return booleans anymore.

Actions
Copy link
 #4

Updated by kraih about 2 years ago

Tina made a qem-bot PR to fix method return values: https://github.com/openSUSE/qem-bot/pull/88

Actions
Copy link
 #5

Updated by tinita about 2 years ago

https://github.com/openSUSE/qem-bot/pull/88 - Fix boolean return values (merged)

Hopefully this fixes it

Actions
Copy link
 #6

Updated by jbaier_cz about 2 years ago

tinita wrote:

https://github.com/openSUSE/qem-bot/pull/88 - Fix boolean return values (merged)

Hopefully this fixes it

Nice job: https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1256805

Actions
Copy link
 #7

Updated by kraih about 2 years ago

The bad pipeline https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1256680 actually shows the case:

  'emu': False,
  'inReview': True,
  'inReviewQAM': [],
  'isActive': True,
Actions
Copy link
 #8

Updated by kraih about 2 years ago

I can't seem to get past validation in tests, very strange case. But i've added tests at least: https://github.com/openSUSE/qem-dashboard/commit/2d3257d4c0e2a98efa61fe897860f86bd7d39385

Actions
Copy link
 #9

Updated by kraih about 2 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #10

Updated by kraih about 2 years ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #11

Updated by kraih about 2 years ago

Thanks everyone who helped!

Actions
Copy link
 #12

Updated by kraih about 2 years ago

To be sure i also looked up the version of JSON::Validator on the production machine and tested with that, but also no problem there:

Information for package perl-JSON-Validator:
--------------------------------------------
Repository     : openSUSE_Leap
Name           : perl-JSON-Validator
Version        : 5.03-bp154.1.23
Actions
Copy link
 #13

Updated by kraih about 2 years ago

  • Status changed from Feedback to Resolved
Actions
Copy link

Also available in: Atom PDF