Project

General

Profile

action #120973

[qem-dashboard] 500 internal server errors reported by qem-bot

Added by kraih 2 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2022-11-25
Due date:
% Done:

0%

Estimated time:

Description

Observation

The qem-bot pipeline https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1256680 started reporting 500 errors today:

2022-11-25 13:47:50 ERROR    Smelt Incidents were not synced to dashboard: error 500
2022-11-25 13:47:50 ERROR    Smelt Incidents were not synced to dashboard: error 500
2022-11-25 13:47:50 ERROR    Smelt Incidents were not synced to dashboard: error 500

A look at the qem-dashboard logs showed that it appears to be responsible for the errors:

Nov 25 12:13:56 qam2 dashboard[17595]: [17595] [e] [Fv4nYiFO8Hb9] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:13:56 qam2 dashboard[17595]: [17595] [e] [qt4QwGntMM0u] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:13:56 qam2 dashboard[12869]: [12869] [e] [YdRxgD1Of09s] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:15:26 qam2 dashboard[26872]: [26872] [e] [y-dXDfJS4mg8] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:15:26 qam2 dashboard[17595]: [17595] [e] [5PdDzxiNwfUM] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:15:26 qam2 dashboard[14450]: [14450] [e] [4K-Md0ngkjF9] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261.
Nov 25 12:17:57 qam2 dashboard[17595]: [17595] [e] [-4Bjs_dgJRr1] DBD::Pg::st execute failed: ERROR:  invalid input syntax for type boolean: "{}" at /home/lurklur/qem-dashboard/lib/Dashboard/Model/Incidents.pm line 261
...

Acceptance criteria

  • AC1: No more 500 errors

Suggestions

  • Look at dashboard input validation
  • Check for recent changes to the qem-bot and possibly do a rollback

History

#1 Updated by kraih 2 months ago

In the dashboard logs the problem was first seen at Nov 25 12:13:56. And the SQL query in question is an update that should only be getting data validated with a JSON schema:

  $db->query(
    'UPDATE incidents SET packages = ?, rr_number = ?, review = ?, review_qam = ?, approved = ?, emu = ?, active = ?
       WHERE id = ?', $incident->{packages}, $incident->{rr_number}, $incident->{inReview}, $incident->{inReviewQAM},
    $incident->{approved}, $incident->{emu}, $incident->{isActive}, $id
  );

#2 Updated by kraih 2 months ago

The dashboard is at least partially at fault here because invalid data gets past the validation, but i suspect that the root cause is a regression in the qem-bot from a patch that has been applied today.

#3 Updated by kraih 2 months ago

It appears some of the qem-bot refactorings from today made certain methods not return booleans anymore.

#4 Updated by kraih 2 months ago

Tina made a qem-bot PR to fix method return values: https://github.com/openSUSE/qem-bot/pull/88

#5 Updated by tinita 2 months ago

https://github.com/openSUSE/qem-bot/pull/88 - Fix boolean return values (merged)

Hopefully this fixes it

#6 Updated by jbaier_cz 2 months ago

tinita wrote:

https://github.com/openSUSE/qem-bot/pull/88 - Fix boolean return values (merged)

Hopefully this fixes it

Nice job: https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1256805

#7 Updated by kraih 2 months ago

The bad pipeline https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1256680 actually shows the case:

  'emu': False,
  'inReview': True,
  'inReviewQAM': [],
  'isActive': True,

#8 Updated by kraih 2 months ago

I can't seem to get past validation in tests, very strange case. But i've added tests at least: https://github.com/openSUSE/qem-dashboard/commit/2d3257d4c0e2a98efa61fe897860f86bd7d39385

#9 Updated by kraih 2 months ago

  • Status changed from New to In Progress

#10 Updated by kraih 2 months ago

  • Status changed from In Progress to Feedback

#11 Updated by kraih 2 months ago

Thanks everyone who helped!

#12 Updated by kraih 2 months ago

To be sure i also looked up the version of JSON::Validator on the production machine and tested with that, but also no problem there:

Information for package perl-JSON-Validator:
--------------------------------------------
Repository     : openSUSE_Leap
Name           : perl-JSON-Validator
Version        : 5.03-bp154.1.23

#13 Updated by kraih 2 months ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF