action #40973: worker retry seems to be bogus - openQA Project (public) - openSUSE Project Management Tool

Actions

Copy link

action #40973

closed

worker retry seems to be bogus

Added by coolo over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

mkittler

Category:

Target version:

Done

Start date:

2018-09-13

Due date:

% Done:

Estimated time:

Description

We had a problem on the openqa webui this morning, so the workers reporting upload progress
went into a 500, but what happened was a mass storm of api calls.

Looking into the worker log, I see

Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -9)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -11)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -366)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -398)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -420)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -398)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 503 response: Service Unavailable (remaining tries: 0)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] Job aborted because web UI doesn't accept updates anymore (likely considers this job dead)

Actions

Copy link

Updated by coolo over 6 years ago

this is how it all starts:

Sep 13 08:39:52 openqaworker13 worker[15163]: [info] 17581: WORKING 2041202
Sep 13 08:40:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:40:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:40:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:40:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:40:22 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:40:27 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -3)
Sep 13 08:40:32 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -4)
Sep 13 08:40:37 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -5)
Sep 13 08:40:42 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -6)
Sep 13 08:40:47 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -7)
Sep 13 08:40:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -8)
Sep 13 08:40:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -9)
Sep 13 08:41:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -10)
Sep 13 08:41:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -11)
Sep 13 08:41:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -12)
Sep 13 08:41:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -13)
Sep 13 08:41:22 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -14)
Sep 13 08:41:27 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -15)
Sep 13 08:41:32 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -16)
Sep 13 08:41:37 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -17)
Sep 13 08:41:42 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -18)
Sep 13 08:41:47 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -19)
Sep 13 08:41:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:41:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -20)
Sep 13 08:41:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:41:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:41:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -21)
Sep 13 08:41:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -22)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -23)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -24)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -3)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -25)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -3)
Sep 13 08:42:23 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:42:23 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -4)
Sep 13 08:42:23 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -26)

Actions

Copy link