action #40973

worker retry seems to be bogus

Added by coolo over 1 year ago. Updated over 1 year ago.

Status:ResolvedStart date:13/09/2018
Priority:NormalDue date:
Assignee:mkittler% Done:

0%

Category:-
Target version:Done
Difficulty:
Duration:

Description

We had a problem on the openqa webui this morning, so the workers reporting upload progress
went into a 500, but what happened was a mass storm of api calls.

Looking into the worker log, I see

Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -9)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -11)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -366)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -398)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -420)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 500 response: Internal Server Error (remaining tries: -398)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] 503 response: Service Unavailable (remaining tries: 0)
Sep 13 11:20:21 openqaworker13 worker[15148]: [error] Job aborted because web UI doesn't accept updates anymore (likely considers this job dead)

History

#1 Updated by coolo over 1 year ago

this is how it all starts:

Sep 13 08:39:52 openqaworker13 worker[15163]: [info] 17581: WORKING 2041202
Sep 13 08:40:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:40:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:40:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:40:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:40:22 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:40:27 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -3)
Sep 13 08:40:32 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -4)
Sep 13 08:40:37 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -5)
Sep 13 08:40:42 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -6)
Sep 13 08:40:47 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -7)
Sep 13 08:40:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -8)
Sep 13 08:40:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -9)
Sep 13 08:41:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -10)
Sep 13 08:41:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -11)
Sep 13 08:41:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -12)
Sep 13 08:41:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -13)
Sep 13 08:41:22 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -14)
Sep 13 08:41:27 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -15)
Sep 13 08:41:32 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -16)
Sep 13 08:41:37 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -17)
Sep 13 08:41:42 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -18)
Sep 13 08:41:47 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -19)
Sep 13 08:41:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:41:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -20)
Sep 13 08:41:52 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:41:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:41:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -21)
Sep 13 08:41:57 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 2)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -22)
Sep 13 08:42:02 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 1)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -23)
Sep 13 08:42:07 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: 0)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -24)
Sep 13 08:42:12 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -1)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -3)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -25)
Sep 13 08:42:17 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -3)
Sep 13 08:42:23 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -2)
Sep 13 08:42:23 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -4)
Sep 13 08:42:23 openqaworker13 worker[15163]: [error] 500 response: Internal Server Error (remaining tries: -26)

#2 Updated by coolo over 1 year ago

  • Status changed from New to Resolved
  • Assignee set to mkittler
  • Target version set to Done

Also available in: Atom PDF