Project

General

Profile

Actions

action #162644

closed

Job fails with api failure: 403 response: timestamp mismatch after brief osd outage

Added by livdywan 6 months ago. Updated 6 months ago.

Status:
Rejected
Priority:
High
Assignee:
Category:
Infrastructure
Start date:
2024-06-20
Due date:
% Done:

0%

Estimated time:
Difficulty:
Tags:

Description

Observation

openQA test in scenario sle-15-SP5-Online-QR-x86_64-create_hdd_minimal_base+sdk@64bit fails in
welcome

The test failed with this reason:

api failure: 403 response: timestamp mismatch - check whether clocks on the local host and the web UI host are in sync

There were also reports of osd being inaccessible around that time.

There was a ton of Updating seen of worker and influxdb/telegraf timing out (sudo journalctl -S '2024-06-20 15:50:00' -U '2024-06-20 16:00:00').

The outage is also seen on grafana: https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=1718889892165&to=1718893680225

Reproducible

Fails since (at least) Build testdev144.8 (current job)

Expected result

Last good: testdev144.8 (or more recent)

Further details

Always latest result in this scenario: latest


Related issues 2 (0 open2 closed)

Related to openQA Infrastructure (public) - action #162038: No HTTP Response on OSD on 10-06-2024 - auto_review:".*timestamp mismatch - check whether clocks on the local host and the web UI host are in sync":retry size:SResolvednicksinger2024-06-10

Actions
Related to Containers and images - action #162014: api failure: 403 response: timestamp mismatchRejected2024-06-10

Actions
Actions #1

Updated by livdywan 6 months ago

  • Description updated (diff)
Actions #2

Updated by okurz 6 months ago

  • Related to action #162038: No HTTP Response on OSD on 10-06-2024 - auto_review:".*timestamp mismatch - check whether clocks on the local host and the web UI host are in sync":retry size:S added
Actions #3

Updated by okurz 6 months ago

  • Related to action #162014: api failure: 403 response: timestamp mismatch added
Actions #4

Updated by okurz 6 months ago

  • Tags set to infra, osd
  • Target version set to Ready
Actions #5

Updated by okurz 6 months ago

  • Description updated (diff)
Actions #6

Updated by okurz 6 months ago

  • Status changed from New to Rejected
  • Assignee set to okurz

merged into #162038

Actions

Also available in: Atom PDF