Project

General

Profile

action #81492

openQA-in-openQA fails with 'function' object has no attribute 'func_name' and label:non_existing asset, candidate for removal or wrong settings

Added by cdywan 7 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2020-12-29
Due date:
2021-01-13
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

New comment in project devel:openQA by okurz:

 Oliver Kurz wrote in project devel:openQA:

test failed, see https://openqa.opensuse.org/tests/overview?version=Tumbleweed&groupid=24

Which gets invoked via Jenkins and reveals the following error (full log attached in case it's wiped):

++ /usr/share/openqa/script/client --host https://openqa.opensuse.org jobs/1533864 200817
[...]
Error initializing plugin Google = keyrings.alt.Google.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 171, in _load_plugins
    init_func = ep.load()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2343, in load
    return self.resolve()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2349, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.6/site-packages/keyrings/alt/Google.py", line 14, in <module>
    import gdata.docs.service
  File "/usr/lib/python3.6/site-packages/gdata/__init__.py", line 27, in <module>
    import atom
  File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 133, in <module>
    CreateClassFromXMLString)
  File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 97, in mark_deprecated
    optional_warn_function.func_name = f.func_name
AttributeError: 'function' object has no attribute 'func_name'
+ osc api -X DELETE /comment/1382814
Error initializing plugin Google = keyrings.alt.Google.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 171, in _load_plugins
    init_func = ep.load()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2343, in load
    return self.resolve()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2349, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.6/site-packages/keyrings/alt/Google.py", line 14, in <module>
    import gdata.docs.service
  File "/usr/lib/python3.6/site-packages/gdata/__init__.py", line 27, in <module>
    import atom
  File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 133, in <module>
    CreateClassFromXMLString)
  File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 97, in mark_deprecated
    optional_warn_function.func_name = f.func_name
AttributeError: 'function' object has no attribute 'func_name'
<status code="ok">
  <summary>Ok</summary>
</status>
+ read -r id
+ osc api '--data=test failed, see https://openqa.opensuse.org/tests/overview?version=Tumbleweed&groupid=24' -X POST /comments/project/devel:openQA
Error initializing plugin Google = keyrings.alt.Google.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 171, in _load_plugins
    init_func = ep.load()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2343, in load
    return self.resolve()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2349, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.6/site-packages/keyrings/alt/Google.py", line 14, in <module>
    import gdata.docs.service
  File "/usr/lib/python3.6/site-packages/gdata/__init__.py", line 27, in <module>
    import atom
  File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 133, in <module>
    CreateClassFromXMLString)
  File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 97, in mark_deprecated
    optional_warn_function.func_name = f.func_name
AttributeError: 'function' object has no attribute 'func_name'
<status code="ok">
  <summary>Ok</summary>
</status>

Further observations

The failed jobs have this comment:

label:non_existing asset, candidate for removal or wrong settings
consoleText.txt (11.3 KB) consoleText.txt cdywan, 2020-12-29 10:44

History

#1 Updated by Xiaojing_liu 7 months ago

I guess this failed because there is no asset opensuse-Tumbleweed-x86_64-20201227-Tumbleweed@64bit.qcow2 in openqa.opensuse.org.
We could see the log from: http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/6848/console

   + latest_published_tw_build=20201227
+ [[ -n 20201227 ]]
++ /usr/share/openqa/script/openqa-cli api --host https://openqa.opensuse.org assets get
++ jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-20201227-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'
+ qcow=null
+ [[ -n null ]]
+ '[' https://openqa.opensuse.org '!=' https://openqa.opensuse.org ']'
++ echo jenkins-trigger-openQA_in_openQA-TW-6848
++ sed -e s/jenkins-trigger-openQA_in_openQA-/:/ -e s/-/./g
+ build=:TW.6848
+ trigger
+ '[' openqa.opensuse.org = openqa.opensuse.org ']'
+ ARGS=OPENQA_HOST=http://openqa.opensuse.org
+ /usr/share/openqa/script/client --host https://openqa.opensuse.org isos post VERSION=Tumbleweed DISTRI=openQA FLAVOR=dev BUILD=:TW.6848 ARCH=x86_64 BACKEND=qemu WORKER_CLASS=qemu_x86_64 HDD_1=null OPENQA_HOST=http://openqa.opensuse.org

The script did not find the asset that named Tumbleweed-x86_64-20201227-Tumbleweed\\@64bit.qcow, so it set the HDD_1=null. And the null caused the download to fail.

Compare to the successful job's log, when the asset exist, the HDD_1 will be its name, such as: http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/6840/console shows:

 /usr/share/openqa/script/openqa-cli api --host https://openqa.opensuse.org assets get
++ jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-20201224-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'
+ qcow=opensuse-Tumbleweed-x86_64-20201224-Tumbleweed@64bit.qcow2
+ [[ -n opensuse-Tumbleweed-x86_64-20201224-Tumbleweed@64bit.qcow2 ]]
+ '[' https://openqa.opensuse.org '!=' https://openqa.opensuse.org ']'
++ echo jenkins-trigger-openQA_in_openQA-TW-6840
++ sed -e s/jenkins-trigger-openQA_in_openQA-/:/ -e s/-/./g
+ build=:TW.6840
+ trigger
+ '[' openqa.opensuse.org = openqa.opensuse.org ']'
+ ARGS=OPENQA_HOST=http://openqa.opensuse.org
+ /usr/share/openqa/script/client --host https://openqa.opensuse.org isos post VERSION=Tumbleweed DISTRI=openQA FLAVOR=dev BUILD=:TW.6840 ARCH=x86_64 BACKEND=qemu WORKER_CLASS=qemu_x86_64 HDD_1=opensuse-Tumbleweed-x86_64-20201224-Tumbleweed@64bit.qcow2 OPENQA_HOST=http://openqa.opensuse.org

#2 Updated by cdywan 7 months ago

  • Category set to Concrete Bugs
  • Status changed from New to In Progress
  • Assignee set to cdywan
  • Priority changed from Normal to Urgent
  • Target version set to Ready

After looking at the jq magic I realized it's not using the latest image but looking for one that doesn't exist... I wonder why it's not using a regex to get the latest one 🤔

  • openqa-cli api --host https://openqa.opensuse.org assets get | jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-[0-9]+-Tumbleweed\\@64bit.qcow"))] | .[0] | .name' seems to get me the image we probably want?
  • We should probably never try to clone a job if we didn't find an image in the first place

Will try and come up with a fix

#4 Updated by openqa_review 7 months ago

  • Due date set to 2021-01-13

Setting due date based on mean cycle time of SUSE QE Tools

#5 Updated by cdywan 7 months ago

The above fix got merged and as expected, Jenkins now fails with Unable to determine qcow image for Tumbleweed build '20210102' (for architecture 'x86_64' and machine '64bit')..

In other words, 20210102 was not found. The server has 20210103, 20201229 and 20200131. But the last published images are 20210102, 20210101, 20201231, 20201229 etc so it seems like we're out of sync. We have no image even though it's listed as published and we got a more recent image. And notice the 20201227 which previously wasn't found is also in that list. But 20201231 was published and had an image so it was temporarily working then.

#6 Updated by cdywan 7 months ago

Hence my suggestion is to pick the most recent image that we have a published build for, which makes it work in the situation described by my previous comment.

https://github.com/os-autoinst/scripts/pull/64

I'm not sure if we could fix the fact that assets can get out of sync. My guess is that clean-ups cause it so we might not be able to fix it if it's a race condition but rely on the fallback.

#7 Updated by cdywan 7 months ago

  • Status changed from In Progress to Feedback

Also available in: Atom PDF