action #81492
closedopenQA-in-openQA fails with 'function' object has no attribute 'func_name' and label:non_existing asset, candidate for removal or wrong settings
Description
Observation¶
New comment in project devel:openQA by okurz:
Oliver Kurz wrote in project devel:openQA:
test failed, see https://openqa.opensuse.org/tests/overview?version=Tumbleweed&groupid=24
Which gets invoked via Jenkins and reveals the following error (full log attached in case it's wiped):
++ /usr/share/openqa/script/client --host https://openqa.opensuse.org jobs/1533864 200817
[...]
Error initializing plugin Google = keyrings.alt.Google.
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 171, in _load_plugins
init_func = ep.load()
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2343, in load
return self.resolve()
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2349, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/python3.6/site-packages/keyrings/alt/Google.py", line 14, in <module>
import gdata.docs.service
File "/usr/lib/python3.6/site-packages/gdata/__init__.py", line 27, in <module>
import atom
File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 133, in <module>
CreateClassFromXMLString)
File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 97, in mark_deprecated
optional_warn_function.func_name = f.func_name
AttributeError: 'function' object has no attribute 'func_name'
+ osc api -X DELETE /comment/1382814
Error initializing plugin Google = keyrings.alt.Google.
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 171, in _load_plugins
init_func = ep.load()
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2343, in load
return self.resolve()
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2349, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/python3.6/site-packages/keyrings/alt/Google.py", line 14, in <module>
import gdata.docs.service
File "/usr/lib/python3.6/site-packages/gdata/__init__.py", line 27, in <module>
import atom
File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 133, in <module>
CreateClassFromXMLString)
File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 97, in mark_deprecated
optional_warn_function.func_name = f.func_name
AttributeError: 'function' object has no attribute 'func_name'
<status code="ok">
<summary>Ok</summary>
</status>
+ read -r id
+ osc api '--data=test failed, see https://openqa.opensuse.org/tests/overview?version=Tumbleweed&groupid=24' -X POST /comments/project/devel:openQA
Error initializing plugin Google = keyrings.alt.Google.
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/keyring/backend.py", line 171, in _load_plugins
init_func = ep.load()
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2343, in load
return self.resolve()
File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2349, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/python3.6/site-packages/keyrings/alt/Google.py", line 14, in <module>
import gdata.docs.service
File "/usr/lib/python3.6/site-packages/gdata/__init__.py", line 27, in <module>
import atom
File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 133, in <module>
CreateClassFromXMLString)
File "/usr/lib/python3.6/site-packages/atom/__init__.py", line 97, in mark_deprecated
optional_warn_function.func_name = f.func_name
AttributeError: 'function' object has no attribute 'func_name'
<status code="ok">
<summary>Ok</summary>
</status>
Further observations¶
The failed jobs have this comment:
label:non_existing asset, candidate for removal or wrong settings
Files
Updated by Xiaojing_liu over 3 years ago
I guess this failed because there is no asset opensuse-Tumbleweed-x86_64-20201227-Tumbleweed@64bit.qcow2 in openqa.opensuse.org.
We could see the log from: http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/6848/console
+ latest_published_tw_build=20201227
+ [[ -n 20201227 ]]
++ /usr/share/openqa/script/openqa-cli api --host https://openqa.opensuse.org assets get
++ jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-20201227-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'
+ qcow=null
+ [[ -n null ]]
+ '[' https://openqa.opensuse.org '!=' https://openqa.opensuse.org ']'
++ echo jenkins-trigger-openQA_in_openQA-TW-6848
++ sed -e s/jenkins-trigger-openQA_in_openQA-/:/ -e s/-/./g
+ build=:TW.6848
+ trigger
+ '[' openqa.opensuse.org = openqa.opensuse.org ']'
+ ARGS=OPENQA_HOST=http://openqa.opensuse.org
+ /usr/share/openqa/script/client --host https://openqa.opensuse.org isos post VERSION=Tumbleweed DISTRI=openQA FLAVOR=dev BUILD=:TW.6848 ARCH=x86_64 BACKEND=qemu WORKER_CLASS=qemu_x86_64 HDD_1=null OPENQA_HOST=http://openqa.opensuse.org
The script did not find the asset that named Tumbleweed-x86_64-20201227-Tumbleweed\\@64bit.qcow, so it set the HDD_1=null. And the null caused the download to fail.
Compare to the successful job's log, when the asset exist, the HDD_1 will be its name, such as: http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/6840/console shows:
/usr/share/openqa/script/openqa-cli api --host https://openqa.opensuse.org assets get
++ jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-20201224-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'
+ qcow=opensuse-Tumbleweed-x86_64-20201224-Tumbleweed@64bit.qcow2
+ [[ -n opensuse-Tumbleweed-x86_64-20201224-Tumbleweed@64bit.qcow2 ]]
+ '[' https://openqa.opensuse.org '!=' https://openqa.opensuse.org ']'
++ echo jenkins-trigger-openQA_in_openQA-TW-6840
++ sed -e s/jenkins-trigger-openQA_in_openQA-/:/ -e s/-/./g
+ build=:TW.6840
+ trigger
+ '[' openqa.opensuse.org = openqa.opensuse.org ']'
+ ARGS=OPENQA_HOST=http://openqa.opensuse.org
+ /usr/share/openqa/script/client --host https://openqa.opensuse.org isos post VERSION=Tumbleweed DISTRI=openQA FLAVOR=dev BUILD=:TW.6840 ARCH=x86_64 BACKEND=qemu WORKER_CLASS=qemu_x86_64 HDD_1=opensuse-Tumbleweed-x86_64-20201224-Tumbleweed@64bit.qcow2 OPENQA_HOST=http://openqa.opensuse.org
Updated by livdywan over 3 years ago
- Category set to Regressions/Crashes
- Status changed from New to In Progress
- Assignee set to livdywan
- Priority changed from Normal to Urgent
- Target version set to Ready
After looking at the jq magic I realized it's not using the latest image but looking for one that doesn't exist... I wonder why it's not using a regex to get the latest one 🤔
openqa-cli api --host https://openqa.opensuse.org assets get | jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-[0-9]+-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'seems to get me the image we probably want?- We should probably never try to clone a job if we didn't find an image in the first place
Will try and come up with a fix
Updated by livdywan over 3 years ago
Updated by openqa_review over 3 years ago
- Due date set to 2021-01-13
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan over 3 years ago
The above fix got merged and as expected, Jenkins now fails with Unable to determine qcow image for Tumbleweed build '20210102' (for architecture 'x86_64' and machine '64bit')..
In other words, 20210102 was not found. The server has 20210103, 20201229 and 20200131. But the last published images are 20210102, 20210101, 20201231, 20201229 etc so it seems like we're out of sync. We have no image even though it's listed as published and we got a more recent image. And notice the 20201227 which previously wasn't found is also in that list. But 20201231 was published and had an image so it was temporarily working then.
Updated by livdywan over 3 years ago
Hence my suggestion is to pick the most recent image that we have a published build for, which makes it work in the situation described by my previous comment.
https://github.com/os-autoinst/scripts/pull/64
I'm not sure if we could fix the fact that assets can get out of sync. My guess is that clean-ups cause it so we might not be able to fix it if it's a race condition but rely on the fallback.
Updated by livdywan over 3 years ago
- Status changed from In Progress to Feedback
Updated by okurz over 3 years ago
- Status changed from Feedback to Resolved