action #91752
closedjenkins: Multiple missing fields and errors in configuration of openQA-in-openQA
Description
Updated by livdywan over 3 years ago
I made an effort going by my "5 minutes rule" finding out if I could fix this easily. I tried to remove unused sections and clear invalid options that don't seem to be enabled.
But I get what looks like validation errors even on e.g. VNC server even though it's not enabled and clearing the field doesn't work either.
Updated by ilausuch over 3 years ago
- Description updated (diff)
- Status changed from New to Workable
Updated by livdywan over 3 years ago
- Blocks action #88754: openQA-in-openQA tests always fail and results do not impact submission pipeline added
Updated by okurz over 3 years ago
- Subject changed from jenkings: Multiple missing fields and errors in configuration to jenkins: Multiple missing fields and errors in configuration of openQA-in-openQA
- Target version set to Ready
Updated by mkittler over 3 years ago
Looks like there's a lot broken. I also often ran into timeouts when clicking though the Jenkins web UI. Could we move this to a GitLab pipeline?
Of course we need to figure out what this Jeninks jobs actually do to move it to GitLab. It looks like Jenkin executes three commands:
trigger-openqa_in_openqa(triggering, http://jenkins.qa.suse.de/view/openQA-in-openQA/job/trigger-openQA_in_openQA-TW/configure)env obs_component=project openqa_groupid=24 monitor-openqa_job devel:openQA(monitoring, http://jenkins.qa.suse.de/view/openQA-in-openQA/job/monitor-openQA_in_openQA-TW/configure)bash -ex < <(curl -s https://raw.githubusercontent.com/os-autoinst/scripts/master/os-autoinst-obs-auto-submit)(submission, http://jenkins.qa.suse.de/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/configure)
Not sure about the VNC configuration and other values in this lengthy HTML form.
Updated by okurz over 3 years ago
mkittler wrote:
Looks like there's a lot broken. I also often ran into timeouts when clicking though the Jenkins web UI. Could we move this to a GitLab pipeline?
I suggested github actions in the past. IMHO the most challenging part is how to trigger github actions based on new packages in OBS. You are welcome to take this ticket here as the opportunity to move from jenkins to another solution but please prefer a public one.
Updated by livdywan over 3 years ago
- Priority changed from Normal to High
So I just realized a curious paradox. This Normal ticket is effectively blocking a High ticket #88754 which is formally exceeding our SLO. I think this means this one should also be High effectively.
Updated by livdywan over 3 years ago
Estimation poker practise:
- Our average estimate seems to be L
- Have a separate spike solution to trigger GitHub actions via OBS
- A screenshot with error messages would avoid having to look into the exact errors on Jenkins manually
- Why is Jenkins so slow? Separate ticket to investigate that
I suggested github actions in the past. IMHO the most challenging part is how to trigger github actions based on new packages in OBS. You are welcome to take this ticket here as the opportunity to move from jenkins to another solution but please prefer a public one.
An reference to how that is currently working would be helpful in the ticket description.
Updated by mkittler over 3 years ago
Looks like the Jenkins instance is now completely broken, I'm only getting error from the NGINX proxy.
I've just had a look: Restarting jenkins.service helped not really. I could briefly access the web UI but it ceased to work shortly again. Judging by the htop output it nevertheless performs some jobs in the background. /var/log/jenkins/jenkins.log shows quite some errors but I haven't been able to make sense of them.
Maybe it is really time to migrate away from Jenkins.
Updated by mkittler over 3 years ago
- Related to coordination #93609: [epic] openqa-bootstrap support on Leap 15.3 added
Updated by mkittler over 3 years ago
- Related to deleted (coordination #93609: [epic] openqa-bootstrap support on Leap 15.3)
Updated by mkittler over 3 years ago
- Blocks coordination #93609: [epic] openqa-bootstrap support on Leap 15.3 added
Updated by okurz over 3 years ago
- Status changed from Workable to In Progress
- Assignee set to okurz
on jenkins.qa.suse.de / is full. I will see what I can do
Updated by okurz over 3 years ago
- Copied to action #94318: http://jenkins.qa.suse.de not reachable added
Updated by okurz over 3 years ago
- Status changed from Blocked to Feedback
Trying to save the profile of "monitor-openQA_in_openQA-TW" manually over ssh:
<?xml version='1.1' encoding='UTF-8'?>
<project>
<actions/>
<description>Person to contact: okurz@suse.de</description>
<keepDependencies>false</keepDependencies>
<properties>
<hudson.plugins.redmine.RedmineProjectProperty plugin="redmine@0.21">
<redmineWebsiteName>progress</redmineWebsiteName>
<projectName>openqav3</projectName>
</hudson.plugins.redmine.RedmineProjectProperty>
<jenkins.model.BuildDiscarderProperty>
<strategy class="hudson.tasks.LogRotator">
<daysToKeep>600</daysToKeep>
<numToKeep>5000</numToKeep>
<artifactDaysToKeep>-1</artifactDaysToKeep>
<artifactNumToKeep>500</artifactNumToKeep>
</strategy>
</jenkins.model.BuildDiscarderProperty>
<com.coravy.hudson.plugins.github.GithubProjectProperty plugin="github@1.29.3">
<projectUrl>https://github.com/okurz/os-autoinst-distri-openQA/</projectUrl>
<displayName></displayName>
</com.coravy.hudson.plugins.github.GithubProjectProperty>
</properties>
<scm class="hudson.scm.NullSCM"/>
<triggers>
<jenkins.triggers.ReverseBuildTrigger>
<spec></spec>
<upstreamProjects>trigger-openQA_in_openQA-TW,</upstreamProjects>
<threshold>
<name>SUCCESS</name>
<ordinal>0</ordinal>
<color>BLUE</color>
<completeBuild>true</completeBuild>
</threshold>
</jenkins.triggers.ReverseBuildTrigger>
</triggers>
<builders>
<hudson.plugins.copyartifact.CopyArtifact plugin="copyartifact@1.41">
<project>trigger-openQA_in_openQA-TW</project>
<filter></filter>
<target></target>
<excludes></excludes>
<selector class="hudson.plugins.copyartifact.TriggeredBuildSelector">
<upstreamFilterStrategy>UseGlobalSetting</upstreamFilterStrategy>
<allowUpstreamDependencies>false</allowUpstreamDependencies>
</selector>
<doNotFingerprintArtifacts>false</doNotFingerprintArtifacts>
</hudson.plugins.copyartifact.CopyArtifact>
<hudson.tasks.Shell>
<command>env obs_component=project openqa_groupid=24 monitor-openqa_job devel:openQA</command>
</hudson.tasks.Shell>
</builders>
<publishers>
<hudson.tasks.ArtifactArchiver>
<artifacts>job_post_response</artifacts>
<allowEmptyArchive>false</allowEmptyArchive>
<onlyIfSuccessful>false</onlyIfSuccessful>
<fingerprint>false</fingerprint>
<defaultExcludes>true</defaultExcludes>
<caseSensitive>true</caseSensitive>
</hudson.tasks.ArtifactArchiver>
</publishers>
</project>
problem seems to be that jenkins changed the format for configuration and some plugins are outdated and hence the config is saved in invalid format, see https://www.jenkins.io/doc/upgrade-guide/2.277/#configuration-form-modernization . Trying to uninstall as many old and unused plugins as possible.
Updated by okurz over 3 years ago
- Status changed from Feedback to Resolved
The problem was caused by an upgrade of jenkins which causes some outdated plugins to make problems and preventing to have jobs. I have followed https://www.jenkins.io/doc/upgrade-guide/2.277/#configuration-form-modernization and uninstalled a big load of plugins and disabled some others. Due to that we have lost some functionality but I think no really relevant ones. We also lost the pipeline view of the "openQA-in-openQA" tests and I don't know which plugin that was but the important functionality including reporting and such should all still be there.