Project

General

Profile

action #91752

jenkins: Multiple missing fields and errors in configuration of openQA-in-openQA

Added by ilausuch 6 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2021-04-26
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

Saving a new configuration is necessary to accomplish #88754. But it's impossible because multiple missing fields and errors in the UI configuration.
It is not possible to save new changes.

Acceptance Criteria

  • AC 1: The configuration can be changed and saved

Suggestions


Related issues

Blocks openQA Project - action #88754: openQA-in-openQA tests always fail and results do not impact submission pipelineResolved2021-02-182021-07-08

Blocks openQA Project - coordination #93609: [epic] openqa-bootstrap support on Leap 15.3Blocked2021-06-07

Copied to openQA Infrastructure - action #94318: http://jenkins.qa.suse.de not reachableResolved2021-04-26

History

#1 Updated by cdywan 6 months ago

I made an effort going by my "5 minutes rule" finding out if I could fix this easily. I tried to remove unused sections and clear invalid options that don't seem to be enabled.

But I get what looks like validation errors even on e.g. VNC server even though it's not enabled and clearing the field doesn't work either.

#2 Updated by ilausuch 6 months ago

  • Description updated (diff)

#3 Updated by ilausuch 6 months ago

  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by cdywan 6 months ago

  • Blocks action #88754: openQA-in-openQA tests always fail and results do not impact submission pipeline added

#5 Updated by okurz 6 months ago

  • Subject changed from jenkings: Multiple missing fields and errors in configuration to jenkins: Multiple missing fields and errors in configuration of openQA-in-openQA
  • Target version set to Ready

#6 Updated by mkittler 6 months ago

Looks like there's a lot broken. I also often ran into timeouts when clicking though the Jenkins web UI. Could we move this to a GitLab pipeline?

Of course we need to figure out what this Jeninks jobs actually do to move it to GitLab. It looks like Jenkin executes three commands:

  1. trigger-openqa_in_openqa (triggering, http://jenkins.qa.suse.de/view/openQA-in-openQA/job/trigger-openQA_in_openQA-TW/configure)
  2. env obs_component=project openqa_groupid=24 monitor-openqa_job devel:openQA (monitoring, http://jenkins.qa.suse.de/view/openQA-in-openQA/job/monitor-openQA_in_openQA-TW/configure)
  3. bash -ex < <(curl -s https://raw.githubusercontent.com/os-autoinst/scripts/master/os-autoinst-obs-auto-submit) (submission, http://jenkins.qa.suse.de/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/configure)

Not sure about the VNC configuration and other values in this lengthy HTML form.

#7 Updated by okurz 6 months ago

mkittler wrote:

Looks like there's a lot broken. I also often ran into timeouts when clicking though the Jenkins web UI. Could we move this to a GitLab pipeline?

I suggested github actions in the past. IMHO the most challenging part is how to trigger github actions based on new packages in OBS. You are welcome to take this ticket here as the opportunity to move from jenkins to another solution but please prefer a public one.

#8 Updated by cdywan 4 months ago

  • Priority changed from Normal to High

So I just realized a curious paradox. This Normal ticket is effectively blocking a High ticket #88754 which is formally exceeding our SLO. I think this means this one should also be High effectively.

#9 Updated by cdywan 4 months ago

Estimation poker practise:

  • Our average estimate seems to be L
  • Have a separate spike solution to trigger GitHub actions via OBS
  • A screenshot with error messages would avoid having to look into the exact errors on Jenkins manually
  • Why is Jenkins so slow? Separate ticket to investigate that

I suggested github actions in the past. IMHO the most challenging part is how to trigger github actions based on new packages in OBS. You are welcome to take this ticket here as the opportunity to move from jenkins to another solution but please prefer a public one.

An reference to how that is currently working would be helpful in the ticket description.

#10 Updated by mkittler 4 months ago

Looks like the Jenkins instance is now completely broken, I'm only getting error from the NGINX proxy.

I've just had a look: Restarting jenkins.service helped not really. I could briefly access the web UI but it ceased to work shortly again. Judging by the htop output it nevertheless performs some jobs in the background. /var/log/jenkins/jenkins.log shows quite some errors but I haven't been able to make sense of them.

Maybe it is really time to migrate away from Jenkins.

#11 Updated by mkittler 4 months ago

#12 Updated by mkittler 4 months ago

#13 Updated by mkittler 4 months ago

#14 Updated by okurz 4 months ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz

on jenkins.qa.suse.de / is full. I will see what I can do

#15 Updated by okurz 4 months ago

  • Copied to action #94318: http://jenkins.qa.suse.de not reachable added

#16 Updated by okurz 4 months ago

  • Status changed from In Progress to Blocked

-> #94318

#17 Updated by okurz 4 months ago

  • Status changed from Blocked to Feedback

Trying to save the profile of "monitor-openQA_in_openQA-TW" manually over ssh:

<?xml version='1.1' encoding='UTF-8'?>
<project>
  <actions/>
  <description>Person to contact: okurz@suse.de</description>
  <keepDependencies>false</keepDependencies>
  <properties>
    <hudson.plugins.redmine.RedmineProjectProperty plugin="redmine@0.21">
      <redmineWebsiteName>progress</redmineWebsiteName>
      <projectName>openqav3</projectName>
    </hudson.plugins.redmine.RedmineProjectProperty>
    <jenkins.model.BuildDiscarderProperty>
      <strategy class="hudson.tasks.LogRotator">
        <daysToKeep>600</daysToKeep>
        <numToKeep>5000</numToKeep>
        <artifactDaysToKeep>-1</artifactDaysToKeep>
        <artifactNumToKeep>500</artifactNumToKeep>
      </strategy>
    </jenkins.model.BuildDiscarderProperty>
    <com.coravy.hudson.plugins.github.GithubProjectProperty plugin="github@1.29.3">
      <projectUrl>https://github.com/okurz/os-autoinst-distri-openQA/</projectUrl>
      <displayName></displayName>
    </com.coravy.hudson.plugins.github.GithubProjectProperty>
  </properties>
  <scm class="hudson.scm.NullSCM"/>
  <triggers>
    <jenkins.triggers.ReverseBuildTrigger>
      <spec></spec>
      <upstreamProjects>trigger-openQA_in_openQA-TW,</upstreamProjects>
      <threshold>
        <name>SUCCESS</name>
        <ordinal>0</ordinal>
        <color>BLUE</color>
        <completeBuild>true</completeBuild>
      </threshold>
    </jenkins.triggers.ReverseBuildTrigger>
  </triggers>
  <builders>
    <hudson.plugins.copyartifact.CopyArtifact plugin="copyartifact@1.41">
      <project>trigger-openQA_in_openQA-TW</project>
      <filter></filter>
      <target></target>
      <excludes></excludes>
      <selector class="hudson.plugins.copyartifact.TriggeredBuildSelector">
        <upstreamFilterStrategy>UseGlobalSetting</upstreamFilterStrategy>
        <allowUpstreamDependencies>false</allowUpstreamDependencies>
      </selector>
      <doNotFingerprintArtifacts>false</doNotFingerprintArtifacts>
    </hudson.plugins.copyartifact.CopyArtifact>
    <hudson.tasks.Shell>
      <command>env obs_component=project openqa_groupid=24 monitor-openqa_job devel:openQA</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers>
    <hudson.tasks.ArtifactArchiver>
      <artifacts>job_post_response</artifacts>
      <allowEmptyArchive>false</allowEmptyArchive>
      <onlyIfSuccessful>false</onlyIfSuccessful>
      <fingerprint>false</fingerprint>
      <defaultExcludes>true</defaultExcludes>
      <caseSensitive>true</caseSensitive>
    </hudson.tasks.ArtifactArchiver>
  </publishers>
</project>

problem seems to be that jenkins changed the format for configuration and some plugins are outdated and hence the config is saved in invalid format, see https://www.jenkins.io/doc/upgrade-guide/2.277/#configuration-form-modernization . Trying to uninstall as many old and unused plugins as possible.

#18 Updated by okurz 4 months ago

  • Status changed from Feedback to Resolved

The problem was caused by an upgrade of jenkins which causes some outdated plugins to make problems and preventing to have jobs. I have followed https://www.jenkins.io/doc/upgrade-guide/2.277/#configuration-form-modernization and uninstalled a big load of plugins and disabled some others. Due to that we have lost some functionality but I think no really relevant ones. We also lost the pipeline view of the "openQA-in-openQA" tests and I don't know which plugin that was but the important functionality including reporting and such should all still be there.

Also available in: Atom PDF