openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-05-28T08:22:13ZopenSUSE Project Management Tool
Redmine openQA Tests - action #93210 (New): [migration][qe-core] sssd openldap/389-ds basic testing, modi...https://progress.opensuse.org/issues/932102021-05-28T08:22:13Ztjyrinki_susetjyrinki+redmine@suse.de
<p>In ticket 89479 the goals for QE Core were fulfilled to modernize directory service testing. However, QE Migration would also like to test sssd with openldap (older SLE) and 389-ds (newer SLE) and testing how the functionality remains after upgrade to newer SLE.</p>
<p>There is a draft design for such tests at <a href="https://confluence.suse.com/display/qasleapac1/Draft+service+check+design+and+implementation" class="external">https://confluence.suse.com/display/qasleapac1/Draft+service+check+design+and+implementation</a> - once there it is fully agreed that we want to go ahead with such architecture, this test, which is probably one of the more important migration tests, should be modified towards that. Whether done by people from QE Core or QE Migration is likely depending on a resource issue, now that developer of 89479 is moving to yet another squad. Community contribution would be of course always welcome as well, although for that the design plan should be incorporated to this ticket. tl;dr; divide the test to 1. install_service, 2. configure_service, 3. enable_service, 4. start_service, 5. check_service, 6. check_function, of which 1-6 are done before distro upgrade and 5-6 after. Maybe it could be possibly to do just division to two steps, and execute either both or just the latter? Ticket will be updated once the draft solidifies.</p>
<a name="Further-Information"></a>
<h2 >Further Information<a href="#Further-Information" class="wiki-anchor">¶</a></h2>
<p>See discussion at <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/12528" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/12528</a> and <a href="https://progress.opensuse.org/issues/89479" class="external">https://progress.opensuse.org/issues/89479</a></p>
<a name="Acceptance-Criteria"></a>
<h2 >Acceptance Criteria<a href="#Acceptance-Criteria" class="wiki-anchor">¶</a></h2>
<p>AC1: Modify test so that it continues to execute as is, but is split so that distro upgrade (migration) tests can be also run using the same module.</p>
openQA Tests - coordination #91199 (Resolved): [security][epic][qem] Add existing security Produc...https://progress.opensuse.org/issues/911992021-04-15T06:26:06Ztjyrinki_susetjyrinki+redmine@suse.de
<p>After comparing test coverage of SLE 15 SP2 and SP3 in opneQA I identified several tests that are exclusive to either pre-release or post-release testing and could be easily used to increase our coverage just by using what we already have.</p>
<p>It is possible that are some errors and some of the tests from the list are already being used. It is also just my opinion, therefor review from other colleagues is more then welcome.</p>
<p>These are the security tests that run only before release and could be used also after the release (product QE -> QEM)</p>
<p>tests/security/grub_auth/grub_authorization.pm<br>
tests/security/ima/evmctl_ima_sign.pm<br>
tests/security/ima/evm_protection_digital_signatures.pm<br>
tests/security/ima/evm_protection_hmacs.pm<br>
tests/security/ima/evm_setup.pm<br>
tests/security/ima/evm_verify.pm<br>
tests/security/ima/ima_appraisal_audit.pm<br>
tests/security/ima/ima_appraisal_digital_signatures.pm<br>
tests/security/ima/ima_appraisal_hashes.pm<br>
tests/security/ima/ima_kernel_cmdline_hash.pm<br>
tests/security/ima/ima_kernel_cmdline_template.pm<br>
tests/security/ima/ima_measurement_audit.pm<br>
tests/security/ima/ima_measurement.pm<br>
tests/security/ima/ima_setup.pm<br>
tests/security/ima/ima_verify.pm<br>
tests/security/krb5/krb5_crypt_nfs_client.pm<br>
tests/security/krb5/krb5_crypt_nfs_server.pm<br>
tests/security/krb5/krb5_crypt_prepare.pm<br>
tests/security/krb5/krb5_crypt_setup_client.pm<br>
tests/security/krb5/krb5_crypt_setup_kdc.pm<br>
tests/security/krb5/krb5_crypt_setup_server.pm<br>
tests/security/krb5/krb5_crypt_ssh_client.pm<br>
tests/security/krb5/krb5_crypt_ssh_server.pm<br>
tests/security/lynis/lynis_analyze_system_audit.pm<br>
tests/security/lynis/lynis_harden_index.pm<br>
tests/security/lynis/lynis_perform_system_audit.pm<br>
tests/security/lynis/lynis_setup.pm<br>
tests/security/mokutil_sign.pm<br>
tests/security/nproc_limits.pm<br>
tests/security/pam/pam_basic_function.pm<br>
tests/security/pam/pam_config.pm<br>
tests/security/pam/pam_login.pm<br>
tests/security/pam/pam_mount.pm<br>
tests/security/pam/pam_su.pm<br>
tests/security/selinux/audit2allow.pm<br>
tests/security/selinux/chcat.pm<br>
tests/security/selinux/chcon.pm<br>
tests/security/selinux/enforcing_mode_setup.pm<br>
tests/security/selinux/fixfiles.pm<br>
tests/security/selinux/print_se_context.pm<br>
tests/security/selinux/restorecon.pm<br>
tests/security/selinux/selinux_setup.pm<br>
tests/security/selinux/selinux_smoke.pm<br>
tests/security/selinux/semanage_boolean.pm<br>
tests/security/selinux/semanage_fcontext.pm<br>
tests/security/selinux/semodule.pm<br>
tests/security/selinux/sestatus.pm<br>
tests/security/selinux/setsebool.pm<br>
tests/security/swtpm/swtpm_env_setup.pm<br>
tests/security/swtpm/swtpm_verify.pm<br>
tests/security/tpm2/tpm2_engine/tpm2_engine_ecdsa_operation.pm<br>
tests/security/tpm2/tpm2_engine/tpm2_engine_info.pm<br>
tests/security/tpm2/tpm2_engine/tpm2_engine_random_data.pm<br>
tests/security/tpm2/tpm2_engine/tpm2_engine_rsa_operation.pm<br>
tests/security/tpm2/tpm2_engine/tpm2_engine_self_sign.pm<br>
tests/security/tpm2/tpm2_env_setup.pm<br>
tests/security/tpm2/tpm2_tools/tpm2_tools_auth.pm<br>
tests/security/tpm2/tpm2_tools/tpm2_tools_encrypt.pm<br>
tests/security/tpm2/tpm2_tools/tpm2_tools_self_contain_tool.pm<br>
tests/security/tpm2/tpm2_tools/tpm2_tools_sign_verify.pm</p>
qe-yam - action #91196 (Rejected): Add existing qe-yast Product QE tests to QEMhttps://progress.opensuse.org/issues/911962021-04-15T06:25:12Ztjyrinki_susetjyrinki+redmine@suse.de
<p>After comparing test coverage of SLE 15 SP2 and SP3 in opneQA I identified several tests that are exclusive to either pre-release or post-release testing and could be easily used to increase our coverage just by using what we already have.</p>
<p>It is possible that are some errors and some of the tests from the list are already being used. It is also just my opinion, therefor review from other colleagues is more then welcome.</p>
<p>These are the YaST tests that run only before release and could be used also after the release (product QE -> QEM)</p>
<p>tests/console/yast2_cmdline.pm<br>
tests/console/yast2_dns_server.pm<br>
tests/console/yast2_kdump.pm<br>
tests/console/yast2_lan_hostname.pm<br>
tests/console/yast2_ntpclient.pm<br>
tests/console/yast2_proxy.pm<br>
tests/console/yast2_rmt.pm<br>
tests/console/yast2_samba.pm<br>
tests/console/yast2_settings.pm<br>
tests/console/yast2_snapper_ncurses.pm<br>
tests/console/yast2_vnc.pm<br>
tests/network/wireguard.pm<br>
tests/security/yast2_apparmor/manually_add_profile.pm<br>
tests/security/yast2_apparmor/scan_audit_logs.pm<br>
tests/security/yast2_apparmor/settings_disable_enable_apparmor.pm<br>
tests/security/yast2_apparmor/settings_toggle_profile_mode.pm<br>
tests/security/yast2_users/add_users.pm</p>
openQA Tests - coordination #91193 (New): [epic][qe-core][qem] Add existing console Product QE te...https://progress.opensuse.org/issues/911932021-04-15T06:23:53Ztjyrinki_susetjyrinki+redmine@suse.de
<p>After comparing test coverage of SLE 15 SP2 and SP3 in opneQA I identified several tests that are exclusive to either pre-release or post-release testing and could be easily used to increase our coverage just by using what we already have.</p>
<p>It is possible that are some errors and some of the tests from the list are already being used. It is also just my opinion, therefor review from other colleagues is more then welcome.</p>
<p>These are the console tests that run only before release and could be used also after the release (product QE -> QEM)</p>
<p><del>tests/console/btrfsmaintenance.pm (runs on mau-filesystem)</del><br>
<del>tests/console/journald_fss.pm</del><br>
<del>tests/console/lvm_thin_check.pm</del><br>
tests/console/ndctl.pm<br>
<del>tests/console/network_hostname.pm</del><br>
tests/console/nvme_checks.pm<br>
<del>tests/console/openvswitch_ssl.pm</del><br>
tests/console/snapper_cleanup_timeline.pm<br>
tests/console/systemd_nspawn.pm<br>
tests/console/verify_default_target.pm<br>
tests/console/verify_network.pm<br>
<del>tests/console/vsftpd.pm</del></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong> All aforementioned tests are scheduled in maintenance tests (aggregated or incidents), in their corresponding categories (When in doubt ask in qe-core channel)</li>
<li><strong>AC2</strong> Cross reference with <a href="https://github.com/ge0r/openQA-module-mapper" class="external">openqa-module-mapper</a> to figure what's already done and what's running where.</li>
<li><strong>AC3</strong> If a test module needs black magic to work (i.e, takes more than half a day), a corresponding ticket is created and it is removed from this list.</li>
</ul>
openQA Tests - action #91178 (Rejected): [kernel-default][qem] Add existing kernel Product QE tes...https://progress.opensuse.org/issues/911782021-04-15T06:14:00Ztjyrinki_susetjyrinki+redmine@suse.de
<p>After comparing test coverage of SLE 15 SP2 and SP3 in opneQA I identified several tests that are exclusive to either pre-release or post-release testing and could be easily used to increase our coverage just by using what we already have.</p>
<p>It is possible that are some errors and some of the tests from the list are already being used. It is also just my opinion, therefor review from other colleagues is more then welcome.</p>
<p>These are the kernel tests that run only before release and could be used also after the release (product QE -> QEM)</p>
<p>tests/kernel/blktests.pm<br>
tests/kernel/kdump.pm<br>
tests/kernel/numa_irqbalance.pm<br>
tests/kernel/pressure_stall_information.pm<br>
tests/kernel/tuned.pm</p>
qe-yam - action #91175 (Rejected): [qe-yast][qem] Add existing Product QE tests to QEMhttps://progress.opensuse.org/issues/911752021-04-15T06:11:54Ztjyrinki_susetjyrinki+redmine@suse.de
<p>After comparing test coverage of SLE 15 SP2 and SP3 in opneQA I identified several tests that are exclusive to either pre-release or post-release testing and could be easily used to increase our coverage just by using what we already have.</p>
<p>It is possible that are some errors and some of the tests from the list are already being used. It is also just my opinion, therefor review from other colleagues is more then welcome.</p>
<p>These are the YaST tests that run only before release and could be used also after the release (product QE -> QEM)</p>
<p>tests/console/vsftpd.pm<br>
tests/console/yast2_cmdline.pm<br>
tests/console/yast2_dns_server.pm<br>
tests/console/yast2_kdump.pm<br>
tests/console/yast2_lan_hostname.pm<br>
tests/console/yast2_ntpclient.pm<br>
tests/console/yast2_proxy.pm<br>
tests/console/yast2_rmt.pm<br>
tests/console/yast2_samba.pm<br>
tests/console/yast2_settings.pm<br>
tests/console/yast2_snapper_ncurses.pm<br>
tests/console/yast2_vnc.pm<br>
tests/security/yast2_apparmor/manually_add_profile.pm<br>
tests/security/yast2_apparmor/scan_audit_logs.pm<br>
tests/security/yast2_apparmor/settings_disable_enable_apparmor.pm<br>
tests/security/yast2_apparmor/settings_toggle_profile_mode.pm<br>
tests/security/yast2_users/add_users.pm</p>
qe-yam - action #91172 (Rejected): Add existing QEM YaST tests to Product QEhttps://progress.opensuse.org/issues/911722021-04-15T06:10:09Ztjyrinki_susetjyrinki+redmine@suse.de
<p>After comparing test coverage of SLE 15 SP2 and SP3 in opneQA I identified several tests that are exclusive to either pre-release or post-release testing and could be easily used to increase our coverage just by using what we already have.</p>
<p>It is possible that are some errors and some of the tests from the list are already being used. It is also just my opinion, therefor review from other colleagues is more then welcome.</p>
<p>These are the YaST tests that run only after release and could be used also before the release (QEM -> product QE)</p>
<p>tests/console/yast2_registration.pm<br>
tests/yast2_gui/yast2_instserver.pm<br>
tests/yast2_gui/yast2_keyboard.pm<br>
tests/yast2_gui/yast2_storage_ng.pm</p>
openQA Tests - action #88687 (New): [qe-tools] Use the TEST_SUITE_SUFFICIENT value to mark packag...https://progress.opensuse.org/issues/886872021-02-17T10:11:04Ztjyrinki_susetjyrinki+redmine@suse.de
<p>Reviewing and actually using the collected TEST_SUITE_SUFFICIENT data together with making template generator aware of it to have templates automatically acknowledge when build time regression test suite is enough.</p>
<p>The current machine readable data for automation is at <a href="https://pes.suse.de/QA_Maintenance/Automation_Progress_Tracking_in_QAM/" class="external">https://pes.suse.de/QA_Maintenance/Automation_Progress_Tracking_in_QAM/</a> and so that should be appended - probably with new sections for indirect and build time ones - as needed with packages that would be now marked as automatically tested.</p>
openQA Tests - coordination #88684 (Feedback): [qe-core][epic][qem] Identify packages that we are...https://progress.opensuse.org/issues/886842021-02-17T10:08:30Ztjyrinki_susetjyrinki+redmine@suse.de
<p>We test some packages indirectly as part of other packages' test. The template generator (<a href="https://gitlab.suse.de/qa-maintenance/metadata/-/tree/master/template_generator" class="external">https://gitlab.suse.de/qa-maintenance/metadata/-/tree/master/template_generator</a>) Update Squad uses does not detect these cases, and does not then tell the update tester there's automatic openQA regression testing.</p>
<p>For example, webkit is used in evolution - if there have been maintenance updates of webkit, was the manual testing more complex than what the evolution automatic test case would be, or not? If the evolution test tests webkit to an enough extent, webkit could be marked as automatically regression tested, and manual testing would no longer be required.</p>
<a name="Acceptance-Criteria"></a>
<h2 >Acceptance Criteria<a href="#Acceptance-Criteria" class="wiki-anchor">¶</a></h2>
<p>AC1: Identify what packages we are regression testing indirectly, and via which tests, check if it's "sufficient" for regression testing of that package: sufficient means at least similar level as in manual testing done in case of updates, see <a href="http://qam.suse.de/" class="external">http://qam.suse.de/</a> database for how the packages was teded manually.<br>
AC2: Mark sufficiently indirectly regression tested packages as automatically tested. This can be done, in the current way, with an entry at PES wiki page (<a href="https://pes.suse.de/QA_Maintenance/Automation_Progress_Tracking_in_QAM/" class="external">https://pes.suse.de/QA_Maintenance/Automation_Progress_Tracking_in_QAM/</a>) - the template generator parses that page to know if there has been automatic regression testing. Add a separate header for indirectly tested packages.</p>
openQA Tests - action #88231 (Feedback): [qe-core][qem][kiwi][regression] Extend kiwi testing to ...https://progress.opensuse.org/issues/882312021-01-26T12:35:53Ztjyrinki_susetjyrinki+redmine@suse.de
<p>There was an lvm update that broke kiwi build:</p>
<p><a href="https://github.com/OSInside/kiwi/issues/1665" class="external">https://github.com/OSInside/kiwi/issues/1665</a></p>
<p>Our basic kiwi testing currently tests simply that kiwi is able to generate an image on x86_64, not enough to catch this kind of problem.</p>
<p>The ticket is not workable at the moment, to be needed to be added to this ticket:</p>
<ul>
<li>link to LVM update with the change that broke kiwi building</li>
<li>description of how to use kiwi in the way that would reveal the brokenness</li>
</ul>
<p>For more information contact mwilck, hmzhao. hmzhao also mentioned we would need specifically someone from kiwi team.</p>
openQA Tests - action #87704 (Resolved): [qe-core][qem][QU] test fails in boot_encrypt on s390x (...https://progress.opensuse.org/issues/877042021-01-13T15:52:23Ztjyrinki_susetjyrinki+redmine@suse.de
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-15-SP2-Online-QR-s390x-lvm-encrypt-separate-boot@s390x-kvm-sle12 fails in<br>
<a href="https://openqa.suse.de/tests/5259382/modules/boot_encrypt/steps/10" class="external">boot_encrypt</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Same as lvm-full-encrypt, but with separate boot not encrypted partition, only installation to not repeat everything again with small risk.<br>
Maintainer: riafarov</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/5256009" class="external">377.1</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/5239855" class="external">376.16</a> (or more recent)</p>
<p>--<br>
Note: also this passing is already using the separated QAM YAML schedule schedule/qam/QR/lvm_encrypt_separate_boot.yaml<br>
--</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Online-QR&machine=s390x-kvm-sle12&test=lvm-encrypt-separate-boot&version=15-SP2" class="external">latest</a></p>
openSUSE admin - tickets #80354 (Rejected): ns3.opensuse.org reports wrong IP to meet.opensuse.orghttps://progress.opensuse.org/issues/803542020-11-25T09:58:45Ztjyrinki_susetjyrinki+redmine@suse.de
<p>The new meet.o.o server was put into place, and it's at 195.135.221.174. However, ns3.opensuse.org disagrees and reports 173, which breaks meet.o.o for everyone (unless using direct IP).</p>
<p>Meanwhile, ns2 and ns4 are also down, possibly something to look at as well and verify if they'll eventually agree on the IP of meet.o.o.</p>
<p>-Timo </p>
openQA Tests - coordination #64126 (New): [qe-core][epic] Identify packages that have automated i...https://progress.opensuse.org/issues/641262020-03-03T13:23:14Ztjyrinki_susetjyrinki+redmine@suse.de
<p>There's both 1) potentially indirect testing of packages that could be marked as tested automatically, and 2) database of TEST_SUITE_SUFFICIENT information from Updates Squad that could be used likewise.</p>
<p>Let's use this ticket to define guidelines for us.</p>
<a name="How"></a>
<h1 >How<a href="#How" class="wiki-anchor">¶</a></h1>
<ul>
<li>How to own our testing better (Where our needs for tooling end, and where the tooling should be provided by external teams)</li>
<li>How to decide whether it should be a package test, an openQA test, or a separate test ran within openQA</li>
<li>How to match this with our test plan</li>
</ul>
openQA Tests - action #57584 (Resolved): [qam] test fails in evolution_setup_servers - dovecot n...https://progress.opensuse.org/issues/575842019-10-01T11:52:32Ztjyrinki_susetjyrinki+redmine@suse.de
<p>After <a href="https://progress.opensuse.org/issues/56273" class="external">https://progress.opensuse.org/issues/56273</a> got fixed, the evolution_setup_servers is now failing, a rerun done first at <a href="https://openqa.suse.de/tests/3397700" class="external">https://openqa.suse.de/tests/3397700</a> and a newer one at <a href="https://openqa.suse.de/tests/3424806" class="external">https://openqa.suse.de/tests/3424806</a></p>
<p>"No provider of 'dovecot' found." at <a href="https://openqa.suse.de/tests/3397700#step/evolution_prepare_servers/18" class="external">https://openqa.suse.de/tests/3397700#step/evolution_prepare_servers/18</a></p>
<p>qam-regression-message is currently disabled, <a href="https://progress.opensuse.org/issues/57260" class="external">https://progress.opensuse.org/issues/57260</a> is about re-enabling it once the issues are all fixed.</p>
openQA Project - action #56789 (New): New needles from git repository not working with openqa-clo...https://progress.opensuse.org/issues/567892019-09-11T08:14:09Ztjyrinki_susetjyrinki+redmine@suse.de
<p>Use case:</p>
<p>To be able to test tests in production (openqa.opensuse.org or openqa.suse.de) before merging the needles and tests themselves, since results have proven to vary on those loaded machines compared to local openQA instance or VM.</p>
<p>It is currently working if there are no new needles involved, but this issue describes the problem with custom needles which would be theoretically supported by openqa-clone-custom-git-refspec but not working in practice.</p>
<p>Problem description:</p>
<p>First of all, documentation [1] says "Path to needles subdirectory to use, defaults to "needles" within PRODUCTDIR. Can be a git repository URL, comparable to CASEDIR", and CASEDIR states "for example <a href="mailto:git@github.com">git@github.com</a>:os-autoinst/os-autoinst-distri-opensuse.git#feature/test".</p>
<p>[1] <a href="https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc" class="external">https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc</a></p>
<p>However,</p>
<pre><code>openqa-clone-custom-git-refspec https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8332 https://openqa.opensuse.org/tests/xxxxxxxx --apikey=XXX --apisecret=XXX NEEDLES_DIR=git@github.com:tjyrinki/os-autoinst-needles-opensuse.git#poo-12345
</code></pre>
<p>results in:</p>
<pre><code>Could not create directory '/var/lib/empty/.ssh'.
Host key verification failed.
fatal: Could not read from remote repository.
</code></pre>
<p>Using https instead (NEEDLES_DIR=<a href="https://github.com/tjyrinki/os-autoinst-needles-opensuse.git" class="external">https://github.com/tjyrinki/os-autoinst-needles-opensuse.git</a>) gets one further, but openQA complains about wrong needles location:</p>
<pre><code>init needles from /var/lib/openqa/pool/15/os-autoinst-needles-opensuse
Needle /var/lib/openqa/pool/15/os-autoinst-needles-opensuse/ssh-login-ok-20160813.json is not under project directory /var/lib/openqa/cache/xxx at /usr/lib/os-autoinst/needle.pm line 63.
</code></pre>
<p>Oliver Kurz suggested playing around with the PRODUCTDIR to try to workaround the forbidding use of (correctly checked out) needles dir which I did but it didn't get any better with trying to set that to eg pool/... directories, which also changes on every run.</p>