openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842016-12-29T09:16:53ZopenSUSE Project Management Tool
Redmine openQA Tests - action #15678 (Resolved): [LTP][OpenQA] misc: acpi_test_dev_callback failshttps://progress.opensuse.org/issues/156782016-12-29T09:16:53Zrpalethorperichard.palethorpe@suse.com
<p>The ltp_acpi tests fails when running a test inside the ltp_acpi_cmds kernel module called acpi_test_dev_callback.</p>
<p><a href="https://openqa.suse.de/tests/686455#step/run_ltp/45" class="external">https://openqa.suse.de/tests/686455#step/run_ltp/45</a></p>
openQA Tests - action #15668 (Resolved): [kernel][LTP][OpenQA] hyperthreading ht_interrupt won't runhttps://progress.opensuse.org/issues/156682016-12-28T15:53:28Zrpalethorperichard.palethorpe@suse.com
<p>Test fails with TCONF claiming system does not have HT enabled. However the CPU has the HT flag and shows 8 processors configured on a 4 core CPU.</p>
<p>The file <code>/testcases/kernel/sched/hyperthreading/ht_interrupt/ht_utils.c</code> (AFAICT) checks the <code>/proc/cpuinfo</code> file for the number of logical processors vs the number sockets/cores/threads and also looks for a line starting with <code>cpu_package</code>. If it finds <code>cpu_package</code> then it assumes that this is a Hyperthreading kernel. No such line is present on my system.</p>
<p>The other tests in the HT runfile, smt_smp_enable and smt_smp_affinity, run and pass, they have their own copies of <code>ht_utils.c</code> which are different.</p>
openQA Tests - action #15652 (Closed): [LTP][OpenQA] commands: mkfs.ntfs missinghttps://progress.opensuse.org/issues/156522016-12-27T11:26:45Zrpalethorperichard.palethorpe@suse.com
<p>The ntfsprogs package appears to be missing in SLES 12 onwards. It exists in OpenSUSE, so we could at least run the tests there, but then we start on the path of installing different sets of packages in SLES and OpenSUSE. Furthermore I am not sure that we care about NTFS support.</p>
openQA Tests - action #15626 (Resolved): [LTP][OpenQA] commands: du -a test failshttps://progress.opensuse.org/issues/156262016-12-22T12:46:21Zrpalethorperichard.palethorpe@suse.com
<p><a href="https://openqa.suse.de/tests/686450#step/run_ltp/111" class="external">https://openqa.suse.de/tests/686450#step/run_ltp/111</a></p>
openQA Tests - action #15624 (Resolved): [LTP][OpenQA] sssd: sss_* commands not foundhttps://progress.opensuse.org/issues/156242016-12-22T12:42:05Zrpalethorperichard.palethorpe@suse.com
<p>It seems that SUSE does not have commands (they are not in sssd-tools) such as sss_useradd. We probably just use regular <code>useradd</code> with the correct PAM modules configured.</p>
openQA Tests - action #15622 (Resolved): [LTP][OpenQA] commands: mail test claims mail fail is no...https://progress.opensuse.org/issues/156222016-12-22T12:36:35Zrpalethorperichard.palethorpe@suse.com
<p><a href="https://openqa.suse.de/tests/686450#step/run_ltp/56" class="external">https://openqa.suse.de/tests/686450#step/run_ltp/56</a></p>
<p><a href="https://openqa.suse.de/tests/latest?flavor=Server-DVD&distri=sle&machine=64bit&test=ltp_commands&version=12-SP3&arch=x86_64" class="external">latest</a></p>
openQA Tests - action #15620 (Rejected): [LTP][OpenQA] Crontab activity not foundhttps://progress.opensuse.org/issues/156202016-12-22T12:17:18Zrpalethorperichard.palethorpe@suse.com
<p>cron_tests.sh checks /var/log/messages and /var/log/cron for crontab activity, but neither file exists on newer systems. The test should probably try using <code>journalctl</code> as well.</p>
openQA Tests - action #15592 (Resolved): [LTP][OpenQA][kernel] LTP Native test runner minor issueshttps://progress.opensuse.org/issues/155922016-12-20T16:06:45Zrpalethorperichard.palethorpe@suse.com
<p>This is an unsorted and unverified list of issues I have identified from running most of the LTP tests on openqa.suse.de</p>
<ul>
<li>[ ] aiodio no junkfile</li>
<li>[ ] cap bounds; fail to exec check_pe</li>
<li>[ ] commands; no crontab activity</li>
<li>[ ] commands; no fail mail</li>
<li>[ ] commands: sssd-lib.sh missing</li>
<li>[ ] commands: du -a failed</li>
<li>[ ] commands; ntfs not found</li>
<li>[ ] controllers; cgroup fail</li>
<li>[ ] controllers; mem+swap not enabled</li>
<li>[ ] controllers; memcg fail</li>
<li>[ ] controllers; subsystem debug not supported</li>
<li>[ ] controllers; 'controllers' test broken</li>
<li>[ ] cpuhotplug support</li>
<li>[ ] random timeouts where serial terminal I/O appears to cease with no error</li>
<li>[ ] ext4, readonly; need big block device</li>
<li>[ ] hyperthreading; supported?</li>
<li>[ ] ima; broken?</li>
<li>[ ] misc; need kernel modules</li>
<li>[ ] misc; acpi broken?</li>
<li>[ ] lvm; unable to make ~/test/growfiles/msdos~ directory</li>
<li>[ ] mm; need more swap space</li>
<li>[ ] numa; broken test?</li>
<li>[ ] rpc; broken tests?</li>
<li>[ ] smack; needs ~/smack~</li>
<li>[ ] syscalls; movepages needs NUMA</li>
<li>[ ] syscalls_ipc; duplicates syscalls?</li>
<li>[ ] msgctl10; fork failed (too ambitious?)</li>
<li>[ ] tirpc; tests broken?</li>
<li>[ ] tpm_tools; commands are missing</li>
<li>[ ] tpm_tools; virtual or physical TPM required</li>
<li>[ ] ensure all non-duplicate, non-networking runtest files are being used</li>
<li>[ ] transfer test configurations to openSUSE</li>
</ul>
<p>Some issues are probably upstream problems and others may be large or hard enough to deserve their own issue entry and proper scrutiny. I will try to fix the easiest ones first and get a better idea of what is required.</p>
openQA Tests - action #15492 (Resolved): Upgrade ppc64le workers to QEMU 2.6.*, i.e. current Leap...https://progress.opensuse.org/issues/154922016-12-14T11:06:38Zrpalethorperichard.palethorpe@suse.com
<a name="observation"></a>
<h2 >observation<a href="#observation" class="wiki-anchor">¶</a></h2>
<p>This job (<a href="https://openqa.suse.de/tests/668033/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/668033/file/autoinst-log.txt</a>) fails because the logfile parameter is not available in the installed version of QEMU on the worker.</p>
<a name="problem"></a>
<h2 >problem<a href="#problem" class="wiki-anchor">¶</a></h2>
<p>Upgrading the QEMU version on the worker will fix this. But for this we would need to update e.g. malbec.arch from SLES 12 SP1 to a more recent version which no one did, maybe for good reasons.</p>
<a name="workaround"></a>
<h2 >workaround<a href="#workaround" class="wiki-anchor">¶</a></h2>
<p>The virtio-console is optional in os-autoinst and is only enabled if the job states 'VIRTIO_CONSOLE=1'. As a workaround disable this setting.</p>
openQA Project - action #14690 (Resolved): Live stream for serial terminalhttps://progress.opensuse.org/issues/146902016-11-08T14:32:21Zrpalethorperichard.palethorpe@suse.com
<p>Replace the live SUT video feed in the OpenQA UI with a scrolling text display when a serial terminal is set as the active console.</p>
<p>Currently when the user selects a serial console a stale screen shot of the last used VNC console is shown. The live log below still updates, but the user experience is significantly degraded.</p>
openQA Project - coordination #14626 (New): [epic] backend and console capabilities interface to ...https://progress.opensuse.org/issues/146262016-11-03T13:28:48Zrpalethorperichard.palethorpe@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Prevent "if/else" in tests needing to distinguish different backends</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> No obvious "if/else" for different types of consoles in os-autoinst-distri-opensuse are necessary anymore</li>
<li><strong>AC2:</strong> Same as <em>AC1</em> for different <em>backends</em></li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Read what had been done in <a href="https://github.com/os-autoinst/os-autoinst/pull/1232">https://github.com/os-autoinst/os-autoinst/pull/1232</a> and <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8718">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8718</a> to define "persistent" consoles</li>
<li>Incorporate content from <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/Utils/Backends.pm">https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/Utils/Backends.pm</a> into os-autoinst as flags on backends rather than if/else in test code</li>
<li>Look for other "if/else" code in test distributions, e.g. os-autoinst-distri-opensuse", distinguishing different backends and consoles to provide as capabilities on backends/consoles</li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<a name="Background"></a>
<h3 >Background<a href="#Background" class="wiki-anchor">¶</a></h3>
<p>In an ideal world all the backends (QEMU, bare metal, Xen) and consoles (VNC, serial or hybrid) would be accessed in a uniform manner by testapi so that the distribution and test writers could write their test case once and then have it run across all available platforms without modification. In practice however different Operating systems, hardware, hypervisors and console combinations differ significantly enough in behaviour that a completely uniform API is not possible without either significantly disadvantaging some platforms or providing support for edge cases in the distribution itself.</p>
<p>While the functions in testapi can be kept mostly uniform in availability and behaviour it requires that the distribution handles changes in the Operating System's behaviour due to the machine (virtual or physical) which it is running on and what user interface (console) is selected. Many things can be abstracted away into the console or backend classes in os-autoinst, however OS specific behaviour can not be without making os-autoinst specific to one type of OS or even Linux distribution. Currently the SUSE os-autoinst distribution handles differences between backends by reading variables to determine which backend or architecture is being used in a variety of different places and performing some particular action for that backend. Unfortunately this doesn't just happen in (suse)distribution.pm or other modules in the lib folder, but throughout the test cases.</p>
<p>The problem with branching on a particular architecture or backend is that the contents of the branch statement may actually apply to a whole class of backends not just one. Thus by restricting it to one particular backend you have missed an opportunity to maximise the benefit of your code, which will lead to duplication of effort. However in some cases it may be wasted effort to try inventing general abstractions when they will only be used in one or two instances, but then that is a universal problem, we just have to make a judgement on each and every case.</p>
<a name="Proposal"></a>
<h3 >Proposal<a href="#Proposal" class="wiki-anchor">¶</a></h3>
<p>At any rate my proposal is to introduce the notion of capabilities which can apply to consoles or backends. Any console or backend should have to declare its capabilities in a standard way which can then be read by the distribution and in some very rare cases, the distribution's test modules. Capabilities should be validated against a central list in the appropriate base class, attempting to access or set a capability which does not exist should be an error. This should make them better structured than simply adding more global variables which already serve this purpose to some extent. A list of quirks could also be maintained to indicate negative platform attributes.</p>
<p>The actual implementation could be done using a Perl map, object mixins from some Perl OO library or something else.</p>
<a name="Further-rambling"></a>
<h3 >Further rambling<a href="#Further-rambling" class="wiki-anchor">¶</a></h3>
<p>In the case of the serial terminal feature, I would remove the new testapi function I have added called is_serial_terminal and instead replace it with one or more console/backend capabilities. Perhaps something like <code>direct_read_text</code> and <code>direct_write_text</code> which indicates to the distribution that we are reading and writing raw text to the terminal, the backend would of course require a serial port capability to activate the console. The Linux VNC text console would have something like <code>redirect_write_text</code> which indicates we can redirect output to the serial port and some other capabilities to indicate that we can use needles and send key presses. Any console on some other OS/hardware combo which doesn't support serial ports will be missing the capabilities which indicate we can do this so either <code>run_script</code> and <code>wait_serial</code> will return an error indicating the missing capability or the distribution will have to implement the testapi functions using some other capabilities or workarounds.</p>
<p>Along with having capabilities comes the idea of having interfaces to take advantage of them, so that a backend, console and distribution with compatible capabilities can be plugged together. Such interfaces and their associated capabilities can be invented and implemented on a rolling basis rather than attempting to do some massive overhaul of the code base. This may slow down feature development for some time, but will eventually speed it up and creates a basis for a backend/console plugin architecture. I am willing to implement this in so far that it is required for <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: Add virtio serial console backend and API (Resolved)" href="https://progress.opensuse.org/issues/14582">#14582</a> and other platform specific features I think that the LTP, and other test suites I may work with, can take advantage of.</p>
<a name="Alternatives"></a>
<h3 >Alternatives<a href="#Alternatives" class="wiki-anchor">¶</a></h3>
<p>The alternatives are to forgo taking advantage of platform specific features or add the occasional function to the testapi like <code>is_serial_terminal</code> and use the existing vars mechanism. At least for what I am currently doing, the latter choice is acceptable to me, but it is not extensible beyond a point. There is also the console proxy feature which allows you to tightly couple your test module to a particular console implementation completely bypassing all layers of abstraction while at the same time obfuscating the code flow using Perl meta programming which should be avoided at least from tests perspective.</p>
<a name="Problems"></a>
<h3 >Problems<a href="#Problems" class="wiki-anchor">¶</a></h3>
<p>It is more difficult to identify and isolate a class of behaviour shared by multiple entities and create an abstraction to encapsulate it than just to write code for a specific case. Sometimes people may attempt to create capabilities when there is no significant advantage to doing so or they may be tempted not to when there clearly is an advantage. The feature will need documenting and require effort on the part of reviewers to learn it and enforce its use. It will probably increase the codebases complexity initially until it has been reasonably taken advantage of. There is the danger of an explosion in capabilities which makes it difficult to write a new distribution which covers multiple platforms without understanding a large number of them. Regressions may be introduced while moving backends and consoles over to this system.</p>
openQA Project - action #14582 (Resolved): Add virtio serial console backend and APIhttps://progress.opensuse.org/issues/145822016-10-31T10:40:37Zrpalethorperichard.palethorpe@suse.com
<p>I have written a new console backend which allows IO through a serial terminal, in particular the virtio console with QEMU, but I am currently thinking about how to generalise it. I started out mostly interested in how virtio could be used to speed up testing, but actually it has turned out to be mostly irrelevant. The most important thing is that communication is done as if a user is typing text into a serial terminal. For platforms which can be controlled entirely (or only) over UART (or UART over USB/Bluetooth/whatever) this opens up quite a few possibilities.</p>
<p>From <a href="https://github.com/os-autoinst/os-autoinst/pull/637:">https://github.com/os-autoinst/os-autoinst/pull/637:</a></p>
<pre><code>This allows the test writer to log into and interact with a serial
terminal directly. Initially this is just for the virtio_console under
QEMU, but can be extended to any serial console.
Presently the only way of sending text to a tty running on a SUT is to
send the keystrokes via VNC. Output from the SUT can be redirected to
and read from, a serial port, however the test writer can not send text
directly to the serial port without circumventing the test API.
This patch introduces a new console backend which can be selected in the
same manner that the existing console backends are, but is limited to
text input and output. This means that there is no video feed, but
entering commands is orders of magnitude faster because the commands are
sent and interpreted as text rather than simulated keystrokes.
This adds the is_serial_terminal subroutine to the testapi which is
exported on request. Otherwise the API should be unchanged and fully
backwards compatible.
</code></pre>
<p>Currently the virtio serial terminal must be requested by the test case. I think this is fine for now, because it is primarily needed for the LTP native runner I am also working on. However after further testing and modifications to the UI to display a text feed rather than a video, I think it can be automatically used when 'root-console' (or similar) is requested. Further improvements and features may include:</p>
<a name="Display-serial-log-in-OpenQA-UI"></a>
<h3 >Display serial log in OpenQA UI<a href="#Display-serial-log-in-OpenQA-UI" class="wiki-anchor">¶</a></h3>
<p>Currently a bitmap feed is displayed, which is nice, and using a serial terminal breaks that. The user can still see, to some extent, what is happening in the virtual machine from the os-autoinst log feed. However it would be nice if they could see the tty log in real time. All that needs to happen is that either os-autoinst relays IO from the serial socket to the UI or the UI reads a log file containing the serial terminal output (e.g. QEMU produces such a file). There is probably a javascript library for handling terminal escape codes, but otherwise the raw text can just be displayed. Alternatively an image could be generated of the text terminal, but that will increase network and processor load.</p>
<a name="Use-serial-terminal-whenever-possible"></a>
<h3 >Use serial terminal whenever possible<a href="#Use-serial-terminal-whenever-possible" class="wiki-anchor">¶</a></h3>
<p>This should be easy to enable, but there are two issues. One is user experience, which I have discussed above. The second is the possibility of bugs caused by subtle differences in the testapi behaviour when switching to a serial terminal. If we explicitly switch a few different tests over to using serial, then we can probably catch most problems without exposing the entire test suite to them at once.</p>
<a name="Implement-it-for-other-platforms"></a>
<h3 >Implement it for other platforms<a href="#Implement-it-for-other-platforms" class="wiki-anchor">¶</a></h3>
<p>To my knowledge, all the backends have some ability to read input from a serial port. Ideally we will want to open a second serial port which OpenQA is not already using, but it should also be possible to use the existing one. It is then a case of generalising the existing code for virtio_console to read from a different socket. If a backend only has one serial port then we perhaps have to come up with a mechanism for ignoring kernel log messages which are usually sent to the serial port.</p>
openQA Project - action #14100 (Rejected): Implement ClientCutText for VNC to speed up sending texthttps://progress.opensuse.org/issues/141002016-10-07T08:47:04Zrpalethorperichard.palethorpe@suse.com
<p>Assuming the backend's VNC server supports *CutText actions we can send text more quickly using the ClientCutText message: <a href="https://tools.ietf.org/html/rfc6143#section-7.5.6" class="external">https://tools.ietf.org/html/rfc6143#section-7.5.6</a></p>
<p>Control flow:</p>
<ol>
<li>Test case calls type_string or perhaps a new call like paste_string</li>
<li>Check the guest is in a state which supports the clipboard</li>
<li>Check the string for any none latin characters or control codes which may break the operation</li>
<li>Send ClientCutText message in VNC.pm</li>
<li>Send the appropriate key sequence to perform paste/yank</li>
</ol>
<p>Similarly ServerCutText can be used to send text in the opposite direction, if the test writer can reliably copy text to the clipboard.</p>
<p>Potential problems:</p>
<ul>
<li>The backends may not support the *CutText operations</li>
<li>It may require a daemon to be running on the guest OS</li>
<li>Not all software supports the clipboard.</li>
</ul>
<p>Advantages:</p>
<ul>
<li>Faster</li>
<li>Won't drop keypresses</li>
<li>May work in most situations</li>
</ul>
<p>I will investigate further if other attempts to speed up text input are not adequate.</p>
openQA Tests - action #13140 (Resolved): Job template organizationhttps://progress.opensuse.org/issues/131402016-08-10T13:59:43Zrpalethorperichard.palethorpe@suse.com
<p>I find the products/opensuse/templates file to big and painful to deal with. Perhaps a templates sub-directory should be created, which we then insert template files into just containing test suits, machines, job groups etc. which are tightly related.</p>
<p>That way someone can view all the information relating to these tests on one or two pages and import the tests without worrying if anything unexpected is going to be added. If someone needs to add all the templates at once then they can write something like ./load_templates templates/*.</p>
<p>Templates provide a way of recording how a machine or test suit should be used, in a transportable and human readable format, but one huge monolithic file is only useful for automated export-import. I used this template during my local installation by following the instructions, but the number of tests it has added was counter productive when trying to learn the system. If new users want to see a complex setup they can look no further than openqa.suse.de.</p>
openQA Project - action #12848 (Resolved): os-autoinst: CDROM assumed to be on SCSI controllerhttps://progress.opensuse.org/issues/128482016-07-25T10:18:24Zrpalethorperichard.palethorpe@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>When running os-autoinst separately from OpenQA. If <code>CDMODEL</code> in <code>vars.json</code> is not set to something beginning with <code>virtio-scsi</code> then it will fail to start the virtual machine.</p>
<pre><code>QEMU: qemu-system-x86_64: -device scsi-cdrom,drive=cd0,bus=scsi0.0: 'scsi-cdrom' is not a valid device model name
</code></pre>
<a name="Reproduction"></a>
<h2 >Reproduction<a href="#Reproduction" class="wiki-anchor">¶</a></h2>
<p>Create a vars.json similar to this</p>
<pre><code>{
"ARCH" : "x86_64",
"BACKEND" : "qemu",
"CASEDIR" : "/home/richie/qa/os-autoinst-distri-opensuse",
"CDMODEL" : "scsi-cdrom",
"DISTRI" : "opensuse",
"ISO" : "/var/lib/openqa/factory/iso/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20160715-Media.iso",
"PRODUCTDIR" : "/home/richie/qa/os-autoinst-distri-opensuse/products/opensuse",
"VNC" : 90,
}
</code></pre>
<p>and run <code>isotovideo</code>. Changing CDROM to <code>virtio-scsi-pci</code> allows isotovideo to run succesfully, however it overwrites the value with <code>scsi-cdrom</code> meaning that it will fail on the next run.</p>
<a name="Discussion"></a>
<h2 >Discussion<a href="#Discussion" class="wiki-anchor">¶</a></h2>
<p>In the file qemu.pm on line 305 it decides whether or not to create a SCSI controller based on the type of drives being used. It looks for drives beginning with <code>virtio-scsi</code>, if none are present then no scsi controller is created. Then on line 548 it assumes the CDROM has the bus address <code>scsi0.0</code>.</p>
<p>It is not clear to me where the value <code>scsi-cdrom</code> comes from. The default for <code>CDMODEL</code> in <code>qemu.pm</code> is <code>virtio-scsi-pci</code>.</p>