QA » openQA Project

Preferably, openQA should use a key or certificate (either provided or self-generated) that should be used as part of the signing process, allowing users to know that their results haven't been tampered with.

There are two versions of this requirement, but the one that's coming from FX is about being able to ensure that hashes submitted in test reports, that are pointing to a set of test execution reports, weren't modified by a third-party.

I have background about how VVSG certifications work and the kind of documents that have to be submitted, in most cases they are part of a test plan+test reports (see this report,

Specifically:

• P25 System Test and Verification, item 1 (likely under NDA)
• P27 11_QA Program and 12_System Change Notes
• P29 Section 3.1 (an example of usability test report is here
• P39 Table 4-1
• P40 Table 4-2

And in the case of FIPS, one has to submit what are the commands (as part of the test plan) and their respective outputs and so on...

In the meantime, I'm looking for a better answer: "Why is it important for the user to archive the results and ensure that they weren't tampered with?"

In the meantime, another (internal) use case that goes in a different direction.

As a Security Engineer, I have to give Auditors evidence that certain programs work in a certain way. It would ease my life, if OpenQA could support me in that way.

For example:
When OpenQA runs a successful maintenance fips command line test under certain conditions, I would not only like to have the result, but also the way how to reproduce it without OpenQA.

The initial idea would be, that commands which are run in OpenQA for actually doing the test can be extracted. The commands may be:

    compiling an input artefact/code
    command line calls of openssl, grep, awk, echo and others
    processing some data generated during the test
    running a compiled code
    show some result state (bash variable of success or fail)


The idea of extraction would be, that the sequence of the test procedure would be available as an asset, perhaps a shell script, with the right input artefacts could reproduce the successful test in another unixish environment.

Of course there could preconditions like "Product XY with installed packages x, y, z and an encrypted root volume", perhaps as a comment within the shellscript. Some of this preconditions can be fullfilled with the script, but others have to be done outside of the testscript, because the have to be done by the external evidence-reproducer (aka auditor)


The value of this proposal is that it is easy for this company to regenerate reproducable evidence, so others do not have to believe our word, but can verify it independently AND it's easy for us to have.

The key for both stories is that there is an external party who is going to execute the test cases in the same way, as stated in the submitted documentation, I think the easiest way to do this, is to have a report generator that uses the os-autoinst log, to produce such thing

szarate wrote:

The key for both stories is that there is an external party who is going to execute the test cases in the same way, as stated in the submitted documentation, I think the easiest way to do this, is to have a report generator that uses the os-autoinst log, to produce such thing

Thank you, that is very valuable information. So far our assumptions were that there was merely a requirement to guarantee the integrity of archived test results regarding accidental corruption during long term storage. Clearly it wouldn't be enough to create checksums for the test output alone, the inputs also need to be auditable and reproducible. That adds a whole new level of complexity to this feature.

okurz wrote:

I am looking for another older ticket about that later request. A test result is only completely reproducible if all steps are followed with same timing, same environment, etc. Expecting a "simple list of commands" automatically generated from log files is just naive and won't fly. @szarate until we can clarify further I'd rather keep this ticket out of the backlog for now.

I have a call with FX tomorrow about his request, but I agree that this needs to be clarified further before it's actually workable.

I am looking for another older ticket about that later request. A test result is only completely reproducible if all steps are followed with same timing, same environment, etc.

The timing is not necessary, but the steps are... still we have to iterate over the requirement and have good discovery before working on it.

kraih wrote:

Clearly it wouldn't be enough to create checksums for the test output alone, the inputs also need to be auditable and reproducible. That adds a whole new level of complexity to this feature.

The more I think about this feature and some others, the more "reproducible buildstests" comes to mind... taking the timing variable out of the equation, there are few things we might be able to do to achieve that... and I feel that we're half way there, but that's a different story, and another whole big can of worms...

/me dreams in Gage repeatability and reproducibility (GR&R) applied to software instead of measuring equipment

I had a good chat with FX some days ago, and one thing that came clear to me is that while this might be a requirement, it is not yet set in stone, and has to be defined better from PM, at this point in time this is more related to notarization of the test artifacts as part of a SBOM.