QA » openQA Project » openQA Tests

I try to test real use cases, helloworld.c does not look like one of them. LTP is at times unstable. But in this case this is most likely a product bug https://bugzilla.suse.com/show_bug.cgi?id=1024050. I don't think we should do anything about the test we have atm.

I think you're true on the real world test. A simple "hello world" does not test enough to make sure gcc works fine.
But the current test tests not only that compilation works but also if LTP runs successfully (which we cannot ensure).
My suggestion would be to use "some complex c program" (to test enough gcc features) which produces a predictable return code to be sure that an actual errors happens because of a gcc bug.

Take a look at these two failures:
https://openqa.suse.de/tests/878180#step/gcc5_Cpp_compilation/21
https://openqa.suse.de/tests/878354#step/gcc5_C_compilation/38

IMHO this test tests not only the core functionality of gcc but more the success of an LTP run itself.

https://openqa.suse.de/tests/887697#step/gcc5_Cpp_compilation/19 and also more in the same scenario in older jobs.

who can do the simple task and check the timeout once on a manually cloned job and then increase the timeout?

this issue is unrelated to question that okurz asked , I open separate issue regarding timeout https://progress.opensuse.org/issues/18692

main idea if this issue is to refactor module to not relay on ltp_tests results while we trying to answer a question "if compiler is working properly?" because this question is totally unrelated to a question "does ltp tests pass?"

To avoid the sin of repetition, I'll just past the IRC conversation FTR:

mnowak: why we can't just try to compile something really tiny in terms of gcc5_Cpp_compilation test ?
mnowak: what the point to compile really huge thing which take some match time ? our goal just to make sure that compilation works , right ?
s/some match/so much/
asmorodskyi, I want GCC from toolchain module to have a decent coverage, because we don't use it anywhere else. stock GCC is used building the distro (I presume). to test it decently we need to compile something and then execute it. test suite seems to me a right way to do both things at once
asmorodskyi, I don't find problems with those test suites so unbearable
mnowak: in my opinion you need to have really good reasons to have a test which run for 3 hours
mnowak: can you please state which "both things" this test covers ?
asmorodskyi, random x86_64 https://openqa.suse.de/tests/894012 did it in 85 minutes. it's just that xgene2 is a slow crap
asmorodskyi, that should read "compilation and execution", we need to run both
mnowak: you mean compilation by gcc and execution of compiled executable to double check that gcc produce something workable , right ?
asmorodskyi, correct
mnowak: I am not big expert in gcc , but I am in doubt that sources of any package can be used to prove that gcc is working. Just brief googling in subject gives me that gcc test suite has more than 50,000 tests which cover it's functionality. And I am suspecting that this tests are sources written in some special way which make gcc use some specific features. Of course you can say that we don't need such deep coverage but than the question what
actually we trying to cover ? Just simple statement "gcc can compile and you can execute resulted binary" ? Then simple HelloWorld.cpp will be enough ?

mnowak: my point is if we want something complex in gcc testing we need clear understand what actually covering and have some gcc test suite which will give us clear answer what is working and what not in gcc . If we don't care so much then why we need to wait 2 hours to understand that gcc "can compile something" ?
asmorodskyi, with the toolchain module's GCC it's the same as with any other component: we need to test it in it's native environment = as the user will run it. GCC's internal test suite is way more unstable (I did QA on GCC before), but would not mind using it in principle.
asmorodskyi, I understand you point. and it's a valid one. but one thing I noticed with various projects like JeOS and CaaSP is that we don't know what should be tested, because no one knows. so we try our best. I don't think picking some test module and showing we don't have a clear plan helps
mnowak: do you have any concerns about replacing "llvm" package which you currently trying to compile with something smaller ? I mean how you can to conclusion that exactly "llvm" package suppose to be compiled ?
mnowak: I am not sure that regular toolchain users compile it , right ?
asmorodskyi, I did not. original author of the test did, if I recall it correctly it was jpupava
jpupava: how you decided to use "llvm" in gcc compilation test ?
asmorodskyi, the reasoning was probably: "lets find something non-trivial". LLVM is non-trivial hence is the answer
mnowak: I am sure that we can find something "non-trivial" which builds for 30 minutes :) how do you think ?
asmorodskyi, frankly, I'd rather keep it as it is and not to "simplify" it just because aarch64 in openQA is in dire situation at the moment
mnowak, aarch64 will always be in a dire situation for something as heavy as this - unless we can justify why we need something THAT big, I think we can find something non-trivial and a lot faster than several hours