Project

General

Profile

Actions

action #18414

closed

[sles][functional] C/Cpp compilation relying on LTP seems very unstable

Added by nicksinger almost 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Enhancement to existing tests
Start date:
2017-03-31
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

Now this one fails because of the asset_script_run: https://openqa.suse.de/tests/861620
LTP tests fail or compilation fails (also in the corresponding ppc64le job in same build).

Reproducible

Fails since (at least) Build 0303 (current job) but looking back at the history we also see similar errors in before

Expected result

simple compilation

Problem

Why need the complex LTP module and not just simple "hello world"

Suggestion

  • Use something more stable?

Further details

Always latest result in this scenario: latest


Related issues 2 (0 open2 closed)

Has duplicate openQA Tests - action #18614: test fails in gcc5_C_compilationRejected2017-04-18

Actions
Copied from openQA Tests - action #18206: [sles][functional]script_run calls in gcc5_Cpp_compilation fail -> assert_script_run?Resolvedmichalnowak2017-03-31

Actions
Actions #1

Updated by nicksinger almost 7 years ago

  • Copied from action #18206: [sles][functional]script_run calls in gcc5_Cpp_compilation fail -> assert_script_run? added
Actions #2

Updated by michalnowak almost 7 years ago

I try to test real use cases, helloworld.c does not look like one of them. LTP is at times unstable. But in this case this is most likely a product bug https://bugzilla.suse.com/show_bug.cgi?id=1024050. I don't think we should do anything about the test we have atm.

Actions #3

Updated by michalnowak almost 7 years ago

  • Status changed from New to In Progress
  • Assignee set to michalnowak

I could actually add the force_cron_run workaround to mitigate the problem. If it's btrfs actually.

Actions #4

Updated by michalnowak almost 7 years ago

  • Status changed from In Progress to Feedback
Actions #5

Updated by nicksinger almost 7 years ago

I think you're true on the real world test. A simple "hello world" does not test enough to make sure gcc works fine.
But the current test tests not only that compilation works but also if LTP runs successfully (which we cannot ensure).
My suggestion would be to use "some complex c program" (to test enough gcc features) which produces a predictable return code to be sure that an actual errors happens because of a gcc bug.

Actions #6

Updated by michalnowak almost 7 years ago

  • Status changed from Feedback to In Progress
  • Assignee deleted (michalnowak)

Passed: https://openqa.suse.de/tests/871719. Unassigning myself as I am not convinced the said LTP test un/stability posses is a problem.

Nick: feel free to close or assign.

Actions #7

Updated by nicksinger almost 7 years ago

Take a look at these two failures:
https://openqa.suse.de/tests/878180#step/gcc5_Cpp_compilation/21
https://openqa.suse.de/tests/878354#step/gcc5_C_compilation/38

IMHO this test tests not only the core functionality of gcc but more the success of an LTP run itself.

Actions #8

Updated by okurz almost 7 years ago

  • Has duplicate action #18614: test fails in gcc5_C_compilation added
Actions #9

Updated by okurz almost 7 years ago

https://openqa.suse.de/tests/887697#step/gcc5_Cpp_compilation/19 and also more in the same scenario in older jobs.

who can do the simple task and check the timeout once on a manually cloned job and then increase the timeout?

Actions #10

Updated by asmorodskyi almost 7 years ago

this issue is unrelated to question that okurz asked , I open separate issue regarding timeout https://progress.opensuse.org/issues/18692

main idea if this issue is to refactor module to not relay on ltp_tests results while we trying to answer a question "if compiler is working properly?" because this question is totally unrelated to a question "does ltp tests pass?"

Actions #11

Updated by michalnowak almost 7 years ago

To avoid the sin of repetition, I'll just past the IRC conversation FTR:

mnowak: why we can't just try to compile something really tiny in terms of gcc5_Cpp_compilation test ?
mnowak: what the point to compile really huge thing which take some match time ? our goal just to make sure that compilation works , right ?
s/some match/so much/
asmorodskyi, I want GCC from toolchain module to have a decent coverage, because we don't use it anywhere else. stock GCC is used building the distro (I presume). to test it decently we need to compile something and then execute it. test suite seems to me a right way to do both things at once
asmorodskyi, I don't find problems with those test suites so unbearable
mnowak: in my opinion you need to have really good reasons to have a test which run for 3 hours
mnowak: can you please state which "both things" this test covers ?
asmorodskyi, random x86_64 https://openqa.suse.de/tests/894012 did it in 85 minutes. it's just that xgene2 is a slow crap
asmorodskyi, that should read "compilation and execution", we need to run both
mnowak: you mean compilation by gcc and execution of compiled executable to double check that gcc produce something workable , right ?
asmorodskyi, correct
mnowak: I am not big expert in gcc , but I am in doubt that sources of any package can be used to prove that gcc is working. Just brief googling in subject gives me that gcc test suite has more than 50,000 tests which cover it's functionality. And I am suspecting that this tests are sources written in some special way which make gcc use some specific features. Of course you can say that we don't need such deep coverage but than the question what
actually we trying to cover ? Just simple statement "gcc can compile and you can execute resulted binary" ? Then simple HelloWorld.cpp will be enough ?

mnowak: my point is if we want something complex in gcc testing we need clear understand what actually covering and have some gcc test suite which will give us clear answer what is working and what not in gcc . If we don't care so much then why we need to wait 2 hours to understand that gcc "can compile something" ?
asmorodskyi, with the toolchain module's GCC it's the same as with any other component: we need to test it in it's native environment = as the user will run it. GCC's internal test suite is way more unstable (I did QA on GCC before), but would not mind using it in principle.
asmorodskyi, I understand you point. and it's a valid one. but one thing I noticed with various projects like JeOS and CaaSP is that we don't know what should be tested, because no one knows. so we try our best. I don't think picking some test module and showing we don't have a clear plan helps
mnowak: do you have any concerns about replacing "llvm" package which you currently trying to compile with something smaller ? I mean how you can to conclusion that exactly "llvm" package suppose to be compiled ?
mnowak: I am not sure that regular toolchain users compile it , right ?
asmorodskyi, I did not. original author of the test did, if I recall it correctly it was jpupava
jpupava: how you decided to use "llvm" in gcc compilation test ?
asmorodskyi, the reasoning was probably: "lets find something non-trivial". LLVM is non-trivial hence is the answer
mnowak: I am sure that we can find something "non-trivial" which builds for 30 minutes :) how do you think ?
asmorodskyi, frankly, I'd rather keep it as it is and not to "simplify" it just because aarch64 in openQA is in dire situation at the moment
mnowak, aarch64 will always be in a dire situation for something as heavy as this - unless we can justify why we need something THAT big, I think we can find something non-trivial and a lot faster than several hours

Actions #12

Updated by okurz over 6 years ago

  • Target version set to Milestone 13
Actions #13

Updated by michalnowak over 6 years ago

  • Status changed from In Progress to Resolved

I believe this was resolved a while back by removing LTP and LLVM compilation & runs from SLE Functional's toolchain by Anton.

Actions

Also available in: Atom PDF