Project

General

Profile

action #55478

Evaluate circleci for openQA

Added by okurz 11 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Organisational
Target version:
Start date:
2019-08-14
Due date:
% Done:

100%

Estimated time:
16.00 h
Difficulty:
Duration:

Description

Motivation

We use Travis CI successfully but having our build environment primarily based on containers anyway we might benefit from a different approach to save time. OBS is also using it


Related issues

Related to openQA Project - action #57050: Turn off travisResolved2019-09-30

Copied to openQA Project - action #56522: Create cron job which will create pull requests with list of current openQA-devel dependencies with versionResolved2019-08-14

History

#1 Updated by coolo 10 months ago

  • Target version set to Ready

The main benefit is the way it stores artefacts (i.e. logs from tests) and that you can ssh into running test containers to debug within the environment. This can make it much faster to turn around a problem

#2 Updated by andriinikitin 10 months ago

  • Status changed from New to In Progress
  • Assignee set to andriinikitin
  • Estimated time set to 16.00 h

#3 Updated by coolo 10 months ago

  • Target version changed from Ready to Current Sprint

#4 Updated by andriinikitin 10 months ago

  • Copied to action #56522: Create cron job which will create pull requests with list of current openQA-devel dependencies with version added

#5 Updated by andriinikitin 10 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

General Considerations

  • circleci provides list of commands which can be used in config. E.g. checkout will check out current branch into project in current environment (e.g. in container) .
  • circleci provides local tool circleci : -- check syntax using command circleci config process .circleci/config.yml -- execute job from .circleci/config.yml: check syntax using command circleci local execute --job testui
  • choose between machine and docker executor. machine allows to manage containers with docker commands, while docker executor allows command only inside container. Officially machine executor is not supported for commands of local circleci tool. E.g. for me checkout command didn't work locally
  • circleci Dashboard allows to restart jobs "with SSH" and easily login into executor using command printed at job start

OpenQA with circleci

  • tests look running much faster (may need detailed analysis) -- unit and integration tests : 20-30 min vs 11 min -- UI tests 20-30 min vs 9 min -- fullstack 15 min vs 10 min
  • it looks circleci environment is quite sensitive to unstable tests and shows failures more often. (copmaring to travis).
  • easy to run tests on local machine using circleci tool. (The downside is ablsence of cache, so local commands despite being simple and explicit do execute quite long, because they need to build image).

TODO

  • codecov may still need work to have it didn't work from circleci (because of fork?)
  • 'documentation generation job' from travis was left out of scope
  • troubleshoot unstable tests

#6 Updated by okurz 10 months ago

hi andriinikitin, could you please use the status "Resolved" rather than "Closed" as we commonly do for all tickets? This is following https://progress.opensuse.org/projects/openqav3/wiki#ticket-workflow

Also, I guess you want to have a follow-up ticket for the TODOs?

#7 Updated by andriinikitin 10 months ago

  • Status changed from Closed to Resolved

#8 Updated by andriinikitin 10 months ago

Also available in: Atom PDF