Project

General

Profile

action #90047

Improve dependency detection in cpanspec

Added by tinita 5 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2021-03-12
Due date:
2021-05-07
% Done:

0%

Estimated time:

Description


Related issues

Related to QA - action #92122: Improve dependency detection via MYMETA.json in cpanspecNew2021-05-04

Related to QA - action #92119: Improve license detection in cpanspecNew2021-05-04

History

#1 Updated by tinita 5 months ago

I just did a little fix to the code reading Makefile.PL to also fetch the TEST_REQUIRES.

We should still add META.json, and additionally check the dynamic_config property - if false, we don't need to read Makefile.PL, Build.PL etc.

#2 Updated by tinita 5 months ago

  • Project changed from QA to openQA Project
  • Category set to Concrete Bugs

#3 Updated by okurz 5 months ago

I think the generic "QA" project is better fitting. Or is it really connected to "openQA"?

#4 Updated by tinita 5 months ago

  • Project changed from openQA Project to QA
  • Category deleted (Concrete Bugs)

Ok

#6 Updated by okurz 5 months ago

ok, what's left to call this ticket "Resolved" after the merged PR?

#7 Updated by okurz 5 months ago

  • Due date set to 2021-03-31

Setting due date based on mean cycle time of SUSE QE Tools

#8 Updated by tinita 5 months ago

What's left? Reading META.json and also decide further actions depending on dynamic_config.

#9 Updated by cdywan 4 months ago

  • Due date changed from 2021-03-31 to 2021-04-16

Bumping due date due to vacation.

#10 Updated by tinita 4 months ago

I ran the current cpanspec on about 20% of the modules (from perl-ACH to perl-Data-Munge) in devel:languages:perl, looked if the changes are expected and committed them to a local git repository.

Now I can make changes to cpanspec and run it on all of those modules to ensure there are no unexpected changes.

#11 Updated by tinita 4 months ago

I'm also creating a Dockerfile, so in the future it will be easier to install the cronjob on a new server and get all the dependencies and environment variables right.

#12 Updated by tinita 4 months ago

  • Due date changed from 2021-04-16 to 2021-04-23

#13 Updated by tinita 4 months ago

I made progress with the dependency handling.

Now the script is reading META.json first.
If META.json is missing or didn't contain any prereqs, META.yml is read and checked for prereqs.

If prereqs have been identified and dynamic_config is false, the step to execute Makefile.PL is skipped.

Also refactored a bit and moved things into subroutines.

Still running the last tests but it looks good.

The next step would be to check the license field, and maybe improve the abstract handling (abstract is a required field in META.json, so we can skip reading it from POD in this case).

#14 Updated by tinita 3 months ago

I ran some more tests and fixed some minor bugs, and also stumbled over a bug related to @INC.
Data::Rand::Obscure has this Build.PL:
https://metacpan.org/source/RKRIMEN/Data-Rand-Obscure-0.021/Build.PL

Because the . has been removed from @INC, what happened with the current cpanspec version is that cpanspec's root path was added to @INC, so it loaded cpanspec's own Makefile.PL instead of the module's one, which lead to the following Require entries https://build.opensuse.org/package/view_file/devel:languages:perl/perl-Data-Rand-Obscure/perl-Data-Rand-Obscure.spec?expand=1:

BuildRequires:  perl-macros
BuildRequires:  perl(Archive::Tar)
BuildRequires:  perl(Archive::Zip)
BuildRequires:  perl(Carp::Clan)
BuildRequires:  perl(IO::Uncompress::Bunzip2)
BuildRequires:  perl(LWP::UserAgent)
BuildRequires:  perl(Module::Build)
BuildRequires:  perl(Object::Tiny)
BuildRequires:  perl(Parse::CPAN::Packages)
BuildRequires:  perl(Pod::Simple::TextContent)
BuildRequires:  perl(Text::Autoformat)
BuildRequires:  perl(YAML)
Requires:       perl(Archive::Tar)
Requires:       perl(Archive::Zip)
Requires:       perl(Carp::Clan)
Requires:       perl(IO::Uncompress::Bunzip2)
Requires:       perl(LWP::UserAgent)
Requires:       perl(Object::Tiny)
Requires:       perl(Parse::CPAN::Packages)
Requires:       perl(Pod::Simple::TextContent)
Requires:       perl(Text::Autoformat)
Requires:       perl(YAML)

My new version moved the code around and didn't put the cpanspec root path to @INC anymore, so I stumbled upon this.

#15 Updated by tinita 3 months ago

I cleaned up the new code a bit, added a debug mode.
Also I added parseable statistics output (YAML :). With this we can parse the logs of the automatic CPAN-* and d:l:p updates.
This will give us easy access to the meta data of the newest releases.

The new code is also a bit faster (if running Makefile.PL/etc. and/or PPI parsing can be skipped, which is more likely the case for newer modules/releases).
A quick benchmark for 50 modules in d:l:p: 12m5.787s (old) vs. 9m20.203s (new).

I will create an extra ticket for the license improvements, and the statistic data that has been collected until then can be used.

I will run this branch on the autoupdate host for a few days and monitor, and if it looks fine, make a PR.

#16 Updated by tinita 3 months ago

I rolled out the code on the autoupdate host and will monitor for the next few days.
I installed logrotate for the logfiles (which is something that I wanted to do a long time ago), so logs from the previous day aren't overwritten anymore.

#17 Updated by tinita 3 months ago

  • Due date changed from 2021-04-23 to 2021-04-30

Moving due date.

Running the current code on a significant amount of packages and comparing results of each package to filter out existing problems or packages that should be left alone took a bit longer than originally expected.

#18 Updated by tinita 3 months ago

  • Status changed from In Progress to Feedback

#19 Updated by tinita 3 months ago

  • Due date changed from 2021-04-30 to 2021-05-07

PR is still running on the autoupdate host for further tests. Looking fine so far.

#20 Updated by tinita 3 months ago

  • Status changed from Feedback to Resolved

Merged PR https://github.com/openSUSE/cpanspec/pull/29

Rolled out master on autoupdate host.

#21 Updated by tinita 3 months ago

  • Related to action #92122: Improve dependency detection via MYMETA.json in cpanspec added

#22 Updated by tinita 3 months ago

  • Related to action #92119: Improve license detection in cpanspec added

Also available in: Atom PDF