Project

General

Profile

Actions
Copy link

tickets #70018

closed

tickets #19900: Consider moving mail lists to mailman 3 + posterious and hyperkitty

Planning for the mailman3 migration

Added by hellcp over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
hellcp
Category:
Core services and virtual infrastructure
Target version:
-
Start date:
2020-08-13
Due date:
% Done:

0%

Estimated time:

Description

Because I'm lazy, I pretty much scripted the entire process of importing the mailing lists

on baloo with an account allowed to read subscriber lists

#!/usr/bin/sh

mkdir -p /tmp/transition
ls /var/spool/mlmmj/ > /tmp/transition/mailing-lists
for i in $(cat /tmp/transition/mailing-lists); do
  cat /srv/www/htdocs/$i/$i-2020-11.mbox > /tmp/transition/$i.mbox
  mlmmj-list -L /var/spool/mlmmj/$i > /tmp/transition/$i.subscribers
  mlmmj-list -L /var/spool/mlmmj/$i -n > /tmp/transition/$i.nomail
  mlmmj-list -L /var/spool/mlmmj/$i -o > /tmp/transition/$i.owners
  mlmmj-list -L /var/spool/mlmmj/$i -m > /tmp/transition/$i.moderators
done

on a local machine for verification and to write $list.mapping file listing mailman list ids and $list.address file listing mailman list address, since mailing lists have to be set up manually, since mailman lacks a few essential settings when creating mailing lists by itself (we basically need to have them setup in postorius to be private, so relying on cli isn't the best of ideas)

#!/usr/bin/sh

rsync --info=progress2 -r hellcp@baloo.infra.opensuse.org:/tmp/transition ./
rsync --info=progress2 -r transition hellcp@mailman3.infra.opensuse.org:/tmp/

on mailman3 with mailman user

#!/usr/bin/sh

for i in $(cat /tmp/transition/mailing-lists); do
  cd /var/lib/mailman/
  mailman addmembers -W -d regular /tmp/transition/$i.subscribers $(cat /tmp/transition/$i.mapping)
  mailman addmembers -W -d disabled /tmp/transition/$i.nomail $(cat /tmp/transition/$i.mapping)
  cd /var/lib/mailman_webui/
  python3 manage.py hyperkitty_import --no-sync-mailman --since 1990-01-01 -l $(cat /tmp/transition/$i.address) /tmp/transition/$i.mbox
  python3 /tmp/transition/modop.py $i
done

and additional python script to add owners and moderators under /tmp/transition/modop.py referenced above

#!/usr/bin/python3

import sys
import os
import mailmanclient

list_name = sys.argv[0]

mapping = ''

with open('/tmp/transition/' + list_name + '.mapping') as mapping_file:
  mapping = mapping_file.read().replace('\n', '')

client = mailmanclient.Client("localhost:8001", "mailman", os.environ['MAILMAN_PASSWORD'])
list = client.get_list(mapping)
with open('/tmp/transition/' + list_name + '.owners') as owners:
  for owner in owners:
    list.add_owner(owner)
with open('/tmp/transition/' + list_name + '.moderators') as moderators:
  for moderator in moderators:
    list.add_moderator(moderator)

The prototype mapping of the old to the new addresses (some of the addresses will only be imported for archives, and their backend mailing lists will be removed soon after)

admin@opensuse.org                             admin@lists.opensuse.org
admin-auto@opensuse.org                        admin-auto@lists.opensuse.org
board@opensuse.org                             board@lists.opensuse.org
cfp@opensuse.org                               cfp@lists.opensuse.org
donations@opensuse.org                         donations@lists.opensuse.org
election-officials@opensuse.org                election-officials@lists.opensuse.org
forums-admin@opensuse.org                      forums-admin@lists.opensuse.org
freighttrain@opensuse.org                      freighttrain@lists.opensuse.org
gsoc-mentors@opensuse.org                      gsoc-mentors@lists.opensuse.org
heroes@opensuse.org                            heroes@lists.opensuse.org
ircops@opensuse.org                            ircops@lists.opensuse.org
journalists@opensuse.org                       journalists@lists.opensuse.org
kernel-bugs@opensuse.org                       kernel-bugs@lists.opensuse.org
kubic-bugs@opensuse.org                        kubic-bugs@lists.opensuse.org
libeconf-devel@opensuse.org                    libeconf-devel@lists.opensuse.org
libyui-commit@opensuse.org                     libyui-commit@lists.opensuse.org
maintenance@opensuse.org                       maintenance@lists.opensuse.org
membership-officials@opensuse.org              membership-officials@lists.opensuse.org
mirror@opensuse.org                            mirror@lists.opensuse.org
mysql-packaging@opensuse.org                   mysql-packaging@lists.opensuse.org
news@opensuse.org                              news@lists.opensuse.org
obs-commits@opensuse.org                       obs-commits@lists.opensuse.org
obs-devel@opensuse.org                         obs-devel@lists.opensuse.org
obs-errors@opensuse.org                        obs-errors@lists.opensuse.org
obs-tests@opensuse.org                         obs-tests@lists.opensuse.org
openqa-admin@opensuse.org                      openqa-admin@lists.opensuse.org
opensuse@opensuse.org                          users@lists.opensuse.org
opensuse-announce@opensuse.org                 announce@lists.opensuse.org
opensuse-arm@opensuse.org                      arm@lists.opensuse.org
opensuse-artwork@opensuse.org                  artwork@lists.opensuse.org
opensuse-autoinstall@opensuse.org              autoinstall@lists.opensuse.org
opensuse-base@opensuse.org                     base@lists.opensuse.org
opensuse-bugs@opensuse.org                     bugs@lists.opensuse.org
opensuse-bugshare@opensuse.org                 bugshare@lists.opensuse.org
opensuse-buildservice@opensuse.org             buildservice@lists.opensuse.org
opensuse-cloud@opensuse.org                    cloud@lists.opensuse.org
opensuse-commit@opensuse.org                   commit@lists.opensuse.org
opensuse-conference@opensuse.org               conference@lists.opensuse.org
opensuse-cz@opensuse.org                       users-cz@lists.opensuse.org
opensuse-de@opensuse.org                       users-de@lists.opensuse.org
opensuse-doc@opensuse.org                      doc@lists.opensuse.org
opensuse-edu@opensuse.org                      edu@lists.opensuse.org
opensuse-el@opensuse.org                       users-el@lists.opensuse.org
opensuse-es@opensuse.org                       users-es@lists.opensuse.org
opensuse-factory@opensuse.org                  factory@lists.opensuse.org
opensuse-factory-mozilla@opensuse.org          factory-mozilla@lists.opensuse.org
opensuse-features@opensuse.org                 features@lists.opensuse.org
opensuse-fr@opensuse.org                       users-fr@lists.opensuse.org
opensuse-gnome@opensuse.org                    gnome@lists.opensuse.org
opensuse-go@opensuse.org                       go@lists.opensuse.org
opensuse-ha@opensuse.org                       ha@lists.opensuse.org
opensuse-haskell@opensuse.org                  haskell@lists.opensuse.org
opensuse-hu@opensuse.org                       users-hu@lists.opensuse.org
opensuse-isdn-de@opensuse.org                  isdn-de@lists.opensuse.org
opensuse-ja@opensuse.org                       users-ja@lists.opensuse.org
opensuse-kde3@opensuse.org                     kde3@lists.opensuse.org
opensuse-kde@opensuse.org                      kde@lists.opensuse.org
opensuse-kde-bugs@opensuse.org                 kde-bugs@lists.opensuse.org
opensuse-kernel@opensuse.org                   kernel@lists.opensuse.org
opensuse-kubic@opensuse.org                    kubic@lists.opensuse.org
opensuse-m17n@opensuse.org                     m17n@lists.opensuse.org
opensuse-marketing@opensuse.org                marketing@lists.opensuse.org
opensuse-offtopic@opensuse.org                 offtopic@lists.opensuse.org
opensuse-openstack@opensuse.org                openstack@lists.opensuse.org
opensuse-optimize@opensuse.org                 optimize@lists.opensuse.org
opensuse-packaging@opensuse.org                packaging@lists.opensuse.org
opensuse-ppc@opensuse.org                      ppc@lists.opensuse.org
opensuse-programming@opensuse.org              programming@lists.opensuse.org
opensuse-project@opensuse.org                  project@lists.opensuse.org
opensuse-proofreading@opensuse.org             proofreading@lists.opensuse.org
opensuse-pt@opensuse.org                       users-pt@lists.opensuse.org
opensuse-python@opensuse.org                   python@lists.opensuse.org
opensuse-releaseteam@opensuse.org              releaseteam@lists.opensuse.org
opensuse-ruby@opensuse.org                     ruby@lists.opensuse.org
opensuse-science@opensuse.org                  science@lists.opensuse.org
opensuse-security@opensuse.org                 security@lists.opensuse.org
opensuse-security-announce@opensuse.org        security-announce@lists.opensuse.org
opensuse-sl@opensuse.org                       users-sl@lists.opensuse.org
opensuse-summit@opensuse.org                   summit@lists.opensuse.org
opensuse-support@opensuse.org                  support@lists.opensuse.org
opensuse-test@opensuse.org                     test@lists.opensuse.org
opensuse-translation@opensuse.org              translation@lists.opensuse.org
opensuse-translation-commit@opensuse.org       translation-commit@lists.opensuse.org
opensuse-translation-de@opensuse.org           translation-de@lists.opensuse.org
opensuse-translation-el@opensuse.org           translation-el@lists.opensuse.org
opensuse-translation-es@opensuse.org           translation-es@lists.opensuse.org
opensuse-translation-fr@opensuse.org           translation-fr@lists.opensuse.org
opensuse-updates@opensuse.org                  updates@lists.opensuse.org
opensuse-virtual@opensuse.org                  virtual@lists.opensuse.org
opensuse-web@opensuse.org                      web@lists.opensuse.org
opensuse-web-de@opensuse.org                   web-de@lists.opensuse.org
opensuse-wiki@opensuse.org                     wiki@lists.opensuse.org
opensuse-xfce@opensuse.org                     xfce@lists.opensuse.org
opensuse-zsystems@opensuse.org                 zsystems@lists.opensuse.org
press@opensuse.org                             press@lists.opensuse.org
release@opensuse.org                           release@lists.opensuse.org
review@opensuse.org                            review@lists.opensuse.org
shop@opensuse.org                              shop@lists.opensuse.org
sourcedvd@opensuse.org                         sourcedvd@lists.opensuse.org
studio-express@opensuse.org                    studio-express@lists.opensuse.org
travel-support@opensuse.org                    travel-support@lists.opensuse.org
tweet-master@opensuse.org                      tweet-master@lists.opensuse.org
ulp-devel@opensuse.org                         ulp-devel@lists.opensuse.org
uyuni-announce@opensuse.org                    announce@lists.uyuni-project.org
uyuni-devel@opensuse.org                       devel@lists.uyuni-project.org
uyuni-users@opensuse.org                       users@lists.uyuni-project.org
uyuni-translation@opensuse.org                 translation@lists.uyuni-project.org
vagrant-admins@opensuse.org                    vagrant-admins@lists.opensuse.org
video@opensuse.org                             video@lists.opensuse.org
weblate-admin@opensuse.org                     weblate-admin@lists.opensuse.org
wicked@opensuse.org                            wicked@lists.opensuse.org
wicked-commit@opensuse.org                     wicked-commit@lists.opensuse.org
wicked-devel@opensuse.org                      wicked-devel@lists.opensuse.org
yast-bugs@opensuse.org                         yast-bugs@lists.opensuse.org
yast-commit@opensuse.org                       yast-commit@lists.opensuse.org
yast-community@opensuse.org                    yast-community@lists.opensuse.org
yast-devel@opensuse.org                        yast-devel@lists.opensuse.org
zypp-commit@opensuse.org                       zypp-commit@lists.opensuse.org
zypp-devel@opensuse.org                        zypp-devel@lists.opensuse.org

This far only the uyuni guys decided to move over to their domain (as visible above), adrianS from OBS team stated they aren't currently interested

For the external $domains:

  • DNS entry: A and AAAA with the IP of proxy.o.o
  • MX DNS entries (mx*.o.o)
  • haproxy config
  • postfix config to ensure $domain mails get routed to mailman3
Actions
Copy link
 #1

Updated by cboltz over 4 years ago

  • Private changed from Yes to No

I see only one export and import of subscribers, but we have "normal" and "nomail" subscribers. I guess you missed one of these groups ;-)

Actions
Copy link
 #2

Updated by pjessen over 4 years ago

hellcp wrote:

on baloo with an account allowed to read subscriber lists

mkdir -p /tmp/transition
ls /var/spool/mlmmj/ > /tmp/transition/mailing-lists
for i in $(cat /tmp/transition/mailing-lists); do
  for j in /var/spool/mlmmj/$i/archive/*; do
    cat /var/spool/mlmmj/$i/archive/$j >> /tmp/transition/$i.mbox
  done

FYI, we don't use the archives in mlmmj, it is being run by mhonarc and they are stored under /srv/www/htdocs.

mlmmj-list -L /var/spool/$i > /tmp/transition/$i.subscribers
done

Don't forget the nomail users.

We should also look at the control/ settings - not all the lists are the same. There will be different owners, moderators etc.

Having it all scripted is cool - makes it easier to rehearse :-)

Actions
Copy link
 #3

Updated by hellcp over 4 years ago

pjessen wrote:

hellcp wrote:

on baloo with an account allowed to read subscriber lists

mkdir -p /tmp/transition
ls /var/spool/mlmmj/ > /tmp/transition/mailing-lists
for i in $(cat /tmp/transition/mailing-lists); do
  for j in /var/spool/mlmmj/$i/archive/*; do
    cat /var/spool/mlmmj/$i/archive/$j >> /tmp/transition/$i.mbox
  done

FYI, we don't use the archives in mlmmj, it is being run by mhonarc and they are stored under /srv/www/htdocs.

Actually, that will only work for public archives, I want to migrate both private and public archives so the users permitted to see the private archives can also check them out in hyperkitty. I might just make a check to do both though, private archives from mlmmj and public from mhonarc

cboltz wrote:

I see only one export and import of subscribers, but we have "normal" and "nomail" subscribers. I guess you missed one of these groups ;-)

pjessen wrote:

mlmmj-list -L /var/spool/$i > /tmp/transition/$i.subscribers
done

Don't forget the nomail users.

We should also look at the control/ settings - not all the lists are the same. There will be different owners, moderators etc.

I don't know if mailman allows to import owners, moderators and nomail users, this far I didn't find anything. It's not much of an issue to import owners and mods manually, when creating the mailing lists, but I don't know an easy way to set multiple users to nomail

Actions
Copy link
 #4

Updated by pjessen over 4 years ago

hellcp wrote:

FYI, we don't use the archives in mlmmj, it is being run by mhonarc and they are stored under /srv/www/htdocs.

Actually, that will only work for public archives, I want to migrate both private and public archives so the users permitted to see the private archives can also check them out in hyperkitty. I might just make a check to do both though, private archives from mlmmj and public from mhonarc

There is no such private/public concept - the mlmmj archives are for retrieving messages by number, but Henne stopped that functionality, at least for most of the lists. I see that some lists still use the archive/ directory, probably because the default setup does not include control/noarchive :-)
Whatever is found in archive/ will also be in the web-browseable archives.

Actions
Copy link
 #5

Updated by hellcp over 4 years ago

  • Description updated (diff)

Update the scripts to migrate moderators, owners and nomail

Actions
Copy link
 #6

Updated by hellcp over 4 years ago

  • Description updated (diff)

Get public archives first before the mlmmj archives

Actions
Copy link
 #7

Updated by hellcp over 4 years ago

  • Description updated (diff)

Add the list of email address mappings

Actions
Copy link
 #8

Updated by hellcp over 4 years ago

  • Description updated (diff)

Add since argument to hyperkitty_import command invocation, to import posts older than the posts currently archived by hyperkitty

Actions
Copy link
 #9

Updated by hellcp over 4 years ago

  • Parent task set to #19900
Actions
Copy link
 #10

Updated by hellcp over 4 years ago

  • Description updated (diff)

Add uyuni translation

Actions
Copy link
 #11

Updated by hellcp over 4 years ago

  • Description updated (diff)

Update the migration scripts

Actions
Copy link
 #12

Updated by hellcp about 4 years ago

This is continued in post mortem issue #77701

Actions
Copy link
 #13

Updated by hellcp about 4 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF