Project

General

Profile

tickets #57104

Helios platform inaccessible

Added by ish.sookun@lasentinelle.mu about 2 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Servers hosted in NBG
Target version:
-
Start date:
Due date:
% Done:

100%

Estimated time:

Description

Dear Heroes,

I am currently getting an "Internal Server Error" message when accessing
elections.opensuse.org.

Grateful if you could investigate.

Regards,

Ish Sookun

(from the Election Committee)

History

#1 Updated by cboltz about 2 years ago

  • Private changed from Yes to No

It looks like someone (I know who...) upgraded the server to Leap 15.1 on 2019-08-29 06:14:04 without telling anybody and without noticing that some helios-related packages don't build for 15.1. Now we have a mix of 42.3 and 15.1 packages, and some interesting[tm] error messages :-(

Hint - next time, before upgrading a server, maybe first ask the person who usually maintains that server (and has a test VM where breakage wouldn't hurt)?

Here's a summary of the problems I found in a quick check.

Helios-related packages are from the infra:elections repo:

# built for 15.1
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_15.1     python-django-secure
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_15.1     python-django-ses
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_15.1     python-django-sslify
# build for 42.3
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_42.3     helios-server
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_42.3     helios-server-uwsgi
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_42.3     python-dj-database-url
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_42.3     python-rollbar
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_42.3     python-validate_email
openSUSE:infrastructure:elections.opensuse.org / openSUSE_Leap_42.3     python-webencodings

Obviously someone[tm] needs to fix the build for these packages (and maybe do a version upgrade?), and then install them on the server.

celeryd refuses to start with this nice error (found in /var/log/messages):

2019-09-19T19:41:28.685525+00:00 sarabi systemd[1]: Started Celery Service.
2019-09-19T19:41:29.146201+00:00 sarabi python[4243]: Traceback (most recent call last):
2019-09-19T19:41:29.146745+00:00 sarabi python[4243]:   File "manage.py", line 10, in <module>
2019-09-19T19:41:29.147121+00:00 sarabi python[4243]:     execute_from_command_line(sys.argv)
2019-09-19T19:41:29.147273+00:00 sarabi python[4243]:   File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
2019-09-19T19:41:29.151245+00:00 sarabi python[4243]:     utility.execute()
2019-09-19T19:41:29.151460+00:00 sarabi python[4243]:   File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute
2019-09-19T19:41:29.151582+00:00 sarabi python[4243]:     django.setup()
2019-09-19T19:41:29.151700+00:00 sarabi python[4243]:   File "/usr/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
2019-09-19T19:41:29.151820+00:00 sarabi python[4243]:     apps.populate(settings.INSTALLED_APPS)
2019-09-19T19:41:29.151936+00:00 sarabi python[4243]:   File "/usr/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
2019-09-19T19:41:29.152060+00:00 sarabi python[4243]:     app_config = AppConfig.create(entry)
2019-09-19T19:41:29.152184+00:00 sarabi python[4243]:   File "/usr/lib/python2.7/site-packages/django/apps/config.py", line 127, in create
2019-09-19T19:41:29.152304+00:00 sarabi python[4243]:     import_module(entry)
2019-09-19T19:41:29.152422+00:00 sarabi python[4243]:   File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
2019-09-19T19:41:29.152540+00:00 sarabi python[4243]:     __import__(name)
2019-09-19T19:41:29.152656+00:00 sarabi python[4243]: ImportError: No module named django
2019-09-19T19:41:29.187190+00:00 sarabi systemd[1]: helios-celeryd.service: Main process exited, code=exited, status=1/FAILURE
2019-09-19T19:41:29.187452+00:00 sarabi systemd[1]: helios-celeryd.service: Unit entered failed state.
2019-09-19T19:41:29.187598+00:00 sarabi systemd[1]: helios-celeryd.service: Failed with result 'exit-code'.

systemctl status helios-server-uwsgi gives me this nice message:

Sep 19 19:48:03 sarabi uwsgi[4461]: Traceback (most recent call last):
Sep 19 19:48:03 sarabi uwsgi[4461]:   File "./wsgi.py", line 5, in <module>
Sep 19 19:48:03 sarabi uwsgi[4461]: application = get_wsgi_application()
Sep 19 19:48:03 sarabi uwsgi[4461]:   File "/usr/lib/python2.7/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
Sep 19 19:48:03 sarabi uwsgi[4461]: django.setup(set_prefix=False)
Sep 19 19:48:03 sarabi uwsgi[4461]:   File "/usr/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
Sep 19 19:48:03 sarabi uwsgi[4461]: apps.populate(settings.INSTALLED_APPS)
Sep 19 19:48:03 sarabi uwsgi[4461]:   File "/usr/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
Sep 19 19:48:03 sarabi uwsgi[4461]: app_config = AppConfig.create(entry)
Sep 19 19:48:03 sarabi uwsgi[4461]:   File "/usr/lib/python2.7/site-packages/django/apps/config.py", line 127, in create
Sep 19 19:48:03 sarabi uwsgi[4461]: import_module(entry)
Sep 19 19:48:03 sarabi uwsgi[4461]:   File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
Sep 19 19:48:03 sarabi uwsgi[4461]: __import__(name)
Sep 19 19:48:03 sarabi uwsgi[4461]: ImportError
Sep 19 19:48:03 sarabi uwsgi[4461]: :
Sep 19 19:48:03 sarabi uwsgi[4461]: No module named django
Sep 19 19:48:03 sarabi uwsgi[4461]: unable to load app 0 (mountpoint='') (callable not found or import error)

For the records: This server is completely setup with salt, and it should stay that way. Therefore please don't do any manual changes.

#2 Updated by mcaj about 2 years ago

  • Assignee set to mcaj

#4 Updated by mcaj about 2 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10

Hi,
We had to update that machine because it run leap 42.3 os.
I had no idea how "up to date" the package helios-server we have.
The leap 42.3 is out of updates and any running machine in DMZ with this os is not secure from my point of view.

Anyway I checked the package and openSUSE:infrastructure:elections.opensuse.org/helios-server was update a year ago :(
I tried to fix builds for leap 15.1 but so far I have a problems with old python dependency packages ...
see the progress here: https://build.opensuse.org/project/show/home:mcaj:branches:openSUSE:infrastructure:elections.opensuse.org

Martin

#5 Updated by cboltz about 2 years ago

I downgraded the elections.o.o VM to Leap 42.3 so that we can run the name vote next week. (Please run a test election to make sure it really works.)

However, this is only a temporary solution, and everybody hates it. We'll need to either fix Helios so that it works with Leap 15.1, or to find and setup a replacement.

#6 Updated by lrupp almost 2 years ago

  • Category set to Servers hosted in NBG
  • Status changed from In Progress to Closed
  • % Done changed from 10 to 100

The original problem is fixed since a while and work has been done to migrate the machine finally.

=> Closing here, to get rid of some old.tickets.

Also available in: Atom PDF