tickets #3704
closedDefective DNS infrastructure for login.opensuse.org / build.opensuse.org
0%
Description
Hi SUSE Admins,
Can someone please take a look at the DNS resolver which provides
service to the OpenSUSE Build Service (OBS)? In particular you need to
check the systems which can originate traffic from the IP address with
the reverse DNS of "login.opensuse.org".
Git traffic from this address was erroneously directed towards the
single US based node of the KDE Anongit network (anongit.kde.org) due
to this defect in your DNS infrastructure. It should have been
directed towards our EU based nodes.
This pattern normally occurs when you are using Google Public DNS
(8.8.8.8), or have a tunnel setup which results in your traffic
appearing to originate in the Americas.
In this instance, the direction of traffic towards a single node
resulted in an effective denial of service attack - as they were
attempting to clone (or otherwise access) many large repositories
simultaneously.
This resulted in everyone in the Americas, Oceania and Asia being
prevented from accessing KDE Git repositories.
I'd therefore also appreciate it if someone could implement throttling
on your side to only fetch one Git repository at a time, particularly
as your systems should be going off reference tarballs which can be
easily obtained from the download.kde.org mirror network.
Regards,
Ben Cooksley
KDE Sysadmin
Updated by Anonymous over 9 years ago
- Private changed from Yes to No
It is probably the IP that our upstream DNS from the company network got.
I will talk to the opensuse-kde team who is pulling snapshots there via git. The files are probably pulled via a source service for snapshot builds.
Updated by Anonymous over 9 years ago
- Category set to OBS
- Assignee set to Anonymous
Updated by luca_b over 9 years ago
bcooksley@kde.org wrote:
Hello Ben,
as your systems should be going off reference tarballs which can be
easily obtained from the download.kde.org mirror network.
This was an unfortunate issue and we apologize for that. What happened is that some OBS repositories offer snapshots pulled off directly from Git. Normally this isn't a problem as the packages are never updated all at once. Unfortunately there was a recent update of all the packages of one repository which triggered this storm of connections over to anongit.kde.org, and caused this issue.
We've noted this to ensure we won't distrupt the normal operations of the KDE anongit mirrors. In particular, for 4.x snapshots, we'll move to tarballs from the Git server.
Updated by Ben_Cook2 over 9 years ago
I wasn't aware this was for snapshots - that would make sense then. I'm aware that other distributions sometimes pull the sources directly from Git to verify tarballs - so thought that could have been the case here.
With regards to the tarballs - they're regenerated once per week.
@darix: where is the "corporate network" based? If it originates traffic from the US, that would explain this. It is also extremely hazardous to KDE infrastructure, so I ask that you partition your DNS infrastructure to ensure European systems handle DNS separately - and thus get pointed to the right systems.
You'll also get performance benefits, as quite a bit of our infrastructure is a very short hop from your systems, rather than being across the Atlantic due to the DNS being wrong.