11:10 a.m.
I think that, before we can make a good decision regarding migration, we
should look at the resources we already have available to us. We have a
dedicated server in Finland with 32GB RAM and a fair amount of disk space
available. We have two dedicated servers generously provided for our use in
Pennsylvania, albeit one earmarked for buildbot service and the other as
yet provisioned (by us) for service.
In light of that, I want to take the server list below in reverse order.
A. Wilcox wrote:
The current systems we run on Integricloud are:
enfys (postgresql) 768 MB RAM 30 GB disk
rarity (these mailing lists) 1536 MB RAM 30 GB disk
mirrormaster 256 MB RAM 1 TB disk
bts (Bugzilla issue tracking) 512 MB RAM 8 GB disk
athdheise (Web server/proxy) 256 MB RAM 4 GB disk
wiki 512 MB RAM 8 GB disk
annwyn (Nextcloud) 512 MB RAM 100 GB disk
chatterbox (Quassel IRC) 512 MB RAM 40 GB disk
At the moment, both chatterbox and annwyn are personal resources. Leaving
aside the discussion as to whether they belong on project infrastructure,
they should be migrated to personal infrastructure unless they are intended
to be made more widely available to Adélie contributors (even if only to
the core and/or infrastructure teams).
athdheise only exists because IntegriCloud was not able to provide IPv4
addresses at a price we were able to pay. It would be retired regardless.
My understanding is that we were planning on retiring the wiki. This would
be an excellent time to do so.
I think that bts should be retired and merged with gitlab, or alternatively
it can be on the same server if retiring it is contra-indicated (e.g. due
to gitlab being unable to provide bug-tracking without a git repo
associated).
mirrormaster would need to be migrated to the Finland server regardless,
since no VPS provider provides block storage in the quantities we need at a
rate we can live with.
The mailing lists should be on hosting separate from our other
infrastructure, since it can and should be usable regardless of the rest of
our infrastructure's dispositions.
That leaves the postgresql server, which should be co-located with gitlab.
I understand that one of our goals is for our infrastructure to not be
subject to architectural issues with x86_64. In principle I agree, but
migrating the majority of our infrastructure from VPSes on a single
dedicated server to VPSes on an unknown number of servers, especially in
today's security environment, carries more risk than ensuring our
infrastructure sits on hardware we know is used only by us.
8:52 p.m.
New subject: Proposal: Replacing Integricloud with Scaleway
On 07/13/19 10:10, Kiyoshi Aman wrote:
I think that, before we can make a good decision regarding migration,
we
should look at the resources we already have available to us. We have a
dedicated server in Finland with 32GB RAM and a fair amount of disk
space available. We have two dedicated servers generously provided for
our use in Pennsylvania, albeit one earmarked for buildbot service and
the other as yet provisioned (by us) for service.
There are a number of reasons that I didn't want to consider running
everything on the Finland server, nor the Pennsylvania one.
1. They are sponsored, i.e. they are not paid directly by us. From
experience, they don't last forever. See what happened with Rack911.
Do I think either of them would do that to us? Probably not. However,
we don't know what the future will hold, and we have no control over the
futures of orgs that aren't us.
2. Putting all of our eggs in one Integricloud basket has already proven
to be a TERRIBLE idea. If we pile all-in to the Leuhta or ionfish
boxes, we're making the same mistake. (Moving GitLab to Finland and
moving its SQL with it was one attempt at making that component
independent from all the others.)
3. Hetzner specifically has garbage-tier ratings for DNSBL / running
MXes. Our mailing lists would be classed as spam.
In light of that, I want to take the server list below in reverse
order.
A. Wilcox wrote:
> The current systems we run on Integricloud are:
>
> enfys (postgresql) 768 MB RAM 30 GB disk
>
> rarity (these mailing lists) 1536 MB RAM 30 GB disk
>
> mirrormaster 256 MB RAM 1 TB disk
>
> bts (Bugzilla issue tracking) 512 MB RAM 8 GB disk
>
> athdheise (Web server/proxy) 256 MB RAM 4 GB disk
>
> wiki 512 MB RAM 8 GB disk
>
> annwyn (Nextcloud) 512 MB RAM 100 GB disk
>
> chatterbox (Quassel IRC) 512 MB RAM 40 GB disk
At the moment, both chatterbox and annwyn are personal resources.
Leaving aside the discussion as to whether they belong on project
infrastructure, they should be migrated to personal infrastructure
unless they are intended to be made more widely available to Adélie
contributors (even if only to the core and/or infrastructure teams).
Yes, the intention was opening up Quassel to other project members. I
was piloting Nextcloud for Adélie on annwyn and I wanted to run one for
all contributors to rectify our current reliance on i.e. bpaste and
imgur. I haven't finalised how this will look yet, hence why there is
no proposal on -project yet :)
athdheise only exists because IntegriCloud was not able to provide
IPv4
addresses at a price we were able to pay. It would be retired regardless.
athdheise is our main web server (www., help., support.) in addition to
being the proxy. It won't be retired. It just won't be overtaxed.
My understanding is that we were planning on retiring the wiki. This
would be an excellent time to do so.
I didn't want to delay the migration off Integricloud until we can save
all the content of the wiki to other places. I wanted to focus on
"getting off of a 250$ per month money sink that is also going down a
lot" before focusing on the process improvements around deprecating the
wiki.
Deprecating the wiki also means we need to set up the GitLab page
runners, which is blocking on me actually setting the thing up because
I'm too busy dealing with Integricloud outages.
Migrating the wiki would take less than three minutes of time (add a
pg_hba auth line, then `scp -pr /srv/www new-wiki:`). It would probably
take a few days to a week before we could retire it.
I think that bts should be retired and merged with gitlab, or
alternatively it can be on the same server if retiring it is
contra-indicated (e.g. due to gitlab being unable to provide
bug-tracking without a git repo associated).
Retiring bts is a hard NAK from me. I can't stand the workflow nor the
UI that GitLab Issues provides.
mirrormaster would need to be migrated to the Finland server
regardless,
since no VPS provider provides block storage in the quantities we need
at a rate we can live with.
Agreed.
The mailing lists should be on hosting separate from our other
infrastructure, since it can and should be usable regardless of the rest
of our infrastructure's dispositions.
It was originally on Vultr, though I'd rather prefer not to do that
again. Not only is the cost high, but enabling MX is a per-VM thing, so
I'd have to get it approved again, which would delay the move. Also,
it's x86 hardware.
That leaves the postgresql server, which should be co-located with
gitlab.
No. Noooooooooooooooooooooooooooooooooooo. No. We are not putting the
PostgreSQL server in Finland. I'm barely comfortable with it going in
Norway for GDPR reasons, but Finland specifically has DRACONIAN
regulations on handling PII, which includes the registrations for bts
and wiki.
I understand that one of our goals is for our infrastructure to not
be
subject to architectural issues with x86_64. In principle I agree, but
migrating the majority of our infrastructure from VPSes on a single
dedicated server to VPSes on an unknown number of servers, especially in
today's security environment, carries more risk than ensuring our
infrastructure sits on hardware we know is used only by us.
I'm not convinced about this.
I would trust an ARM KVM much more than an x86 KVM, especially given
what I know about how both architectures treat hypervisor and VM
separation. And given what I know about how easy it is to exfiltrate
data from an x86 KVM from another guest, vs how hard it is to do the
same on ARM.
References can be provided if desired, but right now I have three more
messages to read and reply to.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
11:58 a.m.
New subject: Proposal: Replacing Integricloud with Scaleway
Hi *,
I strongly agree with Aerdan here. In my opinion, the risks of moving to
VPSes on hardware shared with other tenants outweight all (perceived or
real) benefits of using aarch64 instead of x86.
Furthermore, I believe our current x86 infra has more than enough
capacity to accomodate all services we require, so forgoing the use of
scw aarch64 vpses would save us money too.
However, I am in favour of migrating away from Integricloud regardless
of the destination to which we'd migrate, be it aarch64 vpses, our
already existing x86 infra, colo'ed x86 or ppc servers, or a cluster of
raspis in someone's basement.
The idea of Integricloud is neat, but their unwillingness to set up a
reliable, redundant uplink even after several outages that could've been
easily avoided, combined with their completely ludicrous prices, makes
them an extremly unattractive option in practice.
Cheers,
Luis
12:28 p.m.
New subject: Proposal: Replacing Integricloud with Scaleway
Apologies if this email is malformed, I had to send it from my phone.
On Jul 13, 2019, at 11:10 AM, Kiyoshi Aman <kiyoshi.aman(a)gmail.com> wrote:
My understanding is that we were planning on retiring the wiki. This
would be an excellent time to do so.
Agreed. We are already moving some stuff to GitLab wikis and that seems appropriate. For
other cases we should move forward on previous proposals to provision project-specific
static websites (perhaps via GitLab static site generation).
I think that bts should be retired and merged with gitlab, or
alternatively it can be on the same server if retiring it is contra-indicated (e.g. due to
gitlab being unable to provide bug-tracking without a git repo associated).
No. GitLab issues are not sufficient for the needs of proper bug tracking.
The mailing lists should be on hosting separate from our other
infrastructure, since it can and should be usable regardless of the rest of our
infrastructure's dispositions.
This is an important point and I agree.
Max
8:57 p.m.
New subject: Proposal: Replacing Integricloud with Scaleway
On 07/13/19 10:58, Luis Ressel wrote:
Hi *,
I strongly agree with Aerdan here. In my opinion, the risks of moving to
VPSes on hardware shared with other tenants outweight all (perceived or
real) benefits of using aarch64 instead of x86.
I suppose I can look into more dedicated hosting. Honestly, at our
level of project income, I wasn't even considering it.
Furthermore, I believe our current x86 infra has more than enough
capacity to accomodate all services we require, so forgoing the use of
scw aarch64 vpses would save us money too.
See my response to Kiyoshi for why I feel this is a Very Bad Idea™.
However, I am in favour of migrating away from Integricloud
regardless
of the destination to which we'd migrate, be it aarch64 vpses, our
already existing x86 infra, colo'ed x86 or ppc servers, or a cluster of
raspis in someone's basement.
I do have a static /27 on AT&T, if everyone can stomach 5 Mbit/s egress.
(That's not a typo, I have 30 down and 5 up at my office until the
bonded pair is healed. ETA: whenever AT&T gets their fingers out of
their ---)
The idea of Integricloud is neat, but their unwillingness to set up
a
reliable, redundant uplink even after several outages that could've been
easily avoided, combined with their completely ludicrous prices, makes
them an extremly unattractive option in practice.
+1.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
9 p.m.
New subject: Proposal: Replacing Integricloud with Scaleway
On 07/13/19 11:28, Max Rees wrote:
> My understanding is that we were planning on retiring the wiki.
This
> would be an excellent time to do so.
Agreed. We are already moving some stuff to GitLab wikis and that seems
appropriate. For other cases we should move forward on previous
proposals to provision project-specific static websites (perhaps via
GitLab static site generation).
As noted, it's quicker to just migrate them and move them off after.
Unless someone else wants to pay for an August at Integricloud :/
> I think that bts should be retired and merged with gitlab, or
> alternatively it can be on the same server if retiring it is
> contra-indicated (e.g. due to gitlab being unable to provide
> bug-tracking without a git repo associated).
No. GitLab issues are not sufficient for the needs of proper bug tracking.
Heavily +1.
> The mailing lists should be on hosting separate from our other
> infrastructure, since it can and should be usable regardless of the
> rest of our infrastructure's dispositions.
This is an important point and I agree.
Consensus seems to be building on this point.
I will look in to alternative hosting for the mailing lists, and perhaps
write a (much smaller!) proposal with the options I find.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
667
days inactive
668
days old
adelie-project@lists.adelielinux.org
5 comments
4 participants
participants (4)
-
A. Wilcox -
Kiyoshi Aman -
Luis Ressel -
Max Rees