4:46 p.m.
Hello all,
During some brief free time today I was thinking about how to go about
implementing some of the upcoming features that I plan to add to abuildd in
the next few weeks. Specifically, I am concerned about the handling of
dependencies.
My current proposal is that each new push to a branch ("push event") or update
to a merge request ("MR event") will create a new "job". The job can
be
composed of one or more "tasks" which would be building a package on a
specific architecture. As an example, if a push event is received that changed
two APKBUILDs, a job would be added that would include tasks to build those
two packages on all applicable architectures.
I do not expect there to be any problems with handling dependencies between
tasks in a single job (intra-job dependencies). However, dependencies between
multiple jobs (inter-job dependencies) is more complex, at least the way I see
it now.
For push events, it could be easy to handle inter-job dependencies as long as
the artifacts for each job are collected and kept for that branch. It is MR
events that I am more concerned about.
I believe that as it stands right now (I cannot check it at the moment), my
abuildd-enqueue implementation does not distinguish between new commits added
to an existing MR, a force push over an existing MR, or a completely new MR.
Thus all added/changed (NB: *not* deleted) packages for a MR would be rebuilt
on each change to the MR.
I think it is possible to implement it such that we keep state for each MR and
thus only perform builds as necessary, but I'd like to hear all of your
thoughts on this.
If there are other questions or considerations you'd like to make, please let
me know. Another feature I thought of during the same period today is that it
should be possible to prioritize jobs based on type (push, MR) and branch, if
applicable. Things such as being able to skip a build that would be caused by a
particular commit (I have seen things like having "[ci-skip]" in the commit
title before) are more difficult in the present implementation, but I'm sure it
can be done at some point.
Max
5:01 p.m.
On 05/09/18 21:46, Max Rees wrote:
Hello all,
During some brief free time today I was thinking about how to go about
implementing some of the upcoming features that I plan to add to abuildd in
the next few weeks. Specifically, I am concerned about the handling of
dependencies.
My current proposal is that each new push to a branch ("push event") or update
to a merge request ("MR event") will create a new "job". The job can
be
composed of one or more "tasks" which would be building a package on a
specific architecture. As an example, if a push event is received that changed
two APKBUILDs, a job would be added that would include tasks to build those
two packages on all applicable architectures.
I do not expect there to be any problems with handling dependencies between
tasks in a single job (intra-job dependencies). However, dependencies between
multiple jobs (inter-job dependencies) is more complex, at least the way I see
it now.
For push events, it could be easy to handle inter-job dependencies as long as
the artifacts for each job are collected and kept for that branch. It is MR
events that I am more concerned about.
I believe that as it stands right now (I cannot check it at the moment), my
abuildd-enqueue implementation does not distinguish between new commits added
to an existing MR, a force push over an existing MR, or a completely new MR.
Thus all added/changed (NB: *not* deleted) packages for a MR would be rebuilt
on each change to the MR.
I think it is possible to implement it such that we keep state for each MR and
thus only perform builds as necessary, but I'd like to hear all of your
thoughts on this.
If there are other questions or considerations you'd like to make, please let
me know. Another feature I thought of during the same period today is that it
should be possible to prioritize jobs based on type (push, MR) and branch, if
applicable. Things such as being able to skip a build that would be caused by a
particular commit (I have seen things like having "[ci-skip]" in the commit
title before) are more difficult in the present implementation, but I'm sure it
can be done at some point.
Max
_______________________________________________
Adélie Development mailing list -- adelie-devel(a)lists.adelielinux.org
To unsubscribe send an email to adelie-devel-leave(a)lists.adelielinux.org
Are you
using a custom Message Broker system, or something that would
integrate with other tools, eg RabbitMQ, etc?
Might be good for 'portability' and interaction with other tooling perhaps.
Just a thought,
Michael/veremitz.
5:10 p.m.
On Wed, Sep 5, 2018 at 5:05 PM M. J. Everitt <m.j.everitt(a)iee.org> wrote:
Are you using a custom Message Broker system, or something that
would
integrate with other tools, eg RabbitMQ, etc?
Might be good for 'portability' and interaction with other tooling perhaps.
The plan is to use MQTT as the communication between the webhook/enqueue
server and the build servers. The build servers will subscribe to a MQTT topic
for their architecture, and the enqueue server will post new jobs/tasks to those
topics as necessary (this is actually the next step in development so I have not
worked out the details yet). When the build server finishes a build - success or
failure, it will post as such to a "completion" topic for its architecture.
I think another thing that needs fleshing out in this area is how
multiple build servers
of the same architecture will be handled - perhaps a build server can
post a message
so that it can "claim" a task before the other same-architecture build
servers get to it.
Max
10 p.m.
On 09/05/18 16:10, Max Rees wrote:
I think another thing that needs fleshing out in this area is how
multiple build servers of the same architecture will be handled -
perhaps a build server can post a message so that it can "claim" a
task before the other same-architecture build servers get to it.
Max
My suggestion would be to have the agent daemon be 'smart enough' to know:
- that a build server is available
- that all build servers are busy
- that a build server has become available
If "busy" and "available" are MQTT messages, then I would assume that
this would not be difficult.
In this design, the agent would be the one that chooses which server to
use for an architecture, and send the job directly to the server which
is first available.
There could likely be an operator-maintained order of servers to choose
when all or multiple are available (sorted by CPU core count or so on).
I am not in any way attached to this proposal; better ones are welcome.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
1:07 a.m.
On Sep 05 09:00 PM, A. Wilcox wrote:
On 09/05/18 16:10, Max Rees wrote:
> I think another thing that needs fleshing out in this area is how
> multiple build servers of the same architecture will be handled -
> perhaps a build server can post a message so that it can "claim" a
> task before the other same-architecture build servers get to it.
>
> Max
My suggestion would be to have the agent daemon be 'smart enough' to know:
- that a build server is available
- that all build servers are busy
- that a build server has become available
If "busy" and "available" are MQTT messages, then I would assume
that
this would not be difficult.
In this design, the agent would be the one that chooses which server to
use for an architecture, and send the job directly to the server which
is first available.
There could likely be an operator-maintained order of servers to choose
when all or multiple are available (sorted by CPU core count or so on).
I am not in any way attached to this proposal; better ones are welcome.
As a terminology nitpick - the name "agent" is already used by
"abuildd-agentd", which is the process that runs on the build servers
that acts as the liaison between MQTT and abuild/pbuild. What you are
actually referring to would be a new, separate program - right now, the
webhook imports the functions it needs from enqueue in order to add
jobs. In this way, the webhook would have a long-lived MQTT client
connection, and in the future there would also be a standalone
command-line version of enqueue that would use the same code but with a
short MQTT client connection so that the command returns as soon as it's
done. I would like to keep the number of long-lived processes we must
maintain to a small number. As it stands, we need to manage an agentd
process for each build server; the webhook server; the MQTT broker; the
PostgreSQL server; and more in the future (at least collection and
the status webpage unless we choose to integrate these two more tightly
with the other components).
Perhaps this can be done purely using MQTT. We can publish retained
messages (i.e. messages that are received immediately on client
subscription) to topics like "servers/<arch>/<name>" when
abuildd-agentd starts on the build server, with a payload of e.g. number
of cores and availability status (idle or busy; generally idle on
startup). The webhook or enqueue CLI subscribes to "servers/#" (any
topic under servers) on *their* startup and thus receive a list of all
currently available build servers. They can then prioritize which build
server to delegate the task to by publishing to e.g.
"tasks/available/<arch>/<name>/<id>" (which abuildd-agentd
will have
subscribed to) for some build server that is currently idle.
When abuildd-agentd receives a new task, it will change the retained
message on "servers/<arch>/<name>" to {nprocs, busy} and begin the
task.
Once it's done, it'll publish the result of the task somewhere (perhaps
"tasks/done/<id>") with a payload of return code and error message, if
applicable. Then the agent can update "servers/<arch>/<name>" to be
idle
again.
When abuildd-agentd shuts down gracefully, it can publish a {nprocs,
offline} retained message at "servers/<arch>/<name>".
We can also setup what MQTT calls a "last will and testament" for each
build server. This allows the client to tell the broker what to do in
case the client's connection dies without sending a proper DISCONNECT
message. In our case, we would want the last will and testament to be to
post a retained message to the "servers/<arch>/<name>" topic with a
payload of {nprocs, offline}. This way, if the broker detects that the
MQTT client of the build server has unexpectedly disconnected, it will
automatically publish an offline message for it.
What if all applicable build servers are unavailable (busy or offline)
when we want to delegate a new task? For the webhook, this should be
easy to get around since it's built to be asynchronous, so we can just
keep polling the list of servers until something's available. In case of
a webhook crash we have the SQL database as a backup. For the enqueue
CLI we would probably just fail by default and have an option to keep
polling for X time or indefinitely.
This is all based on about 30 minutes of reading MQTT documentation late
at night, so hopefully it all works.
Max
1:53 a.m.
On Sep 06 01:07 AM, Max Rees wrote:
What if all applicable build servers are unavailable (busy or
offline)
when we want to delegate a new task? For the webhook, this should be
easy to get around since it's built to be asynchronous, so we can just
keep polling the list of servers until something's available. In case of
a webhook crash we have the SQL database as a backup. For the enqueue
CLI we would probably just fail by default and have an option to keep
polling for X time or indefinitely.
One thing I'd like to add here: in order to support restarts of the
webhook without losing all of that state, we could initially set the
state of the tasks in the SQL database to be something like "new" which
would then be changed to "building" once it has been properly delegated.
Max
9:55 p.m.
On 09/05/18 15:46, Max Rees wrote:
I believe that as it stands right now (I cannot check it at the
moment), my
abuildd-enqueue implementation does not distinguish between new commits added
to an existing MR, a force push over an existing MR, or a completely new MR.
Thus all added/changed (NB: *not* deleted) packages for a MR would be rebuilt
on each change to the MR.
That is probably for the best, as a MR is going to be a single unit of
work. I can understand that if, say, XFCE or elementary was added as a
MR, there might be a lot of busywork if a single package is changed.
But there could be non-obvious (or subtly non-declared) dependencies
between them all, which is why I think a clean rebuild for every MR
event is not only acceptable, but desireable.
It is, after all, just "machine time" right?
If there are other questions or considerations you'd like to
make, please let
me know.
If a build server goes down for maintenance, is that architecture "lost"
- i.e. will it require manual intervention / building of what was pushed
while that server was down?
Another feature I thought of during the same period today is that it
should be possible to prioritize jobs based on type (push, MR) and branch, if
applicable. Things such as being able to skip a build that would be caused by a
particular commit (I have seen things like having "[ci-skip]" in the commit
title before) are more difficult in the present implementation, but I'm sure it
can be done at some point.
Definitely prioritise 1-STABLE, then current, then feature branches,
then MRs from contributors, then MRs from the general public. At least,
that's how I would put it in a hierarchy.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
10:36 p.m.
On Sep 05 08:55 PM, A. Wilcox wrote:
On 09/05/18 15:46, Max Rees wrote:
> I believe that as it stands right now (I cannot check it at the moment), my
> abuildd-enqueue implementation does not distinguish between new commits added
> to an existing MR, a force push over an existing MR, or a completely new MR.
> Thus all added/changed (NB: *not* deleted) packages for a MR would be rebuilt
> on each change to the MR.
That is probably for the best, as a MR is going to be a single unit of
work. I can understand that if, say, XFCE or elementary was added as a
MR, there might be a lot of busywork if a single package is changed.
But there could be non-obvious (or subtly non-declared) dependencies
between them all, which is why I think a clean rebuild for every MR
event is not only acceptable, but desireable.
It is, after all, just "machine time" right?
This is actually something I had not considered with respect to changing
dependencies between packages, so indeed it would be best to always
rebuild the whole MR. Duly noted.
> If there are other questions or considerations you'd like to
make, please let
> me know.
If a build server goes down for maintenance, is that architecture "lost"
- i.e. will it require manual intervention / building of what was pushed
while that server was down?
I was also thinking about this earlier. I think the current design would
allow for a build server to go down at any time, and once it comes back
up (really, once abuildd-agentd comes back up on the build server) it
can just reconnect to the MQTT broker and receive any jobs that it
missed while offline. Based on my currently limited understanding of
MQTT, this can be accomplished by using persistent sessions and a
QOS >= 1. I guess that begs the question of what happens when the MQTT
broker goes down - I will have to research what the brokers do in such
a case, i.e. whether they save those persistent sessions to disk on
shutdown. Otherwise we would need to add something to the SQL or just
deal with such cases manually.
> Another feature I thought of during the same period today is
that it
> should be possible to prioritize jobs based on type (push, MR) and branch, if
> applicable. Things such as being able to skip a build that would be caused by a
> particular commit (I have seen things like having "[ci-skip]" in the
commit
> title before) are more difficult in the present implementation, but I'm sure it
> can be done at some point.
Definitely prioritise 1-STABLE, then current, then feature branches,
then MRs from contributors, then MRs from the general public. At least,
that's how I would put it in a hierarchy.
For STABLE, will there be corresponding branch(es) for it? I assume
current will correspond to master. In this case we can set priority
levels for each project based on type (push, MR, manual - more on that
later), branch, and user. I currently have a setting that allows the
project to specify which branches it should watch for push events on.
MRs from contributors - how best to implement that? We could start
adding privileged GitLab API calls but I am slightly opposed to that. So
far I have not needed any privileged APIs, just whatever GitLab sends me
in the webhook payload. I already have access control implemented based
on the pushing user so that you can implement allow/deny lists. This can
probably be extended to the MRs along with adding priorities - I doubt
GitLab allows colons in usernames, so we could do something like
username:20 or whatever. Similarly with branches. We could also do
something like glob matching so that you could have "stable/*" with
priority 1, "master" with priority 5, "feature/*" with priority 10,
etc.
Something I also haven't mentioned yet is there is also functionality
already that allows users to manually request builds for a MR by adding
a comment with a keyword to the MR (the default is "[build]"). This has
its own separate access control, but it ultimately generates an MR
event.
Max
978
days inactive
979
days old
adelie-devel@lists.adelielinux.org
7 comments
3 participants
participants (3)
-
A. Wilcox -
M. J. Everitt -
Max Rees