x86_64 builder - catastrophic disk failure
by A. Wilcox
Hi,
The x86_64 builder (ciall.foxkit.us) has had a catastrophic disk failure
at around 23:44 CST 18 February 2018. During attempted recovery, there
was a power failure in the Tulsa metro at 04:14:47 CST 19 February 2018
that lasted over two hours; regrettably, much longer than the 45 minute
backup power runtime. Due to these events occuring so close together,
the entire /srv (package staging area), /usr/src (aports git trees that
had a few uncommitted patches), and buildroot / (where abuild
configuration lived) trees are completely unrecoverable.
The /opt/virtual tree, where the Adélie Portability Virtual Machines
lived (including NetBSD-based bryan.foxkit.us, OpenBSD-based
georgie.foxkit.us, and Arch-based fran.foxkit.us), may be recoverable -
it is not yet known at this time.
Anyone who had any data on bryan, georgie, or fran should consider that
data gone barring an exceptional recovery effort. As their primary
purpose was for portability testing only, no backups were ever taken of
their virtual disks.
The /home tree on ciall was in the process of its monthly back up at the
time of the disk failure and it appears only a few files (from my own ~)
were lost. No other user data was lost.
As x86_64 is a tier 1 architecture, and the Platform Team is not
currently considering deprecating it, this unexpected, catastrophic,
poorly-timed sequence of failures may cause the alpha5 release date to
slip again. This is regretful and unfortunate. Moving forward, we will
need to discuss if architecture buildroots and configurations should be
stored in Git; if a Git repository isn't deemed a proper solution, they
will at least be added to the monthly back up jobs.
Sincerely,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months
Roadmap to alpha5
by A. Wilcox
Hi,
I've recently completely overhauled our roadmap pages on the wiki and
there is a specific page for alpha5:
https://wiki.adelielinux.org/wiki/Project:Roadmap/1.0-ALPHA5
Since there has been some confusion around all this, I'm going to
clarify the best I can now.
Packaging:
* All builders are in frozen state until the adelie-integration branch
is merged into Alpine's aports.git. This is to ensure we don't mix up
package origins. Once the merge is complete, the builders will be
blanked and mass-rebuild for alpha5 using our system/ as the primary
repo and Alpine aports.git as a secondary repo, instead of a fork.
* As such, system/ is also frozen until the merge is completed, with the
exception of moving packages from main/ in preparation for the 'thaw'.
This is the reason why easy-kernel mc4 is not yet shipping on any
architecture, and why easy-kernel mc3 has not shipped on x86_64.
* This also means while user/ is not frozen on paper, no package changes
will be built until the builders are unfrozen, so there is no work
happening in user/ either.
* Once the merge is complete, we will need to review the packages that
have not had any discussion yet on the MLs (of which there are 18). We
will also need to discuss with Alpine maintainers about the "terrible
tests" category of packages that we did not merge, and whether they are
interested in shipping them with tests or if we need to take ownership.
* Once all of that is done, we need to focus on GStreamer and whether we
need to ship our own GStreamer packages.
* Then we can focus on the Python 3 packages I noted in aports mail 6.
* That will leave us with four packages with ppc64 (BE) fixes,
gnome-doc-utils and pax-utils that we need to fix on our side, and
libgsf, harfbuzz, and lame, which have outstanding questions for Alpine
to answer.
* Once all of the above points are done, the builders will unfreeze and
we will never have to deal with this stuff again.
KDE:
* KF 5.43 is out, but I don't feel comfortable bumping 70 packages when
none of them will be built until a week (or more) later.
* Plasma 5.12 LTS is out. Bug fix 2 (5.12.2) will be released on
Tuesday, the 20th of February.[1] This gives us plenty of time to test
it and back out to 5.8 if it is not yet stable. If we have the time
after the Alpine merge is finished, I feel we should consider bumping in
this release.
* Education packages should be easy once builders are unfrozen.
Documentation:
* The Developer Guide[2] is nearing 50% completion. Once the abuild man
page PR[3] is finished, it should be fairly easy for me to write chapter
4. After some initial outlines for the other guides, this will mean our
documentation targets will be met. I don't foresee any delays here.
If anyone has any questions, please feel free to ask.
Best,
--arw
[1]: https://community.kde.org/Schedules/Plasma_5
[2]: https://code.foxkit.us/adelie/docs/tree/master/src/devel
[3]: https://github.com/alpinelinux/abuild/pull/33
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months
Adding cmd:which (or similar) to build-tools
by A. Wilcox
Hi,
At least main/i3lock and main/libvpx (and likely more) in Alpine
aports.git depend on `which`. This is provided by BusyBox on Alpine
systems, but is not provided in adelie-base because it is not POSIX and
is mostly useless bloat.
I am, however, considering adding cmd:which as a dependency for
build-tools. I'd like to set debianutils-which as the primary provider
since their utility is light (just a wrapper for POSIX command(1))
compared to the full-blown which (~ 100 KB).
The only alternative that I can think of is maintaining i3lock and
libvpx in user or system and adding cmd:which to their makedepends. I
am open to further alternatives.
Let's discuss.
Best,
--arw
--
A. Wilcox (awilfox)
Open-source programmer (C, C++, Python)
https://code.foxkit.us/u/awilfox/
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months
Seven and the terrible tests
by A. Wilcox
Hi there (xpost to both Adélie and Alpine dev lists),
There are seven packages that we (Adélie) ship with tests enabled. We
do not know if Alpine would be able to use them, however, due to some
really bizarre problems we've encountered:
* main/attr
The test suite's Perl uses an outdated form of regular expression that
cause modern Perl to crash. There are also assumptions about the libc
that do not hold in musl. For this reason, we have two very large
patches to make the tests pass. I am not sure if these are acceptable.
* main/cmake
The test suite was disabled on 21 September with a commit log message
"disable testsuite, blocking builders on multiple archs". We haven't
had any issues with it on x86, x86_64, ppc, or ppc64. Are there any
logs about what happened? I would be willing to dig in.
* main/kyua
The test suite requires the builders to have a sysctl set:
kernel.core_pattern = core.%p
Otherwise, the test fails because 'core' is overwritten during the test
of handling core dumps.
* main/openldap
The test suite takes about two hours on our builders. The fastest was
ppc64, 1h48m; next was x86_64, 2h3m; next was ppc at 2h4m; last was x86
at 2h41m.
It does pass.
* main/parted
We ported the test suite to Python 3, rewriting large parts of it in the
process. We have not started a conversation upstream to determine if
this will can be merged or not.
* main/tzdata
The tzdata package needs 'sp', a SGML validator from the mid 90s. I
spent a full day making it compile with a modern compiler. Since it
needs the w3.org DTDs as well, I had to override www.w3.org in
/etc/hosts; they throttle requests to take 5 minutes due to lots of
spurious bot traffic trying to download the DTD files.
* main/zsh
The comment on Alpine's side is that "As of 5.4.2 - 3 tests fails". We
have four tests failing, and we work around that by disabling them:
+ # Does not work with musl due to UTF-8
+ rm "$builddir"/Test/A03quoting.ztst
+ # Does not work with musl due to locale
+ rm "$builddir"/Test/B03print.ztst
+ # Not guaranteed to work portably (requires atime)
+ rm "$builddir"/Test/C02cond.ztst
+ # PowerPC-specific failure
+ rm "$builddir"/Test/V09datetime.ztst
make test
We will maintain our own copies of the packages that have test suites
that Alpine is not willing to merge.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months
Packages that fail test suites on Alpine [corrected]
by A. Wilcox
The following packages pass their test suites on Adélie and fail on Alpine:
main/autoconf
main/c-ares (ppc64le, s390x only)
main/cmake
main/diffutils
main/glib-networking
main/gsl
main/libnih
main/libsndfile (armhf only)
main/libtool
main/orbit2
main/rsync
main/sed
main/snappy
main/strace
main/upower
main/wayland
It'd be very nice to resolve these issues so that the packages can be
tested for Alpine as well.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead
3 years, 2 months
Packages that fail test suites on Alpine
by A. Wilcox
The following packages pass their test suites on Adélie and fail on Alpine:
main/autoconf
main/c-ares (ppc64le, s390x only)
main/cmake
main/diffutils
main/glib-networking
main/gsl
main/orbit2
main/rsync
main/sed
main/snappy
main/strace
main/wayland
It'd be very nice to resolve these issues so that the packages can be
tested for Alpine as well.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead
3 years, 2 months
main/ -> system/ vs aports.git fork [2]
by A. Wilcox
[This list is continued from the previous email.]
gdb:
I could likely manage to bring Alpine and Adélie gdb packages even, but
effort. Their ppc64le fix applies to *all* PPC, and that means turning
a conditional into a case statement. They aren't specifying DISTRO_*
for package version or bug URL. It also needs to be modernised. I will
probably work on this when we are in beta. Alpha cycles are too stressful.
gdk-pixbuf:
We add jasper for JPEG2000 support. It is buggy on 64-bit musl (the
test suite even crashes 1 every 8-10 times) because jasper needs better
patching (stack overflows). I don't really want to shove this upstream
until jasper itself is working better. If someone else will maintain it
in system/ we can do that, otherwise we may drop JPEG2000 support until
jasper is working better. There is also the python-compat issue.
gettext:
ABI breakage. I don't even know how or why. Alpine ships a different
gettext ABI that is not the same as upstream. I removed the ABI patch
from our build because we don't have that issue, and that patch caused
alpha2 packages to segfault. We really can't ship Alpine gettext, so we
will probably need to move this to system/.
glib:
We carry six (!) patches to make the test suite pass on musl. Some of
them should probably be sent upstream. We also ship xattr support via
attr-dev and replace Python 2 with Python 3. I'm already tracking glib
upstream for the constructors issue so I can probably maintain it in
system/.
gnome-doc-utils:
We ship a hilariously broken version with Python 3 just to satisfy the
few packages left that still depend on it. This has been deprecated
upstream for years, so if we have to put it in system/, it won't need
maintenance.
gobject-introspection:
Py2->3, and a confusing comment, prevent me from merging this upstream.
The package seems to work correctly but I don't understand the intent of
enabling a -dev package after saying "dont bother separate -dev".
gpgme:
We need the Qt5 version for KDE, and again Alpine doesn't have Qt5 in
main/, so this needs to probably move to system/. I'd almost be tempted
to move it to user/, but I'd be worried about conflicting.
graphviz:
Not compatible with Python 3 so I ripped out the Python bindings and
replaced them with Guile bindings. I have a feeling Alpine wouldn't
appreciate that, though I didn't see any packages with py2-gv as a
dependency in aports.
grep:
We need grep in /bin instead of /usr/bin for Java. We will be forced to
move this to system/ if we don't maintain a fork of aports.git.
gstreamer (and gst-plugins-*):
Alpine has made a mistake with ldpath for GStreamer 1.0, setting it to
/usr/lib/gstreamer-.0 instead of /usr/lib/gstreamer-1.0. This error
would mean every package using GStreamer would need to be rebuilt
upstream if that was fixed. On the other paw, I REALLY do not want to
ship such a blatant error if we can help it. While I would regret
having to maintain all the GStreamer packages, it may be necessary.
[The next 10 will be in the next email.]
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months
main/ -> system/ + user/ vs aports.git fork [5]
by A. Wilcox
[This list is continued from the previous email.]
rsync:
We enable xattr support via attr-dev. We split -openrc out. We also
have the test suite working correctly.
sdl:
We enable NLS.
syslinux:
We do not use mkinitfs so we need to move this to system/ if we do not
fork aports.git.
util-linux:
In addition to Python 2->3 we enable partx, mesg, write, and wall, which
are requirements of POSIX but disabled by Alpine for being unnecessary
bloat. This is almost certainly going to need to moved to system/ if we
don't fork.
v4l-utils:
We are using the better-maintained Qt5 port of V4L, which again, needs
the Qt5 packages that are only in community/. I may suggest to Alpine
to move v4l-utils to community/ anyway; then our changes could be
upstreamed.
vim:
We ship a default vimrc. This will have to move to system/.
wpa_supplicant:
We depend on D-Bus for KDE integration.
xf86-video-ati:
We disable GLAMOR extensions in our driver. I don't remember why or
what issue that solves. If there's no good reason to do that, we can
just use the upstream xf86-video-ati package.
xfsprogs:
We ship patches that don't apply to the new version. If they are
unnecessary, we can simply ship the Alpine 4.14.0. Otherwise, we need
to port them to 4.14.0.
[Statistics and other information will be in the next and final email.]
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months
main/ -> system/ + user/ vs aports.git fork [1]
by A. Wilcox
Currently we have a "fork" of aports.git. It's very difficult to rebase
what we have, so it definitely needs to "restart" imo. We can move
aports.git to aports-historic.git and then re-clone from Alpine to make
it cleaner, but I think the better thing is to pull all the packages
that we want to keep different from Alpine and put them in system/ in
packages.git, leaving our aports.git fork strictly for changes we wish
to upstream.
Of course, since we can no longer merge in upstream changes, we would
therefore be tracking all the updates ourselves. This is why I would
like to be conservative.
These are the packages with changes that, for whatever reason, we cannot
merge upstream to Alpine. We should discuss what to do with them.
audacious (and audacious-plugins):
We are shipping the Qt variant. Alpine are shipping the Gtk variant.
Since Alpine doesn't have Qt5 in main/ we unfortunately cannot upstream
this change. We could rename it audacious-qt and ship it in user/,
where even Alpine users could enjoy it, but it would need a maintainer.
bash:
We have the -binsh virtual, the patch for bashrc location, and our own
bashrc file. None of these make sense for upstream. Unless I hear a
good argument for offering it in user/, I'm pretty sure this should be
in system/.
binutils:
We ship 2.29 instead of 2.28. We ship seven patches for everything from
tests to wrong behaviour with MIPS targets. ncopa did not seem
interested in bumping binutils until GCC is also bumped. This could be
moved to system/ since we are already maintaining it ourselves.
busybox:
We have an /sbin/init virtual. I added busybox as a provider.
Honestly, this was primarily for Alpine's benefit. Since they don't
have a reason to ship a /sbin/init virtual I think this change can be
discarded; if anyone wants to use busybox as /sbin/init, please speak up
now.
check:
They have a checkdepends of gawk which is satisfied by our mawk. Very
annoying as I don't want to ship gawk at all, but I suppose we can live
with it if it means we don't have to fork.
coreutils:
We have a bunch of hacks to support cross-compile, and a 32-bit PowerPC
patch. They don't release very often so I am fine with moving this to
system/.
exiv2:
We are tracking 0.26 which is not yet released on their official site
because it includes fixes for big-endian that are irrelevant to Alpine.
I know the maintainers personally and I am fine with maintaining this
myself in system/.
ffmpeg:
We enable a LOT more plugins: libcdio, ladspa, lzma, libspeex, freetype,
wavpack, libwebp, and pulseaudio output support. Almost none of those
are in Alpine main/ so it is impossible to add them to ffmpeg
dependencies upstream. I really don't want to maintain ffmpeg myself
since it is a frequent security flyer; if someone else wants to maintain
it in system/ then that is fine. As someone who *uses* three of those
plugins on my own system,
freetype:
Our freetype-profile.sh differs (we enable infinality by default). We
have no other changes. Alpine upstream temporarily did re-enable
infinality by default, but they did not like it (I believe there was
some bug in their XFCE 4 maybe?) so they reverted it.
gcc:
Our gcc is wildly different from Alpine, so much so that this probably
belongs in system/ even if we do end up keeping an aports.git fork. In
addition, I don't feel comfortable with jumping to GCC 8 and Alpine has
been talking about it for retpoline support. This is going in system/
unless someone has a very good argument why it shouldn't.
I'm limiting each email to 10 packages. See you in the next message!
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months
main/ -> system/ + user/ vs aports.git fork [6]
by A. Wilcox
Once the discussions for these packages are finished and we have a plan
for what we will do with them, we can implement it and then ship alpha5.
This will give us a very solid foundation to build on for alpha6 and
then beta1.
The following packages must be moved to system/ no matter what we do:
linux-headers (we ship arches that Alpine do not)
llvm4 (Alpine only ship llvm5 now)
openssl (arch stuff like linux-headers)
The following packages must be moved to system/ due to dependency on
`cmd:which`:
i3lock
libvpx
The following packages must be moved to system/ due to Python 2 changes
that would break Alpine packages:
boost
gamin
libevent
libxml2
libxslt
mesa
postgresql
py-dbus
py-mako
weechat
The following packages must be moved to system/ due to us requiring them
to be built with PAM support:
consolekit2
openssh
polkit
sudo
The following packages must be moved to system/ if Alpine does not
change LibreSSL back to OpenSSL for the provider of libssl and libcrypto:
abuild
apr-util
ca-certificates
clamav
cryptsetup
cups
curl
cyrus-sasl
git
heimdal
hexchat
iputils
irssi
krb5
libfetch
libssh2
libwebsockets
links
mariadb
mosquitto
mutt
neon
rtmpdump
ssmtp
vde2
We will also move poppler-qt5 to user/; this should have never been
added to main/ in our fork.
The following packages have test suites that would break on Alpine and
will either be moved to system/ or marked !check:
attr (Perl 5 issues)
kyua (Requires sysctl change)
openldap (Requires two hours on PPC)
parted (Experimental Python 3 changes to test suite)
tzdata (Requires package `sp` which is in Adélie user/ only)
zsh (Does not seem to work on Alpine)
This large amount of churn is the reason why alpha5 is being delayed so
much further than expected. We will be able to move much faster after
these issues are sorted.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
3 years, 2 months