Tested with Nyxt 2 Pre-release 4
In this article we are going to talk continuous testing and packaging in Common Lisp. The goal is to automate:
Report coding errors on every push, including pull requests. This prevents unseen, long-standing breakages, and also helps with pull requests since the system will automatically report failing tests, compilation warnings, etc.
Ease the release process by automating package builds. This allows us to release often with a higher level of guarantees.
Since Nyxt is currently hosted on GitHub, we are leveraging GitHub Actions for our continuous integration. Even though we are using GitHub Actions, we've designed our process to be platform independent. As a result, our GitHub-specific code remains small and trivial.
Some Common Lisp compilers are particularly good at code analysis:
No false positives, all warnings are meaningful.
They catch many errors at compile time, including typing errors.
SBCL and CCL are two high-quality compilers that fit the bill. While only SBCL is officially supported to build Nyxt, CCL proves useful in catching some error and warning classes that SBCL misses. In particular, CCL is better at type-checking class slots.
Let's review what we want to automate to increase our quality assurance:
Build: The project compiles with all compilers.
Testing: All test suites pass with all compilers.
Code checking: No warnings are reported with any compiler.
In practice, the last two points suffice since the code checking is done by building the project.
Allow me to emphasize the importance of code checking: since the compilers don't report false positives, this forces us to have warning-free code at all time, which is a big boost for quality assurance!
First we need to set up the GitHub action in this YAML file.
We install the Nyxt dependencies using the image package manager, here APT.
Then we install Roswell, a tool that will allow us to easily install the desired Common Lisp compilers since the host system package managers may not have them. Roswell includes Quicklisp, so we will leverage this to install all the Common Lisp dependencies of Nyxt.
Some dependencies are missing from Quicklisp, so we fetch them via our Makefile dedicated rule, then register their location as per the ASDF API.
Finally, we run the following:
We load the tests which builds Nyxt itself. If the Nyxt build fails, the error is reported as part of the workflow.
We exit with a custom non-zero error code, here 17. This can help identify the cause of the error in case the output gets confusing.
The rest of the test controls happens in the nyxt.asd file. Our main
:nyxt system lists all the tests so that call
(asdf:test-system :nyxt) effectively runs the whole test suite.
Each individual test system is specified in the following form:
We've rolled out our own helper function
nyxt-run-test to factor some recurring code.
We've added some knobs that we can control externally:
NYXT_TESTS_NO_NETWORKenvironment variable is set, the test is not run. This is useful to disable tests require a network connection. Some build systems (like Guix) disable network connectivity during builds.
NXYT_TESTS_ERROR_ON_FAILis set, the process will exit with a non-zero error code, which will cause the build system, or the integration pipeline to fail and report. This is necessary because otherwise the ASDF test operation does not "fail" in the sense that the process returns the 0 error code by default.
In the YAML file, we set this variable to
yes, thus externally commanding our test suite to reflect its error on the pipeline.
Automatic code checking
Finally, our last step in our YAML file is very simple: it loads a file and executes the
compilation-conditions function on systems we want to test, here
nyxt and the renderers.
Let's look at this file more closely:
load-system-silentlyon the recursive dependencies of the system as returned by
We do this to pre-compile the Nyxt dependencies, thus ensuring that when we compile Nyxt the compiler only reports warnings related to Nyxt and not its dependencies.
load-system-silentlyis like ASDF's
load-systembut muffles the output to keep the pipeline output shorter.
An amazing feature of Common Lisp is that the
compilefunction is built into the language, which allows us to control the compilation process in Common Lisp itself!
Thus we collect all conditions that are not redefinitions:
Thanks to @phoe for this tip!
Note to the attentive reader:
redefinition-pis not the proper way to check if a condition is a redefinition. The redefinition condition type is not portable, so the code should be different between SBCL and CCL, but it turns out that this "hack" works in our case.
Finally, we report the conditions to the standard output (which will display in the continuous testing web interface) and return a custom non-zero error code.
Since none of us at Atlas are using a Debian-based distribution such as Ubuntu, and since these distributions are among the most popular platforms, it has become increasingly necessary to automate the process of distributing a pre-built
linux-packaging is a real life saver: it abstracts the tasks of packaging for various package managers (as November 2020,
.rpm and pacman's formats are supported) in a consistent Common Lisp interface that sits on top of ASDF.
linux-packaging automatically guesses the operating system dependencies for the FFI libraries and statically links the FFI-generated objects into the Lisp image. No more problem distributing Osicat!
As for continuous testing, we have a YAML file which has roughly the same steps except that we don't leverage Roswell here because we are going to build our own SBCL compiler: indeed,
linux-packaging requires SBCL to be built with the non-default
Notice the new dependency line:
Ruby is required for
fpm and SBCL for… rebuilding SBCL!
Not much here since everything happens in the build-ubuntu-package.sh script:
- We first install
- Then we build
- We install Quicklisp manually since this time we don't have Roswell.
- We install
- Pitfall: We install a modern version of ASDF since
linux-packagingrequires a version that's more recent than the one shipped with SBCL (at least as of 2.0.10).
- As for continuous testing, we register the current directory in the ASDF registry so that it finds the Nyxt ASDF systems.
Finally, the package build happens in the last command:
sbcl \ --eval '(setf *debugger-hook* (lambda (c h) (declare (ignore h)) (format t "~A~%" c) (sb-ext:quit :unix-status -1)))' \ --load ~/quicklisp/setup.lisp \ --eval "(ql:quickload :linux-packaging)" \ --eval "(ql:quickload :nyxt)" \ --eval "(ql:quickload :nyxt-ubuntu-package)" \ --eval "(asdf:make :nyxt-ubuntu-package)" \ --quit
quickload will drag all
nyxt dependencies. Then we load and make an ASDF system that's dedicated to the creation of the package.
This system is declared in a separate file because otherwise it would make Nyxt depend on
linux-packaging, which the end user does not need.
Let's review it:
Here we declare that loading this system will trigger the
linux-packaging:build-op operation which will generate a
Notice that there is only one knob to control the type of package we want to produce. It's enough to change this value to
linux-packaging:rpm to produce an RPM!
Here we list the Common Lisp systems we want to include in our image. Since
nyxt/gtk depends on everything else, it's the only system that we need to list.
:entry-point is the same we use to build Nyxt from the
:package-name "nyxt" :version #.(asdf:system-version (asdf:find-system :nyxt)) :author #.(asdf:system-author (asdf:find-system :nyxt)) :homepage #.(asdf:system-homepage (asdf:find-system :nyxt)) :description #.(asdf:system-description (asdf:find-system :nyxt)) :license #.(asdf:system-license (asdf:find-system :nyxt))
Here we list all the metadata for the
.deb. Since we decided to store this system in a separate
.asd file, it can't automatically inherit from the metadata of the
nyxt system. So we need to use a reader macro to explicitly tell ASDF to look for the metadata of Nyxt.
The attentive reader may have noticed that I said that
linux-packaging automatically derived the operating system dependencies for FFI packages. Indeed, we don't need to declare that WebKitGTK is a dependency here for instance.
However, some of our code has optional dependencies. While WebKitGTK would work without
glib-networking it would have limited functionality, like no HTTPS support.
Other dependencies include those that are typically used by Common Lisp libraries that depend on executables, like trivial-clipboard which depends on an external clipboard program like
:additional-files (("assets/nyxt.desktop" . "usr/share/applications/") ("assets/nyxt_16x16.png" . #p"usr/share/icons/hicolor/16x16/apps/nyxt.png") ("assets/nyxt_32x32.png" . #p"usr/share/icons/hicolor/32x32/apps/nyxt.png") ("assets/nyxt_128x128.png" . #p"usr/share/icons/hicolor/128x128/apps/nyxt.png") ("assets/nyxt_256x256.png" . #p"usr/share/icons/hicolor/256x256/apps/nyxt.png") ("assets/nyxt_512x512.png" . #p"usr/share/icons/hicolor/512x512/apps/nyxt.png")) :build-pathname "nyxt")
Finally, we list all the assets to include in the
.desktop file, etc.
build-pathname field is the name of the produced executable which will be automatically stored to
/usr/bin in the package.
Guix test and packaging
We have a Guix recipe to build Nyxt using the Guix package manager. There are many benefits in providing a Guix package, among others it allows us to provide a create a Guix pack which is an self-containing, portable tarball that can be unpacked and run on any operating system with a Linux kernel.
Another benefit of packaging for Guix is that it uses its own Common Lisp packages instead of Quicklisp to manage the Common Lisp dependencies. This validates our quality assurance one step further.
SBCL supports coverage reporting thanks to its
sb-coverextension. We hope to make use of it to provide the most exhaustive test suite possible.
Thanks for reading!
Florian Margaine for his awesome work on linux-packaging.
@phoe for his Common Lisp tips.