In this article we are going to talk continuous testing and packaging in Common Lisp. The goal is to automate:

• Report coding errors on every push, including pull requests. This prevents unseen, long-standing breakages, and also helps with pull requests since the system will automatically report failing tests, compilation warnings, etc.

• Ease the release process by automating package builds. This allows us to release often with a higher level of guarantees.

Since Nyxt is currently hosted on GitHub, we are leveraging GitHub Actions for our continuous integration. Even though we are using GitHub Actions, we've designed our process to be platform independent. As a result, our GitHub-specific code remains small and trivial.

Testing

Some Common Lisp compilers are particularly good at code analysis:

• No false positives, all warnings are meaningful.

• They catch many errors at compile time, including typing errors.

SBCL and CCL are two high-quality compilers that fit the bill. While only SBCL is officially supported to build Nyxt, CCL proves useful in catching some error and warning classes that SBCL misses. In particular, CCL is better at type-checking class slots.

Let's review what we want to automate to increase our quality assurance:

• Build: The project compiles with all compilers.

• Testing: All test suites pass with all compilers.

• Code checking: No warnings are reported with any compiler.

In practice, the last two points suffice since the code checking is done by building the project.

Allow me to emphasize the importance of code checking: since the compilers don't report false positives, this forces us to have warning-free code at all time, which is a big boost for quality assurance!

Automatic build

First we need to set up the GitHub action in this YAML file.

In short:

• We install the Nyxt dependencies using the image package manager, here APT.

• Then we install Roswell, a tool that will allow us to easily install the desired Common Lisp compilers since the host system package managers may not have them. Roswell includes Quicklisp, so we will leverage this to install all the Common Lisp dependencies of Nyxt.

• Some dependencies are missing from Quicklisp, so we fetch them via our Makefile dedicated rule, then register their location as per the ASDF API.

Automatic testing

Finally, we run the following:

(handler-bind (#+asdf3.2 (asdf:bad-system-name (function MUFFLE-WARNING)))
  (handler-case (ql:quickload :nyxt/tests)
    (error (a) (format t "caught error ~s~%~a~%" a a) (uiop:quit 17))))

(asdf:test-system :nyxt)

We load the tests which builds Nyxt itself. If the Nyxt build fails, the error is reported as part of the workflow.

We exit with a custom non-zero error code, here 17. This can help identify the cause of the error in case the output gets confusing.

The rest of the test controls happens in the nyxt.asd file. Our main :nyxt system lists all the tests so that call (asdf:test-system :nyxt) effectively runs the whole test suite.

Each individual test system is specified in the following form:

(asdf:defsystem nyxt/tests
  :depends-on (nyxt prove)
  :perform (asdf:test-op (op c)
                         (nyxt-run-test c "tests/")
                         (nyxt-run-test c "tests-network-needed/" :network-needed-p t)))

We've rolled out our own helper function nyxt-run-test to factor some recurring code.

(defun nyxt-run-test (c path &key network-needed-p)
  (and (or (not network-needed-p)
           (not (uiop:getenv "NYXT_TESTS_NO_NETWORK")))
       (not (funcall (read-from-string "prove:run")
                     (asdf:system-relative-pathname c path)))
       (uiop:getenv "NYXT_TESTS_ERROR_ON_FAIL")
       (uiop:quit 18)))

We've added some knobs that we can control externally:

• If the NYXT_TESTS_NO_NETWORK environment variable is set, the test is not run. This is useful to disable tests require a network connection. Some build systems (like Guix) disable network connectivity during builds.

• If NXYT_TESTS_ERROR_ON_FAIL is set, the process will exit with a non-zero error code, which will cause the build system, or the integration pipeline to fail and report. d This is necessary because otherwise the ASDF test operation does not "fail" in the sense that the process returns the 0 error code by default.

  In the YAML file, we set this variable to yes, thus externally commanding our test suite to reflect its error on the pipeline.

Automatic code checking

Finally, our last step in our YAML file is very simple: it loads a file and executes the compilation-conditions function on systems we want to test, here nyxt and the renderers.

Let's look at this file more closely:

• First compilation-conditions calls load-system-silently on the recursive dependencies of the system as returned by list-dependencies.

  We do this to pre-compile the Nyxt dependencies, thus ensuring that when we compile Nyxt the compiler only reports warnings related to Nyxt and not its dependencies.

  load-system-silently is like ASDF's load-system but muffles the output to keep the pipeline output shorter.

• An amazing feature of Common Lisp is that the compile function is built into the language, which allows us to control the compilation process in Common Lisp itself!

  Thus we collect all conditions that are not redefinitions:

  (handler-bind ((warning (lambda (c)
                            (unless (redefinition-p c)
                              (push c conditions)))))
    (asdf:load-system system :force t))

  Thanks to @phoe for this tip!

  Note to the attentive reader: redefinition-p is not the proper way to check if a condition is a redefinition. The redefinition condition type is not portable, so the code should be different between SBCL and CCL, but it turns out that this "hack" works in our case.

• Finally, we report the conditions to the standard output (which will display in the continuous testing web interface) and return a custom non-zero error code.

Packaging

Since none of us at Atlas are using a Debian-based distribution such as Ubuntu, and since these distributions are among the most popular platforms, it has become increasingly necessary to automate the process of distributing a pre-built .deb package

Automating packaging is no simple task, for this we leverage the linux-packaging Common Lisp library, which itself is based on fpm.

linux-packaging is a real life saver: it abstracts the tasks of packaging for various package managers (as November 2020, .deb, .rpm and pacman's formats are supported) in a consistent Common Lisp interface that sits on top of ASDF.

Beside wrapping fpm, linux-packaging automatically guesses the operating system dependencies for the FFI libraries and statically links the FFI-generated objects into the Lisp image. No more problem distributing Osicat!

As for continuous testing, we have a YAML file which has roughly the same steps except that we don't leverage Roswell here because we are going to build our own SBCL compiler: indeed, linux-packaging requires SBCL to be built with the non-default --with-sb-linkable-runtime option.

Notice the new dependency line:

sudo apt-get install -y ruby ruby-dev rubygems dpkg-dev sbcl curl git-core zlib1g-dev

Ruby is required for fpm and SBCL for… rebuilding SBCL!

Not much here since everything happens in the build-ubuntu-package.sh script:

• We first install fpm.
• Then we build SBCL with the --with-sb-linkable-runtime --with-sb-dynamic-core options.
• We install Quicklisp manually since this time we don't have Roswell.
• We install linux-packaging.
• Pitfall: We install a modern version of ASDF since linux-packaging requires a version that's more recent than the one shipped with SBCL (at least as of 2.0.10).
• As for continuous testing, we register the current directory in the ASDF registry so that it finds the Nyxt ASDF systems.

Finally, the package build happens in the last command:

sbcl \
    --eval '(setf *debugger-hook* (lambda (c h) (declare (ignore h)) (format t "~A~%" c) (sb-ext:quit :unix-status -1)))' \
    --load ~/quicklisp/setup.lisp \
    --eval "(ql:quickload :linux-packaging)" \
    --eval "(ql:quickload :nyxt)" \
    --eval "(ql:quickload :nyxt-ubuntu-package)" \
    --eval "(asdf:make :nyxt-ubuntu-package)" \
    --quit

quickload will drag all linux-packaging and nyxt dependencies. Then we load and make an ASDF system that's dedicated to the creation of the package.

This system is declared in a separate file because otherwise it would make Nyxt depend on linux-packaging, which the end user does not need.

Let's review it:

(defsystem "nyxt-ubuntu-package"
  :defsystem-depends-on ("linux-packaging")
  :class "linux-packaging:deb"
  :build-operation "linux-packaging:build-op"

Here we declare that loading this system will trigger the linux-packaging:build-op operation which will generate a linux-packaging:deb package.

Notice that there is only one knob to control the type of package we want to produce. It's enough to change this value to linux-packaging:rpm to produce an RPM!

:depends-on ("nyxt/gtk")
:entry-point "nyxt:entry-point"

Here we list the Common Lisp systems we want to include in our image. Since nyxt/gtk depends on everything else, it's the only system that we need to list.

The :entry-point is the same we use to build Nyxt from the Makefile.

:package-name "nyxt"
:version #.(asdf:system-version (asdf:find-system :nyxt))
:author #.(asdf:system-author (asdf:find-system :nyxt))
:homepage #.(asdf:system-homepage (asdf:find-system :nyxt))
:description #.(asdf:system-description (asdf:find-system :nyxt))
:license #.(asdf:system-license (asdf:find-system :nyxt))

Here we list all the metadata for the .deb. Since we decided to store this system in a separate .asd file, it can't automatically inherit from the metadata of the nyxt system. So we need to use a reader macro to explicitly tell ASDF to look for the metadata of Nyxt.

:additional-dependencies ("glib-networking"
                          "gsettings-desktop-schemas"
                          "xclip"
                          "enchant"
                          "notify-osd")

The attentive reader may have noticed that I said that linux-packaging automatically derived the operating system dependencies for FFI packages. Indeed, we don't need to declare that WebKitGTK is a dependency here for instance.

However, some of our code has optional dependencies. While WebKitGTK would work without glib-networking it would have limited functionality, like no HTTPS support.

Other dependencies include those that are typically used by Common Lisp libraries that depend on executables, like trivial-clipboard which depends on an external clipboard program like xclip.

:additional-files (("assets/nyxt.desktop" . "usr/share/applications/")
                   ("assets/nyxt_16x16.png" . #p"usr/share/icons/hicolor/16x16/apps/nyxt.png")
                   ("assets/nyxt_32x32.png" . #p"usr/share/icons/hicolor/32x32/apps/nyxt.png")
                   ("assets/nyxt_128x128.png" . #p"usr/share/icons/hicolor/128x128/apps/nyxt.png")
                   ("assets/nyxt_256x256.png" . #p"usr/share/icons/hicolor/256x256/apps/nyxt.png")
                   ("assets/nyxt_512x512.png" . #p"usr/share/icons/hicolor/512x512/apps/nyxt.png"))
:build-pathname "nyxt")

Finally, we list all the assets to include in the .deb: icons, .desktop file, etc.

The build-pathname field is the name of the produced executable which will be automatically stored to /usr/bin in the package.

Future work

• Guix test and packaging

  We have a Guix recipe to build Nyxt using the Guix package manager. There are many benefits in providing a Guix package, among others it allows us to provide a create a Guix pack which is an self-containing, portable tarball that can be unpacked and run on any operating system with a Linux kernel.

  Another benefit of packaging for Guix is that it uses its own Common Lisp packages instead of Quicklisp to manage the Common Lisp dependencies. This validates our quality assurance one step further.

• Code coverage

  SBCL supports coverage reporting thanks to its sb-cover extension. We hope to make use of it to provide the most exhaustive test suite possible.

Thanks for reading!

Special thanks

Florian Margaine for his awesome work on linux-packaging.

@phoe for his Common Lisp tips.