Containers are a solution to the ever-present dependency problem: how do you make sure that the user has all of the software needed to run the program you are shipping? In general terms, containers work by bundling a specific operating system, alongside other necessary software, and running the target program using them.<\/p>\n
While Docker has become the most-used containerisation platform, Singularity is interesting for a couple of reasons:<\/p>\n
- \n
- It does not require root access for the most common workflows. This means that users can simply add their image to a high-performance computing cluster and do the necessary work without needing an administrator password.<\/li>\n
- Singularity uses the Docker image format, which means that it works with the myriad of available bundles on the Docker Hub<\/a>.<\/li>\n<\/ul>\n
Therefore, a computer running Singularity on top of a minimal Linux build can host virtually any software, regardless of its dependencies or even the operating system the software was built for, as long as you can make or obtain a working image.<\/p>\n
What is the Yocto Project?<\/h2>\n
The Yocto Project is an open-source, mostly MIT-licensed collaboration that makes it easier to create and build Linux-based embedded systems. It offers a mature, fully automated Linux build system, supporting all major embedded architectures, alongside recipes for various commonly-used software.<\/p>\n
Unifying the world of embedded Linux is a daunting task. The Yocto Project has done a stellar job of it, but it is not user-friendly. Things can and oftentimes do go wrong. Errors in a file can create errors of an apparently unrelated origin. Thankfully, the Yocto user manual<\/a> provides immense detail concerning common tasks, and the Yocto Project mailing lists<\/a> are publicly available and can be of great help.<\/p>\n
Getting started with Yocto<\/h2>\n
If you are new to the Yocto Project, I strongly recommend<\/i> reading the entirety of the Yocto Project Quickstart<\/a> and following its steps until you can successfully emulate a basic image using QEMU. By doing this, you will become familiar with some of the basic tasks, and if something goes wrong it will be significantly easier to find help online. Once this have succeeded, you will have a known working state onto which you can apply the steps within this guide.<\/p>\n
Note:<\/strong> I prefer keeping the build directory<\/a> and my layers outside the poky directory, in order to make changes easier to track using git submodules. This is my directory structure:<\/p>\n
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Tree-of-layer-directory.png\")
<\/p>\nI initialise the build environment by running:<\/p>\n
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Initializing-build-directory.png\")
<\/p>\nContrast with the approach in the quickstart guide, where the build directory resides within poky<\/a> and layers are cloned within the latter.<\/a> I recommend following my approach, for consistency with the rest of this guide.<\/p>\n
Creating your own image<\/h2>\n
In this section I will detail the process of writing an image using software from within the Yocto project. The end product will be a fully-featured Linux build containing the Singularity containerisation engine.<\/p>\n
Obtaining the necessary layers<\/h3>\n
Since our target platform is the Raspberry Pi, you must have the Raspberry Pi BSP<\/a> in our layer directory. Obtain it by cloning the repository:<\/p>\n
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Cloning-meta-raspberrypi.png\")
<\/p>\nThe link to the remote used in the command can be found within the webpage linked above. For reference, here is the command used:
\ngit clone git:\/\/git.yoctoproject.org\/meta-raspberrypi<\/code]><\/code><\/p>\n<\/code>
\nOnce you have cloned a layer, you need to tell the build system to use it. You can do that using:
\nbitbake-layers add-layer ..\/meta-raspberrypi\/<\/code>
\nHere it is in action:<\/p>\n![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Adding-the-meta-raspberrypi-layer.png\")
<\/p>\nWARNING:<\/strong> Running the
bitbake-layers<\/code> tool outside of the build directory makes it fail and print out a massive stack trace. Make sure you are within the build directory and that you reference the layers from this directory.<\/p>\nClick here for further information about
bitbake-layers<\/code>.<\/a><\/p>\nInspecting the README.md file<\/a> of meta-raspberrypi reveals that some extra layers are required: the meta-oe, meta-multimedia, meta-python and meta-networking from within meta-openembedded. Clone the meta-openembedded layer using the remote in README.md and add the necessary layers using
bitbake-layers add-layer<\/code>. The order is important:<\/em> Adding meta-networking before meta-python causes an error:<\/p>\n![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Error-in-meta-networking.png\")
<\/p>\nThankfully,
bitbake-layers<\/code> told us exactly what is wrong. The error is fixed by adding meta-python before you add meta-networking:<\/p>\n![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Succesfully-adding-meta-networking.png\")
<\/p>\nWe can see what layers we are currently using by running
bitbake-layers show-layers<\/code> . At this stage, you should have the same layers as shown below. Make sure you have all these layers before proceeding.<\/p>\n![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Output-of-bitbake-layers-show-layers.png\")
<\/p>\nmeta-virtualisation<\/h4>\n
The recipe for Singularity resides within meta-virtualisation<\/a>. Clone this layer and add it to the build system as shown above. The README.md file<\/a>\u00a0reveals that this layer has further dependencies: openembedded-core, meta-filesystems, oe-meta-go and meta-selinux. The README.md file also provides links to the remote repositories of these dependencies, therefore adding them to your build should be straightforward.<\/p>\n
NOTE:\u00a0<\/strong>The dependency on openembedded-core is fulfilled by the layers automatically added from within poky. You should skip cloning and adding this layer. Poky can replace openembedded-core because it was developed off of OpenEmbedded.<\/a><\/p>\n
Once you have added the layers required by meta-virtualisation, you should be left with this:\u00a0
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/All-of-the-needed-dependencies.png\")
<\/p>\nOnce you are at this stage, you are ready to create your own image. Important takeaways:<\/p>\n
- \n
- Make sure you fully read the README files of any layer you plan on using, and that you obtain the layers it depends upon.<\/li>\n
- Do not forget to add any layer you want to use by using bitbake-layers add-layer.<\/li>\n<\/ul>\n
Writing the custom image recipe<\/h3>\n
In order to keep everything organised, and to make it easy to share the work, I like to keep my images inside a layer<\/a>, even though you can obtain the same result by modifying local.conf<\/a>. The easiest way to create a layer is by using the
yocto-layer<\/code> script:<\/a><\/p>\n![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Creating-a-layer-using-yocto-layer.png\")
<\/p>\nyocto-layer create <\/code> generates the minimally required files to create a layer, leaving it to the user to populate it with recipes.<\/p>\nMake sure you make the build system aware of the layer by using
bitbake-layers add <\/code>. Remember that you must be inside the build directory to successfully run the bitbake-layers script. Additionally, give a sensible name to your layer! The OpenEmbedded build system does not support file or directory names that contain spaces.<\/p>\nIt is conventional to store image recipes under a recipes-core\/images directory, inside the layer folder. Therefore, we will follow this convention. The image itself should be a .bb file. I have named mine ‘computenodes-image.bb’ . Here is what the directory structure should look like:
\n![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Directory-tree-for-computenodes-image.png\")
<\/p>\ncomputenodes-image.bb <\/code> must contain the following lines:<\/p>\n\r\nSUMMARY = "Basic image, containing Singularity"\r\nLICENSE = "MIT"\r\n# Base the image upon a mostly complete Linux build\r\ninclude recipes-extended\/images\/core-image-full-cmdline.bb\r\n\r\n# Install the Singularity containerization platform\r\n# Also install kernel modules, as detailed in the BSP that is being used\r\nIMAGE_INSTALL_append = " \\\r\n singularity \\\r\n kernel-modules \\\r\n "\r\n\r\n# Allocate ~1 extra GB to pull containers\r\nIMAGE_ROOTFS_EXTRA_SPACE = "1000000"\r\n<\/pre>\n
This recipe is based upon rpi-hwup-image<\/a> from meta-raspberrypi. Files preceded by a ‘#’ are comments. The important changes are the fact that it includes core-image-full-cmdline, as opposed to core-image-minimal. This change provides a more complete Linux environment, as opposed to a system which is not capable of doing much more than booting. We are additionally installing Singularity and allocating more SD card space for the containers.<\/p>\n
The SUMMARY variable<\/a> is meant to succinctly describe the product of the recipe.\u00a0The LICENSE variable<\/a> specifies the license under which the source code \/ recipe is licensed. It must be included inside any recipe. Here is some more general information about how OpenEmbedded handles licensing.<\/a><\/p>\n
The include keyword includes all of the text of the mentioned recipe inside the file. BitBake has multiple ways of sharing functionality between files<\/a>. Additionally, IMAGE_INSTALL must be used with care<\/a>. While IMAGE_ROOTFS_EXTRA_SPACE is more or less self-explanatory, the Yocto documentation has a section on it.<\/a><\/p>\n
Building the image<\/h3>\n
To inform the build system that you want to target a Raspberry Pi, you must add the following line to
build\/conf\/local.conf<\/code>:
\nMACHINE = \"raspberrypi3\"
\n<\/code>
\nNow, you are finally ready to build. Runbitbake computenodes-image <\/code> and, after waiting for the build to be complete, you can find the end product underbuild\/tmp\/deploy\/images\/raspberrypi3 <\/code>. In my case, the image is calledcomputenodes-image-raspberrypi3.rpi-sdimg<\/code> . You can then flash the image using Etcher<\/a>. If Etcher does not recognise the file as a valid OS image, change .rpi-sdimg to .img inside the filename<\/em>. Alternatively, you can achieve the same result with a symbolic link tocomputenodes-image-raspberrypi3.rpi-sdimg<\/code> if the link’s filename ends with .img .<\/p>\nTesting the image<\/h3>\n
Obtain access to the terminal by plugging in a monitor and keyboard or, if you know the IP address of the Pi, you can SSH into it. Once you’re there, you can test Singularity:<\/p>\n
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Singularity-selftest.png\")
<\/p>\nIf the selftest succeeds, try pulling a container from the Docker hub:<\/p>\n
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Pulling-Alpine-container.png\")
<\/p>\nIt works! Let us open a shell inside the container.<\/p>\n
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/Shell-inside-the-Alpine-container.png\")
<\/p>\nAnother example:<\/p>\n
![\"\"](\"https:\/\/computenodes.net\/wp-content\/uploads\/2017\/08\/contents-of-etc-forwardslash-issue.png\")
<\/p>\nIt is clear that the environment inside the container is different from the one outside.<\/p>\n
Conclusion<\/h2>\n
We have looked at creating a basic image using the Yocto project. The process involves cloning the necessary layers and their dependencies, creating a new layer, writing the image recipe and, finally, building it. Our test image successfully booted, and the Singularity containerisation engine works without a flaw.<\/p>\n","protected":false},"excerpt":{"rendered":"
This guide serves as a basic introduction to using the Yocto project for building a custom Linux image for the Raspberry Pi. The Singularity containerisation engine is bundled within this image as an example, and because it is a nifty piece of software. In writing this, I assume the reader has some working knowledge of
Continue reading Bundling Singularity within a Yocto image for the Raspberry Pi<\/span> <\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7,8],"tags":[],"class_list":["post-78","post","type-post","status-publish","format-standard","hentry","category-interns","category-raspberry-pi","col-1"],"_links":{"self":[{"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/posts\/78","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/comments?post=78"}],"version-history":[{"count":95,"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/posts\/78\/revisions"}],"predecessor-version":[{"id":204,"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/posts\/78\/revisions\/204"}],"wp:attachment":[{"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/media?parent=78"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/categories?post=78"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/computenodes.net\/wp-json\/wp\/v2\/tags?post=78"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}