Developing in Linux Containers¶
Most often in our development cycle, we edit the sources and run some test
and/or builds by using
make [ref] before we commit. This
cycle is simple and perfect but might fail in some aspects we should not
The environment in which we run all our development processes matters!
The Makefile & ./manage and the Python environment (make install) encapsulate a lot for us, but these tools do not have access to all prerequisites. For example, there may have dependencies on packages that are installed on developer’s desktop, but usually are not preinstalled on a server or client system. Another example is; settings have been made to the software on developer’s desktop that would never be set on a production system.
Linux Containers are isolate environments, we use them to not mix up all the prerequisites from various projects on developer’s desktop.
The scripts from DevOps tooling box can divide in those to install and maintain software
and the script
with we can scale our installation, maintenance or even development tasks over a stack of isolated containers / what we call the:
Before you can start with containers, you need to install and initiate LXD once:
$ snap install lxd $ lxd init --auto
And you need to clone from origin or if you have your own fork, clone from your fork:
$ cd ~/Downloads $ git clone https://github.com/searxng/searxng.git searxng $ cd searxng
The SearXNG suite config consists of several images, see
LXC_SUITE=(... near by git://utils/lxc-searxng.env#L19.
For this blog post we exercise on a archlinux image. The container of this
image is named
Lets build the container, but be sure that this container does not already exists, so first lets remove possible old one:
$ sudo -H ./utils/lxc.sh remove searxng-archlinux $ sudo -H ./utils/lxc.sh build searxng-archlinux
$ sudo -H ./utils/lxc.sh install suite searxng-archlinux $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL ... [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng
In such a SearXNG suite admins can maintain and access the debug log of the services quite easy.
In the example above the SearXNG instance in the container is wrapped to
http://n.n.n.140/searxng to the HOST system. Note, on your HOST system, the
IP of your
searxng-archlinux container is different to this example. To
test the instance in the container from outside of the container, in your WEB
browser on your desktop just open the URL reported in your installation
Usually you open a root-bash using
sudo -H bash. In case of LXC containers
open the root-bash in the container is done by the
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash INFO: [searxng-archlinux] bash [root@searxng-archlinux SearXNG]$
[root@searxng-archlinux ...] signals, that you are the root user
in the container (GUEST). To debug the running SearXNG instance use:
$ ./utils/searxng.sh instance inspect ... use [CTRL-C] to stop monitoring the log ...
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect ... use [CTRL-C] to stop monitoring the log ...
Back in the browser on your desktop open the service http://n.n.n.140/searxng
and run your application tests while the debug log is shown in the terminal from
above. You can stop monitoring using
CTRL-C, this also disables the “debug
option” in SearXNG’s settings file and restarts the SearXNG uwsgi application.
$ ./utils/searxng.sh instance cmd bash -l (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd /usr/local/searxng
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd /usr/local/searxng
[searxng@searxng-archlinux] signals that you are logged in as system
searxng in the
searxng-archlinux container and the python virtualenv
(searxng-pyenv) environment is activated.
In this section we will see how to change the “Fully functional SearXNG suite” from a LXC container (which is quite ready for production) into a developer suite. For this, we have to keep an eye on the Step by step installation:
SearXNG setup in:
SearXNG user’s home:
SearXNG software in:
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng
With the command above, we stopped the SearXNG uWSGI-App in the archlinux container.
The uWSGI-App for the archlinux distros is configured in
git://utils/templates/etc/uwsgi/apps-archlinux/searxng.ini, from where at
least you should attend the settings of
env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml http = 127.0.0.1:8888 chdir = /usr/local/searxng/searxng-src/searx virtualenv = /usr/local/searxng/searxng-pyenv pythonpath = /usr/local/searxng/searxng-src
If you have read the Good to know you remember, that each container
shares the root folder of the repository and the command
handles relative path names transparent.
To wrap the SearXNG installation in the container into a developer one, we
simple have to create a symlink to the transparent repository from the
desktop. Now lets replace the repository at
searxng-src in the container
with the working tree from outside of the container:
$ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old $ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src
Now we can develop as usual in the working tree of our desktop system. Every time the software was changed, you have to restart the SearXNG service (in the container):
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
Remember: In containers, work as usual .. here are just some examples from my daily usage:
To inspect the SearXNG instance (already described above):
$ ./utils/searx.sh inspect service
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searx.sh inspect service
Run Makefile & ./manage, e.g. to test inside the container:
$ make test
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test
To install all prerequisites needed for a Buildhosts:
$ ./utils/searxng.sh install buildhost
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost
To build the docs on a buildhost Buildhosts:
$ make docs.html
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html
We build up a fully functional SearXNG suite in a archlinux container:
$ sudo -H ./utils/lxc.sh build searxng-archlinux $ sudo -H ./utils/lxc.sh install suite searxng-archlinux ... Developer install? (wraps source from HOST into the running instance) [YES/no]
To wrap the suite into a developer one answer
YES (or press Enter).
link SearXNG's sources to: /share/SearXNG ========================================= mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup" ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src" ls -ld /usr/local/searxng/searxng-src |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG
On code modification the instance has to be restarted (see uWSGI maintenance):
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
To access HTTP from the desktop we installed nginx for the services inside the container:
$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
To get information about the SearxNG suite in the archlinux container we can use:
$ sudo -H ./utils/lxc.sh show suite searxng-archlinux [searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/ [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744] [searxng-archlinux] uWSGI: [searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket [searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env: [searxng-archlinux] GIT_URL : https://github.com/searxng/searxng [searxng-archlinux] GIT_BRANCH : master [searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng [searxng-archlinux] SEARXNG_PORT : 8888 [searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1