libvirt_cloud/README.md

Spin up libvrt instances from a cloud-init capable base that can be
maintained using chroot on qcow2, and by ansible using libvirt_qemu.

This builds on two projects:
1) bash scripts to take a Gentoo qcow2 nightly box and use it as a base layer
for another qcow image that you spin up in libvirt, overlaying the base layer
for testing. You can then throw away that image for each test (if you like);
it takes less than a minute to make the base qcow2 layer, and other minute
to make the overlay layer and spin it up.
https://github.com/earlruby/create-vm/

2) python scripts to take a Gentoo stage3 and portage to build a qcow2 box and
use it as a base layer for a cloud-init capable qcow image.
https://github.com/NucleaPeon/gentooimgr/

The huge advantage of this is that you can maintain the base image
under chroot, so you can build test instances that have no incoming
network and still maintain them using tools like ansible, which
support maintaing chroot instances. The problem of using 1) gentooimgr
is that the configuration code of the install step is written in
Python, rather than handled by a more capable tool like ansible.

The problem of using 2) gentooimgr is that the base box may be old (it
is) and you may want to use an existing gentoo kernel and initramfs,
and the base box may be missing qemu-quest-agent (it is). *Much* worse
is that to use cloud-init you have to load rust and 344 crates from
God only knows where that you'll never audit, just to get oauth: no
thanks Google.

You can use ansible to maintain the base layer using chroot, and use
it again to create and spin up the test instance. And as we install
qemu-quest-agent in the base layer, you can manage the test instance
with ansible using libvirt guest-agent, even if the test instance
allows no incoming network.

For now, the code is written to build Gentoo base images, but it could
be extended to other bases. It can build a Gentoo base image from a
Gentoo system.  or another system with a Gentoo system mounted on some
directory, that can be chrooted into, and files copied from. It may be
able to build from a Debian without a mounted Gentoo filesystem, but this
is currently untested.

## Workflow

1) We build the qcow2 base image that we can maintain by chroot
mounting the disk, so we can sort out problems conveniently. We started
doing that by using EarlRuby's python approach, but later rewrote an
old ansible role by https://github.com/agaffney/ansible-gentoo_install/
That ansible role is in roles/ansible-gentoo_install/tasks/ which is
executed as an included role by the toxcore role.

It's a very basic piece of coding that works on a local connection
and is run on the host by the build_base Makefile target. It starts
out as a local connection play, and run chroot internally when it needs it.
You must set these variable in a host in hosts.yml in the linux_chroot_group:
          BOX_NBD_DEV: nbd1
          BOX_NBD_MP: /mnt/gentoo
          BOX_NBD_FILES: "/i/data/Agile/tmp/Topics/GentooImgr"
          BOX_NBD_BASE_QCOW: "/g/Agile/tmp/Topics/GentooImgr/gentoo.qcow2"
This role is slow and may take an hour or more;
It will build the BOX_NBD_BASE_QCOW.

As a safety feature you must create and open the qcow base image before
running the roles: the roles do not use qemu-nbd -c or qemu-nbd -d by
design. You may also choose to download the gentoo latest stage3 and
portage files to the directory specified in hosts.ynl as BOX_NBD_FILES
These

2) We build the qcow2 overlay image that we can maintain by libvirt.
It is run on the host by the build_overlay Makefile target which runs
/usr/local/bin/toxcore_build_overlay_qcow.bash. It gets its parameters from
the hosts.yml file from the host called gentoo1 in the linux_libvirt_group.

## Roles

There are 3 ansible roles:

1. base : The base role sets up the basics and is required to be run.
   It sets up the essential parameters to run roles on the host or client.
   Check the settings in roles/base/defaults/main.yml before running the role.
   
2. proxy : The proxy role sets up the networking with proxies,
   and is required to be run, even if you don't use a proxy.
   It sets proxying and installs basic packages on the host or client.
   Check the settings in roles/proxy/defaults/main.yml before running the role.

3. toxcore : 
   This role sets up the software to run libvirt on the host.
   Check the settings in roles/toxcore/defaults/main.yml before running the role.
   
   In addition, toxcore calls an included role ansible-gentoo_install.
   This is an updated version of the abandonned
   https://github.com/agaffney/ansible-gentoo_install/ This role,
   when run on the host, builds the Gentoo base qcow image. As a safety
   feature, you must create the qcow2 image and activate it with:
    
   The host creates the base qcow2 image and then creates the overlay
   image. When both are created, it install Tox software on the host and
   client.

    modprobe nbd
    qemu-img $BOX_NBD_BASE_QCOW 20G
    qemu-nbd -c $BOX_NBD_DEV $BOX_NBD_BASE_QCOW

    and put these values into the hosts.yml file in the pentoo or devuan
    target, depending on your host operating system. The filesytem that
    holds base qcow2 $BOX_NBD_BASE_QCOW must have at least 12G available,
    and may grow to almost 20G.

After you have finished building the base qcow2 image, you will want
to dismount it with qemu-nbd -d $BOX_NBD_DEV. Be careful and look
in /proc/partitions to see if it is still there after you dismount it:
it the partition is busy the dismount will fail silently, and you can
get into trouble if partprobe complains. You may have to reboot,
or it may resolve itself. Wierd.

Each role has been conditionalized to run with different connections.
 
## Connection Types

There are 3 ansible connection types:

1. localhost : ( ansible_connection == local )
   Running the roles with a local connection will setup the host to
   be able to run the software. The toxcore role will build the
   toxcore software on the localhost and runs tests on it. It will also
   build the base qcow2 image that will underly the overlay box.
   
2. chroot : ( ansible_connection == chroot )
   When you have built the base box, you can chroot mount the qcow2 image
   with qemu-nbd, so all of the configuring of the base can be done without
   a network. A local bash script then builds the overlay qcow2 instance,
   in less than a minute.
   
3. remote :  ( ansible_connection == libvirt_qemu )
   The base box provides the libvirt_qemu connection that is be used
   to run ansible roles on the overlay image. The toxcore role will
   build the toxcore software in the overlay and runs tests on it.
   
All of the 3 roles can all be run with all 3 connection types.

## Stages

There are 4 stages to building an instance:

1. Setup the localhost : 
   set up the host up with the software and settings needed to build boxes.
   
2. Build the base qcow2 base box :
   
3. Build the overlay qcow2 instance : 

4. Test the overlay instance :

## Hosts.yml targets

## Makefile targets

all: install lint build check run test

1. install

2. lint

3. build

4. check

5. run

6. test

## Simplest usage

On Ansibles from 2.10 and later, you will need the community plugins installed.

### Downloaded base qcow

### Created and copuied base qcow on a Gentoo system

### Created and copied base qcow on a non-Gentoo system with a Gentoo mounted

## Advanced Usage

### GentooImgr: Gentoo Image Builder for Cloud and Turnkey ISO installers

There is a modified version of https://github.com/NucleaPeon/gentooimgr/
where we've modified the code a little to do use Python logging. We can
still use it for the build stage, but we think the install stage is better
done using ansible, hence the libvirt_cloud playbook.

The code is in src/ansible_gentooimgr The code is being supported as
an ansible module using library/ansible_gentooimgr.py which is a work
in progress; the idea is to use it for the build and status actions,
but handle the install tasks using ansible.

### ansible_local.bash

We have a script that calls ansible to run our play: ansible_local.yml
but sets some defaults

[ -l limit ]
[ -c connection ]
[ --skip comma,separated]
[ --tags comma,separated]
[ --check]
[ --diff]
[ --step]
[ --verbose 0-3] higher number, more debugging
roles - base and proxy roles will always be run.