Use the image

Option 1: Use the Image Directly (not recommended)

Upload packer image using virsh

ls -al *.qcow2

virsh -c qemu+ssh://user@host/system vol-upload --pool default --vol preseeded-armhf-uefi-image-preseed-image-xxxx-xx-xx-xx-xx.qcow2 --file preseeded-armhf-uefi-image-preseed-image-xxxx-xx-xx-xx-xx.qcow2

Create the ARM VM using Virtual Machine Manager

Launch “Virtual Machine Manager” (virt-manager from the command line).
Select ‘Edit|Prefrences’
Make sure ‘Enable XML editing’ is checked
Select ‘File|New Virtual Machine’
Select ‘Import existing disk image’
Change ‘Architecture options’ to Architecture: ‘arm’, Machine Type: ‘virt-2.12’. (virt-3.0 and virt-3.1 are known to not work with this guide; newer and older versions likely will work).
Select ‘Browse…’, select a virtual image you uploaded above, and select ‘Choose Volume’.
1. Alternatively, if you want to use this image for more than one virtual machine, then create a new virtual hard drive and use the uploaded virtual image as a ‘backing store’.
Set the operating system to ‘Debian10’
Select ‘Forward’
Configure the amount of memory and cpus (max 4) and select ‘Forward’
Set the VM name and check ‘Customize configuration before install`
Select the appropriate network device for your virtual hosting setup.
Click ‘Finish’
Change ‘Firmware’ to ‘Custom: /usr/share/AAVMF/AAVMF32_CODE.fd’ and click ‘Apply’.
Select ‘Begin installation’
The system will drop to an EFI prompt.

Configure EFI to Boot Debian

Execute bcfg add 0 FS0:EFI\debian\grubarm.efi "Linux"
Execute reset
VM should reboot into Debian GNU/Linux.

Option 2: Feed the Image to a Packer Provisioning Run

You need a few files:

Output From the Preseed Stage

This guide assumes you have copied the …_VARS.fd, hard drive image (….qcow2), and kernel (vmlinuz) and initramfs (initrd.img) into /home/user/Documents/Artifacts.

Packer Template

You could name this qemu-ansible-armhf-blog-uefi-template.json

{
    "variables": {
        "accelerator": "none",
        "admin_password": "Should fail unless overridden via other variable input.",
        "admin_user": "example-admin",
        "armhf_kernel": "",
        "armhf_initrd": "",
        "build_time": "{{isotime "2006-01-02-15-04"}}",
        "domain": "",
        "hostname": "",
        "iso_checksum": "",
        "iso_checksum_type": "sha512",
        "iso_src_url_prefix": "",
        "iso_name": "",
        "machine_type": "virt",
        "memory_size": "1024",
        "os_disk_size": "8192",
        "output_compression": "true",
        "output_format": "qcow2",
        "provisioning_groups": "",
        "ssh_boot_password": "example-provision-password",
        "uefi_firmware_CODE_path": "/usr/share/AAVMF/AAVMF32_CODE.fd",
        "uefi_firmware_VARS_path": "",
        "vm_name_suffix": "-os.qcow2"
    },
    "sensitive-variables": [
        "admin_password",
        "ssh_boot_password"
    ],
    "builders": [
        {
            "type": "qemu",
            "accelerator": "{{ user `accelerator` }}",
            "cpus": 4,
            "disable_vnc": true,
            "disk_compression":"{{ user `output_compression` }}",
            "disk_image": true,
            "disk_size": "{{ user `os_disk_size` }}",
            "headless": true,
            "format": "{{ user `output_format` }}",
            "iso_checksum": "{{ user `iso_checksum_type` }}:{{ user `iso_checksum` }}",
            "iso_target_extension": "qcow2",
            "iso_url": "{{ user `iso_src_url_prefix` }}/{{ user `iso_name` }}",
            "machine_type": "{{ user `machine_type` }}",
            "memory": "{{ user `memory_size` }}",
            "net_device": "virtio-net-pci",
            "output_directory": "output/output-armhf-uefi-{{ user `hostname` }}-{{user `build_time`}}",
            "qemuargs": [
                [ "-display", "none" ],
                [ "-kernel", "{{ user `armhf_kernel` }}" ],
                [ "-initrd", "{{ user `armhf_initrd` }}" ],
                [ "-boot", "menu=off,order=cd,strict=on" ],
                [ "-serial", "mon:pty" ],
                [ "-machine", "{{ user `machine_type` }},accel={{ user `accelerator` }},usb=off,dump-guest-core=off,gic-version=2,pflash0=pflash0-format,pflash1=pflash1-format" ],
                [ "-drive", "file=output/output-armhf-uefi-{{ user `hostname` }}-{{user `build_time`}}/{{ .Name }},if=virtio,cache=writeback,discard=ignore,format=qcow2" ],
                [ "-blockdev", "driver=file,filename=/usr/share/AAVMF/AAVMF32_CODE.fd,node-name=pflash0-storage,auto-read-only=on,discard=unmap" ],
                [ "-blockdev", "node-name=pflash0-format,read-only=on,driver=raw,file=pflash0-storage" ],
                [ "-blockdev", "driver=file,filename={{ user `uefi_firmware_VARS_path` }},node-name=pflash1-storage,auto-read-only=on,discard=unmap" ],
                [ "-blockdev", "node-name=pflash1-format,read-only=off,driver=raw,file=pflash1-storage" ],
                [ "-append", "elevator=noop noresume root=/dev/vda4" ]
            ],
            "qemu_binary": "qemu-system-arm",
            "shutdown_command": "su -c '( sleep 10 && echo {{ user `admin_password` }} ) | sudo -u root -S shutdown -P now' {{ user `admin_user` }}",
            "ssh_password": "{{ user `ssh_boot_password` }}",
            "ssh_timeout": "10m",
            "ssh_username": "root",
            "use_backing_file": false,
            "vm_name": "{{ user `hostname` }}{{ user `vm_name_suffix` }}"
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "expect_disconnect": true,
            "inline": [
                "hostnamectl set-hostname {{ user `hostname` }}",
                "sed -i -e '1,$s/preseed-image/{{ user `hostname` }}/g' /etc/hosts",
                "systemctl reboot"
            ]
        },
        {
            "type": "ansible",
            "groups": "{{ user `provisioning_groups` }}",
            "host_alias": "{{ user `hostname` }}.{{ user `domain` }}",
            "playbook_file": "playbook-armhf-uefi-blog-example.yml",
            "user": "root"
        }
    ]
}

A Packer ‘var-file’

You could name this arm-devel-blog-uefi-var-file.json

{
    "accelerator": "tcg",
    "admin_user": "example-admin",
    "armhf_kernel": "/home/user/Documents/Artifacts/preseeded-armhf-uefi-image-preseed-image-2020-11-11-10-29/vmlinuz-4.19.0-12-armmp-lpae",
    "armhf_initrd": "/home/user/Documents/Artifacts/preseeded-armhf-uefi-image-preseed-image-2020-11-11-10-29/initrd.img-4.19.0-12-armmp-lpae",
    "domain": "example.net",
    "hostname": "arm-devel",
    "iso_checksum": "6aeecc54be02d3cf51e65dd6f592c7f6b4c3d4ad7663cbdd824f2d895f5509bcee9ec9c858d79b7bb97071a0f24a4ab6144c992fe8a6101d96fa8d0922645532",
    "iso_checksum_type": "sha512",
    "iso_src_url_prefix": "file:///home/user/Documents/Artifacts/preseeded-armhf-uefi-image-preseed-image-2020-11-11-10-29/",
    "iso_name": "preseed-image-armhf-uefi-buster-packer.qcow2",
    "machine_type": "virt-2.12",
    "memory_size": "2048",
    "os_disk_size": "8192",
    "output_compression": "true",
    "output_format": "qcow2",
    "provisioning_groups": "apt_no_proxy,devel_hosts,dhcp,unattended_upgraders,first_regular_user,vm_guest_type_kvm,vm_role_guest,not_wsl",
    "uefi_firmware_CODE_path": "/usr/share/AAVMF/AAVMF32_CODE.fd",
    "uefi_firmware_VARS_path": "/home/user/Documents/Artifacts/preseeded-arm-uefi-image-preseed-image-2020-11-11-10-29/armhf-uefi-debian-10.6_VARS.fd",
    "vm_name_suffix": "-os.qcow2"
}

Admin Password Var File

You could call this password-var-file.json. I recommend you do NOT place this version control, and that you make it readable by the user only (e.g. chmod 600 password-var-file.json), and that you delete it when finished creating the image.

{
    "admin_password": "example-admin-password"
}

Ansible Playbook and Support Files

The following is a very simple ansible playbook for demonstration purposes as the use of Ansible is beyond the scope of this article.
Also note that Packer can use many provisioners, so if you don’t like Ansible you have other choices.
NB Password really shouldn’t be included in playbooks.
- You should use the ansible ‘vault’ with encrypted passwords.
This playbook assumes you have an SSH public/private keypair in your home directory’s .ssh subdirectory. If this is not true, please generate one with ssh-keygen -t rsa.
You should name this playbook-armhf-uefi-blog-example.yml.

- hosts: all
  vars:
    admin_user_name: example-admin
    admin_user_password: example-admin-password
    admin_groups:
    - sudo
    - adm
    - operator
    - staff
    first_user_name: example-user
    first_user_password: example-user-password
    first_user_groups:
    - users

  tasks:
    - name: Configure local admin user
      tags:
        - admin_user
        - ssh
      become: yes
      block:
        - name: Configure group for local admin user
          group:
            name: "{{ admin_user_name }}"
            system: yes
            state: present
        - name: Add local admin user system groups
          group:
            name: "{{ item }}"
            system: yes
            state: present
        loop: "{{ admin_groups | union(['{{ admin_user_name }}'] ) }}"
    - name: Configure local admin user
      user:
        name: "{{ admin_user_name }}"
        password: "{{ admin_user_password | password_hash('sha512') }}"
        system: yes
        create_home: yes
        expires:
        shell: "/bin/bash"
        group: "{{ admin_user_group_name | default(admin_user_name) }}"
        groups: "{{ admin_groups }}"
        state: present
  - name: Configure SSH authorized keys for local admin user
    vars:
        admin_user_public_key:
        - ~/.ssh/id_rsa.pub
        authorized_key:
            user: "{{ admin_user_name }}"
            state: present
            key: "{{ lookup('file', lookup('first_found', admin_user_public_key)) }}"
            exclusive: True

  - name: Install QEMU guest agent
    apt:
    name: "qemu-guest-agent"
    state: present

  - name: Configure local regular user
    tags:
    - regular_user
    - ssh
    become: yes
    block:
      - name: Configure group for first regular user
        group:
        name: "{{ first_user_name }}"
        state: present
      - name: Add first regular user system groups
        group:
            name: "{{ item }}"
            system: yes
            state: present
        loop: "{{ first_user_groups }}"
      - name: Configure first regular user
        user:
            name: "{{ first_user_name }}"
            password: "{{ first_user_password | password_hash('sha512') }}"
            comment: "{{ first_user_comment | default(omit) }}"
            create_home: yes
            expires:
            shell: "/bin/bash"
            group: "{{ first_user_name }}"
            groups: "{{ first_user_groups }}"
            state: present
      - name: Configure SSH authorized keys for local admin user
        vars:
            first_user_public_key:
            - ~/.ssh/id_rsa.pub
            authorized_key:
            user: "{{ first_user_name }}"
            state: present
            key: "{{ lookup('file', lookup('first_found', first_user_public_key)) }}"
            exclusive: True

  - name: Install packages for debian family hosts
    become: yes
    vars:
        devel_packages:
        -  autoconf
        -  autoconf-doc
        -  automake
        -  autopoint
        -  autotools-dev
        -  build-essential
        -  debootstrap
        -  fakechroot
        -  fakeroot
        -  gawk
        -  libncurses-dev
        -  libncurses5-dev
        -  libncursesw5-dev
        -  quilt
    apt:
        name: "{{ devel_packages }}"
        state: present

  - name: Add ifupdown configs as appropriate
    vars:
        iface_fragments:
        enp1s0:
        inet6_type: auto
        become: yes
    block:
      - name: Add ifupdown and related packages
        apt:
          name:
            - ifupdown
            - ethtool
            - isc-dhcp-client
          state: present
      - name: Add configuration for ifupdown
        block:
        - name: Add base interfaces file
          copy:
            src: interfaces
            dest: /etc/network/interfaces
            owner: root
            group: root
            mode: 0644
  - name: Create interfaces fragments directory
    file:
        path: /etc/network/interfaces.d
        owner: root
        group: root
        mode: 0755
        state: directory
  - name: Add interfaces fragments files
    template:
        src: interfaces.d.j2
        dest: "/etc/network/interfaces.d/{{ item.key }}"
        owner: root
        group: root
        mode: 0644
    loop: "{{ iface_fragments|default({})|dict2items }}"

  - name: Configure SSH
    become: yes
    block:
      - name: Add final sshd_config
        copy:
            dest: /etc/ssh/sshd_config
            owner: root
            group: root
            mode: 0755
            src: "sshd_config.sample"

  - name: Configure local root user
    tags:
    - root_user
    become: yes
    user:
      name: "{{ root_user_name | default('root') }}"
      password: "{{ root_user_password | default('*') }}"
      state: present

Support File: SSH Server Config

This should be in the files subdirectory and named sshd_config.sample.

# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
PermitRootLogin no
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunnelled clear text passwords, change to no here!
#PasswordAuthentication yes
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server

Support File: Network Configuration (interfaces) File

This should be in the templates subdirectory and named interfaces.d.j2

# {{ ansible_managed }}

auto {{ item.key }}
allow-hotplug {{ item.key }}

iface {{ item.key }} inet {{ item.value['inet_type'] | default((inventory_hostname in groups['dhcp']) | ternary ('dhcp','static')) }}
{% if item.value['bridge_ports'] is defined %}
  bridge_ports {{ item.value['bridge_ports'] }}
{% endif -%}
{% if item.value['bridge_vlan_aware'] is defined %}
  bridge_vlan_aware {{ item.value['bridge_vlan_aware'] | ternary('on','off') }}
{% endif -%}
{% if item.value['addresses'] is defined -%}
{%- for aitem in item.value['addresses'] %}
  address {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['gateways'] is defined -%}
{%- for aitem in item.value['gateways'] %}
  gateway {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['ups'] is defined -%}
{%- for aitem in item.value['ups'] %}
  up {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['downs'] is defined -%}
{%- for aitem in item.value['downs'] %}
  down {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['extrav4block'] is defined %}
  {{ item.value['extrav4block'] }}
{% endif %}

{% if item.value['inet6_type'] is defined %}
iface {{ item.key }} inet6 {{ item.value['inet6_type'] }}
{% if item.value['bridge_ports'] is defined %}
  bridge_ports {{ item.value['bridge_ports'] }}
{% endif -%}
{%- if item.value['bridge_vlan_aware'] is defined %}
  bridge_vlan_aware {{ item.value['bridge_vlan_aware'] | ternary('on','off')}}
{% endif -%}
{% if item.value['v6addresses'] is defined -%}
{%- for aitem in item.value['v6addresses'] %}
  address {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['v6gateways'] is defined -%}
{%- for aitem in item.value['v6gateways'] %}
  gateway {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['v6ups'] is defined -%}
{%- for aitem in item.value['v6ups'] %}
  up {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['v6downs'] is defined -%}
{%- for aitem in item.value['v6downs'] %}
  down {{ aitem }}
{% endfor -%}
{% endif -%}
{% if item.value['extrav6block'] is defined %}
 {{ item.value['extrav6block'] }}
{% endif -%}
{% endif -%}

Optional: A serial terminal

If you want to watch the virtual machines boot screens you will need a serial terminal program that works with a Linux pty as the serial input/output. picocom is a good choice. Unlike the preseed phase, very little actually happens on this terminal.
I recommend omitting PACKER_LOG=1 in the command below and not bothering to watch the terminal; this will also make the provisioning output much easier to read.

Execute Packer Command

Execute

PACKER_LOG=1 packer build -var-file arm-devel-blog-uefi-var-file.json -var-file password-var-file.json qemu-ansible-armhf-blog-uefi-template.json

The packer command will take a while (probably over ten minutes). To watch the progress point your serial terminal program at the PTY device with baud rate 115200, pointed to by the line Qemu stdout: char device redirected to /dev/pts/xin the packer output. ‘x’ will be a number. For example picocom -b 115200 /dev/pts/3 if x was 3.

Use the Image

Upload instance image using virsh

ls -al *.qcow2

virsh -c qemu+ssh://user@host/system vol-create-as --pool default --name preseeded-armhf-no-efi-image-preseed-image-xxxx-xx-xx-xx-xx.qcow2 --format qcow2 --allocation <size-from-ls> --capacity <size-from-ls>

virsh -c qemu+ssh://user@host/system vol-upload --pool default --vol preseeded-armhf-uefi-image-preseed-image-xxxx-xx-xx-xx-xx.qcow2 --file preseeded-armhf-uefi-image-preseed-image-xxxx-xx-xx-xx-xx.qcow2

Create the ARM VM instance using Virtual Machine Manager

Launch “Virtual Machine Manager” (virt-manager from the command line).
Select ‘Edit|Prefrences’
Make sure ‘Enable XML editing’ is checked
Select ‘File|New Virtual Machine’
Select ‘Import existing disk image’
Change ‘Architecture options’ to Architecture: ‘arm’, Machine Type: ‘virt-2.12’. (virt-3.0 and virt-3.1 are known to not work with this guide; newer and older versions likely will work).
Select ‘Browse…’, select a virtual image you uploaded above, and select ‘Choose Volume’.
1. Alternatively, if you want to use this image for more than one virtual machine, then create a new virtual hard drive and use the uploaded virtual image as a ‘backing store’.
Set the operating system to ‘Debian10’
Select ‘Forward’
Configure the amount of memory and cpus (max 4) and select ‘Forward’
Set the VM name and check ‘Customize configuration before install`
Select the appropriate network device for your virtual hosting setup.
Click ‘Finish’
Change ‘Firmware’ to ‘Custom: /usr/share/AAVMF/AAVMF32_CODE.fd’ and click ‘Apply’.
Select ‘Begin installation’
The system will drop to an EFI prompt.

Configure EFI to Boot Debian Instance

Execute bcfg boot add 0 FS0:EFI\debian\grubarm.efi "Linux"
Execute reset
VM should reboot into Debian GNU/Linux.