Linux Jails With VNET

Look online for information about how to run a Linux jail with VNET and you’ll find very few resources. And of course some of the advice appears obsolete. I managed to do this recently with FreeBSD 14.1 and Debian 12, and here is the setup.

The instructions in the FreeBSD Handbook are good. I found it useful to walk through the progression of creating an ordinary thin jail, a VNET jail, a Linux jail, and finally a Linux jail with VNET.

Beware one typo, in the first of the two configuration steps for a Linux jail. The path setting should be path="/usr/local/jails/containers/ubuntu". (The example is missing /containers.

The first configuration step lets you connect to the jail to run debootstrap and install the Linux userland in a directory for later use with chroot. The second configuration adds the mounts needed for Linux to the chroot directory. In effect, you create an ordinary jail, set up a subdirectory in it to run Linux userland, and then chroot there to use Linux.

It’s not in fact necessary to log into the jail to run debootstrap. If you have it installed on the host machine you can run and provide the full path to the jail’s chroot directory as seen from the host. That’s a bit less mysterious, though it does require running debootstrap outside the confines of the jail.

Because of this split-level treatment, you have a free hand to adjust the configuration of the jail itself without disrupting the chroot Linux. When I tried that to add VNET, it just worked.

My final configuration for the jail looked like this:

debv {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

  # PATH/HOSTNAME
  host.hostname = "${name}";
  path = "/usr/local/jails/containers/${name}";

  # VNET/VIMAGE
  vnet;
  vnet.interface = "${epair}b";

  # NETWORKS/INTERFACES
  $id = "159";
  $ip = "10.0.6.${id}/16";
  $gateway = "10.0.0.1";
  $bridge = "bridge0";
  $epair = "epair${id}";

  # ADD TO bridge INTERFACE
  exec.prestart  = "/sbin/ifconfig ${epair} create up";
  exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
  exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
  exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
  exec.start    += "/sbin/route add default ${gateway}";
  exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
  exec.poststop += "/sbin/ifconfig ${epair}a destroy";

  # MOUNT
  mount += "devfs     $path/compat/debv/dev     devfs     rw  0 0";
  mount += "tmpfs     $path/compat/debv/dev/shm tmpfs     rw,size=1g,mode=1777  0 0";
  mount += "fdescfs   $path/compat/debv/dev/fd  fdescfs   rw,linrdlnk 0 0";
  mount += "linprocfs $path/compat/debv/proc    linprocfs rw  0 0";
  mount += "linsysfs  $path/compat/debv/sys     linsysfs  rw  0 0";
  mount += "/tmp      $path/compat/debv/tmp     nullfs    rw  0 0";
  mount += "/home     $path/compat/debv/home    nullfs    rw  0 0";
}

This simply combines the mounts neede for Linux with the underlying jail setup from the VNET example.

Some resources say that you need to pull ifconfig and dhclient from /rescue into the jail in order to use VNET. Presumably that is needed if you avoid the chroot and operate in Linux at the top level of the jail installation. And perhaps that approach gives you faster and broader network connectivity. But to get up and running immediately, the split-level approach is simple and just works.