Setting up ZoneMinder NVR in a FreeBSD VNET Jail with Intel VAAPI Acceleration

[ccb056]

I set up a ZoneMinder test NVR inside a hardened FreeBSD VNET jail named jailnvr. The goal was to view several Dahua IP cameras from a browser without exposing the cameras directly, and to make use of the Intel iGPU for H.264/H.265 hardware acceleration.

This was done on a Supermicro embedded system with:

Code: Select all

Intel Core i3-7100U
Intel HD Graphics 620
FreeBSD 15.x
VNET jails
Dahua 8MP IP cameras

Final design:

Code: Select all

FreeBSD host
    -> Intel i915kms DRM driver
    -> /dev/dri/renderD128
    -> jailnvr VNET jail
        -> ZoneMinder
        -> nginx
        -> PHP-FPM
        -> fcgiwrap
        -> MySQL 8.4
        -> FFmpeg
        -> Intel VAAPI iHD driver
        -> Dahua RTSP cameras

1. Enable Intel iGPU on the FreeBSD host

The first step was getting the Intel iGPU visible on the FreeBSD host.

Installed the Intel DRM driver and Kaby Lake firmware:

Code: Select all

pkg install drm-kmod gpu-firmware-intel-kmod-kabylake

Loaded the Intel driver:

Code: Select all

kldload i915kms

Verified that /dev/dri appeared:

Code: Select all

ls -l /dev/dri

Expected result:

Code: Select all

card0 -> ../drm/0
renderD128 -> ../drm/128

The important device for VAAPI video acceleration is:

Code: Select all

/dev/dri/renderD128

Made the driver load at boot:

Code: Select all

sysrc kld_list+="i915kms"

2. Create the jailnvr VNET jail

Created a thick jail:

Code: Select all

bsdinstall jail /usr/local/jails/jailnvr

Copied DNS config into the jail:

Code: Select all

cp /etc/resolv.conf /usr/local/jails/jailnvr/etc/resolv.conf

Added jailnvr to:

Code: Select all

/etc/jail.conf

My jail block:

Code: Select all

jailnvr {
    host.hostname = "jailnvr";
    path = "/usr/local/jails/jailnvr";

    vnet;
    vnet.interface = "jailnvr0";

    exec.clean;
    exec.system_user = "root";
    exec.jail_user = "root";

    exec.prestart = "ifconfig epair13 create";
    exec.prestart += "ifconfig epair13a descr jailnvr-host";
    exec.prestart += "ifconfig epair13a up";
    exec.prestart += "ifconfig bridge0 addm epair13a";
    exec.prestart += "ifconfig epair13b name jailnvr0";

    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";

    exec.poststop = "ifconfig bridge0 deletem epair13a";
    exec.poststop += "ifconfig epair13a destroy";

    mount.devfs;
    devfs_ruleset = 13;

    persist;

    allow.raw_sockets = 0;
    allow.set_hostname = 0;

    # Needed for ZoneMinder PHP streaming/semaphores.
    allow.sysvipc = 1;

    allow.mount = 0;
    allow.mount.devfs = 0;
    allow.mount.nullfs = 0;
    allow.mount.procfs = 0;
    allow.mount.tmpfs = 0;

    allow.chflags = 0;
    enforce_statfs = 2;
    children.max = 0;
}

The important difference from my more hardened database/web jails is:

Code: Select all

allow.sysvipc = 1;

Without this, ZoneMinder live view produced errors like:

Code: Select all

Failed to get semaphore for key ...
It is likely that your php does not have the sysv semaphore extension either installed or enabled.

The PHP sysvsem module was installed, but the jail itself also needed SysV IPC enabled.


3. Configure devfs rules for DHCP and GPU passthrough

The jail needs bpf devices for DHCP and /dev/dri for GPU access.

Edited:

Code: Select all

/etc/devfs.rules

Final rules:

Code: Select all

[devfsrules_jaildb=10]
add include $devfsrules_jail
add path 'bpf*' unhide

[devfsrules_jailwww=11]
add include $devfsrules_jail
add path 'bpf*' unhide

[devfsrules_jailmon=12]
add include $devfsrules_jail
add path 'bpf*' unhide

[devfsrules_jailnvr=13]
add include $devfsrules_jail
add path 'bpf*' unhide

add path 'dri' unhide
add path 'dri/*' hide
add path 'dri/card0' unhide mode 0666
add path 'dri/renderD128' unhide mode 0666

add path 'drm' unhide
add path 'drm/*' hide
add path 'drm/0' unhide mode 0666
add path 'drm/128' unhide mode 0666

Reloaded devfs:

Code: Select all

service devfs restart

Inside the jail, verified:

Code: Select all

jexec jailnvr ls -l /dev/dri
jexec jailnvr ls -l /dev/drm

Expected:

Code: Select all

/dev/dri/card0 -> ../drm/0
/dev/dri/renderD128 -> ../drm/128

/dev/drm/0
/dev/drm/128

Originally I exposed all of /dev/drm/*, but narrowed it down to only the two nodes required by the symlinks.


4. Configure jailnvr rc.conf

Edited:

Code: Select all

/usr/local/jails/jailnvr/etc/rc.conf

Contents:

Code: Select all

hostname="jailnvr"

ifconfig_jailnvr0="SYNCDHCP"

sshd_enable="NO"
sendmail_enable="NONE"
syslogd_flags="-ss"
clear_tmp_enable="YES"

dumpdev="NO"
moused_nondefault_enable="NO"

Added the jail to the host jail list:

Code: Select all

sysrc jail_list+=" jailnvr"

Started jail:

Code: Select all

service jail start jailnvr

Verified:

Code: Select all

jls
jexec jailnvr hostname
jexec jailnvr ifconfig jailnvr0

The jail received:

Code: Select all

192.168.11.236

5. Install FFmpeg, VAAPI tools, and Intel media driver inside jailnvr

Entered the jail:

Code: Select all

jexec jailnvr /bin/sh

Installed FFmpeg and VAAPI tools:

Code: Select all

pkg install ffmpeg libva-utils

Installed the Intel VAAPI media driver:

Code: Select all

pkg install libva-intel-media-driver

I originally tested both the newer iHD driver and the legacy i965 driver. Both worked, but I kept the newer iHD driver and removed the legacy driver later.

Verified FFmpeg hardware acceleration support:

Code: Select all

ffmpeg -hide_banner -hwaccels

Output included:

Code: Select all

vaapi
drm

Verified VAAPI with iHD:

Code: Select all

env LIBVA_DRIVER_NAME=iHD vainfo --display drm --device /dev/dri/renderD128

This showed support for H.264 and HEVC/H.265 profiles, including HEVC Main and Main10.


6. Test Dahua RTSP video acceleration

Tested one Dahua 4K H.265 main stream:

Code: Select all

env LIBVA_DRIVER_NAME=iHD ffmpeg \
  -hide_banner \
  -rtsp_transport tcp \
  -hwaccel vaapi \
  -hwaccel_device /dev/dri/renderD128 \
  -i 'rtsp://USER:[email protected]:554/cam/realmonitor?channel=1&subtype=0' \
  -f null -

This worked, but CPU was around 33%.

Then tested with VAAPI output surfaces:

Code: Select all

env LIBVA_DRIVER_NAME=iHD ffmpeg \
  -hide_banner \
  -rtsp_transport tcp \
  -hwaccel vaapi \
  -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi \
  -i 'rtsp://USER:[email protected]:554/cam/realmonitor?channel=1&subtype=0' \
  -f null -

CPU dropped to around 2%.

Software decode was around 90% CPU.

So the VAAPI path was definitely working.


7. Install ZoneMinder

Installed ZoneMinder first and let pkg pull in dependencies:

Code: Select all

pkg install zoneminder-php84

Checked installed packages:

Code: Select all

pkg info | egrep -i 'zoneminder|nginx|php84|mysql84|fcgiwrap|ffmpeg|libva'

ZoneMinder pulled in PHP and MySQL client pieces, but not the full web/database stack I wanted.

Installed the missing pieces:

Code: Select all

pkg install nginx fcgiwrap mysql84-server

Enabled services:

Code: Select all

sysrc mysql_enable=YES
sysrc nginx_enable=YES
sysrc php_fpm_enable=YES
sysrc fcgiwrap_enable=YES
sysrc zoneminder_enable=YES

Started MySQL:

Code: Select all

service mysql-server start

Verified:

Code: Select all

mysql -e "SELECT VERSION();"

8. Create ZoneMinder database

Located the schema:

Code: Select all

find /usr/local -name 'zm_create.sql'

Result:

Code: Select all

/usr/local/share/zoneminder/db/zm_create.sql

Created database and user:

Code: Select all

mysql <<'EOF'
CREATE DATABASE zm CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'zmuser'@'localhost' IDENTIFIED BY 'REDACTED_PASSWORD';
GRANT ALL PRIVILEGES ON zm.* TO 'zmuser'@'localhost';
FLUSH PRIVILEGES;
EOF

Imported schema:

Code: Select all

mysql zm < /usr/local/share/zoneminder/db/zm_create.sql

Verified tables:

Code: Select all

mysql zm -e "SHOW TABLES;"

Verified user access:

Code: Select all

mysql -u zmuser -p zm -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='zm';"

9. Configure ZoneMinder database settings

Edited:

Code: Select all

/usr/local/etc/zm.conf

Set:

Code: Select all

ZM_DB_HOST=localhost
ZM_DB_NAME=zm
ZM_DB_USER=zmuser
ZM_DB_PASS=REDACTED_PASSWORD

ZM_WEB_USER=www
ZM_WEB_GROUP=www

Locked down permissions:

Code: Select all

chown root:www /usr/local/etc/zm.conf
chmod 640 /usr/local/etc/zm.conf

10. Configure PHP

Copied production PHP config:

Code: Select all

cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

Edited:

Code: Select all

/usr/local/etc/php.ini

Set:

Code: Select all

date.timezone = America/Chicago
memory_limit = 512M
upload_max_filesize = 64M
post_max_size = 64M
cgi.fix_pathinfo = 0

Started PHP-FPM:

Code: Select all

service php_fpm start

PHP-FPM listened on:

Code: Select all

127.0.0.1:9000

11. Configure fcgiwrap to run as www

Initially fcgiwrap ran as nobody, which caused issues with ZoneMinder streaming.

Set rc variables:

Code: Select all

sysrc fcgiwrap_user=www
sysrc fcgiwrap_group=www
sysrc fcgiwrap_socket_owner=www
sysrc fcgiwrap_socket_group=www
sysrc fcgiwrap_socket_mode=0775

Restarted:

Code: Select all

service fcgiwrap restart

Verified:

Code: Select all

sockstat | grep fcgiwrap
ls -l /var/run/fcgiwrap/fcgiwrap.sock

Expected:

Code: Select all

www fcgiwrap ...
/var/run/fcgiwrap/fcgiwrap.sock owned by www

12. Configure nginx

The final working nginx config used:

Code: Select all

user www;

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile on;
    keepalive_timeout 65;

    error_log /var/log/nginx-zm-error.log info;
    access_log /var/log/nginx-zm-access.log;

    server {
        listen 80;
        server_name jailnvr 192.168.11.236;

        location = / {
            return 302 /zm/;
        }

        location /zm/cache/ {
            alias /var/cache/zoneminder/;
        }

        location /zm/ {
            alias /usr/local/www/zoneminder/;
            index index.php index.html;
            try_files $uri $uri/ /zm/index.php?$query_string;
        }

        location ~ ^/zm/(.+\.php)$ {
            alias /usr/local/www/zoneminder/$1;

            if (!-f $request_filename) {
                return 404;
            }

            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $request_filename;
            fastcgi_param SCRIPT_NAME /zm/$1;
            fastcgi_index index.php;
            fastcgi_pass 127.0.0.1:9000;
        }

        location ~ ^/cgi-bin/(.*)$ {
            gzip off;
            fastcgi_buffering off;

            include fastcgi_params;

            fastcgi_param SCRIPT_FILENAME /usr/local/www/zoneminder/cgi-bin/$1;
            fastcgi_param SCRIPT_NAME /cgi-bin/$1;
            fastcgi_param QUERY_STRING $query_string;
            fastcgi_param REQUEST_METHOD $request_method;
            fastcgi_param CONTENT_TYPE $content_type;
            fastcgi_param CONTENT_LENGTH $content_length;

            fastcgi_read_timeout 300;

            fastcgi_pass unix:/var/run/fcgiwrap/fcgiwrap.sock;
        }

        location ~ /\.(ht|git|svn) {
            deny all;
        }
    }
}

Important note:

ZoneMinder’s console thumbnails requested:

Code: Select all

/cgi-bin/nph-zms

not:

Code: Select all

/zm/cgi-bin/nph-zms

The nginx error log showed:

Code: Select all

open() "/usr/local/etc/nginx/html/cgi-bin/nph-zms" failed

Adding the /cgi-bin/ block fixed thumbnails.

Tested nginx:

Code: Select all

nginx -t

Restarted:

Code: Select all

service nginx restart

13. Start ZoneMinder

Started ZoneMinder:

Code: Select all

service zoneminder start

Checked status:

Code: Select all

service zoneminder status

Checked processes:

Code: Select all

ps aux | grep -i '[z]m'

Opened web UI:

Code: Select all

http://jailnvr/zm/

The web GUI loaded successfully.


14. Camera setup

Added Dahua cameras using the main stream:

Code: Select all

rtsp://CAMERA_IP:554/cam/realmonitor?channel=1&subtype=0

Username/password were placed in the monitor Source tab rather than embedded in the URL.

For my use case I decided I did not want recording or motion detection, only viewing.

Final style of monitor settings:

Code: Select all

Function: Monitor
Enabled: 1
Capturing: Always
Analysing: None
Recording: None
Decoding: KeyFrames+Ondemand
Decoder: hevc_vaapi
DecoderHWAccelName: vaapi
DecoderHWAccelDevice: /dev/dri/renderD128
Method: TCP
Width: 3840
Height: 2160

Verified monitor settings from MySQL:

Code: Select all

mysql zm -e 'SELECT Id,Name,`Function`,Enabled,Capturing,Analysing,Recording,Decoding,Width,Height FROM Monitors;'

For view-only use, the important fields are:

Code: Select all

Analysing = None
Recording = None

That prevents motion detection and recording.

Confirmed disk was not growing:

Code: Select all

du -sh /var/db/zoneminder

Result was tiny:

Code: Select all

13K /var/db/zoneminder

15. Problems encountered

Problem: /dev/dri missing on host

Fix:

Code: Select all

pkg install drm-kmod gpu-firmware-intel-kmod-kabylake
kldload i915kms
ls -l /dev/dri

Then expose /dev/dri and /dev/drm to the jail with devfs rules.


Problem: VAAPI failed

Initial vainfo output showed:

Code: Select all

va_openDriver() returns -1

Fix:

Code: Select all

pkg install libva-intel-media-driver

Then:

Code: Select all

env LIBVA_DRIVER_NAME=iHD vainfo --display drm --device /dev/dri/renderD128

Problem: thumbnail/live stream did not work

Symptoms:

Code: Select all

ZoneMinder UI worked but thumbnails did not.
Logs showed nph-zms / zms socket errors.

Fixes:

Code: Select all

1. Run fcgiwrap as www.
2. Enable allow.sysvipc=1 in jailnvr.
3. Add nginx location for /cgi-bin/nph-zms.

Problem: PHP semaphore errors

Log:

Code: Select all

Failed to get semaphore for key ...
It is likely that your php does not have the sysv semaphore extension either installed or enabled.

Fix:

Code: Select all

php84-sysvsem was already installed.
The real issue was the jail itself:

allow.sysvipc = 1;

After restarting the jail, sysvsem worked.


Problem: ZoneMinder requested /cgi-bin/nph-zms, not /zm/cgi-bin/nph-zms

nginx log:

Code: Select all

GET /cgi-bin/nph-zms?scale=1&mode=jpeg&frames=1&monitor=1...

Fix:

Code: Select all

location ~ ^/cgi-bin/(.*)$ {
    gzip off;
    fastcgi_buffering off;

    include fastcgi_params;

    fastcgi_param SCRIPT_FILENAME /usr/local/www/zoneminder/cgi-bin/$1;
    fastcgi_param SCRIPT_NAME /cgi-bin/$1;
    fastcgi_param QUERY_STRING $query_string;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param CONTENT_TYPE $content_type;
    fastcgi_param CONTENT_LENGTH $content_length;

    fastcgi_read_timeout 300;

    fastcgi_pass unix:/var/run/fcgiwrap/fcgiwrap.sock;
}

16. Performance notes

With 7 Dahua 8MP main streams configured for viewing only, top showed the system mostly idle:

Code: Select all

CPU roughly 90% idle
Each zmc process around 1-2% CPU
No swap usage
No meaningful storage growth

When actively viewing a 4K camera in the browser, CPU usage rises significantly because ZoneMinder streams through zms as JPEG/MJPEG-style frames.

This is different from the raw FFmpeg VAAPI test, where decoded frames stayed on the GPU.

The best FFmpeg hardware acceleration test was:

Code: Select all

env LIBVA_DRIVER_NAME=iHD ffmpeg \
  -hide_banner \
  -rtsp_transport tcp \
  -hwaccel vaapi \
  -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi \
  -i 'rtsp://USER:PASSWORD@CAMERA_IP:554/cam/realmonitor?channel=1&subtype=0' \
  -f null -

That used very little CPU.

ZoneMinder live browser viewing is heavier because it must produce web-viewable output.


17. Current final state

The final working stack:

Code: Select all

FreeBSD host
    i915kms loaded
    /dev/dri/renderD128 available

jailnvr
    VNET jail
    DHCP address 192.168.11.236
    /dev/dri/renderD128 exposed
    ZoneMinder 1.38.1
    nginx
    PHP-FPM
    fcgiwrap running as www
    MySQL 8.4
    FFmpeg 8.1
    Intel iHD VAAPI driver
    7 Dahua cameras configured

ZoneMinder is currently being used as a browser-based camera viewer, not a recorder.

Final view-only monitor concept:

Code: Select all

Capturing: Always
Analysing: None
Recording: None
Decoding: KeyFrames+Ondemand

This gives working thumbnails and live view without writing video to disk.