meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
vm:proxmox:lxc:google_coral [2024/12/07 20:07] – created niziakvm:proxmox:lxc:google_coral [2024/12/08 16:06] (current) – niziak
Line 1: Line 1:
-====== Google Coral passthrough ======+====== Google Coral USB in LXC ======
  
 to Frigate container to Frigate container
  
 +===== Google Coral =====
 +
 +After power-up Google Coral is in boot mode:
  
 <code bash> <code bash>
-# lsusb 
 Bus 002 Device 005: ID 1a6e:089a Global Unichip Corp.  Bus 002 Device 005: ID 1a6e:089a Global Unichip Corp. 
 </code> </code>
  
 +It needs software to run. So Proxmox host or LXC must upload firmware.
 +After successful init Google Coral changes its USB id to:
 +<code bash>
 +Bus 002 Device 006: ID 18d1:9302 Google Inc. 
 +</code>
 +
 +Frigate contains Coral firmware and can init it.
 +
 +===== Issues to solve =====
 +
 +Coral:
 +  * Frigate needs access to e.g.: ''/dev/bus/usb/002/005''
 +  * Most solution on Internet provides workarounds:
 +    * ugly and unsafe solution like:''chmod 666'' by udev and unprivileged LXC
 +    * still unsafe using ''lxc.idmap'' to give access from LXC to devices in ''video'' and ''render'' group (I don't want to give so wide access).
 +    * one time solution - like ''chown /dev/bus/usb/002/*'' before container start by LXC hook.
 +      * this solution cannot survive lots of scenarios (udev reload, usb reset, usb cable reconnect)
 +
 +iGPU acceleration:
 +  * Frigate needs access to ''/dev/dri/renderD128'' - givin access to LXC simply works
 +
 +iGPU performance monitors:
 +
 +
 +===== Setup =====
 +
 +==== docker-compose ====
 +
 +<file yaml docker-compose.yml>
 +services:
 +  frigate:
 +    privileged: true  # still in unpriv LXC so don't care
 +    devices:
 +      - /dev/bus/usb:/dev/bus/usb  # Passes the USB bus
 +      - /dev/dri/renderD128:/dev/dri/renderD128  # For intel hwaccel
 +</file>
 +
 +==== LXC ====
  
 <file ini /etc/pve/lxc/307.conf> <file ini /etc/pve/lxc/307.conf>
-features: nesting=1+features: fuse=1, nesting=1
 unprivileged: 1 unprivileged: 1
 lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU
Line 17: Line 57:
 lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU
 lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render) lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render)
-lxc.hook.pre-start: sh -c "chown 100000:111000 /dev/dri/renderD128" # create a host gid for lxc_gpu_shares +</file>
-lxc.hook.pre-start: sh -c "chown -R 100000:111002 /dev/bus/usb/002" # create a host gid for lxc_usb2_shares+
  
 +In Frigate's LXC shell:
 +<code bash>
 +groupadd -g 11000 lxc_gpu_shares
 +groupadd -g 11002 lxc_usb2_shares
 +usermod -aG lxc_gpu_shares,lxc_usb2_shares root
 +</code>
 +
 +==== Host ====
 +
 +On Proxmox host:
 +
 +<file ini /etc/udev/rules.d/71-coral.rules>
 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="1a6e", ATTRS{idProduct}=="089a", MODE="0664", OWNER="100000", GROUP="111002"
 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9302", MODE="0664", OWNER="100000", GROUP="111002"
 </file> </file>
 +
 +<file ini /etc/udev/rules.d/99-igpu.rules>
 +KERNEL=="renderD128", MODE="0664", OWNER="100000", GROUP="111000"
 +</file>
 +
 +<code bash>
 +udevadm control --reload-rules && udevadm trigger
 +</code>
 +
 +
 +===== ERROR: Unable to poll intel GPU stats: Failed to initialize PMU! (Permission denied) =====
 +
 +This error is "informational".
 +Rendering on ''/dev/dri/renderD128'' **works** but container lacks the necessary permissions to access the performance monitoring unit. 
 +Frigate wants this to show GPU usage statistics.
 +
 +Reproduce in LXC:
 +<code bash>
 +# apt install intel-gpu-tools
 +# intel_gpu_top
 +Failed to initialize PMU! (Permission denied)
 +</code>
 +
 +Host Kernel prevents access to performance events subsystem for unprivileged users. Security level is controller by sysctl:
 +<code bash>
 +# sysctl kernel.perf_event_paranoid
 +kernel.perf_event_paranoid = 4
 +</code>
 +
 +where values:
 +  * ''-1'' Allow use of (almost) all events by all users. Ignore mlock limit after ''perf_event_mlock_kb'' without ''CAP_IPC_LOCK''
 +  * ''>=0'' Disallow ftrace function tracepoint by users without ''CAP_SYS_ADMIN''. Disallow raw tracepoint access by users without ''CAP_SYS_ADMIN''
 +  * ''>=1'' Disallow CPU event access by users without ''CAP_SYS_ADMIN''
 +  * ''>=2'' Disallow kernel profiling by users without ''CAP_SYS_ADMIN''
 +
 +<code bash>
 +sysctl --write kernel.perf_event_paranoid=NEWVALUE
 +</code>
 +
 +I suppose this is not possible to use unpriv LXC and enable ''CAP_SYS_ADMIN'' (multiple trials with ''lxc.cap.drop'' and ''lxc.cap.keep'').
 +It works when <code bash>sysctl --write kernel.perf_event_paranoid=0</code> which is still safer (there were kernel vulnerabilities) than giving container ''CAP_SYS_ADMIN''.
 +
 +When Frigate is running and performance events are accessible in LXC, command <code bash>intel_gpu_top</code> show nicely formatted text statistics:
 +
 +{{:vm:proxmox:lxc:pasted:20241208-144931.png}}
 +
 +===== references =====
 +
 +  * [[https://github.com/Bytelake/Coral-in-LXC|Coral-in-LXC]]
 +  * [[https://www.reddit.com/r/frigate_nvr/comments/1cr9akm/frigate_inside_lxc_on_proxmox_google_coral_usb/|Frigate inside LXC on Proxmox [Google Coral USB, iGPU passthrough] ]]
 +  * [[https://github.com/google-coral/edgetpu/issues/536|Coral USB changing ID and Vendor
 +#536]]
 +
 +
 +