meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
vm:proxmox:lxc:google_coral [2024/12/07 21:37] – niziakvm:proxmox:lxc:google_coral [2024/12/08 16:06] (current) – niziak
Line 11: Line 11:
 </code> </code>
  
-It needs software to run. So host or LXC must upload firmware. +It needs software to run. So Proxmox host or LXC must upload firmware. 
-After successfull init Google Coral changes its USB id to:+After successful init Google Coral changes its USB id to:
 <code bash> <code bash>
 Bus 002 Device 006: ID 18d1:9302 Google Inc.  Bus 002 Device 006: ID 18d1:9302 Google Inc. 
 </code> </code>
  
-Frigate contains Coral firmware and if access from LXC to USB dev is given it can init Google Coral upon startup.+Frigate contains Coral firmware and can init it.
  
-===== docker-compose =====+===== Issues to solve ===== 
 + 
 +Coral: 
 +  * Frigate needs access to e.g.: ''/dev/bus/usb/002/005'' 
 +  * Most solution on Internet provides workarounds: 
 +    * ugly and unsafe solution like:''chmod 666'' by udev and unprivileged LXC 
 +    * still unsafe using ''lxc.idmap'' to give access from LXC to devices in ''video'' and ''render'' group (I don't want to give so wide access). 
 +    * one time solution - like ''chown /dev/bus/usb/002/*'' before container start by LXC hook. 
 +      * this solution cannot survive lots of scenarios (udev reload, usb reset, usb cable reconnect) 
 + 
 +iGPU acceleration: 
 +  * Frigate needs access to ''/dev/dri/renderD128'' - givin access to LXC simply works 
 + 
 +iGPU performance monitors: 
 + 
 + 
 +===== Setup ===== 
 + 
 +==== docker-compose ====
  
 <file yaml docker-compose.yml> <file yaml docker-compose.yml>
 +services:
 +  frigate:
 +    privileged: true  # still in unpriv LXC so don't care
     devices:     devices:
-      - /dev/bus/usb:/dev/bus/usb # Passes the USB Coral, needs to be modified for other versions +      - /dev/bus/usb:/dev/bus/usb  # Passes the USB bus 
-      - /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware+      - /dev/dri/renderD128:/dev/dri/renderD128  # For intel hwaccel
 </file> </file>
  
-===== LXC config =====+==== LXC ====
  
 <file ini /etc/pve/lxc/307.conf> <file ini /etc/pve/lxc/307.conf>
-features: nesting=1+features: fuse=1, nesting=1
 unprivileged: 1 unprivileged: 1
 lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU
Line 36: Line 57:
 lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU
 lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render) lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render)
-lxc.hook.pre-start: sh -c "chown 100000:111000 /dev/dri/renderD128" # create a host gid for lxc_gpu_shares 
-lxc.hook.pre-start: sh -c "chown -R 100000:111002 /dev/bus/usb/002" # create a host gid for lxc_usb2_shares 
 </file> </file>
  
-In LXC shell:+In Frigate's LXC shell:
 <code bash> <code bash>
 groupadd -g 11000 lxc_gpu_shares groupadd -g 11000 lxc_gpu_shares
Line 47: Line 66:
 </code> </code>
  
-Issue: +==== Host ====
-  * ''lxc.hook'' changes permission only once just before LXC starts. LXC must be restarted to trigger ''lxc.hook'' Drawback: +
-    * Any further host USB reconnects / udevadm reload cause USB permisions change to default.  +
-    * any reinit of Coral device cause of the same device to appear on the same bus with new device ID, and new default permissions. +
- +
-Fix for issue: +
-  * use udev to always set correct permissions:+
  
 On Proxmox host: On Proxmox host:
  
-<file /etc/udev/rules.d/71-coral.rules> +<file ini /etc/udev/rules.d/71-coral.rules> 
-SUBSYSTEMS=="usb", ATTRS{idVendor}=="1a6e", ATTRS{idProduct}=="089a", MODE="0664", USER="100000", GROUP="111002" +SUBSYSTEMS=="usb", ATTRS{idVendor}=="1a6e", ATTRS{idProduct}=="089a", MODE="0664", OWNER="100000", GROUP="111002" 
-SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9302", MODE="0664", USER="100000", GROUP="111002"+SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9302", MODE="0664", OWNER="100000", GROUP="111002"
 </file> </file>
  
-<file /etc/udev/rules.d/99-igpuchmod.rules> +<file ini /etc/udev/rules.d/99-igpu.rules> 
-KERNEL=="renderD128", MODE="0664", USER="100000", GROUP="111000"+KERNEL=="renderD128", MODE="0664", OWNER="100000", GROUP="111000"
 </file> </file>
  
Line 70: Line 83:
 </code> </code>
  
 +
 +===== ERROR: Unable to poll intel GPU stats: Failed to initialize PMU! (Permission denied) =====
 +
 +This error is "informational".
 +Rendering on ''/dev/dri/renderD128'' **works** but container lacks the necessary permissions to access the performance monitoring unit. 
 +Frigate wants this to show GPU usage statistics.
 +
 +Reproduce in LXC:
 +<code bash>
 +# apt install intel-gpu-tools
 +# intel_gpu_top
 +Failed to initialize PMU! (Permission denied)
 +</code>
 +
 +Host Kernel prevents access to performance events subsystem for unprivileged users. Security level is controller by sysctl:
 +<code bash>
 +# sysctl kernel.perf_event_paranoid
 +kernel.perf_event_paranoid = 4
 +</code>
 +
 +where values:
 +  * ''-1'' Allow use of (almost) all events by all users. Ignore mlock limit after ''perf_event_mlock_kb'' without ''CAP_IPC_LOCK''
 +  * ''>=0'' Disallow ftrace function tracepoint by users without ''CAP_SYS_ADMIN''. Disallow raw tracepoint access by users without ''CAP_SYS_ADMIN''
 +  * ''>=1'' Disallow CPU event access by users without ''CAP_SYS_ADMIN''
 +  * ''>=2'' Disallow kernel profiling by users without ''CAP_SYS_ADMIN''
 +
 +<code bash>
 +sysctl --write kernel.perf_event_paranoid=NEWVALUE
 +</code>
 +
 +I suppose this is not possible to use unpriv LXC and enable ''CAP_SYS_ADMIN'' (multiple trials with ''lxc.cap.drop'' and ''lxc.cap.keep'').
 +It works when <code bash>sysctl --write kernel.perf_event_paranoid=0</code> which is still safer (there were kernel vulnerabilities) than giving container ''CAP_SYS_ADMIN''.
 +
 +When Frigate is running and performance events are accessible in LXC, command <code bash>intel_gpu_top</code> show nicely formatted text statistics:
 +
 +{{:vm:proxmox:lxc:pasted:20241208-144931.png}}
  
 ===== references ===== ===== references =====