meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
vm:proxmox:lxc:google_coral [2024/12/07 21:08] – niziakvm:proxmox:lxc:google_coral [2024/12/08 16:06] (current) – niziak
Line 11: Line 11:
 </code> </code>
  
-It needs software to run. So host or LXC must upload firmware. +It needs software to run. So Proxmox host or LXC must upload firmware. 
-After successfull init Google Coral changes its USB id to:+After successful init Google Coral changes its USB id to:
 <code bash> <code bash>
 Bus 002 Device 006: ID 18d1:9302 Google Inc.  Bus 002 Device 006: ID 18d1:9302 Google Inc. 
 </code> </code>
  
-Frigate contains Coral firmware and if access from LXC to USB dev is given it can init Google Coral upon startup.+Frigate contains Coral firmware and can init it.
  
-===== LXC config =====+===== Issues to solve ===== 
 + 
 +Coral: 
 +  * Frigate needs access to e.g.: ''/dev/bus/usb/002/005'' 
 +  * Most solution on Internet provides workarounds: 
 +    * ugly and unsafe solution like:''chmod 666'' by udev and unprivileged LXC 
 +    * still unsafe using ''lxc.idmap'' to give access from LXC to devices in ''video'' and ''render'' group (I don't want to give so wide access). 
 +    * one time solution - like ''chown /dev/bus/usb/002/*'' before container start by LXC hook. 
 +      * this solution cannot survive lots of scenarios (udev reload, usb reset, usb cable reconnect) 
 + 
 +iGPU acceleration: 
 +  * Frigate needs access to ''/dev/dri/renderD128'' - givin access to LXC simply works 
 + 
 +iGPU performance monitors: 
 + 
 + 
 +===== Setup ===== 
 + 
 +==== docker-compose ==== 
 + 
 +<file yaml docker-compose.yml> 
 +services: 
 +  frigate: 
 +    privileged: true  # still in unpriv LXC so don't care 
 +    devices: 
 +      - /dev/bus/usb:/dev/bus/usb  # Passes the USB bus 
 +      - /dev/dri/renderD128:/dev/dri/renderD128  # For intel hwaccel 
 +</file> 
 + 
 +==== LXC ====
  
 <file ini /etc/pve/lxc/307.conf> <file ini /etc/pve/lxc/307.conf>
-features: nesting=1+features: fuse=1, nesting=1
 unprivileged: 1 unprivileged: 1
 lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU lxc.cgroup2.devices.allow: c 226:128 rwm # iGPU
Line 28: Line 57:
 lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir,mode=664 # USB Coral TPU
 lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render) lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0 # iGPU (u=root g=render)
-lxc.hook.pre-start: sh -c "chown 100000:111000 /dev/dri/renderD128" # create a host gid for lxc_gpu_shares 
-lxc.hook.pre-start: sh -c "chown -R 100000:111002 /dev/bus/usb/002" # create a host gid for lxc_usb2_shares 
 </file> </file>
  
-Issues: +In Frigate's LXC shell: 
-  * ''lxc.hook'' changes permission only once just before LXC starts. Any further host USB reconnects / udevadm reload cause USB permisions change. LXC must be restarted to trigger ''lxc.hook''+<code bash> 
 +groupadd -g 11000 lxc_gpu_shares 
 +groupadd -g 11002 lxc_usb2_shares 
 +usermod -aG lxc_gpu_shares,lxc_usb2_shares root 
 +</code> 
 + 
 +==== Host ==== 
 + 
 +On Proxmox host: 
 + 
 +<file ini /etc/udev/rules.d/71-coral.rules> 
 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="1a6e", ATTRS{idProduct}=="089a", MODE="0664", OWNER="100000", GROUP="111002" 
 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9302", MODE="0664", OWNER="100000", GROUP="111002" 
 +</file> 
 + 
 +<file ini /etc/udev/rules.d/99-igpu.rules> 
 +KERNEL=="renderD128", MODE="0664", OWNER="100000", GROUP="111000" 
 +</file> 
 + 
 +<code bash> 
 +udevadm control --reload-rules && udevadm trigger 
 +</code> 
 + 
 + 
 +===== ERROR: Unable to poll intel GPU stats: Failed to initialize PMU! (Permission denied) ===== 
 + 
 +This error is "informational". 
 +Rendering on ''/dev/dri/renderD128'' **works** but container lacks the necessary permissions to access the performance monitoring unit.  
 +Frigate wants this to show GPU usage statistics. 
 + 
 +Reproduce in LXC: 
 +<code bash> 
 +# apt install intel-gpu-tools 
 +# intel_gpu_top 
 +Failed to initialize PMU! (Permission denied) 
 +</code> 
 + 
 +Host Kernel prevents access to performance events subsystem for unprivileged users. Security level is controller by sysctl: 
 +<code bash> 
 +# sysctl kernel.perf_event_paranoid 
 +kernel.perf_event_paranoid = 4 
 +</code> 
 + 
 +where values: 
 +  * ''-1'' Allow use of (almost) all events by all users. Ignore mlock limit after ''perf_event_mlock_kb'' without ''CAP_IPC_LOCK'' 
 +  * ''>=0'' Disallow ftrace function tracepoint by users without ''CAP_SYS_ADMIN''. Disallow raw tracepoint access by users without ''CAP_SYS_ADMIN'' 
 +  * ''>=1'' Disallow CPU event access by users without ''CAP_SYS_ADMIN'' 
 +  * ''>=2'' Disallow kernel profiling by users without ''CAP_SYS_ADMIN'' 
 + 
 +<code bash> 
 +sysctl --write kernel.perf_event_paranoid=NEWVALUE 
 +</code> 
 + 
 +I suppose this is not possible to use unpriv LXC and enable ''CAP_SYS_ADMIN'' (multiple trials with ''lxc.cap.drop'' and ''lxc.cap.keep''). 
 +It works when <code bash>sysctl --write kernel.perf_event_paranoid=0</code> which is still safer (there were kernel vulnerabilities) than giving container ''CAP_SYS_ADMIN''. 
 + 
 +When Frigate is running and performance events are accessible in LXC, command <code bash>intel_gpu_top</code> show nicely formatted text statistics: 
 + 
 +{{:vm:proxmox:lxc:pasted:20241208-144931.png}} 
 + 
 +===== references ===== 
 + 
 +  * [[https://github.com/Bytelake/Coral-in-LXC|Coral-in-LXC]] 
 +  * [[https://www.reddit.com/r/frigate_nvr/comments/1cr9akm/frigate_inside_lxc_on_proxmox_google_coral_usb/|Frigate inside LXC on Proxmox [Google Coral USB, iGPU passthrough] ]] 
 +  * [[https://github.com/google-coral/edgetpu/issues/536|Coral USB changing ID and Vendor 
 +#536]]