meta data for this page
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| vm:proxmox:ceph:performance [2023/07/03 20:32] – niziak | vm:proxmox:ceph:performance [2025/01/08 19:05] (current) – niziak | ||
|---|---|---|---|
| Line 8: | Line 8: | ||
| ===== Performance tips ===== | ===== Performance tips ===== | ||
| + | |||
| + | Ceph is build for scale and works great in large clusters. In small cluster every node will be heavily loaded. | ||
| + | |||
| * adapt PG to number of OSDs to spread traffic evenly | * adapt PG to number of OSDs to spread traffic evenly | ||
| * use '' | * use '' | ||
| * enable '' | * enable '' | ||
| + | ==== performance on small cluster ==== | ||
| + | * [[https:// | ||
| + | * number of PG should be power of 2 (or middle between powers of 2) | ||
| + | * same utilization (% full) per device | ||
| + | * same number of PG per OSD := same number of request per device | ||
| + | * same number of primary PG per OSD = read operations spread evenly | ||
| + | * primary PG - original/ | ||
| + | * use relatively more PG than for big cluster - better balance, but handling PGs consumes resources (RAM) | ||
| + | * i.e. for 7 OSD x 2TB PG autoscaler recommends 256 PG. After changing to 384 IOops drastivally increases and latency drops. | ||
| + | Setting to 512 PG wasn't possible because limit of 250PG/OSD. | ||
| - | ==== Issues ==== | + | === balancer |
| - | === auth: unable to find a keyring === | + | <code bash> |
| - | + | ceph mgr module enable balancer | |
| - | It is not possible to create ceph OSD neither from WebUI nor cmdline: | + | ceph balancer |
| - | + | ceph balancer mode upmap | |
| - | < | + | |
| - | Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring / | + | |
| - | | + | |
| - | 2021-01-28T10: | + | |
| </ | </ | ||
| - | <file init / | + | === CRUSH reweight === |
| - | [client] | + | |
| - | | + | |
| - | [mds] | + | If possible use '' |
| - | | + | |
| - | </ | + | |
| - | ceph.conf Variables | + | Override default CRUSH assignment. |
| - | * **$cluster** - cluster name. For proxmox it is '' | + | |
| - | * **$type** - daemon process '' | + | |
| - | * **$id** - daemon or client indentifier. For '' | + | |
| - | * **$host** - hostname where the process is running | + | |
| - | * **$name** - Expands to $type.$id. I.e: '' | + | |
| - | * **$pid** - Expands to daemon pid | + | |
| - | **SOLUTION: | ||
| - | <code bash>cp / | ||
| - | alternative to try: change ceph.conf | ||
| - | === Unit -.mount is masked. | + | === PG autoscaler |
| - | < | + | Better to use in warn mode, to do not put unexpected load when PG number will change. |
| - | Running command: / | + | < |
| - | stderr: Failed to start ceph-osd@2.service: Unit -.mount is masked. | + | ceph mgr module enable pg_autoscaler |
| - | --> | + | #ceph osd pool set <pool> pg_autoscale_mode < |
| + | ceph osd pool set rbd pg_autoscale_mode warn | ||
| </ | </ | ||
| - | It was caused by '' | + | It is possible to set desired/target size of pool. This prevents autoscaler to move data every time new data are stored. |
| - | * [[https:// | + | |
| - | * [[https://bugs.debian.org/ | + | |
| - | * [[https:// | + | |
| - | **Solution: | + | ==== check cluster balance ==== |
| - | <code bash>systemctl | + | <code bash> |
| + | ceph -s | ||
| + | ceph osd df # shows standard deviation | ||
| + | </ | ||
| - | To list runtime masked units: | + | no tools to show primary PG balancing. Tool on https://github.com/JoshSalomon/Cephalocon-2019/blob/master/pool_pgs_osd.sh |
| - | <code bash>ls -l /var/run/systemd/system | grep mount | grep '/dev/null' | cut -d ' ' -f 11</code> | + | |
| - | To unescape systemd unit names: | ||
| - | <code bash> systemd-escape -u ' | ||
| + | ==== performance on slow HDDs ==== | ||