apt-get install cryptsetup-bin

Enable HW acceleration. Which is a bit slower than software :P

NOTE: From Kernel 4.2 cesa driver was completely rewritten to support DMA, and old mv_cesa driver was removed in kernel 4.15

modprobe mv_cesa
cat /proc/crypto | grep mv_cesa -B 2 -A 7

Is providing only:

1. hmac(sha1)
2. sha1
3. cbc(aes)
4. ecb(aes)

There are also additional kernel modules optimised for ARM:

1. sha1_arm
2. aes_arm

modprobe mv_cesa
cat /proc/crypto | grep cesa -B 2 -A 7

Is providing:

1. hmac(sha1)
2. hmac(md5)
3. sha1
4. md5
5. cbc(aes)
6. ecb(aes)
7. cbc(des3_ede)
8. ecb(des3_ede)
9. cbc(des)
10. ecb(des)

There are also additional kernel modules optimised for ARM:

1. sha1_arm
2. aes_arm

CESA:

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 512MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=1304KiB/s,w=424KiB/s][r=326,w=106 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=13292: Mon Dec 14 09:40:05 2020
  read: IOPS=119, BW=479KiB/s (491kB/s)(384MiB/820117msec)
   bw (  KiB/s): min=  200, max= 1544, per=100.00%, avg=479.15, stdev=86.28, samples=1640
   iops        : min=   50, max=  386, avg=119.71, stdev=21.58, samples=1640
  write: IOPS=39, BW=160KiB/s (164kB/s)(128MiB/820117msec); 0 zone resets
   bw (  KiB/s): min=   31, max=  464, per=100.00%, avg=159.60, stdev=45.77, samples=1640
   iops        : min=    7, max=  116, avg=39.82, stdev=11.45, samples=1640
  cpu          : usr=0.63%, sys=2.77%, ctx=171146, majf=2, minf=18
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98308,32764,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=479KiB/s (491kB/s), 479KiB/s-479KiB/s (491kB/s-491kB/s), io=384MiB (403MB), run=820117-820117msec
  WRITE: bw=160KiB/s (164kB/s), 160KiB/s-160KiB/s (164kB/s-164kB/s), io=128MiB (134MB), run=820117-820117msec

Disk stats (read/write):
    dm-0: ios=98257/33380, merge=0/0, ticks=1571720/67740, in_queue=1639460, util=100.00%, aggrios=98759/33422, aggrmerge=377/24, aggrticks=1539323/51980, aggrin_queue=1611556, aggrutil=99.99%
  sda: ios=98759/33422, merge=377/24, ticks=1539323/51980, in_queue=1611556, util=99.99%

ARM:

root@nsa310:~# ./pool_bench.sh
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=1424KiB/s,w=492KiB/s][r=356,w=123 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=801: Mon Dec 14 12:15:02 2020
  read: IOPS=119, BW=478KiB/s (490kB/s)(384MiB/822072msec)
   bw (  KiB/s): min=  152, max= 1512, per=100.00%, avg=478.05, stdev=86.75, samples=1644
   iops        : min=   38, max=  378, avg=119.45, stdev=21.70, samples=1644
  write: IOPS=39, BW=159KiB/s (163kB/s)(128MiB/822072msec); 0 zone resets
   bw (  KiB/s): min=    8, max=  528, per=100.00%, avg=159.29, stdev=46.21, samples=1644
   iops        : min=    2, max=  132, avg=39.75, stdev=11.56, samples=1644
  cpu          : usr=0.68%, sys=2.80%, ctx=131888, majf=2, minf=84
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=98308,32764,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=478KiB/s (490kB/s), 478KiB/s-478KiB/s (490kB/s-490kB/s), io=384MiB (403MB), run=822072-822072msec
  WRITE: bw=159KiB/s (163kB/s), 159KiB/s-159KiB/s (163kB/s-163kB/s), io=128MiB (134MB), run=822072-822072msec

Disk stats (read/write):
    dm-0: ios=98268/33396, merge=0/0, ticks=1581580/64370, in_queue=1645950, util=100.00%, aggrios=98848/33436, aggrmerge=732/2870, aggrticks=1550548/48653, aggrin_queue=1620936, aggrutil=100.00%
  sda: ios=98848/33436, merge=732/2870, ticks=1550548/48653, in_queue=1620936, util=100.00%

cryptsetup benchmark

Each cipher was tested with following steps:

• luksFormat /dev/sda5
• luksOpen /dev/sda5 sda5
• benchmarks described in table below on /dev/mapper/sda5
• create ext4fs on /dev/mapper/sda5
• the same benchmarks but on mounted ext4 (writing/reading from file).

REMARKS:

1. For XTS, only half of key is used, so for 128b cipher I need to specify -s 256.
2. Ext4 by default was created with lazy_init, to speed up creation process, but it can make impact on tests.
3. Before each test, flush by sync && echo 3 > …/drop_caches was issued.

Copy using dd if=src_file of=dst_file conv=fsync

“It will synchronize output data and metadata just before finishing”

Comparison SW & HW with loaded system

stress -v -c 1

(SW only)

Enable low complexity ciphers if device is used locally.

ssh -Q cipher localhost | paste -d , -s

/etc/ssh/sshd_config

# enable all ciphers!
# obtained with ssh -Q cipher localhost | paste -d , -s
Ciphers 3des-cbc,blowfish-cbc,cast128-cbc,arcfour,arcfour128,arcfour256,aes128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com