meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
vm:proxmox:storage [2020/04/22 14:38] – niziakvm:proxmox:storage [2020/04/26 16:57] (current) – niziak
Line 1: Line 1:
 ====== Storage ====== ====== Storage ======
 +
 +===== Terms =====
 +  * **shared** 
 +    * do not set local storage as shared, because content on each node is different
 +    * One major benefit of storing VMs on shared storage is the ability to live-migrate running machines without any downtime, as all nodes in the cluster have direct access to VM disk images. There is no need to copy VM image data, so live migration is very fast in that case.
 +  * **thin-provisioning** - allocates block when they are written.
 +
 +===== Content types =====
 +
 +^  content types     ^  type    ^ description                   ^ path ^
 +|   Disk image       |  images  | KVM-Qemu VM images (VM disks) | ''local:230/example-image.raw'' | 
 +|  ISO image         |   iso    |                               | ''local:iso/debian-501-amd64-netinst.iso'' |
 +| Container template |  vztmpl  |                               | ''local:vztmpl/debian-5.0-joomla_1.5.9-1_i386.tar.gz'' |
 +| VZDump backup file |  backup  |                                |
 +| Container          |  roodir  |  Allow to store container data |
 +|                    |  none    | prevent using block device directly for VMs (to create LVM on top) | |
 +| Snippets           | snippets | aka hookscripts (for example guest hook scripts) | ''<storage>:snippets/<file>'' |
 +
 +''iscsi-storage:0.0.2.scsi-14f504e46494c4500494b5042546d2d646744372d31616d61''
 +
 +==== File level storage dir layout ====
 +  * images - (VM images) - ''images/<VMID>/''
 +    * raw, qcow2, vmdk
 +  * iso - (ISO images) - ''template/iso/''
 +  * vztmpl - (Container templates) - ''template/cache/''
 +  * backup - (Backup files) - ''dump/''
 +  * snippets - (Snippets) - ''snippets/''
 +
 +===== Default storage for ZFS  =====
 +  * **local**: file-level storage - you can upload iso images and place backups there.
 +  * **local-zfs**: this is used to store VM images.
 +
 +Note: Both resides on the same zfs pool.
 +
 +<file | /etc/pve/storage.cfg>
 +dir: local
 +        path /var/lib/vz
 +        content iso,vztmpl,backup
 +
 +zfspool: local-zfs
 +        pool rpool/data
 +        sparse 1
 +        content images,rootdir
 +</file>
 +
  
 ===== Storage types ===== ===== Storage types =====
 +  * File level storage
 +    * poll types:
 +      * **directory** shared: **NO**
 +      * **glusterFS** shared: **YES**
 +      * **nfs** shared: **YES**
 +      * **cifs** shared: **YES**
 +    * features
 +      * any POSIX compatible filesystem pointed by path
 +      * no snapshot by FS: VMs with qcow2 are capable to do snapshots
 +      * any content type:
 +      * virtual disk images, containers, templates, ISO images, backup files
 +  * Block level storage
 +    * ''iscsidirect'' iSCSI user mode (''libiscsi2'')
 +      * content types: images
 +      * format: raw, shared: YES, no snapshots, no clones
 +    * ''iscsi'' 
 +      * content types: images, none
 +      * format: raw, shared: YES, no snapshots, no clones
 +    * ''LVM'' 
 +      * possible to create on top of iSCSI LUN to get managable disk space
 +      * content types: images, rootdir
 +      * format: raw, shared: YES (iSCSI), no snapshots, no clones
 +    * ''LVM thin''
 +      * new thin volume type on top of existing LVM VG
 +      * thin-provisioning
 +      * content types: images, rootdir 
 +      * format: raw, shared: NO, snapshots, clones
 +    * ''ZFS over iSCSI'' - to use **ZFS from remote** system using iSCSI 
 +      * benefits of ZFS:
 +        * for VMs: zfs volume per VM, live snaphots, cloning
 +        * thin provision
 +    * ''ZFS''
 +      * local node ZFS
 +      * content types: images, rootdir 
 +      * format: raw, subvol; shared: NO, snapshots YES, clones YES
 +
 +
 +===== poll type =====
   * Network storage   * Network storage
     * LVM Group on iSCSI     * LVM Group on iSCSI
Line 17: Line 100:
     * ZFS     * ZFS
  
-===== Content types ===== + 
-  * tbd +
-  * +
 ===== Local storage ===== ===== Local storage =====
  
Line 28: Line 110:
  
  
-===== iSCSI ===== 
-Proxmox doc recommends: 
- 
-  iSCSI is a block level type storage, and provides no management interface. So it is usually best to export one big LUN, and setup LVM on top of that LUN. You can then use the LVM plugin to manage the storage on that iSCSI LUN. 
- 
-iSCSI target/direct is capable to store only following content types 
-  * images 
-  * none 
- 
-Use LUNs directly - use directly as VM disk without putting LVM volume on it 
- 
-==== NAS326 CHAP issue ==== 
- 
-NAS326 requires CHAP authentication and initiator user name. 
-There are 2 options to use NAS326: 
-  * disable CHAP on NAS326 
-  * enable CHAP on Proxmox 
- 
-Proxmox initiator name can be found in file: ''/etc/iscsi/initiatorname.iscsi'' 
- 
-=== disable CHAP on NAS326 === 
-  * create the LUN(s) and target via the webgui 
-  * login to your zyxel via ssh as root 
-  * targetcli (this will open a shell where you can manage iscsi, use tab completion to get around in it) 
-  * ls (to get an overview) 
-  * cd /iscsi/iqn.2018-03.com.zyxel:targetname.randomstring/tpg1/ (again use tab completion, so cd /iscsi/iqn<tab><tab<tab<tab> etc. etc.) 
-  * set attribute authentication=0 
-  * set attribute generate_node_acls=1 
-  * set attribute demo_mode_write_protect=0 
-  * I also deleted the ACLS by doing; cd acls, delete iqn<tab><tab> 
-  * exit 
-  * targetcli saveconfig (normally if you exit targetcli, it will autosave, so this is just in case) 
- 
-=== use CHAP on Proxmox === 
- 
-Logout and remove all failed trials to connect to NAS326.  
-Especially if IPv6 was enabled on NAS326, proxmox detect two send_targets: one for IPv4 and one for IPv6 (not reachable). 
-After disabling IPv6 on NAS326, please delete IPv6 target portal: 
-<code bash> 
-targetcli ls 
-targetcli /iscsi/iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux/tpg1/portals ls 
-targetcli /iscsi/iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux/tpg1/portals/ 'delete fd57::be99:11ff:fe06:18b0 3260' 
-targetcli saveconfig 
-</code> 
- 
-<code bash> 
-ls /etc/iscsi/nodes 
- 
-# logout 
-iscsiadm -m node -u -T "iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux" --portal 192.168.28.150 
-iscsiadm -m node -u -T "iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux" --portal fd57::be99:11ff:fe06:18b0 
-# remove 
-iscsiadm -m node -o delete -T "iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux" --portal 192.168.28.150 
-iscsiadm -m node -o delete -T "iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux" --portal fd57::be99:11ff:fe06:18b0 
-</code> 
- 
-Uncomment and set following config lines: 
-<file conf | /etc/iscsi/iscsid.conf> 
-node.session.auth.authmethod = CHAP 
-# get initiator name from /etc/iscsi/initiatorname.iscsi 
-node.session.auth.username = iqn.1993-08.org.debian:01:4dad9d97a329 
-node.session.auth.password = my_chap_password_for_NAS326 
-</file> 
  
-Now disovery should return only one IPv4 target: 
-<code bash> 
-# iscsiadm -m discovery -t sendtargets -p 192.168.28.150 
-192.168.28.150:3260,1 iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux 
  
-# list config options +==== storage for ISOs ====
-iscsiadm -m node -o show+
  
-# login +By default it is stored in ''local'' storage. It is possible to upload ISOs directly using SCP to ''/var/lib/vs/template/iso''.
-# iscsiadm -m node --login +
-Logging in to [iface: default, target: iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux, portal: 192.168.28.150,3260] (multiple) +
-Login to [iface: default, target: iqn.2020-04.com.zyxel:nas326-iscsi-pve1-isos-target.tjlintux, portal: 192.168.28.150,3260] successful.+
  
-# check new block device 
-cat /proc/partitions 
-</code> 
  
-   
  
-iSCSI+LVM supports HA and Live Migration of VMs --> mark LVM storage as shared