Setup LVMCache for monitor.zappa.cx with RAID-1 on SSD
- Orig LV: monitor-disk F3_2TB -wi-ao---- 6.00g
- Done
To change cache mode
To remove cache
Setup LVMCache and ThinPool for caching multiple LV's with RAID-1 on SSD

Setup LVMCache for monitor.zappa.cx with RAID-1 on SSD

Orig LV: monitor-disk F3_2TB -wi-ao---- 6.00g

SSD PV's has been added to F3_2TB VG and SSD tags has been added.

bigbang:~# pvs -o +tags
  PV                VG     Fmt  Attr PSize   PFree   PV Tags
  /dev/cciss/c0d0p2 F3_2TB lvm2 a--    1.79t 734.65g HDD    
  /dev/sdb1         F3_2TB lvm2 a--  400.00g 400.00g SSD    
  /dev/sdc1         F3_2TB lvm2 a--  400.00g 400.00g SSD    
bigbang:~#

Create a 2-way RAID1 cache data LV (1GB)

bigbang:~# lvcreate --type raid1 -m 1 -n monitorCDLV -L 1G F3_2TB @SSD
  Logical volume "monitorCDLV" created.
bigbang:~#

Create a 2-way RAID1 cache metadata LV (8MB minimum or 1000 times smaller than cache LV)

bigbang:~# lvcreate --type raid1 -m 1 -n monitorCMLV -L 8M F3_2TB @SSD
  Logical volume "monitorCMLV" created.
bigbang:~#

Create a cache pool LV combining cache data LV and cache metadata LV

bigbang:~# lvconvert --type cache-pool --poolmetadata F3_2TB/monitorCMLV F3_2TB/monitorCDLV
  WARNING: Converting logical volume F3_2TB/monitorCDLV and F3_2TB/monitorCMLV to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert F3_2TB/monitorCDLV and F3_2TB/monitorCMLV? [y/n]: y
  Converted F3_2TB/monitorCDLV to cache pool.
bigbang:~#

Create a cached LV by combining the cache pool LV and origin LV

bigbang:~# lvconvert --type cache --cachepool F3_2TB/monitorCDLV F3_2TB/monitor-disk
Do you want wipe existing metadata of cache pool volume F3_2TB/monitorCDLV? [y/n]: y
  Logical volume F3_2TB/monitor-disk is now cached.
bigbang:~#

Done

To change cache mode

bigbang:~# lvs -o +cachemode F3_2TB/monitor-disk 
  LV           VG     Attr       LSize Pool          Origin               Data%  Meta%  Move Log Cpy%Sync Convert CacheMode   
  monitor-disk F3_2TB Cwi-aoC--- 6.00g [monitorCDLV] [monitor-disk_corig] 13.33  1.90            0.00             writethrough
bigbang:~#

bigbang:~# lvchange --cachemode writeback F3_2TB/monitor-disk
  Logical volume F3_2TB/monitor-disk changed.
bigbang:~#

bigbang:~# lvs -o +cachemode F3_2TB/monitor-disk 
  LV           VG     Attr       LSize Pool          Origin               Data%  Meta%  Move Log Cpy%Sync Convert CacheMode
  monitor-disk F3_2TB Cwi-aoC--- 6.00g [monitorCDLV] [monitor-disk_corig] 13.37  2.83            5.02             writeback
bigbang:~#

bigbang:~# lvchange --cachemode writethrough F3_2TB/monitor-disk
  Flushing 107 blocks for cache F3_2TB/monitor-disk.
  Flushing 79 blocks for cache F3_2TB/monitor-disk.
  Logical volume F3_2TB/monitor-disk changed.
bigbang:~#

bigbang:~# lvs -o +cachemode F3_2TB/monitor-disk 
  LV           VG     Attr       LSize Pool          Origin               Data%  Meta%  Move Log Cpy%Sync Convert CacheMode   
  monitor-disk F3_2TB Cwi-aoC--- 6.00g [monitorCDLV] [monitor-disk_corig] 13.37  2.83            0.00             writethrough
bigbang:~#

To remove cache

Make sure it is in writethrough - just to be safe. Also check that no blocks are dirty.

bigbang:~# lvconvert --uncache F3_2TB/monitor-disk
  Logical volume "monitorCDLV" successfully removed
  Logical volume F3_2TB/monitor-disk is not cached.
bigbang:~#

Setup LVMCache and ThinPool for caching multiple LV's with RAID-1 on SSD

Setting up ThinPool (size 500GB on the HW RAID-1 HDD and metadata on LVM mirrored SSD's)

Create 500GB LV for ThinPool0 on the HW raided HDD's

bigbang:~# lvcreate -n ThinPool0 -L 500G F3_2TB @HDD
  Logical volume "ThinPool0" created.
bigbang:~#

Create 1GB LV for ThinPool0_meta using LVM RAID-1 on the SSD's

bigbang:~# lvcreate --type raid1 -m 1 -n ThinPool0_meta -L 1G F3_2TB @SSD
  Logical volume "ThinPool0_meta" created.
bigbang:~#

Create the ThinPool0 by combining data data and metadata LV's

bigbang:~# lvconvert --type thin-pool --poolmetadata F3_2TB/ThinPool0_meta F3_2TB/ThinPool0
  Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data.
  WARNING: Converting F3_2TB/ThinPool0 and F3_2TB/ThinPool0_meta to thin pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert F3_2TB/ThinPool0 and F3_2TB/ThinPool0_meta? [y/n]: y
  Converted F3_2TB/ThinPool0 and F3_2TB/ThinPool0_meta to thin pool.
bigbang:~#

ThinPool0 has now been created, LV's using it can be created like so

# lvcreate -n ThinLV -V VirtualSize --thinpool ThinPool0 F3_2TB

Snapshots are created without specifying the size as it is automatically using the data from the ThinPool0 LV for storing the difference.

# lvcreate -n ThinLV_s1 -s F3_2TB/ThinLV

Adding caching for the ThinPool0

Create a 2-way RAID1 cache data LV (50GB)

bigbang:~# lvcreate --type raid1 -m 1 -n ThinPool0_CDLV -L 50G F3_2TB @SSD
  Logical volume "ThinPool0_CDLV" created.
bigbang:~#

Create a 2-way RAID1 cache metadata LV (8MB minimum or 1000 times smaller than cache LV)

bigbang:~# lvcreate --type raid1 -m 1 -n ThinPool0_CMLV -L 50M F3_2TB @SSD
  Rounding up size to full physical extent 52.00 MiB
  Logical volume "ThinPool0_CMLV" created.
bigbang:~#

Create a writeback cached LV by combining the cache pool LV cache metadata and origin LV

bigbang:~# lvconvert --type cache --cachemode writeback --cachepool ThinPool0_CDLV --poolmetadata ThinPool0_CMLV F3_2TB/ThinPool0
  WARNING: Converting F3_2TB/ThinPool0_CDLV and F3_2TB/ThinPool0_CMLV to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert F3_2TB/ThinPool0_CDLV and F3_2TB/ThinPool0_CMLV? [y/n]: y
  Converted F3_2TB/ThinPool0_CDLV and F3_2TB/ThinPool0_CMLV to cache pool.
  WARNING: Cached thin pool's data cannot be currently resized and require manual uncache before resize!
  Logical volume F3_2TB/ThinPool0_tdata is now cached.
bigbang:~#

Removing cache from ThinPool0

Change to writethrough before separating cache pool from cache LV

bigbang:~# lvchange --cachemode writethrough F3_2TB/ThinPool0_tdata
  Flushing 0 blocks for cache F3_2TB/ThinPool0_tdata.
  Logical volume F3_2TB/ThinPool0_tdata changed.

Separate and keep the cache pool from a cache LV.

bigbang:~# lvconvert --splitcache F3_2TB/ThinPool0
  Logical volume F3_2TB/ThinPool0_tdata is not cached and cache pool F3_2TB/ThinPool0_CDLV is unused.

Attach cache LV to previously separated cache pool

bigbang:~# lvconvert --type cache --cachemode writeback --cachepool F3_2TB/ThinPool0_CDLV F3_2TB/ThinPool0
Do you want wipe existing metadata of cache pool F3_2TB/ThinPool0_CDLV? [y/n]: y
  WARNING: Cached thin pool's data cannot be currently resized and require manual uncache before resize!
  Logical volume F3_2TB/ThinPool0_tdata is now cached.

Extending size of ThinPool0

Remove caching from the thinpool

As it does not seem to be possible to extend a cached thinpool, we first need to remove the cache. Follow first two steps above from section “Removing cache from ThinPool0” (do not re-attach the cache of course)

Some - before information

bigbang:~# lvdisplay -a F3_2TB/ThinPool0_tdata
  --- Logical volume ---
  Internal LV Name       ThinPool0_tdata
  VG Name                F3_2TB
  LV UUID                Fyc9Md-c1XD-xpib-dJDw-Obr4-SYL5-G88yqS
  LV Write Access        read/write
  LV Creation host, time bigbang, 2019-10-12 14:17:30 +0200
  LV Cache pool name     ThinPool0_CDLV
  LV Cache origin name   ThinPool0_tdata_corig
  LV Status              available
  # open                 1
  LV Size                500.00 GiB
  Cache used blocks      99.99%
  Cache metadata blocks  18.54%
  Cache dirty blocks     0.00%
  Cache read hits/misses 644218163 / 453243621
  Cache wrt hits/misses  1046297083 / 31441523
  Cache demotions        1455727
  Cache promotions       1455727
  Current LE             128000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:36

bigbang:~# lvchange --cachemode writethrough F3_2TB/ThinPool0_tdata
  Flushing 55 blocks for cache F3_2TB/ThinPool0_tdata.
  Logical volume F3_2TB/ThinPool0_tdata changed.

bigbang:~# lvconvert --splitcache F3_2TB/ThinPool0
  Logical volume F3_2TB/ThinPool0_tdata is not cached and cache pool F3_2TB/ThinPool0_CDLV is unused.

Now extend the ThinPool0 LV

bigbang:~# lvextend -L 1t F3_2TB/ThinPool0
  Size of logical volume F3_2TB/ThinPool0_tdata changed from 500.00 GiB (128000 extents) to 1.00 TiB (262144 extents).
  Logical volume F3_2TB/ThinPool0_tdata successfully resized.

And now reattach the cache

Do the last step from section “Removing cache from ThinPool0” to reattach the cache to the thinpool

bigbang:~# lvconvert --type cache --cachemode writeback --cachepool F3_2TB/ThinPool0_CDLV F3_2TB/ThinPool0
Do you want wipe existing metadata of cache pool F3_2TB/ThinPool0_CDLV? [y/n]: y
  WARNING: Cached thin pool's data cannot be currently resized and require manual uncache before resize!
  Logical volume F3_2TB/ThinPool0_tdata is now cached.

And some “after” information

bigbang:~# lvdisplay -a F3_2TB/ThinPool0_tdata
  --- Logical volume ---
  Internal LV Name       ThinPool0_tdata
  VG Name                F3_2TB
  LV UUID                Fyc9Md-c1XD-xpib-dJDw-Obr4-SYL5-G88yqS
  LV Write Access        read/write
  LV Creation host, time bigbang, 2019-10-12 14:17:30 +0200
  LV Cache pool name     ThinPool0_CDLV
  LV Cache origin name   ThinPool0_tdata_corig
  LV Status              available
  # open                 1
  LV Size                1.00 TiB
  Cache used blocks      0.13%
  Cache metadata blocks  12.46%
  Cache dirty blocks     10.12%
  Cache read hits/misses 0 / 36
  Cache wrt hits/misses  3663 / 1530
  Cache demotions        0
  Cache promotions       1067
  Current LE             262144
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:36

Table of Contents