Speeding Up RAID Operations on Synology NAS

There are a few things you can do to speed up the process of RAID operations, SSH to your NAS as admin and enter the following commands (change md3 to your device):

# echo 100000 > /proc/sys/dev/raid/speed_limit_min
# echo 32768 > /sys/block/md3/md/stripe_cache_size



The first command increases the minimum “goal” rebuild speeds for when there’s non-rebuild activity. 

The second command increases the stripe cache size which increases sync performance by allowing a larger cache to synchronise the read and write operations on the array.  This is only available for RAID 5 volumes and it does decrease the amount of available system RAM but I never saw 100% utilisation on the DiskStation.

You can monitor the process with the following commands:

# cat /proc/mdstat
# cat /sys/block/md5/md/stripe_cache_size
# cat /proc/sys/dev/raid/speed_limit_min

It also works for speeding up volume consistency checks as they both read the same config. Remember the commands above will only set those options until the NAS is rebooted.

Verifying the status of RAID arrays

We can check the status of the arrays on the system with:

1	cat /proc/mdstat

or

1	mdadm --detail /dev/md0

The output of this command will look like:

1 2 3 4 5 6 7 8 9 10

cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU] md1 : active raid1 sdb3[1] sda3[0] 19542976 blocks [2/2] [UU] md2 : active raid1 sdb4[1] sda4[0] 223504192 blocks [2/2] [UU]

Speed up Synology volume reshaping

Google brought me to this and this forum post which suggested this command for checking the status and estimated runtime:

root@DiskStation:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid5 sdd6[2] sda6[0] sdb6[1]
      1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=====>...............]  reshape = 25.1% (491263360/1953494784) finish=5198.4min speed=4687K/sec

md2 : active raid5 sdd5[4] sda5[0] sdc5[5] sdb5[1]
      5846049792 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid1 sdd2[3] sda2[0] sdb2[1] sdc2[2]
      2097088 blocks [4/4] [UUUU]

md0 : active raid1 sdd1[3] sda1[0] sdb1[1] sdc1[2]
      2490176 blocks [4/4] [UUUU]

unused devices: <none>

It showed about 5200 minutes left. That would be almost 87 hours, or a little bit over 3½ days. Given that some people reported runtimes of 16 days, I assumed I was lucky already.
However, the speed indicator of the above command showed values around 4-6 MB/sec which is rather slow for hard drives capable of 100 MB/sec and more.
People mentioned running services on the DiskStation to be a problem, especially indexing services for e.g. the PhotoStation. So I stopped every service I didn’t need (from the Package Manager) and paused those services which appeared at the top for the top command, i.e. which were using the CPU:

root@DiskStation:~# synoservice --pause synomkthumbd
root@DiskStation:~# synoservice --pause pkgctl-FileStation
root@DiskStation:~# synoservice --pause scemd
root@DiskStation:~# synoservice --pause synorelayd
root@DiskStation:~# synoservice --pause synoindexd

(Just remember to --resume them after everything is done.)
Now to the tweaking of the lvm parameters. Searched for a site explaining the different parameters and found this nice page which explains the commands and their effects in terms of speed and memory consumption.
The sysctl dev.raid.speed_limit_max already showed a reasonable value of 200000 (200 MB/sec) but I raised it to 500 MB/sec - just to be sure.

sysctl -w dev.raid.speed_limit_max=500000

As expected, this didn’t do anything. On to the next parameter from that page - thestripe_cache_size. First, check the current (default) value:

root@DiskStation:~# cat /sys/block/md3/md/stripe_cache_size
256

This means the stripe cache is 256 pages of 4096 Bytes each = 1 MiB per disk. Not very much. So increased it as per the recommendation to 4096 pages = 16 MiB per disk:

echo 4096 > /sys/block/md2/md/stripe_cache_size
echo 4096 > /sys/block/md3/md/stripe_cache_size

(md2 and md3 are my raid5 devices. See mdstat output above.)
This brought the biggest effect. The estimated runtime left immediately fell to around 1300 minutes, fluctuating between 1100 and 1800 minutes:

root@DiskStation:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid5 sdd6[2] sda6[0] sdb6[1]
      1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=====>...............]  reshape = 26.0% (509001728/1953494784) finish=1338.0min speed=17992K/sec

That was already impressive, cutting down the overall time from 3.5 days to 22 hours. And we had another parameter to tweak: the read-ahead. Again, we first want to figure out the default value:

root@DiskStation:~# blockdev --getra /dev/md3
384

Since the value is in “sectors” of 512 Bytes, that comes in at 192 KiB. As we have plenty of RAM (upgraded my DS to 8GB), set this to the recommended 32 MiB:

root@DiskStation:~# blockdev --setra 65536 /dev/md3
root@DiskStation:~# blockdev --setra 65536 /dev/md2

And that gave another small boost:

root@DiskStation:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid5 sdd6[2] sda6[0] sdb6[1]
      1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=====>...............]  reshape = 26.1% (511295616/1953494784) finish=1001.1min speed=24008K/sec

Another 300 minutes (5 hours) shaved off the remaining time. With a remaining time of less than 17 hours for something which initially needed 3.5 days, was very pleased.
And the process seems to have sped up even more as it is now - 9.5 hours later - nearing completion:

root@DiskStation:~# cat /proc/mdstat 
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md3 : active raid5 sdd6[2] sda6[0] sdb6[1]
      1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [===================>.]  reshape = 95.9% (1874314432/1953494784) finish=20.1min speed=65521K/sec

UPDATE

After replacing the second disk, there was another reshape going on, but this time, the speed lingered at 20-30 MB/sec which isn’t much given that it went up to 80 MB/sec for the first disk.
However, after some more research, found this comment which explained how the system only speeds up to speed_limit_max when there’s no other disk I/O going on and otherwise keeps the speed at speed_limit_min. That made sense as the default speed_limit_min is 10000 (or 10 MB/sec) and already increased it to 20000 yesterday.
Why the Synology thought there was other activity and kept the transfer speed down, don’t know.
But now increased the minimum speed further to 60 MB/s:

sysctl -w dev.raid.speed_limit_min=60000

And suddenly the reshaping speed also increased to around 60 MB/sec. Slightly increased it further until it ended up at a value of 90000 (90 MB/s) and a real speed of around 80-90 MB/sec.
This cut the remaining time down from 12 hours to 3.