Bug #85

3.0.0 beta 3 Zil not used

Added by David Ramsay over 3 years ago. Updated over 3 years ago.

Status:Feedback Start:May 10, 2010
Priority:Normal Due date:
Assigned to:- % Done:

0%
Category:-
Target version:-

Description

Here's the setup: Zil: mirrored 32G SSDs Cahce: dual 128G SSDs Main Pool: 6 1.5 TB Drives in a raid-z2 setup Backup store: 1 1.5TB drive used as a datadump OSpool: 2 mirrored 160GB drives iSCSI transport: Comstar uname -a: SunOS store1.zfs 5.11 NexentaOS_134c i86pc i386 i86pc Solaris Raid card: dual LSI 9240-8i in jbod configuration server specs: Dual Intel E5520 @ 2.27GHz 32GB Ram 10/100/1000 Nic to storage network

Problem: The zil never gets used in disk writes, i.e. while writing data either via iscsi or direct to the server the zil never shows activity nor affects performance.

Data: here comes a lot of test data. zpool iostat -z aggr0 3 capacity operations bandwidth pool alloc free read write read write

aggr0 32.3G 10.8T 0 0 0 0 raidz2 32.3G 10.8T 0 0 0 0 c5t9d1 - - 0 0 0 0 c5t10d1 - - 0 0 0 0 c5t11d1 - - 0 0 0 0 c5t12d1 - - 0 0 0 0 c5t14d1 - - 0 0 0 0 c5t13d1 - - 0 0 0 0 mirror 4K 29.7G 0 0 0 0 c4t8d1 - - 0 0 0 0 c4t11d1 - - 0 0 0 0 cache - - - - - - c4t9d1 3.51G 116G 0 0 0 0 c4t10d1 3.77G 115G 0 0 0 0

            capacity     operations    bandwidth

pool alloc free read write read write

aggr0 32.3G 10.8T 10 134 27.2K 365K raidz2 32.3G 10.8T 10 134 27.2K 365K c5t9d1 - - 6 67 3.47K 102K c5t10d1 - - 10 26 7.43K 106K c5t11d1 - - 4 74 3.30K 112K c5t12d1 - - 4 28 2.15K 102K c5t14d1 - - 10 23 7.10K 102K c5t13d1 - - 6 49 4.13K 111K mirror 4K 29.7G 0 0 0 0 c4t8d1 - - 0 0 0 0 c4t11d1 - - 0 0 0 0 cache - - - - - - c4t9d1 3.51G 116G 0 0 0 0 c4t10d1 3.77G 115G 0 0 1.32K 0

            capacity     operations    bandwidth

pool alloc free read write read write

aggr0 32.3G 10.8T 3 580 7.02K 1.30M raidz2 32.3G 10.8T 3 580 7.02K 1.30M c5t9d1 - - 0 49 319 379K c5t10d1 - - 3 48 2.03K 385K c5t11d1 - - 2 44 1.40K 388K c5t12d1 - - 2 38 1.09K 387K c5t14d1 - - 3 46 1.56K 386K c5t13d1 - - 0 50 319 388K mirror 4K 29.7G 0 0 0 0 c4t8d1 - - 0 0 0 0 c4t11d1 - - 0 0 0 0 cache - - - - - - c4t9d1 3.51G 116G 0 0 0 0 c4t10d1 3.77G 115G 0 0 0 0

iostat -x 3 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b cmdk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 cmdk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd4 4.0 21.3 7.0 83.0 1.1 2.1 126.4 26 90 sd5 4.7 20.7 7.5 55.0 1.1 1.6 106.5 23 83 sd6 3.3 18.0 6.9 29.0 0.1 1.0 52.8 7 65 sd7 3.0 21.3 1.7 75.2 2.6 1.8 181.0 93 98 sd8 4.7 15.3 7.4 43.3 0.4 1.5 94.7 12 71 sd9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd11 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd12 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd13 2.3 15.7 6.2 23.2 0.2 0.1 17.9 4 8 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b cmdk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 cmdk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd3 1.7 0.7 5.7 0.0 0.0 0.0 0.0 0 0 sd4 4.7 2.7 7.5 1.3 0.0 0.1 11.6 0 5 sd5 4.7 2.3 7.5 1.3 0.0 0.1 14.4 0 5 sd6 3.3 2.7 6.5 1.3 0.0 0.1 11.6 0 7 sd7 6.7 3.0 13.7 1.3 0.1 0.2 28.9 11 14 sd8 4.7 2.3 7.7 1.3 0.0 0.1 7.4 0 5 sd9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd10 3.3 1.3 11.4 0.0 0.0 0.0 0.0 0 0 sd11 1.7 0.7 5.7 0.0 0.0 0.0 0.0 0 0 sd12 3.3 1.3 11.4 0.0 0.0 0.0 0.0 0 0 sd13 2.3 2.3 6.0 1.3 0.0 0.0 2.1 1 0

test data: zfs: fresh aggr no zil, no cache, no dedup, no compression, on box root@store1:~# zpool status pool: aggr0 state: ONLINE scrub: none requested config:

    NAME         STATE     READ WRITE CKSUM
    aggr0        ONLINE       0     0     0
      raidz2-0   ONLINE       0     0     0
        c5t9d1   ONLINE       0     0     0
        c5t10d1  ONLINE       0     0     0
        c5t11d1  ONLINE       0     0     0
        c5t12d1  ONLINE       0     0     0
        c5t14d1  ONLINE       0     0     0
        c5t13d1  ONLINE       0     0     0

root@store1:~# dd if=/dev/urandom of=/aggr0/crap1.test bs=512 count=1800K 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 31.1391 seconds, 30.3 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/aggr0/crap2.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.248 seconds, 61.9 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/root/crap1.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.8757 seconds, 59.4 MB/s root@store1:~# dd if=/root/crap1.test of=/aggr0/crap3.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.3111 seconds, 61.6 MB/s bonnie root@store1:~# bonnie -s 100 -r 50 -d /aggr0 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03b ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP store1.zfs 100M +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ 27320 100 +++++ +++ +++++ +++ store1.zfs,100M,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16,+++++,+++,+++++,+++,+++++,+++,27320,100,+++++,+++,+++++,+++

fresh aggr zil, no cache, no dedup, no compression, on box root@store1:~# zpool status aggr0 pool: aggr0 state: ONLINE scrub: none requested config:

    NAME         STATE     READ WRITE CKSUM
    aggr0        ONLINE       0     0     0
      raidz2-0   ONLINE       0     0     0
        c5t9d1   ONLINE       0     0     0
        c5t10d1  ONLINE       0     0     0
        c5t11d1  ONLINE       0     0     0
        c5t12d1  ONLINE       0     0     0
        c5t14d1  ONLINE       0     0     0
        c5t13d1  ONLINE       0     0     0
    logs
      mirror-1   ONLINE       0     0     0
        c4t8d1   ONLINE       0     0     0
        c4t11d1  ONLINE       0     0     0

root@store1:~# dd if=/dev/urandom of=/aggr0/crap1.test bs=512 count=1800K 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 29.6727 seconds, 31.8 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/aggr0/crap2.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.1723 seconds, 62.2 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/root/crap1.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 17.1579 seconds, 55.0 MB/s root@store1:~# dd if=/root/crap1.test of=/aggr0/crap3.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 14.727 seconds, 64.1 MB/s

bonnie root@store1:~# bonnie -s 100 -r 50 -d /aggr0 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03b ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP store1.zfs 100M +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ 23092 99 +++++ +++ 13151 100 store1.zfs,100M,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16,+++++,+++,+++++,+++,+++++,+++,23092,99,+++++,+++,13151,100

fresh aggr zil, cache, no dedup, no compression, on box root@store1:~# zpool status aggr0 pool: aggr0 state: ONLINE scrub: none requested config:

    NAME         STATE     READ WRITE CKSUM
    aggr0        ONLINE       0     0     0
      raidz2-0   ONLINE       0     0     0
        c5t9d1   ONLINE       0     0     0
        c5t10d1  ONLINE       0     0     0
        c5t11d1  ONLINE       0     0     0
        c5t12d1  ONLINE       0     0     0
        c5t14d1  ONLINE       0     0     0
        c5t13d1  ONLINE       0     0     0
    logs
      mirror-1   ONLINE       0     0     0
        c4t8d1   ONLINE       0     0     0
        c4t11d1  ONLINE       0     0     0
    cache
      c4t9d1     ONLINE       0     0     0
      c4t10d1    ONLINE       0     0     0

root@store1:~# dd if=/dev/urandom of=/aggr0/crap1.test bs=512 count=1800K 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 29.9784 seconds, 31.5 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/aggr0/crap2.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 14.7929 seconds, 63.8 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/root/crap1.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 16.2444 seconds, 58.1 MB/s root@store1:~# dd if=/root/crap1.test of=/aggr0/crap3.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.0895 seconds, 62.5 MB/s bonnie root@store1:~# bonnie -s 100 -r 50 -d /aggr0 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03b ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP store1.zfs 100M 158597 100 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 26341 99 +++++ +++ +++++ +++ 28779 100 +++++ +++ +++++ +++ store1.zfs,100M,158597,100,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16,26341,99,+++++,+++,+++++,+++,28779,100,+++++,+++,+++++,+++

fresh aggr zil, cache, dedup, no compression, on box root@store1:~# zpool status aggr0 pool: aggr0 state: ONLINE scrub: none requested config:

    NAME         STATE     READ WRITE CKSUM
    aggr0        ONLINE       0     0     0
      raidz2-0   ONLINE       0     0     0
        c5t9d1   ONLINE       0     0     0
        c5t10d1  ONLINE       0     0     0
        c5t11d1  ONLINE       0     0     0
        c5t12d1  ONLINE       0     0     0
        c5t14d1  ONLINE       0     0     0
        c5t13d1  ONLINE       0     0     0
    logs
      mirror-1   ONLINE       0     0     0
        c4t8d1   ONLINE       0     0     0
        c4t11d1  ONLINE       0     0     0
    cache
      c4t9d1     ONLINE       0     0     0
      c4t10d1    ONLINE       0     0     0

root@store1:~# dd if=/dev/urandom of=/aggr0/crap1.test bs=512 count=1800K 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 30.6629 seconds, 30.8 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/aggr0/crap2.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.0573 seconds, 62.7 MB/s root@store1:~# dd if=/root/crap1.test of=/aggr0/crap3.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.1529 seconds, 62.3 MB/s root@store1:~# dd if=/root/crap1.test of=/aggr0/crap3.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.8964 seconds, 59.4 MB/s

bonnie root@store1:~# bonnie -s 100 -r 50 -d /aggr0 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03b ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP store1.zfs 100M 202057 100 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ 24991 99 +++++ +++ +++++ +++ store1.zfs,100M,202057,100,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16,+++++,+++,+++++,+++,+++++,+++,24991,99,+++++,+++,+++++,+++

fresh aggr zil, cache, dedup, compression, on box root@store1:~# zpool status aggr0 pool: aggr0 state: ONLINE scrub: none requested config:

    NAME         STATE     READ WRITE CKSUM
    aggr0        ONLINE       0     0     0
      raidz2-0   ONLINE       0     0     0
        c5t9d1   ONLINE       0     0     0
        c5t10d1  ONLINE       0     0     0
        c5t11d1  ONLINE       0     0     0
        c5t12d1  ONLINE       0     0     0
        c5t14d1  ONLINE       0     0     0
        c5t13d1  ONLINE       0     0     0
    logs
      mirror-1   ONLINE       0     0     0
        c4t8d1   ONLINE       0     0     0
        c4t11d1  ONLINE       0     0     0
    cache
      c4t9d1     ONLINE       0     0     0
      c4t10d1    ONLINE       0     0     0

errors: No known data errors root@store1:~# dd if=/dev/urandom of=/aggr0/crap1.test bs=512 count=1800K 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 31.3156 seconds, 30.1 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/aggr0/crap2.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.3419 seconds, 61.5 MB/s root@store1:~# dd if=/root/crap1.test of=/aggr0/crap3.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 15.0122 seconds, 62.9 MB/s root@store1:~# dd if=/aggr0/crap1.test of=/root/crap1.test 1843200+0 records in 1843200+0 records out 943718400 bytes (944 MB) copied, 16.4975 seconds, 57.2 MB/s bonnie root@store1:~# bonnie -s 100 -r 50 -d /aggr0 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03b ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP store1.zfs 100M +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ 20153 99 18396 100 +++++ +++ 10516 100 store1.zfs,100M,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16,+++++,+++,+++++,+++,20153,99,18396,100,+++++,+++,10516,100

History

Updated by Werner Fouché over 3 years ago

Hi,

The latest version of Nexenta is 3.0.2. I'm also testing Nexenta, and it works fine for me so far. Maybe you should give the latest version a try. 3.0.0 beta 3 definitely would not be at the same quality level as 3.0.2.

Regards,

Werner

Updated by Dmitry Yusupov over 3 years ago

  • Status changed from New to Feedback

Could you please test this with 3.0.2-1 updates (will be released later today/tomorrow)?

Updated by David Ramsay over 3 years ago

I'll give the new version a try, I have installed opensolaris svn_134 and the problem dose not occur. The Zil show's IO as soon as you write to the aggregate.

Also available in: Atom PDF