Sign in to follow this  
Followers 0
Kevin OBrien

Nexenta All-Flash for VM Compression and Deduplication

11 posts in this topic

As the StorageReview Enterprise Test Lab grows at a great clip in both devices being reviewed as well as storage consumed through our large-scale testing, the need for some efficiency is starting to be realized. Currently for just VMs used in our production environment, we use over 12TB of Tier1 storage through a couple of all-flash arrays to host our day to day activities. In most cases there is a lot of similar data ripe for a good data reduction method, since we have many S2012-based VMs with minimal changes across them, and plenty of Linux-based VMs for large-scale tests such as VMmark. As we build up new tests, a storage array with data reduction capabilities is turning to a must.

To that end we're building up a storage array leveraging Nexenta 4.0.3 now on a 24-bay Supermicro SAS3 platform with flash as the backend storage. We're leveraging two dual-port Emulex 8Gb FC cards, 3 LSI 9300-8i HBAs to connect the storage, two WD Xe 900GB HDDs in RAID1 for OS storage, and the rest is experimenting with different SSDs for the main storage pool.

Currently we've been experimenting with the following SSD pools, although using some different volume settings across all of them.

8 x 480GB Micron M500DC (4 mirrored pairs)

16 x 480GB Micron M500DC (8 mirrored pairs)

6 x 800GB Micron M500DC (3 mirrored pairs)

post-47697-0-15397900-1416160935_thumb.p

The performance requirements on this particular build are rather interesting. The only thing we care about is VM migration speed. It needs to be fast for that to move to different arrays we are testing. If its fast in FIO but not this, it won't matter since thats not the metric we care about. To that end, deduplication and compression enabled don't have a huge performance impact, while sync does. With sync enabled VM migration speed tops out around 100MB/s, while it being disabled for the volume brings us up to about 800MB/s-1GB/s. That is the comfort zone that makes everyone happy.

post-47697-0-81476200-1416160910_thumb.p

The second part in all of this is what if any gains does Nexenta offer in this project, key area being data reduction. To answer that, we will be breaking the lab down into homogenous chunks and measuring space consumed at the ESXi host level to the amount used at the dataset level in Nexenta to pinpoint those numbers.

Share this post


Link to post
Share on other sites

Apparently I did something the system didn't like and the management went offline during a huge data migration and the volume on the ESXi host vanished. Logging in through SSH I was able to issue a reboot and bring it back online. Thankfully data is still intact, but the zvol mapping was gone. Recreated that and got the volume to appear on the ESXi side where the VMs on that datastore still functioned. Really odd, but I guess thats what testing this stuff out first, before rolling it out in production is all about.

Share this post


Link to post
Share on other sites

Built a better form of the array adjusting some settings and made things more stable. First round of VMs to check data reduction included the OlioWeb and OlioDB VMs we use for VMmark.

1.8TB Raw used

16.63x data reduction as seen by the Nexenta zvol

96.40GB Allocated at the volume level.... huge benefit!

Share this post


Link to post
Share on other sites

Next up is the DS2Web and DS2DB VMs.

1.47TB Raw used

2.77x Dedupe

157GB allocated at the volume level... so again a huge benefit.

It looks like this group had a big part of its data reduction helped by compression over dedupe.

Share this post


Link to post
Share on other sites

MailServer and VirtualClients were the least compressible and dedupable

2.17TB Raw used

2.09x Dedupe

612GB allocated at the volume level

Share this post


Link to post
Share on other sites

Next up is the DS2Web and DS2DB VMs.

1.47TB Raw used

2.77x Dedupe

157GB allocated at the volume level... so again a huge benefit.

It looks like this group had a big part of its data reduction helped by compression over dedupe.

Hello, you can just connect through ssh go to the expert mode:

option expert_mode =1

!bash

and enter this command to know the compression ratio:

zfs get refcompressratio

Share this post


Link to post
Share on other sites

Thanks, I'll poll that on my next round!

And because you array is full of ssd you can also make some modificationat the kernel level to enable the caching of ssd ... if you ssd are protected against power lost. I did that for my own lab (intel dc s3500 and intel dc s3700):

vi /kernel/drv/sd.conf ->

sd-config-list=

« ATA INTEL SSDSC2BB24″, »physical-block-size:4096,throttle-max:32, disksort:false, cache-nonvolatile:true »,

« ATA INTEL SSDSC2BA10″, »physical-block-size:4096,throttle-max:32, disksort:false, cache-nonvolatile:true »;

So here i force the physical block size to 4k, i put the throttle-max to 32 because the ssd are sata and the max queue of sata is 32, i said that they are ssd and not disk and finally i enable the cache of my ssd drive.

this command will enable the new parameter:

update_drv -vf sd

To find the id of your ssd you have to run this command first:

echo « ::walk sd_state | ::grep ‘.!=0′ | ::print struct sd_lun un_sd | \

::print struct scsi_device sd_inq | ::print struct scsi_inquiry \

inq_vid inq_pid » | mdb -k

Don't forget you have to do that before creating your pool.

The only thing i can say is play with zfs because it's just an awesome file system. The best out there

Share this post


Link to post
Share on other sites

Stuck in an airport with flight delays... might as well test another stage.

Windows 2012 and Windows 8.1 VMs used around the lab

1.01TB Raw

4.92x dedupe

128GB allocated at the zvol level

Share this post


Link to post
Share on other sites

Kind of strange but during the migration off the Nexenta storage datastore I had the system crash again. WebGUI goes offline with this error:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Looking at dmesg its telling me this in a few spots that a zpool crashed, but not which one and not which drive had a problem.

http://illumos.org/msg/ZFS-8000-D3

A reboot of the Nexenta box brought the zpool back online, with a target volume in place... but the mapping was gone. Just like last time. Make a new mapping and storage is back online and functional.

Any ideas where a log might be telling me exactly which device failed? After the reboot no errors are logged anywhere and no device is showing problems.

zpool status

pool: performance1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
performance1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t500A07510C0A9D23d0 ONLINE 0 0 0
c0t500A07510C0A9C9Bd0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c0t500A07510C0A9CCCd0 ONLINE 0 0 0
c0t500A07510C0A9D1Cd0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c0t500A07510C0A9CEAd0 ONLINE 0 0 0
c0t500A07510C0A9C6Ed0 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
c0t500A07510C0A9D07d0 ONLINE 0 0 0
c0t500A07510C0A9D2Fd0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
c0t500A07510C0A9C6Bd0 ONLINE 0 0 0
c0t500A07510C0A9CC1d0 ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
c0t500A07510C0A9CB1d0 ONLINE 0 0 0
c0t500A07510C0A9CD5d0 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
c0t500A07510C0A9CAAd0 ONLINE 0 0 0
c0t500A07510C0A9CA3d0 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
c0t500A07510C0A9CC7d0 ONLINE 0 0 0
c0t500A07510C0A9CB6d0 ONLINE 0 0 0
errors: No known data errors
pool: syspool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
syspool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t50000C0F0225E7BCd0s0 ONLINE 0 0 0
c0t50000C0F02D95B14d0s0 ONLINE 0 0 0
errors: No known data errors
admin@myhost:~#

Share this post


Link to post
Share on other sites

Kind of strange but during the migration off the Nexenta storage datastore I had the system crash again. WebGUI goes offline with this error:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Looking at dmesg its telling me this in a few spots that a zpool crashed, but not which one and not which drive had a problem.

http://illumos.org/msg/ZFS-8000-D3

A reboot of the Nexenta box brought the zpool back online, with a target volume in place... but the mapping was gone. Just like last time. Make a new mapping and storage is back online and functional.

Any ideas where a log might be telling me exactly which device failed? After the reboot no errors are logged anywhere and no device is showing problems.

zpool status

pool: performance1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
performance1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t500A07510C0A9D23d0 ONLINE 0 0 0
c0t500A07510C0A9C9Bd0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c0t500A07510C0A9CCCd0 ONLINE 0 0 0
c0t500A07510C0A9D1Cd0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c0t500A07510C0A9CEAd0 ONLINE 0 0 0
c0t500A07510C0A9C6Ed0 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
c0t500A07510C0A9D07d0 ONLINE 0 0 0
c0t500A07510C0A9D2Fd0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
c0t500A07510C0A9C6Bd0 ONLINE 0 0 0
c0t500A07510C0A9CC1d0 ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
c0t500A07510C0A9CB1d0 ONLINE 0 0 0
c0t500A07510C0A9CD5d0 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
c0t500A07510C0A9CAAd0 ONLINE 0 0 0
c0t500A07510C0A9CA3d0 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
c0t500A07510C0A9CC7d0 ONLINE 0 0 0
c0t500A07510C0A9CB6d0 ONLINE 0 0 0
errors: No known data errors
pool: syspool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
syspool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t50000C0F0225E7BCd0s0 ONLINE 0 0 0
c0t50000C0F02D95B14d0s0 ONLINE 0 0 0
errors: No known data errors
admin@myhost:~#

If you go in expert mode you will have access to the classic solaris shell.

So you will find the issue in /var/log or /var/adm or /var/crash.

Hum just a question is your memory ok?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0