last week I had an issue after Firmware update and restarting one ESXi server. I did some checks and noticed that one datastore appeared as “not consumed” and was inaccessible. I could not reach the VMs on that datastore of course.
Reaching out to VMware support, they immediately understood what was the issue.
Problem
One or more Datastore volume in vcenter appears as “not consumed” in storage devices. and you cant access the VMs on this datastore.
Cause
Missing partition table on the storage device.
To further investigate this, open SSH to one of your ESxi servers and run this piece of code:
offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done
Output:
A bad partition table will look like this and will be missing the partition table. —– This Datastore was not consumed in vc
/vmfs/devices/disks/naa.
gpt
1336746 255 63 21474836480
Checking offset found at 2048:
0200000 d00d c001
0200004
1400000 f15e 2fab
1400004
0140001d 44 53 5f 50 52 4f 44 5f 30 31 00 00 00 00 00 00 |DS_PROD_01……|
0140002d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |…………….|
—
A good partition table will look like this. see the bolded partition table. note the partition table highlighted in yellow:
/vmfs/devices/disks/naa.
gpt
1336746 255 63 21474836480
1 2048 21474836446 AA31E02A400F11DB9590000C2911D1
Checking offset found at 2048:
0200000 d00d c001
0200004
1400000 f15e 2fab
1400004
0140001d 44 53 36 5f 50 52 4f 44 5f 31 32 00 00 00 00 00 |DS6_PROD_12…..|
0140002d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |…………….|
Fix
Disclaimer: Vmware recommends to not perform the fix by yourself and open a support ticket at Vmware
now that you know that VMware advises not to do this on your own. To fix this, we need to recreate the partition table.
Run partedUtil getUsableSectors to find the last usable sector on the volume naa.xxxxxxxxxx
[root@esx006:~] partedUtil getUsableSectors /vmfs/devices/disks/naa.6742b0f0000004fb0000000000160b9d
34 21474836446
The output we got: 21474836446 – this number is the last usable sector on the volume. We need this for the next command.
Now run partedUtil setptbl using the last usable sector.
Note that 2048 is always the first usable sector and is the same for vmfs5/6
your command should look something like this:
partedUtil setptbl /vmfs/devices/disks/naa.6742b0f0000004fb0000000000160b9d gpt "1 2048 21474836446 AA31E02A400F11DB9590000C2911D1B8 0"
Thats it. now run the first code again to recheck for missing partition tables on other datastores and on the specific datastore you originally had the “not consumed” issue.
Greetings! I came across your site trying to solve a problem with my datastore disappearing into thin air.
I have a 54TB datastore (“54tb-md1200” local DAS RAID array) on my home lab host (Dell R620). I rebooted the host today and when it came back up, the datastore was gone. RAID controller says the array is fine. Storage Devices shows the array is detected but “Not Consumed”. Partition Details shows it’s all still there. However nothing I do in the GUI or the command line will force it to find and mount the datastore.
Used CLI to check partition details and they show no errors (using this as reference: https://silentcrash.com/2021/07/datastore-volume-in-vcenter-appears-as-not-consumed-in-storage-devices/). Used trial version of DiskInternals VMFS Recovery and it finds the datastore with directory structure intact. I can browse to the directory holding the VMDK files and they’re all there. The trial version won’t let me copy them anywhere though. So it doesn’t look like the partition table is bad. It’s just…not finding it.
I’m kind of at my wits end here. Everything I can see says the datastore is there and fine yet ESXi 8.0 will neither find it nor mount it. Any ideas? Much appreciated and thank you in advance for any assistance!
sounds like a vcenter / esxi bug to me. I guess you tried a storage adapters rescan via the GUI
https://kb.vmware.com/s/article/1003988
via the ESXi
esxcli storage core adapter rescan -a
or
esxcli storage core adapter rescan –all
That’s one of the first things I tried. No luck. And I highly doubt it’s a bug. This hardware has been running like this as ESXi 7.0 for a few years without incident, and a few years before that with ESXi 6.7 and 6.5. The current ESXi 8.0 is something I put on it *after* the array became unreachable in the hopes that it might resolve the situation, but no luck. I also connected this array to my other server (Dell R640, PERC H830, ESXi 8.0) and got identical results. It’s not the R620. It’s not the PERC H810. It’s not the physical array itself.
I suspect the issue is something wrong with the volume or partition table, or perhaps some kind of lock on the datastore (if ESXi uses such a thing) but I can’t find anything.
My best guess is that esxi 8 is not compatible with the hardware. I do hope you installed an esx version downloaded from the vendor website.
Good luck
It is compatible. Even if it wasn’t, I tried this with ESXi 7.0 and got the same results. The array was running fine under ESXi 7.0 for over a year when this happened. I re-installed ESXi 7.0 to see if that fixed it. When it didn’t, I did a fresh install of ESXi 8.0 to see if that made any difference. Neither did. I also hooked this same array up to my R640 with PERC H830 (both of which are supported configs) and got same results. This isn’t a compatibility issue.