Advanced Storage Issues in ESXi

18. Datastore Becomes Read-Only

Scenario:

• An ESXi host suddenly marks a VMFS datastore as read-only, preventing VMs from writing to the disk.

Possible Causes:

• Storage device path failure.
• VMFS corruption.
• ESXi lost access to the LUN.

Solution Steps:

1. Check if the Datastore is Marked as Read-Only bashCopyEditvmkfstools -P /vmfs/volumes/DatastoreName
   • If output shows “Read-Only”, continue with recovery.
2. Rescan the Storage Adapters bashCopyEditesxcli storage core adapter rescan --all
3. Check for Active Paths bashCopyEditesxcli storage core path list | grep "Active"
   • If no active paths exist, storage connectivity is lost.
4. Try to Remount the Datastore bashCopyEditesxcli storage filesystem mount -l DatastoreName
5. Check for VMFS Errors and Repair bashCopyEditvmkfstools -y /vmfs/devices/disks/naa.6xxxxxxxxx
6. Restart Storage Services bashCopyEdit/etc/init.d/hostd restart /etc/init.d/vpxa restart
7. If All Fails, Restore from Backup or Recreate the Datastore bashCopyEditesxcli storage filesystem unmount -l DatastoreName esxcli storage vmfs extent delete -l DatastoreName

---

19. ESXi Host Loses Access to iSCSI Storage (APD/PDL Errors)

Scenario:

• The ESXi host loses connection to iSCSI storage and shows APD (All Paths Down) or PDL (Permanent Device Loss) errors.

Solution Steps:

1. Check Storage Paths bashCopyEditesxcli storage core path list
   • If no active paths, the storage is disconnected.
2. Restart the iSCSI Service bashCopyEdit/etc/init.d/iscsi restart
3. Rescan iSCSI Targets bashCopyEditesxcli storage core adapter rescan --adapter=vmhbaXX
4. Verify Firewall Rules (If iSCSI is Blocked) bashCopyEditesxcli network firewall ruleset list
5. Reconnect iSCSI Targets Manually bashCopyEditesxcli iscsi adapter discovery sendtarget add -a <Storage_IP>
6. If PDL Condition Exists, Remove the LUN and Re-add It bashCopyEditesxcli storage core device detach -d naa.xxxxxxxx esxcli storage core device set -d naa.xxxxxxxx -O off

---

20. High Storage Latency Causing VM Performance Issues

Scenario:

• VMs running on an ESXi host experience high I/O latency, affecting performance.
• esxtop shows high DAVG/cmd values.

Solution Steps:

1. Check Storage Performance in esxtop bashCopyEditesxtop
   • Press d for disk statistics.
   • Check DAVG/cmd (Disk Average Latency):
     • Below 10ms = Good
     • 10-20ms = Moderate
     • Above 20ms = Bad
2. Check for Storage Congestion bashCopyEditesxcli storage core stats get
3. Identify High I/O VMs bashCopyEditvscsiStats -l
4. Reduce Storage Load
   • Move VMs to another datastore using Storage vMotion.
   • Enable Storage I/O Control (SIOC) to balance I/O.
5. Verify Multipathing Policy bashCopyEditesxcli storage nmp device list
6. Upgrade Storage Firmware and Drivers
   • Check if the storage controller is on VMware’s HCL (Hardware Compatibility List).

---

Advanced Network Issues in ESXi

21. VM Network Latency or Packet Loss

Scenario:

• VMs experience high ping latency or packet drops.
• vMotion and iSCSI traffic is slow.

Solution Steps:

1. Check Physical NIC Statistics bashCopyEditesxcli network nic stats get -n vmnicX
   • Look for high droppedTx or droppedRx.
2. Check for Network Congestion in esxtop bashCopyEditesxtop
   • Press n for network stats.
3. Verify MTU Mismatch (If using Jumbo Frames) bashCopyEditesxcli network ip interface list
   • Ensure all devices (VMkernel, vSwitch, and physical switch) have the same MTU.
4. Ensure vSwitch and Port Group Configurations are Correct bashCopyEditesxcli network vswitch standard list esxcli network vswitch standard portgroup list
5. Check VLAN Tagging Issues
   • Ensure VLAN IDs match between ESXi and the physical switch.
6. Enable RSS (Receive Side Scaling) for Better Performance bashCopyEditesxcli system settings advanced set -o /Net/RSS -i 1

---

22. vMotion Fails Due to Network Misconfiguration

Scenario:

• vMotion fails with “Host not reachable” error.
• Migration is slow or gets stuck.

Solution Steps:

1. Check if vMotion is Enabled on the VMkernel Port bashCopyEditesxcli network ip interface tag list
2. Test vMotion Network Connectivity bashCopyEditvmkping -I vmk1 <destination_host_IP>
3. Verify MTU Configuration bashCopyEditesxcli network ip interface list
   • Ensure Jumbo Frames (9000 MTU) is configured correctly.
4. Check for Firewall Blocking vMotion Traffic bashCopyEditesxcli network firewall ruleset list
   • Enable vMotion if necessary: bashCopyEditesxcli network firewall ruleset set -e true -r vmotion
5. Try vMotion Again Using CLI bashCopyEditvim-cmd vmsvc/getallvms vim-cmd vmsvc/migrate <VM_ID> <Target_Host_ID>

---

23. ESXi Host Loses Network Connectivity After vSwitch Change

Scenario:

• After changing vSwitch or VLAN settings, the ESXi host loses network access.

Solution Steps:

1. Check Physical NIC Status bashCopyEditesxcli network nic list
2. Verify vSwitch Configuration bashCopyEditesxcli network vswitch standard list
3. Reconfigure the VMkernel Network Manually bashCopyEditesxcli network ip interface remove -i vmk0 esxcli network ip interface add -i vmk0 -p "Management Network" -i 192.168.1.10 -n 255.255.255.0 -g 192.168.1.1
4. Restart the Network Stack bashCopyEdit/etc/init.d/networking restart