VMware HP AMS service bug

Recently I ran into a problem with a HP DL380 G8 VMware ESX 5.5 host. It was not possible to logon trough SSH and we could not vMotion Virtual Machines anymore. It seemed like the SSH daemon stopped working. Starting the SSH deamon again did not work. In the vmkernel.log file the following error was displayed:

WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd

I discovered VMware KB2085618 article with the title “ESXi host cannot initiate vMotion or enable services and reports the error: Heap globalCartel-1 already at its maximum size.Cannot expand” which sounded exactly like our problem and seems to be caused by a memory leak in the HP-AMS service.

The KB article suggests to log in trough SSH and stop the HP-AMS service. The problem was the SSH daemon was not running anymore and could not be started.

We had to plan a maintenance window to reboot the ESX host. After the reboot we can connect trough SSH to the host and disable the HP-AMS service by following this steps:

  • Log in to the host using SSH.
  • Run this command to stop the HP service (does not persist on reboot): /etc/init.d/hp-ams.sh stop
  • Run this command to remove the VIB: esxcli software vib remove -n hp-ams
  • Reboot the host.

After removal the problem did not acquire again.

VMware PowerCLI script to add storage and VLAN’s

When deploying multiple VMware hosts you can use the Template feature in vSphere. If you’re not licenced to use the template feature in vSphere you can use the VMware PowerCLI PowerShell commands to quickly add the strorage and the networks.

First thing you got to do is connect to the host:

  • Connect-VIServer -Server esx.jvr.local

To add VLAN’s to a vSwitch use the following command:

  • get-vmhost -name esx.jvr.local | Get-VirtualSwitch -name vSwitch0 | new-VirtualPortGroup -name “LAN” -vlanid 0
  • get-vmhost -name esx.jvr.local | Get-VirtualSwitch -name vSwitch0 | new-VirtualPortGroup -name “PVS” -vlanid 2310

To add NFS storage use the following commands:

  • get-vmhost -name esx.jvr.local | New-Datastore -Nfs -Name ESX1 -Path /NAS_ESX1 -NfsHost 10.10.10.50

This can also be done at the cluster level. The following command’s can be used:

  • Get-Cluster “CLUSTER_1” | Get-VMHost | Get-VirtualSwitch -name vSwitch1 | New-VirtualPortGroup -name “PVS” -VLanId 2310
  • Get-Cluster “CLUSTER_1” | Get-VMHost | Get-VirtualSwitch -name vSwitch1 | New-VirtualPortGroup -name “HB” -VLanId 2309