Well it looks as though this issue is still alive and well. I worked with VMWare Support to identify the root cause of the failures and after checking VMKwarning.log it was confirmed that it was yet another issue with the hp ams service. In our case we run a scratch off a different partition and this took 82 days, almost exactly to surface. Here's the details on the build. I'm still awaiting feedback from HP on the "correct" way to remediate the issue. Frankly, this is my 2nd go around with similar issues that have hazarded my environment in 3 months and I'm seriously considering just removing it altogether.
Here's details on our affected build
HP DL560 G8
ESXi 5.5.0 189274
FW 02-2014 B
HP Offline Management Bundle v 1.7-13
AMS V 550.9.6.0-12.1198610
Build Original HP Custom CD 5.5x
Reported Errors on Affected Hosts
"Console" Can't Fork
VMotion Timed Out
VCenter Host Configuration Security Profile Access "Call "HostImageConfigManager.QueryHostAcceptanceLevel" for object "imageConfigManager-252705" on vCenter Server "your vcenters fqdn" failed.
vmwarning.log "2014-10-14T20:20:53.668Z cpu39:46241)WARNING: Heap: 4128: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41801ffe5429"
Avamar 7.01 Backup and Restore Failure, Restore Error 10011 Failed to write to disk, Backup failure Snapshot failure,
Recommended Remediation
1) Lower memory overhead on effected hosts by shutting off 25 % or more of VMs
2) Login to Tech support mode
3) Run the following commands
•Stop the service via SSH or the ESXi Shell: /etc/init.d/hp-ams.sh stop
•Disable the service at boot via SSH or the ESXi Shell: chkconfig hp-ams.sh off
< At this point host operations should return to normal, backups, vmotion>
No immediate host reboot has been required thus far
Long Term:
I'm trying to work with HP to identify a Solid Remediation process.
It has been stipulated by HP's support team that you must remove the vib and reboot prior to updating the AMS. What may be happening is that not all effectid components are removed and simply applying the subsequent patch or full bundle may not refresh all components
Here's a list of resources on the subject matter thus far.
VMWare KB2085618
HP mmr_kc-0120902 http://tinyurl.com/mxkt9es
HP FW http://tinyurl.com/k6hebz4