I’ve received a bunch of questions from people around what happens at each vSphere memory state after writing the article around breaking up large pages and introducing a new memory state in vSphere 6.0. Note, that the below is about vSphere 6.0 only, the “Clear” memory state does not exist before 6.0. Also, note that there is an upper and lower boundary when transitioning between states, this means that you will not see actions triggered at the exact specified threshold, but slightly before or after passing that threshold.
I created a simple table that shows what happens when. Note that minFree itself not a fixed number but rather a sliding scale and the value will depend on the host memory configuration.
Memory state |
Threshold | Actions performed |
High | 400% of minFree | Break Large Pages when below Threshold (wait for next TPS run) |
Clear | 100% of minFree | Break Large Pages and actively call TPS to collapse pages |
Soft | 64% of minFree | TPS + Balloon |
Hard | 32% of minFree | TPS + Compress + Swap |
Low | 16% of minFree | Compress + Swap + Block |
First of all, note that when you enter the “Hard” state the balloon driver stops and “Swap” and “Compress” take over. This is something I never really realized, but it is important to know as it means that when memory fills up fast you will see a short period of ballooning and then jump to compressing and swapping immediately. You may ask yourself what is this “block” thing. Well this is the worst situation you can find yourself in and it is the last resort, as my colleague Ishan described it:
The low state is similar to the hard state. In addition, to compressing and swapping memory pages, ESX may block certain VMs from allocating memory in this state. It aggressively reclaims memory from VMs, until ESX moves into the hard state.
I hope this makes it clear which action is triggered at which state, and also why the “Clear” state was introduced and the “High” state changed. It provides more time for the other actions to do what they need to do: free up memory to avoid blocking VMs from allocating new memory pages.
Sreejesh says
Thanks Duncan.
Jon says
Really digging the v6 beta, and these post of yours are beyond value.
Christian Dadswell (@chrisdadswell) says
Thanks Duncan, an interesting couple of posts on the subject.
Justin Adams says
Since minFree is now a moving target, I put together a formula for it. This formula is founded on the assumption that your host has MORE than 28GB of memory, which for production hosts, that should be an easy constraint to to adhere to. 🙂
Working on this assumption, the 6%, 4%, and 2% ranges will always add up to 901MB.
245.76 – 6%
327.68 – 4%
327.68 – 2%
Adds up to:
901 MB
So that only leaves us with the 1% of what’s left, that’s the moving target. You only need to input your Host’s installed memory amount.
M = Installed Host Memory
.01((M-28)*1024) + 901 = minFree (in MB)
Duncan, I worked this a couple of times, pretty sure it would hole true for all cases in which the host has more than 28GB of installed memory. There is a bit of variance from the numbers presented in your your post on June 14, 2013, but that is only due to the fact that I strictly worked in MB and rounded a little closer.