SYMPTOMS:I’ve been troubleshooting a problem heavily for the past three days and on and off for several years now involving Microsoft’s Small Business Server 2011 product. Here are the symptoms:
-
- RAM utilization is excessive, remaining at 99-100% constantly, primarily rising after boot in a few weeks, and in some extreme cases, within an hour of boot
- The server has plenty of RAM (16+ GB)
- The number of users/size of the Exchange store is not commensurate with the RAM burn
- Tuning Exchange and SQL database cache size seems to have no effect on the symptoms
- Task manager and Resource Monitor do not indicate any clear culprits
- Process Explorer and RAM MAP do not provide exceedingly obvious pieces of information (except in ONE specific case that I’ll discuss later, but it wasn’t very obvious)
- Task Manager basically shows the RAM is fully consumed and none is Cached or Available.
- EVERYTHING is slowed down to a crawl
- CPU was cool under 20%
TROUBLESHOOTING:
So in troubleshooting this problem, I did the most obvious things such as:
-
- Look for anything consuming lots of RAM
- Ensure antivirus exceptions were set properly
- Allowed Windows to manage the page file
- Look for heavy processes (CPU, disk, and RAM utilization)
- CHKDSK
- Limit Exchange and SQL cache sizes
- Shut down superfluous services
- Engaged additional technical resources
RESOLUTION:
None of this yielded any help until I started reviewing Process Monitor. I (for whatever reason) double-clicked the RAM graph at the to of the screen and all of a sudden noticed that Cache WS was consuming all of the RAM on the server. I was not familiar with the term so I began an online search that resulted in me finding this forum post: http://social.technet.microsoft.com/Forums/en-US/7f87deaf-7cfb-4b40-ae88-04b42b074a0e/cache-ws-ram-issue-on-windows-server-2012 The post was related to Windows Server 2012 consuming over 80 GB of RAM but I know that under the hood, 2008 R2 and 2012 are fairly similar so I read on. I found a reference to someone suggesting that the File Cache could be the problem and a reference to an article on how to limit it: http://social.technet.microsoft.com/wiki/contents/articles/6925.how-to-prevent-filecache-to-eat-up-all-the-ram.aspx Basically, the long and short of it is that on x64 systems, the File Cache is not limited and can grow beyond the system’s RAM, causing insane amounts of swapping. This, in turn, makes the system unresponsive and unusable. Limiting the File Cache is one way to resolve the issue. One other way is to remove mapped drives that may contain large files. I opted to limit the cache with a utility found here: http://www.softpedia.com/progDownload/SetSystemFileCacheSize-Download-138752.html There is a detailed blog discussing the issue, if you haven’t already burned out yourself and Google trying to find an answer to it. The moral of the story…. There is such a thing as too much cache. Additional reading ONLY found AFTER I discovered that File Cache was the issue (note how few there are):
CREDITS:
I’m gonna just go ahead and thank my Lord and Savior, Jesus Christ, for this. Several years worth of troubleshooting and the answer was in a tiny little stat under Process Explorer. Once I found that stat and was able to start researching it, the resolution came together.