tag:blogger.com,1999:blog-20038672.post2238515399456302539..comments2024-03-28T00:58:29.187-04:00Comments on Robert Haas: Linux Memory ReportingRobert Haashttp://www.blogger.com/profile/08393677427643988650noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-20038672.post-10371695135139830372012-05-24T21:07:29.371-04:002012-05-24T21:07:29.371-04:00htop - fist impression => great advice => gr...htop - fist impression => great advice => great tool => great help - thanks<br />PeteAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-20038672.post-30031671117874932942012-01-30T12:33:36.167-05:002012-01-30T12:33:36.167-05:00Bob,
Can you check out htop and post an update as ...Bob,<br />Can you check out htop and post an update as to whether htop gets the checks and balances as far as memory reporting goes?<br /><br />http://htop.sourceforge.net/index.php?page=main<br /><br />I enjoyed reading your post, but wanted to suggest htop as an alternative to top<br /><br />@gsvoltgsvolthttp://gsvolt.wordpress.comnoreply@blogger.comtag:blogger.com,1999:blog-20038672.post-25801263114882468282012-01-26T16:48:09.465-05:002012-01-26T16:48:09.465-05:00I do confess I struggle on interpreting these valu...I do confess I struggle on interpreting these values on a regular basis. What I hate most is that SHM is accounted for as "cached" by each and every tool I know... even /proc/meminfo!<br /><br />However, on the plus side:<br />I totally love atop (http://www.atoptool.nl). It is based on Linux' process accounting, gives very detailed information on almost every aspect of the system and its processes, has a comparably small footprint (compared to top at least) etc.<br /><br />It can save snapshots of the system (default every 10 mins.) which sum up the usage in the past timespan, incl. exited processes.<br />Especially that last feature has saved my back many times, as it is very easy to point at the process/user that was consuming most memory. Comes in very handy when you dip into swapspace or experience OOM situations.<br /><br />As a bonus, you get disk I/O and network traffic (even per process with some patches), context switches, migrations etc.<br /><br />Best regards,<br /><br />NickGunnar "Nick" Bluthnoreply@blogger.comtag:blogger.com,1999:blog-20038672.post-6372691375929478592012-01-17T18:08:03.162-05:002012-01-17T18:08:03.162-05:00There's yet another tool I like better than to...There's yet another tool I like better than top and `free -m`: <a href="http://htop.sourceforge.net/" rel="nofollow">htop</a>.<br />But of course I know nothing of its accuracy :-)Diegohttps://www.blogger.com/profile/18052138336869925472noreply@blogger.comtag:blogger.com,1999:blog-20038672.post-58417993957960460972012-01-16T04:37:58.318-05:002012-01-16T04:37:58.318-05:00The only reliable way of doing memory accounting o...The only reliable way of doing memory accounting on linux i know of is using /proc/PID/clear_refs and parsing /proc/PID/smaps afterwards.gebihttps://www.blogger.com/profile/02726867824497339744noreply@blogger.comtag:blogger.com,1999:blog-20038672.post-30382615011542003752012-01-16T01:30:21.386-05:002012-01-16T01:30:21.386-05:00You've inspired me to write up what I know (an...You've inspired me to write up what I know (and have discovered) about the various sorts of memory usage information that you can get from Linux with top, ps, and smem. I've posted the result as <a href="http://utcc.utoronto.ca/~cks/space/blog/linux/LinuxMemoryStats" rel="nofollow">Linux memory stats</a>.<br /><br />(The short version is that the question is ambiguous but you can get various sorts of useful information, especially with smem.)Chris Siebenmannhttp://utcc.utoronto.ca/~cks/space/blog/noreply@blogger.comtag:blogger.com,1999:blog-20038672.post-16932838711741657712012-01-14T12:33:49.351-05:002012-01-14T12:33:49.351-05:00This was a very helpful blog post. It highlights ...This was a very helpful blog post. It highlights the vague and inconsistent method developers have used to show memory.<br /><br />There are various methods for representing memory that is shared, either via SysV shared memory, fork's copy-on-write, or shared libraries. Does every process get charged the full amount, or do they split it among themselves, e.g. if five processes use shared memory, is each process charged 20% of the total size? (If another prcocess attaches, does your percentage decrease?) What happens when you map in a large shared memory area but only access part of it? When do you stop using that memory?<br /><br />There doesn't seem to be any clear answers because the answer depends on what you want to do with the number. What hasn't happened is anyone really laying out these issues and defining exactly what their tool does. What is happening now is that each developer is deciding on their own how to answer these questions, and it isn't making it out into the documentation, probably because few users would even understand the explanation.<br /><br />This uncertainty has lingered for years, and I don't see it geting resolved anytime soon, unless someone steps up to the plate, defines what these numbers mean, and then records how each tool handles them.<br /><br />As Robert described, PostgreSQL isn't alone in having this issue, but maybe we have to take the lead in explaining it. In an ideal world, someone would define these tradeoffs, and every tool would use the same terms, and allow output matching each of the posssible use cases.Bruce Momjianhttps://www.blogger.com/profile/07875088787463864011noreply@blogger.comtag:blogger.com,1999:blog-20038672.post-43678094339015810792012-01-13T06:02:32.439-05:002012-01-13T06:02:32.439-05:00See this LWN article on PSS/USS. USS is just the u...See <a href="http://lwn.net/Articles/230975/" rel="nofollow">this LWN article</a> on PSS/USS. USS is just the unshared page count, i.e. memory returned when process is killed.<br /><br />Random fact: Chrome's also in the same multiprocess boat, and its built-in task manager and about:memory page <a href="https://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/14d57bcf09fc5ddd" rel="nofollow">use USS/PSS for accounting</a>.<br /><br />I think what would be interesting and is missing from this picture is a tool that presents USS for a <em>set</em> of processes. This would be useful for multiprocess applications like PostgreSQL, Chrome, etc. Simply summing the processes' USS would undercount, as any shared pages would be left unaccounted for. The hypothetical tool would need to determine which pages are shared exclusively among the processes belonging to the set in question (probably fine to assume a tree, so just pointing it at a parent process would work).Yang Zhanghttp://yz.mit.edu/noreply@blogger.comtag:blogger.com,1999:blog-20038672.post-75169540192358517622012-01-13T05:45:53.273-05:002012-01-13T05:45:53.273-05:00PostgreSQL already has functions like "pg_dat...PostgreSQL already has functions like "pg_database_size" which give information which can also be obtained from the operating system. <br /><br />Likewise, PostgreSQL functions concerning memory usage can be considered; this would be operating system independent, avoiding the issues with Linux. Of course, allocator overhead (space) cannot be taken into account, but leaks would still show up. If the information is maintained not per each allocation but per the larger units of allocation you mentioned, the slowdown might be acceptable.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-20038672.post-68136114401801825572012-01-12T13:21:39.944-05:002012-01-12T13:21:39.944-05:00I haven't run across smem before, but it does ...I haven't run across smem before, but it does look like their idea of proportional set size (PSS) might be a useful concept. smem's notion of RSS is the the same as what top reports as RES, so it has the same pros and cons. There's also a USS column which, at least in my copy of smem, isn't documented, so it's hard to know whether that's useful or not.<br /><br />On my system, the "real" memory utilization of httpd, as determine by starting and stopping it, is about 11M. The total of the PSS columns for httpd is 7871kB and the total of the USS columns is 5624kB. So PSS alone is clearly less than the real memory utilization, and PSS+USS is slightly more (but maybe that's rounding error?).<br /><br />In short: seems interesting, but it's hard to tell what it's really doing. It has a blizzard of command-line options that are described with phrases like "show library-oriented view" but there's no clear definition of what that actually means, which makes it not that useful IMV.Robert Haashttps://www.blogger.com/profile/08393677427643988650noreply@blogger.comtag:blogger.com,1999:blog-20038672.post-57084334242893274822012-01-12T12:26:02.580-05:002012-01-12T12:26:02.580-05:00I use smem to get an idea of the memory usage of p...I use smem to get an idea of the memory usage of processes. Do you think this tool gives a correct view on memory usage?Anonymousnoreply@blogger.com