Performance Tuning WebWizard

Expanation of Memory and Paging

The Memory of your machine hardly needs any expanation but how it is used is complex. In modern computer systems the memory is relatively inexpensive and so is often the second thing to be upgraded (disks first).

The Memory is used for:

code - the actual programs [can be paged from the filesystem]
data - the data of a program
- stack - for making function calls and local variables
- heap - for dynamically requested data
shared memory - shared between processes
library code and data
AIX "buffer cache" - reading and writing journal file system files [always paged to/from the filesystem]
AIX O.S. kernel including its data
Network buffers for sending and receiving data on the network
Memory mapped files [always paged to/from the filesystem]

... and you thought you understood memory !!!

When real phyiscal memory is full, AIX does a number of things to work around this problem. AIX uses virtual memory (this is physically on the hard disk allocated for paging as paging space). AIX uses virtual memory by temporarily writing pages of memory to the paging space. This frees up real memory for running the system.

AIX is a very advanced version of UNIX. This means is does not have fixed sizes for above list of "things" that use memory but these are dynamically allocated. This means AIX does not have to be rebooted to change their sizes.

AIX always seems low in Memory

The above does mean that AIX "seems" to slowly use up all memory regardless of how much it really needs. What is actually happening is that as users and programs read files from the file systems AIX keeps these for as long as possible in case it can reuse the memory version and thus avoid disk I/O. To prove this is happening you can steal memory from the system with the rmss (say 25% of RAM) command and then give it back. This will appear as genuinely unused real memory that will then slowly be allocated until AIX again "seems" to use all the RAM. If this takes 60 seconds or longer then its not really used. If this took 5 seconds to be used up then you are short of RAM.

Memory or Paging Bottleneck ???

Criteria:

Paging space low - when paging space greater than 70% active
Paging bound - when paging volume %busy greater than 30% of the IO and paging logical volume to paging volume activity greater than 40%
Thrashing - rising page outs, CPU wait and run queue.

Not a criteria:

Free Memory is zero or very low - this is normal with AIX (see above)

In a multi-user environment memory usage is always likely to be very peaky - meaning that users demands of the memory are in lots of small unpredictable bursts. To cater for this we need to keep space always free in the paging space, so that small peaks in workload can be handled immediately. Failure to do this will cause totally unexpected program/process crashes - it will appear that AIX randomly choises which process to crash. Actually, it will be the next process to request any extra memory or when trying to start new programs.

Memory and Paging monitor Statistics

The basic Memory and Paging usage is on the Main Screen of nmon.
The statisics comprises the Real Memory, Virtual Memory and Paging rate activity.

Memory - Real and Virtual

Free means that it is not in use. AIX slowly uses Real memory if it is used for caching files and does not free it unless it is low on free space. This means that there is free memory but it will not show up here.
Files AIX slowly uses Real memory if it is used for caching files and does not free it unless it is low on free space. This reduces I/O and increases performance. But it means that there appears to be less and less free memory the longer AIX is running.
Total for Real this is the RAM in the machine. For Virtual its the amount of paging space.

Note: that typically the Virtual Memory (paging space use lsps -a) should be about two times Real memory (lsdev -Cc memory).

Paging

This section includes the paging activity. Paging is a result of running out of Real memory so that currently unused parts of Real memory are written to disk to make free room in memory.

pgfaults the number of time AIX found it needed a page of data or program in memory and it was not found in memory.
pgin the number of times per second that a page needed to be read in to be used
pgout the number of times per second that a page needed to be read out to make more free space
pgsin paging to actual paging space (paging can also be done to binary files or memory mapped files in the normal filesystem)
pgsout ditto

Notes:

AIX pages are always 4k bytes (4096 bytes).
Small amount of paging are normal and not worring, as programs regularly change their working set over time.
What is small is hard to define? Generally - 50 to 200 pages per second is OK on multiple paging disks.

What can you do to tune Memory?

If you find the memory is 100% used:

This is normal for AIX because it always keeps whats in memory if it can in case it needs it later and thus saves time.

If you find the system paging too much or even thrashing:

Spread Paging Disk I/O to more disks this will speed up paging but note this will not be used immediately, only new processes will make use of new paging space unless you reboot.
Check there are no "run-away" processes using too much memory - see tuning processes
Reduce the number of processes, often database and the like have options to use less - see tuning processes
Reduce the memory used by applications - often database allocate memory for database caches and this can be reuded
Plan for an upgrade!!

Free	means that it is not in use. AIX slowly uses Real memory if it is used for caching files and does not free it unless it is low on free space. This means that there is free memory but it will not show up here.
Files	AIX slowly uses Real memory if it is used for caching files and does not free it unless it is low on free space. This reduces I/O and increases performance. But it means that there appears to be less and less free memory the longer AIX is running.
Total	for Real this is the RAM in the machine. For Virtual its the amount of paging space.

pgfaults	the number of time AIX found it needed a page of data or program in memory and it was not found in memory.
pgin	the number of times per second that a page needed to be read in to be used
pgout	the number of times per second that a page needed to be read out to make more free space
pgsin	paging to actual paging space (paging can also be done to binary files or memory mapped files in the normal filesystem)
pgsout	ditto