Body

AIX Performance Tools
Reference Sheet

CPU bound when %user+%sys greater than 80%

Disk I/O bound when %iowait greater than 40%

Application disk bound when %tm_act greater than 70%

Paging space low when paging space greater than 70% acive

Paging bound paging volume %busy greater than 30% of the IO and paging logical volume to paging volume activity greater than 40%

Thrashing rising page outs, CPU wait and run queue.

rmss - Reduced Memory System Simulator

rmss -p -c M -r

-p print the current value
-c M change to M size (in Mbytes)
-r restore all memory to use
Examples:
rmss -c 32 Change available memory to 32 Mbytes
rmss -r Undo the above
vmtune - Virtual Memory Tuning

vmtune -p min -P max -f min -F max -r min -R max

-p min min percentage of memory reserved for file pages (default 20%)
-P max max percentage of memory reserved for file pages (default 80%)
-f min number of pages on free list below which page stealling starts (default 120)
-F max number of pages on free list above which page stealling stops (default 128)
-r min min number of pages to be read ahead after when sequential access detected
-R max max number of pages to be read ahead after when sequential access detected

schedtune - Processes Scheduling Tuning

schedtune -h sys -p proc -w wait -m multi -e grace -f ticks

-t time_slice -D (default)

Flag & Default Meaning
-h 6 Sets system wide criteria for process suspension begins and end (thrashing)
-p 4 Sets per-process criteria for determining process suspension begins and end
-w 1 Seconds to wait before trashing ended
-e 2 Seconds exempt after suspension
-f 10 Clock tick waited after fork failure
-t 0 Clock tick interrupts before dispather called
-D Restore default values
Examples:
schedtune -t5 set time slice to 50 tcks
schedtune Report current settings
Code: /usr/lpp/bos/samples
Compiled: make schedtune

Async I/O

Set via smit/smitty or chdev

minservers min. # of kernel processes
maxserver max. # of kernel processes
maxrequests max # of in progress and waiting requests
serverpriority priority of kernel processes (try 39)
state has to be available to work

svmon -System Virtual Memory Monitor

svmon -G -Pnsa pid... -Pnsa[upg][count] -S sid... -i secs count

-G Global report
-P[nsa] pid.. Process report n=non-sys s-system a=both
-S[nsa][upg][x] Segment report nsa as above + u==real-mem p=pinned g=paging x=top x items
-S sid... Segment report on particular segments
-i secs count Repeat report at interval second&count times
-D sid... Detailed report
Examples:
svmon -G Global / General stats
svmon -Pa 215 Process report for process 215
svmon -Ssu 10 Top ten system segments in real-mem order
svmon -D 340d Detailed report on a particular segment
size in pages (4096)
inuse in-use
free not inuse included rmss pages
pin pined (locked by app.)
work pages in working segments
pers pages in persistent segments
clnt pages inclient segments
pg space paging space
Note: pages can be in more than one process

filemon - File I/O monitor

filemon -i file -o file -d -Tn -P -v _O levels

Example: filemon -O all -o file.out
do work load here... finish with trcstop

#MBs total # of Mbytes transfer during run
#opns # of times the file was opened
#rpgs # of 4K page reads
#wpgs # of 4K page written
#wrs # of write calls
persistent paged from file system
working paged from paging space
util percentage busy
KB/s average data transfer rate

netpmon - Network Monitor

netpmon -o file -Tn -P -v -Oreport-type

-o outputfile put the output to file not stdout
-T n Set output buffer size (default 64000)
-P Force monitor process into pinned memory
-v Verbose (default only top 20 processes)
-O cpu, dd(device driver), so(socket), nfs, all
Example: netpmon -O all -o net.out
do network load here ... finish with trcstop

ncheck - inode check

ncheck [-a][-i inodenumber...] [-s] [filesystem]

-a all including . and ..
-i inode find the file(s) with these inode no.
-s list special and set UID files
Example:
ncheck -a / List all files in /
ncheck -i 2194 /tmp find name for inode 2194 in /tmp

PS - Process State

-l long listing
-f full listing
-u user list only user's processes (-u fred)
-e every user's processes
-t ttyno processes attached to tty (-t 03)
-p pid list the process number N

Examples:
ps -fu jim List the user jim processes in full
ps -lt 04 List all processes on terminal tty04
ps -fe List all processes

Output Meaning
PID/PPID Process IDentity&Parent Process IDentity
S State= Running Sleeping Waiting Zombie Terminating Kernel Intermediate X=growing
UID/USER User IDentity/User name
C CPU recent use value (part of priority)
STIME Start time of process
PRI Priority (higher means less priority)
NI NIce value (part of priority) default 20
ADDR ADDRess, of stack ( segment no)
SZ SiZe of process in 1K pages
CMD COMmanD the user typed (-f for more)
WCHAN Event awaited for (kernel address)
TTY Terminal processes in connected to (- = none)
TIME Minutes and Seconds of CPU time
SSIZ Size of kernel stack
PGIN # of pages paged in
SIZE Virtual size of data section in 1K's
RSS Real memory (resident set) size of process 1K's
LIM Soft limit on memory (see setrlimit) xx=none
TSIZ Size of text (shared text program) image
TRS Size of resident set (real memory) of test
%CPU Percentage of CPU used since started
%MEM Percentage of real memory used

iostat - Disk I/O statistics

iostat interval count

interval number of second between outputs
count number of times to output
Warning: ignore the first line (average since reboot)
Examples
iostat 10 20 20 lines output with 10 seconds between each
Output
%tm_act Percentage of time active
Kbps K bytes per second transfered
tps Transfers per second
msps Milli-second per seek (if available)
Kb_read Total K bytes read ( likewise for write)

vmstat - Virtual Memory Management Stats

vmstat interval count

interval number of second between outputs
count number of times to output
warning: ignore the first line (average since reboot)
Examples
vmstat 10 20 20 lines output with 10 seconds between each
r # of processes on run queue per second
b # of processes awaiting paging in per second
avm active virtual memory pages in page space
fre real memory pages on the free list
re Page reclaims, free but claimed before reused
pi paged in (per second)
po paged out (per sceond)
fr pages freed (page replacement) (per second)
sr pages per second scanned for replacement
cy complete scans of page table
in device interrupts per second
sy system calls per second
cs CPU context switches per second
us User CPU time percentage
sys System CPU time percentage
id CPU idle percentage (nothing to do)
wa CPU waiting for pending local Disk i/o

sar - system activity reporter Immediate: sar -A interval number

Collect: sar -A -o savefile interval number >/dev/null

Report: sar -A -f savefile -i secs

-s HH[:MM[:SS]] -e HH[:MM[:SS]]

-A All stats to be collected/reported
-o savefile Collect stats to binary file
-f savefile Report stats from binary file
-i secs Report at secs interval from binary file
-s and -e Report stats only between these times

Examples:
sar 10 100 Report now at 10 secs intervals
sar -A -o fred 10 6
>/dev/null
Collect data into fred
sar -A -f fred Report on the data
sar -A -f fred -s 10:30
-e 10:45
Report on 15 minutes from 10:30 a.m.
sar -A -f fred -i60 Report 1 minute interval rather that 10 seconds as collected

sar - CPU
%usr %sys Percent of time in user / kernel mode
%wio %idle Percent of time waiting for disk io/idle
Buffer Cache
bread/s bwrit/s Block I/O per second
lread/s lwrit/s Logical I/O per sec (hopefully cached
pread/s pwrit/s Raw disk I/O (not buffer cached)
%rcache %wcache Percentage hit on cache
Kernel
exec/s fork/s sread/s swrite/s rchar/s wchars/s scall/s Calls per second of these system calls sread/write system calls (cache, raw, tty or network).
scall is the total system calls

msg/s sema/s IPC for messages and semaphores
kexit/s ksched/s kproc-ov/s Process exits, process switches and process-overload (hit proc thresholds)
runq-sz Avg. process on run queue
%runocc Percent. of time with process on queue
swap-sz Avg. process waiting for page in
%swap-occ Percent. of time with process on queue
cycles/s # of page replace search of all pages
faults/s # of page faults (might not need I/O)
slots # of free pages on paging spaces
odio/s # of non-paging disk I/O per second
file-ov, proc-ov # of times these table overflow per sec
file-sz inode-sz proc-sz Entries in the tables
pswch/s Process switches per second
canch/s outch/s rawch/s Characters per second on terminal lines
rcvin/s xmtin/s Receive and transmit interrupts per second

-p	print the current value
-c M	change to M size (in Mbytes)
-r	restore all memory to use
Examples:
rmss -c 32	Change available memory to 32 Mbytes
rmss -r	Undo the above

-p min	min percentage of memory reserved for file pages (default 20%)
-P max	max percentage of memory reserved for file pages (default 80%)
-f min	number of pages on free list below which page stealling starts (default 120)
-F max	number of pages on free list above which page stealling stops (default 128)
-r min	min number of pages to be read ahead after when sequential access detected
-R max	max number of pages to be read ahead after when sequential access detected

Flag & Default	Meaning
-h 6	Sets system wide criteria for process suspension begins and end (thrashing)
-p 4	Sets per-process criteria for determining process suspension begins and end
-w 1	Seconds to wait before trashing ended
-e 2	Seconds exempt after suspension
-f 10	Clock tick waited after fork failure
-t 0	Clock tick interrupts before dispather called
-D	Restore default values
Examples:
schedtune -t5	set time slice to 50 tcks
schedtune	Report current settings
Code:	/usr/lpp/bos/samples
Compiled:	make schedtune

minservers	min. # of kernel processes
maxserver	max. # of kernel processes
maxrequests	max # of in progress and waiting requests
serverpriority	priority of kernel processes (try 39)
state	has to be available to work

-G	Global report
-P[nsa] pid..	Process report n=non-sys s-system a=both
-S[nsa][upg][x]	Segment report nsa as above + u==real-mem p=pinned g=paging x=top x items
-S sid...	Segment report on particular segments
-i secs count	Repeat report at interval second&count times
-D sid...	Detailed report
Examples:
svmon -G	Global / General stats
svmon -Pa 215	Process report for process 215
svmon -Ssu 10	Top ten system segments in real-mem order
svmon -D 340d	Detailed report on a particular segment
size	in pages (4096)
inuse	in-use
free	not inuse included rmss pages
pin	pined (locked by app.)
work	pages in working segments
pers	pages in persistent segments
clnt	pages inclient segments
pg space	paging space
Note:	pages can be in more than one process

Example:	filemon -O all -o file.out do work load here... finish with trcstop
#MBs	total # of Mbytes transfer during run
#opns	# of times the file was opened
#rpgs	# of 4K page reads
#wpgs	# of 4K page written
#wrs	# of write calls
persistent	paged from file system
working	paged from paging space
util	percentage busy
KB/s	average data transfer rate

-o outputfile	put the output to file not stdout
-T n	Set output buffer size (default 64000)
-P	Force monitor process into pinned memory
-v	Verbose (default only top 20 processes)
-O	cpu, dd(device driver), so(socket), nfs, all
Example:	netpmon -O all -o net.out
	do network load here ... finish with trcstop

-a	all including . and ..
-i inode	find the file(s) with these inode no.
-s	list special and set UID files
Example:
ncheck -a /	List all files in /
ncheck -i 2194 /tmp	find name for inode 2194 in /tmp

-l	long listing
-f	full listing
-u user	list only user's processes (-u fred)
-e	every user's processes
-t ttyno	processes attached to tty (-t 03)
-p pid	list the process number N

Examples:
ps -fu jim	List the user jim processes in full
ps -lt 04	List all processes on terminal tty04
ps -fe	List all processes

Output	Meaning
PID/PPID	Process IDentity&Parent Process IDentity
S	State= Running Sleeping Waiting Zombie Terminating Kernel Intermediate X=growing
UID/USER	User IDentity/User name
C	CPU recent use value (part of priority)
STIME	Start time of process
PRI	Priority (higher means less priority)
NI	NIce value (part of priority) default 20
ADDR	ADDRess, of stack ( segment no)
SZ	SiZe of process in 1K pages
CMD	COMmanD the user typed (-f for more)
WCHAN	Event awaited for (kernel address)
TTY	Terminal processes in connected to (- = none)
TIME	Minutes and Seconds of CPU time
SSIZ	Size of kernel stack
PGIN	# of pages paged in
SIZE	Virtual size of data section in 1K's
RSS	Real memory (resident set) size of process 1K's
LIM	Soft limit on memory (see setrlimit) xx=none
TSIZ	Size of text (shared text program) image
TRS	Size of resident set (real memory) of test
%CPU	Percentage of CPU used since started
%MEM	Percentage of real memory used

interval	number of second between outputs
count	number of times to output
Warning:	ignore the first line (average since reboot)
Examples
iostat 10 20	20 lines output with 10 seconds between each
Output
%tm_act	Percentage of time active
Kbps	K bytes per second transfered
tps	Transfers per second
msps	Milli-second per seek (if available)
Kb_read	Total K bytes read ( likewise for write)

-A	All stats to be collected/reported
-o savefile	Collect stats to binary file
-f savefile	Report stats from binary file
-i secs	Report at secs interval from binary file
-s and -e	Report stats only between these times

Examples:
sar 10 100	Report now at 10 secs intervals
sar -A -o fred 10 6 >/dev/null	Collect data into fred
sar -A -f fred	Report on the data
sar -A -f fred -s 10:30 -e 10:45	Report on 15 minutes from 10:30 a.m.
sar -A -f fred -i60	Report 1 minute interval rather that 10 seconds as collected

sar - CPU
%usr %sys	Percent of time in user / kernel mode
%wio %idle	Percent of time waiting for disk io/idle
Buffer Cache
bread/s bwrit/s	Block I/O per second
lread/s lwrit/s	Logical I/O per sec (hopefully cached
pread/s pwrit/s	Raw disk I/O (not buffer cached)
%rcache %wcache	Percentage hit on cache
Kernel
exec/s fork/s sread/s swrite/s rchar/s wchars/s scall/s	Calls per second of these system calls sread/write system calls (cache, raw, tty or network). scall is the total system calls
msg/s sema/s	IPC for messages and semaphores
kexit/s ksched/s kproc-ov/s	Process exits, process switches and process-overload (hit proc thresholds)
runq-sz	Avg. process on run queue
%runocc	Percent. of time with process on queue
swap-sz	Avg. process waiting for page in
%swap-occ	Percent. of time with process on queue
cycles/s	# of page replace search of all pages
faults/s	# of page faults (might not need I/O)
slots	# of free pages on paging spaces
odio/s	# of non-paging disk I/O per second
file-ov, proc-ov	# of times these table overflow per sec
file-sz inode-sz proc-sz	Entries in the tables
pswch/s	Process switches per second
canch/s outch/s rawch/s	Characters per second on terminal lines
rcvin/s xmtin/s	Receive and transmit interrupts per second