发信人: coobear(冥思苦想...)
整理人: hlzhan(2002-09-22 14:37:56), 站内信件
|
2.3.4. The day cometh, truss is your friend
truss(1M) is a utility which executes a specified command and produces a trace of the system calls it performs, the signals it receives, and the machine faults it incurs.
truss can also follow the execution of an existing process. truss is a very useful to tool to narrow down what resources an application is requesting from the kernel that are slow or are used to excess.
If you don't know about truss, then read the man page and have a play. The -m option is very useful for showing faults such as page faults. The -c option gives a summary of system calls, faults, and signals and the cumulative times spent in each system call type and number of failed system calls.
2.3.5. Lockstat - contention for resources
Kernel locks protect data structures multiple updates and controls access to resources such as disk, network and various kernel caches.
lockstat, which appeared in 2.6, executes a command and reports all kernel lock activity for the duration of the command, irrespective of the
process or device which made the request for a lock. In Solaris 8 it allows the kernel to be profiled, similar to the undocumented netstat -k option available in previous releases of Solaris. See the lockstat(1M)man page. the option -s 10 reports the stack of the kernel threads contenting on each lock.
2.3.6. Trapstat
trapstat is a tool to provide run-time trap statistics on UltraSPARCs running an otherwise stock Solaris kernel. For I-TLB and D-TLB misses, trapstat can optionally display the amount of time spent in the operating system's TLB miss handler. For interrupt vector traps, trapstat can optionally display the interrupting device. Trapstat will be part of Solaris 9 and a binary is available for use on previous releases.
2.3.7. gprof - Application Profiling
For home grown C, C++ and FORTRAN applications, try compiling with -xpg, execute the program with a typical workload that demonstrates the performance problem. Run gprof on the generated tmon.out file. This will show where the application is spending its most time.
Forte Workshop has a number of useful tools such as the analyzer which provide a graphical representation of where the application is spending its time. See workshop documentation and the book "Techniques For Optimizing Applications: High Performance Computing" by Rajat Garg and Ilya Sharapov for further details.
2.3.8. proc tools
The proc tools are utilities that exercise features of /proc reporting attributes of a process such as; pstack, the call stack; ptree, a tree of process relationships; pfiles, a list of open file descriptors; and pldd, a list of dynamic libraries in use by the running process. See man proc(1).
3. Some Commonly Asked Questions and some suggestions
3.1. What does 64-bit Sizing and Capacities buy me?
From a performance point of view, the ability to run 64-bit applications has two main benefits. One is that much larger problems can be solved efficiently using a bigger process address space; the other is that integer arithmetic computations get to use 64-bit registers and operations. Overall, programs get slightly larger due to larger pointer values in code and data structures. This in turn means that CPU caches are a little less likely to have enough cache lines, and a slight slowdown might occur in programs that could run just as well in a 32-bit environment.
Kernel thread stacks are 16k, rather than 8k, though the effect is usually negligible.
3.2. Free Memory
Examining a Solaris system to determine the amount of memory free has traditionally be an area of confusion.
For Solaris releases before 8, to look for a shortage of memory, do not rely upon the "free" column or the sr column. The value in the fr column is not an indication of a lack of memory. The page cache holding on to pages in case they may be needed again. The VM subsystem will only reclaim memory when needed.
Much has been written on this subject in the SunWorld articles and "Sun Performance and Tuning". To determine if there is a lack of memory examine at the 12th column, "sr", or scan rate and in conjunction with I/O traffic to the swap partitions(with iostat -P) on disk. The sr column may have high figures if a large amount of I/O is being generated through the filesystem and the page scanner needs to run to free up pages for I/O.
The pageout scanner runs only when the free list shrinks below a threshold (lotsfree in pages). Any process or file inactive and not locked in memory may be paged out. The size of the freelist will appear to shrink, and will remain at that value(lotsfree). The page daemon will start to scan for memory to be reclaim from the page cache, exited and idle processes when the amount on the freelist drops below the lotsfree threshold. There is no way for the "free" value to grow much above the threshold, because there is no way to get the page scanner to work to reclaim memory beyond the threshold. It is more efficient for pages to be left in the page cache, than needlessly put on the free list.
Solaris 8 cleans up much confusion in this area and implements a more efficient algorithm within the segmap driver to provide the pages required for I/O. The fr column in vmstat really does now mean memory which is free and not used by the page cache. The -p option has been added to vmstat to give a more accurate breakdown of paging behaviour.
For individual processes, the pmap command reports the address space layout of an individual process(-x option is useful).
3.3. Priority paging
Priority paging was introduced with Solaris 7 and was backported to Solaris 2.6 (kernel patch 105181-09) and Solaris 2.5.1 (kernel patch 103640-25).
Priority paging provides an improved paging algorithm which can significantly enhance system response when the file system is being used. Priority paging introduces a new additional water mark, cachefree. The paging parameters are now:
minfree < desfree < lotsfree < cachefree
By default the new behaviour is turned off in Solaris 2.5.1, 2.6 and 7, so it is important to enable this functionality on systems that are paging noticeably. cachefree is set to lotsfree if priority_paging is not enabled. If it is enabled then cachefree is set to 2 times lotsfree by default.
Adjusting this parameter tends to make switching between windows on desktop systems faster, and is a big help for systems running databases that read large files into memory from the filesystem. For system which do a large amount of I/O through a filesystem, speed increases of several hundred percent have been seen for compute-intensive jobs with a large dataset.
Solaris 8 uses a different algorithm which removes the limiting factor of previous releases where the page scanner had to scan for memory to supply the segmap driver with memory in which to place I/O. All pages that the segmap no longer uses are put on a list allowing immediate reuse. Do not set priority_paging in Solaris 8. In addition, Solaris 8 should not require tuning of virtual memory parameters, except on large systems where setting fastscan and maxpgio to higher values may be useful.
For more information on priority paging refer to the following URLs:
http://www.sunworld.com/swol-11-1998/swol-11-perf.html
http://www.sun.com/sun-on-net/performance/priority_paging.html
INFODOC ID: 17946: has more details on prority paging.
3.4. Intimate Shared Memory (ISM)
ISM provides for the shared memory to be locked in memory and cannot be paged out. Memory management data structures that are normally created on a per process basis are created once and then shared by every process. In Solaris 2.6 a further optimisation takes place as the kernel tries to find 4-Mbyte contiguous blocks of physical memory that can be used as large pages to map the shared memory. This greatly reduces memory management unit overhead. (p.333 "Performance and Tuning - Java and the Internet") By default, applications such as Oracle, Informix, and Sybase use a special flag to specify that they want intimate shared memory (ISM).
Intimate shared memory is an important optimisation that makes more efficient use of the kernel and hardware resources involved in the implementation of virtual memory and provides a means of keeping heavily used shared pages locked in memory.
Intimate shared memory is enabled by default and there is no need to edit the /etc/system file to turn on this feature. With a currently patched kernel, turning off ISM can cause system degradation and possibly a hang condition. In addition database configuration files, such as Oracle's init.ora file, should not have "use_ism=false" which turns it off.
3.5. Swap Configurations Related to Shared Memory
To understand swap configurations related to shared memory see Sunworldonline, "Swap space implementation part 2" by Jim Mauro:
The two primary considerations in setting swap space size are:
a. Have enough memory to avoid swapping in common operation
b. Have enough swap to get a crash dump
3.6. Interprocess Communication Parameters (IPC)
The values for the following IPC parameters need to be determined by the Database Administrator (DBA). Sun Solution Centres can not give recommendations for what the actual IPC parameter settings should be. These values are application dependent.
It is *very* easy to mis-type the /etc/system setting for IPC parameters which can have a significant performance impact on the application. Make a trawl through /var/adm/messages for a message of the form:
genunix: [ID 492708 kern.notice] sorry, variable 'seminfo_semopn' is not defined in the 'semsys'
which indicates a typo in the line. Grep for "sorry".
Solaris 8 has more reasonable defaults for these values than previous releases.
For Solaris releases previous to 2.6 more swap space (as "backing store") is needed for shared memory. Using swap -l, divide the block numbers by 2 to get megabytes. There should be at least 2 times the amount of swap available for allocated shared memory (shmmax).
Here is the default and maximum values for shmmax:
Default Maximum
shmmax 1048576 (Meg) 4294967295 (4GB) 2.5.1, 2.6, 32bit solaris 7
2147483647 (2GB) 2.5 or lower
Solaris 2.6 shmmax and shmmin are unsigned ints (32 bit). Solaris 7 "32-bit" shmmax and shmmin are unsigned ints (32 bit). Solaris 7 "64-bit" shmmax and shmmin are unsigned longs (64 bit). In all cases, shmmni and shmseg are signed ints (31 bit).
shmmax limits the maximum size of a shared memory segment, which is the largest value which can be requested of shmget(2). The resource it controls is not preallocated. It is allocated on demand.
Solaris 7 and 8 64-bit breaks the 4GB barrier. The maximum size is theoretical. The actual settings need to be based on the system resources like memory and database sizes and configurations. The maximum size of the segment itself (shmmax) is an upper limit.
Appendix
A. IPC articles in SunSolve SRDBs and InfoDoc Collections
There are numerous articles that have been written by the Sun Solution Centres on the subject of IPC parameters. They are available on the SunSolve web site (sunsolve.sun.com) after logging in as a contract customer. Here is a partial list:
If modifications to the /etc/system file do not seem to have taken effect, read the following:
SRDB ID: 12824: sysdef -i does not report IPC parameters set
in /etc/system
For general information on the IPC parameters:
INFODOC ID: 13421: Shared Memory Commands Explained
INFODOC ID: 6328: All about Shared Memory Parameters in 2.X
INFODOC ID: 2270: Understanding semaphores, seminfo_ semaphore info
INFODOC ID: 13523: Semaphores Explained
SRDB ID: 12075: How to configure the IPC semaphores and shared memory
SRDB ID: 5288: How to determine the IPC semaphore parameter values
INFODOC ID: 2273: Kernel tuning parameters for message queues
INFODOC ID: 7241: Determine the message queue parameters
For debugging problems:
SRDB ID: 12174: How to check how much shared memory is used by system
INFODOC ID: 13480: How to use the UNDO feature in semaphore operations
SRDB ID: 16985: A process using shared memory has terminated, but
swap space doesn't seem to get reclaimed.
|
|