strace on unix systems is good utility to trace systems calls from a specific process. I use it a lot to troubleshoot performance issues and especially detect memory leaks.
So a little history , strace is re-do by a sun employee on the old trace utility. The typical way to use strace is to find a pid (process id) of a particular process you want to trace.
So what I typically end up doing is a
ps aux –sort rss|grep httpd
to sort the process by their rss (resident set size) . In simple words sort the process list by the non-swapped memory utilization of the process.
You can then choose the pid of higher rss process.
Another way which I typically use is to use the apache server-status page. And find the request you want to trace.
To strace the process its fairly straightforward.
strace -p <pid>
strace -ttp <pid> if you want to get the time taken for system call.
Reading the output of an strace is very similar to reading assembly code. You have to think link the cpu of the machine. And realize the small individual steps required to paint the bigger picture.
A couple of the common things to look out for are:
1. a lot of time (), getimeofday () calls these are sometimes called by bad programming where a programmer has the date or time functions being called.
2. a lot xstat64 (lstat64, fstat64, stat64) calls usually made to check attributes on the filesystem.
3. brk, munmap if several of these occur consecutively it usually points to a memory leak problem in that area of the code
memory