strace - trace system calls and signals


   strace  [-CdffhikqrtttTvVxxy]  [-In] [-bexecve] [-eexpr]...  [-acolumn]
   [-ofile]  [-sstrsize]  [-Ppath]...  -ppid...  /  [-D]  [-Evar[=val]]...
   [-uusername] command [args]

   strace  -c[df]  [-In]  [-bexecve]  [-eexpr]...  [-Ooverhead] [-Ssortby]
   -ppid... / [-D] [-Evar[=val]]... [-uusername] command [args]


   In the simplest case strace runs the specified command until it  exits.
   It  intercepts  and  records  the  system  calls  which are called by a
   process and the signals which are received by a process.  The  name  of
   each  system  call,  its  arguments and its return value are printed on
   standard error or to the file specified with the -o option.

   strace is a  useful  diagnostic,  instructional,  and  debugging  tool.
   System administrators, diagnosticians and trouble-shooters will find it
   invaluable for solving problems with programs for which the  source  is
   not  readily available since they do not need to be recompiled in order
   to trace them.  Students, hackers and the overly-curious will find that
   a  great  deal  can  be  learned about a system and its system calls by
   tracing even ordinary programs.  And programmers will find  that  since
   system  calls  and  signals  are  events that happen at the user/kernel
   interface, a close examination of this boundary is very useful for  bug
   isolation, sanity checking and attempting to capture race conditions.

   Each  line  in the trace contains the system call name, followed by its
   arguments in  parentheses  and  its  return  value.   An  example  from
   stracing the command "cat /dev/null" is:

   open("/dev/null", O_RDONLY) = 3

   Errors (typically a return value of -1) have the errno symbol and error
   string appended.

   open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)

   Signals are printed as signal symbol and decoded siginfo structure.  An
   excerpt from stracing and interrupting the command "sleep 666" is:

   sigsuspend([] <unfinished ...>
   --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=...} ---
   +++ killed by SIGINT +++

   If  a  system call is being executed and meanwhile another one is being
   called from a different thread/process then strace will try to preserve
   the  order  of  those  events  and  mark  the  ongoing  call  as  being
   unfinished.  When the call returns it will be marked as resumed.

   [pid 28772] select(4, [3], NULL, NULL, NULL <unfinished ...>
   [pid 28779] clock_gettime(CLOCK_REALTIME, {1130322148, 939977000}) = 0
   [pid 28772] <... select resumed> )      = 1 (in [3])

   Interruption of a (restartable) system call by  a  signal  delivery  is
   processed  differently  as  kernel  terminates the system call and also
   arranges its immediate reexecution after the signal handler completes.

   read(0, 0x7ffff72cf5cf, 1)              = ? ERESTARTSYS (To be restarted)
   --- SIGALRM ... ---
   rt_sigreturn(0xe)                       = 0
   read(0, "", 1)                          = 0

   Arguments are printed in symbolic form with a  passion.   This  example
   shows the shell performing ">>xyzzy" output redirection:

   open("xyzzy", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3

   Here  the  third  argument of open is decoded by breaking down the flag
   argument into its three bitwise-OR constituents and printing  the  mode
   value in octal by tradition.  Where traditional or native usage differs
   from ANSI or POSIX, the latter forms are  preferred.   In  some  cases,
   strace output has proven to be more readable than the source.

   Structure  pointers  are  dereferenced and the members are displayed as
   appropriate.  In all cases arguments are formatted in the  most  C-like
   fashion  possible.   For  example,  the  essence  of the command "ls -l
   /dev/null" is captured as:

   lstat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0

   Notice how the 'struct stat' argument  is  dereferenced  and  how  each
   member  is  displayed  symbolically.   In  particular,  observe how the
   st_mode member is carefully decoded into a bitwise-OR of  symbolic  and
   numeric values.  Also notice in this example that the first argument to
   lstat is an input to the system call and  the  second  argument  is  an
   output.   Since  output  arguments  are not modified if the system call
   fails, arguments may not always be dereferenced.  For example, retrying
   the  "ls  -l"  example  with a non-existent file produces the following

   lstat("/foo/bar", 0xb004) = -1 ENOENT (No such file or directory)

   In this case the porch light is on but nobody is home.

   Character pointers are dereferenced and printed  as  C  strings.   Non-
   printing  characters  in strings are normally represented by ordinary C
   escape codes.  Only the first strsize (32 by default) bytes of  strings
   are  printed;  longer  strings  have an ellipsis appended following the
   closing quote.  Here is a line from "ls -l" where the getpwuid  library
   routine is reading the password file:

   read(3, "root::0:0:System Administrator:/"..., 1024) = 422

   While  structures are annotated using curly braces, simple pointers and
   arrays  are  printed  using  square  brackets  with  commas  separating
   elements.   Here  is  an example from the command "id" on a system with
   supplementary group ids:

   getgroups(32, [100, 0]) = 2

   On the other hand, bit-sets are also shown using  square  brackets  but
   set  elements  are  separated  only  by  a  space.   Here  is the shell
   preparing to execute an external command:

   sigprocmask(SIG_BLOCK, [CHLD TTOU], []) = 0

   Here the second argument is a  bit-set  of  two  signals,  SIGCHLD  and
   SIGTTOU.   In  some  cases the bit-set is so full that printing out the
   unset elements is more valuable.  In that case, the bit-set is prefixed
   by a tilde like this:

   sigprocmask(SIG_UNBLOCK, ~[], NULL) = 0

   Here the second argument represents the full set of all signals.


   -c          Count  time,  calls, and errors for each system call
               and report a summary on  program  exit.   On  Linux,
               this  attempts  to  show system time (CPU time spent
               running in the kernel)  independent  of  wall  clock
               time.   If  -c  is  used with -f or -F (below), only
               aggregate totals for all traced processes are kept.

   -C          Like  -c  but  also  print  regular   output   while
               processes are running.

   -D          Run  tracer process as a detached grandchild, not as
               parent of the  tracee.   This  reduces  the  visible
               effect  of  strace  by  keeping  the tracee a direct
               child of the calling process.

   -d          Show some debugging output of strace itself  on  the
               standard error.

   -f          Trace   child  processes  as  they  are  created  by
               currently  traced  processes  as  a  result  of  the
               fork(2),  vfork(2)  and clone(2) system calls.  Note
               that -p PID -f will attach all  threads  of  process
               PID  if  it  is multi-threaded, not only thread with
               thread_id = PID.

   -ff         If  the  -o  filename  option  is  in  effect,  each
               processes trace is written to where pid
               is the numeric process id of each process.  This  is
               incompatible  with  -c,  since no per-process counts
               are kept.

   -F          This option is now obsolete  and  it  has  the  same
               functionality as -f.

   -h          Print the help summary.

   -i          Print  the  instruction  pointer  at the time of the
               system call.

   -k          Print  the  execution  stack  trace  of  the  traced
               processes  after  each  system  call (experimental).
               This option is available only  if  strace  is  built
               with libunwind.

   -q          Suppress  messages  about  attaching, detaching etc.
               This happens automatically when output is redirected
               to a file and the command is run directly instead of

   -qq         If given twice, suppress messages about process exit

   -r          Print a relative timestamp upon entry to each system
               call.  This records the time difference between  the
               beginning of successive system calls.

   -t          Prefix each line of the trace with the time of day.

   -tt         If  given  twice,  the time printed will include the

   -ttt        If given thrice, the time printed will  include  the
               microseconds and the leading portion will be printed
               as the number of seconds since the epoch.

   -T          Show the time spent in system calls.   This  records
               the  time  difference  between the beginning and the
               end of each system call.

   -w          Summarise the time difference between the  beginning
               and  end  of  each  system  call.  The default is to
               summarise the system time.

   -v          Print unabbreviated versions of  environment,  stat,
               termios,  etc.   calls.   These  structures are very
               common in calls and so the default behavior displays
               a  reasonable subset of structure members.  Use this
               option to get all of the gory details.

   -V          Print the version number of strace.

   -x          Print all non-ASCII strings  in  hexadecimal  string

   -xx         Print all strings in hexadecimal string format.

   -y          Print   paths   associated   with   file  descriptor

   -yy         Print protocol specific information associated  with
               socket file descriptors.

   -a column   Align  return  values  in a specific column (default
               column 40).

   -b syscall  If specified syscall is reached, detach from  traced
               process.    Currently,   only   execve   syscall  is
               supported.  This option is useful  if  you  want  to
               trace  multi-threaded  process and therefore require
               -f, but don't want to trace  its  (potentially  very
               complex) children.

   -e expr     A  qualifying expression which modifies which events
               to trace or how to trace them.  The  format  of  the
               expression is:


               where  qualifier  is  one of trace, abbrev, verbose,
               raw,  signal,  read,  or  write  and  value   is   a
               qualifier-dependent  symbol  or number.  The default
               qualifier  is  trace.   Using  an  exclamation  mark
               negates  the  set  of  values.  For example, -e open
               means literally -e trace=open which  in  turn  means
               trace  only  the  open  system  call.   By contrast,
               -e trace=!open means  to  trace  every  system  call
               except  open.   In  addition, the special values all
               and none have the obvious meanings.

               Note that some shells use the exclamation point  for
               history  expansion even inside quoted arguments.  If
               so, you must escape the  exclamation  point  with  a

   -e trace=set
               Trace  only  the specified set of system calls.  The
               -c option is useful  for  determining  which  system
               calls  might  be  useful  to  trace.   For  example,
               trace=open,close,read,write  means  to  only   trace
               those  four  system  calls.   Be careful when making
               inferences about the user/kernel boundary if only  a
               subset  of  system  calls  are being monitored.  The
               default is trace=all.

   -e trace=file
               Trace all system calls which take a file name as  an
               argument.   You can think of this as an abbreviation
               for  -e trace=open,stat,chmod,unlink,...   which  is
               useful   to   seeing   what  files  the  process  is
               referencing.  Furthermore,  using  the  abbreviation
               will  ensure  that  you don't accidentally forget to
               include a call like  lstat  in  the  list.   Betchya
               woulda forgot that one.

   -e trace=process
               Trace   all   system  calls  which  involve  process
               management.  This is useful for watching  the  fork,
               wait, and exec steps of a process.

   -e trace=network
               Trace all the network related system calls.

   -e trace=signal
               Trace all signal related system calls.

   -e trace=ipc
               Trace all IPC related system calls.

   -e trace=desc
               Trace all file descriptor related system calls.

   -e trace=memory
               Trace all memory mapping related system calls.

   -e abbrev=set
               Abbreviate  the  output from printing each member of
               large structures.  The default is  abbrev=all.   The
               -v option has the effect of abbrev=none.

   -e verbose=set
               Dereference  structures  for  the  specified  set of
               system calls.  The default is verbose=all.

   -e raw=set  Print raw, undecoded arguments for the specified set
               of  system  calls.   This  option  has the effect of
               causing all arguments to be printed in  hexadecimal.
               This  is  mostly  useful  if  you  don't  trust  the
               decoding or you need  to  know  the  actual  numeric
               value of an argument.

   -e signal=set
               Trace  only  the  specified  subset of signals.  The
               default is signal=all.  For example, signal =! SIGIO
               (or  signal=!io)  causes  SIGIO  signals  not  to be

   -e read=set Perform a full hexadecimal and ASCII dump of all the
               data  read  from  file  descriptors  listed  in  the
               specified  set.   For  example,  to  see  all  input
               activity   on   file   descriptors   3   and  5  use
               -e read=3,5.  Note that this is independent from the
               normal  tracing  of the read(2) system call which is
               controlled by the option -e trace=read.

   -e write=set
               Perform a full hexadecimal and ASCII dump of all the
               data  written  to  file  descriptors  listed  in the
               specified set.   For  example,  to  see  all  output
               activity   on   file   descriptors   3   and  5  use
               -e write=3,5.  Note that this  is  independent  from
               the normal tracing of the write(2) system call which
               is controlled by the option -e trace=write.

   -I interruptible
               When strace can be interrupted by signals  (such  as
               pressing  ^C).   1: no signals are blocked; 2: fatal
               signals   are   blocked   while   decoding   syscall
               (default);  3:  fatal  signals  are  always  blocked
               (default if '-o FILE PROG'); 4:  fatal  signals  and
               SIGTSTP  (^Z)  are  always  blocked  (useful to make
               strace -o FILE PROG not stop on ^Z).

   -o filename Write the trace output to the file  filename  rather
               than  to  stderr.   Use if -ff is used.
               If the argument begins with '|' or with '!' then the
               rest of the argument is treated as a command and all
               output is piped  to  it.   This  is  convenient  for
               piping  the  debugging  output  to a program without
               affecting the redirections of executed programs.

   -O overhead Set  the  overhead  for  tracing  system  calls   to
               overhead   microseconds.    This   is   useful   for
               overriding the default heuristic  for  guessing  how
               much  time  is  spent  in mere measuring when timing
               system calls using the -c option.  The  accuracy  of
               the  heuristic  can  be  gauged  by  timing  a given
               program run  without  tracing  (using  time(1))  and
               comparing  the  accumulated  system call time to the
               total produced using -c.

   -p pid      Attach to the process with the process  ID  pid  and
               begin  tracing.   The trace may be terminated at any
               time  by  a  keyboard  interrupt  signal   (CTRL-C).
               strace  will  respond  by  detaching itself from the
               traced process(es) leaving  it  (them)  to  continue
               running.   Multiple -p options can be used to attach
               to many processes in addition to command  (which  is
               optional  if  at  least one -p option is given).  -p
               "`pidof PROG`" syntax is supported.

   -P path     Trace only system calls accessing path.  Multiple -P
               options can be used to specify several paths.

   -s strsize  Specify  the  maximum  string  size  to  print  (the
               default  is  32).   Note  that  filenames  are   not
               considered strings and are always printed in full.

   -S sortby   Sort  the  output of the histogram printed by the -c
               option by the specified criterion.  Legal values are
               time, calls, name, and nothing (default is time).

   -u username Run   command  with  the  user  ID,  group  ID,  and
               supplementary groups of username.   This  option  is
               only  useful  when  running  as root and enables the
               correct execution of setuid and/or setgid  binaries.
               Unless   this  option  is  used  setuid  and  setgid
               programs are executed without effective privileges.

   -E var=val  Run command with var=val in its list of  environment

   -E var      Remove  var  from  the inherited list of environment
               variables before passing it on to the command.


   When command exits, strace exits with the same exit status.   If
   command is terminated by a signal, strace terminates itself with
   the same signal, so that strace can be used as a wrapper process
   transparent  to  the invoking parent process.  Note that parent-
   child relationship (signal stop notifications, getppid()  value,
   etc)  between  traced  process  and its parent are not preserved
   unless -D is used.

   When using -p, the exit status of strace is  zero  unless  there
   was an unexpected error in doing the tracing.


   If  strace  is  installed  setuid to root then the invoking user
   will be able to attach to and trace processes owned by any user.
   In  addition  setuid  and  setgid  programs will be executed and
   traced with the correct effective privileges.  Since only  users
   trusted  with full root privileges should be allowed to do these
   things, it only makes sense to install strace as setuid to  root
   when  the users who can execute it are restricted to those users
   who have this trust.  For example, it makes sense to  install  a
   special  version  of strace with mode 'rwsr-xr--', user root and
   group trace, where members of the trace group are trusted users.
   If  you  do  use this feature, please remember to install a non-
   setuid version of strace for ordinary lusers to use.


   ltrace(1), time(1), ptrace(2), proc(5)


   It is a pity that so much tracing clutter is produced by systems
   employing shared libraries.

   It  is instructive to think about system call inputs and outputs
   as data-flow across the  user/kernel  boundary.   Because  user-
   space and kernel-space are separate and address-protected, it is
   sometimes possible to make deductive  inferences  about  process
   behavior using inputs and outputs as propositions.

   In  some  cases,  a  system call will differ from the documented
   behavior or have a different name.  For example,  on  System  V-
   derived  systems  the  true time(2) system call does not take an
   argument and the stat function is  called  xstat  and  takes  an
   extra  leading  argument.   These  discrepancies  are normal but
   idiosyncratic characteristics of the system call  interface  and
   are accounted for by C library wrapper functions.

   Some   system   calls   have   different   names   in  different
   architectures and personalities.  In these  cases,  system  call
   filtering  and  printing uses the names that match corresponding
   __NR_*  kernel  macros  of   the   tracee's   architecture   and
   personality.   There  are two exceptions from this general rule:
   arm_fadvise64_64(2)  ARM  syscall   and   xtensa_fadvise64_64(2)
   Xtensa syscall are filtered and printed as fadvise64_64(2).

   On  some  platforms  a  process  that is attached to with the -p
   option may observe a spurious  EINTR  return  from  the  current
   system call that is not restartable.  (Ideally, all system calls
   should  be  restarted  on  strace  attach,  making  the   attach
   invisible  to the traced process, but a few system calls aren't.
   Arguably, every instance of such  behavior  is  a  kernel  bug.)
   This  may  have  an  unpredictable  effect on the process if the
   process takes no action to restart the system call.


   Programs that use the setuid bit do not have effective  user  ID
   privileges while being traced.

   A traced process runs slowly.

   Traced  processes  which  are descended from command may be left
   running after an interrupt signal (CTRL-C).

   The -i option is weakly supported.


   The original strace was written by Paul Kranenburg for SunOS and
   was  inspired by its trace utility.  The SunOS version of strace
   was ported to Linux and enhanced by Branko Lankester,  who  also
   wrote  the  Linux  kernel  support.   Even  though Paul released
   strace 2.5 in 1992, Branko's work was based on Paul's strace 1.5
   release  from 1991.  In 1993, Rick Sladkey merged strace 2.5 for
   SunOS and the second release of strace for Linux, added many  of
   the  features of truss(1) from SVR4, and produced an strace that
   worked on both platforms.  In 1994 Rick ported  strace  to  SVR4
   and  Solaris  and wrote the automatic configuration support.  In
   1995 he ported strace to Irix and tired of writing about himself
   in the third person.


   Problems  with  strace  should be reported to the strace mailing
   list at <>.

                              2010-03-30                         STRACE(1)


Personal Opportunity - Free software gives you access to billions of dollars of software at no cost. Use this software for your business, personal use or to develop a profitable skill. Access to source code provides access to a level of capabilities/information that companies protect though copyrights. Open source is a core component of the Internet and it is available to you. Leverage the billions of dollars in resources and capabilities to build a career, establish a business or change the world. The potential is endless for those who understand the opportunity.

Business Opportunity - Goldman Sachs, IBM and countless large corporations are leveraging open source to reduce costs, develop products and increase their bottom lines. Learn what these companies know about open source and how open source can give you the advantage.

Free Software

Free Software provides computer programs and capabilities at no cost but more importantly, it provides the freedom to run, edit, contribute to, and share the software. The importance of free software is a matter of access, not price. Software at no cost is a benefit but ownership rights to the software and source code is far more significant.

Free Office Software - The Libre Office suite provides top desktop productivity tools for free. This includes, a word processor, spreadsheet, presentation engine, drawing and flowcharting, database and math applications. Libre Office is available for Linux or Windows.

Free Books

The Free Books Library is a collection of thousands of the most popular public domain books in an online readable format. The collection includes great classical literature and more recent works where the U.S. copyright has expired. These books are yours to read and use without restrictions.

Source Code - Want to change a program or know how it works? Open Source provides the source code for its programs so that anyone can use, modify or learn how to write those programs themselves. Visit the GNU source code repositories to download the source.


Study at Harvard, Stanford or MIT - Open edX provides free online courses from Harvard, MIT, Columbia, UC Berkeley and other top Universities. Hundreds of courses for almost all major subjects and course levels. Open edx also offers some paid courses and selected certifications.

Linux Manual Pages - A man or manual page is a form of software documentation found on Linux/Unix operating systems. Topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts.