/sys/kernel/debug/tracing/trace overwrites itself after a few seconds. How can I collect it without duplicates?

When setting up the function_graph tracer in Linux (Ubuntu 18), the trace that is stored at /sys/kernel/debug/tracing/trace only stores a couple of seconds before overwriting itself.

As the period might be variable, I cannot be saving it with for example

cat /sys/kernel/debug/tracing/trace >> total_trace

Because it might produce duplicates which are not acceptable during postprocessing. Even worst, it might miss some information.

Is there a way I can open the file and just pipe all the new incoming info to another one?


Asked By: Gaston


From ftrace documentation:


    The output is the same as the "trace" file but this
    file is meant to be streamed with live tracing. 
    Reads from this file will block until new data is
    retrieved.  Unlike the "trace" file, this file is a
    consumer.  This means reading from this file causes
    sequential reads to display more current data.  Once
    data is read from this file, it is consumed, and
    will not be read again with a sequential read. 
    The "trace" file is static, and if the tracer is not
    adding more data, it will display the same
    information every time it is read. 
    This file will not disable tracing while being read.

Or use trace-cmd(1), and specifically trace-cmd-show
From its man page:

-p                        (lower case ‘P’)

    Instead of displaying the contents of the "trace" file, use
    the "trace_pipe" file. The difference between the two is that
    the "trace" file is static. That is, if tracing is stopped,
    the "trace" file will show the same contents each time.

        The "trace_pipe" file is a consuming read, where a read of the file
        will consume the output of what was read and it will not read the
        same thing a second time even if tracing is stopped. This file
        als will block. 
        If no data is available, trace-cmd show will stop
        and wait for data to appear.

Also see Knio’s answer to How to set the buffer size
for trace_pipe in ftrace?
(on Stack Overflow)
to see how you can change the size and behavior of the trace file.

Answered By: aviro


  1. cat trace_pipe
  2. cat trace_pipe | grep nfs
  3. cat trace_pipe > kernel_trace.txt
Answered By: Sagar Singh
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.