Why cached reads are slower than disk reads in hdparm –direct?

Question

Why cached reads are slower than disk reads in hdparm –direct?

I am trying to interpret this result of hdparm:

janus@behemoth ~ $ sudo hdparm  -Tt --direct /dev/nvme0n1

/dev/nvme0n1:
 Timing O_DIRECT cached reads:   2548 MB in  2.00 seconds = 1273.69 MB/sec
 Timing O_DIRECT disk reads: 4188 MB in  3.00 seconds = 1395.36 MB/sec

I do not understand how the cached reads can be slower than the direct disk reads. If I drop the –direct, I get what I would have expect: the disk reads are slower than the cached ones:

janus@behemoth ~ $ sudo hdparm  -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   22064 MB in  2.00 seconds = 11042.86 MB/sec
 Timing buffered disk reads: 2330 MB in  3.00 seconds = 776.06 MB/sec

(Although it says “buffered disk reads” now).

Can somebody explain to me what is going on?

Asked By: Alejandro DC

||

Source

Answer 1

Per hdparm man page:

--direct

Use the kernel´s "O_DIRECT" flag when  performing  a  -t  timing
test.   This  bypasses  the  page cache, causing the reads to go
directly from the drive into hdparm's buffers,  using  so-called
"raw"  I/O.  In many cases, this can produce results that appear
much faster than the usual page cache method,  giving  a  better
indication of raw device and driver performance.

It so explains why hdparm -t --direct may be faster than hdparm -t. It also says that --direct only applies to the -t test, not to the -T test which is not supposed to involve the disk (see below).

-T 

Perform timings of cache reads for benchmark and comparison pur‐
poses.   For  meaningful  results,  this  operation  should   be  
repeated  2-3  times  on  an otherwise inactive system (no other
active processes) with at least a couple of  megabytes  of  free
memory.   This  displays  the speed of reading directly from the 
Linux buffer cache without disk  access.   This  measurement  is  
essentially  an  indication  of the throughput of the processor,
cache, and memory of the system under test.

I guess -T works by reading the same cached part of the disk. But your --direct prevents this. So, logically, you should have the same results with -t --direct as with -T --direct.

Answered By: xhienne

Answer 2

Based on my research, --direct sends any data acquired through the hard drive at least once. That means if the system cache procures data at 10GB/sec but the poor hard drive controller bogs down at 400MB/sec, 400MB/sec is what you will get.

As a refresher

-Tt --direct Does two runs:
- 1. reads the system cache, then sends it to the hard drive controller and back. This can be useful perhaps for testing SATA connections and max buffer speeds.
- 1. Reads the disk without it’s buffer.
-T reads the system cache directly, tests processor and raid driver overhead
-t reads the disk with the buffer enabled

Answered By: Ray Foss