Why cached reads are slower than disk reads in hdparm –direct?
I am trying to interpret this result of hdparm:
janus@behemoth ~ $ sudo hdparm -Tt --direct /dev/nvme0n1 /dev/nvme0n1: Timing O_DIRECT cached reads: 2548 MB in 2.00 seconds = 1273.69 MB/sec Timing O_DIRECT disk reads: 4188 MB in 3.00 seconds = 1395.36 MB/sec
I do not understand how the cached reads can be slower than the direct disk reads. If I drop the –direct, I get what I would have expect: the disk reads are slower than the cached ones:
janus@behemoth ~ $ sudo hdparm -Tt /dev/nvme0n1 /dev/nvme0n1: Timing cached reads: 22064 MB in 2.00 seconds = 11042.86 MB/sec Timing buffered disk reads: 2330 MB in 3.00 seconds = 776.06 MB/sec
(Although it says “buffered disk reads” now).
Can somebody explain to me what is going on?
hdparm man page:
--direct Use the kernel´s "O_DIRECT" flag when performing a -t timing test. This bypasses the page cache, causing the reads to go directly from the drive into hdparm's buffers, using so-called "raw" I/O. In many cases, this can produce results that appear much faster than the usual page cache method, giving a better indication of raw device and driver performance.
It so explains why
hdparm -t --direct may be faster than
hdparm -t. It also says that
--direct only applies to the
-t test, not to the
-T test which is not supposed to involve the disk (see below).
-T Perform timings of cache reads for benchmark and comparison pur‐ poses. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. This displays the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.
-T works by reading the same cached part of the disk. But your
--direct prevents this. So, logically, you should have the same results with
-t --direct as with
Based on my research,
--direct sends any data acquired through the hard drive at least once. That means if the system cache procures data at 10GB/sec but the poor hard drive controller bogs down at 400MB/sec, 400MB/sec is what you will get.
As a refresher
-Tt --directDoes two runs:
- reads the system cache, then sends it to the hard drive controller and back. This can be useful perhaps for testing SATA connections and max buffer speeds.
- Reads the disk without it’s buffer.
-Treads the system cache directly, tests processor and raid driver overhead
-treads the disk with the buffer enabled