journalctl shows very old boots which are not recycled

Question

journalctl shows very old boots which are not recycled

The system has very old boots (2 and 3 years old), which are not recycled and I haven’t been able to vacuum them:

$ journalctl --list-boots --no-pager
-16 53baf678f0d749d6b390afea4a3ef96b Wed 2014-04-02 22:07:26 IDT—Wed 2014-04-02 22:46:08 IDT
-15 60a54132f5c8450d9b33a77819a037d1 Thu 2014-04-03 00:04:50 IDT—Thu 2014-04-03 12:30:21 IDT
-14 24b65a7e589d4479bf5020b98b8120b7 Wed 2015-04-01 03:10:01 IDT—Wed 2015-04-01 08:35:21 IDT
-13 43398d6d74c849bcb359a2d3963f4aaa Wed 2015-04-08 00:26:31 IDT—Wed 2015-04-08 00:26:31 IDT
-12 51b28f394cbb4699b2c4098297f73b2e Mon 2017-07-24 18:28:02 IDT—Mon 2017-07-24 19:08:37 IDT
-11 67467a640fb5413189f9cd518a56f668 Tue 2017-07-25 01:21:00 IDT—Mon 2017-07-24 22:54:40 IDT
-10 1370875e2f2c4b3c80c82901367b0835 Tue 2017-07-25 03:05:18 IDT—Tue 2017-07-25 01:18:51 IDT
-9 462c24a6b4cd487c834e121240bb880c Tue 2017-07-25 13:16:02 IDT—Tue 2017-07-25 18:50:32 IDT
-8 970d61bd3a6f455bb67a7a77c788b930 Wed 2017-07-26 00:01:58 IDT—Wed 2017-07-26 02:37:09 IDT
-7 dc33b354faa64c7c981da25eb9b77bde Wed 2017-07-26 12:45:04 IDT—Wed 2017-07-26 17:01:02 IDT
-6 1bb69b41c09c40aea412714b09678cf2 Wed 2017-07-26 20:01:12 IDT—Wed 2017-07-26 18:43:34 IDT
-5 9a6ed1de771d4056b8be15409dfe06f4 Wed 2017-07-26 23:18:25 IDT—Tue 2017-08-01 00:04:56 IDT
-4 e3eba22761bc470ca9bd1d9004478ad1 Tue 2017-08-01 13:12:55 IDT—Tue 2017-08-01 18:40:22 IDT
-3 02d288fc10714e0592b24ea1cbaf60e4 Tue 2017-08-01 23:00:01 IDT—Wed 2017-08-02 01:29:53 IDT
-2 3230c51e8792424aaec920fa15fa96c0 Wed 2017-08-02 12:54:43 IDT—Wed 2017-08-02 18:09:52 IDT
-1 10621f49412c43cf976ab30555e6eb36 Wed 2017-08-02 22:37:36 IDT—Wed 2017-08-02 20:22:57 IDT
0 d2e38bd2d96b4027ac14e132638561fb Wed 2017-08-02 23:23:07 IDT—Thu 2017-08-03 00:03:01 IDT

I tried vacuuming by time:

$ sudo journalctl --vacuum-time=1years

But nothing was deleted.

I tried by files:

$ sudo journalctl --vacuum-files=12

Deleted archived journal /var/log/journal/b78deda26d9b4740a6bc52f31d993baf/system@412dc1544fb841d4909752bffd03e810-0000000000000001-00055511dc545ff3.journal (16.0M).
Deleted archived journal /var/log/journal/b78deda26d9b4740a6bc52f31d993baf/user-1000@15238a40d91c40fabe10b0bec7f53a23-00000000000009ef-0005550f5956d667.journal (25.0M).
Deleted archived journal /var/log/journal/b78deda26d9b4740a6bc52f31d993baf/system@412dc1544fb841d4909752bffd03e810-000000000000b685-00055528569c7855.journal (16.0M).
Deleted archived journal /var/log/journal/b78deda26d9b4740a6bc52f31d993baf/user-1000@15238a40d91c40fabe10b0bec7f53a23-000000000000b5cd-0005552854d417fd.journal (25.0M).
Vacuuming done, freed 82.0M of archived journals on disk.

It deleted some boots but not the ones I expected:

$ journalctl --list-boots --no-pager
-9 53baf678f0d749d6b390afea4a3ef96b Wed 2014-04-02 22:07:26 IDT—Wed 2014-04-02 22:46:08 IDT
-8 60a54132f5c8450d9b33a77819a037d1 Thu 2014-04-03 00:04:50 IDT—Thu 2014-04-03 12:30:21 IDT
-7 24b65a7e589d4479bf5020b98b8120b7 Wed 2015-04-01 03:10:01 IDT—Wed 2015-04-01 08:35:21 IDT
-6 43398d6d74c849bcb359a2d3963f4aaa Wed 2015-04-08 00:26:31 IDT—Wed 2015-04-08 00:26:31 IDT
-5 9a6ed1de771d4056b8be15409dfe06f4 Wed 2017-07-26 23:23:00 IDT—Tue 2017-08-01 00:04:56 IDT
-4 e3eba22761bc470ca9bd1d9004478ad1 Tue 2017-08-01 13:12:55 IDT—Tue 2017-08-01 18:40:22 IDT
-3 02d288fc10714e0592b24ea1cbaf60e4 Tue 2017-08-01 23:00:01 IDT—Wed 2017-08-02 01:29:53 IDT
-2 3230c51e8792424aaec920fa15fa96c0 Wed 2017-08-02 12:54:43 IDT—Wed 2017-08-02 18:09:52 IDT
-1 10621f49412c43cf976ab30555e6eb36 Wed 2017-08-02 22:37:36 IDT—Wed 2017-08-02 20:22:57 IDT
 0 d2e38bd2d96b4027ac14e132638561fb Wed 2017-08-02 23:23:07 IDT—Thu 2017-08-03 00:09:37 IDT

How can I get rid of those ancient boots from 2014 and 2015? Why are they preserved?

Update

As per derobert’s suggestion listing the directory returns the following:

$ ls -lt /var/log/journal/b78deda26d9b4740a6bc52f31d993baf/ | tail
-rw-r-----+ 1 root systemd-journal 26214400 Aug  2 22:44 user-1000@15238a40d91c40fabe10b0bec7f53a23-0000000000026307-000555aff155c595.journal
-rw-r-----+ 1 root systemd-journal 16777216 Aug  2 22:44 system@412dc1544fb841d4909752bffd03e810-0000000000026340-000555b006ce5e6d.journal
-rw-r-----+ 1 root systemd-journal 26214400 Aug  1 15:04 user-1000@15238a40d91c40fabe10b0bec7f53a23-000000000001d951-0005558547686519.journal
-rw-r-----+ 1 root systemd-journal  8388608 Aug  1 15:04 system@412dc1544fb841d4909752bffd03e810-000000000001d952-000555854768665a.journal
-rw-r-----+ 1 root systemd-journal 26214400 Jul 30 12:10 user-1000@15238a40d91c40fabe10b0bec7f53a23-0000000000015378-0005553e36e51b82.journal
-rw-r-----+ 1 root systemd-journal  8388608 Jul 30 12:03 system@412dc1544fb841d4909752bffd03e810-0000000000015393-0005553e40691231.journal
-rw-r-----+ 1 root systemd-journal  8388608 Apr 18  2015 user-1002.journal
-rwxr-xr-x+ 1 root systemd-journal  8388608 Apr  8  2015 user-42.journal
-rwxr-xr-x+ 1 root systemd-journal  8388608 Apr  1  2015 user-995.journal
-rwxr-xr-x+ 1 root systemd-journal  8388608 Apr  3  2014 user-1001.journal

Asked By: dchrome

||

Source

Answer 1

The new binary logs on Linux operating systems do not work in the way that the old binary logs did.

The old binary logs were /var/log/wtmp and /var/log/btmp. At system bootstrap an entry would be written to wtmp with the username reboot, and at shutdown an entry would be written to wtmp with the username shutdown. Finding the times that the system was rebooted was a matter of using the last reboot and last shutdown commands to print out these entries.

The new binary logs are the systemd journal, and they do not have such entries.

Instead, every journal record has a field named the boot ID. You can see this with the -o verbose option to journalctl. A boot ID is generated by the kernel at bootstrap, and systemd-journald applies the current boot ID, taken from the kernel, to every log record as it is adding it to the journal.

To implement the list-boots functionality, journalctl scans the entire journal, reading the timestamps and boot IDs of every record, and noting the earliest and latest timestamps associated with each unique boot ID.

There are no explicit boot log entries to vacuum, as was the case with the old binary logs. Instead you have to get rid of all entries stamped with the boot IDs that you no longer want to see.

Your problem is that because you are a privileged user, the entire journal also includes the per-user journals: those user-1001.journal and user-42.journal files that you can see in your directory listing. And as you can also see, you have per-user journals for some users that have not been touched since 2014 and 2015. Those are where the journal entries are that have these old boot IDs.

As the manual says, “vacuuming” only operates on archived journal files, not on the currently active files. But as you can further see those various per-user journals dated 2014 and 2015 are the still currently active files for those users. Ironically, they never grew large enough to be archived.

You have two options:

Use the --system option to restrict journalctl --list-boots to just the system files and have it not read the per-user files.
Use journalctl --rotate to force the per-user journals to be archived (as well as the system journal — cave!), so that they will be “vacuum”-able. If your journalctl does not have this, you have to send a signal to the systemd-journald process, as described on its manual page.

Of course, before doing that you might want to investigate why your gdm user account has a per-user journal at all, and what user ID 995 was doing on April Fools’ Day 2015. ☺

You might also want to see what results from journalctl -n 20 _BOOT_ID=53baf678f0d749d6b390afea4a3ef96b and journalctl --reverse -n 20 _BOOT_ID=53baf678f0d749d6b390afea4a3ef96b and their equivalents for the other boot IDs.

Answered By: JdeBP