math problem or variable value problem im not sure

I am having an issue comparing epoch times. I’m really not sure what I’m doing wrong. I have been tasked with identifying and actioning files beyond and during certain date ranges.

In this case, it’s a cleanup script. It takes a few parameters to do the search and then checks the dates. My dates are like this:

earliest date we can delete a file -> date -d "-30days" +"%s"

earliest date we can compress a file -> date -d "-14days" +"%s"

get the file last modified time: stat -c "%y" "/path/to/file"

get the epoch from that last modified time: date -d "<from the obtained time from stat>" +"%s"

The math goes like this:

if (( FILE_TIMESTAMP >= EARLIEST_DELETE_DATE ))
then
    delete file;
elif (( FILE_TIMESTAMP >= EARLIEST_COMPRESS_DATE ))
then
    compress file;
else
    echo "file skipped as it wasnt in teh date ranges";
fi

(edit to the above code: I removed the check for <= EARLIEST_DELETE_DATE here)

The first if statement works as expected. The second works halfway – it does identify that the timestamp is greater than the compress date but the check for less than the delete date appears to be returning false as that section of code never gets run.

Am I doing the math wrong or do I have the variables wrong?

Also, I know my posts are crap. I’ll delete my account after this so as to lessen the strain on the community.

Some updates:

Looking at my debug log output I see this:

[Time: ][Thread: 883837][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:99] [Method: manageFileLifecycle.sh] - FILE_DATESTAMP -> 2023-05-07 03:20:29.000000000 -0400 [Time: ][Thread: 883837][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:100] [Method: manageFileLifecycle.sh] - FILE_TIMESTAMP -> 1683444029

The file shown is older than 6 months but was not triggered by the >= delete clause. not sure what I’m doing wrong

For @markp-fuso:

declare -- FILE_TIMESTAMP="1688178017"
declare -i EARLIEST_DELETE_DATE="1697037095"
declare -i EARLIEST_COMPRESS_DATE="1698419495"
FILE_TO_PROCESS -> /opt/cws/log/S93206319.log, FILE_DATESTAMP -> 2023-06-30 22:20:17.000000000 -0400 FILE_TIMESTAMP -> 1688178017, EARLIEST_DELETE_DATE -> 1697037095, EARLIEST_COMPRESS_DATE -> 1698419495

In the log output I see this file being skipped, even though it shouldn’t have been.

Also, I must apologize – I cannot share the whole script as it is company property =(

More information:

Specific file pulled that ISN’T working:

declare -- FILE_TIMESTAMP="1679107439"
declare -i EARLIEST_DELETE_DATE="1697040305"
declare -i EARLIEST_COMPRESS_DATE="1698422705"
FILE_TO_PROCESS -> /opt/cws/log/S93382366.log, FILE_DATESTAMP -> 2023-03-17 22:43:59.000000000 -0400 FILE_TIMESTAMP -> 1679107439, EARLIEST_DELETE_DATE -> 1697040305, EARLIEST_COMPRESS_DATE -> 1698422705
file skipped as it wasnt in teh date ranges

and

[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:93] [Method: manageFileLifecycle.sh] - processRequestedFile -> enter
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:94] [Method: manageFileLifecycle.sh] - Provided arguments: /opt/cws/log/S93382366.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:95] [Method: manageFileLifecycle.sh] - ERROR_COUNT -> 0
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:96] [Method: manageFileLifecycle.sh] - FILE_TO_PROCESS -> /opt/cws/log/S93382366.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:97] [Method: manageFileLifecycle.sh] - FILE_CORE_NAME -> S93382366
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:98] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT -> log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:99] [Method: manageFileLifecycle.sh] - FILE_DATESTAMP -> 2023-03-17 22:43:59.000000000 -0400
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:100] [Method: manageFileLifecycle.sh] - FILE_TIMESTAMP -> 1679107439
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:107] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT and FILE_CORE_NAME do not match - executing with a file extension
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:139] [Method: manageFileLifecycle.sh] - File has extension log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:298] [Method: manageFileLifecycle.sh] - RETURN_CODE -> 0
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:299] [Method: manageFileLifecycle.sh] - processRequestedFile -> exit

A file that appears to have at least hit the delete clause:

Output from the typeset:

declare -- FILE_TIMESTAMP="1703308529"
declare -i EARLIEST_DELETE_DATE="1697040305"
declare -i EARLIEST_COMPRESS_DATE="1698422705"
FILE_TO_PROCESS -> /opt/cws/log/S20504215.log, FILE_DATESTAMP -> 2023-12-23 00:15:29.000000000 -0500 FILE_TIMESTAMP -> 1703308529, EARLIEST_DELETE_DATE -> 1697040305, EARLIEST_COMPRESS_DATE -> 1698422705

Output from the debug log:

[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:93] [Method: manageFileLifecycle.sh] - processRequestedFile -> enter
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:94] [Method: manageFileLifecycle.sh] - Provided arguments: /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:95] [Method: manageFileLifecycle.sh] - ERROR_COUNT -> 0
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:96] [Method: manageFileLifecycle.sh] - FILE_TO_PROCESS -> /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:97] [Method: manageFileLifecycle.sh] - FILE_CORE_NAME -> S20504215
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:98] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT -> log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:99] [Method: manageFileLifecycle.sh] - FILE_DATESTAMP -> 2023-12-23 00:15:29.000000000 -0500
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:100] [Method: manageFileLifecycle.sh] - FILE_TIMESTAMP -> 1703308529
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:107] [Method: manageFileLifecycle.sh] - FILE_CORE_EXT and FILE_CORE_NAME do not match - executing with a file extension
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:139] [Method: manageFileLifecycle.sh] - File has extension log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:143] [Method: manageFileLifecycle.sh] - File timestamp is greater than or equal to the delete date. File -> /opt/cws/log/S20504215.log, Timestamp -> 1703308529, EARLIEST_DELETE_DATE -> 1697035331
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:148] [Method: manageFileLifecycle.sh] - Deleting file /opt/cws/log/S20504215.log as it falls after the earliest delete date range.
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: processRequestedFile:149] [Method: manageFileLifecycle.sh] - EXEC: deleteRequestedFile /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:462] [Method: manageFileLifecycle.sh] - deleteRequestedFile -> enter
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:463] [Method: manageFileLifecycle.sh] - Provided arguments: /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:464] [Method: manageFileLifecycle.sh] - REMOVABLE_FILE -> /opt/cws/log/S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:465] [Method: manageFileLifecycle.sh] - REMOVABLE_FILE_NAME -> S20504215.log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:466] [Method: manageFileLifecycle.sh] - REMOVABLE_FILE_PATH -> /opt/cws/log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:470] [Method: manageFileLifecycle.sh] - Checking if /opt/cws/log exists...
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:516] [Method: manageFileLifecycle.sh] - Returning to previous directory /opt/cws/log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:520] [Method: manageFileLifecycle.sh] - Previous directory: /opt/cws/log, New working directory: /opt/cws/log
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:530] [Method: manageFileLifecycle.sh] - RETURN_CODE -> 1
[Time: ][Thread: 8066][Log: debug.log][Level: DEBUG] - [File: deleteRequestedFile:531] [Method: manageFileLifecycle.sh] - deleteRequestedFile -> exit

The goal is to delete any file older than 30 days from the current date, compress anything older than 14 days from the current date but not older than 30 days from the current date (these entries would fall under the delete) and anything younger than 14 days is left alone.

Asked By: Kevin Huntly

||
Source

You have your logic reversed. You’re treating those timestamps as if they were ages. Should be:

if (( FILE_TIMESTAMP < EARLIEST_DELETE_DATE ))
then
    delete file
elif (( FILE_TIMESTAMP < EARLIEST_COMPRESS_DATE ))
then
    compress file
else
    echo "file skipped as it wasn't in the date ranges"
fi

You could also do it portably with find:

find -- "$file" -prune -mtime +29 -exec rm -f -- {} + -o 
  -mtime +13 -exec gzip -- {} + -o 
  -exec echo Skipping {} +

Note that despite the --, that fails for values of $file that start with - (or are (, ), !). gzip also has a problem with the file called - (which it treats as meaning standard input instead; same for GNU stat).

-mtime +29 is true for files whose age rounded down to an integer number of days is strictly greater than 29, so that’s files that are 30 days old or older.

With the find of BSDs, both issues/confusions can be addressed by doing:

find -f "$file" -prune -mtime +30d -exec rm -f -- {} + -o 
  -mtime +14d -exec gzip -- {} + -o 
  -exec echo Skipping {} +

(note the d suffix).

While with recent versions of GNU find, one can do:

find  -files0-from <(printf '%s' "$file") -prune 
  ! -newermt '30 days ago' -exec rm -f -- {} + -o 
  ! -newermt '14 days ago' -exec gzip -- {} + -o 
  -printf 'Skipping %pn'

Do not user -delete in place of -exec rm -f {} + as -delete implies -depth and do not use -exec ... ';' in place of -exec ... + in the first 2 -execs in those commands as then the success/failure of the command would be used in the find logic.

All those commands can take more than one file to process (with BSD find, use -f "$file1" -f "$file2"...).

Answered By: Stéphane Chazelas

At the end of the question we have the following rules:

  • delete any file older than 30 days from the current date
  • compress any file older than 14 days from the current date but not older than 30 days from the current date (these entries would fall under the delete)
  • any file younger than 14 days is left alone

From the latest update we’re given two sets of epoch values:

filename: /opt/cws/log/S93382366.log

        FILE_TIMESTAMP="1679107439"      # Fri Mar 17 21:43:59 CDT 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

    code result: ISN'T working (assuming this means code generates "file skipped as it wasnt in the date ranges")

expected result: delete file (more than 30 days old; NOTE: compression should have occurred a long time ago)

-----------

filename: /opt/cws/log/S20504215.log

        FILE_TIMESTAMP="1703308529"      # Fri Dec 22 23:15:29 CST 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

    code result: delete file

expected result: file skipped as it wasn't in the date ranges (less than 14 days old; actually, file has timestamp that is 42 days in the future when today = Nov 10 2023)

Primary issue with the code:

  • an ‘older’ epoch date is actually going to be less than a ‘newer’ epoch date so …
  • the code needs to be testing for epochs that are less than a desired threshold

Correcting the current code to implement the 3 desired rules:

if (( FILE_TIMESTAMP < EARLIEST_DELETE_DATE ))
then
    echo "delete file"
elif (( FILE_TIMESTAMP < EARLIEST_COMPRESS_DATE ))
then
    echo "compress file"
else
    echo "file skipped as it wasn't in the date ranges";
fi

If we apply this to our 2 sample files/epoch-tuples we get:

filename: /opt/cws/log/S93382366.log

        FILE_TIMESTAMP="1679107439"      # Fri Mar 17 21:43:59 CDT 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

code result: delete file

-----------

filename: /opt/cws/log/S20504215.log

        FILE_TIMESTAMP="1703308529"      # Fri Dec 22 23:15:29 CST 2023
  EARLIEST_DELETE_DATE="1697040305"      # Wed Oct 11 11:05:05 CDT 2023
EARLIEST_COMPRESS_DATE="1698422705"      # Fri Oct 27 11:05:05 CDT 2023

code result: file skipped as it wasn't in the date ranges
Answered By: markp-fuso
Categories: Answers Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.