large-files

Delete huge directory that causes all commands to hang

Delete huge directory that causes all commands to hang How do I delete this large directory? stat session/ File: ‘session/’ Size: 321540096 Blocks: 628040 IO Block: 4096 directory Device: 903h/2307d Inode: 11149319 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-09-29 14:34:40.910894275 +0200 Modify: 2022-09-29 14:35:09.598400050 +0200 Change: 2022-09-29 14:35:09.598400050 …

Total answers: 3

How copytruncate actually works?

How copytruncate actually works? we would like to understand copytruncate before rotating the file using logrotate with below configuration: /app/syslog-ng/custom/output/all_devices.log { size 200M copytruncate dateext dateformat -%Y%m%d-%s rotate 365 sharedscripts compress postrotate /app/syslog-ng/sbin/syslog-ng-ctl reload endscript } RHEL 7.x, 8GB RAM, 4 VCpu Question: How does logrotate truncate the file, when syslog-NG already opened file for …

Total answers: 1

How to find duplicate lines in many large files?

How to find duplicate lines in many large files? I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free. My goal: I want to find all all duplicate lines across two or more files and also the names of the …

Total answers: 5

Replace string in a huge (70GB), one line, text file

Replace string in a huge (70GB), one line, text file I have a huge (70GB), one line, text file and I want to replace a string (token) in it. I want to replace the token <unk>, with another dummy token (glove issue). I tried sed: sed ‘s/<unk>/<raw_unk>/g’ < corpus.txt > corpus.txt.new but the output file …

Total answers: 14

Why are these files in an ext4 volume fragmented?

Why are these files in an ext4 volume fragmented? I have a 900GB ext4 partition on a (magnetic) hard drive that has no defects and no bad sectors. The partition is completely empty except for an empty lost+found directory. The partition was formatted using default parameters except that I set the number of reserved filesystem …

Total answers: 2

Transferring large (8 GB) files over ssh

Transferring large (8 GB) files over ssh I tried it with SCP, but it says “Negative file size”. >scp matlab.iso xxx@xxx:/matlab.iso matlab.iso: Negative file size Also tried using SFTP, worked fine until 2 GB of the file had transferred, then stopped: sftp> put matlab.iso Uploading matlab.iso to /home/x/matlab.iso matlab.iso -298% 2021MB -16651.-8KB/s 00:5d o_upload: offset …

Total answers: 3

Basic sed command on large one-line file: couldn't re-allocate memory

Basic sed command on large one-line file: couldn't re-allocate memory I have a 250 MB text file, all in one line. In this file I want to replace a characters with b characters: sed -e “s/a/b/g” < one-line-250-mb.txt It fails with: sed: couldn’t re-allocate memory It seems to me that this kind of task could …

Total answers: 3

Merge nonzero blocks of huge (sparse) file A into huge file B

Merge nonzero blocks of huge (sparse) file A into huge file B I have two partial disk images from a failing hard drive. File B contains the bulk of the disk’s contents, with gaps where sector reads failed. File A is the result of telling ddrescue to retry all the failed sectors, so it is …

Total answers: 1

Viewing large image on Linux

Viewing large image on Linux I need to view a large (50000×40000 px) png image on Linux. Unfortunately most tools (eog, convert etc.) either crashes or fail with note about too little memory. Is there a way to view this image (I would prefer to see both the resized image and details)? Asked By: Maciej …

Total answers: 1

cat line X to line Y on a huge file

cat line X to line Y on a huge file Say I have a huge text file (>2GB) and I just want to cat the lines X to Y (e.g. 57890000 to 57890010). From what I understand I can do this by piping head into tail or viceversa, i.e. head -A /path/to/file | tail -B …

Total answers: 8

largefile feature at creating file-system

largefile feature at creating file-system Is useful to use -T largefile flag at creating a file-system for a partition with big files like video, and audio in flac format? I tested the same partition with that flag and without it, and using tune2fs -l [partition], I checked in “Filesystem features” that both have “large_file” enabled. …

Total answers: 2

Grepping over a huge file performance

Grepping over a huge file performance I have FILE_A which has over 300K lines and FILE_B which has over 30M lines. I created a bash script that greps each line in FILE_A over in FILE_B and writes the result of the grep to a new file. This whole process is taking over 5+ hours. I’m …

Total answers: 4

How to remove duplicate lines inside a text file?

How to remove duplicate lines inside a text file? A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions while (preferably, but this can …

Total answers: 10

Is there a way to modify a file in-place?

Is there a way to modify a file in-place? I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don’t have enough disk space for another file), specifically I want to grep and ignore some patterns — is there a way to do this without using …

Total answers: 11

What linux editor can open a 200G file for editing within a minute or two?

What linux editor can open a 200G file for editing within a minute or two? I’m looking for an editor that will open the file in chunks (not try to read the whole file into memory) as I’m trying to hand-edit a 200G file. Asked By: Clint Priest || Source This may not be exactly …

Total answers: 1

Number of files per directory

Number of files per directory I have a directory with about 100000 small files (each file is from 1-3 lines, each file is a text file). In size the directory isn’t very big (< 2GB). This data lives in a professionally administered NFS server. The server runs Linux. I think the filesystem is ext3, but …

Total answers: 1

Diffing two big text files

Diffing two big text files I have two big files (6GB each). They are unsorted, with linefeeds (n) as separators. How can I diff them? It should take under 24h. Asked By: Jonas Lejon || Source The most obvious answer is just to use the diff command and it is probably a good idea to …

Total answers: 4