Archieve Small Files

To decrease number of small file, keep the folder structure, tar all files less than 1G

$ git clone https://github.com/ld32/archiveSmallFiles.git

$ export PATH=$PWD/archiveSmallFiles/bin:$PATH

If you don't have sudo permission, please using the guild:

https://github.com/ld32/archiveSmallFiles/blob/master/README_noSudo.md

Practice with testing data

# Start an interactive job, create a working directory and go to it: 
$ srun -p short -t 2:0:0 --mem 2G --pty /bin/bash 
$ cd $HOME
$ mkdir -p tarTesting 
$ cd tarTesting

# Prepare a testing data: 
$ createTestData.sh
...
Test data generation complete. It is in /home/xyz/tarTesting/TestingData. 
One folder and one file are set to not readable, so that you can test the scripts.

# To scan folders (with 1 process):
$ sudoScanFolders.sh TestingData/ 1 65 2>&1 | tee scan.log
...
Folder count matches expected value: 65. Total folders found: 65.
Scan results (should match with the numbers in starfish, try this useful tool: convertStarfishPath.sh):
Total number of folders:
65
And total number of files:
365

# To archive using tar:
archiveFolders.sh tar local pass1

# Some folders will not work due to permission or some other reason. 
# Let's find the folders not done yet:
$ findFoldersNotDoneArchiving.sh pass1
...
Actual folders: 65
Done folders: 63
Not done folders: 2
Not done folders are saved to pass2/folders.txt.
Please review logs and see what is the issue: 
$ cat pass1/tarError* 

#If there is permission issues, please run (with 4 processes): 
$ sudoCorrectPermission.sh pass2 4

#Aftet that, you can run the next pass now:
$ archiveFolders.sh tar local pass2

#Check again: 
$ findFoldersNotDoneArchiving.sh pass2
...
Actual folders: 2
Done folders: 2

# To check archives (We use pass1 because pass1 has full list of folders):
$ checkArchives.sh tar local pass1

# Some folders might not work due to permission or some other reason. 
# Let's find the folders not done yet:
$ findFoldersNotDoneChecking.sh pass1
...
pass1
Archive checking results:
Actual folders: 65
Done folders: 63
Total number of original files:
365
Total number of files if we  untar all the data (should be the same as untarred file count):
348
Total number of files after tarring (should be the same the number of files in tarred folder in starfish):
57
Not done folders: 2
Not done folders are saved to pass3/folders.txt.

#Please review logs and see what is the issue:
$ cat pass1/tarError*

#If there is permission issues, please run:
$ sudoCorrectPermission.sh pass3 4

#Aftet that, you can archive them: 
$ archiveFolders.sh tar local/sbatch pass3

#Aftet that, you can run the next pass now:
$ checkArchives.sh tar local pass3

Then:
$ findFoldersNotDoneChecking.sh pass3

# To randomly un-archieve 10 folder and compare with original using diff command:
randomUnArchiveToCheck.sh tar pass1 10

Un-Archive

# quickly untar from interactive commandline: 
$ find dataFolder -name "*.tar" -print0 | xargs -0 -P 4 -I {} sh -c 'tar --overwrite -xf "$1" -C "$(dirname "$1")"; rm $1 ${1/.tar/.md5sum}' _ {}

# this code snippet takes care of both cases: 
 $ find $sPath -maxdepth 1 -mindepth 1 \( -type f -o -type l \) ! -name "*.md5sum" -print0 | xargs -0 -I {} sh -c '
                if [[ "$1" == *.tar ]]; then
                    if tar -tf "$1" | grep -qxF "${1%.tar}.md5sum" || [ -f ${1%.tar}.md5sum ]; then
                            sudo tar --exclude ".md5sum" --overwrite -xf "$1" -C "$2"
                        else
                            sudo cp -a "$1" "$2/"
                        fi
                    fi    
            ' _ {} $dPath

# Using scripts
$ unArchives.sh tar local pass2

Working with real data

# Start an interactive job, create a working directory where you 
# want to keep the archived data and go to it: 
$ srun -p short -t 12:0:0 --mem 8G --pty /bin/bash 
$ cd /some/big/storage/
$ mkdir -p tarFprReal 
$ cd tarForReal

# Check Starfish website and find the actul folder count, 
# for example 5500000 folders, 
# then scan folders using 20 processes:
$ sudoScanFolders.sh /n/data1/xyz/.snapshot/daily.2026.2.1/someData 20 5500000 2>&1 | tee scan.log
...
Folder count matches expected value: 55000000. 
Total folders found: 55000000.
Parallel scan is done
Warning: Found 5500000 folders, which is more than 100000. 
Let me split the folder list into 6 parts:
pass1/folders_part_1
pass1/folders_part_2
pass1/folders_part_3
pass1/folders_part_4
pass1/folders_part_5
pass1/folders_part_6

# To archive using tar using Slurm jobs, each job run 10k folders:
archiveFolders.sh tar sbatch pass1

# Some folders might not work due to permission or some other reason. 
# Let's find the folders not done yet:
$ findFoldersNotDoneArchiving.sh pass1
...
Actual folders: 5500000
Done folders: 5499999
Not done folders: 1
Not done folders are saved to pass2/folders.txt.
Please review logs and see what is the issue: 
$ cat pass1/tarError* 

# If there is permission issues, please run: 
$ sudoCorrectPermission.sh pass2 4

# Aftet that, you can run the next pass now: pass2
$ archiveFolders.sh tar sbatch pass2

# To check archives:
$ checkArchives.sh tar local1 pass

# Some folders might not work due to permission or some other reason. 
# Let's find the folders not done yet:
$ findFoldersNotDoneChecking.sh pass1
...
Actual folders: 5500000
Done folders: 5499999
Not done folders: 1
Not done folders are saved to pass2/folders.txt.
Please review logs and see what is the issue: 
$ cat pass1/tarError* 

# If there is permission issues, please run: 
$ sudoCorrectPermission.sh pass2 4

# Aftet that, you can run the next pass now: pass2
$ checkArchives.sh tar sbatch pass2

# To randomly un-archieve 10 folder and compare with original:
$ randomUnarchiveToCheck.sh tar pass1 10

# When we help with lab data, the tarred data 
# need set proper ownership and permission.
# If folder list is not very large, 
# set ownership and permission using 4 processes: 
$ sudoSetOwnerPermission.sh pass1 4

# Or if folder list is huge, run 6 interactive 
# jobs and each job run 1 subset folders: 
$ srun -p short --mem 8G -t 5:0:0 --pty /bin/bash
$ sudoSetOwnerPermissionParts.sh pass1 4 1

# srun -p short --mem 8G -t 5:0:0 --pty /bin/bash
$ sudoSetOwnerPermissionParts.sh pass1 4 2

$ srun -p short --mem 8G -t 5:0:0 --pty /bin/bash
$ sudoSetOwnerPermissionParts.sh pass1 4 3

$ srun -p short --mem 8G -t 5:0:0 --pty /bin/bash
$ sudoSetOwnerPermissionParts.sh pass1 4 4

$ srun -p short --mem 8G -t 5:0:0 --pty /bin/bash
$ sudoSetOwnerPermissionParts.sh pass1 4 5

$ srun -p short --mem 8G -t 5:0:0 --pty /bin/bash
$ sudoSetOwnerPermissionParts.sh pass1 4 6

# Sometimes, an .snapshot may outdated. 
# To updata .snapshot version with the folder paths: 
$ updateSnapshotVersion.sh