Commit Graph

35 Commits

Author SHA1 Message Date
Colin Hebert
b788358600
Support Hardlinks (Any number) - Update (#70)
* Support hardlink groups, add debug

-Adds debug functionality with extended details
-Supports detecting inode groups for hardlink processing.
-Pulls files and sorts by, then groups by inode group with awk
-Checks all files in an inode group's counts when calculating skipping counts
-Removes existing skip hardlink flag
-Removes hardlinks and recreates them directly after the balance copy/delete/move operation per inode group to minimize 'downtime'

* Update README.md to denote hardlink support

Adds details around the debug flag, hardlink support, removed --skip-hardlinks functionality, and temporary files used during the script processing.

* Add additional hardlink group database notes

* typo

* Flip default debug

* Fix space handling in paths

* Fix echo bug

Removed the 'recreating hardlinks' echo for inode groups of 1 file.

* Introduce echo_debug

* Reintroduce cp documentation

* Remove unnecessary -e flag

* Remove unused flag

* Fix broken test due to | in the filename

* Add ignore on temporary inode files

* Update tests to assert the hardlinks are still working

* Fix ShellCheck issues

* Fix pasting issue

* Make grep compatible with BSD/MacOS

* Print whole line in debug

* Fix stat for BSD

* Fix stat for BSD part 2

* Fix shellcheck entry

* Fix stats working with FreeBSD

* Use bash's double brackets for consistency sake

* Expand to a string

---------

Co-authored-by: undaunt <31376520+undaunt@users.noreply.github.com>
2025-04-15 21:12:22 +02:00
Adi
97d11fdb0b
1.5x speed increase by using cmp instead of crc32 (#61)
* 2x speed increase by using cmp instead of crc32

* Remove redundant cmp check
2025-01-18 11:48:12 +01:00
Adi
9b233fdeaf
Replace MD5 with CRC32 for 2x checksum speed (#57)
* Replace MD5 with CRC32 for 2x checksum speed

* Use % builtin instead of awk for better performance
2025-01-09 17:00:01 +01:00
meo
393cbe2a12
FreeBSD fix and test (#55)
* add: test on freebsd

* test fix vm

* fix: transform to lowercase

transform the `OSTYPE` to a lowercase and store it inside `OSName`

* add: caching of the vm
2025-01-06 00:43:43 +01:00
meo
bd8d447150
Test mac os (#53)
* add: test on MacOS

Add test for MacOS in Github action

Maybe see this to add test for FreeBSD :
https://github.com/marketplace/actions/freebsd-vm

* add: log to test

adding log to the test for better understanding

* fix: shellcheck

remove the yellow color that is unused in the test

* fix: fail on lower-casting

the `"${variable,,}"` seam to not work on macOS so y remove it for the `skip_hardlink_flag` because it's always lowercase

* fix: macOS

remove all the lower-case casting

* fix(perf): going to `stat` remove `lsattr` macOS

The command `lsattr` is not supported on macOS so I have remove it
switch to `stat` instead of `ls` for better performance and no subshell
use of parameter expansion with  `%` in place of `awk` for performance and no subshell

* fix: forgot to remove variable
2024-12-17 00:33:35 +01:00
meo
922531829e
refactor: add regularity to spaces (#52)
1) switch all the miscellaneous indentation to 4 spaces

2) remove the space before the parenthesis of function

3) add indentation inside of if

4) remove miscellaneous spaces at the end of lines
2024-12-14 02:13:11 +01:00
Moss Hope
4a6fb83c3d
freebsd and macos fixes (#44)
freebsd and macos share stat and sed history with some differences from
gnu

tested on freebsd and linux, mac just verified with man pages and stack
overflow

Co-authored-by: mosshope <moss@lararium.houseofdusk.online>
2024-10-04 19:07:04 +02:00
Karol Kozlowski
9cea80a9c7
Remove dependence on perl (#41)
* Remove dependence on perl

* Fix shellcheck SC2004

* Fix shellcheck SC2004
2024-08-29 17:06:00 +02:00
Oliver Conzen
15672d2a02
Remove extraneous flags (#38) 2024-06-17 03:17:41 +02:00
Markus Ressel
979edb265c
add --reflink=never to cp on linux (#34) 2023-11-11 11:08:36 +01:00
Gregory Zdanowski-House
2276727d6f
Add rebalancing start date (#25) 2023-09-04 17:21:11 +02:00
Kenneth Ballard
b550a57615
Using perl to generate the percent text. "bc" isn't available on TrueNAS SCALE. (#24) 2023-08-21 18:10:01 +02:00
johnpyp
973855c049
--skip-hardlinks flag (#22)
* feat: --skip-hardlinks flag

Adds a --skip-hardlinks flag that will not process any files that have an active hardlink.

Without a pre-scan to identify hardlink targets and relink them after copies, all this would result in is duplication.
You'd then have to run another de-duplicator, which doesn't have knowledge of the balancing, and could arbitrarily undo
the balancing work.

* doc: .rebalance suffix -> .balance suffix

* doc: add --skip-hardlinks parameter to README

* test: add unit tests for --skip-hardlink
2023-06-16 16:36:43 +02:00
Markus Ressel
9faf865850
Fix help string 2023-01-27 00:44:42 +01:00
Markus Ressel
d62e6e0be7
added folder structure for testing (#7)
* added folder structure for testing
added testing.sh script
added testing github action

* delete rebalancing file for testing

* run multiple variants of the command

* fix "integer expression expected"

* added assertions

* output logged data
2022-03-29 04:02:13 +02:00
Markus Ressel
b5fcf1ad0c
-F, --fixed-strings PATTERNS are strings (#8)
-x, --line-regexp         match only whole lines
2022-03-29 03:41:31 +02:00
Markus Ressel
40ebafc233
ignore file if missing on disk (#5) 2020-07-31 16:09:45 +02:00
Markus Ressel
fbf83777fc ignore shellcheck warnings for ls 2020-07-31 15:57:12 +02:00
Markus Ressel
c070f4a178 fixed passes parameter, added shorthands 2020-07-30 20:05:27 +02:00
Markus Ressel
b258b6aa3b added ability to disable passes tracking 2020-07-30 17:45:43 +02:00
Markus Ressel
3d02d6c274 added -checksum and -passes parameters, added logic to skip checksum, added logic to limit max passes, print configuration on startup, added help message 2020-07-30 17:38:02 +02:00
Markus Ressel
67a75a52b3 added gitignore, shellcheck fix 2020-07-30 16:29:27 +02:00
Markus Ressel
ea7a5501b3 keep track of rebalances in a 'database' file 2020-07-30 16:27:57 +02:00
Markus Ressel
af0e156c9f use case sensitive comparison for checksum 2020-07-30 15:51:57 +02:00
Markus Ressel
39fe636b8e remove unnecessary md5 for easier debugging 2020-07-30 15:51:03 +02:00
Markus Ressel
c85713ab47 added checksum test to ensure file attributes, permissions and content match 2020-07-30 15:30:11 +02:00
Markus Ressel
704e7d4390 added comments 2020-07-28 17:31:10 +02:00
Markus Ressel
d9744e191a print Done! in green 2020-07-28 17:28:32 +02:00
Markus Ressel
418b965fc1 print progress in green 2020-07-28 17:26:40 +02:00
Markus Ressel
7004f91611 added progress output 2020-07-28 17:10:35 +02:00
Markus Ressel
dfdae73d76 case insensitive comparison 2020-07-28 16:52:04 +02:00
Markus Ressel
fecf86b1cc case insensitive comparison 2020-07-28 16:51:53 +02:00
Markus Ressel
6ebd749f5e added os type distinction 2020-07-28 16:32:39 +02:00
Markus Ressel
1b0c6795f9 added empty lines before 'Done!' message 2020-07-28 16:23:07 +02:00
Markus Ressel
1da3101ca9 renamed script file 2020-07-28 04:27:35 +02:00