Merge branch 'develop'

Before, the hook would allow pushing binary files to newly created empty
repos. Now we handle and test this in our test suite.

git-hooks/README.md

@@ -1,41 +0,0 @@
-# Git Hooks
-
-## Reject Merge Commits
-
-The `blender_merged_hook` script rejects merge commits to branches were we don't want them. It is installed as an `pre-receive` hook in the following repositories:
-
-- `blender/blender`
-- `blender/blender-addons`
-- `blender/blender-addons-contrib`
-- `blender/blender-assets`
-- `blender/blender-benchmarks`
-- `blender/blender-developer-docs`
-- `blender/blender-manual`
-- `blender/blender-test-data`
-- `blender/lib-linux_x64`
-- `blender/lib-macos_arm64`
-- `blender/lib-macos_x64`
-- `blender/lib-source`
-- `blender/lib-windows_arm64`
-- `blender/lib-windows_x64`
-- `brecht/test-hooks`
-
-## Deny Binary Files
-
-The `deny-binary` script rejects commit that add binary files that should have been tracked as Git LFS object instead. We also provide some tests, which can be run using Docker Compose inside our Gitea container image. It has been installed as a `pre-receive` hook in the following repositories:
-
-- `blender/blender`
-- `blender/blender-assets`
-- `blender/blender-benchmarks`
-- `blender/blender-developer-docs`
-- `blender/blender-manual`
-- `blender/blender-test-data`
-- `blender/lib-linux_x64`
-- `blender/lib-macos_arm64`
-- `blender/lib-macos_x64`
-- `blender/lib-source`
-- `blender/lib-windows_arm64`
-- `blender/lib-windows_x64`
-- `studio/dogwalk`
-
-To circumvent the rejection you can add `override restrictions` anywhere in the commit message.

hooks/README.md

@@ -0,0 +1,80 @@
+# Git Hooks
+
+## Tests
+
+For our `deny_binary` Git hook, we have several test cases. They can be run locally:
+
+```bash
+$ ./test/run_deny_binary_tests.sh 
+== Running tests in directory deny_binary ==
+Running 10_initial_text_file_test.sh... ok
+Running 11_add_text_file_test.sh... ok
+Running 20_initial_binary_file_test.sh... ok
+Running 21_add_binary_file_test.sh... ok
+Running 22_add_suspicious_binary_file_test.sh... ok
+Running 30_add_binary_lfs_file_test.sh... ok
+Running 40_add_binary_file_and_convert_test.sh... ok
+Running 50_migrate_legacy_file_to_lfs_test.sh... ok
+Running 51_move_legacy_file_with_binary.sh... ok
+All tests passed.
+```
+
+Or they can be run using the Gitea image to do end-to-end testing (see `compose.yml` for changing the image).
+
+```bash
+$ docker compose up
+...
+Attaching to gitea_deny_binary_tests
+gitea_deny_binary_tests  | == Running tests in directory deny_binary ==
+gitea_deny_binary_tests  | Running 10_initial_text_file_test.sh... ok
+gitea_deny_binary_tests  | Running 11_add_text_file_test.sh... ok
+gitea_deny_binary_tests  | Running 20_initial_binary_file_test.sh... ok
+gitea_deny_binary_tests  | Running 21_add_binary_file_test.sh... ok
+gitea_deny_binary_tests  | Running 22_add_suspicious_binary_file_test.sh... ok
+gitea_deny_binary_tests  | Running 30_add_binary_lfs_file_test.sh... ok
+gitea_deny_binary_tests  | Running 40_add_binary_file_and_convert_test.sh... ok
+gitea_deny_binary_tests  | Running 50_migrate_legacy_file_to_lfs_test.sh... ok
+gitea_deny_binary_tests  | Running 51_move_legacy_file_with_binary.sh... ok
+gitea_deny_binary_tests  | All tests passed.
+gitea_deny_binary_tests exited with code 0
+```
+
+## Reject Merge Commits
+
+The `blender_merged_hook` script rejects merge commits to branches were we don't want them. It is installed as an `pre-receive` hook in the following repositories:
+
+- `blender/blender`
+- `blender/blender-addons`
+- `blender/blender-addons-contrib`
+- `blender/blender-assets`
+- `blender/blender-benchmarks`
+- `blender/blender-developer-docs`
+- `blender/blender-manual`
+- `blender/blender-test-data`
+- `blender/lib-linux_x64`
+- `blender/lib-macos_arm64`
+- `blender/lib-macos_x64`
+- `blender/lib-source`
+- `blender/lib-windows_arm64`
+- `blender/lib-windows_x64`
+- `brecht/test-hooks`
+
+## Deny Binary Files
+
+The `deny_binary` script rejects commit that add binary files that should have been tracked as Git LFS object instead. We also provide some tests, which can be run using Docker Compose inside our Gitea container image. It has been installed as a `pre-receive` hook in the following repositories:
+
+- `blender/blender`
+- `blender/blender-assets`
+- `blender/blender-benchmarks`
+- `blender/blender-developer-docs`
+- `blender/blender-manual`
+- `blender/blender-test-data`
+- `blender/lib-linux_x64`
+- `blender/lib-macos_arm64`
+- `blender/lib-macos_x64`
+- `blender/lib-source`
+- `blender/lib-windows_arm64`
+- `blender/lib-windows_x64`
+- `studio/dogwalk`
+
+To circumvent the rejection you can add `override restrictions` anywhere in the commit message.

hooks/compose.yml

@@ -1,6 +1,6 @@
 services:
   gitea_deny_binary_tests:
-    image: ghcr.io/blender/gitea:v1.25.2
+    image: ghcr.io/blender/gitea:v1.25.3
     container_name: gitea_deny_binary_tests
     working_dir: /workspace
     command: bash -c "./test/run_deny_binary_tests.sh"

hooks/deny_binary

@@ -28,9 +28,17 @@ while read oldrev newrev refname; do
     continue
   fi
 
-  # Set oldrev to HEAD if this is branch creation.
   if [ "$oldrev" = "$nullsha" ]; then
-    oldrev="HEAD"
+    if [ $(git rev-parse HEAD) = "HEAD" ]; then
+      # Iterate over all commits if there is no HEAD.
+      # This happens when the repo has just been initalized
+      rev_range=${newrev}
+    else
+      # Don't iterate over any commits already in the default branch of the repo
+      rev_range=HEAD..${newrev}
+    fi
+  else
+    rev_range=${oldrev}..${newrev}
   fi
 
   # Check for branches and tags, but not pull requests in refs/pull. Otherwise Gitea
@@ -44,12 +52,14 @@ while read oldrev newrev refname; do
   fi
 
   # Loop over each commit.
-  for commit in $(git rev-list --objects ${oldrev}..${newrev} |
+  for commit in $(git rev-list --objects ${rev_range} --filter=object:type=commit); do
-    git cat-file --batch-check='%(objectname) %(objecttype) %(objectsize) %(rest)' | grep commit | awk '{print $1}'); do
 
     # Get list of potentially binary files in this commit
-    mapfile -t binary_files < <(git log --diff-filter=d --pretty=format:%H -M100% --numstat ${commit}^! | grep -e "-	-	\w" | awk '{print $3}')
+    mapfile -t binary_files < <(
-    
+      git log --diff-filter=d --pretty=format:%H -M100% --numstat ${commit}^! |
+      awk -F'\t' '$1=="-" && $2=="-" {print $3}'
+    )
+
     if [ ${#binary_files[@]} -eq 0 ]; then
       continue
     fi
@@ -62,6 +72,12 @@ while read oldrev newrev refname; do
 
     rejected_files=()
     for file in "${binary_files[@]}"; do
+      if [[ $file == *" => "* ]]; then
+        # If the name contains " => " then this probably means the file was moved.
+        # If it was just moved, then it must still be a binary file.
+        rejected_files+=("$file")
+        continue
+      fi
       # Check if it's an LFS pointer
       # Use handle_pipefails as we may run into SIGPIPE error code
       if git show "${commit}:${file}" | git lfs pointer --check --stdin || handle_pipefails $?; then

hooks/test/deny_binary/10_initial_text_file_test.sh

@@ -17,7 +17,7 @@ setup_temp_git_identity
 ORIGIN_REPO_DIR=`setup_bare_origin_repository`
 WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 echo "Hello, World!" > "${WORK_GIT_DIR}/README.txt"
 git -C "${WORK_GIT_DIR}" add README.txt

hooks/test/deny_binary/11_add_text_file_test.sh

@@ -17,7 +17,7 @@ setup_temp_git_identity
 ORIGIN_REPO_DIR=`setup_bare_origin_repository`
 WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 echo "Hello, World!" > "${WORK_GIT_DIR}/README.txt"
 git -C "${WORK_GIT_DIR}" add README.txt

hooks/test/deny_binary/20_initial_binary_file_test.sh

@@ -2,9 +2,7 @@
 
 # Test: Add binary file to an empty repository and try to push.
 #
-# It is expected to pass as part of a logic which allows to branch off
+# It should fail as we should block binary files from be pushed.
-# existing branches where it is possible to have binary files that were
-# added prior to migration to Git LFS.
 
 set -e
 
@@ -18,12 +16,12 @@ setup_temp_git_identity
 ORIGIN_REPO_DIR=`setup_bare_origin_repository`
 WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 dd if=/dev/zero of="${WORK_GIT_DIR}/data.bin" bs=1 count=32k
 git -C "${WORK_GIT_DIR}" add data.bin
 git -C "${WORK_GIT_DIR}" commit --message "Initial commit"
-if ! git -C "${WORK_GIT_DIR}" push; then
+if git -C "${WORK_GIT_DIR}" push; then
   exit 1
 fi
 

hooks/test/deny_binary/21_add_binary_file_test.sh

@@ -17,7 +17,7 @@ setup_temp_git_identity
 ORIGIN_REPO_DIR=`setup_bare_origin_repository`
 WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 echo "Hello, World!" > "${WORK_GIT_DIR}/README.txt"
 git -C "${WORK_GIT_DIR}" add README.txt

hooks/test/deny_binary/22_add_suspicious_binary_file_test.sh

@@ -16,7 +16,7 @@ setup_temp_git_identity
 ORIGIN_REPO_DIR=`setup_bare_origin_repository`
 WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 echo "Hello, World!" > "${WORK_GIT_DIR}/README.txt"
 git -C "${WORK_GIT_DIR}" add README.txt

hooks/test/deny_binary/23_force_add_binary_file_test.sh

@@ -0,0 +1,29 @@
+#!/bin/sh
+
+# Test: Add binary file to an empty repository and push it by bypassing the hook.
+#
+
+set -e
+
+SCRIPT=$(readlink -f "$0")
+SCRIPT_PATH=$(dirname "$SCRIPT")
+
+. ${SCRIPT_PATH}/functions
+
+setup_temp_git_identity
+
+ORIGIN_REPO_DIR=`setup_bare_origin_repository`
+WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
+
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
+
+dd if=/dev/zero of="${WORK_GIT_DIR}/data.bin" bs=1 count=32k
+git -C "${WORK_GIT_DIR}" add data.bin
+git -C "${WORK_GIT_DIR}" commit --message "Initial commit, override restrictions"
+if ! git -C "${WORK_GIT_DIR}" push; then
+  exit 1
+fi
+
+echo
+echo "Test passed!"
+exit 0

hooks/test/deny_binary/30_add_binary_lfs_file_test.sh

@@ -17,7 +17,7 @@ setup_temp_git_identity
 ORIGIN_REPO_DIR=`setup_bare_origin_repository`
 WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 echo "Hello, World!" > "${WORK_GIT_DIR}/README.txt"
 git -C "${WORK_GIT_DIR}" add README.txt

hooks/test/deny_binary/40_add_binary_file_and_convert_test.sh

@@ -18,7 +18,7 @@ setup_temp_git_identity
 ORIGIN_REPO_DIR=`setup_bare_origin_repository`
 WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 echo "Hello, World!" > "${WORK_GIT_DIR}/README.txt"
 git -C "${WORK_GIT_DIR}" add README.txt
@@ -36,8 +36,6 @@ git -C "${WORK_GIT_DIR}" lfs track "*.bin"
 git -C "${WORK_GIT_DIR}" add .gitattributes
 git -C "${WORK_GIT_DIR}" commit --message "Track .bin files with Git LFS"
 
-git -C "${WORK_GIT_DIR}" lfs migrate import --no-rewrite --yes data.bin
-
 if git -C "${WORK_GIT_DIR}" push; then
   exit 1
 fi

hooks/test/deny_binary/50_migrate_legacy_file_to_lfs_test.sh

@@ -33,7 +33,7 @@ if ! git -C "${WORK_GIT_DIR}" push; then
   exit 1
 fi
 
-install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny-binary" "pre-receive"
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
 
 git -C "${WORK_GIT_DIR}" lfs install
 git -C "${WORK_GIT_DIR}" lfs track "*.bin"

hooks/test/deny_binary/51_move_legacy_file_with_binary.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+
+# Test: Show rejected files correctly when moving binary files
+#
+# This test is expected to fail since the binary file is supposed 
+# to be covered by Git LFS. This test verifies the reject message
+# for correctly parsing and displaying the file move.
+
+set -e
+
+SCRIPT=$(readlink -f "$0")
+SCRIPT_PATH=$(dirname "$SCRIPT")
+
+. ${SCRIPT_PATH}/functions
+
+setup_temp_git_identity
+
+ORIGIN_REPO_DIR=`setup_bare_origin_repository`
+WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
+
+dd if=/dev/zero of="${WORK_GIT_DIR}/data.bin" bs=1 count=32k
+git -C "${WORK_GIT_DIR}" add data.bin
+git -C "${WORK_GIT_DIR}" commit --message "Initial commit"
+if ! git -C "${WORK_GIT_DIR}" push; then
+  exit 1
+fi
+
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
+
+mkdir -p "${WORK_GIT_DIR}/bindata"
+git -C "${WORK_GIT_DIR}" mv data.bin bindata/data.bin
+git -C "${WORK_GIT_DIR}" commit --message "Move binary file"
+
+set +e
+PUSH_OUTPUT=$(git -C "${WORK_GIT_DIR}" push 2>&1)
+PUSH_STATUS=$?
+set -e
+
+if [ "${PUSH_STATUS}" -eq 0 ] || echo "${PUSH_OUTPUT}" | grep -q "fatal:"; then
+  exit 1
+fi
+
+echo
+echo "Test passed!"
+exit 0

hooks/test/deny_binary/52_branch_legacy_file_iteration_test.sh

@@ -0,0 +1,50 @@
+#!/bin/sh
+
+# Test: Ensure that we don't iterate over legacy files when branching
+#
+# Ensure that we don't iterate over legacy commits when pushing new branches
+
+set -e
+
+SCRIPT=$(readlink -f "$0")
+SCRIPT_PATH=$(dirname "$SCRIPT")
+
+. ${SCRIPT_PATH}/functions
+
+setup_temp_git_identity
+
+ORIGIN_REPO_DIR=`setup_bare_origin_repository`
+WORK_GIT_DIR=`clone_repository "${ORIGIN_REPO_DIR}"`
+
+dd if=/dev/zero of="${WORK_GIT_DIR}/data.bin" bs=1 count=32k
+git -C "${WORK_GIT_DIR}" add data.bin
+git -C "${WORK_GIT_DIR}" commit --message "Initial commit"
+if ! git -C "${WORK_GIT_DIR}" push; then
+  exit 1
+fi
+
+install_hook "${ORIGIN_REPO_DIR}" "${SCRIPT_PATH}/../../deny_binary" "pre-receive"
+
+git -C "${WORK_GIT_DIR}" lfs install
+git -C "${WORK_GIT_DIR}" lfs track "*.bin"
+git -C "${WORK_GIT_DIR}" add .gitattributes
+touch "${WORK_GIT_DIR}/data.bin"
+git -C "${WORK_GIT_DIR}" commit --message "Track .bin files with Git LFS"
+
+if ! git -C "${WORK_GIT_DIR}" push; then
+  exit 1
+fi
+
+git -C "${WORK_GIT_DIR}"  branch new_branch
+git -C "${WORK_GIT_DIR}"  switch new_branch
+touch "${WORK_GIT_DIR}/text_file"
+git -C "${WORK_GIT_DIR}" add text_file
+git -C "${WORK_GIT_DIR}" commit --message "Add regular text file"
+
+if ! git -C "${WORK_GIT_DIR}" push --set-upstream origin new_branch; then
+  exit 1
+fi
+
+echo
+echo "Test passed!"
+exit 0

hooks/test/run_deny_binary_tests.sh

@@ -8,3 +8,4 @@ SCRIPT_PATH=$(dirname "$SCRIPT")
 . "${SCRIPT_PATH}/common/functions"
 
 run_tests_in_directory "${SCRIPT_PATH}/deny_binary"
+rm -rf "${SCRIPT_PATH}/deny_binary/tmp"