Skip to content

Run the upstream test suite in CI (fixes #8) #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sylvestre merged 19 commits into from
Feb 17, 2024
Merged

Run the upstream test suite in CI (fixes #8) #13

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
15ba584
Naive script that runs relevant upstream tests against our diffutils …
oSoMoN Feb 9, 2024
37ae1c8
Also report skipped tests
oSoMoN Feb 10, 2024
95b8e5d
Pass the build profile as parameter
oSoMoN Feb 10, 2024
ddb3169
Run the upstream test suite in CI
oSoMoN Feb 10, 2024
2739956
Set $TERM to please tput
oSoMoN Feb 10, 2024
6fee076
More explicit comments
oSoMoN Feb 10, 2024
143359d
Do not fail when run in CI, the report is merely informative (at leas…
oSoMoN Feb 11, 2024
149c044
Allow running a subset of tests by setting the TESTS environment vari…
oSoMoN Feb 12, 2024
76dce3d
Export detailed test results to a JSON file and expose it as an artif…
oSoMoN Feb 12, 2024
faa810f
Log stdout and stderr for each test run
oSoMoN Feb 13, 2024
55bc85d
Add a comment to justify not including the contents of subdirectories
oSoMoN Feb 13, 2024
fed876b
Add relevant metadata to the JSON output
oSoMoN Feb 13, 2024
553ac75
Include a URL to the test script on the upstream cgit webUI
oSoMoN Feb 15, 2024
a526208
Add a script to print the test results
oSoMoN Feb 15, 2024
3a83bbf
Move helper scripts for running the upstream test suite in a tests/ s…
oSoMoN Feb 15, 2024
63388ad
Print a summary after running all tests
oSoMoN Feb 15, 2024
5ccb762
Simplify codeblock function
oSoMoN Feb 16, 2024
90d9826
Document the test scripts, and cosmetic changes
oSoMoN Feb 16, 2024
3f86d9c
Rename the task
sylvestre Feb 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Print a summary after running all tests
  • Loading branch information
@oSoMoN
oSoMoN committed Feb 15, 2024
commit 63388ad78add65f8e7d6c285dbd08dc86681e2bc
15 changes: 13 additions & 2 deletions tests/run-upstream-testsuite.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,17 @@ else
echo -e '\n\nprinttests:\n\t@echo "${TESTS}"' >> Makefile.am
tests=$(make -f Makefile.am printtests)
fi
echo "Running $(echo "$tests" | wc -w) tests"
total=$(echo "$tests" | wc -w)
echo "Running $total tests"
export LC_ALL=C
export KEEP=yes
exitcode=0
timestamp=$(date -Iseconds)
urlroot="$gitserver/cgit/diffutils.git/tree/tests/"
passed=0
failed=0
skipped=0
normal="$(tput sgr0)"
for test in $tests
do
result="FAIL"
Expand All @@ -72,17 +77,23 @@ do
done
json="${json%,}}},"
cd - > /dev/null
[[ "$result" = "PASS" ]] && (( passed++ ))
[[ "$result" = "FAIL" ]] && (( failed++ ))
else
result="SKIP"
(( skipped++ ))
json+="{\"test\":\"$test\",\"url\":\"$url\",\"result\":\"$result\"},"
fi
color=2 # green
[[ "$result" = "FAIL" ]] && color=1 # red
[[ "$result" = "SKIP" ]] && color=3 # yellow
printf " %-40s $(tput setaf $color)$result$(tput sgr0)\n" "$test"
done
json="\"tests\":[${json%,}]"
echo ""
echo "Summary: TOTAL: $total / $(tput setaf 2)PASS$normal: $passed / $(tput setaf 1)FAIL$normal: $failed / $(tput setaf 3)SKIP$normal: $skipped"
echo ""

json="\"tests\":[${json%,}]"
metadata="\"timestamp\":\"$timestamp\","
metadata+="\"revision\":\"$rev\","
metadata+="\"upstream-revision\":\"$upstreamrev\","
Expand Down