Skip to content

Run the upstream test suite in CI (fixes #8) #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sylvestre merged 19 commits into from
Feb 17, 2024
Merged

Run the upstream test suite in CI (fixes #8) #13

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
15ba584
Naive script that runs relevant upstream tests against our diffutils …
oSoMoN Feb 9, 2024
37ae1c8
Also report skipped tests
oSoMoN Feb 10, 2024
95b8e5d
Pass the build profile as parameter
oSoMoN Feb 10, 2024
ddb3169
Run the upstream test suite in CI
oSoMoN Feb 10, 2024
2739956
Set $TERM to please tput
oSoMoN Feb 10, 2024
6fee076
More explicit comments
oSoMoN Feb 10, 2024
143359d
Do not fail when run in CI, the report is merely informative (at leas…
oSoMoN Feb 11, 2024
149c044
Allow running a subset of tests by setting the TESTS environment vari…
oSoMoN Feb 12, 2024
76dce3d
Export detailed test results to a JSON file and expose it as an artif…
oSoMoN Feb 12, 2024
faa810f
Log stdout and stderr for each test run
oSoMoN Feb 13, 2024
55bc85d
Add a comment to justify not including the contents of subdirectories
oSoMoN Feb 13, 2024
fed876b
Add relevant metadata to the JSON output
oSoMoN Feb 13, 2024
553ac75
Include a URL to the test script on the upstream cgit webUI
oSoMoN Feb 15, 2024
a526208
Add a script to print the test results
oSoMoN Feb 15, 2024
3a83bbf
Move helper scripts for running the upstream test suite in a tests/ s…
oSoMoN Feb 15, 2024
63388ad
Print a summary after running all tests
oSoMoN Feb 15, 2024
5ccb762
Simplify codeblock function
oSoMoN Feb 16, 2024
90d9826
Document the test scripts, and cosmetic changes
oSoMoN Feb 16, 2024
3f86d9c
Rename the task
sylvestre Feb 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Include a URL to the test script on the upstream cgit webUI
  • Loading branch information
@oSoMoN
oSoMoN committed Feb 15, 2024
commit 553ac752ecb667e82a242e29fe8ce45d2b69c377
8 changes: 6 additions & 2 deletions run-upstream-testsuite.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ tempdir=$(mktemp -d)
cd "$tempdir"

# Check out the upstream test suite
testsuite="https://git.savannah.gnu.org/git/diffutils.git"
gitserver="https://git.savannah.gnu.org"
testsuite="$gitserver/git/diffutils.git"
echo "Fetching upstream test suite from $testsuite"
git clone -n --depth=1 --filter=tree:0 "$testsuite" &> /dev/null
cd diffutils
Expand Down Expand Up @@ -48,14 +49,17 @@ export LC_ALL=C
export KEEP=yes
exitcode=0
timestamp=$(date -Iseconds)
urlroot="$gitserver/cgit/diffutils.git/tree/tests/"
for test in $tests
do
result="FAIL"
url="$urlroot$test?id=$upstreamrev"
# Run only the tests that invoke `diff`, because other binaries aren't implemented yet
if ! grep -E -s -q "(cmp|diff3|sdiff)" "$test"
then
sh "$test" 1> stdout.txt 2> stderr.txt && result="PASS" || exitcode=1
json+="{\"test\":\"$test\",\"result\":\"$result\","
json+="\"url\":\"$url\","
json+="\"stdout\":\"$(base64 -w0 < stdout.txt)\","
json+="\"stderr\":\"$(base64 -w0 < stderr.txt)\","
json+="\"files\":{"
Expand All @@ -70,7 +74,7 @@ do
cd - > /dev/null
else
result="SKIP"
json+="{\"test\":\"$test\",\"result\":\"$result\"},"
json+="{\"test\":\"$test\",\"url\":\"$url\",\"result\":\"$result\"},"
fi
color=2 # green
[[ "$result" = "FAIL" ]] && color=1 # red
Expand Down