Patch #42688
closed
Run system tests on GitHub CI
0%
Description
This patch enables Redmine's system tests to run on GitHub CI.
- Uses Chrome and ChromeDriver pre-installed on GitHub Actions' Ubuntu images.
- Enables
continue-on-errorfor a period, so that system test failures do not cause the entire test suite to fail. - Confirmed all tests pass with patches #42422 and #42687 applied.
Details¶
Use Chrome/ChromeDriver pre-installed on Ubuntu images¶
GitHub Actions' Ubuntu images come with Chrome and ChromeDriver pre-installed, and these seem to be generally updated to the latest stable versions.
I believe this is the best option unless there's a need to test against specific or multiple versions of Chrome.
I considered the following alternative methods, but still believe this approach is the best:
- Using
selenium/standalone-chromeimage as a service container: As mentioned in Mounting volumes to retrieve downloaded files , correctly enabling download functionality requires setting up a directory with appropriate permissions before the container starts. However, when using service containers, it's not possible to perform operations before the container starts. - Directly running
npx @puppeteer/browsers installto set up Chrome (Chrome for testing) and ChromeDriver: This requires additional setup, such as adding the installed Chrome and ChromeDriver paths to the system's PATH. - Installing with
apt install google-chrome-stable: This requires additional configuration forapt install, such as registering package repositories.
Enable continue-on-error¶
Once patches #42422 and #42687 are applied, system tests should be reasonably stable, but may still occasionally fail. Therefore, for a period, continue-on-error will be enabled so that system test failures do not cause the entire test suite to fail.
System test failures can be identified by errors outputted on the job summary page as shown below. Screenshots of failures can also be downloaded.
Total test time increases by about 2 minutes¶
Running system tests will increase the total test time by approximately 2 minutes. I plan to propose test parallelization to address this in the future.
Files
Related issues
Updated by Marius BĂLTEANU 29 days ago
- Is duplicate of Feature #42609: Run system tests on Github CI added
Updated by Marius BĂLTEANU 29 days ago
- Is duplicate of deleted (Feature #42609: Run system tests on Github CI)
Updated by Marius BĂLTEANU 29 days ago
- Has duplicate Feature #42609: Run system tests on Github CI added
Updated by Marius BĂLTEANU 29 days ago
- Tracker changed from Feature to Patch
- Status changed from New to Resolved
- Assignee set to Marius BĂLTEANU
- Target version set to 5.1.9
Updated by Marius BĂLTEANU 28 days ago
It seems that all the tests fail with a blank page in the screenshot.
Also, I'm wondering if we really need to run the system tests on the matrix or it should be enough to run them only on one Ruby version and one database, I don't expect to find compatibility issues which these kind of tests.
Updated by Katsuya HIDAKA 28 days ago
Thank you for committing the patch.
It seems that all the tests fail with a blank page in the screenshot.
I believe the test failures can be resolved with the fix I mentioned in #42422#note-12.
Updated by Katsuya HIDAKA 28 days ago
Also, I'm wondering if we really need to run the system tests on the matrix or it should be enough to run them only on one Ruby version and one database, I don't expect to find compatibility issues which these kind of tests.
Actually, I had the same thought when considering the introduction of system tests.
The current implementation was proposed as the simplest way to integrate system tests into the CI.
That said, I also think it's sufficient to run the tests against just one database. It's unlikely that differences between databases would affect user interactions in these tests.
As an alternative approach, I'm thinking of splitting the system tests into a separate job and running them only on the latest stable Ruby version with PostgreSQL. This should also help reduce the overall test execution time.
Updated by Marius BĂLTEANU 27 days ago
Katsuya HIDAKA wrote in #note-8:
[..]
That said, I also think it's sufficient to run the tests against just one database. It's unlikely that differences between databases would affect user interactions in these tests.
As an alternative approach, I'm thinking of splitting the system tests into a separate job and running them only on the latest stable Ruby version with PostgreSQL. This should also help reduce the overall test execution time.
Sounds very good! In the meantime, I've merged them to stable branches as their are.
Updated by Katsuya HIDAKA 26 days ago
- File job-summary-after-separating-system-tests.png job-summary-after-separating-system-tests.png added
- File Extract-system-tests-to-a-separate-job-run-only-on-SQLite3-and-latest-stable-Ruby.patch Extract-system-tests-to-a-separate-job-run-only-on-SQLite3-and-latest-stable-Ruby.patch added
I've merged them to stable branches as their are.
I appreciate your work.
I've attached a patch to extract the system tests to a separate job:
- Extracted system tests into a separate job in the workflow
- Configured to run only on the latest stable Ruby (3.4) with SQLite3
- Created a reusable composite action for test environment setup
You can see the results here:
https://github.com/hidakatsuya/redmine/actions/runs/15086836056
The changes have successfully reduced the overall CI execution time while maintaining test coverage.
Updated by Katsuya HIDAKA 26 days ago
- File Extract-system-tests-to-a-separate-job-run-only-on-SQLite3-and-latest-stable-Ruby.patch Extract-system-tests-to-a-separate-job-run-only-on-SQLite3-and-latest-stable-Ruby.patch added
I've updated the patch to simplify the artifact name:
- name: system-test-screenshots-ruby3.4-sqlite3
+ name: system-test-screenshots
Updated by Marius BĂLTEANU 18 days ago
- Status changed from Resolved to Closed
Committed both changes and merged to stable branches, thanks again Katsuya HIDAKA for working on this! The CI from Github looks very nice now.
Updated by Katsuya HIDAKA 18 days ago
Sorry, the current implementation causes system tests to fail on 5.1-stable and 6.0-stable. We need to specify a Ruby version compatible with each branch for the system tests.
I have created a patch to address the issue in #42770. I would appreciate it if you could take a look and commit it.