HookTest resets all listeners - core tests run inconsistently with plugins
When a test runner is invoked and the tests include either
test/unit/lib/redmine/hook_test.rb, there is a side effect that all hooks classes / listeners are cleared. It is probably related to hook listeners being stored in class variables.
This may affect other tests, but most importantly, it makes it difficult to verify redmine core integrity with plugins in place. While plugins' patches are still applied, the listeners are not guaranteed and the results differ between e.g.
rake test and
More info about the context below.
Although this might not be necessarily a defect - maybe core tests are not supposed to run with plugins - the consequences of plugin developers not running the core tests are terrible and affect the reputation of whole Redmine ecosystem.What we've observed:
- We have run Redmine core tests with plugins we use.
- It's even difficult to find a plugin that doesn't break anything in the core.
- A logical objection would be "plugins modify core functionality, so it's normal that core tests are not passing". But this turned out NOT to be the case. In fact:
- Around 50% of the errors and failures are that the implementation is not robust enough to withstand testing.
- And the other 50% are user-reproducible bugs, which would have been discovered if core tests had been run.
- In the end, we have not observed any error or failure caused by intentional plugin behavior.
Based on your feedback, I can provide specific examples and will be happy to update plugin_tutorial to encourage core testing.
Updated by Martin Cizek almost 3 years ago
We currently use the functions in the attached script to work around the issue.
At the moment, the only way around is to temporarily rename the tests with plugin-breaking side effects. In Rails 6.2, there is `DEFAULT_TEST_EXCLUDE` env that can be used instead of renaming files. But avoiding the side effect completely would still be the very very best. :)