We ownClouders believe that our extensive automatic testing, that we foster since ownCloud 10 times, is one of the most important pillars of the stability and quality of ownClouds products such as Infinte Scale.
Recently we made a big milestone in the Infinite Scale codebase that we want to report about as we think it is important and really interesting.
How it Started
We were starting with UI testing using Nightwatch and Selenium. There was an integration with our products called the test middleware for creating preparatory steps in test.
That was great, but at some point we were seeing downsides, such as
- test execution was slow
- no auto-waiting selectors
- it was hard to debug as developer
- and hard to debug when running in CI
With that, we ended up with a large number of tests for negative scenarios and low priority functions, which further down the road led to a large number of skipped tests, because there were more and more tests expected to fail for several reasons.
Restructuring
To raise efficiency in testing, we decided to restructure testing a bit and use a new testing framework.
In phase one, we stopped writing tests in the “old world” and started to write new tests with Playwright. That was evaluated in the beginning with the very concrete task to automatize the smoke testing of Infinte Scale, and it turned out to be good.
The middleware that was used before could be retired by replacing it with the Playwright API Testing. Also, tests could now be written just like user stories.
It was also way easier to run tests in the CI with different deployment scenarios, such as different IDP or app-provider configurations, for example WOPI with OnlyOffice or Collabora.
This turned out with a lot of benefits for us:
- better test runtime performance
- less flaky tests because of autowaiting
- nice cross platform and -browser capabilities
- support for easier debugging through tracing
Very amazing are the different contexts for test users, which allow us to test multi-user collaboration for example collaborative editing on documents in Infinite Scale.
In phase two, all the previous UI tests were ported to the new platform.
That was finalized recently and we are super excited to have this new testing framework in place.
If you are interested, you can find out more about Infinite Scale Testing. There is even a testing workshop available for interested developers. Testing pays off!
We are pretty excited about this great progress. What do you think?