When working with application that uses machine learning to drive decision. Testing becomes very challenging. For examples, systems like review spam detection, images categorization applications and etc. So as a testers, how do you write your tests for these applications? What kind of automation framework we should use?
I recall that I tested one application which using machine learning algo to detect spam reviews from user submissions. The way I constructed my test plan, I broke down into 3 parts:
- Technical functionalities: all the test cases/automation which check the most basic technical functionalities. (if all these test passed, means the application runs and have some output, but it won’t tell you if the review spam is catching the right stuff.)
- Established baseline reviews for both spam/not spam: Manually create 50 reviews that are “spammy” and 50 legit reviews. Run through the application to make sure they did it correctly. We automated using a simple PHP program to run through this step
- Established an on going process (what I call AI for testing) to automatically feed more spam/not spam reviews from the output of the application and feed it back to step #2.
This framework did help us to find a few regression bugs. Not perfect, but yet it works.