Originally published at: https://mycroft.ai/blog/the-voight-kampff-test/
No, we don’t foresee Mycroft software running in replicants any time soon. But it is a fun name for a new test suite we have been working hard at over the past couple of months.
User experience is important to us. It suffers when a change is made to one part of the software that inadvertently causes breakage elsewhere in the stack. These issues can be hard to find using manual interaction with a device.
Our goal with this test suite is to build an automated way to ensure a user’s interactions with the voice assistant software are always meeting expectations.
Voight Kampff Integration Test Framework
We chose Python Behave, a Behavior Driven Development (BDD) framework for this project. The primary benefit to this framework over a unit testing framework like pytest, is that it focuses on behaviours and expectations using a very English-like representation of the test conditions. This provides a clear test and acceptance criteria that anyone in our team and the broader Community can understand and contribute to.
Voight Kampff tests are split into Features. Each Feature may have one or more Scenarios, and each Scenario will have multiple Steps.
Taking a quick example of a Weather Skill. A Feature of this Skill might be the ability to report the current weather conditions. A Scenario within that Feature is that a user asks for the current weather. The Steps to test this Scenario might then be:
- Given an English speaking user
- When the user says “tell me the weather”
- Then “my-weather-skill” should reply with “Right now, it’s overcast clouds and 32 degrees.”
As you can see this is quite simple to understand, and enables anyone to participate in running and expanding our test coverage.
How do I get started?
The first iteration of Voight Kampff will be included in our next release of Mycroft-core (v20.2.2). If you’re on the dev branch, you will already have access to it.
This is all part of our testing road map, which we’ll outline in a future post. Currently it injects messages representing the output of the speech-to-text step into the system. Then it inspects the messages that would be fed into the text-to-speech step. Future iterations will expand the scope to include the speech-to-text and text-to-speech engines with the aim to create complete end-to-end tests.
If you want to learn more about Voight Kampff, including how to use these tests for your own Skills, check out our expanding documentation.