Originally published at: Voight Kampff: How You Can Help #addatest - Mycroft
Last week we introduced the Voight Kampff Integration Test Framework. Our internal team has been busy over the last month implementing this and creating base tests for Mycroft’s most used Skills.
This has significantly expanded our test coverage and has already uncovered over a hundred failing tests. That is at least 100 actions that are not working as they should be and I’m sure there are more out there. So we need your help!
“But I’m not a developer” I hear some of you say. That is great! We need people that don’t know their Python from their Snake Case.
1. Tell us when something needs fixing and #addatest
The best way that you can help us, is to let us know when something doesn’t work the way you expected. This might be a phrase that Mycroft didn’t understand, or a bug in one of the components.
Whatever it is, we want to #addatest so we can not just fix it, but make sure it stays fixed. Adding new tests for our Skills is a great way for anyone in the community to help. It doesn’t require any knowledge of Python or how our Skills work.
Let’s quickly revisit what an example Voight Kampff test looks like.
Given an English speaking user
When the user says “tell me the weather”
Then “my-weather-skill” should reply with “Right now, it’s overcast clouds and 32 degrees.”
If you can understand these three lines, then you can help Mycroft by writing new tests. It’s as simple as Given – When – Then.
Anytime you find something that isn’t working as expected, we want to know about it. To submit new tests you can:
- Reply to this Voight Kampff thread in the Forums
- Add an issue to the relevant Skill on Github and tag it with #addatest, or
- In post in Mycroft Chat and include the hashtag #addatest
2. Add tests to your own Skills
If you’re a Skill author yourself, then adding tests to your Skills is not just a good development practice, it also helps us to make sure that all of the Skills in the Marketplace are operating as expected.
Voight Kampff allows us to test your Skill alongside all of the default Skills and any combination of other Skills. We do this when you first submit it, and anytime a change is made to Mycroft Core. This prevents some change in the system, causing problems with your Skill.
To see how you can get started with Voight Kampff, check out our new documentation.
3. Add new Steps
As we’ve seen Voight Kampff is made up of Steps. There are many existing Steps that will be common for many if not all Skills such as, “When the user says {phrase}”.
Your Skill however might need a Step that doesn’t yet exist. It might be for functionality that is specific to your Skill, or we may not have thought of it yet.
Consider the Timer Skill. One thing we would want to test is that the user can cancel a timer. To do this, there has to be a timer that exists for us to cancel it. We need to define the state of the Scenario, for which we use a Given Step.
Given a 10 minute timer is set
This Step is perfect for our Timer Skill, but would not be useful for most other Skills. The great news is that we can define Custom Steps for our Skill without needing to add them to Mycroft Core. You can learn all about Custom Steps in the documentation.
If you have a Custom Step that you think would be useful for others and should be added to the default Steps, let us know about it. Join us in the Forums, the ~Skills channel of Mycroft Chat, or create an issue on Github.
4. Fix a failing test
As I mentioned earlier, Voight Kampff has already been very successful at uncovering some bugs in our existing Skills. If you have some experience in Python, we’d love your help in squashing these bugs one-by-one.
Any known failing test is tagged with “@xfail”. This enables us to keep the tests running, but not block our continuous integration pipeline in the process. It also has the benefit that we can quickly find bugs to fix.
Here you can see all the tests that are currently tagged as failing.
If you are looking for a way to help out, fixing one of these failing tests is a massive help to Mycroft and to everyone else in the Community that will use it.