CI/CD runners and unit tests for skills

Archive - Mycroft Project Archive - Former Random Mycroft Category
1

Hello, I wonder how unit testing in Mycroft skills works. Especially if a CI/CD runner (Docker) performs the unit tests for a skill. In my example case, the runner complains that it does not know the “mycroft” module (error).

The documentation has a section for mycroft-core unit tests, but there is nothing for skills yet, I think. Could anyone please point me in the right direction?

2

Hey ScienceGuy,

We’ve been looking at this a little bit recently. There haven’t “traditionally” been unit tests in Skills but especially when you get to more complex Skills they become very important.

First, to make sure the mycroft module and all of the goodies that come with Mycroft are available, you want to have the Mycroft venv activated eg:

source ~/mycroft-core/.venv/bin/activate

And depending on what you want to add unit tests for, you might need to restructure your Skill a bit.

The News Skill has some unit tests for it’s station module which is easy enough with a “normally structured” Mycroft Skill.

But to test your Skills class you would probably need to move that class to it’s own module. I’ve done a quick example on a branch of the HelloWorldSkill here:

Because of the way we load Skills into Mycroft - at present you still need to have the top-level __init__.py file containing the create_skill() function.

Be good to hear how this fits in with your development workflow and whether it meets the needs of your project?

3

Actually, the skill class needs to be in __init__ as well. Core looks for dialog and vocabulary files in a directory relative to where the skill class resides. We would like to fix this at some point, so all the skill code can go in a skill module, but it might be a while before we get to that project.

4

Yep so in this example - all the dialog/vocab/locale directories are moved into the skill subdirectory.

I’m not sure if there will be other issues arise from doing this…

5

Hi all, please check my PR #77 in skill-homeassistant. I am proposing GitHub Actions workflow with few linters and vk tests with allure report.

1 Like
6

That is awesome work Tony!
Apologies for so many comments - it’s a big PR

I’ve been thinking that it would be great to have some standard Github Actions workflows that any Skill could use and run on Microsoft’s metal. A lot of our stuff runs on Jenkins which has it’s own benefits, but the current infrastructure isn’t made to scale for all community Skills.

1 Like
7

I managed to get a GitLab pipeline working that performs the unit tests for my skill. The skill has a “python” folder that contains the test descriptions. This is the job’s code, ideas welcome.

# Unit tests for a Mycroft skill
# Uses an Ubuntu image to install a Mycroft core
# Runs the Mycroft setup because it installs dependencies
# Uses the Mycroft venv to install the skill using msm
unit-tests:
  stage: test
  image: ubuntu:18.04
  before_script:
    - apt-get update && apt-get upgrade
    - apt-get install -y sudo
    - TZ=Europe
    - apt-get install -y tzdata
    - apt-get install git -y
    - git version
    - echo "Ubuntu preparations completed."
    - git clone "$MYCROFT_REPOSITORY_URL"
    - cd mycroft-core
    # Run this with root only in the container.
    - bash dev_setup.sh --allow-root
    - echo "Mycroft dev setup completed."
    - source .venv/bin/activate
    - python --version
    - msm install "https://$DEPLOY_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH"
    - echo "Skill installed, ready for unit testing."
  script:
    - cd ..
    - python -m unittest
  only:
    changes:
      - python/**/*
      - .gitlab-ci.yml
2 Likes
8

Hi @ScienceGuy check my PR to Home Assistant skill, I managed to incorporate both behave tests and unittests into GitHub Actions workflow and output their results to Allure Report.

2 Likes