Abandoning Python 2.7 and adopting Python 3.4+ | What you need to know as a Community Developer


#1

Originally published at: http://mycroft.ai/blog/python-2-7-python-3-4/

Are you a Community Developer? Have you previously written Skills for the Mycroft ecosystem? Know Python, and considering writing Mycroft Skills?

Then you need to read this blog post to understand a key change to Python language support in Skills, and what you need to do to ensure the Skills you maintain are compliant.

We’re moving from Python 2.7 to Python 3.4+. What does this mean?

Over time, programming languages evolve to meet the needs of their community. That evolution often means that newer versions are not backward compatible. This blog post by core Python developer Brett Cannon explains in more detail why Python moved to version 3. When it began development in late 2015, Mycroft software was still in its infancy - as was Python 3.4+. The decision was made at the time to standardize on Python 2.7. Over the intervening two and a half years, Python 3.4+ has become the de facto standard for Python software development. If Mycroft does not move now to support Python 3.4+, we will become further and further behind the industry standard.

Python 2.7 and Python 3.4+ are not interchangeable; code that works in Python 2.7 might not work in a Python 3.4+ environment. This means we essentially have to choose a time at which to “rip the band-aid off” and drop support for Python 2.7 and adopt Python 3.4+, otherwise we would need to support both versions of the language side by side; this support overhead is not maintainable.

What additional benefits does Python 3.4+ have?

Python 3.4+ introduces several new features that make development easier and less error prone. Key among them is out-of-the-box Unicode support - which is a must-have for Mycroft being able to support languages other than English in the future. Python 3.4+ also handles many functions which return array-like data structures differently, with many now returning iterable objects instead of list data types; this feature provides additional flexibility in data handling. This blog post, from Sebastian Raschka, highlights other key language differences, and is well worth a read. Lisa Tagliaferri from Digital Ocean also has a great write-up on the changes.

What does this mean for me as a Community Developer?

From the 18.2.6b release, mycroft-core is switching to Python 3.4+. If you have Skills that have not been tested under Python 3.4+, then they may not work as intended from version 18.2.6b onwards. We recommend that you take the following action:
  • Implement automated testing for your Skill- We’ve recently implemented automated testing tools to help you ensure that your Skill works as intended, and which helps Mycroft assure the quality of Skills that are made available to end users. From 18.2.6b, the automated testing tools will test against Python 3.4+.
  • Run your Skill through an automated tool that checks Python 2.7 => Python 3.4+ changes - there are several of these tools around, however we’ve had best results with this one from Juan Carlos Garcia.
If you haven’t already, it’s worth reading up on our Skill Branching changes as well.

In particular, we’ve seen the following common issues:

Python Future also has a great cheat sheet that helps understand the differences between the two versions.

And this 2-to-3 software applies a set of fixers to help auto-translate your Python 2.7 code to Python 3.x.

Virtual environment

As part of this switch we are also migrating from virtualenv to the more lightweight venv. venv was introduced as part of the standard library in Python 3. This offers the same type of features as virtualenv but doesn’t require external dependencies, meaning that it is easier to troubleshoot and support.

I’m a Skill Author. How do I test my Skill under Python 3?

We’re presuming that you do Skill development using a Mycroft for Linux install, installed through GitHub. To test your Skill under Python 3.4+, do the following:
  • In the terminal, navigate to the directory where you installed mycroft-core
  • Next, you will pull down the dev version of mycroft-core using the command mycroft-core$ git pull origin dev
  • Next, you need to run the dev-setup.sh script by typing mycroft-core$ ./dev-setup.sh
  • This will establish your Python 3.4+ environment
  • Next, you need to test that your Skill works. Ordinarily, your Skill will be located at /opt/mycroft/skills. However, if your Skill is not already installed, install it by using the command mycroft-core$ ./msm/msm install [URL for your Skill’s git repository]
  • Test your Skill by ensuring that you trigger every Intent that the Skill has, including any Stop or Pause Intents, and checking that they have the intended result. If an Intent in your Skill is not functioning, then you are likely to hear the phrase An error was encountered in Your Skill Name when the Intent is triggered.
  • Use the resources above to examine your code for Python 2.7 -> Python 3.4+ incompatibilities and resolve them.
  • You will then need to resubmit your Skill to the 18.02 branch. Read more here about Skill Branching.

I’m using another Skill Author’s Skill, and it doesn’t work under Python 3.4+. What do I do?

If you encounter a Skill from another Skill Author that does not function correctly under Python 3.4+, then we recommend that you create an Issue in GitHub under that Skill’s repository. We’re very happy to provide assistance that Skill Authors may need in making their Skills function correctly under Python 3.4+, however we do not directly control Community-Developed Skills.

Where can you go to get assistance?

If this all sounds a little confusing, we totally understand. There’s a bit of a learning curve involved in becoming a Mycroft Community Developer, but we’re here to help. You can get assistance in our Mattermost Chat - the Skills channel is going to be the most helpful, or on our Forum.

#2

#3

@KathyReid I just started working on a Mycroft skill to control a Magic Mirror running on a RPi3b. At this point, because I have really just started, it seems to me from this post I should re-install the mycroft-core from the dev branch to this MagicMirror build before I get any deeper. I looks like there is a Python3-full branch on Github as well. Just to be clear, if I re-install mycroft-core from the dev branch will I then be in a Python 3.4+ environment?


#4

Short answer is “yes”, but if you wait a few hours or so, the Python 3.x goodness will be promoted to master and your Device should update automatically :+1:


#5

@KathyReid Awesome thanks! Will I need to run the ./udpate.sh or will it just happen automatically?


#6

Ah, I see you have a Linux install. In this case you’ll need to do a git pull origin master then run ./update-dev.sh :thumbsup:


#7

@KathyReid Worked like a charm! Thank you! Now to finish the Magic-Mirror-Voice-Control-skill.

Cheers!


#8

So, the python3-full branch was actually left over from a while ago. I’ve just removed it to prevent confusion. You can actually get Python 3 in the dev branch of mycroft-core. This is done done via a slight modification of Kathy’s command:

git checkout dev  # Makes sure we are on the right branch
git pull origin dev  # Update the dev branch with the latest changes
./dev_setup.sh  # Update/install the new dependencies for Python 3 mycroft-core

Also, sometime around tomorrow, we plan on publishing the Python 3 changes which will bring them the the stable master branch.


#9

Neither dev nor master branches contain mycroft-core/test/integrationtests/skills/skill_developers_testrunner.py yet the public documentation instructs us to run it before submitting. I am confused.


#10

Thanks for flagging this @jrwarwick. My apologies, in a reversal of usual state of affairs, my documentation is a little ahead of where development is. PR #1527 contains all the automated testing code and framework for unit tests. This isn’t yet merged into the dev branch because some of the Python 3 pieces have not yet been verified. We’ll get it merged soon, and apologies again for the confusion.


#11

Thanks for the clarification, Kathy. However, I would like to point out it isn’t just the documentation that is “ahead”. There has been at least one public announcement that highlighted the “recently implemented automated testing tools to help you ensure that your Skill works as intended…” .

I hope that I do not sound like one who is looking a gift horse in the mouth, I am grateful indeed for the early warning and the time to look into and think about what might need to be changed. Just letting you know that at least one person had the overall impression that we should already be using the tools to get ahead of the change.


#12

Totally fair and reasonable feedback @jrwarwick - internally I need to make sure I’m better synced with other teams so we’re sending consistent and accurate messaging.