Reloading submodules?

I’m developing a skill and I’ve noticed that it does a handy reload whenever I edit and save my __init__.py file. However for the files that I’m importing in that file, edits don’t seem to take effect. For example:

# __init__.py
from mycroft.skills import MycroftSkill

from .thing import mything


class MySkill(MycroftSkill):
    def do_the_thing(self):
        ...

If I make an edit to the MySkill class, the file is reloaded, and the new code is executed when my skill is triggered. However, if I edit thing.py, wait a minute or so, and trigger my skill, the old code is triggered as if it hasn’t changed.

Is this a known/expected behaviour? Is there a work-around that I should be following? Is it just common practise to put all of your logic into __init__.py?

Hi,

it has been common to put all the code in the __init__.py despite it not being ideal, due to this limitation.

There is an old WIP fix, but it may reload the submodules in the incorrect order if there are nested module imports. I was looking into this some more last week to see if I could get it simplified and updated.

A quick workaround may be to manually enforce the reload

from importlib import reload

from . import my_submodule
reload(my_submodule)

Not Ideal but I think it should work.

i’m looking to either simplify the old PR using find module or checking if modules such as lazy_reload works for the Mycroft case.

That is really unfortunate, as it throws a wrench into continued development of any skill of substantial size. The one I’m working on currently has 4 external imports along with the main __init__.py, and managing all of this in one Great Big File would be brutal.

I’ll try putting your work-around into my skill’s __init__() method and see if that does the job. Thanks.

1 Like

Timing is everything, I was going to post this exact question yesterday. I am having this exact issue during my development.
Currently what I am doing is deleting the submodule compile directory before I msm update my latest build.

/opt/mycroft/skills/cpkodi-skill.pcwii/__pycache__
1 Like

Hey my skill is talking to Kodi too! Well, Kodi and a few other things. I know it’s off-topic, but I gotta know: what’s yours do?

Here’s mine (work in progress)

@danielquinn, I already have a Kodi skilll that works very well. I am in the process of updating it to support the common play architecture. I am definitly interested in what you might be developing.
Currently mine supports, music / youtube(music/videos) / movies, these are already working, my next release will include TV-shows and I am also planning the ability to cast any items from the kodi library to a chromecast enabled device.
My original work is here

Non-functioning new work is here…

1 Like

Well there’s certainly a lot of overlap between the two projects. Have a look at kodi.py for the most relevant stuff, but note also that my “Majel” skill is also using the Common Play Framework, so you can reference that if you get stuck (though it’s pretty straightforward, they’ve done a good job there).

My project only does movies & tv, and doesn’t use Kodi directly, but only as a indexer/search engine for locally-stored stuff. In that way, the lookup for episodes etc. may be handy (I’m using kodijson unless you’ve got a better idea?).

I am just using requests for all my json. I had been using kodipydent in the past but found it didn’t have all the api functions available so I began doing my own. example…

def get_requested_movies(kodi_path, search_words):
    """
        Searches the Kodi Library for movies that contain all the words in movie_name
        first we build a filter that contains each word in the requested phrase
    """
    filter_key = []
    for each_word in search_words:
        search_key = {
            "field": "title",
            "operator": "contains",
            "value": each_word.strip()
        }
        filter_key.append(search_key)
    # Make the request
    json_header = {'content-type': 'application/json'}
    method = "VideoLibrary.GetMovies"
    kodi_payload = {
        "jsonrpc": "2.0",
        "method": method,
        "id": 1,
        "params": {
            "properties": [
                "file",
                "thumbnail",
                "fanart"
            ],
            "filter": {
                "and": filter_key
            }
        }
    }
    try:
        kodi_response = requests.post(kodi_path, data=json.dumps(kodi_payload), headers=json_header)
        LOG.info(kodi_response.text)
        movie_list = json.loads(kodi_response.text)["result"]["movies"]
        LOG.info('GetReqeustedMovies found: ' + str(movie_list))
        # remove duplicates
        clean_list = []  # this is a dict
        for each_movie in movie_list:
            movie_title = str(each_movie['label'])
            info = {
                "label": each_movie['label'],
                "movieid": each_movie['movieid'],
                "fanart": each_movie['fanart'],
                "thumbnail": each_movie['thumbnail'],
                "filename": each_movie['file']
            }
            if movie_title.lower() not in str(clean_list).lower():
                clean_list.append(info)
            else:
                if len(each_movie['label']) == len(movie_title):
                    print('found duplicate')
                else:
                    clean_list.append(info)
        return clean_list  # returns a dictionary of matched movies
    except Exception as e:
        print(e)
        return None

I’ll redouble my efforts to get the submodule reloading working without hitches. (I’m moving house so my spare time is sadly limited)

Bigger skills really need to be able to use multiple files.

2 Likes

Thanks for your efforts @forslund, both in the code itself and in posting here!

1 Like

If you figure this out and it works for you, Please let me know. In the meantime I will continue to delete my cache directory.

It does! Here’s the top of my file to see how I’ve implemented it:

import concurrent.futures
import os
import sys

from importlib import reload
from typing import Union

from mycroft import intent_handler
from mycroft.skills.common_play_skill import CPSMatchLevel, CommonPlaySkill

from .bookmarker import Bookmarker
from .exceptions import BookmarkNotFoundError, MediaNotFoundException
from .kodi import Kodi
from .streaming_services import StreamingServices
from .youtube import YouTube


# Hack to work around the auto-reloader not recognising files other than
# __init__.py: https://community.openconversational.ai/t/8879

reload(sys.modules["majel-skill.bookmarker"])
reload(sys.modules["majel-skill.kodi"])
reload(sys.modules["majel-skill.streaming_services"])
reload(sys.modules["majel-skill.youtube"])

# /hack

Note that reload() is looking for a module, rather than a string, and if you’re importing the modules with relative paths as I’ve done here, I found the cleanest way to do this was to import sys and use the name of the module to lookup the actual module in sys.modules.

Also note that I decided to put this at the top of the file (rather than inside __init__()) as this will ensure that the modules will be reloaded whenever the file is reloaded (ie. when it’s edited and the auto-reloader does its thing) rather than reloading whenever the class is instantiated.

2 Likes

This seems to work for me. Thanks

for each_module in sys.modules:
    if "kodi_tools" in each_module:
        LOG.info("Attempting to reload Kodi_tools Module: " + str(each_module))
        reload(sys.modules[each_module])
1 Like

I’ve updated the PR with something that seem to work (both according to theory and practice from what I’ve seen). If you want to try it out check the the feature/importlib branch.

Let me know if there are other issues that I’ve missed.

1 Like

This is now in dev and will be included in the next major release, thanks for flagging it Daniel, and of course to Ake for the fix :slight_smile:

Wow, that’s really great guys, thanks for the hard work!

1 Like