Creating my first skill with essentially no experience - Mycroft MagicMirror skill

Sounds really cool. Looking forward to hearing more of your progress!

1 Like

Hi there @dmwilsonkc thank you so much for the update!

Just to confirm, you mean that this PR on the mycroft-skills repo? If so, we’ve marked that one as an ‘override’ so we can override the normal testing that goes on with Skills, because the Skill is difficult to replicate without installing the Magic Mirror component. Normally what we do here is get someone else from the Skills Team, or a Community member, to vouch for the Skill if we can’t test it, but I’m not sure if there is anyone else out there running the Magic Mirror Skill?

I’m not sure how to go about sending all the Utterances and responses to the MMM-Mycroft module. @forslund may have some ideas on this one?

@KathyReid Thanks for the reply. So… I had a chance to visit the PlexPod and the Mycroft office space in Kansas City which was totally cool. I have been talking with @eric-mycroft and he asked if I could drop by Maker Faire Kansas City and visit with others about my experience with writing a Mycroft skill (with essentially no experience). Eric took the time to get a working copy of the MagicMirror installed on a RPi. He also installed the magic-mirror-voice-control-skill on a Mark 1 which was attached to the same network as the MagicMirror on the RPi. It did connect, but there was an issue that has since been corrected in the code. At some point, Eric can do a git pull from my repo to the Mark 1 and test it. He’s a busy guy, and I’m not in any hurray, but he will test it at some point.

I am now looking to extend the MagicMirror - Mycroft connection by creating a MMM-Mycroft module for the MagicMirror, which will provide a better Mycroft experience when it comes to the MagicMirror. Similar to the MMM-Kalliope module i referenced in the earlier post. My hope is to get some of the MagicMirror builders interested in Mycroft, and using it for a robust AI experience on their mirror builds. It seems to me that the MagicMirror community are the type of people that would be interested in an open-source, privacy-centric, AI.

I am stuck however. I’m not sure what the best way is to have essentially the same interaction that the bottom half of the CLI has, but displayed on the MagicMirror. So, once the wake word is triggered, display text of what Mycroft hears, and then the text response. I know how to get the text to the mirror, by sending a notification to the MMM-Remote-Control module, but how do I get Mycroft to send the notifications everytime the wake word is triggered and everytime Mycroft responds?

@forslund do you have any suggetions?

Super helpful! Thanks @dmwilsonkc, I will see if @eric-mycroft can test it for us - although I know he has some vacation time coming up, so it may need to wait until he returns from holidays.

Yeah, he’s been super helpful. I’m in no hurray. I’m sure it is a much deserved vacation. He was working pretty hard on Sunday.:grinning:

Hey friends. I have every intention of adding Magic Mirror to our dashboard TV here at the office and linking up the Mark I for control. I am OOO Thursday-Thursday coming up. May break some time off tonight to give it a go, otherwise it’ll probably be July 6 or later.

1 Like

@KathyReid @forslund @eric-mycroft Here’s another update on the magic-mirror-voice-control-skill. Well… actually I didn’t make any changes to the skill, but I did add a Mycroft Icon on the MagicMirror and text of what Mycroft hears and text of what Mycroft responds. At this point, it’s a complete hack, not a skill. I’m attaching a short video to give you an idea of what the interaction between the user and Mycroft look like on the MagicMirror.

On the MagicMirror, I am using the MMM-kalliope module to receive the text from Mycroft through a requests.post to the MagicMirror’s ip address at the default port/kalliope.

Now here’s the hack, and I would of course prefer another way, but I’m kind of stuck on how to do this in a skill. I added a couple of lines of code to the /home/pi/mycroft-core/mycroft/client/speech/main.py:

def handle_wakeword(event):
    LOG.info("Wakeword Detected: " + event['utterance'])
    ws.emit(Message('recognizer_loop:wakeword', event))
    voiceurl = 'http://192.168.3.126:8080/kalliope'  #<----- Add these lines, using your mirror's ip
    voice_payload = {"notification":"KALLIOPE", "payload": "Listening"}  #<-------
    r = requests.post(url=voiceurl, data=voice_payload) #<-------

def handle_utterance(event):
    LOG.info("Utterance: " + str(event['utterances']))
    context = {'client_name': 'mycroft_listener'}
    if 'ident' in event:
        ident = event.pop('ident')
        context['ident'] = ident
    ws.emit(Message('recognizer_loop:utterance', event, context))
    utterance = str(event['utterances'])        #<--------- I added these 6 lines of code
    utterance = utterance.replace("['", "")    #<--------- to send what Mycroft hears
    utterance = utterance.replace("']", "")    #<--------- formatted without ['example']
    voiceurl = 'http://192.168.3.126:8080/kalliope'  #<----- to a hard coded url
    voice_payload = {"notification":"KALLIOPE", "payload": utterance}  #<-------
    r = requests.post(url=voiceurl, data=voice_payload)   #<----- again, I know this is not ideal

It does work as you can see from the video, but the timing is slow. It is convenient from the perspective that if Mycroft doesn’t hear you quite right, you realize what it heard.

The other hack was to the /home/pi/mycroft-core/mycroft/client/text/main.py:

def handle_speak(event):
    global chat
    utterance = event.data.get('utterance')
    utterance = TTS.remove_ssml(utterance)
    voiceurl = 'http://192.168.3.126:8080/kalliope'     #<- I only needed to add three lines
    voice_payload = {"notification":"KALLIOPE", "payload": utterance}  #<--- here
    r = requests.post(url=voiceurl, data=voice_payload)   #<---- and here
    if bSimple:
        print(">> " + utterance)
    else:
        chat.append(">> " + utterance)
    draw_screen()

Again a hard coded url, at this point it is just a hack.
Is this even possible using a skill???
if so, how would the skill get triggered??
I’m looking for a little advice.

Cheers!
Dave

I am creating a skill that is forwarding the remainder portion of my utterance to another computer through a udp connection. The computer “daemon” will be running adapt - parser to parse the reminder in the daemon. The mycroft skill will parse the original command “ask the computer to…” Then pass on the remainder.
Not sure this is helpful or not. Good Luck. If I ever get some time I will be playing with the magic mirror project.

@pcwii Thanks for the idea. I’m currently trying to capture every utterance including when the wake word is detected. I just am not sure if that can be done in a skill. The UDP idea could really be useful with multiple devices I think. In my example, multiple mirrors. So I could say show weather on the entry way mirror, for example, and have a broadcast message to all mirrors, but have them programmed to listen for their own notification. Cool idea.

How would you see UDP working to display “Listening” for example, every time Mycroft hears the wake word? That’s my current dilemma.

The only way I can see this working is to modify the wake word listener code to send a UDP packet once the wake word is triggered. Maybe somewhere around when the listening sound is played. Not sure in the code where this is and if we can find a clean method to enable / disable this in the mycroft config we may be able to submit a pull request for it. I actually feel there is some value in creating a process to “echo” the sst to other devices so that it can be processed there. This would permit a workable communications channel to/from other “mycroft compatible” devices. It is fairly easy to remotely send mycroft commands using websockets (https://github.com/pcwii/testing_stuff/blob/master/send_to_mycroft.py) but I don’t think it is as easy to intercept commands it is processing.

You should be able to move those things into your skill by listening for the wake words, STT result and the speak messages on the bus.

This is done using self.add_event(MESSAGE_TYPE, self.handler_method) where MESSAGE_TYPE is a string and self.handler_method is a method you create for handling the message.

The message types for the above cases (if I’ve understood them correctly) is:
Wakeword detected: “recognizer_loop:record_begin”
STT result: “recognizer_loop:utterance”
Mycroft speaks: “speak”

It would look something like this in your skill:


[...]

    def initialize(self):
        self.add_event('recognizer_loop:record_begin', self.handle_listen)
        self.add_event('recognizer_loop:utterance', self.handle_utterance)
        self.add_event('speak', self.handle_speak)

    def handle_listen(self, message):
        voiceurl = 'http://192.168.3.126:8080/kalliope'  #<----- Here you can use self.settings['ip']
        voice_payload = {"notification":"KALLIOPE", "payload": "Listening"} 
        r = requests.post(url=voiceurl, data=voice_payload)

    def handle_utterance(self, message):
        utterance = str(event['utterances'])
        utterance = utterance.replace("['", "")
        utterance = utterance.replace("']", "")
        voiceurl = 'http://192.168.3.126:8080/kalliope'  #<----- again with self.settings
        voice_payload = {"notification":"KALLIOPE", "payload": utterance} 
        r = requests.post(url=voiceurl, data=voice_payload)

    def handle_speak(self, message):
        voiceurl = 'http://192.168.3.126:8080/kalliope'
        voice_payload = {"notification":"KALLIOPE", "payload": utterance}
        r = requests.post(url=voiceurl, data=voice_payload)

The speed will still be about the same as previously though…

2 Likes

@forslund You are the MAN!!! Dude, that is exactly what I was hoping for!! I’m having a few issues with the latest update for MagicMirror (my current install is broken - badly!). Once I revert back to a previous image I will give this a try.

Thanks Åke, you really are the MAN!

2 Likes

@forslund thanks to your suggestion here is the code that ended up working to display both what Mycroft hears and what Mycroft says, along with a wake word “listening” indicator.

def initialize(self)
    self.add_event('recognizer_loop:wakeword', self.handle_listen)
    self.add_event('recognizer_loop:utterance', self.handle_utterance)
    self.add_event('speak', self.handle_speak)

def handle_listen(self, message):
    voice_payload = {"notification":"KALLIOPE", "payload": "Listening"}
    r = requests.post(url=self.voiceurl, data=voice_payload)

def handle_utterance(self, message):
    utterance = message.data.get('utterances')
    voice_payload = {"notification":"KALLIOPE", "payload": utterance}
    r = requests.post(url=self.voiceurl, data=voice_payload)

def handle_speak(self, message):
    speak = message.data.get('utterance')
    voice_payload = {"notification":"KALLIOPE", "payload": speak}
    r = requests.post(url=self.voiceurl, data=voice_payload)

The only issue that I am having is that the user utterance gets displayed on the mirror after the text of what Mycroft says. I am not sure if there is a way to send the user utterance to the mirror before Mycroft’s speak utterance so that it makes more sense. I’m not sure if there is an earlier messagebus event during which I could capture the user utterance. When I get time, I will try doing another short video to demonstrate what the skill does now. The good news is that it’s not a core-hack anymore and the requests.post happens from the skill itself. Although it still is delayed from how fast Mycroft reacts to the utterance.

Thanks again for your guidance!

2 Likes

@dmwilsonkc,
Congratulations on the amazing progress on your MagicMirror skill. I have a question about capturing the utterances. I have used your code snippet above to provide notifications to kodi in my kodi-skill and have noticed that I will receive the message "Listening!, and I will receive mycroft’s response but what I don’t see are my utterances. Are you having the same issue or do your utterances get returned in the event?
thanks,

1 Like

@pcwii Thanks. So, I am sort of having the same problem. I haven’t quite figured out why, but my utterances are being displayed after the Mycroft utterances in most cases. The only time my utterances are displayed before Mycroft’s are when Mycroft has to “get information” to answer my request. So for example, when I say “What’s the weather for tomorrow?” and Mycroft has to go to the internet to get the answer, my utterance appears first. But if the response is either hard coded or in a dialog file, Mycroft’s utterance appears first, then my utterance follows. Also, the “action” happens before anything. So for example, if I say “Hey Mycroft” the chirp sounds to indicate the listening state, I respond “hide clock”, the “listening” notification then appears, the MagicMirror clock gets hidden, then Mycroft responds with one of the pre-configured dialog responses audibly, Mycroft’s response appears in text on the mirror followed by my utterance.

I am not quite sure why the timing of each event is being processed at different rates. If I had to guess after a quick browse of the as much code as I could peruse, it might have something to do with the converse() portion of the code when handling the user utterances. It looks like there is a programmed delay before the utterance is passed to the adapt intent parser. That is just a guess and may not be the case at all.

The other peculiarity is that when I hacked the /home/pi/mycroft-core/mycroft/client/speech/main.py
and the /home/pi/mycroft-core/mycroft/client/text/main.py as i did in reply 56 above, it worked perfectly with no timing issues other than it was just delayed a little from Mycroft’s actual audible response as you can see in the video I posted. The user utterance appeared before Mycroft’s utterance every single time. So maybe it has something to do with the way my skill processes the messagebus notifications. @forslund, do you have any ideas?

In my case I am seeing the listing notification then the response, but never the utterance. Not sure why at this point.

@pcwii Did you copy the code? Or type it? The reason I ask is it needs to be

utterance = message.data.get('utterances')

utterances with an ‘s’ at the end. It doesn’t work without the ‘s’. I typed it wrong the first time myself.

Could the issue be in the event registration.


Or here?

Thanks.

@pcwii The only reason I could thinks of at this point is your if statement. If it is not triggered, obviously it would not send the notification. Try it without the if statement to see if it works. Then you’ll know.

utterance = message.data.get(‘utterances’) <---- shouldn’t this be utterance (without s)?

I saw what @dmwilsonkc wrote but in a skill of mine it looks like this:
response = message.data.get(“utterance”)

And it seems to work just fine