Mycroft Skills etiquette: How should they interact?

Mycroft Project Skills
#7

^Agree this could be more concise, perhaps just “Playing song” or “Here’s song” (or even “You got it!” …might be nice to have a little variation in the responses, especially if it’s a frequently played and/or quickly accessible song).

^Agree with this as well.
In terms of the general “Hey Mycroft… Stop.” command to terminate skills, I think we could make that language a little more natural. (This would involve giving Mycroft a new activation phrase to listen for in addition to “Hey Mycroft”, and I have no idea how hard that is programmatically. This idea may very well be unfeasible.)
For any skill that’s awaiting a termination command, like a timer, or for any skill that gives a verbal/visual reply to a question (e.g. “What’s the date?” / “What’s the capital of France?”), it would be great for Mycroft to recognize the termination command “Thank you Mycroft.”

  • Use case 1: “Hey Mycroft … Set a 5 minute timer.” [Timer goes off. It’s beeping loudly.] Instead of saying, “Hey Mycroft…” [wait for Mycroft sound to indicate that it’s listening] “…Stop.” the user could simply say, “Thank you Mycroft” and the timer would end.
  • Use case 2: “Hey Mycroft… What’s the longest city name in the world?” Mycroft replies something like: “The longest city name in the world is in Wales, called ‘llanfairpwllgwyngyllgogerychwyrndro—’ .” The user realizes they have made a terrible mistake, so they simply say, “Thank you Mycroft.” and Mycroft immediately stops talking.

(I noticed in the video that when asked for the date, Mycroft replied with the entire date including year, and also left the visual reply on screen for what seemed like an excessively long time. I realized it might be helpful if, as soon as the user had gotten the information they needed, they could say, “Thank you Mycroft” and terminate both Mycroft’s verbal reply and the and visual reply displayed on-screen.)

Lastly, it would be nice if there was some sort of visual indicator that Mycroft is ready to receive commands. For example, Google Assistant has a colorful border around part of the screen, which changes depending on whether its state is “ready”, “actively listening”, or “processing command” (see image below). If Mycroft had some similar visual indicator, it would help users avoid the frustration/confusion of giving Mycroft a new command while it’s still working on processing the previous one.
For example, imagine there’s a pulsating blue bar at the bottom edge of the screen to indicate Mycroft is ready. The users says “Hey Mycroft…” The blue bar lights up to indicate Mycroft is listening. The user continues, “…play song.” The blue bar undergoes some sort of animation to indicate Mycroft is processing the request. The music player appears on screen, and Mycroft says, “Playing song by artist.” Music begins to play, and the blue bar returns to its original pulsating state, indicating that Mycroft is now ready to receive further commands.

image
image1600×800 136 KB

2 Likes
#8

All of those are excellent ideas. I especially like “Thank you Mycroft” to instantly stop the current action

4 Likes
#9

One thing to note is that “Thank you Mycroft” should always make Mycroft stop talking, but should not always terminate the skill. For example, if a timer is started, and the user for whatever reasons says, “Thank you Mycroft,” before the timer has run out, that probably should not terminate the timer. As another example, if the user says, “Hey Mycroft, play song,” and Mycroft starts to reply, “Playing song by artist,” if the user says, “thank you Mycroft,” that should make Mycroft stop talking, but it should not prevent the music from playing.

4 Likes
#10

Thanks for all your feedback already!

There’s no need to limit suggestions to what is possible in the current implementation. We are building a roadmap so if we can’t get to a particularly good idea right now it will get put on the plan.

Also I should point out that we don’t expect us to reach a consensus on how Mycroft should behave. Our immediate goal is to establish a reasonable default behavior. But ultimately we want your Mycroft (or whatever you call it) to develop a “personality” that suits you. This can be accomplished in a variety of ways at the technical level- configuration options, machine learning and installation of alternative skill implementations, for example. But for now we’re not worried about that: we’re establishing the user requirements. Implementation details come much later.

So keep those ideas coming. And if you find your opinion in the minority, just remember that we’re not looking for one ring to rule them all. We are looking to enable infinite diversity in infinite combinations.

1 Like
#11

Well, commands both to ‘stop the current skill’ and also to ‘stop talking but otherwise continue’ would be useful. What those commands should be is perhaps an implementation detail

1 Like
#12

I agree that the “play music” response is too verbose and a simple “sure” or “ok” response would be fine. Would be nice to have some contextual feedback only if the song or artist is not found, maybe with some options listed for play selection.

1 Like
#13

I am not sure if it is a detail. The most intuitive way, in my view, is to say “stop” when you want that your assistant stops what it currently does in a narrow sense. If it talks, it should stop talking. If it displays some specific content, it should stop displaying that specific content. If it plays music, it should stop playing music, etc…

The semantic ambiguity of “stop” may very well require that Mycroft understands much more context - and ultimately also personal context (that can become personal data if Mycroft collects enough to finger print users). So, I think that skill etiquette depends on/interacts with data privacy at least.

Probably, it also depends on accountability, e.g. when an assistant does something for you that you rely on. For instance, you may want Mycroft to order 10 pizzas for your birthday party. While Mycroft utters that it successfully sent the order, you quickly realize you want someting else and tell Mycroft to stop. Does that mean it should stop uttering or cancelling the order? Should there be a different “default” for skills that perform something that may cost you money?

2 Likes
#14

In lack of context, a layered cake of “skill hierarchy” could be applied. It’s generally obvious that when music is playing and the alarm goes off, a single “stop” command is directed at the alarm.

At least do I not see any case where the opposite would be expected.

I do realise that implementing “skill hierarchy” generally is no simple thing, maybe it could be a personal setting in each skill?

1 Like
#15

I think a lot of people lose track of what day of the month it is. Very few lose track of what year it is. I think it is generally a good policy to minimise clutter in mycroft responses. If I want to quickly check what the date is, then ideal response is “today is the 28th of September”. I most likely know what day of the week it is and what year of the century it is. I can ask specifically for that info on the rare occasions I need it. Long winded responses from a voice assistant quickly grows old. The music request should definitely not be answered preceded with a long winded announcement unless as pointed out it is taking an unusually long time
I like the nesting of active contexts in the demo. Alarm going off is active, then stop applies to that and not to background contexts. Could have some catchall as suggested such as “mycroft quiet” if I need immediate silence to talk to someone in the room or answer my phone. (might be a nice skill for mycroft to recognise a phone ringing)

2 Likes
#16

When there’s a display then anything ambiguous or generic (like “stop”) should go to the displaying application.

If the displaying skill doesn’t handle it but others do then “do you want to stop timer1, egg timer or music” seems reasonable rather than stopping something you didn’t want stopped.

When the timer goes off it would be nice if Mycroft entered listening mode whilst it is sounding as either you’re not there to hear so it won’t matter or you’re likely to want to speak and acknowledge the timer.

some other comments that I would vote for:

  • Not speaking the year
  • Music response is too long (though 10s is an eternity)
  • Visual feedback of listening

Nitpick: The timer display of -ve numbers is quite geeky if you think about it.
It should ideally say “10s ago” and maybe change colour? I’m not sure if there is space to put “went off 10s ago”.

1 Like
#17

Time and date were fine.
If the music could be found “quickly”, skip the intro announcement.
Stopping the timer and music:
The timer should continue until verbally stopped (good).
The issue, thoush, is how should Mycroft interpret potentially ambiguous commands?
If the timer was still running and the music was still playing and the command was “Hey Mycroft Stop”, Mycroft would not know which skill to stop. Relying on what is displayed is not an great option as it may be running on a display-less device or the display can’t be seen for some reason. If it can’t reasonably be determined which skill the command is meant for, Mycroft could come back and say “Stop timer or stop music?” and wait for the response.

About the discussion about using “Thank you Mycroft” to stop a response, it seems inconsistent with how Mycroft interactions work. Unless Mycroft asks for clarification, it is simplest if any/all commands start with “Hey Mycroft…” In the case of a rambling response to a badly stated question, simply “Hey Mycroft, stop”. In the middle of a response, there is little ambiguity as to what needs to be stopped.

1 Like
#18

Thanks for all the awesome feedback everyone!

It’s clear that there’s never going to be 1 set of behaviour that works for everyone. We’re all different and we have different expectations.

As Michael said - in the first round we want to implement good default behaviour. Something that works well for most people and can be manually modified if you choose eg through a setting change. In time the system should learn different users preferences and adapt itself - but this is a much longer term ambition :smiley:

I won’t reply to everything directly but there’s a few things I wanted to drill into a little more…

Stopping things

Some intents are nice and simple to infer - “stop the music” you would expect to… stop the music. It gets less clear when looking at generic phrases like “stop”. If a timer or alarm is actively beeping then we might assume that this rather than the music is what should be stopped. But what happens if we have a Timer displayed on the screen and music playing in the background? Should we ask which thing to stop? Stop the most recently activated process (ie LIFO)? Have a defined priority order - Expired thing > active timer > music > other? or something else?

It is relatively easy to add more back and forth conversation and/or settings to handle situations like this - but the more we can infer these correctly and not need to bother the user with more detail, the better the experience.

As you said in a follow up post, I can see myself saying this when a Timer has expired or even if Mycroft is being a bit long winded and I want to cut off the speech. But if I asked for some music, it started playing, and I said “thank you Mycroft” - the intent is not for the music to stop. Can you think of other examples where “thank you” is or isn’t a termination intent/command?

Undoing actions

This is a really interesting scenario both in terms of what “stop” should do but also around what actions should be “undo-able”? Would love to hear more thoughts or examples of things that are important to undo.

Expired Timer display

Definitely not a nitpick - this is great feedback! We want to hear it all - big and small :slight_smile:

Visual indicator of system state

Currently the LED’s on top of the unit perform this function but the intention is to have this reflected on the screen as well. You can see the start of this if you set your device to grab the “latest” updates via home.mycroft.ai. Are there other states or information that you think could be communicated through this other than “ready”, “actively listening”, or “processing command”?

#19

Stopping: it’s indeed nice if Mycroft could infer what you’re trying to stop, rather than (always) add questions. But note that there are options in between as well:

  1. it could infer, yet you’d still have the option to say “that’s not what I meant” (or some command like that), to nip in in the bud
  2. it could infer, and ask “OK?” (when in doubt) to get explicit confirmation
  3. only if pretty clueless it would indeed have to just ask what to stop

Thank you I think that’ll do wonders if it effectively means “shut up”, but shouldn’t extend beyond that; gets way too subltle = confusing.

Also means that the actual “stop” could be “stop action” (not just stop confirming). If it’s confirming that 10 pizza order, I suppose there might be a lot of text, like total price, when they’ll be there, etc. If that order is somehow a mistake and I want to stop it going through, I do not need further confusion around the commands…

Quick to glance Please don’t bury information in extensive text (hope that simply won’t fit). I’ve been surprised while visiting the USA how much text there is in trafic everywhere! In Europe, we use road signs for that. IMHO that’s much more readable at a glance and less dependent on knowing a language well.

It might be a tad geeky, but a “-X s” (in red?) is also very clear and remains practical for those who use timers all day long, as it is concise and quick to read. Which also means easy to read for kids who don’t read that well yet, anyone who should wear glassess for proper reading but is still at breakfast, folks who don’t speak the language very well, etc.

(Anyone confused a first time, will see what it means anyway by staring at it for litterally an extra second - “Oh, it goes UP again…”)

Processing command Sound like a really good set of three simple indicators, that just didn’t come across yet over this video medium.

1 Like
#20

Good discussion around LIFO/stack of context vs. specifying context. It seems best to me that Mycroft handle both.

My remark is actually mainly about the video content. For completeness, the video could anticipate the variety and demonstrate both. So perhaps after the demo as is, the user then pauses a moment and starts a different song, and starts another timer, and this time at expiration, utters “Mycroft, stop music” and then a little pause and then “Mycroft, stop timer”. And finally, user then asks Mycroft how long the timer ran, to which Mycroft replies “five minutes and twenty seconds.” (Adding the absolute value of the time overage to original duration).

1 Like
#21

Concerning “undo”, I think, actions that cause one of the following should be undo-able:

  • result in immediate or future payments (to protect from financial loss)
  • result in configuration changes (to avoid misconfigs that drive users crazy, e.g., changing the language to something you do not speak → requires a universal keyword for undo)
  • control events for Internet of Things devices (to mitigate damage or other harmful/unwanted results)

These would be my top priorities. The second one is universal for all skills, while the first and second concern specific skill types only (but these are the really useful ones, I guess).

1 Like
#22

It may be helpful to see this as another instance of skill-based intent recognition, giving skills a chance to handle things first.

For example, the timer skill may be able to handle “stop timer {name}” to stop a timer by name. It may also handle just “stop”, but this should trigger a fallback if no timer exists. Similarly, the music skill could handle “stop music” or just “stop” (trigger fallback if no song is playing).

If the skills “fall back” in LIFO order of use, it would produce the expected behavior.
With a song playing, then a timer set and beeping:

  1. “stop” will disengage the current timer
  2. “stop” again will pause the music

The benefit of this approach is you could still say “stop music” to leave the timer alone. Additionally, Mycroft itself may have a meta skill that can handle phrases like “stop everything”.

1 Like
#24

The commentary around the word “stop” actually reminds me of a surprising interaction I had with a proprietary home-assistant device I had while at a friend’s place.

My friend has much of his A/V gear hooked up to his HA device, so we were able to tell it to turn on the projector and start streaming a movie to it. At some point, we put some pizzas in the oven and asked the HA device to set a timer to tell us when they were cooked, and resumed watching the movie.

At some point during the movie something interesting happened, so we wanted to pause the movie to discuss the interesting thing that happened in the movie, but our attempt to pause the movie didn’t work. Puzzled, we tried again; it worked the second time.

After several minutes discussing whatever plot point had taken our interest, we resumed the movie. Some time later, we noticed a burning smell; we jumped up and pulled the pizzas out just in time! They were extra toasted around the edge, but still edible.

I’m sure you can tell what happened: our first attempt to pause the movie actually paused our cooking timer!

In this case, I don’t think his HA device didn’t have a screen, so it could be difficult for the device to choose what we wanted to pause. But even a display might not help; the HA device may not be able to determine what it is we are focused on and therefore what should be paused.

Perhaps an interaction like this would have helped:

  • Me: “pause”
  • HA: “the movie, or the timer?”

I have no idea what you would do if there were multiple times, either with or without a movie or background music… A conundrum to be sure!

PS: if the HA device did choose one thing when there were multiple candidates, perhaps some unambiguous feedback would help; for example, the situation above would have been easy for us to resolve if my friend’s HA device told us “I’ve paused your timer”. I don’t recall there being any feedback at all, or if there was it was drowned out by the movie…

3 Likes
#25

I don’t do CrossFit anymore, but when I did the following would have been great.
An example might be have Mycroft start workout music playlist, start a timer for 3 minutes of say jumping rope with a 5 sec count down and alarm at the end of the 3 min, then 1 minute of rest with 5 sec count down and alarm at end of minute, then 2 minutes of burpees with countdown and alarm, and repeat this a total of five times.
“Mycroft start a workout with music, exercise announcements, countdown timers and alarms.”
“Ok. What music do you want?”
“World’s best workout playlist”
“Ok. What is your first exercise?”
“Jumping rope”
“Ok. How long?”
“3 minutes with 5 second count down at end”
“Got it. Next exercise (and time with 5 sec count down)?”
“Rest (for 1 minute with 5 sec count down)”
“Roger that. Next exercise (and time)?”
“Burpees for 3 minutes with 5 sec countdown”
“Ok. Next exercise?”
“Repeat that 5 times and stop all but let the music play”

That would have been really nice to have.
If Mycroft would have called split times every minute (2 minutes left) and in last minute every 30 sec that would have been really nice.
If Mycroft could be connected to a heart rate monitor it could call out/display your heart rate occasionally.
If you count your reps, Mycroft records your count, and at the end tell you how many you did, and where your heart rate was, and what your heart rate did while you recovered after the work out.
Or maybe there is an app I don’t know about that does all this, and you just say “Mycroft, run world’s best workout app”

If you like your workout, you can name it and save it and not have to recreate it. Share workouts with friends. Work out together at the same time (option for separate playlists) or asynchronisely.
Also Mycroft could save results of workouts if you count reps out loud - first round 30 pushups in 1 min, heart rate X to X during 1 minute rest, 20 burpees in 1 minute, heart rate …, second round: pushups, heart rate, burpees, heart rate, 3 rd round, etc. Look for progress over time, compete with friends.

Could Mycroft sync to multiple headphones and play different playlists on the different headphones so people can have their own tunes?

1 Like
#26

I did not expect the year when asking for the day.

I expected the music to just play, not to receive a confirmation.

When the timer was being set, I did not expect the music to disappear, I expected it to continue with a lowered volume. Like when I’m listening to music on my phone, the sound is just lowered to signal a notification.

For the stop command, I expected a question : “Stop what?” or “The music or the timer?”

1 Like
#27

Thanks for the extra feedback all - it’s all very helpful in determining the behaviour we should expect. This will be used to inform the design of the technical processes sitting behind the scenes.

I’ve just posted another video and would love your feedback on this video too.

Thanks!