Mark 1 - Enclosure eyes, screen, and button not working

General Discussion
1

We weren’t going to trouble anyone else with this issue, until it happened to our 2nd of 3 Mark 1 units.

Seemingly out of the blue, the eyes, screen and button of the Mark1 enclosure do not function. However, the mic and speaker still function normally, and the device itself runs just fine (responds to wake words, transcribes, uses skills correctly). Pushing or turning the enclosure button does not result in any audible response, and the eyes and screen are never active.

I tried burning a new image from here to a new SD card, results are the same.

What could be the problem here?

I can post logs, although I don’t yet know which are relevant.

Thanks

2

Hey @zach_neongecko sorry to heart his has happened to two out of 3 units.

We’ve seen this a couple of times, and it usually just means that we need to run an upgrade on the unit itself. If there is a hardware failure with the Micro SD card, this usually manifests as a kernel panic, and the mic and speaker don’t function properly.

Are you able to SSH into the Mark 1?

Let me know if so and I can provide some further instructions - or jump into our Chat;
https://chat.mycroft.ai/community/channels/troubleshooting
and if I’m not around, another Mycrofter will be around to help.

Best, Kathy

3

Hi @KathyReid

To answer your question, yes, I am able to SSH into the Mark1, as well as access it via keyboard + HDMI/DSI display.

Also, I wanted to clarify that my mic and speaker actually are working correctly - it is all of the other enclosure components that seemingly aren’t active.

I didn’t get any response from the Chat to Troubleshooting earlier - it seems there hasn’t been any action there past noon today.

Thanks

4

Thanks @zach_neongecko that information is super helpful.

If the other enclosure components are not working, we may need to upgrade the unit. From the SSH command line, type:

sudo apt-get update && sudo apt-get upgrade

Make a noe of any packages that are flagged for upgrade.

Please let me know how you go - I’m @kathy-mycroft in Chat.

5

I had a similar issue this weekend due to some tinkering. In my version the eyes are on, but not animated, and stuck on yellow. I seem to have it resolved now. Here’s what happened, and how (I think) I got it all working again:

  • I updated my Mark 1 after many months gathering dust… I apologized to him later.
  • I found that Pianobar wasn’t working and:
    • through some testing discovered that the TLS fingerprint validation was failing
    • verified that the key was actually valid like so (stolen from somewhere now lost):

      openssl s_client -connect [PandoraTunerURL]443 < /dev/null 2> /dev/null | openssl x509 -noout -fingerprint | tr -d ‘:’ | cut -d’=’ -f2
      openssl s_client -connect [secondPandoraTunerURL]:443 < /dev/null 2> /dev/null | openssl x509 -noout -fingerprint | tr -d ‘:’ | cut -d’=’ -f2
      - (sorry, I can’t post two links so I removed the pandora tune urls infavor of the below page links)

    • Found the following articles:
      1. https://bbs.nextthing.co/t/a-better-way-of-building-pianobar-for-pandora-radio/8385
      2. https://bbs.nextthing.co/t/solved-pianobar-wont-connect-to-pandora/8092/3
      • I followed the steps in 1 first, then all the non-bold steps in LinuxHippy’s detailed comment on 2(third comment on that page)
      • I also needed to use the apt-file stuff from Kolban’s comment on 2 to grab avfiltergraph.h (with the libavfilter-dev package) in order to ‘make pianobar’
    • THAT DID IT!!! Pianobar now worked, and even better it didn’t seem to adversely affect anything I tried…
  • …Until I next rebooted, apparently. Now:
    • all the voice control and audio functionality still worked great (including pianobar, woot!), except the top button and screen didn’t work at all. Also the eyes were stuck on yellow and not animating. So, not great…
    • I searched around and found this very page and:
      • was able to SSH into the Mark 1
      • considered the ‘sudo apt-get…’ command from KathyReid but was afraid that it may negate my pianobar fix, so:
        • I grabbed all my apt and dpkg logs (I couldn’t find a log that would show the ‘git clone’ details, not sure if it logs), as well as a bunch of others for posterity, and plan to figure out which packages that I updated mattered to Mycroft… sometime… more certainly if it ever matters again.
        • I removed and re-added mycroft-core, like so:

          sudo apt-get remove mycroft-core
          sudo apt-get install mycroft-core

        • …and a reboot later, Mycroft was smiling at me again with the right colored eyes and the button was again working correctly as well. Best of all pianobar still works!

So… It is still an open question whether I needed to go through the hoops for pianobar or if there was a simpler fix, but since everything seems to be right with my Mycroft again, I thought I would share. I’m sure at some point I’ll end up with a completely new image, and hoping this holds until then. :smile:

I also replaced the mycroft-update cronjob with the following monstrosity for better log formatting w/timestamping:
0 4 * * * ( echo -e “\n\n-------------------” ; echo -e “date '+%Y-%m-%d %H:%M:%S'” ; echo -e “-------------------\n\nAPT-GET UPDATE\n--------------\n\n” ; /usr/bin/apt-get update ; echo -e “\nAPT-GET UPGRADE\n---------------\n\n” ; /usr/bin/apt-get install --only-upgrade mycroft-core mimic -y ) >> /var/log/mycroft-update.log

6

Great debugging work, @necroid. We never got to the bottom of the Yellow Eyes problem - it seems to have become much less frequent since the 18.02 release, so we haven’t done a huge amount of digging to diagnose it - your notes are incredibly helpful.

What are your thoughts on us changing the crontab entries on mycroft-core to permanently add more verbose logging?

7

Thanks @KathyReid. I think that sounds like great idea. :smiley:

I wouldn’t use my crontab though, it’s super ugly. I’d do a quick bash script like:

#!/bin/bash

#This script updates the yum inventory and
#upgrades the mycroft-core and mimic packages.

#Log Location

LOG=“/var/log/mycroft-update.log”

#Packages to install

PACKAGES=“mycroft-core mimic”

DATE=date '+%Y-%m-%d %H:%M:%S'

#Log the session date stamp

echo -e “\n-------------------\n$DATE Beginning Mycroft Update\n-------------------\n” >> $LOG

#Update

echo -e “—\n-APT-GET UPDATE\n--------------\n” >> $LOG
apt-get update >> $LOG

#Upgrade

echo -e “\n—\n-APT-GET UPGRADE\n--------------\n” >> $LOG
apt-get install --only-upgrade $PACKAGES -y >> $LOG

and then call it from crontab instead.

On the topic, I like the format of the wifi-setup log timestamp, but the rest of the mycroft-* logs don’t have the date portion.

1 Like