Unable to Install Custom Precise Network

I have a picroft speaker that works well for me but not for my daughters. So we trained a version of “hey mycroft” using the precise tools and generated the .pb file from the .net. Before conversion, we tested it both against the samples, which gave 97% accuracy (and only false negatives), and with a live mic to confirm that the wake word worked for all three of us, which it did, with no issues.

I then used the precise install instructions to modify the user conf file, like so:

  "max_allowed_core_version": 20.8,

  "listener": {
    "wake_word": "hey mycroft"
  },

  "hotwords": {
    "hey mycroft": {
      "module": "precise",
      "local_model_file": "/home/pi/mycroft-precise/hey-computer.pb",
      "sensitivity": 0.3,
      "trigger_level": 7
    }
  }
}

I restarted the mycroft service and it immediately woke without a prompt and then refused to wake despite repeated attempts:

2021-01-23 23:17:05.904 | INFO     |  8359 | mycroft.client.speech.listener:create_wake_word_recognizer:328 | Creating wake word engine
2021-01-23 23:17:05.907 | INFO     |  8359 | mycroft.client.speech.listener:create_wake_word_recognizer:351 | Using hotword entry for hey mycroft
2021-01-23 23:17:05.910 | INFO     |  8359 | mycroft.client.speech.hotword_factory:load_module:403 | Loading "hey mycroft" wake word via precise
2021-01-23 23:17:07.529 | INFO     |  8359 | mycroft.client.speech.listener:create_wakeup_recognizer:365 | creating stand up word engine
2021-01-23 23:17:07.531 | INFO     |  8359 | mycroft.client.speech.hotword_factory:load_module:403 | Loading "wake up" wake word via pocketsphinx
2021-01-23 23:17:07.664 | INFO     |  8359 | __main__:on_ready:175 | Speech client is ready.
2021-01-23 23:17:07.752 | INFO     |  8359 | mycroft.messagebus.client.client:on_open:114 | Connected
2021-01-23 23:17:32.470 | INFO     |  8359 | mycroft.session:get:74 | New Session Start: 6e3ef801-c0fc-4e8c-b791-4ba5dd8bde90
2021-01-23 23:17:32.474 | INFO     |  8359 | __main__:handle_wakeword:67 | Wakeword Detected: hey mycroft
Playing WAVE '/home/pi/mycroft-core/mycroft/res/snd/start_listening.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Stereo
2021-01-23 23:17:32.915 | INFO     |  8359 | __main__:handle_record_begin:37 | Begin Recording...
2021-01-23 23:17:35.941 | INFO     |  8359 | __main__:handle_record_end:45 | End Recording...
2021-01-23 23:17:37.320 | ERROR    |  8359 | mycroft.client.speech.listener:transcribe:239 | list index out of range
2021-01-23 23:17:37.321 | ERROR    |  8359 | mycroft.client.speech.listener:transcribe:240 | Speech Recognition could not understand audio
2021-01-23 23:18:15.268 | INFO     |  8359 | __main__:on_stopping:179 | Speech service is shutting down...
Warning: Failed to load parameters from /home/pi/mycroft-precise/hey-computer.pb.params

hey-computer.pb.params is at present with permissions at that location, right next to hey-computer.pb.

The long and the short of it is that I have a network I’ve trained that works perfectly in testing but not at all when installed. Any assistance would be very helpful.

Hey computer or hey mycroft?

I’m guessing you’re recording, modeling, and testing from a different platform than picroft? If so, try recording some samples from normal distance/usage patterns on picroft and test on the model. The amount of data you use is also somewhat proportional to the relative accuracy of your model.

Remote possibility: There’s a quirk about precise startup that it likes to download the model if it thinks it’s not there. Check to see that the model it’s loading is yours (check hash sums or sizes or?).

You’ll want to add "save_utterances": "true" to your listener config as well (they get dumped into /tmp by default). These you can sift through and add to wake or not wake word data as relevant, this eliminated almost all of my false positives after a week or so.

The network I trained is hey-computer.pb, which I’m loading using the “hey mycroft” hotword. It didn’t work when I named the hotword something different, either. All of the recorded samples were done on the picroft (with the public domain sounds added per the instructions) and the net was trained on the picroft.

Your second suggestion might be it. How do I check that the model is being loaded properly and/or hash? I do keep the custom net files in a different location to avoid collision.

As far the last, I don’t think it’s a model problem, since testing the net directly with precise-test was almost perfect.

Try ls -l and then sha1sum on both files to get a quick idea if they’re the same. I edited the hotword factory file to log the model name, I don’t think it shows up in normal logging otherwise.

I understand how to hash, I’m unclear on where the model is loaded to in the system. Is there documentation for any of this anywhere (other than the precise conf tutorial)? I can’t find anything but if there is that might clear some things up.

You mention this:
"local_model_file": "/home/pi/mycroft-precise/hey-computer.pb",
which would be the first place I’d check. By default precise looks in ~/.mycroft/precise/ for things, though.

So I took a break from this and then went back and tried. I’m still extremely unclear about the checksum suggestion. It’s not that I think the model is being overwritten, I believe the configuration is simply not loading my model. I think the voice log bears this out:

2021-02-05 14:29:01.570 | INFO     |   635 | mycroft.client.speech.listener:create_wake_word_recognizer:328 | Creating wake word engine
2021-02-05 14:29:01.572 | WARNING  |   635 | mycroft.client.speech.listener:create_wake_word_recognizer:341 | Wakeword doesn't have an entry falling backto old listener config
2021-02-05 14:29:01.575 | INFO     |   635 | mycroft.client.speech.hotword_factory:load_module:403 | Loading "hello computer" wake word via precise
2021-02-05 14:29:11.654 | INFO     |   635 | mycroft.client.speech.hotword_factory:load_module:429 | precise is taking too long to load
2021-02-05 14:29:11.656 | INFO     |   635 | mycroft.client.speech.hotword_factory:load_module:403 | Loading "hello computer" wake word via pocketsphinx
2021-02-05 14:29:11.919 | INFO     |   635 | mycroft.client.speech.listener:create_wakeup_recognizer:365 | creating stand up word engine
2021-02-05 14:29:11.922 | INFO     |   635 | mycroft.client.speech.hotword_factory:load_module:403 | Loading "wake up" wake word via pocketsphinx
2021-02-05 14:29:12.000 | INFO     |   635 | __main__:on_ready:175 | Speech client is ready.
2021-02-05 14:29:12.005 | INFO     |   635 | mycroft.messagebus.client.client:on_open:114 | Connected
2021-02-05 14:29:14.917 | WARNING  |   635 | mycroft.client.speech.hotword_factory:initialize:418 | Could not found find model for hello computer on precise.
2021-02-05 14:30:18.706 | INFO     |   635 | mycroft.session:get:74 | New Session Start: 02f20044-721e-4d31-a06b-ed20b661482e
2021-02-05 14:30:18.708 | INFO     |   635 | __main__:handle_wakeword:67 | Wakeword Detected: hello computer
Playing WAVE '/home/pi/mycroft-core/mycroft/res/snd/start_listening.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Stereo
2021-02-05 14:30:19.396 | INFO     |   635 | __main__:handle_record_begin:37 | Begin Recording...
2021-02-05 14:30:29.383 | INFO     |   635 | __main__:handle_record_end:45 | End Recording...
2021-02-05 14:30:29.386 | INFO     |   635 | mycroft.client.speech.mic:listen:744 | Recording utterance
2021-02-05 14:30:32.594 | ERROR    |   635 | mycroft.client.speech.listener:transcribe:239 | list index out of range
2021-02-05 14:30:32.598 | ERROR    |   635 | mycroft.client.speech.listener:transcribe:240 | Speech Recognition could not understand audio
2021-02-05 15:14:11.242 | INFO     |   635 | __main__:on_stopping:179 | Speech service is shutting down...

This makes no sense, because looking at the user conf:

{
  "max_allowed_core_version": 20.8,

  "listener": {
    "device_name": "pulse",
    "sample_rate": 16000,
    "record_wake_words": true,
    "save_utterances": true,
    "wake_word_upload": {
      "disable": true,
      "url":"https://training.mycroft.ai/precise/upload"
    },
    "mute_during_output":true,
    "duck_while_listening": 0.3,
    "phoneme_duration": 120,
    "multiplier": 1.0,
    "energy_ratio": 1.5,
    "wake_word": "hello computer",
    "stand_up_word": "wake up"
    },
  "precise": {
    "dist_url": "https://github.com/MycroftAI/precise-data/raw/dist/{arch}/latest",
    "model_url": "https://raw.githubusercontent.com/MycroftAI/precise-data/models/{wake_word}.tar.gz"
  },
  "hotwords": {
    "hey computer": {
      "module": "precise",
      "lang": "en-us",
      "local_model_file": "/home/pi/mycroft-precise/hey-computer4.pb",
      "sensitivity": 0.7,
      "trigger_level": 2
    },
    "wake up": {
      "module": "pocketsphinx",
      "phonemes": "W EY K . AH P",
      "threshold": 1e-20,
      "lang": "en-us"
    }
  }
}

I can’t see anything that’s wrong, yet it is clearly not using my trained model. This is borne out by actual use, where we have a val_acc of .98 and live testing on the network is perfect, but when I try to load the model, it “feels” like the default model; ie, my children’s voices are more or less ignored.

D’oh. Mixed up the wake word names between the listener and hotword modules. Will see if that fixes it.

1 Like

This:
Could not found find model for hello computer on precise.

seems to be the main issue, did the config change fix it? I don’t think it’s wrong (if the path is right). I would also trim the model_url or adjust it to point at your model.

For good measure check if you have /etc/mycroft/mycroft.conf, as well as ~/.mycroft/mycroft.conf and if they vary.

Fixing the conf fixed it, in that it is now pointing at the correct model. However, there is some disconnect between the listener and the model because despite having 99% accuracy, it is literally impossible to get the wake word to activate. I think I found the issue; there’s an error thrown when it’s loading the model complaining about the params file:

2021-02-06 22:52:23.849 | INFO     |   652 | mycroft.client.speech.hotword_factory:load_module:403 | Loading "hello computer" wake word via precise
Warning: Failed to load parameters from /home/pi/mycroft-precise/hey-computer9.pb.params

Which makes sense, except:

pi@picroft:~/mycroft-precise $ ls -l ./hey-computer9*
-rw-r--r-- 1 pi pi      3 Feb  6 22:07 ./hey-computer9.epoch
-rw-r--r-- 1 pi pi  36352 Feb  6 22:07 ./hey-computer9.net
-rw-r--r-- 1 pi pi    224 Feb  6 21:58 ./hey-computer9.net.params
-rw-r--r-- 1 pi pi  26640 Feb  6 22:12 ./hey-computer9.pb
-rw-r--r-- 1 pi pi    224 Feb  6 22:12 ./hey-computer9.pb.params
-rw-r--r-- 1 pi pi 479577 Feb  6 22:12 ./hey-computer9.pbtxt

So it’s loading the model but for some reason can’t get to the params file in the same location. So I figure I’ll change the user conf and make a copy of the model and params file in the same spot that the default model lives, to rule out some kind of access issue:

"local_model_file": "/home/pi/.mycroft/precise/hey-computer9.pb"

pi@picroft:~ $ ls -l ~/.mycroft/precise/hey-computer9.pb*
-rw-r--r-- 1 pi pi 26640 Feb  6 23:16 /home/pi/.mycroft/precise/hey-computer9.pb
-rw-r--r-- 1 pi pi   224 Feb  6 23:16 /home/pi/.mycroft/precise/hey-computer9.pb.params

Ok, so restarting should have it look for both files in a place I know it’s comfortable looking for them.

2021-02-06 23:18:04.914 | INFO     |   682 | mycroft.client.speech.hotword_factory:load_module:403 | Loading "hello computer" wake word via precise
Warning: Failed to load parameters from /home/pi/mycroft-precise/hey-computer9.pb.params

What.

The warning for the params file isn’t an issue.

The percentage you keep referring to is based only on the data you fed it, which may or may not be related to real world accuracy depending where the data came from.