For Those Having Trouble Training a New Wake Word

General Discussion
#12

Thanks for your reply. I’ll try re-training with fewer not-wake-word samples. Heck, I’ve come this far, why not?

98% of the not-wake-word samples are from the google speech commands, so thankfully I didn’t waste too much time there.

The training & test data shape was recommended in @sparkyvision 's mycroft-precise-tips you linked to. Note where the author claims ~51k not wake words and 300 wake words (they also recommend against using sox). @baconator 's data shape sounds similar, see localcroft/Precise.md at master · el-tocino/localcroft · GitHub . (dang, suddenly I’m hungry for bacon) .

Hopefully I’m using the phrase “data shape” correctly. When I say that I’m referring to the balance–the number of wake-word and not-wake-word samples.

What you are recommending (roughly the same number of wake-word and not-wake-word samples) is way different so I’d love to understand why. Unfortunately I’m pretty much lost when you mention kw / not_kw, labels, class, binary, spectra graphs, cross entropy, over-fit, under-fit. I get bits and pieces of what you wrote, but this just really isn’t my field. I guess I’ll need to read up on Tensorflow as you recommend.

The precise-test output I included was after 150 epochs. I tried stopping after 6 just as an experiment because acc and val_acc were hitting 1.000 and staying there. I also tried 300 epochs and a few different invocations of precise-train. I haven’t been able to train a model that works.

#13

Here’s yet another try without google speech commands samples. Everything is still coming back as true negatives.

Training:

$ precise-train custom-hey-mycroft.net data/ -e 150
Data: <TrainData wake_words=421 not_wake_words=885 test_wake_words=102 test_not_wake_words=217>
Loading wake-word...

       
Loading not-wake-word...

       
Loading wake-word...

       
Loading not-wake-word...

       
Inputs shape: (620, 29, 13)
Outputs shape: (620, 1)
Test inputs shape: (217, 29, 13)
Test outputs shape: (217, 1)
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
net (GRU)                    (None, 20)                2040      
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 21        
=================================================================
Total params: 2,061
Trainable params: 2,061
Non-trainable params: 0
_________________________________________________________________
Train on 620 samples, validate on 217 samples
Epoch 1/150

620/620 [==============================] - 0s 370us/step - loss: 0.5040 - acc: 0.6597 - val_loss: 0.2961 - val_acc: 0.8479
Epoch 2/150

620/620 [==============================] - 0s 38us/step - loss: 0.3600 - acc: 0.7726 - val_loss: 0.2368 - val_acc: 0.9171
Epoch 3/150

620/620 [==============================] - 0s 38us/step - loss: 0.3130 - acc: 0.8226 - val_loss: 0.2005 - val_acc: 0.9447
Epoch 4/150

620/620 [==============================] - 0s 40us/step - loss: 0.2709 - acc: 0.8516 - val_loss: 0.1739 - val_acc: 0.9631
Epoch 5/150

620/620 [==============================] - 0s 39us/step - loss: 0.2492 - acc: 0.8710 - val_loss: 0.1538 - val_acc: 0.9677
Epoch 6/150

620/620 [==============================] - 0s 39us/step - loss: 0.2329 - acc: 0.8774 - val_loss: 0.1382 - val_acc: 0.9770
Epoch 7/150

620/620 [==============================] - 0s 39us/step - loss: 0.2070 - acc: 0.9129 - val_loss: 0.1252 - val_acc: 0.9770
Epoch 8/150

620/620 [==============================] - 0s 39us/step - loss: 0.1939 - acc: 0.9161 - val_loss: 0.1139 - val_acc: 0.9816
Epoch 9/150

620/620 [==============================] - 0s 39us/step - loss: 0.1862 - acc: 0.9113 - val_loss: 0.1043 - val_acc: 0.9816
Epoch 10/150

620/620 [==============================] - 0s 39us/step - loss: 0.1660 - acc: 0.9177 - val_loss: 0.0967 - val_acc: 0.9816
Epoch 11/150

620/620 [==============================] - 0s 40us/step - loss: 0.1610 - acc: 0.9419 - val_loss: 0.0894 - val_acc: 0.9816
Epoch 12/150

620/620 [==============================] - 0s 39us/step - loss: 0.1785 - acc: 0.9016 - val_loss: 0.0829 - val_acc: 0.9816
Epoch 13/150

620/620 [==============================] - 0s 39us/step - loss: 0.1425 - acc: 0.9387 - val_loss: 0.0776 - val_acc: 0.9816
Epoch 14/150

620/620 [==============================] - 0s 40us/step - loss: 0.1483 - acc: 0.9323 - val_loss: 0.0728 - val_acc: 0.9862
Epoch 15/150

620/620 [==============================] - 0s 39us/step - loss: 0.1506 - acc: 0.9258 - val_loss: 0.0680 - val_acc: 0.9862
Epoch 16/150

620/620 [==============================] - 0s 39us/step - loss: 0.1386 - acc: 0.9419 - val_loss: 0.0636 - val_acc: 0.9862
Epoch 17/150

620/620 [==============================] - 0s 41us/step - loss: 0.1133 - acc: 0.9613 - val_loss: 0.0591 - val_acc: 0.9862
Epoch 18/150

620/620 [==============================] - 0s 39us/step - loss: 0.1186 - acc: 0.9468 - val_loss: 0.0558 - val_acc: 0.9908
Epoch 19/150

620/620 [==============================] - 0s 39us/step - loss: 0.1226 - acc: 0.9484 - val_loss: 0.0527 - val_acc: 0.9908
Epoch 20/150

620/620 [==============================] - 0s 39us/step - loss: 0.1223 - acc: 0.9500 - val_loss: 0.0500 - val_acc: 0.9908
Epoch 21/150

620/620 [==============================] - 0s 40us/step - loss: 0.1102 - acc: 0.9532 - val_loss: 0.0473 - val_acc: 0.9908
Epoch 22/150

620/620 [==============================] - 0s 40us/step - loss: 0.1043 - acc: 0.9613 - val_loss: 0.0450 - val_acc: 0.9908
Epoch 23/150

620/620 [==============================] - 0s 39us/step - loss: 0.1005 - acc: 0.9548 - val_loss: 0.0427 - val_acc: 0.9908
Epoch 24/150

620/620 [==============================] - 0s 40us/step - loss: 0.1054 - acc: 0.9500 - val_loss: 0.0407 - val_acc: 0.9908
Epoch 25/150

620/620 [==============================] - 0s 39us/step - loss: 0.1088 - acc: 0.9468 - val_loss: 0.0387 - val_acc: 0.9908
Epoch 26/150

620/620 [==============================] - 0s 40us/step - loss: 0.0952 - acc: 0.9484 - val_loss: 0.0370 - val_acc: 0.9908
Epoch 27/150

620/620 [==============================] - 0s 42us/step - loss: 0.0843 - acc: 0.9645 - val_loss: 0.0354 - val_acc: 0.9908
Epoch 28/150

620/620 [==============================] - 0s 39us/step - loss: 0.0851 - acc: 0.9710 - val_loss: 0.0338 - val_acc: 0.9908
Epoch 29/150

620/620 [==============================] - 0s 40us/step - loss: 0.0824 - acc: 0.9613 - val_loss: 0.0320 - val_acc: 0.9908
Epoch 30/150

620/620 [==============================] - 0s 39us/step - loss: 0.0855 - acc: 0.9694 - val_loss: 0.0305 - val_acc: 0.9908
Epoch 31/150

620/620 [==============================] - 0s 41us/step - loss: 0.0826 - acc: 0.9613 - val_loss: 0.0291 - val_acc: 0.9908
Epoch 32/150

620/620 [==============================] - 0s 39us/step - loss: 0.0721 - acc: 0.9710 - val_loss: 0.0276 - val_acc: 0.9908
Epoch 33/150

620/620 [==============================] - 0s 39us/step - loss: 0.0832 - acc: 0.9661 - val_loss: 0.0264 - val_acc: 0.9908
Epoch 34/150

620/620 [==============================] - 0s 40us/step - loss: 0.0659 - acc: 0.9726 - val_loss: 0.0254 - val_acc: 0.9908
Epoch 35/150

620/620 [==============================] - 0s 39us/step - loss: 0.0657 - acc: 0.9726 - val_loss: 0.0242 - val_acc: 0.9954
Epoch 36/150

620/620 [==============================] - 0s 39us/step - loss: 0.0677 - acc: 0.9677 - val_loss: 0.0232 - val_acc: 0.9954
Epoch 37/150

620/620 [==============================] - 0s 40us/step - loss: 0.0584 - acc: 0.9839 - val_loss: 0.0222 - val_acc: 0.9954
Epoch 38/150

620/620 [==============================] - 0s 39us/step - loss: 0.0631 - acc: 0.9758 - val_loss: 0.0213 - val_acc: 0.9954
Epoch 39/150

620/620 [==============================] - 0s 39us/step - loss: 0.0576 - acc: 0.9790 - val_loss: 0.0202 - val_acc: 0.9954
Epoch 40/150

620/620 [==============================] - 0s 40us/step - loss: 0.0578 - acc: 0.9726 - val_loss: 0.0193 - val_acc: 0.9954
Epoch 41/150

620/620 [==============================] - 0s 41us/step - loss: 0.0598 - acc: 0.9774 - val_loss: 0.0185 - val_acc: 0.9954
Epoch 42/150

620/620 [==============================] - 0s 39us/step - loss: 0.0505 - acc: 0.9806 - val_loss: 0.0177 - val_acc: 0.9954
Epoch 43/150

620/620 [==============================] - 0s 39us/step - loss: 0.0590 - acc: 0.9823 - val_loss: 0.0169 - val_acc: 1.0000
Epoch 44/150

620/620 [==============================] - 0s 41us/step - loss: 0.0618 - acc: 0.9758 - val_loss: 0.0162 - val_acc: 1.0000
Epoch 45/150

620/620 [==============================] - 0s 39us/step - loss: 0.0630 - acc: 0.9694 - val_loss: 0.0156 - val_acc: 1.0000
Epoch 46/150

620/620 [==============================] - 0s 39us/step - loss: 0.0482 - acc: 0.9839 - val_loss: 0.0148 - val_acc: 1.0000
Epoch 47/150

620/620 [==============================] - 0s 40us/step - loss: 0.0384 - acc: 0.9871 - val_loss: 0.0142 - val_acc: 1.0000
Epoch 48/150

620/620 [==============================] - 0s 40us/step - loss: 0.0474 - acc: 0.9790 - val_loss: 0.0138 - val_acc: 1.0000
Epoch 49/150

620/620 [==============================] - 0s 39us/step - loss: 0.0421 - acc: 0.9887 - val_loss: 0.0132 - val_acc: 1.0000
Epoch 50/150

620/620 [==============================] - 0s 40us/step - loss: 0.0442 - acc: 0.9855 - val_loss: 0.0125 - val_acc: 1.0000
Epoch 51/150

620/620 [==============================] - 0s 40us/step - loss: 0.0490 - acc: 0.9855 - val_loss: 0.0119 - val_acc: 1.0000
Epoch 52/150

620/620 [==============================] - 0s 39us/step - loss: 0.0402 - acc: 0.9887 - val_loss: 0.0114 - val_acc: 1.0000
Epoch 53/150

620/620 [==============================] - 0s 41us/step - loss: 0.0415 - acc: 0.9871 - val_loss: 0.0109 - val_acc: 1.0000
Epoch 54/150

620/620 [==============================] - 0s 39us/step - loss: 0.0301 - acc: 0.9903 - val_loss: 0.0103 - val_acc: 1.0000
Epoch 55/150

620/620 [==============================] - 0s 39us/step - loss: 0.0373 - acc: 0.9855 - val_loss: 0.0099 - val_acc: 1.0000
Epoch 56/150

620/620 [==============================] - 0s 40us/step - loss: 0.0406 - acc: 0.9823 - val_loss: 0.0096 - val_acc: 1.0000
Epoch 57/150

620/620 [==============================] - 0s 40us/step - loss: 0.0379 - acc: 0.9887 - val_loss: 0.0092 - val_acc: 1.0000
Epoch 58/150

620/620 [==============================] - 0s 39us/step - loss: 0.0359 - acc: 0.9903 - val_loss: 0.0088 - val_acc: 1.0000
Epoch 59/150

620/620 [==============================] - 0s 39us/step - loss: 0.0341 - acc: 0.9919 - val_loss: 0.0083 - val_acc: 1.0000
Epoch 60/150

620/620 [==============================] - 0s 41us/step - loss: 0.0354 - acc: 0.9935 - val_loss: 0.0078 - val_acc: 1.0000
Epoch 61/150

620/620 [==============================] - 0s 39us/step - loss: 0.0296 - acc: 0.9887 - val_loss: 0.0076 - val_acc: 1.0000
Epoch 62/150

620/620 [==============================] - 0s 39us/step - loss: 0.0350 - acc: 0.9823 - val_loss: 0.0072 - val_acc: 1.0000
Epoch 63/150

620/620 [==============================] - 0s 40us/step - loss: 0.0320 - acc: 0.9855 - val_loss: 0.0069 - val_acc: 1.0000
Epoch 64/150

620/620 [==============================] - 0s 41us/step - loss: 0.0315 - acc: 0.9903 - val_loss: 0.0066 - val_acc: 1.0000
Epoch 65/150

620/620 [==============================] - 0s 40us/step - loss: 0.0278 - acc: 0.9903 - val_loss: 0.0063 - val_acc: 1.0000
Epoch 66/150

620/620 [==============================] - 0s 41us/step - loss: 0.0265 - acc: 0.9919 - val_loss: 0.0059 - val_acc: 1.0000
Epoch 67/150

620/620 [==============================] - 0s 40us/step - loss: 0.0293 - acc: 0.9903 - val_loss: 0.0056 - val_acc: 1.0000
Epoch 68/150

620/620 [==============================] - 0s 40us/step - loss: 0.0226 - acc: 0.9935 - val_loss: 0.0054 - val_acc: 1.0000
Epoch 69/150

620/620 [==============================] - 0s 40us/step - loss: 0.0201 - acc: 0.9935 - val_loss: 0.0051 - val_acc: 1.0000
Epoch 70/150

620/620 [==============================] - 0s 40us/step - loss: 0.0212 - acc: 0.9887 - val_loss: 0.0050 - val_acc: 1.0000
Epoch 71/150

620/620 [==============================] - 0s 39us/step - loss: 0.0248 - acc: 0.9919 - val_loss: 0.0047 - val_acc: 1.0000
Epoch 72/150

620/620 [==============================] - 0s 40us/step - loss: 0.0225 - acc: 0.9952 - val_loss: 0.0044 - val_acc: 1.0000
Epoch 73/150

620/620 [==============================] - 0s 41us/step - loss: 0.0157 - acc: 0.9935 - val_loss: 0.0043 - val_acc: 1.0000
Epoch 74/150

620/620 [==============================] - 0s 40us/step - loss: 0.0187 - acc: 0.9919 - val_loss: 0.0041 - val_acc: 1.0000
Epoch 75/150

620/620 [==============================] - 0s 39us/step - loss: 0.0161 - acc: 0.9968 - val_loss: 0.0040 - val_acc: 1.0000
Epoch 76/150

620/620 [==============================] - 0s 40us/step - loss: 0.0249 - acc: 0.9903 - val_loss: 0.0038 - val_acc: 1.0000
Epoch 77/150

620/620 [==============================] - 0s 40us/step - loss: 0.0140 - acc: 0.9984 - val_loss: 0.0036 - val_acc: 1.0000
Epoch 78/150

620/620 [==============================] - 0s 40us/step - loss: 0.0208 - acc: 0.9935 - val_loss: 0.0035 - val_acc: 1.0000
Epoch 79/150

620/620 [==============================] - 0s 41us/step - loss: 0.0171 - acc: 0.9952 - val_loss: 0.0033 - val_acc: 1.0000
Epoch 80/150

620/620 [==============================] - 0s 40us/step - loss: 0.0177 - acc: 0.9935 - val_loss: 0.0031 - val_acc: 1.0000
Epoch 81/150

620/620 [==============================] - 0s 39us/step - loss: 0.0149 - acc: 0.9984 - val_loss: 0.0030 - val_acc: 1.0000
Epoch 82/150

620/620 [==============================] - 0s 39us/step - loss: 0.0132 - acc: 0.9952 - val_loss: 0.0029 - val_acc: 1.0000
Epoch 83/150

620/620 [==============================] - 0s 40us/step - loss: 0.0115 - acc: 1.0000 - val_loss: 0.0028 - val_acc: 1.0000
Epoch 84/150

620/620 [==============================] - 0s 39us/step - loss: 0.0101 - acc: 0.9984 - val_loss: 0.0026 - val_acc: 1.0000
Epoch 85/150

620/620 [==============================] - 0s 40us/step - loss: 0.0105 - acc: 1.0000 - val_loss: 0.0025 - val_acc: 1.0000
Epoch 86/150

620/620 [==============================] - 0s 43us/step - loss: 0.0149 - acc: 0.9984 - val_loss: 0.0024 - val_acc: 1.0000
Epoch 87/150

620/620 [==============================] - 0s 40us/step - loss: 0.0137 - acc: 0.9952 - val_loss: 0.0023 - val_acc: 1.0000
Epoch 88/150

620/620 [==============================] - 0s 40us/step - loss: 0.0080 - acc: 0.9984 - val_loss: 0.0022 - val_acc: 1.0000
Epoch 89/150

620/620 [==============================] - 0s 41us/step - loss: 0.0131 - acc: 0.9968 - val_loss: 0.0021 - val_acc: 1.0000
Epoch 90/150

620/620 [==============================] - 0s 40us/step - loss: 0.0105 - acc: 0.9968 - val_loss: 0.0020 - val_acc: 1.0000
Epoch 91/150

620/620 [==============================] - 0s 40us/step - loss: 0.0107 - acc: 0.9952 - val_loss: 0.0020 - val_acc: 1.0000
Epoch 92/150

620/620 [==============================] - 0s 40us/step - loss: 0.0097 - acc: 1.0000 - val_loss: 0.0019 - val_acc: 1.0000
Epoch 93/150

620/620 [==============================] - 0s 39us/step - loss: 0.0057 - acc: 1.0000 - val_loss: 0.0018 - val_acc: 1.0000
Epoch 94/150

620/620 [==============================] - 0s 39us/step - loss: 0.0069 - acc: 0.9984 - val_loss: 0.0017 - val_acc: 1.0000
Epoch 95/150

620/620 [==============================] - 0s 39us/step - loss: 0.0065 - acc: 1.0000 - val_loss: 0.0016 - val_acc: 1.0000
Epoch 96/150

620/620 [==============================] - 0s 40us/step - loss: 0.0081 - acc: 0.9984 - val_loss: 0.0016 - val_acc: 1.0000
Epoch 97/150

620/620 [==============================] - 0s 39us/step - loss: 0.0098 - acc: 1.0000 - val_loss: 0.0015 - val_acc: 1.0000
Epoch 98/150

620/620 [==============================] - 0s 39us/step - loss: 0.0055 - acc: 1.0000 - val_loss: 0.0014 - val_acc: 1.0000
Epoch 99/150

620/620 [==============================] - 0s 41us/step - loss: 0.0072 - acc: 1.0000 - val_loss: 0.0013 - val_acc: 1.0000
Epoch 100/150

620/620 [==============================] - 0s 39us/step - loss: 0.0054 - acc: 0.9984 - val_loss: 0.0013 - val_acc: 1.0000
Epoch 101/150

620/620 [==============================] - 0s 39us/step - loss: 0.0086 - acc: 1.0000 - val_loss: 0.0012 - val_acc: 1.0000
Epoch 102/150

620/620 [==============================] - 0s 40us/step - loss: 0.0078 - acc: 0.9984 - val_loss: 0.0011 - val_acc: 1.0000
Epoch 103/150

620/620 [==============================] - 0s 40us/step - loss: 0.0058 - acc: 1.0000 - val_loss: 0.0010 - val_acc: 1.0000
Epoch 104/150

620/620 [==============================] - 0s 40us/step - loss: 0.0039 - acc: 0.9968 - val_loss: 9.8235e-04 - val_acc: 1.0000
Epoch 105/150

620/620 [==============================] - 0s 40us/step - loss: 0.0039 - acc: 1.0000 - val_loss: 9.3719e-04 - val_acc: 1.0000
Epoch 106/150

620/620 [==============================] - 0s 40us/step - loss: 0.0030 - acc: 1.0000 - val_loss: 8.9144e-04 - val_acc: 1.0000
Epoch 107/150

620/620 [==============================] - 0s 40us/step - loss: 0.0044 - acc: 0.9984 - val_loss: 8.3908e-04 - val_acc: 1.0000
Epoch 108/150

620/620 [==============================] - 0s 39us/step - loss: 0.0030 - acc: 1.0000 - val_loss: 7.9770e-04 - val_acc: 1.0000
Epoch 109/150

620/620 [==============================] - 0s 40us/step - loss: 0.0036 - acc: 1.0000 - val_loss: 7.4814e-04 - val_acc: 1.0000
Epoch 110/150

620/620 [==============================] - 0s 39us/step - loss: 0.0055 - acc: 0.9984 - val_loss: 6.9113e-04 - val_acc: 1.0000
Epoch 111/150

620/620 [==============================] - 0s 40us/step - loss: 0.0024 - acc: 1.0000 - val_loss: 6.4932e-04 - val_acc: 1.0000
Epoch 112/150

620/620 [==============================] - 0s 40us/step - loss: 0.0035 - acc: 1.0000 - val_loss: 6.0726e-04 - val_acc: 1.0000
Epoch 113/150

620/620 [==============================] - 0s 40us/step - loss: 0.0042 - acc: 0.9984 - val_loss: 5.8828e-04 - val_acc: 1.0000
Epoch 114/150

620/620 [==============================] - 0s 39us/step - loss: 0.0049 - acc: 1.0000 - val_loss: 5.3590e-04 - val_acc: 1.0000
Epoch 115/150

620/620 [==============================] - 0s 39us/step - loss: 0.0063 - acc: 0.9968 - val_loss: 5.1206e-04 - val_acc: 1.0000
Epoch 116/150

620/620 [==============================] - 0s 39us/step - loss: 0.0037 - acc: 1.0000 - val_loss: 4.6828e-04 - val_acc: 1.0000
Epoch 117/150

620/620 [==============================] - 0s 39us/step - loss: 0.0022 - acc: 1.0000 - val_loss: 4.3992e-04 - val_acc: 1.0000
Epoch 118/150

620/620 [==============================] - 0s 40us/step - loss: 0.0022 - acc: 1.0000 - val_loss: 4.0873e-04 - val_acc: 1.0000
Epoch 119/150

620/620 [==============================] - 0s 40us/step - loss: 0.0026 - acc: 0.9984 - val_loss: 3.5682e-04 - val_acc: 1.0000
Epoch 120/150

620/620 [==============================] - 0s 40us/step - loss: 0.0023 - acc: 1.0000 - val_loss: 3.1697e-04 - val_acc: 1.0000
Epoch 121/150

620/620 [==============================] - 0s 38us/step - loss: 0.0011 - acc: 1.0000 - val_loss: 3.0181e-04 - val_acc: 1.0000
Epoch 122/150

620/620 [==============================] - 0s 38us/step - loss: 0.0015 - acc: 1.0000 - val_loss: 2.8702e-04 - val_acc: 1.0000
Epoch 123/150

620/620 [==============================] - 0s 39us/step - loss: 0.0037 - acc: 0.9984 - val_loss: 2.6929e-04 - val_acc: 1.0000
Epoch 124/150

620/620 [==============================] - 0s 38us/step - loss: 0.0022 - acc: 1.0000 - val_loss: 2.3655e-04 - val_acc: 1.0000
Epoch 125/150

620/620 [==============================] - 0s 40us/step - loss: 0.0035 - acc: 0.9984 - val_loss: 2.2102e-04 - val_acc: 1.0000
Epoch 126/150

620/620 [==============================] - 0s 39us/step - loss: 0.0030 - acc: 1.0000 - val_loss: 1.9317e-04 - val_acc: 1.0000
Epoch 127/150

620/620 [==============================] - 0s 39us/step - loss: 4.2609e-04 - acc: 1.0000 - val_loss: 1.8849e-04 - val_acc: 1.0000
Epoch 128/150

620/620 [==============================] - 0s 40us/step - loss: 0.0020 - acc: 1.0000 - val_loss: 1.7967e-04 - val_acc: 1.0000
Epoch 129/150

620/620 [==============================] - 0s 39us/step - loss: 0.0019 - acc: 1.0000 - val_loss: 1.7138e-04 - val_acc: 1.0000
Epoch 130/150

620/620 [==============================] - 0s 40us/step - loss: 0.0017 - acc: 1.0000 - val_loss: 1.6300e-04 - val_acc: 1.0000
Epoch 131/150

620/620 [==============================] - 0s 40us/step - loss: 0.0015 - acc: 1.0000 - val_loss: 1.5270e-04 - val_acc: 1.0000
Epoch 132/150

620/620 [==============================] - 0s 40us/step - loss: 0.0013 - acc: 1.0000 - val_loss: 1.4148e-04 - val_acc: 1.0000
Epoch 133/150

620/620 [==============================] - 0s 39us/step - loss: 9.3766e-04 - acc: 1.0000 - val_loss: 1.3763e-04 - val_acc: 1.0000
Epoch 134/150

620/620 [==============================] - 0s 39us/step - loss: 3.7757e-04 - acc: 1.0000 - val_loss: 1.3148e-04 - val_acc: 1.0000
Epoch 135/150

620/620 [==============================] - 0s 40us/step - loss: 0.0014 - acc: 1.0000 - val_loss: 1.2069e-04 - val_acc: 1.0000
Epoch 136/150

620/620 [==============================] - 0s 40us/step - loss: 9.2301e-04 - acc: 1.0000 - val_loss: 1.1077e-04 - val_acc: 1.0000
Epoch 137/150

620/620 [==============================] - 0s 38us/step - loss: 9.0810e-04 - acc: 1.0000 - val_loss: 1.0398e-04 - val_acc: 1.0000
Epoch 138/150

620/620 [==============================] - 0s 39us/step - loss: 0.0014 - acc: 1.0000 - val_loss: 8.3460e-05 - val_acc: 1.0000
Epoch 139/150

620/620 [==============================] - 0s 40us/step - loss: 0.0012 - acc: 1.0000 - val_loss: 7.9932e-05 - val_acc: 1.0000
Epoch 140/150

620/620 [==============================] - 0s 40us/step - loss: 0.0019 - acc: 1.0000 - val_loss: 7.8661e-05 - val_acc: 1.0000
Epoch 141/150

620/620 [==============================] - 0s 38us/step - loss: 0.0017 - acc: 1.0000 - val_loss: 7.4887e-05 - val_acc: 1.0000
Epoch 142/150

620/620 [==============================] - 0s 39us/step - loss: 0.0011 - acc: 1.0000 - val_loss: 7.3465e-05 - val_acc: 1.0000
Epoch 143/150

620/620 [==============================] - 0s 39us/step - loss: 1.4882e-04 - acc: 1.0000 - val_loss: 7.2583e-05 - val_acc: 1.0000
Epoch 144/150

620/620 [==============================] - 0s 39us/step - loss: 5.4213e-04 - acc: 1.0000 - val_loss: 7.0473e-05 - val_acc: 1.0000
Epoch 145/150

620/620 [==============================] - 0s 39us/step - loss: 0.0025 - acc: 0.9984 - val_loss: 7.2666e-05 - val_acc: 1.0000
Epoch 146/150

620/620 [==============================] - 0s 40us/step - loss: 0.0020 - acc: 1.0000 - val_loss: 7.1411e-05 - val_acc: 1.0000
Epoch 147/150

620/620 [==============================] - 0s 40us/step - loss: 0.0020 - acc: 1.0000 - val_loss: 6.6285e-05 - val_acc: 1.0000
Epoch 148/150

620/620 [==============================] - 0s 40us/step - loss: 0.0019 - acc: 1.0000 - val_loss: 6.3273e-05 - val_acc: 1.0000
Epoch 149/150

620/620 [==============================] - 0s 40us/step - loss: 4.8341e-04 - acc: 1.0000 - val_loss: 6.1387e-05 - val_acc: 1.0000
Epoch 150/150

620/620 [==============================] - 0s 41us/step - loss: 0.0015 - acc: 1.0000 - val_loss: 5.5593e-05 - val_acc: 1.0000

Testing:

$ precise-test custom-hey-mycroft.net data/
Loading wake-word...

       
Loading not-wake-word...

       
Data: <TrainData wake_words=421 not_wake_words=885 test_wake_words=102 test_not_wake_words=217>
=== False Positives ===


=== False Negatives ===


=== Counts ===
False Positives: 0
True Negatives: 217
False Negatives: 0
True Positives: 0


=== Summary ===
217 out of 217
100.00%

0.00% false positives
0.00% false negatives
#14

I would be more inclined to use sox to augment your KW and make more than the other way.

If you google you can cope with imbalance Classification on imbalanced data  |  TensorFlow Core but don’t think Precise employs such mechanisms.

Only reason I am saying is because splitting notkw into noise (non voice kw) & notkw (voice kw) is a simple and easy why by just adding the noise ‘label’/‘class’ to the model.
What I started saying even with that I have always struggled still with overfitting KW and 2x sounds like of the start and end of your KW can further balance but have much more cross entropy so forcing the model to work much harder in training and end with a far more accurate model of use.

I would download Multilingual Spoken Words | MLCommons for your language for words and have a few of as many words you can also just picking words with a similar syllable count and length as words much longer or shorter are not really that much use.

Dunno must be Precise or your setup but if you want to get really to grips with KWS then tensorflow is by Google and the publish a repo specifically dedicated to KWS.

You could learn much there but purely down to its dataset manner Precise is always going to be oddly named.

PS what is your KW as have you played anyback to see exactly what you have?

#15

Are you certain that your files are in the exact right format? Precise-train is extremely particular about the format, samples, mono files, etc. I’ve had issues where it would ignore huge chunks of files because they were in stereo.

The fact that it’s completely failling points to something other than the auditory quality of your data, and to something more fundamental.

Edit: additionally, you’re getting extremely high acc values very quickly, which smells wrong to me.

1 Like
#17

Assuming KW means “keyword”, the thing I’m trying to detect. In Mycroft land seems like they use the term “wake word”.

I’m trying to train a model to recognize “hey mycroft” – that’s my wake word. The included/default Precise model for “hey mycroft” only detects my voice (white male) and fails to activate for many other people that try (other genders, ages, etc).

And I assume KWS is “keyword spotting”? Sigh, this is super overwhelming. I’m just a hobbyist with plenty else to do.

Have I spot-checked my data to listen to it to check that it sounds as I’d expect? Yes.

#18

I think so. The file command says every one of my wav files are RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 16000 Hz, and ffprobe says all contain Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, 1 channels, s16, 256 kb/s.

The fact that it’s completely failling points to something other than the auditory quality of your data, and to something more fundamental.

Agreed.

Edit: additionally, you’re getting extremely high acc values very quickly, which smells wrong to me.

Agreed.

#19

Dunno as its strange but ‘Hey Mycroft’ if Mycroft actually released that as a dataset it would make your life easier as you would have a datum and also you could just add to that than having to start from scratch.

Apols if its all been a bit much as my additions are merely curiosity to why Precise training sets itself up to be less than it could by applying a binary class model.
So ignore all that and sure someone can help.

PS install audacity and have a look at what you have in comparison to your not-kw, something is not right though but you know that.

#20

Are they .riff files? I believe Precise expects the extension .wav. (I realize riff is just a container format for a variety of data.)

I’ve never seen riff come up in format discussions before, so I’m sort of singling that out as sus, as the kiddos say.

You’re welcome to upload some of your data somewhere and I can try to look at it when I get a chance.

Edit: I’m just a hobbyist too, don’t worry. If I can get it, you can get it. There’s just something goofy going on here.

#21

That’s a good idea, I’ll give that a shot!

install audacity and have a look at what you have in comparison to your not-kw, something is not right though but you know that

Yeah, I did the recordings in audacity in the first place (following sparkyvision’s handy guide) and tried to clean them up as much as possible before training. I also trimmed silence at the beginnings and ends of the samples I made.

So I tried precise-test hey-mycroft.pb data/ to test the default/included/built-in wake word model against my test data set. This looks better, and matches my experience with the default model missing some wake word activations (at least, when the voice sounds like mine).

I can’t train though, I guess precise-train can’t work on hey-mycroft.pb?

$ precise-train hey-mycroft.pb data/
Using TensorFlow backend.
Loading from hey-mycroft.pb...
Warning: Unknown model type,  hey-mycroft.pb
Traceback (most recent call last):
  File "/home/user/mycroft-precise/.venv/bin/precise-train", line 33, in <module>
    sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-train')())
  File "/home/user/mycroft-precise/precise/scripts/base_script.py", line 43, in run_main
    script = cls(args)
  File "/home/user/mycroft-precise/precise/scripts/train.py", line 87, in __init__
    self.model = create_model(args.model, params)
  File "/home/user/mycroft-precise/precise/model.py", line 70, in create_model
    model = load_precise_model(model_name)
  File "/home/user/mycroft-precise/precise/model.py", line 54, in load_precise_model
    return load_keras().models.load_model(model_name)
  File "/home/user/mycroft-precise/.venv/lib/python3.7/site-packages/keras/models.py", line 237, in load_model
    with h5py.File(filepath, mode='r') as f:
  File "/home/user/mycroft-precise/.venv/lib/python3.7/site-packages/h5py/_hl/files.py", line 408, in __init__
    swmr=swmr)
  File "/home/user/mycroft-precise/.venv/lib/python3.7/site-packages/h5py/_hl/files.py", line 173, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 88, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

Is there an easy way to convert this .pb file to a .net file that works with precise-train? I searched around and tried a few methods and none worked for me.

#22

All the sample files have a .wav extension.

I’ve never seen riff come up in format discussions before, so I’m sort of singling that out as sus, as the kiddos say.

I used ffmpeg in the same Ubuntu 18.04 VM I’ve been using for training and testing. :man_shrugging:

You’re welcome to upload some of your data somewhere and I can try to look at it when I get a chance.

Oh sure, awesome! Ok, I’ll message you a link.

Edit: I’m just a hobbyist too, don’t worry. If I can get it, you can get it. There’s just something goofy going on here.

Thanks, I needed this!

#23

You can try these hey-marvin 10k KW
https://drive.google.com/file/d/1v8xfbj7bILcHn8KkTSs6G-twd3glnsbo/view?usp=sharing

hey-marvin2 100k KW
https://drive.google.com/file/d/1lm-wKZAYIGKQxtm2t6Vkrlv935ccJJfP/view?usp=sharing

Might not be the KW you want but gives you a datum to get used to training.

notkw-10k
https://drive.google.com/file/d/1V1rMgZKTUItZbzat_ZfDArg493kCkL0B/view?usp=sharing

notkw 100k
https://drive.google.com/file/d/1rxdzAS1t49tYvv8GkdwOzvamhdu3XMIC/view?usp=sharing

util-scripts
https://drive.google.com/file/d/18DMNdrwBUXw3lgDysoarfgw94rublGGx/view?usp=sharing

https://storage.googleapis.com/public-datasets-mswc/audio/en.tar.gz

#24

For starters you should read the precise tips from eltocino, one of the first guides that has been used and validated by the community over the years

recently we got these cool repos i havent tried yet, but i know of people having great results with these

and

#25

What are these? What are you suggesting? Sorry, I’m totally lost.

#26

Thanks @JarbasAl . I’ve read el tocino’s guide and I have sunk a bunch of time into my training data already. I guess I’ll check those out if I just can’t get training to work with my existing data.

#27

Ready made KW & !KW 10k of each, then same sets but 100k of each for good measure.
Formatted just need training the 1st or 2nd pair is your choice.
All wavs from MLcommons, if the don’t train and do as last time then its your setup not your dataset.
If they do train then its your dataset and you should get an idea of how a reasonable set should train.
They are not Big Data good but as good a ready made binary dataset, minus noise that you will find.

#28

I used these. I put 80% in wake-word and 20% in test/wake-word.

I used these and did a similar 80/20 split between not-wake-word and test/not-wake-word.

Worked. Perfectly. Dang! I guess the problem is with my data. Thank you! Back to the drawing board.

$ precise-test hey-marvin.net hey-marvin/ 2>/dev/null 
Loading wake-word...
Loading not-wake-word...
Data: <TrainData wake_words=8000 not_wake_words=9108 test_wake_words=2000 test_not_wake_words=2277>
=== False Positives ===
hey-marvin/test/not-wake-word/2819015076embroideredcommon_voice_en_18751486.wav
hey-marvin/test/not-wake-word/2827423258theobaldcommon_voice_en_18984515.wav
hey-marvin/test/not-wake-word/2606411025indianapoliscommon_voice_en_19116959.wav

=== False Negatives ===
hey-marvin/test/wake-word/c913d758-e6d2-4a35-b930-e0ebd11656f1cats.wav
hey-marvin/test/wake-word/d367a887-994e-449d-987e-22b36a5099dbcatstb.wav

=== Counts ===
False Positives: 3
True Negatives: 2274
False Negatives: 2
True Positives: 1998


=== Summary ===
4272 out of 4277
99.88%

0.13% false positives
0.10% false negatives
1 Like
#29

Found one problem: the wav files must all be in the top-level directories (wake-word, not-wake-word, test/wake-word, test/not-wake-word) during training. I must have missed that in one of the guides!

This seems like a way better starting point for my training / data cleanup:

$ precise-test custom-hey-mycroft.net custom-hey-mycroft/ 2>/dev/null 
Loading wake-word...
Loading not-wake-word...
Data: <TrainData wake_words=421 not_wake_words=885 test_wake_words=102 test_not_wake_words=217>
=== False Positives ===
custom-hey-mycroft/test/not-wake-word/room-noise-03-46.wav
custom-hey-mycroft/test/not-wake-word/a-01_S-25.wav
custom-hey-mycroft/test/not-wake-word/e-01_S-4.wav
custom-hey-mycroft/test/not-wake-word/room-noise-03-20.wav
custom-hey-mycroft/test/not-wake-word/m-01_S-12.wav
custom-hey-mycroft/test/not-wake-word/m-01_S-47.wav
custom-hey-mycroft/test/not-wake-word/room-noise-03-48.wav

=== False Negatives ===
custom-hey-mycroft/test/wake-word/a-02_S-11.wav
custom-hey-mycroft/test/wake-word/a-01_S.wav
custom-hey-mycroft/test/wake-word/d-02_S-23.wav
custom-hey-mycroft/test/wake-word/a-02_S-10.wav
custom-hey-mycroft/test/wake-word/a-02_S-36.wav
custom-hey-mycroft/test/wake-word/m-01_S-37.wav
custom-hey-mycroft/test/wake-word/a-02_S-26.wav
custom-hey-mycroft/test/wake-word/a-01_S-11.wav
custom-hey-mycroft/test/wake-word/m-02_S-10.wav
custom-hey-mycroft/test/wake-word/a-01_S-54.wav
custom-hey-mycroft/test/wake-word/m-02_S-4.wav
custom-hey-mycroft/test/wake-word/m-02_S-12.wav
custom-hey-mycroft/test/wake-word/a-01_S-4.wav

=== Counts ===
False Positives: 7
True Negatives: 210
False Negatives: 13
True Positives: 89


=== Summary ===
299 out of 319
93.73%

3.23% false positives
12.75% false negatives

precise-listen doesn’t work yet (it doesn’t seem to recognize when I say the wake word).

Update: adding all the speech_commands_v0.01 helped a bit… I am at “99.9%” with fewer false positives and fewer false negatives. precise-listen works sometimes! I’m a little suspicious of this USB mic through a VM setup, so I’m going to try this fresh model on my actual picroft.

#30

You prob need to turn on Precise-collect so you can play back the KW it received which is a bit hard when not receiving any I guess.
Maybe it does have options to give max amplitude and avg of incoming wav, dunno as not a precise fan and don’t use it.

PS stop using audacity and manual editing and pip install sox and check Welcome to pysox’s documentation! — pysox 1.4.2 documentation as your latest results are not good.
Maybe being forced to do it programatically making sure length, format, sr and levels are right and not full of distortion, may highlight possible problems more?

You can always connect to your vm and arecord -D plughw:1 -V mono -r 16000 -f S16_LE -c 1 /dev/null or whatever device index is your mic and the -V will display a VU meter on the cli. It doesn’t record anything but is just a level check.

PS VM or container?

Why Mycroft don’t have a high quality preformatted dataset available which hey-marvin sufficed is a mystery to me but at least it served its purpose.
Maybe start again but switch and make a hey-marvin by adding more samples from family members.

I did a rough and ready recording ‘boutique’ a while back GitHub - StuartIanNaylor/Dataset-builder: KWS dataset builder for Google-streaming-kws
That prompts on screen and the code there for augmentation and stuff will give you a load of tips.

Also just the text sentences to grab !kw from are as concise as you can get as they are ‘phonetic pangrams’ of nonsense sentences that have just about every phone & allophone in a single sentence.

#31

Not good how?

VM. I followed GitHub - sparky-vision/mycroft-precise-tips: mycroft-precise-tips , which recommends audacity over sox.

Seriously? Argggghhhh.

#32 
=== Summary ===
299 out of 319
93.73%

3.23% false positives
12.75% false negatives

Prob me being a KWS snob but for me the above results are atrocious, even on the dataset I posted I would still be drilling down and cleaning when 99.88%.
The KW where the only remnants a harddrive mishap of formatting the wrong one and losing loads of work as would of had a much cleaner !kw than just grabbing a selection fro MLCommons for you, but hey. Likely on my own models with my own dataset I would get 100% which actually means very little as your feeding what its just been trained on and its should be near 100% as it should of heard those already as in a real life environment with conditions and input it hasn’t been trained on things will just get worse.

Still guess it doesn’t matter as with a binary model such as Precise when you ran Hey-Marvin KW & !KW it was just voice samples and the cross entropy was quite high so the input KW had to be quite close to the KW label.
Because any single label has no memory of any individual elements just a graph of its input adding !voice will lower the cross entropy and move further away from KW so making it less accurate.
Its the catch-22 of a binary model such as this and trying to place so much variance in a single label and so little variance in another.

So if you are happy with that go with the flow and just find out what signal precise is getting from your mic and if that is good enough for you its good enough for you.
Sparky recommends recording via audacity but to be honest when you hand manipulating that qty of files you are just bound to get something wrong and that is where sox comes in for post processing and augmentation.
I am still kicking myself for losing all my scripts as its somehow more painful to dev them a 2nd time even though can remember the gist.
I am making a model of my own in the next couple of days and will be starting with those scripts again so will share when I have them done as with a sulk been dodging it.