How to pass python BytesIO() stream to Precise Runner?

General Discussion
1

Hello,

I’ve been testing Precise as a wake-word engine for my companion robot project (GitHub - ScottMonaghan/robud: Loveable Accessible Autonomous Robot).

If I create a stream object using pyaudio.open, I’m able to specify my input device, pass it into the precise runner in python and it works great!

The problem is, I can’t allow the precise python script to take full control of my audio card because it’s used by several other processes.

The way that I handle this is that I publish the raw input bytestream with the Moquitto MQTT messaging system.

With other tools I’m able to subscribe to that stream and then use it the audio however I wish, but with precise I’ve been having trouble.

I’m able to subscribe and write the stream to a BytesIO object, and I can specify the BytesIO object as the stream when initializing my runner.

But when I start the runner, I get an error stating there are 0 frames.

Can anyone offer any advice on how to achieve this?

Here are the related python snippets:

def on_message_audio_input_data(client:mqtt.Client, userdata, message):
        #print("audio recieived")
        audio_buffer:BytesIO = userdata["audio_buffer"]
        audio_buffer.write(message.payload)

    #Create audio buffer
    audio_buffer = BytesIO()

    client_userdata = {"audio_buffer":audio_buffer}
    mqtt_client = mqtt.Client(client_id=MQTT_CLIENT_NAME,userdata=client_userdata)
    mqtt_client.connect(MQTT_BROKER_ADDRESS)
    logger.info('MQTT Client Connected')
    mqtt_client.subscribe(TOPIC_AUDIO_INPUT_DATA)
    mqtt_client.message_callback_add(TOPIC_AUDIO_INPUT_DATA, on_message_audio_input_data)
    logger.info('Subscribed to ' + TOPIC_AUDIO_INPUT_DATA)
    #stream.start_stream()
    logger.info('Waiting for messages...')
    mqtt_client.loop_start()

    #Create audio buffer
    #audio_buffer = BytesIO()

    engine = PreciseEngine('/home/robud/Downloads/precise-engine/precise-engine', '/home/robud/src/precise-data/hey-mycroft.pb')

    runner = PreciseRunner(engine=engine, stream=audio_buffer, on_activation=lambda: print('activated!'))
    #runner.start()

    # Sleep forever
    from time import sleep
    while True:
        if audio_buffer.getbuffer().nbytes > CHUNK and not(runner.running): #2 bytes for each 16bit frame
            logger.info('runner started')
            runner.start()
        sleep(0.1)
2

Is this the correct place to ask this question?

3

probably better on github.

4

For anyone interested, I was able to get this working.

By reviewing the code in runner.py (mycroft-precise/runner.py at dev · MycroftAI/mycroft-precise · GitHub) I realized I could bypass the PreciseRunner object altogether.

Here is the snippet of code that shows how it all comes together. Let me know if anyone has any questions:

    client_userdata = {}
    mqtt_client = mqtt.Client(client_id=MQTT_CLIENT_NAME,userdata=client_userdata)
 
    def on_message_audio_input_data(client:mqtt.Client, userdata, message):
        if engine.proc: #this is the Popen external process run for the precise engine. It takes a bit to fully open so this prevents errors before it fully starts.
            chunk = message.payload
            handle_predictions(chunk)

    def handle_predictions(chunk):
        """Check Precise process output"""
        prob = engine.get_prediction(chunk)
        if detector.update(prob):
           mqtt_client.publish(topic=TOPIC_WAKEWORD_DETECTED, payload="True",qos=2)
           logger.info("Wakeword Detected!")

    #initialize mqtt client
    mqtt_client.connect(MQTT_BROKER_ADDRESS)
    logger.info('MQTT Client Connected')
    mqtt_client.subscribe(TOPIC_AUDIO_INPUT_DATA)
    mqtt_client.message_callback_add(TOPIC_AUDIO_INPUT_DATA, on_message_audio_input_data)   
    logger.info('Subscribed to ' + TOPIC_AUDIO_INPUT_DATA)
    logger.info('Waiting for messages...')
    
    #initialize precise engine
    engine = PreciseEngine('/home/robud/Downloads/precise-engine/precise-engine', '/home/robud/src/precise-data/hey-mycroft.pb')
    detector = TriggerDetector(CHUNK, SENSITIVITY, TRIGGER_LEVEL)
    engine.start()

    mqtt_client.loop_forever()
2 Likes
5

Next step, train my own wakeword!

1 Like