Memory leak running latest neon/ovos?

General Discussion
1

My MKII with 2GB is already low on memory from the start.
With the Neon/Ovos I seem to be running out of memory after a couple of days.
I already upped the 1GB swap file to 3GB.
I see mainly the neon-gui.py process eating 90MB/hour
I manually do this at certain intervals:

(date;free -h; sudo ps aux --sort -%mem |head -n 10) |tee -a ~/memusage.txt

And this is what is showing so far after reset yesterday and no wakeword events were used:

Tue May 16 04:06:20 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.6Gi 39Mi 97Mi 291Mi 191Mi
Swap: 3.0Gi 209Mi 2.8Gi

Tue May 16 04:43:23 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.6Gi 30Mi 97Mi 245Mi 136Mi
Swap: 3.0Gi 244Mi 2.8Gi

Tue May 16 05:19:28 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 60Mi 97Mi 177Mi 105Mi
Swap: 3.0Gi 260Mi 2.7Gi

Tue May 16 06:11:52 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 15Mi 97Mi 180Mi 63Mi
Swap: 3.0Gi 296Mi 2.7Gi

Tue May 16 07:11:41 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 18Mi 97Mi 187Mi 73Mi
Swap: 3.0Gi 400Mi 2.6Gi
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neon 583 2.1 16.7 3132108 309256 ? Ssl 15:40 4:30 /home/neon/venv/bin/python /usr/libexec/neon-audio.py
neon 584 4.5 13.1 2760244 241348 ? Ssl 15:40 9:30 /home/neon/venv/bin/python /usr/libexec/neon-gui.py
neon 387 24.3 12.9 2981356 238200 ? Ssl 15:39 51:27 ovos-shell
neon 643 5.9 10.5 2851512 193748 ? Ssl 15:41 12:23 /home/neon/venv/bin/python /usr/libexec/neon-enclosure.py
neon 644 7.3 10.2 3793316 188820 ? Ssl 15:41 15:19 /home/neon/venv/bin/python /usr/libexec/neon-skills.py
root 582 2.5 8.9 1904716 164188 ? Ssl 15:40 5:26 /home/neon/venv/bin/python /usr/libexec/neon-admin-enclosure.py
neon 585 3.4 6.7 3034096 124664 ? Ssl 15:40 7:22 /home/neon/venv/bin/python /usr/libexec/neon-speech.py
neon 546 6.6 5.1 1200372 94992 ? Ssl 15:40 14:02 /home/neon/venv/bin/python /usr/libexec/neon-messagebus.py
neon 669 21.7 4.7 1034260 88372 ? Sl 15:41 45:34 /home/neon/.local/share/precise-engine/precise-engine /home/neon/.local/share/neon/hey-mycroft.pb 2048

Tue May 16 07:55:56 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 25Mi 97Mi 195Mi 89Mi
Swap: 3.0Gi 482Mi 2.5Gi
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neon 583 2.0 15.6 3139276 288560 ? Ssl 15:40 5:16 /home/neon/venv/bin/python /usr/libexec/neon-audio.py
neon 584 4.4 14.9 2780980 274416 ? Ssl 15:40 11:20 /home/neon/venv/bin/python /usr/libexec/neon-gui.py
neon 387 24.1 13.1 2981356 242284 ? Ssl 15:39 61:53 ovos-shell
neon 643 5.8 11.2 2865592 207844 ? Ssl 15:41 14:50 /home/neon/venv/bin/python /usr/libexec/neon-enclosure.py
neon 644 7.2 10.2 3793316 188408 ? Ssl 15:41 18:23 /home/neon/venv/bin/python /usr/libexec/neon-skills.py
root 582 2.5 9.6 1918540 178160 ? Ssl 15:40 6:23 /home/neon/venv/bin/python /usr/libexec/neon-admin-enclosure.py
neon 546 6.5 5.1 1200372 94992 ? Ssl 15:40 16:49 /home/neon/venv/bin/python /usr/libexec/neon-messagebus.py
neon 585 3.4 4.7 3034096 87932 ? Ssl 15:40 8:44 /home/neon/venv/bin/python /usr/libexec/neon-speech.py
neon 669 21.7 3.7 1034260 69584 ? Sl 15:41 55:13 /home/neon/.local/share/precise-engine/precise-engine /home/neon/.local/share/neon/hey-mycroft.pb 2048

Tue May 16 09:38:14 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 13Mi 97Mi 196Mi 77Mi
Swap: 3.0Gi 620Mi 2.4Gi
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neon 584 4.3 18.6 2823220 342988 ? Ssl 15:40 15:33 /home/neon/venv/bin/python /usr/libexec/neon-gui.py
neon 387 24.0 13.2 2981356 243440 ? Ssl 15:39 86:00 ovos-shell
neon 643 5.7 13.0 2897848 239412 ? Ssl 15:41 20:32 /home/neon/venv/bin/python /usr/libexec/neon-enclosure.py
neon 583 1.9 12.3 3156172 227568 ? Ssl 15:40 7:02 /home/neon/venv/bin/python /usr/libexec/neon-audio.py
root 582 2.3 11.4 1951052 210536 ? Ssl 15:40 8:33 /home/neon/venv/bin/python /usr/libexec/neon-admin-enclosure.py
neon 644 7.1 9.5 3793316 175952 ? Ssl 15:41 25:27 /home/neon/venv/bin/python /usr/libexec/neon-skills.py
neon 546 6.4 5.1 1200372 94496 ? Ssl 15:40 23:14 /home/neon/venv/bin/python /usr/libexec/neon-messagebus.py
neon 585 3.3 3.1 3034096 57316 ? Ssl 15:40 11:53 /home/neon/venv/bin/python /usr/libexec/neon-speech.py
neon 669 21.7 2.7 1034260 49752 ? Sl 15:41 77:32 /home/neon/.local/share/precise-engine/precise-engine /home/neon/.local/share/neon/hey-mycroft.pb 2048

Tue May 16 10:13:12 PM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 23Mi 97Mi 193Mi 78Mi
Swap: 3.0Gi 688Mi 2.3Gi
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neon 584 4.3 19.9 2837556 368176 ? Ssl 15:40 17:00 /home/neon/venv/bin/python /usr/libexec/neon-gui.py
neon 643 5.7 13.3 2911672 245004 ? Ssl 15:41 22:29 /home/neon/venv/bin/python /usr/libexec/neon-enclosure.py
neon 387 23.9 13.1 2981356 241752 ? Rsl 15:39 94:14 ovos-shell
neon 583 1.9 12.7 3161548 235260 ? Ssl 15:40 7:38 /home/neon/venv/bin/python /usr/libexec/neon-audio.py
root 582 2.3 12.1 1964620 223560 ? Ssl 15:40 9:18 /home/neon/venv/bin/python /usr/libexec/neon-admin-enclosure.py
neon 644 7.1 9.4 3793316 174680 ? Ssl 15:41 27:53 /home/neon/venv/bin/python /usr/libexec/neon-skills.py
neon 546 6.4 5.1 1200372 94280 ? Ssl 15:40 25:25 /home/neon/venv/bin/python /usr/libexec/neon-messagebus.py
neon 585 3.3 2.0 3034096 37756 ? Ssl 15:40 12:58 /home/neon/venv/bin/python /usr/libexec/neon-speech.py
neon 669 21.7 0.9 1034260 17028 ? Sl 15:41 85:09 /home/neon/.local/share/precise-engine/precise-engine /home/neon/.local/share/neon/hey-mycroft.pb 2048

Wed May 17 03:25:48 AM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 13Mi 96Mi 224Mi 107Mi
Swap: 3.0Gi 1.2Gi 1.8Gi
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neon 584 4.2 30.3 3167284 558536 ? Ssl May16 29:49 /home/neon/venv/bin/python /usr/libexec/neon-gui.py
root 582 2.2 15.5 2062884 286940 ? Ssl May16 15:54 /home/neon/venv/bin/python /usr/libexec/neon-admin-enclosure.py
neon 643 5.6 14.1 3009720 261236 ? Ssl May16 39:58 /home/neon/venv/bin/python /usr/libexec/neon-enclosure.py
neon 644 7.0 8.6 3793316 159188 ? Ssl May16 49:33 /home/neon/venv/bin/python /usr/libexec/neon-skills.py
neon 387 23.7 5.9 2981428 109688 ? Ssl May16 167:51 ovos-shell
neon 583 1.8 5.9 3211980 109528 ? Ssl May16 12:59 /home/neon/venv/bin/python /usr/libexec/neon-audio.py
neon 546 6.3 4.9 1200372 90568 ? Ssl May16 44:57 /home/neon/venv/bin/python /usr/libexec/neon-messagebus.py
neon 585 3.2 0.7 3034096 13384 ? Ssl May16 22:36 /home/neon/venv/bin/python /usr/libexec/neon-speech.py
neon 669 21.7 0.7 1034260 13216 ? Sl May16 153:20 /home/neon/.local/share/precise-engine/precise-engine /home/neon/.local/share/neon/hey-mycroft.pb 20

Hmm this doesn’t read as easy as full lines.
Don’t know how long this info is available: https://termbin.com/udl4h

2

After nearly 4 more hours of sleep :

Wed May 17 03:25:48 AM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 13Mi 96Mi 224Mi 107Mi
Swap: 3.0Gi 1.2Gi 1.8Gi

Wed May 17 07:18:22 AM PDT 2023
total used free shared buff/cache available
Mem: 1.8Gi 1.7Gi 12Mi 104Mi 201Mi 75Mi
Swap: 3.0Gi 1.6Gi 1.4Gi

400 Megabyte more swap in use in nearly 4 hours.

This is unusable after 15 hours of uptime !

I restarted neon service:

time sudo systemctl restart neon

real 0m23.044s
user 0m0.023s
sys 0m0.039s

Look at how much RAM was freed:

           total        used        free      shared  buff/cache   available

Mem: 1.8Gi 300Mi 1.2Gi 52Mi 346Mi 1.5Gi
Swap: 3.0Gi 133Mi 2.9Gi

complete overview:

https://termbin.com/l5h7

$ sudo systemctl status neon
â—Ź neon.service - Neon A.I. Software stack.
Loaded: loaded (/lib/systemd/system/neon.service; enabled; preset: enabled)
Active: active (exited) since Wed 2023-05-17 07:26:08 PDT; 10min ago
Process: 7874 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 7874 (code=exited, status=0/SUCCESS)
CPU: 5ms

May 17 07:26:08 neon systemd[1]: Starting Neon A.I. Software stack…
May 17 07:26:08 neon systemd[1]: Finished Neon A.I. Software stack…

After the first voice command:

(venv) neon@neon:~$ free -h
total used free shared buff/cache available
Mem: 1.8Gi 1.6Gi 38Mi 105Mi 262Mi 161Mi
Swap: 3.0Gi 207Mi 2.8Gi

3

Turns out only the ALPHA versions (pre-release) have an issue.
Thanks to Daniel McKnight @neon-daniel:matrix.org eye to detail this has now been resolved.
Also thanks to the others on the channel who chimed in like @JarbasAl (and others).
Thanks for your patience and the ultra fast finding/solving of the issue!

3 Likes