Pi3B 64bit OS implementation

cbxbiker61 · January 5, 2019, 7:33pm

I have Mycroft running on Pi3B with 64 bit OS. Due to the type of processing that Mycroft does I figured that the 64bit implementation would be faster and more responsive. Now I just need to power it down and mount it with the google AIY kit parts in my Adabot robot head. Looks like it will run great.

FYI, the Pi running 64bit OS benchmarks much stronger with sysbench vs the 32bit OS.

baconator · January 5, 2019, 7:38pm

Which distro did you use? And curious how it goes with the aiy bits.

cbxbiker61 · January 5, 2019, 7:46pm

I’m actually using my own Distro that I’ve developed over quite a series of years. I started it just before the 64 bit x86_64 transition was common. The cool thing about having my own distro is that I can implement changes immediately. For example all of my systems are running Python 3.7.2 and kernel 4.20.

I have another box that I tested the AIY kit with, based off the stock images. That one works fine. I’m pretty sure that I won’t have any major hurdles getting the AIY bits working on the 64bit OS.

If I was to recommend a common distro to go with it would be Arch. Arch is philosophically very close to my distro.

KathyReid · January 7, 2019, 7:05am

It would definitely be interesting to see how Mycroft runs on some sort of 64-bit Raspbian-type OS. There’s been a bit of discussion about this on the Raspberry Pi forums at;
https://www.raspberrypi.org/forums/viewtopic.php?t=208314

For now though, I’d be a little hesitant to use something other than a widely-supported Raspberry Pi distro as the base for Picroft - as it is we struggle to try and support all the different audio combinations that can be used with Picroft - implementing a more niche type of OS might generate additional support load.

cbxbiker61 · January 7, 2019, 10:56am

I have mycroft speaking and responding to requests from the command line. From what I see the google voice hat overlay is working properly (kernel 4.14.91). The google voice hat overlay doesn’t work properly with kernel 4.19 and 4.20 yet, which really isn’t an important issue at this point. The audio devices should work as well on 64bit as 32bit. Real-world testing is the only way to know for sure.

My sticking point at this time is that the PortAudio lib is throwing an error, which is preventing the mic from working. Not sure if that is 64 bit specific or not. I’m not too excited on the dependency on PortAudio that comes with pyaudio. In this case you end up going pyaudio->portaudio->alsa->pulseaudio, which seems like an unnecessarily long path to capture audio. I may look at replacing pyaudio with a direct path to pulseaudio, rather than trying to debug the PortAudio problem.

Expression ‘paInvalidSampleRate’ failed in ‘…/Source/src/hostapi/alsa/pa_linux_alsa.c’, line: 2043
Expression ‘PaAlsaStreamComponent_InitialConfigure( &self->capture, inParams, self->primeBuffers, hwParamsCapture, &realSr )’ failed in ‘…/Source/src/hostapi/alsa/pa_linux_alsa.c’, line: 2714
Expression ‘PaAlsaStream_Configure( stream, inputParameters, outputParameters, sampleRate, framesPerBuffer, &inputLatency, &outputLatency, &hostBufferSizeMode )’ failed in ‘…/Source/src/hostapi/alsa/pa_linux_alsa.c’, line: 2838

cbxbiker61 · January 7, 2019, 11:02am

BTW, I would be surprised if Pi 4 doesn’t go 64bit right out of the gate. When that happens Pi 3 and Pi 4 will probably have a common 64bit base distro.

The alternative SBC’s that are running 64bit are making the 32bit Pi’s look pretty anemic. I don’t think the Pi Foundation can ignore that any longer.

cbxbiker61 · January 9, 2019, 9:43am

Got it working…for one thing I took pulseaudio out of the loop. Then I fine-tuned my asound.conf.

On another note, I just got my Khadas Vim2 8 core up and running Linux 4.20.0 with my distro. I’ll use the Khadas Vim2 as my build server for aarch64.

cbxbiker61 · January 31, 2019, 6:21am

Comparing apples to apples…

***** 64bit pi 3b 1.2GHz
~16% cpu usage waiting for command
344M used
5 min load 0.60

***** 32bit pi 3b 1.2GHz
~23% cpu usage waiting for command
267M used
5 min load 0.82

So yes, 64bit does have a performance advantage…and the memory footprint is within the capabilities of the Pi 3A+.

j1nx · January 31, 2019, 7:26am

I have read somewhere that EXT4 r/w is a bit slower on 32bit then it is on 64bit. Can you test / validate that as well?

Above does look promising though!

cbxbiker61 · January 31, 2019, 12:51pm

I can’t really test that easily…I use f2fs which is superior on flash due to the way that it distributes writes. This makes the flash hold up better. I started using f2fs way back when it first came out and it’s been good to me.

Better sd cards have the most impact on performance on these sbc’s. Most recently I’ve started to use 64G Samsung Evo Plus in the gray packaging (the white packaging was slower). Previously I was quite happy with SanDisk Ultra.

cbxbiker61 · January 31, 2019, 12:57pm

I should add that all of the current filesystems use 64bits to store sector locations. So that would imply that they should all operate a bit faster on 64bit archs vs 32bit. I just don’t know that it would be noticeable in real-world scenarios.

j1nx · January 31, 2019, 1:33pm

Just remember where I read what I said;

github.com

home-assistant/hassos/blob/dev/Documentation/boards/raspberrypi.md

# Raspberry PI

Supported Hardware:

| Device | Board | 
|--------|-----------|
| Raspberry Pi A+/B/B+| rpi |
| Raspberry Pi Zero | rpi |
| Raspberry Pi Zero W | rpi0-w |
| Raspberry Pi 2 B | rpi2 |
| Raspberry Pi 3 B/B+ | rpi3 / rpi3-64 |

## Limitation 64bit

The 64bit version is under development by RPi-Team. It work very nice but it could have some impacts. Actual we see that the SDcard access with ext4 are a bit slower than on 32bit.

## Serial console

For access to terminal over serial console, add `console=ttyAMA0,115200` to `cmdline.txt` and `enable_uart=1`, `dtoverlay=pi3-disable-bt` into `config.txt`. GPIO pins are: 6 = GND / 8 = UART TXD / 10 = UART RXD.

This file has been truncated. show original

No clue if it is valid or not, nor how old that statement is…

cbxbiker61 · January 31, 2019, 9:47pm

I just ordered 3 of the Model 3A+. I plan to swap these into my Mycroft boxes. Memory should be fine and the updated WiFi module is more important in this case than the ethernet.

cbxbiker61 · January 31, 2019, 9:56pm

Here is an iozone run with 64G Samsung Evo Plus micro-sd using f2fs on 64bit Pi 3B. The 4k write/random write speeds are phenomenal for micro-sd.

random random kB reclen write rewrite read reread read writ 51200 4 3218 3347 8055 8038 7386 2987 51200 512 5275 11283 22617 22736 22751 8762 51200 16384 6522 10536 23101 23087 23086 11955

cbxbiker61 · January 31, 2019, 10:00pm

Trying to get the formatting right, anyway 3218 random write speed and 2987 random 4k write speed.

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write     read
           51200       4     3218     3347     8055     8038     7386     2987                                                                
           51200     512     5275    11283    22617    22736    22751     8762                                                                
           51200   16384     6522    10536    23101    23087    23086    11955

cbxbiker61 · January 31, 2019, 10:07pm

These are some parameters I tune on my sbc’s with micro-sd that may improve performance.

/etc/tmpfiles.d % cat 99-sbc.conf                                                                                             
w /proc/sys/vm/dirty_writeback_centisecs - - - - 1500
w /proc/sys/vm/swappiness - - - - 1

cbxbiker61 · February 4, 2019, 1:37am

Interesting note.

Linux 4.19.x is now working with the Google voice hat. So I switched to 4.19.19.
I also installed GLibC 2.29.

Now my 5min load is down to 0.50, quite a bit better than the 32bit kernel at 0.82.

j1nx · February 4, 2019, 1:28pm

Interesting to hear that glibc 2.29 brings you so much!? For my MycroftOS project I also use the default Buildroot glibc 2.29.1 now, however do have the change to bump to 2.30.

Not benchedmarked anything as of yet.

Boot times of Buildroot are incredible fast anyway.

Aculimic · February 9, 2019, 7:13pm

Since you are running Mycroft in a nonstadard environment, do you have any interest in harnessing OpenCL? Someone has it running on a Pi (https://hackaday.com/2019/01/24/running-opencl-on-a-raspberry-pi-gpu) and I can’t help but think it will make them speedier.

I suspect getting the 64bit libraries working with Mycroft is a separate skill set from adapting the python, but I have not seen any mention of this development on the forum, and I wanted to make someone with more than “Hello World” python skills aware of the possibility.

cbxbiker61 · February 10, 2019, 3:55am

Thanks for the link.

I’ve been interested in OpenCL in general for quite a while. It’s been underutilized even on those platforms that support it, i.e. Linux/AMD. One of the applications that I had hoped would eventually be open-sourced was an OpenCL HEVC encoder. That hasn’t happened yet as far as I know.

I’ll play around a bit with the code.

Just today I got the RasPi userspace programs to build cleanly on 64bit. vcgencmd seems to be working. My first goal with the 64bit userspace is to get the raspicam working with hardware jpeg encode (mjpgstreamer). Once I have that working I’ll be able to migrate my OctoPrint devices from 32 to 64 bits.