I’m trying to install mycroft-core on my openmediavault server and keep running into a “Failed to set up virtualenv for mycroft, exiting setup.” error when I run dev_setup.sh
I am ssh’d into the OMV server with a user that has a home directory enabled where I am trying follow the Linux installation instructions. I had to edit the script to change all source commands to . since the source command doesn’t seem to work on OMV. Here is the output from dev_setup.sh whenever I try to run it:
$ bash dev_setup.sh
Installing packages...
Installing packages for Debian/Ubuntu/Mint...
Reading package lists... Done
Building dependency tree
Reading state information... Done
autoconf is already the newest version (2.69-10).
automake is already the newest version (1:1.15-6).
bison is already the newest version (2:3.0.4.dfsg-1+b1).
build-essential is already the newest version (12.3).
flac is already the newest version (1.3.2-1).
git is already the newest version (1:2.11.0-3+deb9u7).
libglib2.0-dev is already the newest version (2.50.3-2+deb9u2).
libicu-dev is already the newest version (57.1-6+deb9u4).
jq is already the newest version (1.5+dfsg-1.3).
libfann-dev is already the newest version (2.2.0+ds-3).
libffi-dev is already the newest version (3.2.1-6).
libtool is already the newest version (2.4.6-2).
mpg123 is already the newest version (1.23.8-1+b1).
libssl-dev is already the newest version (1.1.0l-1~deb9u1).
pkg-config is already the newest version (0.29-4+b1).
portaudio19-dev is already the newest version (19.6.0-1).
pulseaudio is already the newest version (10.0-1+deb9u1).
pulseaudio-utils is already the newest version (10.0-1+deb9u1).
python3-setuptools is already the newest version (33.1.1-1).
python3 is already the newest version (3.5.3-1).
python3-dev is already the newest version (3.5.3-1).
screen is already the newest version (4.5.0-6).
swig is already the newest version (3.0.10-1.1).
curl is already the newest version (7.52.1-5+deb9u12).
libjpeg-dev is already the newest version (1:1.5.1-2+deb9u1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1842k 100 1842k 0 0 1539k 0 0:00:01 0:00:01 --:--:-- 1538k
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
Collecting pip==20.0.2
Using cached pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Uninstalling pip-20.0.2:
Successfully uninstalled pip-20.0.2
Successfully installed pip-20.0.2
WARNING: You are using pip version 20.0.2; however, version 20.2.4 is available.
You should consider upgrading via the '${VIRTUALENV_ROOT}/bin/python -m pip install --upgrade pip' command.
Failed to set up virtualenv for mycroft, exiting setup.
I checked the bash script and found the install_venv function seems to be what triggers that error if things don’t go right. However, manually running the python3 -m venv "${VIRTUALENV_ROOT}/" --without-pip command from terminal doesn’t throw any errors if I swap the env variable for the actual location (actually doesn’t seem to do anything, just returns). And pip does exist at the ${VIRTUALENV_ROOT}/bin location. So I’m not sure what is causing the error.
Any ideas what the problem is or what to check next?
Ah, that’s my bad. Sorry for the red herring there. $virtualenv_root does resolve and I used the resolved value when manually running the python3 venv command and checking the folder pip exists in. It was late and I replaced it in the output with the variable name not wanting the username posted but not realizing that what I wrote might be misleading.
I created and ran your test script. Output is right here and I believe it is what is expected:
There, redacted the user name in a more useful manner And I get the same output from the script whether I use the first expression for $TOP or the expression in your refactor.
Thanks, I hadn’t gotten to researching that line yet. I thought it might be some way of storing a bool for whether or not the file exists. My shell skills are on the minimal side.
After checking on the server again, it seems you’re right but I don’t know why it’s not executable. My linux terminal skills are also only rudimentary but this is the output I got when trying to get it working. It seems that the user I’m logged in as owns the file and it is marked executable but I get a Permission denied when attempting to execute it.
$ ls -l ./.venv/bin
total 52
-rw-r--r--+ 1 FoxSly42 users 2183 Nov 8 19:21 activate
-rw-r--r--+ 1 FoxSly42 users 1299 Nov 8 19:21 activate.csh
-rw-r--r--+ 1 FoxSly42 users 2463 Nov 8 19:21 activate.fish
drwxrws---+ 2 FoxSly42 users 4096 Nov 8 21:58 bin
-rwxrwxr-x+ 1 FoxSly42 users 290 Nov 8 17:44 easy_install
-rwxrwxr-x+ 1 FoxSly42 users 290 Nov 8 17:44 easy_install-3.5
drwxrws---+ 2 FoxSly42 users 4096 Nov 8 21:58 include
drwxrws---+ 3 FoxSly42 users 4096 Nov 8 21:58 lib
lrwxrwxrwx 1 FoxSly42 users 3 Nov 8 21:58 lib64 -> lib
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip3
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip3.5
lrwxrwxrwx 1 FoxSly42 users 7 Nov 8 17:44 python -> python3
lrwxrwxrwx 1 FoxSly42 users 16 Nov 8 17:44 python3 -> /usr/bin/python3
-rw-rw----+ 1 FoxSly42 users 130 Nov 8 21:58 pyvenv.cfg
-rwxrwxr-x+ 1 FoxSly42 users 268 Nov 8 17:44 wheel
$ chmod +x ./.venv/bin/pip
$ ls -l ./.venv/bin
total 52
-rw-r--r--+ 1 FoxSly42 users 2183 Nov 8 19:21 activate
-rw-r--r--+ 1 FoxSly42 users 1299 Nov 8 19:21 activate.csh
-rw-r--r--+ 1 FoxSly42 users 2463 Nov 8 19:21 activate.fish
drwxrws---+ 2 FoxSly42 users 4096 Nov 8 21:58 bin
-rwxrwxr-x+ 1 FoxSly42 users 290 Nov 8 17:44 easy_install
-rwxrwxr-x+ 1 FoxSly42 users 290 Nov 8 17:44 easy_install-3.5
drwxrws---+ 2 FoxSly42 users 4096 Nov 8 21:58 include
drwxrws---+ 3 FoxSly42 users 4096 Nov 8 21:58 lib
lrwxrwxrwx 1 FoxSly42 users 3 Nov 8 21:58 lib64 -> lib
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip3
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip3.5
lrwxrwxrwx 1 FoxSly42 users 7 Nov 8 17:44 python -> python3
lrwxrwxrwx 1 FoxSly42 users 16 Nov 8 17:44 python3 -> /usr/bin/python3
-rw-rw----+ 1 FoxSly42 users 130 Nov 8 21:58 pyvenv.cfg
-rwxrwxr-x+ 1 FoxSly42 users 268 Nov 8 17:44 wheel
$ ./.venv/bin/pip --version
-sh: 8: ./.venv/bin/pip: Permission denied
$ whoami
FoxSly42
$ /srv/dev-disk-by-label-TOSHIBAEXT/home/FoxSly42/mycroft-core/.venv/bin/pip
-sh: 11: /srv/dev-disk-by-label-TOSHIBAEXT/home/FoxSly42/mycroft-core/.venv/bin/pip: Permission denied
$ ./.venv/bin/pip install
-sh: 13: ./.venv/bin/pip: Permission denied
Looks like the user owns all files and directories in the tree…
$ ls -lR ./.venv/bin
./.venv/bin:
total 52
-rw-r--r--+ 1 FoxSly42 users 2183 Nov 8 19:21 activate
-rw-r--r--+ 1 FoxSly42 users 1299 Nov 8 19:21 activate.csh
-rw-r--r--+ 1 FoxSly42 users 2463 Nov 8 19:21 activate.fish
drwxrws---+ 2 FoxSly42 users 4096 Nov 8 21:58 bin
-rwxrwxr-x+ 1 FoxSly42 users 290 Nov 8 17:44 easy_install
-rwxrwxr-x+ 1 FoxSly42 users 290 Nov 8 17:44 easy_install-3.5
drwxrws---+ 2 FoxSly42 users 4096 Nov 8 21:58 include
drwxrws---+ 3 FoxSly42 users 4096 Nov 8 21:58 lib
lrwxrwxrwx 1 FoxSly42 users 3 Nov 8 21:58 lib64 -> lib
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip3
-rwxrwxr-x+ 1 FoxSly42 users 281 Nov 8 18:23 pip3.5
lrwxrwxrwx 1 FoxSly42 users 7 Nov 8 17:44 python -> python3
lrwxrwxrwx 1 FoxSly42 users 16 Nov 8 17:44 python3 -> /usr/bin/python3
-rw-rw----+ 1 FoxSly42 users 130 Nov 8 21:58 pyvenv.cfg
-rwxrwxr-x+ 1 FoxSly42 users 268 Nov 8 17:44 wheel
./.venv/bin/bin:
total 16
-rw-r--r--+ 1 FoxSly42 users 2183 Nov 8 21:58 activate
-rw-r--r--+ 1 FoxSly42 users 1299 Nov 8 21:58 activate.csh
-rw-r--r--+ 1 FoxSly42 users 2463 Nov 8 21:58 activate.fish
lrwxrwxrwx 1 FoxSly42 users 7 Nov 8 21:58 python -> python3
lrwxrwxrwx 1 FoxSly42 users 77 Nov 8 21:58 python3 -> /srv/dev-disk-by-label-TOSHIBAEXT/home/FoxSly42/mycroft-core/.venv/bin/python3
./.venv/bin/include:
total 0
./.venv/bin/lib:
total 4
drwxrws---+ 3 FoxSly42 users 4096 Nov 8 21:58 python3.5
./.venv/bin/lib/python3.5:
total 4
drwxrws---+ 2 FoxSly42 users 4096 Nov 8 21:58 site-packages
./.venv/bin/lib/python3.5/site-packages:
total 0
I just realized while typing this that it looks like python3 is a link to /usr/bin/python3 which is definitely owned by the root user and $opt_python in the script was evaluating to python3. Is that possibly the problem?
I’m pretty sure that it would not be a good idea to chown /usr/bin/python3 to the current user. Any ideas how I would go about fixing the link or getting a locally owned install of python for the virtual environment? I thought that was what the setup script already does.
That is not what i meant. An example: my venv/pip is sitting /home/gui/mycroft-core/.venv/bin/pip. If i chown gui to root:root and call /home/gui/mycroft-core/.venv/bin/pip i get the permission denied, too. In those NAS cases it is not unlikely that ```“dev-disk-by-label-TOSHIBAEXT” is not accessible to the user that want to use pip
This is what i would do anyway, since 3.5 is a bad idea (and you should also thinking about raising the system python in the near future)
2 pathways
dev_setup.sh with the argument --python 3.7 (the most easy thing but never used it as such tbh)
pyenv → (modify .bashrc) pyenv install 3.7, and then set pyenv local 3.7 in the base directory (this is before you create the venv, so you have to relink which should look like python3 → /home/###/.pyenv/versions/3.7.9/bin/python3)
Yet this wouldn’t encompass the problem described above (the script is only checking pip, not what’s happening down the drain)
Ah okay. Yes, it appears that is the case. The NAS disk is owned by root. Thanks, I didn’t realize that execution and ownership up the tree worked that way.
$ ls -l /srv
drwxr-xr-x 11 root root 4096 Nov 8 16:55 dev-disk-by-label-TOSHIBAEXT
Is it safe/okay to just chown the NAS folder to the FoxSly42 user? Or would the “right” way to do it be like adding FoxSly42 to the root group to give access?
And thanks for the python 3.7 recommendation. I noticed the deprecation warnings but hadn’t thought through dealing with upgrading. Probably better to get that taken care of sooner rather than later.
Alright, I think I got it solved. Just got the dev_setup.sh script to run and it looks like it successfully installed and built everything.
So I tried chown-ing all the directories up the tree to the current user and I still was getting permission denied errors for pip. I didn’t really like changing ownership on those folders anyways so no complaints reverting all that.
It seems like another part of the problem is that OMV by default mounts NAS or data disks using the noexec option. I first tried changing the /etc/openmediavault/config.xml file to remove the noexec option from the mntent entry for that disk. After doing that I tried both running omv-mkconf fstab while logged in as root (I’m on OMV 4, on OMV 5 the command would be omv-salt deploy run fstab) and restarting the server. Neither of those seemed to work and /proc/mounts still listed the noexec flag for the drive and sharedfolders.
So I took a different approach that seems to work. I ssh’d to the server as the user I’m trying to install mycroft with and created a folder on / (the root directory) and then created a symlink in my user’s home directory to that folder. After that I began the mycroft install directions inside the symlinked folder in my home directory and the dev_setup.sh script ran fine.
I’m thinking the symlink folder option is probably a better solution than making it run on the NAS disk anyways.
Thanks for all the help!
P.S. running the dev_setup.sh from the symlink folder also meant that I didn’t have to change all the source commands in it to . as mentioned in the original post. Script ran just fine as it was.
And I get the same output from the script whether I use the first expression for $TOP or the expression in your refactor.
My shell skills are on the minimal side.