Installing and Enabling pyDE1

Overview

To enhance security, the pyDE1 executables run as a dedicated user, one with limited privilege. When possible, only read access is granted to the files. None of the executables or configuration files should be writable by pyde1. In this guide, they are set to root ownership.

Warning

Accessing the database as any user other than pyde1 may cause files to be written by that user, preventing access by pyde1 [1]

There are sh scripts provided that should make the process relatively straightforward. As they are often run as root with sudo, read them and convince yourself that they are going to do what you expect, before running them blindly.

Unless specifically noted, all scripts are to be run with root privilege using sudo.

Walk-Through

The scripts have been broken down into relatively small operations. This allows them to be easily re-run should an error occur.

The scripts are not distributed in the pip package. They are present in the pyDE1 git repo, in the install/ directory. The repo can be cloned, or individual files downloaded. Make sure that the _config file is in the same directory as the shell scripts. Generally no changes need to be made to the _config file.

# Copyright © 2021 Jeff Kletsky. All Rights Reserved.
#
# License for this software, part of the pyDE1 package, is granted under
# GNU General Public License v3.0 only
# SPDX-License-Identifier: GPL-3.0-only

# sourced by install scripts

PYDE1_USER=pyde1
PYDE1_GROUP="${PYDE1_USER}"
VENV_PATH="/home/${PYDE1_USER}/venv/pyde1"

10-create-user.sh

This script will create the pyde1 user if it does not exist and give it access to the bluetooth group.

#!/usr/bin/sh -e

# Copyright © 2021 Jeff Kletsky. All Rights Reserved.
#
# License for this software, part of the pyDE1 package, is granted under
# GNU General Public License v3.0 only
# SPDX-License-Identifier: GPL-3.0-only

. "$(dirname $0)"/_config

if [ -z "$SUDO_USER" ] ; then
  >&2 echo "Script must be run with sudo"
elif [ "$SUDO_UID" =  0 ] ; then
  >&2 echo "Script must be run with sudo by a normal user"
fi

# Create the pyde1 user if they do not yet exist.

if getent passwd "$PYDE1_USER" ; then
  echo "User $PYDE1_USER already exists"
else
  echo "Creating user $PYDE1_USER"
  adduser --system --group "$PYDE1_USER"
fi
usermod -a -G bluetooth "$PYDE1_USER"
id "$PYDE1_USER"

20-create-dirs.sh

This script will create the following directories and set their ownership and permissions:

/var/log/pyde1
/var/lib/pyde1

#!/usr/bin/sh -e

# Copyright © 2021, 2022 Jeff Kletsky. All Rights Reserved.
#
# License for this software, part of the pyDE1 package, is granted under
# GNU General Public License v3.0 only
# SPDX-License-Identifier: GPL-3.0-only

. "$(dirname $0)"/_config

echo "Creating target directories"

mkdir -p /var/log/pyde1
chown $PYDE1_USER /var/log/pyde1

mkdir -p /var/lib/pyde1
chown $PYDE1_USER /var/lib/pyde1

ls -ld /var/log/pyde1
ls -ld /var/lib/pyde1

30-populate-venv

This creates a root-owned, Python virtual environment (“venv”). It then updates pip and setuptools and adds the pyDE1 package and its dependencies to the venv.

Note

If you installed a non-default version of Python, such as 3.9 or 3.10 on an install of Raspberry Pi OS based on “Buster”, you will need to explicitly reference that version when creating the venv.

python -m venv $VENV_PATH would need to be edited to explicitly to refer to your chosen version. References after . $VENV_PATH/bin/activate should not need modification, as that sets python to refer to the one in the venv.

#!/usr/bin/sh -e

# Copyright © 2021 Jeff Kletsky. All Rights Reserved.
#
# License for this software, part of the pyDE1 package, is granted under
# GNU General Public License v3.0 only
# SPDX-License-Identifier: GPL-3.0-only

. "$(dirname $0)"/_config

echo "Creating Python venv at $VENV_PATH"

if ! dpkg --get-selections | egrep '^python3-venv\s+install$' ; then
  apt install python3-venv
fi

mkdir -p $VENV_PATH

python -m venv $VENV_PATH

. $VENV_PATH/bin/activate

pip install -U pip
pip install -U setuptools
pip install pyDE1
pip list

# "Where is pyDE1?"
python -c \
  'import importlib.resources ; print(importlib.resources.files("pyDE1"))'

40-config-files.sh

This copies the config files from the location where pip installed them in the venv and into /usr/local/etc/pyde1. It will make a timestamped backup of any file that would be overwritten.

Note

Some of the configuration files may contain sensitive credentials, such as MQTT and Visualizer usernames and passwords. These files are set to root:pyde1 ownership with no other read access.

It also copies the pyde1.service and pyde1-visualizer.service files, similarly making backups. These files are edited in place to adjust for the specifics of the local install from the previous steps. The editor (sed) backs up the original version with a .bak suffix.

Rather than run disconnect-btid.sh directly from the install, it is copied to /usr/local/bin/pyde1-disconnect-btid.sh. This script is run by pyde1.service to help clean up any “stale” Bluetooth connections related to a prior run that may have terminated ungracefully.

#!/usr/bin/sh -e

# Copyright © 2021, 2023 Jeff Kletsky. All Rights Reserved.
#
# License for this software, part of the pyDE1 package, is granted under
# GNU General Public License v3.0 only
# SPDX-License-Identifier: GPL-3.0-only

. "$(dirname $0)"/_config

mkdir -p /usr/local/bin/pyde1
mkdir -p /usr/local/etc/pyde1

CP_BACKUP="cp -v --backup --suffix=$(date +'.%Y%m%d_%H%M')"

$CP_BACKUP ${PYDE1_ROOT}/services/config/* /usr/local/etc/pyde1/

# As the config files may contain credentials,
# make them unreadable to anyone but root and pyde1

chown root:${PYDE1_GROUP} /usr/local/etc/pyde1/*
chmod 640 /usr/local/etc/pyde1/*

$CP_BACKUP ${PYDE1_ROOT}/services/unit-files/* /usr/local/etc/pyde1/
chown root:root /usr/local/etc/pyde1/*.service
chmod 644 /usr/local/etc/pyde1/*.service

for f in /usr/local/etc/pyde1/*.service ; do
  # This is rather fragile. TODO: Consider a template
  sed -i'.bak'  \
    -e "s|^User=.*|User=${PYDE1_USER}|" \
    -e "s|^Group=.*|Group=${PYDE1_GROUP}|" \
    -e "s|/home/pyde1/venv/pyde1|${VENV_PATH}"
    $f
done

ls -l /usr/local/etc/*

Adjust Config Files to Suit

Examine the various config files /usr/local/etc/pyde1/*.conf and edit as needed.

Changes that are commonly needed include:

In pyde1.conf

mqtt:
- USERNAME
- PASSWORD
de1:
- LINE_FREQUENCY

In pyde1-visualizer.conf

mqtt:
- USERNAME
- PASSWORD
visualizer:
- USERNAME
- PASSWORD

After completing the edits, ensure that they are readable by the pyde1 group and not by anyone else, other than root.

ls -l *.conf
-rw-r----- 1 root pyde1 3555 Nov  3 18:18 pyde1.conf
-rw-r----- 1 root pyde1 1789 Nov  3 18:18 pyde1-replay.conf
-rw-r----- 1 root pyde1 2308 Nov  3 18:18 pyde1-visualizer.conf

If needed, the ownership and permissions can be corrected with

sudo chown root:pyde1 *.conf
sudo chmod 640 *.conf

50-enable-services.sh

This links the service definitions in /usr/local/etc/pyde1 for the pyde1.service and the pyde1-visualizer.service to where systemd (the “startup manager” for Debian) knows about them, enables the services, and restarts them. Unless they are explicitly disabled, they will start on every boot.

Further information on service management can be found with

man systemctl
man journalctl

#!/usr/bin/sh -e

# Copyright © 2021 Jeff Kletsky. All Rights Reserved.
#
# License for this software, part of the pyDE1 package, is granted under
# GNU General Public License v3.0 only
# SPDX-License-Identifier: GPL-3.0-only

for service in /usr/local/etc/pyde1/pyde1.service \
           /usr/local/etc/pyde1/pyde1-visualizer.service ; do

  systemctl link -f $service
  systemctl daemon-reload
  service_name=$(basename $service)
  systemctl enable $service_name
  systemctl restart $service_name

done

Log Rotation

The standard log-rotation utility on Debian is logrotate with configuration in /etc/logrotate.d/

One configuration that rotates daily, compresses, and retains 60 days’ of logs is

/var/log/pyde1/pyde1.log {
    daily
    missingok
    rotate 60
    compress
    delaycompress
    notifempty
    create
}

/var/log/pyde1/visualizer.log {
    daily
    missingok
    rotate 60
    compress
    delaycompress
    notifempty
    create
}

Both the mosquitto and nginx packages install self-named config into /etc/logrotate.d/