Docker

Synology ships a custom Linux distribution that does not come with a full-fledged package manager. It does, on the other hand, have a Container Manager, which is a Docker interface. So the first adaptation is to use Docker instead of podman, which means managing permissions.

sudo synogroup --add docker
sudo chown :docker /var/run/docker.sock

sudo synouser --add git none "git user" 0 noreply@nodomain none
sudo passwd -ld git 

git:x:1000:65536:git user:/var/services/homes/git:/bin/sh

SSH Passthrough

Note above that we do not change the shell, which is a departure from the DataLad instructions. Synology appears to have custom PAM modules that prevent the use of arbitrary programs as shells, as opposed to the more standard pam_shells (8).

Therefore we need to use AuthorizedKeysCommand in sshd_config (5) to do all of the work for us.

/usr/local/bin/forgejo-keys

#!/bin/sh

PROXY=/usr/local/bin/docker
CONTAINER=forgejo
KEYS_COMMAND="/usr/local/bin/gitea keys -c /etc/gitea/app.ini"

$PROXY exec $CONTAINER $KEYS_COMMAND $@ \
    | sed -e 's@/@'"${PROXY} exec -i -e SSH_ORIGINAL_COMMAND $CONTAINER /@"

This script passes arguments to gitea keys inside the container, which returns a command that can be run inside the container, for example:

command="/usr/local/bin/gitea --config=/etc/gitea/app.ini serv key-1",<ssh-options> ssh-ed25519 <pubkey>

The sed script finds the first / and inserts a docker exec prefix, for example:

command="/usr/local/bin/docker exec -i -e SSH_ORGINAL_COMMAND forgejo /usr/local/bin/gitea --config=/etc/gitea/app.ini serv key-1",<ssh-options> ssh-ed25519 <pubkey>

To ensure that this is usable by sshd, set the permissions correctly:

sudo chown root:root /usr/local/bin/forgejo-keys
sudo chmod 755 /usr/local/bin/forgejo-keys

Finally, update sshd_config:

/etc/ssh/sshd_config

...

Match User git
    AuthorizedKeysCommandUser git
    AuthorizedKeysCommand /usr/local/bin/forgejo-keys -e git -u %u -t %t -k %k

Reverse proxy

The only remaining thing is to set up Synology’s reverse proxy through their Control Panel:

A little bit more configuration is needed to get a LetsEncrypt certificate working. If I flesh out this post into a full how-to, I will include that as well.

References

• Rebuilding When a File’s Checksum Changes (which almost worked…)

Validity

This was written in January 2023, superseding previous posts from 2020 and 2019.

As of this writing, Python 3.7 is approaching its end-of-life and many packages have already set a minimum version of Python 3.8. This document should be superseded or disregarded no later than the Python 3.9 end-of-life. If you cite this as a justification for your behavior, please stop doing so at that time.

Summary

• Describe all of your project metadata in pyproject.toml
• Use hatchling to build sdists and wheels
• Use hatch-vcs to extract the version from git and, optionally, write into a module
• Test your packages with a build-test-deploy workflow in CI

A Cookiecutter template implementing these suggestions can be found at https://github.com/effigies/cookiecutter-packaging.

Perspective

My position in the Python ecosystem will color my perspective and approach to packaging, and, presumably, how much weight you give what I think.

I work primarily on neuroimaging packages. I am currently the lead maintainer of NiBabel and keep the lights on at Nipype 1.x while the transition to Pydra progresses. I also maintain PyBIDS, fMRIPrep and a few other packages closely related to the these.

These packages have different requirements. A couple worth mentioning are

1. The nipy packages have historically included git hash references and version information to allow users to provide detailed debugging information. Preserving this while updating the infrastructure has taken effort.
2. Pydra aims to keep a similar import infrastructure to nipype, so that from nipype.interfaces import fsl becomes from pydra.tasks import fsl, while separating out the tasks into task packages. pydra.tasks.fsl is provided by a pydra-fsl package.

A couple years ago, I became a maintainer of versioneer, which until a few years ago was the way in Python to extract versions from Git history. As a result, I’ve become a bit more familiar with the internals and development trajectory of setuptools.

On the whole, I believe the Python packaging ecosystem is moving in a positive direction. The changes in this document over previous versions reflect the evolution of the standards and tooling more than in my philosophy. In particular, the standardization of declarative package metadata and editable installs mean that the choice of build backends is now less fraught, and we are free to choose the simplest that gets the job done.

Desiderata

Motivating my recommendations are a few desiderata, in rough order of importance:

1. Installation should work, from source, on fairly old systems. Debian Stable (11; “bullseye”) is my touchstone here.
2. Prefer declarative syntax, and limit dynamic metadata, as much as possible.
3. Enable revision-based versions, with minimal opportunity for error.
4. Limit custom code to absolute minimum. (Partially redundant with limiting dynamic metadata.)

To operationalize (1), the following approaches should all install correctly:

• pip install .
• python -m build && pip install dist/*.tar.gz
• python -m build && pip install dist/*.whl

And development/editable mode should work:

• pip install -e .

To operationalize (3), all of the above should produce an install with the same version string, and setting the version should be done from a version control tag if possible. Assuming a git repository, the following should also work:

• git archive -o archive.tar.gz $TAG && pip install archive.tar.gz

Recommendations

I recommend using hatchling and hatch-vcs.

[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

[project]
name = "project-name"
description = "A package"
readme = "README.md"
requires-python = ">=3.8"
license = { file="LICENSE" }
authors = [
  { name="You", email="your@email.tld" },
]
classifiers = [
  "Programming Language :: Python :: 3",
]
dependencies = []
dynamic = ["version"]

[project.urls]
"Homepage" = "https://github.com/your/package"

[tool.hatch.version]
source = "vcs"

import project_name
print(project_name.__version__)

[tool.hatch.build.hooks.vcs]
version-file = "project_name/_version.py"

try:
    from ._version import version as __version__
except ImportError:
    pass

node: $Format:%H$
node-date: $Format:%cI$
describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$
ref-names: $Format:%D$

.git_archival.txt  export-subst

[tool.hatch.build.targets.sdist]
exclude = [".git_archival.txt"]

[tool.hatch.build.targets.wheel]
packages = ["project_name"]
exclude = [
    "project_name/data/test_data",
]

Notes on new build systems and legacy operating systems

The standardization of build frontends and backends has dramatically reduced my concerns about backwards compatibility. As long as a user is able to upgrade pip with pip install --upgrade --user pip, essentially any build system becomes available. The main place where this could become problematic is on systems that can’t fetch from PyPI due to network access restrictions. In this case, distributing pre-built wheels that do not need any backend processing is probably cleanest.

Continuous Integration

Whether you use the recommended approach or not, it’s worth checking that your packages build correctly. In addition to users of your packages, you may have third-party packagers that will prepare your package to be installed via conda or a Linux distribution-specific package manager. Testing your packages outside your repository reduces the chances of distributing broken packages.

The following is a minimal GitHub Actions specification for testing packages, but it should be easily translatable to other continuous integration services.

on:
  push:
    branches:
      - master
      - maint/*
    tags:
      - "*"
  pull_request:
    branches:
      - master
      - maint/*

defaults:
  run:
    shell: bash

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v4
        with:
          python-version: 3
      - run: pip install --upgrade build twine
      - name: Build sdist and wheel
        run: python -m build
      - run: twine check dist/*
      - name: Upload sdist and wheel artifacts
        uses: actions/upload-artifact@v3
        with:
          name: dist
          path: dist/
      - name: Build git archive
        run: mkdir archive && git archive -v -o archive/archive.tgz HEAD
      - name: Upload git archive artifact
        uses: actions/upload-artifact@v3
        with:
          name: archive
          path: archive/

  test-package:
    runs-on: ubuntu-latest
    needs: [build]
    strategy:
      matrix:
        package: ['wheel', 'sdist', 'archive']
    steps:
      - name: Download sdist and wheel artifacts
        if: matrix.package != 'archive'
        uses: actions/download-artifact@v3
        with:
          name: dist
          path: dist/
      - name: Download git archive artifact
        if: matrix.package == 'archive'
        uses: actions/download-artifact@v3
        with:
          name: archive
          path: archive/
      - uses: actions/setup-python@v4
        with:
          python-version: 3
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"
      - name: Update pip
        run: pip install --upgrade pip
      - name: Install wheel
        if: matrix.package == 'wheel'
        run: pip install dist/*.whl
      - name: Install sdist
        if: matrix.package == 'sdist'
        run: pip install dist/*.tar.gz
      - name: Install archive
        if: matrix.package == 'archive'
        run: pip install archive/archive.tgz
      - name: Install test extras
        run: pip install project-name[test]
      - name: Run tests
        run: pytest --doctest-modules -v --pyargs project_name

  publish:
    runs-on: ubuntu-latest
    needs: [test-package]
    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
    steps:
      - uses: actions/download-artifact@v3
        with:
          name: dist
          path: dist/
      - uses: pypa/gh-action-pypi-publish@release/v1
        with:
          user: __token__
          password: ${{ secrets.PYPI_API_TOKEN }}

(Apologies for the lack of newlines. They seem to be removed by the syntax highlighter.)

To use the publish step, you will need to add a PyPI token to your encrypted secrets.

References

• Python Packaging User Guide - Declaring project metadata

Validity

This was written in July 2020, superseding this gist from 2019.

As of this writing, Python 3.5 is approaching its end-of-life and many packages have already set a minimum version of Python 3.6. This document should be superseded or disregarded no later than the Python 3.7 end-of-life. If you cite this as a justification for your behavior, please stop doing so at that time.

Summary

• For versioning, use versioneer
• For describing your build requirements, use pyproject.toml
• For all static (and some dynamic-ish) metadata, use setup.cfg
• A very small setup.py for dynamic metadata and to tie everything together
• Use MANIFEST.in for files that cannot otherwise be included in the sdist

Perspective

My position in the Python ecosystem will color my perspective and approach to packaging, and, presumably, how much weight you give what I think.

I am most active with the NIPY collection of projects, generally related to neuroimaging and neuroscience. I am currently the lead maintainer of NiBabel and do a reasonable amount of maintenance work for Nipype, PyBIDS, fMRIPrep and a few other packages closely related to the aforementioned.

I am not an active developer in CPython, PyPA or any packaging-related tools. I have not followed deep arguments, but have relied mostly on PEPs, documentation and sporadic searches to identify the current state of the art.

So my perspective is less concerned with what packaging should become, and more with what works today and where things appear to be heading, as I look to prepare or update packaging infrastructure for several tools. Infrastructure I hope to stop thinking about so much.

Desiderata

Motivating my recommendations are a few desiderata, in rough order of importance:

1. Installation should work, from source, on fairly old systems. Debian Stable (10; “buster”) is my touchstone here.
2. Prefer declarative syntax, and limit dynamic metadata, as much as possible.
3. Enable revision-based versions, with minimal opportunity for error.
4. Limit custom code to absolute minimum. (Partially redundant with limiting dynamic metadata.)

To operationalize (1), the following approaches should all install correctly:

• pip install .
• python setup.py sdist && pip install dist/*.tar.gz
• python setup.py bdist_wheel && pip install dist/*.whl
• python setup.py install

And development/editable mode should work:

• python setup.py develop
• pip install -e .

This also means that newer, better build systems that do not rely on setuptools are not really under consideration here.

To operationalize (3), all of the above should produce an install with the same version string, and setting the version should be done from a version control tag if possible. Assuming a git repository, the following should also work:

• git archive -o archive.tar.gz $TAG && pip install archive.tar.gz

Recommendations

I recommend on a setuptools-based approach, using setup.cfg to declare as much of the metadata as possible, along with an OPTIONAL pyproject.toml laid out in PEP 518. Versioneer is used to handle versioning.

[build-system]
requires = ["setuptools >= 30.3.0", "wheel"]

[metadata]
url = https://github.com/your/package
author = You
author_email = your@email.tld
maintainer = You
maintainer_email = your@email.tld
description = A package
long_description = file:README.rst
long_description_content_type = text/x-rst; charset=UTF-8
license = GPL
classifiers =
    Programming Language :: Python

[options]
python_requires = >= 3.6
install_requires =
packages = find:

[options.package_data]
* =
    data/*

[options.extras_require]
doc =
    sphinx
test =
    pytest
    coverage
all =
    %(doc)s
    %(test)s

#!/usr/bin/env python
import sys
from setuptools import setup
import versioneer

# Give setuptools a hint to complain if it's too old a version
# 30.3.0 allows us to put most metadata in setup.cfg
# Should match pyproject.toml
SETUP_REQUIRES = ['setuptools >= 30.3.0']
# This enables setuptools to install wheel on-the-fly
SETUP_REQUIRES += ['wheel'] if 'bdist_wheel' in sys.argv else []

if __name__ == '__main__':
    setup(name='package',
          version=versioneer.get_version(),
          cmdclass=versioneer.get_cmdclass(),
          setup_requires=SETUP_REQUIRES,
          )

[versioneer]
VCS = git
style = pep440
versionfile_source = package/_version.py
versionfile_build = package/_version.py
tag_prefix =
parentdir_prefix = 

pip install versioneer
versioneer install

Dependencies

The minimum setuptools version needed for setup.cfg to work is 30.3.0, although more fields have been defined since then, and the minimum pip needed for PEP 518 compatibility is 10.0.0.

Notes on new build systems and legacy operating systems

As noted above, setuptools was the only system under serious consideration, simply because it has long been the standard to run python setup.py. Until pip 10+ is universal, alternative build systems will create headaches that I don’t want to deal with.

CentOS 7, for instance still packages pip 7.1 and setuptools 0.9.8, which means the above will not work out of the box (though this may be changing… I’m having a hard time reading pkgs.org). However, sticking with setuptools and setup_requires ensures that a user will at least be told to upgrade setuptools.

References

Validity

This document has been superseded as of July 2020.

This was written in July 2019. As of this writing Python 2.7 and Python 3.5 still have not reached end-of-life. This document should be superseded or disregarded no later than the Python 3.6 end-of-life. If you cite this as a justification for your behavior, please stop doing so at that time.

Summary

• For versioning, use versioneer
• For describing your build requirements, use pyproject.toml
• For all static (and some dynamic-ish) metadata, use setup.cfg
• A very small setup.py for dynamic metadata and to tie everything together
• Go ahead and keep MANIFEST.in for now

Perspective

My position in the Python ecosystem will color my perspective and approach to packaging, and, presumably, how much weight you give what I think.

I am most active with the NIPY collection of projects, generally related to neuroimaging and neuroscience. I am currently the lead maintainer of NiBabel and do a reasonable amount of maintenance work for Nipype, PyBIDS, fMRIPrep and a few other packages closely related to the aforementioned.

I am not an active developer in CPython, PyPA or any packaging-related tools. I have not followed deep arguments, but have relied mostly on PEPs, documentation and sporadic searches to identify the current state of the art.

So my perspective is less concerned with what packaging should become, and more with what works today and where things appear to be heading, as I look to prepare or update packaging infrastructure for several tools. Infrastructure I hope to stop thinking about so much.

Desiderata

Motivating my recommendations are a few desiderata, in rough order of importance:

1. Installation should work, from source, on fairly old systems. Debian Stable (9) is my touchstone here.
2. Prefer declarative syntax, and limit dynamic metadata, as much as possible.
3. Enable revision-based versions, with minimal opportunity for error.
4. Limit custom code to absolute minimum. (Partially redundant with limiting dynamic metadata.)

To operationalize (1), the following approaches should all install correctly:

• python setup.py install
• pip install .
• python setup.py sdist && pip install dist/*.tar.gz
• python setup.py bdist_wheel && pip install dist/*.whl

And development/editable mode should work:

• python setup.py develop
• pip install -e .

This also means that newer, better build systems that do not rely on setuptools are not really under consideration here.

To operationalize (3), all of the above should produce an install with the same version string, and setting the version should be done from a version control tag if possible. Assuming a git repository, the following should also work:

• git archive -o archive.tar.gz $TAG && pip install archive.tar.gz

Recommendations

I have settled on a setuptools-based approach, using setup.cfg to declare as much of the metadata as possible, along with pyproject.toml laid out in PEP 518. Versioneer is used to handle versioning.

[build-system]
requires = ["setuptools >= 30.3.0", "wheel"]

[metadata]
url = https://github.com/your/package
author = You
author_email = your@email.tld
maintainer = You
maintainer_email = your@email.tld
description = A package
long_description = file:README.rst
long_description_content_type = text/x-rst; charset=UTF-8
license = GPL
classifiers =
    Programming Language :: Python

[options]
python_requires = >= 3.5
install_requires =
test_requires =
    pytest
    coverage
packages = find:
include_package_data = True

[options.package_data]
* =
    data/*

[options.extras_require]
doc =
    sphinx
test =
    pytest
    coverage
all =
    %(doc)s
    %(test)s

#!/usr/bin/env python
import sys
from setuptools import setup
import versioneer

# Give setuptools a hint to complain if it's too old a version
# 30.3.0 allows us to put most metadata in setup.cfg
# Should match pyproject.toml
SETUP_REQUIRES = ['setuptools >= 30.3.0']
# This enables setuptools to install wheel on-the-fly
SETUP_REQUIRES += ['wheel'] if 'bdist_wheel' in sys.argv else []

if __name__ == '__main__':
    setup(name='package',
          version=versioneer.get_version(),
          cmdclass=versioneer.get_cmdclass(),
          setup_requires=SETUP_REQUIRES,
          )

[versioneer]
VCS = git
style = pep440
versionfile_source = package/_version.py
versionfile_build = package/_version.py
tag_prefix =
parentdir_prefix = 

pip install versioneer
versioneer install

Dependencies

The minimum setuptools version needed for setup.cfg to work is 30.3.0, although more fields have been defined since then, and the minimum pip needed for PEP 518 compatibility is 10.0.0.

Notes on new build systems and legacy operating systems

As noted above, setuptools was the only system under serious consideration, simply because it has long been the standard to run python setup.py. Until pip 10+ is universal, alternative build systems will create headaches that I don’t want to deal with.

CentOS 7, for instance still packages pip 7.1 and setuptools 0.9.8, which means the above will not work out of the box. However, sticking with setuptools and setup_requires ensures that a user will at least be told to upgrade setuptools.

References