Using the Python API

You can use req2flatpak’s python api to write a custom python script that makes use of req2flatpak’s functionality. This allows to programmatically tweak and tune the behavior as needed.

Example

You can use the following code as an example to get started with your script. The code demonstrates how to generate a flatpak-builder build module in order to install python packages on a specific flatpak target platform.

platforms = [PlatformFactory.from_string("cp310-x86_64")]
requirements = RequirementsParser.parse_file("requirements.txt")
releases = PypiClient.get_releases(requirements)
downloads = {
    DownloadChooser.wheel_or_sdist(release, platform)
    for release in releases
    for platform in platforms
}
build_module = FlatpakGenerator.build_module(requirements, downloads)

The above code uses req2flatpak to generate a build module in five steps:

define the target platforms,
specify the python packages to be installed,
query PyPi about available downloads,
choose downloads that are compatible with the target platforms,
generate the flatpak-builder build module.

…if you include the resulting build module in a flatpak build, the module will install the required packages.

You will benefit from a writing a custom script (in contrast to simply using req2flatpak’s commandline interface) if you want to change the modify or tweak the behavior. In a custom script, you have all the freedom in the world to modify each step as you see fit. For example:

you may want to query other package indices instead of pypi,
you may prefer wheels or sdists for certain packages, or
you may want to exclude specific packages.

The following subsections explain in detail how you can use req2flatpak’s python api in your custom script. For further inspiration you can also have a look at how req2flatpak’s main() method is implemented.

Specifying Target Platforms

A target platform describes everything that pip (or req2flatpak) needs to know when choosing package downloads to install on this platform.

Platforms

Target platforms are represented in req2flatpak as a dataclass, as follows.

class req2flatpak.Platform(python_version: List[str], python_tags: List[str])

Represents a target platform for python package installations.

python_tags: List[str]: A list of platform tags, similar to packaging.tags.sys_tags().

python_version: List[str]: A list of python version numbers, similar to platform.python_version_tuple().

There are many options how to create platform objects. You can create platform objects any way you wish in your script. And you can use functionality from req2flatpak, as described below.

PlatformFactory

The PlatformFactory provides methods for creating platform objects.

For example:

platform = PlatformFactory.from_string("cp310-x86_64")

Documentation of all methods provided by the PlatformFactory class:

class req2flatpak.PlatformFactory

Provides methods for creating platform objects.

classmethod from_current_interpreter() → Platform

Returns a platform object that describes the current interpreter and system.

This method requires the packaging package to be installed because functionality from this package is used to generate the list of tags that are supported by the current platform.

The platform object returned by this method obviously depends on the specific python interpreter and system architecture used to run the req2flatpak script. The reason why is that this method reads platform properties from the current interpreter and system.

classmethod from_python_version_and_arch(minor_version: int | None = None, arch='x86_64') → Platform

Returns a platform object that roughly describes a cpython installation on linux.

The tags in the platform object are a rough approximation, trying to match what packaging.tags.sys_tags would return if invoked on a linux system with cpython. No guarantees are made about how closely this approximation matches a real system.

Parameters:

minor_version – the python 3 minor version, specified as int. Defaults to the current python version.
arch – either “x86_64” or “aarch64”.

classmethod from_string(platform_string: str) → Platform | None

Returns a platform object by parsing a platform string.

Parameters:: platform_string – A string specifying python version and system architecture. The string format is “{python_version}-{system_architecture}”. For example: “cp39-x86_64” or “cp310-aarch64”. Acceptable values are the same as in from_python_version_and_arch().

Specifying Package Requirements

A requirement describes the name and exact version of a python package that shall be installed.

Requirements

Package requirements are represented in req2flatpak as a dataclass, as follows:

class req2flatpak.Requirement(package: str, version: str)

Represents a python package requirement.

package: str: A python package name.

version: str: The exact version of the package.

RequirementsParser

There are many options for how to create a requirement object in your code. One option is to use methods from req2flatpak’s RequirementsParser class, as in the following example:

requirements = RequirementsParser.parse_file("requirements.txt")

Documentation of all methods provided by the RequirementsParser class:

class req2flatpak.RequirementsParser

Parses requirements.txt files in a very simple way.

This methods expects all versions to be pinned, and it does not resolve dependencies.

classmethod parse_file(file) → List[Requirement]: Parses a requirements.txt file into a list of Requirement objects.

classmethod parse_string(requirements_txt: str) → List[Requirement]: Parses requirements.txt string content into a list of Requirement objects.

It is important to note that req2flatpak’s RequirementsParser expects all package versions to be fully specified. For example, it will not accept a package version specification such as requests >= 2.0. Instead, it expects all versions to be pinned using the == operator, such as, e.g., s``requests == 2.0``.

The reason for this is that req2flatpak, by design, does not resolve or freeze dependencies. You can use other tools like pip-compile or poetry export to resolve and freeze dependencies and to export them into a requirements.txt file.

Querying Package Indices for Available Releases

A release describes what is offered by package index (or multiple indices) for a given package requirement.

Release

Releases are represented in req2flatpak as a dataclass, as follows:

class req2flatpak.Release(package: str, version: str, downloads: ~typing.List[~req2flatpak.Download] = <factory>)

Represents a package release as name, version, and downloads.

downloads: List[Download]

package: str: A python package name.

version: str: The exact version of the package.

In other words, a release object combines the name and version of a required package with a list of available downloads. You can create release objects in your code as you wish, or you can use req2flatpak’s PypiClient for this purpose.

PypiClient

The PypiClient class allows to query the “PyPi” python package index about available releases. For example:

releases = PypiClient.get_releases(requirements)

Documentation of all methods provided by the PypiClient class:

class req2flatpak.PypiClient

Queries package information from the PyPi package index.

cache: dict | Shelf = {}: A dict-like object for caching responses from PyPi.

classmethod get_release(req: Requirement) → Release: Queries pypi regarding available releases for this requirement.

classmethod get_releases(reqs: Iterable[Requirement]) → List[Release]: Queries pypi regarding available releases for these requirements.

Caching: PypiClient caches responses to reduce traffic when querying PyPi. By default, a simple dict is used as an in-memory cache. To improve caching, you can use a persistent cache as follows:

import shelve

with shelve.open("pypi_cache.tmp") as cache:
    PypiClient.cache = cache
    releases = PypiClient.get_releases(requirements)

The above code instantiates a persistent shelve.Shelf cache using pypi_cache.tmp as filename. The code then configures the PypiClient class to use the shelf for caching, and then uses the cache when querying PyPi.

Clients for other package indices instead of Pypi are not included in req2flatpak. You are free, of course, to implement your own clients for additional package indices in you own script.

Choosing Compatible Downloads

req2flatpak provides methods for choosing the best compatible download for a given target platform.

Background Information

Choosing compatible downloads is important because, generally speaking, only a subset of downloads are compatible with a given target platform. For example, a release may contain wheels for older python interpreters or different system architectures.

In python packaging, the compatibility requirements of any download are encoded in the filename as a set of “tags”, as explained, for example, in pypa’s packaging documentation about tags. A download is compatible with a target platform if (at least) one of the package tags equals one of the target platform’s system tags.

Choosing the “best” compatible download is important because multiple downloads may be compatible, but only one download is needed for installing a package. Pip’s algorithm for choosing the best download uses ranked platform tags; the best download is the one that matches the highest-ranked platform tag.

Multiple implementations exist in related work for choosing the best compatible download:

Pip’s internal behavior is the official reference implementation, but pip recommends against accessing internal pip apis from outside packages, which makes it difficult to re-use the implementation.
Pyodide’s implementation of its find_matching_wheels function is easy to understand and can serve for inspiration.
req2flatpak includes its own implementation in the DownloadChooser class.

DownloadChooser

The DownloadChooser class provides the following methods for filtering compatible downloads and for choosing the “best” download. For example:

# choose the best wheel for a target platform:
wheel = DownloadChooser.wheel(release, platform)

Documentation of all methods provided by the DownloadChooser class:

class req2flatpak.DownloadChooser

Provides methods for choosing package downloads.

This class implements logic for filtering wheel and sdist downloads that are compatible with a given target platform.

classmethod matches(download: Download, platform_tag: str) → bool: Returns whether a download is compatible with a target platform tag.

classmethod downloads(release: Release, platform: Platform, wheels_only=False, sdists_only=False) → Iterator[Download]

Yields suitable downloads for a specific platform.

The order of downloads matches the order of platform tags, i.e., preferred downloads are returned first.

classmethod wheel(release: Release, platform: Platform) → Download | None: Returns the preferred wheel download for this release.

classmethod sdist(release: Release) → Download | None: Returns the source package download for this release.

classmethod wheel_or_sdist(release: Release, platform: Platform) → Download | None: Returns a wheel or an sdist for this release, in this order of preference.

classmethod sdist_or_wheel(release: Release, platform: Platform) → Download | None: Returns an sdist or a wheel for this release, in this order of preference.

Generating a Build Module for flatpak-builder

The output of req2flatpak is a build module, intended to be processed by flatpak-builder as part of a build manifest.

flatpak-builder creates flatpak packages by processing build manifests. A manifest can consist of multiple build modules. Each module adds to the things to be included in the resulting flatpak package. The build modules generated by req2flatpak are no different; when processed by flatpak-builder, they install required python packages to include them into the flatpak package.

FlatpakGenerator

req2flatpak’s FlatpakGenerator class provides methods for generating a build module that will instruct flatpak-builder to install the required python packages.

Example usage:

# generate a flatpak build module:
build_module = FlatpakGenerator.build_module(requirements, downloads)

Documentation of all methods provided by the FlatpakGenerator class:

class req2flatpak.FlatpakGenerator

Provides methods for generating a flatpak-builder build module.

static build_module(requirements: Iterable[Requirement], downloads: Iterable[Download], module_name='python3-package-installation', pip_install_template: str = 'pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} --no-build-isolation ') → dict: Generates a build module for inclusion in a flatpak-builder build manifest.

classmethod build_module_as_str(*args, **kwargs) → str

Generate JSON build module for inclusion in a flatpak-builder build manifest.

The args and kwargs are the same as in build_module()

classmethod build_module_as_yaml_str(*args, **kwargs) → str

Generate YAML build module for inclusion in a flatpak-builder build manifest.

The args and kwargs are the same as in build_module()

Saving the Generated Build Module

You can easily save a generated build module as a .json file using built-in functionality from python’s standard library. For example:

# example showing how to export a build module to json
import json

# write the json data to file:
with open("build-module.json", "w") as outfile:
 json.dump(build_module, outfile, indent=2)

The above code assumes a build module has been generated and is stored as a dict in the variable build_module. The code shows how to serialize the dict and save it to an output file.