Skip to content

Commit 547d872

Browse files
authored
eccodeslib: add build-eccodeslib.yml for riscv64 wheels (#1694)
* **Package**: `eccodeslib` * **Version**: `2.48.0.26` * **Source**: https://github.com/ecmwf/eccodes * **Docs**: https://confluence.ecmwf.int/display/ECC/ecCodes+Home Compiles ecCodes, ECMWF's C library for decoding and encoding GRIB/BUFR meteorological data, with no Python bindings of its own. Upstream publishes no riscv64 wheel. Mirrors the CMake flags of ecmwf/eccodes's own [`python_wrapper/buildconfig`](https://github.com/ecmwf/eccodes/blob/2.48.0/python_wrapper/buildconfig), since upstream builds this wheel with a private tool (`ecmwf-actions/reusable-workflows`' `python-wrapper-wheel.yml` plus its "wheelmaker" image), not a public CI workflow. **Differs from upstream** - Single py3-none wheel, not a per-interpreter matrix - libeccodes.so is byte-identical across upstream's published cp310-cp314 wheels. - `ENABLE_ECKIT_GEO=0`, not upstream's `1` - eckit has no riscv64 build yet; only drops the optional geo-iterator backend. - libaec and OpenJPEG built from source, libpng from the manylinux image's dnf repos, replacing upstream's private "cxx-deps" image. **Testing** - Installs the built wheel and calls `findlibs.load("eccodes")` to confirm the library and its vendored codecs resolve and dlopen correctly. **License**: Apache-2.0. The wheel also vendors libpng, OpenJPEG and libaec (all BSD-style licences); their licence texts are added under `dist-info/licenses` since this port builds them separately from eccodes itself.
1 parent 446629c commit 547d872

1 file changed

Lines changed: 241 additions & 0 deletions

File tree

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
---
4+
# Upstream builds eccodeslib via a private tool (ecmwf-actions/reusable-workflows'
5+
# python-wrapper-wheel.yml + the "wheelmaker" image), not a public workflow, so this
6+
# replays ecmwf/eccodes's own python_wrapper/buildconfig CMAKE_PARAMS directly. The
7+
# built libeccodes.so is byte-identical across upstream's cp310..cp314 wheels (no
8+
# Python C-API use), so this builds one py3-none wheel instead of a per-interpreter
9+
# matrix. ENABLE_ECKIT_GEO is set to its CMake default (OFF; upstream forces it ON)
10+
# since eckit has no riscv64 build yet; this only drops the optional geo-iterator
11+
# backend, not core GRIB/BUFR support.
12+
name: Build eccodeslib wheels (riscv64)
13+
14+
on:
15+
workflow_dispatch:
16+
inputs:
17+
version:
18+
description: 'eccodeslib version to build (e.g. 2.48.0.26); the eccodes git tag is this with the trailing build-counter component dropped'
19+
required: true
20+
default: '2.48.0.26'
21+
pull_request:
22+
paths:
23+
- '.github/workflows/build-eccodeslib.yml'
24+
25+
concurrency:
26+
group: ${{ github.workflow }}-${{ inputs.version || '2.48.0.26' }}-${{ github.head_ref || github.run_id }}
27+
cancel-in-progress: true
28+
29+
permissions:
30+
contents: read # to fetch code (actions/checkout)
31+
32+
env:
33+
PACKAGE_VERSION: ${{ inputs.version || '2.48.0.26' }}
34+
MANYLINUX_RISCV64_IMAGE: quay.io/pypa/manylinux_2_39_riscv64
35+
ECBUILD_VERSION: '3.15.2'
36+
OPENJPEG_VERSION: '2.5.2'
37+
LIBAEC_VERSION: '1.1.4'
38+
39+
jobs:
40+
setup:
41+
uses: $/.github/workflows/_setup.yml
42+
43+
build_wheels:
44+
needs: [setup]
45+
name: Build eccodeslib ${{ inputs.version || '2.48.0.26' }} py3-none-manylinux_riscv64
46+
runs-on: ubuntu-24.04-riscv
47+
timeout-minutes: 420
48+
steps:
49+
- name: Derive the eccodes git tag from the package version
50+
run: echo "ECCODES_VERSION=$(echo "$PACKAGE_VERSION" | sed -E 's/\.[0-9]+$//')" >> "$GITHUB_ENV"
51+
52+
- name: Checkout eccodes ${{ env.ECCODES_VERSION }}
53+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
54+
with:
55+
repository: ecmwf/eccodes
56+
ref: ${{ env.ECCODES_VERSION }}
57+
persist-credentials: false
58+
59+
- name: Record the resolved commit
60+
run: echo "ECCODES_COMMIT=$(git rev-parse HEAD)" >> "$GITHUB_ENV"
61+
62+
- name: Write the wheel packaging script
63+
run: |
64+
cat > build_wheel.py <<'PY'
65+
import hashlib
66+
import os
67+
import sys
68+
import zipfile
69+
from base64 import urlsafe_b64encode
70+
71+
PKG = "eccodeslib"
72+
VERSION, PLATFORM_TAG, INSTALL_DIR, LICENSES_DIR, OUT_DIR = sys.argv[1:6]
73+
74+
WHEEL_TAG = f"py3-none-{PLATFORM_TAG}"
75+
DIST_INFO = f"{PKG}-{VERSION}.dist-info"
76+
77+
METADATA = f"""Metadata-Version: 2.4
78+
Name: {PKG}
79+
Version: {VERSION}
80+
Summary: Compiled ecCodes C library (GRIB/BUFR decoding and encoding), no Python bindings
81+
Home-page: https://confluence.ecmwf.int/display/ECC/ecCodes+Home
82+
Project-URL: Source, https://github.com/ecmwf/eccodes
83+
License-Expression: Apache-2.0
84+
"""
85+
86+
WHEEL = f"""Wheel-Version: 1.0
87+
Generator: python-wheels-riscv64-port
88+
Root-Is-Purelib: false
89+
Tag: {WHEEL_TAG}
90+
"""
91+
92+
os.makedirs(OUT_DIR, exist_ok=True)
93+
wheel_path = os.path.join(OUT_DIR, f"{PKG}-{VERSION}-{WHEEL_TAG}.whl")
94+
records = []
95+
96+
def write_file(zf, arcname, path):
97+
with open(path, "rb") as f:
98+
data = f.read()
99+
info = zipfile.ZipInfo(arcname)
100+
info.external_attr = (os.stat(path).st_mode & 0xFFFF) << 16
101+
zf.writestr(info, data, zipfile.ZIP_DEFLATED)
102+
digest = urlsafe_b64encode(hashlib.sha256(data).digest()).rstrip(b"=").decode()
103+
records.append(f"{arcname},sha256={digest},{len(data)}")
104+
105+
def write_bytes(zf, arcname, data):
106+
zf.writestr(arcname, data)
107+
digest = urlsafe_b64encode(hashlib.sha256(data).digest()).rstrip(b"=").decode()
108+
records.append(f"{arcname},sha256={digest},{len(data)}")
109+
110+
with zipfile.ZipFile(wheel_path, "w", zipfile.ZIP_DEFLATED) as zf:
111+
for root, _dirs, files in os.walk(INSTALL_DIR):
112+
for name in sorted(files):
113+
path = os.path.join(root, name)
114+
rel = os.path.relpath(path, INSTALL_DIR)
115+
write_file(zf, f"{PKG}/{rel}", path)
116+
117+
for name in sorted(os.listdir(LICENSES_DIR)):
118+
write_file(zf, f"{DIST_INFO}/licenses/{name}", os.path.join(LICENSES_DIR, name))
119+
120+
write_bytes(zf, f"{DIST_INFO}/METADATA", METADATA.encode())
121+
write_bytes(zf, f"{DIST_INFO}/WHEEL", WHEEL.encode())
122+
write_bytes(zf, f"{DIST_INFO}/top_level.txt", f"{PKG}\n".encode())
123+
124+
record_name = f"{DIST_INFO}/RECORD"
125+
zf.writestr(record_name, "\n".join(records + [f"{record_name},,"]) + "\n")
126+
127+
print(wheel_path)
128+
PY
129+
130+
- name: Build eccodes and its vendored codecs, then package the wheel
131+
run: |
132+
docker run --rm \
133+
-v "$(pwd)":/workspace \
134+
--workdir /workspace \
135+
-e PACKAGE_VERSION="$PACKAGE_VERSION" \
136+
-e ECCODES_COMMIT="$ECCODES_COMMIT" \
137+
-e ECBUILD_VERSION="$ECBUILD_VERSION" \
138+
-e OPENJPEG_VERSION="$OPENJPEG_VERSION" \
139+
-e LIBAEC_VERSION="$LIBAEC_VERSION" \
140+
"$MANYLINUX_RISCV64_IMAGE" \
141+
bash -c '
142+
set -euo pipefail
143+
dnf -y -q install libpng-devel
144+
mkdir -p /tmp/deps /tmp/build /tmp/install/eccodeslib /tmp/licenses
145+
146+
curl -fsSL "https://github.com/ecmwf/ecbuild/archive/refs/tags/$ECBUILD_VERSION.tar.gz" | tar xz -C /tmp/build
147+
cmake -S "/tmp/build/ecbuild-$ECBUILD_VERSION" -B /tmp/build/ecbuild-build -D CMAKE_INSTALL_PREFIX=/tmp/deps
148+
cmake --build /tmp/build/ecbuild-build -j "$(nproc)"
149+
cmake --install /tmp/build/ecbuild-build
150+
151+
mkdir -p /tmp/build/libaec-src
152+
curl -fsSL "https://github.com/Deutsches-Klimarechenzentrum/libaec/releases/download/v$LIBAEC_VERSION/libaec-$LIBAEC_VERSION.tar.gz" | tar xz --strip-components=1 -C /tmp/build/libaec-src
153+
cmake -S /tmp/build/libaec-src -B /tmp/build/libaec-build \
154+
-D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/tmp/deps -D CMAKE_INSTALL_LIBDIR=lib \
155+
-D BUILD_STATIC_LIBS=OFF -D BUILD_TESTING=OFF
156+
cmake --build /tmp/build/libaec-build -j "$(nproc)"
157+
cmake --install /tmp/build/libaec-build
158+
159+
mkdir -p /tmp/build/openjpeg-src
160+
curl -fsSL "https://github.com/uclouvain/openjpeg/archive/refs/tags/v$OPENJPEG_VERSION.tar.gz" | tar xz --strip-components=1 -C /tmp/build/openjpeg-src
161+
cmake -S /tmp/build/openjpeg-src -B /tmp/build/openjpeg-build \
162+
-D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_INSTALL_PREFIX=/tmp/deps -D CMAKE_INSTALL_LIBDIR=lib \
163+
-D BUILD_CODEC=OFF
164+
cmake --build /tmp/build/openjpeg-build -j "$(nproc)"
165+
cmake --install /tmp/build/openjpeg-build
166+
167+
cmake -S . -B /tmp/build/eccodes-build \
168+
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
169+
-D CMAKE_PREFIX_PATH=/tmp/deps \
170+
-D CMAKE_INSTALL_PREFIX=/tmp/install/eccodeslib \
171+
-D CMAKE_INSTALL_LIBDIR=lib64 \
172+
-D Python3_EXECUTABLE=/opt/python/cp312-cp312/bin/python3 \
173+
-D ENABLE_PNG=1 \
174+
-D ENABLE_JPG=1 -D ENABLE_JPG_LIBJASPER=0 -D ENABLE_JPG_LIBOPENJPEG=1 -D OPENJPEG_DIR=/tmp/deps \
175+
-D ENABLE_NETCDF=0 -D ENABLE_FORTRAN=0 \
176+
-D ENABLE_EXAMPLES=0 -D ENABLE_BUILD_TOOLS=1 \
177+
-D ENABLE_INSTALL_ECCODES_DEFINITIONS=0 -D ENABLE_INSTALL_ECCODES_SAMPLES=0 \
178+
-D ENABLE_ECCODES_THREADS=1 -D ENABLE_MEMFS=1 -D ENABLE_ECKIT_GEO=0 \
179+
-D ENABLE_AEC=1
180+
cmake --build /tmp/build/eccodes-build -j "$(nproc)"
181+
cmake --install /tmp/build/eccodes-build
182+
183+
cp /tmp/deps/lib/libopenjp2.so* /tmp/install/eccodeslib/lib64/
184+
cp /tmp/deps/lib/libaec.so* /tmp/install/eccodeslib/lib64/
185+
cp "$(find /usr/lib* -name "libpng16.so.16*" | head -1)" /tmp/install/eccodeslib/lib64/
186+
187+
cp LICENSE /tmp/licenses/eccodes.txt
188+
cp AUTHORS /tmp/licenses/
189+
curl -fsSL "https://raw.githubusercontent.com/uclouvain/openjpeg/v$OPENJPEG_VERSION/LICENSE" -o /tmp/licenses/openjpeg.txt
190+
curl -fsSL "https://raw.githubusercontent.com/MathisRosenhauer/libaec/v$LIBAEC_VERSION/LICENSE.txt" -o /tmp/licenses/libaec.txt
191+
cp /usr/share/licenses/libpng/LICENSE /tmp/licenses/libpng.txt
192+
193+
printf "__version__ = \"%s\"\n__commit_hash__ = \"%s\"\nfindlibs_dependencies = []\n" "$PACKAGE_VERSION" "$ECCODES_COMMIT" > /tmp/install/eccodeslib/__init__.py
194+
195+
python3 build_wheel.py "$PACKAGE_VERSION" manylinux_2_39_riscv64 /tmp/install/eccodeslib /tmp/licenses wheelhouse
196+
'
197+
198+
- name: Check the wheel ships the extensions and the bundled licences
199+
run: |
200+
python3 - wheelhouse/*.whl <<'EOF'
201+
import sys, zipfile
202+
203+
names = zipfile.ZipFile(sys.argv[1]).namelist()
204+
assert any(n.endswith("lib64/libeccodes.so") for n in names), names
205+
for licence in ("eccodes.txt", "openjpeg.txt", "libaec.txt", "libpng.txt"):
206+
assert any(
207+
n.endswith(f".dist-info/licenses/{licence}") for n in names
208+
), (licence, names)
209+
EOF
210+
211+
- name: Smoke-test the built library
212+
run: |
213+
sudo apt-get update -qq
214+
sudo apt-get install -y -qq --no-install-recommends python3-venv
215+
python3 -m venv .venv
216+
. .venv/bin/activate
217+
# gotcha 234: stock Ubuntu 24.04 apt pip (24.0) has zero manylinux_*_riscv64
218+
# tags, so it rejects the riscv64 wheel outright without upgrading first.
219+
pip install -q --upgrade pip
220+
pip install -q findlibs wheelhouse/*.whl
221+
python3 -c "
222+
import findlibs
223+
lib = findlibs.load('eccodes')
224+
print(lib)
225+
"
226+
227+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
228+
with:
229+
name: eccodeslib-${{ inputs.version || '2.48.0.26' }}-py3-none-manylinux_riscv64
230+
path: wheelhouse/*.whl
231+
if-no-files-found: error
232+
233+
publish:
234+
name: Publish eccodeslib ${{ inputs.version || '2.48.0.26' }}
235+
needs: [setup, build_wheels]
236+
permissions:
237+
contents: write
238+
pull-requests: write
239+
uses: $/.github/workflows/_publish-wheel.yml
240+
with:
241+
artifact-pattern: eccodeslib-${{ inputs.version || '2.48.0.26' }}-*-manylinux_riscv64

0 commit comments

Comments
 (0)