zj/tms-calamares
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

first

Browse Source
This commit is contained in:
zj
2025-11-23 16:05:51 +08:00
commit a923c1098d
364 changed files with 529674 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

145
calamares/ci/RELEASE.md Normal file
View File

@@ -0,0 +1,145 @@
# Calamares Release Process
<!-- SPDX-FileCopyrightText: 2015 Teo Mrnjavac <teo@kde.org>
SPDX-FileCopyrightText: 2017 Adriaan de Groot <groot@kde.org>
SPDX-License-Identifier: GPL-3.0-or-later
-->
> Calamares releases are now rolling when-they-are-ready releases.
> Releases are made from *calamares* and tagged there. When, in future,
> LTS releases resume, these steps may be edited again.
>
> Most things are automated through the release script [RELEASE.sh](RELEASE.sh)
## (0) During a release cycle
* Fetch latest translations from Transifex. We only push / pull translations
from *calamares* branch, so longer-lived branches (e.g. 3.1.x) don't get
translation updates. This is to keep the translation workflow simple.
The script automatically commits changes to the translations. It's ok
to do this during a release cycle. Run `sh ci/txpull.sh`
to fetch translations and commit the changes in one go.
* Push the strings to Transifex. From a checkout, run `ci/txpush.sh`
* Update the list of enabled translation languages in `CMakeLists.txt`.
Check the [translation site][transifex] for the list of languages with
fairly complete translations, or use `ci/txstats.py --edit` for an automated
suggestion. If there are changes, commit them.
## (1) Preparation
* Double-check the *CALAMARES_VERSION* value at the top of `CMakeLists.txt`.
* Set *CALAMARES_RELEASE_MODE* to `ON` in `CMakeLists.txt`.
* Edit `CHANGES-*` and set the date of the release. Pick the right
file for the release-stream.
* Commit both. This is usually done with commit-message
*Changes: pre-release housekeeping*.
## (2) Release Preparation
* Make sure all tests pass.
```
make
make test
```
Note that *all* means all-that-make-sense. The partition-manager tests need
an additional environment variable to be set for some tests, which will
destroy an attached disk. This is not always desirable. There are some
sample config-files that are empty and which fail the config-tests.
Note that the release script (see below) also runs the tests and
will bail out if any fail.
* Make sure the translations are up-to-date. There is logic to check
for changes in translations: a movable tag *translations* indicates
when translations were last pushed, and the logic tries to enforce a
week of latency between push-translations and a release, to allow
translators to catch up. Run `ci/txcheck.sh` to confirm this.
Run `ci/txcheck.sh --cleanup` to tidy up afterwards, and possibly pass
`-T` to the release script to skip the translation-age check if you
feel it is warranted.
* Run the helper script `ci/RELEASE.sh` or follow steps below.
The script checks:
- for uncommitted local changes,
- if translations are up-to-date and translators
have had enough time to chase new strings,
- that the build is successful (with gcc and clang, if available),
- tests pass,
- tarball can be created,
- tarball can be signed.
On success, it prints out a suitable signature- and SHA256 blurb
for use in the release announcement.
## (3) Release
Follow the instructions printed by the release script.
* Push the tags.
* Create a new release on Codeberg.
* Upload tarball and signature.
* Publish release article on `calamares.io`.
* Close associated milestone if it's entirely done.
* Update topic on `#calamares:kde.org` Matrix channel.
## (4) Post-Release
* Bump the version number in `CMakeLists.txt` in *CALAMARES_VERSION*.
* Set *CALAMARES_RELEASE_MODE* back to `OFF`.
* Add a placeholder entry for the next release in `CHANGES-*` with date
text *not released yet*. See the text below, "Placeholder Release".
Add the placeholder to the right file for the release-stream.
* Commit and push that, usually with the message
*Changes: post-release housekeeping*.
# Related Material
> This section isn't directly related to any specific release,
> but bears on all releases.
## GPG Key Maintainence
Calamares uses GPG Keys for signing the tarballs and some commits
(tags, mostly). Calamares uses the **maintainer's** personal GPG
key for this. This section details some GPG activities that the
maintainer should do with those keys.
- Signing sub-key. It's convenient to use a signing sub-key specifically
for the signing of Calamares. To do so, add a key to the private key.
It's recommended to use key expiry, and to update signing keys periodically.
- Run `gpg -K` to find the key ID of your personal GPG secret key.
- Run `gpg --edit-key <keyid>` to edit that personal GPG key.
- In gpg edit-mode, use `addkey`, then pick a key type that is *sign-only*
(e.g. type 4, *RSA (sign only)*), then pick a keysize (3072 seems ok
as of 2020) and set a key expiry time, (e.g. in 18 months time).
- After generation, the secret key information is printed again, now
including the new signing subkey:
```
ssb rsa3072/0xCFDDC96F12B1915C
created: 2020-07-11 expires: 2022-01-02 usage: S
```
- Update the `RELEASE.sh` script with a new signing sub-key ID when a new
one is generated. Also announce the change of signing sub-key (e.g. on
the Calmares site or as part of a release announcement).
- Send the updated key to keyservers with `gpg --send-keys <keyid>`
- Optional: sanitize the keyring for use in development machines.
Export the current subkeys of the primary key and keep **only** those
secret keys around. There is documentation
[here](https://blog.tinned-software.net/create-gnupg-key-with-sub-keys-to-sign-encrypt-authenticate/)
but be careful.
- Export the public key material with `gpg --export --armor <keyid>`,
possibly also setting an output file.
- Upload that public key to the relevant Codeberg profile.
- Upload that public key to the Calamares site.
## Placeholder Release Notes
```
# 3.2.XX (unreleased) #
This release contains contributions from (alphabetically by first name):
- No external contributors yet
## Core ##
- No core changes yet
## Modules ##
- No module changes yet
```

190
calamares/ci/RELEASE.sh Executable file
View File

@@ -0,0 +1,190 @@
#! /bin/sh
#
# SPDX-FileCopyrightText: 2018 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
### USAGE
#
# Release script for Calamares
#
# This attempts to perform the different steps of the RELEASE.md
# document automatically. It's not tested on other machines or
# setups other than [ade]'s development VM.
#
# Assumes that the version in CMakeLists.txt has been bumped,
# and that a release of that version is desired.
#
# None of the "update stuff" is done by this script; in preparation
# for the release, you should have already done:
# * updating the version
# * pulling translations
# * updating the language list
# * switching to the right branch
# The release can fail for various reasons: doesn't build, tests fail,
# or the string freeze has been violated.
#
# You can influence the script a little with these options:
# * `-B` do not build (before tagging)
# * `-P` do not package (tag, sign, tarball)
# * `-T` do not respect string freeze
# * '-b' do not build-and-test tarball
#
# The build / package settings can be influenced via environment variables:
# * BUILD_DEFAULT set to `false` to avoid first build with gcc
# * BUILD_CLANG set to `false` to avoid second build with clang
# * BUILD_ONLY set to `true` to break after building
# * TEST_TARBALL set to 'false' to skip build-and-test phase after tarring
# * QT_VERSION set to nothing (uses default), 5 or 6
#
### END USAGE
test -d .git || { echo "Not at top-level." ; exit 1 ; }
test -d src/modules || { echo "No src/modules." ; exit 1 ; }
which cmake > /dev/null 2>&1 || { echo "No cmake(1) available." ; exit 1 ; }
test -z "$BUILD_DEFAULT" && BUILD_DEFAULT=true
test -z "$BUILD_CLANG" && BUILD_CLANG=true
test -z "$BUILD_ONLY" && BUILD_ONLY=false
test -z "$TEST_TARBALL" && TEST_TARBALL=true
STRING_FREEZE=true
while getopts "hBbPT" opt ; do
case "$opt" in
h|\?)
sed -e '1,/USAGE/d' -e '/END.USAGE/,$d' < "$0"
return 0
;;
B)
BUILD_DEFAULT=false
BUILD_CLANG=false
;;
b)
TEST_TARBALL=false
;;
P)
BUILD_ONLY=true
;;
T)
STRING_FREEZE=false
;;
esac
done
if $STRING_FREEZE ; then
sh ci/txcheck.sh || { echo "! String freeze failed." ; exit 1 ; }
fi
# Via environment, not command-line
case "$QT_VERSION" in
5) extra_cmake_args="-DWITH_QT6=OFF" ;;
6) extra_cmake_args="-DWITH_QT6=ON" ;;
"") extra_cmake_args="" ;;
*) echo "Invalid QT_VERSION environment '${QT_VERSION}'" ; exit 1 ; ;;
esac
### Setup
#
#
BUILDDIR=$(mktemp -d ./cala-tmp-XXXXXX)
# This is the signing key ID associated with the the maintainer Adriaan de Groot,
# which is used to create all "verified" tags in the Calamares repo.
KEY_ID="55734316C0AE465B"
# Try to make gpg cache the signing key, so we can leave the process
# to run and sign.
rm -f CMakeLists.txt.gpg
gpg -s -u $KEY_ID CMakeLists.txt
test -f CMakeLists.txt.gpg || { echo "Could not sign (check GPG key validity)"; exit 1 ; }
### Get version number for this release
#
# Do this early, in a clean build-dir, since it doesn't cost much.
# Redirect stderr from CMake script mode, because the message()
# in CMakeLists.txt that prints the version, goes to stderr.
rm -rf "$BUILDDIR"
mkdir "$BUILDDIR" || { echo "Could not create build directory." ; exit 1 ; }
V=$( cd "$BUILDDIR" && cmake -P ../CMakeLists.txt 2>&1 )
test -n "$V" || { echo "Could not obtain version in $BUILDDIR ." ; exit 1 ; }
### Build with default compiler
#
#
if test "x$BUILD_DEFAULT" = "xtrue" ; then
rm -rf "$BUILDDIR"
mkdir "$BUILDDIR" || { echo "Could not create build directory." ; exit 1 ; }
( cd "$BUILDDIR" && cmake .. $extra_cmake_args && make -j4 ) || { echo "Could not perform test-build in $BUILDDIR." ; exit 1 ; }
( cd "$BUILDDIR" && make test ) || { echo "Tests failed in $BUILDDIR ." ; exit 1 ; }
fi
### Build with clang
#
#
if test "x$BUILD_CLANG" = "xtrue" ; then
if which clang++ > /dev/null 2>&1 ; then
# Do build again with clang
rm -rf "$BUILDDIR"
mkdir "$BUILDDIR" || { echo "Could not create build directory." ; exit 1 ; }
( cd "$BUILDDIR" && CC=clang CXX=clang++ cmake .. $extra_cmake_args && make -j4 ) || { echo "Could not perform test-build in $BUILDDIR." ; exit 1 ; }
( cd "$BUILDDIR" && make test ) || { echo "Tests failed in $BUILDDIR (clang)." ; exit 1 ; }
fi
fi
if test "x$BUILD_ONLY" = "xtrue" ; then
echo "Builds completed, release stopped. Build remains in $BUILDDIR ."
exit 1
fi
if test -f "$BUILDDIR/CMakeCache.txt" ; then
# Some build has created it, so that's good
:
else
# Presumably -B was given; just do the cmake part
rm -rf "$BUILDDIR"
mkdir "$BUILDDIR" || { echo "Could not create build directory." ; exit 1 ; }
( cd "$BUILDDIR" && cmake .. $extra_cmake_args ) || { echo "Could not run cmake in $BUILDDIR ." ; exit 1 ; }
fi
### Create signed tag
#
git tag -u "$KEY_ID" -m "Release v$V" "v$V" || { echo "Could not sign tag v$V." ; exit 1 ; }
### Create the tarball
#
#
TAR_V="calamares-$V"
TAR_FILE="$TAR_V.tar.gz"
git archive -o "$TAR_FILE" --prefix "$TAR_V/" "v$V" || { echo "Could not create tarball." ; exit 1 ; }
test -f "$TAR_FILE" || { echo "Tarball was not created." ; exit 1 ; }
SHA256=$(sha256sum "$TAR_FILE" | cut -d" " -f1)
### Build the tarball
#
#
if test "x$TEST_TARBALL" = "xtrue" ; then
D=$(date +%Y%m%d-%H%M%S)
TMPDIR=$(mktemp -d ./cala-tar-XXXXXX)
test -d "$TMPDIR" || { echo "Could not create tarball-build directory." ; exit 1 ; }
tar xzf "$TAR_FILE" -C "$TMPDIR" || { echo "Could not unpack tarball." ; exit 1 ; }
test -d "$TMPDIR/$TAR_V" || { echo "Tarball did not contain source directory." ; exit 1 ; }
( cd "$TMPDIR/$TAR_V" && cmake . && make -j4 && make test ) || { echo "Tarball build failed in $TMPDIR ." ; exit 1 ; }
fi
gpg -s -u $KEY_ID --detach --armor $TAR_FILE # Sign the tarball
### Cleanup
#
rm -rf "$BUILDDIR" # From test-builds
rm -rf "$TMPDIR" # From tarball
### Print subsequent instructions
#
#
cat <<EOF
# Next steps for this release:
git push origin v$V
# Upload tarball $TAR_FILE and the signature $TAR_FILE.asc
# Announce on the website (and the releases page of the forge).
# SHA256: $SHA256
EOF
exit 0

92
calamares/ci/abicheck.sh Executable file
View File

@@ -0,0 +1,92 @@
#! /bin/sh
#
# SPDX-FileCopyrightText: 2021 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# Compares the ABI of the current working tree with the ABI
# from a base-version. Uses libabigail for the actual comparison.
#
# To use the tool, just run the script. It will build Calamares at
# least once, maybe twice (if it needs the base-version ABI information
# and hasn't cached it).
# The build settings can be influenced via environment variables:
# * QT_VERSION set to nothing (uses default), 5 or 6
case "$QT_VERSION" in
5) extra_cmake_args="-DWITH_QT6=OFF" ;;
6) extra_cmake_args="-DWITH_QT6=ON" ;;
"") extra_cmake_args="" ;;
*) echo "Invalid QT_VERSION environment '${QT_VERSION}'" ; exit 1 ; ;;
esac
# The base version can be a tag or git-hash; it will be checked-out
# in a worktree.
#
# Note that the hash here corresponds to v3.3.3 . That was a release
# with hidden visibility enabled and a first step towards more-stable ABI.
BASE_VERSION=8741c7ec1a94ee5f27e98ef3663d1a8f4738d2c2
### Build a tree and cache the ABI info into ci/
#
#
do_build() {
LABEL=$1
SOURCE_DIR=$2
BUILD_DIR=build-abi-$LABEL
rm -rf $BUILD_DIR
rm -f $BUILD_DIR.log
echo "# Running CMake for $LABEL"
cmake -S $SOURCE_DIR -B $BUILD_DIR -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-Og -g -gdwarf" -DCMAKE_C_FLAGS="-Og -g -gdwarf" $extra_cmake_args > /dev/null 2>&1
test -f $BUILD_DIR/Makefile || { echo "! failed to CMake $LABEL" ; exit 1 ; }
echo "# Running make for $LABEL"
# Two targets make knows about at top-level
if make -C $BUILD_DIR -j12 calamares calamaresui > $BUILD_DIR.log 2>&1
then
ls -1 $BUILD_DIR/libcalamares*.so.*
# Copy the un-versioned files; .so is a symlink to the just-built one
for lib in $BUILD_DIR/libcalamares*.so
do
cp $lib ci/`basename $lib`.$LABEL
done
rm -rf $BUILD_DIR $BUILD_DIR.log
echo "# .. build successful for $LABEL"
else
echo "! failed to build $LABEL"
exit 1
fi
}
### Build current tree and get ABI info
#
#
do_build current .
### Build ABI base version
#
# We cache this to save on some build time, if we are chasing a
# single branch from an unchanging base version.
#
if test -f ci/libcalamares.so.$BASE_VERSION
then
# The ABI version is cached, so we're good
:
else
git worktree remove --force tree-abi-$BASE_VERSION > /dev/null 2>&1
git worktree add tree-abi-$BASE_VERSION $BASE_VERSION > /dev/null 2>&1 || { echo "! could not create worktree for $BASE_VERSION" ; exit 1 ; }
do_build $BASE_VERSION tree-abi-$BASE_VERSION
git worktree remove --force tree-abi-$BASE_VERSION > /dev/null 2>&1
fi
### Compare & Report
#
# abidiff compares the Application Binary Interfaces (ABI) of two
# shared libraries in ELF format. It emits a meaningful report describing
# the differences between the two ABIs.
#
# -l prints only the leaf changes, leaving out explanations of why.
#
abidiff -l ci/libcalamares.so.$BASE_VERSION ci/libcalamares.so.current

17
calamares/ci/astylerc Normal file
View File

@@ -0,0 +1,17 @@
# SPDX-FileCopyrightText: 2014 Aurélien Gâteau <agateau@kde.org>
# SPDX-FileCopyrightText: 2019 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
# Do not create a backup file
suffix=none
indent=spaces=4
# Brackets
style=break
add-braces
# Spaces
pad-paren-in
pad-header
align-pointer=type

68
calamares/ci/build.sh Executable file
View File

@@ -0,0 +1,68 @@
#! /bin/sh
#
# Generic build. The build is driven by environment variables:
# - SRCDIR (e.g. /src)
# - BUILDDIR (e.g. /build)
# - CMAKE_ARGS (e.g. "-DWITH_QT6=ON -DCMAKE_BUILD_TYPE=Debug")
#
# If SRCDIR is not set, it is assumed to be the directory above
# wherever this script is being run from (this script is in ci/).
#
# If BUILDDIR is not set, and /build exists (e.g. in the recommended
# Docker setup) then /build is used.
#
# If CMAKE_ARGS is not set, but the script is given an argument
# that exists as a workflow (e.g. "nightly-opensuse-qt6" or
# "nightly-debian.yml") and yq is installed, then the CMAKE_ARGS
# are extracted from that workflow file.
#
# If CMAKE_ARGS is not set, and the argument to the script is "-",
# then assume a build should be done under $SRCDIR/build,
# with default arguments.
#
# Summary, pick one:
# - set environment variables, run "build.sh"
# - set no variables, run "build.sh <workflow-name>"
# - set no variables, run "build.sh -"
if test -z "$SRCDIR" ; then
_d=$(dirname "$0" )
_d=$(dirname "$_d" )
test -f "$_d/CMakeLists.txt" && SRCDIR="$_d"
fi
if test -z "$BUILDDIR" ; then
test -d "/build" && BUILDDIR=/build
fi
if test -z "$CMAKE_ARGS" -a "x-" = "x$1" ; then
BUILDDIR="$SRCDIR/build"
CMAKE_ARGS="-DBOGUS=unused"
fi
if test -z "$CMAKE_ARGS" -a -n "$1" ; then
_d="$SRCDIR/.github/workflows/$1"
test -f "$_d" || _d="$SRCDIR/.github/workflows/$1.yml"
test -f "$_d" || { echo "! No workflow $1" ; exit 1 ; }
if test -x "$(which yq)" ; then
CMAKE_ARGS=$(yq ".env.CMAKE_ARGS" "$_d")
else
CMAKE_ARGS=$(python3 -c 'import yaml ; f=open("'$_d'","r"); print(yaml.safe_load(f)["env"]["CMAKE_ARGS"]);')
fi
fi
# Sanity check
test -n "$BUILDDIR" || { echo "! \$BUILDDIR not set" ; exit 1 ; }
test -n "$SRCDIR" || { echo "! \$SRCDIR not set" ; exit 1 ; }
mkdir -p "$BUILDDIR"
test -f "$SRCDIR/CMakeLists.txt" || { echo "! Missing $SRCDIR/CMakeLists.txt" ; exit 1 ; }
BUILD_MESSAGE="No commit info"
test -n "$GIT_HASH" && BUILD_MESSAGE=$( git log -1 --abbrev-commit --pretty=oneline --no-decorate "$GIT_HASH" )
echo "::" ; echo ":: $BUILD_MESSAGE" ; echo "::"
cmake -S "$SRCDIR" -B "$BUILDDIR" -G Ninja $CMAKE_ARGS || exit 1
ninja -C "$BUILDDIR" || exit 1
ninja -C "$BUILDDIR" install || exit 1
echo "::" ; echo ":: $BUILD_MESSAGE" ; echo "::"

102
calamares/ci/calamaresstyle Executable file
View File

@@ -0,0 +1,102 @@
#!/bin/sh
#
# SPDX-FileCopyrightText: 2014 Aurélien Gâteau <agateau@kde.org>
# SPDX-FileCopyrightText: 2019 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# Apply Calamares-style formatting to sources. Requires clang-format-15.
#
# You can pass in directory names, in which case the files
# in that directory (NOT below it) are processed.
#
# If the environment variable CLANG_FORMAT is set to a (full path) and
# that path is executable, it will be used if possible.
#
LANG=C
LC_ALL=C
LC_NUMERIC=C
export LANG LC_ALL LC_NUMERIC
BASEDIR=$(dirname $0)
TOPDIR=$( cd $BASEDIR/.. && pwd -P )
test -d "$BASEDIR" || { echo "! Could not determine base for $0" ; exit 1 ; }
test -d "$TOPDIR" || { echo "! Cound not determine top-level source dir" ; exit 1 ; }
test -f "$TOPDIR/.clang-format" || { echo "! No .clang-format support files in $TOPDIR" ; exit 1 ; }
# Start with CLANG_FORMAT, if it is specified
CF_VERSIONS=""
if test -n "$CLANG_FORMAT" && test -x "$CLANG_FORMAT" ; then
CF_VERSIONS="$CLANG_FORMAT"
fi
# And a bunch of other potential known versions of clang-format, newest first
CF_VERSIONS="$CF_VERSIONS clang-format-17"
CF_VERSIONS="$CF_VERSIONS clang-format-16 clang-format-16.0.6 "
CF_VERSIONS="$CF_VERSIONS clang-format15 clang-format-15 "
# Generic name of clang-format
CF_VERSIONS="$CF_VERSIONS clang-format"
for _cf in $CF_VERSIONS
do
# Not an error if this particular clang-format isn't found
CF=$( which $_cf 2> /dev/null || true )
test -n "$CF" && break
done
test -n "$CF" || { echo "! No clang-format ($CF_VERSIONS) found in PATH"; exit 1 ; }
test -x "$CF" || { echo "! $CF is not executable."; exit 1 ; }
### CLANG-FORMAT-WRANGLING
#
# Version 7 and earlier doesn't understand all the options we would like.
# Version 12 handled lambdas nicely and was the norm for Calamares 3.2.
# Version 13 was also ok.
# Version 14 behaves differently with short-functions-in-class,
# spreading functions out that 13 keeps on one line. To avoid
# ping-pong commits, forbid 14.
# Version 15 is available on recent-ish Ubuntus and FreeBSD, pick it.
# It also supports inserting braces, which is the one thing we kept
# astyle around for.
# Version 16 is available on openSUSE and is ok as well.
# Version 17 is available on FreeBSD and KaOS and is ok as well.
format_version=`"$CF" --version | tr -dc '[^.0-9]' | cut -d . -f 1`
case "$format_version" in
15|16|17 )
:
;;
* )
echo "! Clang-format version '$format_version' unsupported, versions 15-17 are ok."
exit 1
;;
esac
### FILE PROCESSING
#
#
set -e
any_dirs=no
for d in "$@"
do
test -d "$d" && any_dirs=yes
done
style_some()
{
if test -n "$*" ; then
$CF -i -style=file "$@"
fi
}
if test "x$any_dirs" = "xyes" ; then
for d in "$@"
do
if test -d "$d" ; then
style_some $( find "$d" -maxdepth 1 -type f -name '*.cpp' -o -name '*.h' )
else
style_some "$d"
fi
done
else
style_some "$@"
fi

135
calamares/ci/configvalidator.py Executable file
View File

@@ -0,0 +1,135 @@
#! /usr/bin/env python3
#
# SPDX-FileCopyrightText: 2020 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
usage = """
Validates a Calamares config file -- YAML syntax -- against a schema.
The schema is also written in YAML syntax, but the schema itself
is JSON-schema. This is possible because all JSON is YAML, and most
YAML is JSON. The limited subset of YAML that Calamares uses is
JSON-representable, anyway.
Usage:
configvalidator.py <schema> <file> ...
configvalidator.py -m <module>
configvalidator.py -x
Exits with value 0 on success, otherwise:
1 on missing dependencies
2 on invalid command-line arguments
3 on missing files
4 if files have invalid syntax
5 if files fail to validate
Use -x as only command-line argument to check the imports only.
Use -m <module> as shorthand for standard paths in src/modules/<module>/
"""
# The schemata originally lived outside the Calamares repository,
# without documented tooling. By putting them in the repository
# with the example files and explicit tooling, there's a better
# chance of them catching problems and acting as documentation.
dependencies = """
Dependencies for this tool are: py-yaml and py-jsonschema.
https://pyyaml.org/
https://github.com/Julian/jsonschema
Simple installation is `pip install pyyaml jsonschema`
"""
ERR_IMPORT, ERR_USAGE, ERR_FILE_NOT_FOUND, ERR_SYNTAX, ERR_INVALID = range(1,6)
### DEPENDENCIES
#
#
try:
from jsonschema import validate, SchemaError, ValidationError
from yaml import safe_load, YAMLError
except ImportError as e:
print(e)
print(dependencies)
exit(ERR_IMPORT)
from os.path import exists
import sys
### INPUT VALIDATION
#
#
if len(sys.argv) < 3:
# Special-case: called with -x to just test the imports
if len(sys.argv) == 2 and sys.argv[1] == "-x":
exit(0)
print(usage)
exit(ERR_USAGE)
if len(sys.argv) == 3 and sys.argv[1] == "-m":
module = sys.argv[2]
schema_file_name = f"src/modules/{module}/{module}.schema.yaml"
config_file_names = [ f"src/modules/{module}/{module}.conf" ]
else:
schema_file_name = sys.argv[1]
config_file_names = sys.argv[2:]
if not exists(schema_file_name):
print(usage)
print("\nSchema file '{}' does not exist.".format(schema_file_name))
exit(ERR_FILE_NOT_FOUND)
for f in config_file_names:
if not exists(f):
print(usage)
print("\nYAML file '{}' does not exist.".format(f))
exit(ERR_FILE_NOT_FOUND)
### FILES SYNTAX CHECK
#
#
with open(schema_file_name, "r") as data:
try:
schema = safe_load(data)
except YAMLError as e:
print("Schema error: {} {}.".format(e.problem, e.problem_mark))
print("\nSchema file '{}' is invalid YAML.".format(schema_file_name))
exit(ERR_SYNTAX)
try:
validate(instance={}, schema=schema)
# While developing the schemata, get full exceptions from schema failure
except SchemaError as e:
print(e)
print("\nSchema file '{}' is invalid JSON-Schema.".format(schema_file_name))
exit(ERR_INVALID)
except ValidationError:
# Just means that empty isn't valid, but the Schema itself is
pass
configs = []
for f in config_file_names:
config = None
with open(f, "r") as data:
try:
config = safe_load(data)
except YAMLError as e:
print("YAML error: {} {}.".format(e.problem, e.problem_mark))
print("\nYAML file '{}' is invalid.".format(f))
exit(ERR_SYNTAX)
if config is None:
print("YAML file '{}' is empty.".format(f))
configs.append(config)
assert len(configs) == len(config_file_names), "Not all configurations loaded."
### SCHEMA VALIDATION
#
#
for c, f in zip(configs, config_file_names):
try:
validate(instance=c, schema=schema)
except ValidationError as e:
print(e)
print("\nConfig file '{}' does not validate in schema.".format(f))
exit(ERR_INVALID)

9
calamares/ci/coverity-model.c Normal file
View File

@@ -0,0 +1,9 @@
/* Model file for Coverity checker.
See https://scan.coverity.com/tune
Calamares doesn't seem to geenerate any false positives,
so the model-file is empty.
SPDX-FileCopyrightText: 2017 Adriaan de Groot <groot@kde.org>
SPDX-License-Identifier: BSD-2-Clause
*/

46
calamares/ci/deps-debian11.sh Executable file
View File

@@ -0,0 +1,46 @@
#! /bin/sh
#
# Install dependencies for the nightly-debian (11) build
#
apt-get update
apt-get -y install git-core jq curl ninja
apt-get -y install \
build-essential \
cmake \
extra-cmake-modules \
gettext \
libatasmart-dev \
libappstreamqt-dev \
libboost-python-dev \
libicu-dev \
libparted-dev \
libpolkit-qt5-1-dev \
libqt5svg5-dev \
libqt5webkit5-dev \
libyaml-cpp-dev \
ninja-build \
os-prober \
pkg-config \
python3-dev \
qtbase5-dev \
qtdeclarative5-dev \
qttools5-dev \
qttools5-dev-tools
# Same name as on KDE neon, different version
apt-get -y install libkpmcore-dev
# Additional dependencies (KF5, +)
apt-get -y install \
libkf5config-dev \
libkf5coreaddons-dev \
libkf5i18n-dev \
libkf5iconthemes-dev \
libkf5parts-dev \
libkf5service-dev \
libkf5solid-dev \
libkf5crash-dev \
libkf5package-dev \
libkf5plasma-dev \
libpwquality-dev \
libqt5webenginewidgets5 \
qtwebengine5-dev
true

13
calamares/ci/deps-endeavouros.sh Executable file
View File

@@ -0,0 +1,13 @@
#! /bin/sh
#
# Install dependencies for building on EndeavourOS
#
# There is no docker image for EndeavoudOS, and the live ISO
# for Cassini Nova is KF5 / Qt5 based, but we can build there.
# It even has most of the build-deps already installed.
pacman -Syu --noconfirm jq
pacman -S --noconfirm git cmake ninja jq || exit 1
pacman -S --noconfirm gcc yaml-cpp icu || exit 1
pacman -S --noconfirm extra-cmake-modules || exit 1
pacman -S --noconfirm python-jsonschema || exit 1

18
calamares/ci/deps-fedora-qt6-boost.sh Executable file
View File

@@ -0,0 +1,18 @@
#! /bin/sh
#
# Install dependencies for the nightly-fedora-qt6-boost build
#
yum install -y bison flex git make cmake gcc-c++ ninja-build
yum install -y yaml-cpp-devel libpwquality-devel parted-devel python-devel gettext gettext-devel python3-pyyaml
yum install -y libicu-devel libatasmart-devel
yum install -y boost-devel
# Qt6/KF6 dependencies
yum install -y qt6-qtbase-devel qt6-linguist qt6-qtbase-private-devel qt6-qtdeclarative-devel qt6-qtsvg-devel qt6-qttools-devel
yum install -y extra-cmake-modules kf6-kcoreaddons-devel kf6-kdbusaddons-devel kf6-kcrash-devel
yum install -y kf6-kconfig-devel kf6-ki18n-devel kf6-kwidgetsaddons-devel kf6-kservice-devel
yum install -y polkit-qt6-1-devel appstream-qt-devel
# Runtime dependencies for QML modules
yum install -y kf6-kirigami2-devel || true
yum install -y qt6-qt5compat-devel || true
true

17
calamares/ci/deps-fedora-qt6.sh Executable file
View File

@@ -0,0 +1,17 @@
#! /bin/sh
#
# Install dependencies for the nightly-fedora-qt6 build
#
yum install -y bison flex git make cmake gcc-c++ ninja-build
yum install -y yaml-cpp-devel libpwquality-devel parted-devel python-devel gettext gettext-devel python3-pyyaml
yum install -y libicu-devel libatasmart-devel
# Qt6/KF6 dependencies
yum install -y qt6-qtbase-devel qt6-linguist qt6-qtbase-private-devel qt6-qtdeclarative-devel qt6-qtsvg-devel qt6-qttools-devel
yum install -y extra-cmake-modules kf6-kcoreaddons-devel kf6-kdbusaddons-devel kf6-kcrash-devel
yum install -y kf6-kconfig-devel kf6-ki18n-devel kf6-kwidgetsaddons-devel kf6-kservice-devel
yum install -y polkit-qt6-1-devel appstream-qt-devel
# Runtime dependencies for QML modules
yum install -y kf6-kirigami2-devel || true
yum install -y qt6-qt5compat-devel || true
true

25
calamares/ci/deps-kaos.sh Executable file
View File

@@ -0,0 +1,25 @@
#! /bin/sh
#
# Install dependencies for building on KaOS
#
# There is no docker image for KaOS, and the live ISO comes
# with many useful things already installed, so the list
# here is short. Use pacman -Syu once to update the whole
# system to get latest libraries.
#
pacman -Syu --noconfirm git cmake ninja # No jq available
pacman -S --noconfirm \
"gcc" \
"boost" \
"qt6-tools" \
"yaml-cpp" \
"kpmcore" \
"icu"
pacman -S --noconfirm \
"extra-cmake-modules" \
"kiconthemes6" \
"kservice6" \
"kio6" \
"kparts6" \
"qt6-webengine"
true

36
calamares/ci/deps-neon.sh Executable file
View File

@@ -0,0 +1,36 @@
#! /bin/sh
#
# Install dependencies for the nightly-neon build
#
apt-get update
apt-get -y install git-core jq ninja
apt-get -y install \
build-essential \
cmake \
extra-cmake-modules \
gettext \
kio-dev \
libatasmart-dev \
libboost-python-dev \
libkf5config-dev \
libkf5coreaddons-dev \
libkf5i18n-dev \
libkf5iconthemes-dev \
libkf5parts-dev \
libkf5service-dev \
libkf5solid-dev \
libkpmcore-dev \
libparted-dev \
libpolkit-qt5-1-dev \
libqt5svg5-dev \
libqt5webkit5-dev \
libyaml-cpp-dev \
ninja-build \
os-prober \
pkg-config \
python3-dev \
qtbase5-dev \
qtdeclarative5-dev \
qttools5-dev \
qttools5-dev-tools
true

22
calamares/ci/deps-opensuse-qt6.sh Executable file
View File

@@ -0,0 +1,22 @@
#! /bin/sh
#
# Install dependencies for the nightly-opensuse-qt6 build
#
# Add a Qt6/KF6 repo
zypper --non-interactive addrepo -f -G https://download.opensuse.org/repositories/KDE:/Unstable:/Frameworks/openSUSE_Factory/KDE:Unstable:Frameworks.repo
zypper --non-interactive addrepo -f -G https://download.opensuse.org/repositories/KDE:/Qt6/openSUSE_Tumbleweed/KDE:Qt6.repo
zypper --non-interactive refresh
zypper --non-interactive up
zypper --non-interactive in git-core jq yq curl ninja
# From deploycala.py
zypper --non-interactive in bison flex git make cmake gcc-c++
zypper --non-interactive in yaml-cpp-devel libpwquality-devel parted-devel python3-devel
zypper --non-interactive in libicu-devel libatasmart-devel
# Qt6/KF6 dependencies
zypper --non-interactive in kf6-extra-cmake-modules
zypper --non-interactive in "qt6-declarative-devel" "cmake(Qt6Concurrent)" "cmake(Qt6Gui)" "cmake(Qt6Network)" "cmake(Qt6Svg)" "cmake(Qt6Linguist)"
zypper --non-interactive in "cmake(KF6CoreAddons)" "cmake(KF6DBusAddons)" "cmake(KF6Crash)"
zypper --non-interactive in "cmake(KF6Parts)" # Also installs KF5 things
zypper --non-interactive in "cmake(PolkitQt6-1)" appstream-qt6-devel
true

50
calamares/ci/deps-opensuse.sh Executable file
View File

@@ -0,0 +1,50 @@
#! /bin/sh
#
# Install dependencies for the nightly-opensuse build
#
zypper --non-interactive up
zypper --non-interactive in git-core jq curl ninja
# From deploycala.py
zypper --non-interactive in \
"bison" \
"flex" \
"git" \
"make" \
"cmake" \
"gcc-c++"
zypper --non-interactive in \
"libqt5-qtbase-devel" \
"libqt5-qtdeclarative-devel" \
"cmake(Qt5LinguistTools)" \
"cmake(Qt5Svg)" \
"cmake(Qt5WebEngine)" \
"cmake(PolkitQt5-1)" \
"yaml-cpp-devel" \
"libpwquality-devel" \
"parted-devel" \
"python311-devel" \
"libboost_headers-devel" \
"libboost_python3-devel"
zypper --non-interactive in \
"extra-cmake-modules" \
"cmake(KF5Crash)" \
"cmake(KF5DBusAddons)" \
"cmake(KF5Package)" \
"cmake(KF5Parts)" \
"cmake(KF5Plasma)" \
"cmake(KF5Service)" \
"cmake(KPMcore)" \
"cmake(LibKWorkspace)"
# Additional dependencies
zypper --non-interactive in \
libicu-devel \
libAppStreamQt-devel \
libatasmart-devel
# Not actual dependencies, but good to have
zypper --non-interactive in python311-PyYAML python311-jsonschema
# vi to edit things inside the docker
zypper --non-interactive in vim
# noto so that running Calamares in the docker is readable
zypper --non-interactive in noto-sans-fonts
true

47
calamares/ci/deps-ubuntu.sh Executable file
View File

@@ -0,0 +1,47 @@
#! /bin/sh
#
# Install dependencies for the nightly-ubuntu (devel) build
# These build dependencies are grabbed directly from the Debian package
#
apt-get update
apt-get -y install git-core jq curl ninja
apt-get -y install \
build-essential \
cmake \
extra-cmake-modules \
gettext \
libappstreamqt5-dev \
libkf5config-dev \
libkf5coreaddons-dev \
libkf5crash-dev \
libkf5i18n-dev \
libkf5iconthemes-dev \
libkf5kio-dev \
libkf5parts-dev \
libkf5plasma-dev \
libkf5service-dev \
libkf5solid-dev \
libkpmcore-dev \
libparted-dev \
libpolkit-qt5-1-dev \
libpwquality-dev \
libqt5svg5-dev \
libqt5webkit5-dev \
libyaml-cpp-dev \
os-prober \
pkg-config \
pkg-kde-tools \
polkitd \
python3-dev \
python3-jsonschema \
python3-yaml \
qml-module-qtquick-layouts \
qml-module-qtquick-privatewidgets \
qml-module-qtquick-window2 \
qml-module-qtquick2 \
qtbase5-dev \
qtdeclarative5-dev \
qtlocation5-dev \
qttools5-dev \
qttools5-dev-tools
true

9
calamares/ci/libcalamares/__init__.py Normal file
View File

@@ -0,0 +1,9 @@
# SPDX-FileCopyrightText: no
# SPDX-License-Identifier: CC0-1.0
#
# Stubs for part of the Python API from libcalamares
# (although the **actual** API is presented through
# Boost::Python, not as a bare C-extension) so that
# pylint doesn't complain about libcalamares internals.
VERSION_SHORT="1.0"

24
calamares/ci/libcalamares/globalstorage.py Normal file
View File

@@ -0,0 +1,24 @@
# SPDX-FileCopyrightText: no
# SPDX-License-Identifier: CC0-1.0
#
# Stubs for part of the Python API from libcalamares
# (although the **actual** API is presented through
# Boost::Python, not as a bare C-extension) so that
# pylint doesn't complain about libcalamares internals.
def count(): return 1
def keys(): return []
def contains(_): return True
def value(key):
if key in ("branding",):
return dict()
if key in ("partitions",):
return list()
return ""
def insert(key, value): pass
def remove(_): pass

15
calamares/ci/libcalamares/job.py Normal file
View File

@@ -0,0 +1,15 @@
# SPDX-FileCopyrightText: no
# SPDX-License-Identifier: CC0-1.0
#
# Stubs for part of the Python API from libcalamares
# (although the **actual** API is presented through
# Boost::Python, not as a bare C-extension) so that
# pylint doesn't complain about libcalamares internals.
configuration = dict()
def setprogress(_): pass
def pretty_name(): return ""
def working_path(): return ""

27
calamares/ci/libcalamares/utils.py Normal file
View File

@@ -0,0 +1,27 @@
# SPDX-FileCopyrightText: no
# SPDX-License-Identifier: CC0-1.0
#
# Stubs for part of the Python API from libcalamares
# (although the **actual** API is presented through
# Boost::Python, not as a bare C-extension) so that
# pylint doesn't complain about libcalamares internals.
def debug(_): pass
def warning(_): pass
def error(_): pass
def gettext_path(): pass
def gettext_languages(): pass
def target_env_call(_): return 0
def check_target_env_call(_): pass
def target_env_process_output(cmd, *args): return 0
def host_env_process_output(cmd, *args): return 0
def mount(device, mountpoint, fstype, options): return 0

155
calamares/ci/txcheck.sh Executable file
View File

@@ -0,0 +1,155 @@
#! /bin/sh
### LICENSE
# === This file is part of Calamares - <https://calamares.io> ===
#
# SPDX-FileCopyrightText: 2019-2020 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# This file is Free Software: you can redistribute it and/or modify
# it under the terms of the 2-clause BSD License.
#
### END LICENSE
### USAGE
#
# Does the translation tag (from a previous txpush) exist?
# This assumes that the release host has also locally done
# a translations push, which works for the current development
# workflow .. but it could be improved by looking for one of
# the typical txpush log messages instead of the tag.
#
# Use --cleanup as an argument to clean things up.
#
# Normal use:
# $ sh ci/txcheck.sh
# If there are differences, fix them and then clean up:
# $ sh ci/txcheck.sh --cleanup
#
### END USAGE
# The files that are translated; should match the contents of .tx/config
TX_FILE_LIST="lang/calamares_en.ts lang/python.pot calamares.desktop"
### COMMAND ARGUMENTS
#
# We need to define tx_cleanup for the --cleanup argument, although it's
# normally used much later in the script.
tx_cleanup()
{
# Cleanup artifacs of checking
git worktree remove --force build-txcheck-head
git worktree remove --force build-txcheck-prev
git branch -D build-txcheck-head > /dev/null 2>&1
}
if test "x$1" = "x--cleanup" ; then
tx_cleanup
exit 0
fi
if test "x$1" = "x--help" ; then
sed -e '1,/USAGE/d' -e '/END.USAGE/,$d' < "$0"
fi
test -z "$1" || { echo "! Usage: txcheck.sh [--cleanup]" ; exit 1 ; }
### FIND EXECUTABLES
#
#
XMLLINT=""
for _xmllint in xmllint
do
$_xmllint --version > /dev/null 2>&1 && XMLLINT=$_xmllint
test -n "$XMLLINT" && break
done
# Distinguish GNU date from BSD date
if date +%s -d "1 week ago" > /dev/null 2>&1 ; then
last_week() { date +%s -d "1 week ago" ; }
else
last_week() { date -v1w +%s; }
fi
# Distinguish GNU SHA executables from BSD ones
if which sha256sum > /dev/null 2>&1 ; then
SHA256=sha256sum
else
SHA256=sha256
fi
### CHECK WORKING DIRECTORY
#
#
if git describe translation > /dev/null 2>&1 ; then
:
else
echo "! No 'translation' tag exists for enforcing the string-freeze."
exit 1
fi
# The tag exists, so now check that there's no unsaved changes
if test `git describe` = `git describe --dirty` ; then
:
else
# Don't want any local changes, since those won't be
# reflected in the worktrees and we might miss a string change.
echo "! There are local changes."
exit 1
fi
# No unsaved changes; enforce a string freeze of one week
DATE_PREV=$( git log -1 translation --date=unix | sed -e '/^Date:/s+.*:++p' -e d )
DATE_HEAD=$( last_week )
test "$DATE_PREV" -le "$DATE_HEAD" || { echo "! Translation tag has not aged enough." ; git log -1 translation ; exit 1 ; }
# Tag is good, check that necessary files exist. The list of
# files is hard-coded, but should match what is in the Transifex config.
test -f ".tx/config" || { echo "! No Transifex configuration is present." ; exit 1 ; }
for f in $TX_FILE_LIST ; do
test -f $f || { echo "! Translation file '$f' does not exist." ; exit 1 ; }
done
### COMPARE TRANSLATIONS
#
#
# The state of translations; assume that sha256 is enough
# to distinguish changed translations when we cat all the
# string sources together.
tx_sum()
{
CURDIR=`pwd`
WORKTREE_NAME="$1"
WORKTREE_TAG="$2"
git worktree add -d $WORKTREE_NAME $WORKTREE_TAG > /dev/null 2>&1 || { echo "! Could not create worktree." ; exit 1 ; }
( cd $WORKTREE_NAME && sh "$CURDIR"/ci/txpush.sh --no-tx ) > /dev/null 2>&1 || { echo "! Could not re-create translations." ; exit 1 ; }
# Remove linenumbers from .ts (XML) and .pot
sed -i'' -e '/<location filename/d' "$WORKTREE_NAME/lang/calamares_en.ts"
sed -i'' -e '/^#: src..*[0-9]$/d' $WORKTREE_NAME/lang/python.pot
_SUM=$( cd $WORKTREE_NAME && cat $TX_FILE_LIST | $SHA256 )
echo "$_SUM"
}
# Check from the translation tag as well
HEAD_SUM=`tx_sum build-txcheck-head ""` || { echo "$HEAD_SUM" ; exit 1 ; }
PREV_SUM=`tx_sum build-txcheck-prev translation` || { echo "$HEAD_SUM" ; exit 1 ; }
# An error message will have come from the shell function
test -d build-txcheck-head || { echo "$HEAD_SUM" ; exit 1 ; }
test -d build-txcheck-prev || { echo "$PREV_SUM" ; exit 1 ; }
if test "$HEAD_SUM" = "$PREV_SUM" ; then
:
else
echo "! Translations have changed."
for f in $TX_FILE_LIST ; do
echo "! $f"
diff -u build-txcheck-prev/$f build-txcheck-head/$f
done
echo "! Run 'txcheck.sh --cleanup' to clean-up before next run"
exit 1
fi
tx_cleanup
exit 0

167
calamares/ci/txpull.sh Executable file
View File

@@ -0,0 +1,167 @@
#!/bin/sh
### LICENSE
# === This file is part of Calamares - <https://calamares.io> ===
#
# SPDX-FileCopyrightText: 2015-2016 Teo Mrnjavac <teo@kde.org>
# SPDX-FileCopyrightText: 2017-2020 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# This file is Free Software: you can redistribute it and/or modify
# it under the terms of the 2-clause BSD License.
#
### END LICENSE
### USAGE
#
# Fetch the Transifex translations for Calamares and incorporate them
# into the source tree, adding commits of the different files.
#
# Run this (occasionally) at the top-level directory to get
# new translations. See also CMakeLists.txt and ci/txstats.py
# for update instructions.
#
### END USAGE
### SANITY CHECKING
#
# The script needs a .tx/config to talk to the Transifex server;
# it also checks that it is run from the top-level of a Calamares
# checkout. In order to use the system overall, you'll also need:
# - ~/.gitconfig (For the git commits this does)
# - ~/.transifexrc (Password token for Transifex)
# - ~/.ssh (For git commits)
#
test -f "CMakeLists.txt" || { echo "! Not at Calamares top-level" ; exit 1 ; }
test -f ".tx/config" || { echo "! Not at Calamares top-level" ; exit 1 ; }
test -f "calamares.desktop" || { echo "! Not at Calamares top-level" ; exit 1 ; }
### FIND EXECUTABLES
#
#
XMLLINT=""
for _xmllint in xmllint
do
$_xmllint --version > /dev/null 2>&1 && XMLLINT=$_xmllint
test -n "$XMLLINT" && break
done
# XMLLINT is optional
### FETCH TRANSLATIONS
#
# Use Transifex client to get translations; this depends on the
# .tx/config file to locate files, and overwrites them in the
# filesystem with new (merged) translations.
export QT_SELECT=5
transifex-client pull --force --all || exit 1
### CLEANUP TRANSLATIONS
#
# Some languages have been deprecated. They may still exist in Transifex,
# so clean them up after pulling.
#
drop_language() {
rm -rf lang/python/"$1" lang/calamares_"$1".ts
grep -v "\\[$1]" calamares.desktop > calamares.desktop.new
mv calamares.desktop.new calamares.desktop
}
# Also fix the .desktop file, which has some fields removed by Transifex.
#
{ cat calamares.desktop.in ; grep "\\[[a-zA-Z_@]*]=" calamares.desktop ; } > calamares.desktop.new
mv calamares.desktop.new calamares.desktop
# And fixup the XML files like in txpush.sh
if test -n "$XMLLINT" ; then
for TS_FILE in lang/calamares_*.ts
do
$XMLLINT --c14n11 "$TS_FILE" | { echo "<!DOCTYPE TS>" ; cat - ; } | $XMLLINT --format --encode utf-8 -o "$TS_FILE".new - && mv "$TS_FILE".new "$TS_FILE"
done
fi
### COMMIT TRANSLATIONS
#
# Produce multiple commits (for the various parts of the i18n
# infrastructure used by Calamares) of the updated translations.
# Try to be a little smart about not committing trivial changes.
# Who is credited with these CI commits
AUTHOR="--author='Calamares CI <groot@kde.org>'"
# Message to put after the module name
BOILERPLATE="Automatic merge of Transifex translations"
git add --verbose lang/calamares*.ts
git commit "$AUTHOR" --message="i18n: [calamares] $BOILERPLATE" | true
rm -f lang/desktop*.desktop
awk '
BEGIN {skip=0;}
/^# Translations/ {skip=1;}
{if (!skip || (length($0)>1 && $0 != "# Translations")) {
skip=0; print $0;
}}' < calamares.desktop > calamares.desktop.new
mv calamares.desktop.new calamares.desktop
# Now group translated key-names (Name, Icon, Description, ..) by sorted
# language key rather than random-ish language-key order (which shuffles
# entries around).
#
# First, the non-translated lines
grep -v '\[.*\]=' calamares.desktop > calamares.desktop.new
# The translated lines:
# - replace (the first) [] by | so we have a consistent field separator
# - sort based on field 2, then 1 (language code, then reversed key-name)
# - replace the first | by [, the first (remaining) | by ]
# Effectively this puts the fields in this order: Name, Icon, Generic Name,
# Comment -- within each language key. This keeps churn down since the
# language codes and key-names are constant.
grep '\[.*\]=' calamares.desktop | sed -e 's/\[/|/' -e 's/\]/|/' | sort -t '|' -k 2,2 -k 1,1r | sed -e 's/|/\[/' | sed -e 's/|/\]/' >> calamares.desktop.new
mv calamares.desktop.new calamares.desktop
git add --verbose calamares.desktop
git commit "$AUTHOR" --message="i18n: [desktop] $BOILERPLATE" | true
# Transifex updates the PO-Created timestamp also when nothing interesting
# has happened, so drop the files which have just 1 line changed (the
# PO-Created line). This applies only to modules which use po-files.
git diff --numstat src/modules | awk '($1==1 && $2==1){print $3}' | xargs git checkout --
# sed either wants -i'' (GNU sed) or -i '' (BSD sed) to
# replace in a file, with no backup extension. Define
# a `reinplace` command to deal with the difference.
if test FreeBSD = `uname` ; then
reinplace() {
sed -i '' "$@"
}
else
reinplace() {
sed -i'' "$@"
}
fi
# Go through the Python modules; those with a lang/ subdir have their
# own complete gettext-based setup.
for MODULE_DIR in $(find src/modules -maxdepth 1 -mindepth 1 -type d) ; do
FILES=$(find "$MODULE_DIR" -name "*.py" -a -type f)
if test -n "$FILES" ; then
MODULE_NAME=$(basename ${MODULE_DIR})
if [ -d ${MODULE_DIR}/lang ]; then
# Convert PO files to MO files
for POFILE in $(find ${MODULE_DIR} -name "*.po") ; do
reinplace '/^"Content-Type/s/CHARSET/UTF-8/' $POFILE
# msgfmt -o ${POFILE%.po}.mo $POFILE
done
git add --verbose ${MODULE_DIR}/lang/*
git commit "$AUTHOR" --message="i18n: [${MODULE_NAME}] $BOILERPLATE" | true
fi
fi
done
for POFILE in $(find lang -name "python.po") ; do
reinplace '/^"Content-Type/s/CHARSET/UTF-8/' $POFILE
# msgfmt -o ${POFILE%.po}.mo $POFILE
done
git add --verbose lang/python*
git commit "$AUTHOR" --message="i18n: [python] $BOILERPLATE" | true

166
calamares/ci/txpush.sh Executable file
View File

@@ -0,0 +1,166 @@
#!/bin/sh
### LICENSE
# === This file is part of Calamares - <https://calamares.io> ===
#
# SPDX-FileCopyrightText: 2015-2016 Teo Mrnjavac <teo@kde.org>
# SPDX-FileCopyrightText: 2017-2020 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# This file is Free Software: you can redistribute it and/or modify
# it under the terms of the 2-clause BSD License.
#
### END LICENSE
### USAGE
#
# Extract translations from Calamares source and send them
# to Transifex. Also (forcibly) updates the git "translation"
# tag to document that source texts were updated and sent;
# this is used by txcheck.sh to ensure that there's enough
# time between updates and releases, and that strings don't
# change between updates and releases.
#
# Run this at the top-level.
#
# Use the --no-tx option to do the extraction, but not the
# pushing-to-Transifex part. This can be useful to check for
# new strings or when testing the tools themselves.
#
### END USAGE
### SANITY CHECKING
#
# The script needs a .tx/config to talk to the Transifex server;
# it also checks that it is run from the top-level of a Calamares
# checkout. In order to use the system overall, you'll also need:
# - ~/.gitconfig (For the git commits this does)
# - ~/.transifexrc (Password token for Transifex)
# - ~/.ssh (For git commits)
#
test -f "CMakeLists.txt" || { echo "! Not at Calamares top-level" ; exit 1 ; }
test -f ".tx/config" || { echo "! Not at Calamares top-level" ; exit 1 ; }
test -f "calamares.desktop" || { echo "! Not at Calamares top-level" ; exit 1 ; }
if test "x$1" = "x--no-tx" ; then
# tx is the transifex command -- eat its arguments and do nothing
tx() {
echo "Skipped tx $*"
}
# txtag is used to tag in git to measure changes -- skip it too
txtag() {
echo "Skipped tx tagging."
}
else
# tx is the regular transifex command
tx() {
transifex-client "$@"
}
# txtag is used to tag in git to measure changes
txtag() {
git tag -f translation
git push --force origin translation
}
fi
### FIND EXECUTABLES
#
#
LUPDATE=""
for _lupdate in lupdate-qt5 lupdate
do
export QT_SELECT=5
$_lupdate -version > /dev/null 2>&1 || export QT_SELECT=qt5
$_lupdate -version > /dev/null 2>&1 && LUPDATE=$_lupdate
test -n "$LUPDATE" && break
done
test -n "$LUPDATE" || { echo "! No working lupdate" ; lupdate -version ; exit 1 ; }
XMLLINT=""
for _xmllint in xmllint
do
$_xmllint --version > /dev/null 2>&1 && XMLLINT=$_xmllint
test -n "$XMLLINT" && break
done
# XMLLINT is optional
if sed --version 2>&1 | grep -q GNU ; then
reinplace() {
sed -i'' "$@"
}
else
reinplace() {
sed -i '' "$@"
}
fi
### CREATE TRANSLATIONS
#
# Use local tools (depending on type of source) to create translation
# sources, then push to Transifex
# Don't pull branding translations in,
# those are done separately.
_srcdirs="src/calamares src/libcalamares src/libcalamaresui src/modules src/qml"
$LUPDATE -no-obsolete $_srcdirs -ts lang/calamares_en.ts
grep '{1' lang/calamares_en.ts && { echo "lupdate has introduced weird markers." ; exit 1 ; }
# Non-Transifex special-cases
#
# - timezone names can be translated, but that's 700+ strings I don't want
# to inflict on translators normally
# - keyboard layouts can be translated, but that's 767 strings
#
# For both of these, the language / translation only needs to be updated
# when the source data is updated, which is very very rarely.
# $LUPDATE -no-obsolete -extensions cxxtr src/libcalamares/locale -ts lang/tz_en.ts
# $LUPDATE -no-obsolete -extensions cxxtr src/modules/keyboard -ts lang/kb_en.ts
if test -n "$XMLLINT" ; then
TS_FILE="lang/calamares_en.ts"
$XMLLINT --c14n11 "$TS_FILE" | { echo "<!DOCTYPE TS>" ; cat - ; } | $XMLLINT --format --encode utf-8 -o "$TS_FILE".new - && mv "$TS_FILE".new "$TS_FILE"
fi
tx push --source -r calamares.calamares || exit 1
tx push --source -r calamares.fdo || exit 1
### PYTHON MODULES
#
# The Python tooling depends on the underlying distro to provide
# gettext, and handles two cases:
#
# - python modules with their own lang/ subdir, for larger translations
# - python modules without lang/, which use one shared catalog
#
PYGETTEXT="xgettext --keyword=_n:1,2 -L python"
SHARED_PYTHON=""
for MODULE_DIR in $(find src/modules -maxdepth 1 -mindepth 1 -type d | sort) ; do
FILES=$(find "$MODULE_DIR" -name "*.py" -a -type f | sort)
if test -n "$FILES" ; then
MODULE_NAME=$(basename ${MODULE_DIR})
if [ -d ${MODULE_DIR}/lang ]; then
${PYGETTEXT} -p ${MODULE_DIR}/lang -d ${MODULE_NAME} -o ${MODULE_NAME}.pot ${MODULE_DIR}/*.py
POTFILE="${MODULE_DIR}/lang/${MODULE_NAME}.pot"
if [ -f "$POTFILE" ]; then
reinplace '/^"Content-Type/s/CHARSET/UTF-8/' "$POTFILE"
tx push --source -r calamares.${MODULE_NAME}
fi
else
SHARED_PYTHON="$SHARED_PYTHON $FILES"
fi
fi
done
if test -n "$SHARED_PYTHON" ; then
${PYGETTEXT} -p lang -d python -o python.pot $SHARED_PYTHON
POTFILE="lang/python.pot"
reinplace '/^"Content-Type/s/CHARSET/UTF-8/' "$POTFILE"
tx push --source -r calamares.python
fi
txtag
exit 0

47
calamares/ci/txreduce.py Executable file
View File

@@ -0,0 +1,47 @@
#! /usr/bin/env python3
#
# SPDX-FileCopyrightText: 2020 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# Reduce a translation file -- generally, a Timezone translation -- by
# dropping untranslated strings. An untranslated string is one that
# has an empty translation **and** is marked unfinished.
#
# This is mostly useful to cut down the size of the source file:
# far and away most of the zones are not translated, and it's just a
# handful of places that get special treatment.
from xml.dom.minidom import parse
import sys
valid = True
dom = parse(sys.argv[1])
for n in dom.getElementsByTagName("translation"):
attrs = n.attributes.keys()
drop = True
if "type" not in attrs:
drop = False
elif "type" in attrs and n.attributes["type"].value != "unfinished":
# In the samples I've seen, only "unfinished" is a valid type;
# once something has been translated, the attribute vanishes (see
# the if branch, above).
print("WARNING ''{!s}'' unknown type".format(n.attributes["type"].value))
drop = False
valid = False
else:
t = n.firstChild
if t is None:
# Unfinished and empty
drop = True
else:
drop = bool(t.data)
if drop:
message = n.parentNode
message.parentNode.removeChild(message)
message.unlink()
if valid:
for line in dom.toxml().split("\n"):
if line.strip():
print(line)

284
calamares/ci/txstats.py Executable file
View File

@@ -0,0 +1,284 @@
#! /usr/bin/env python3
#
# SPDX-FileCopyrightText: 2018 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# Uses the Transifex API to get a list of enabled languages,
# and outputs CMake settings for inclusion into CMakeLists.txt.
#
# This is a Python3 script.
#
# Run it with a -v command-line option to get extra output on
# actual translation percentages.
import sys
import os
import argparse
class TXError(Exception):
pass
class TransifexGetter(object):
"""
Get language data from Transifex.
The object does all the work in __init__, after that
the only relevant data is .languages, a dictionary
of language data.
"""
def __init__(self):
token = self.get_tx_credentials()
if token is None:
raise TXError("Could not get Transifex API token")
import requests
base_url = "https://rest.api.transifex.com/resource_language_stats"
project_filter = "filter[project]=o:calamares:p:calamares"
resource_filter = "filter[resource]=o:calamares:p:calamares:r:calamares"
url = base_url + "?" + project_filter.replace(":", "%3A") + "&" + resource_filter.replace(":", "%3A")
headers = {
"accept": "application/vnd.api+json",
"authorization": "Bearer " + token
}
r = requests.get(url, headers=headers)
if r.status_code != 200:
raise TXError("Could not get Transifex data from API")
j = r.json()
data = j["data"]
self.languages = dict()
for d in data:
translated_count = d["attributes"]["translated_strings"]
total_count = d["attributes"]["total_strings"]
language_key = d["relationships"]["language"]["data"]["id"]
assert language_key.startswith("l:")
language_key = language_key[2:]
self.languages[language_key] = dict(translated=dict(stringcount=translated_count, percentage=(translated_count / total_count)))
def get_tx_credentials(self):
"""
Gets the API token out of the user's .transifexrc (this is supposed
to be secure).
"""
import configparser
import os
txconfig_name = os.path.expanduser("~/.transifexrc")
try:
with open(txconfig_name, "r") as f:
parser = configparser.ConfigParser()
parser.read_file(f)
return parser.get("https://app.transifex.com", "password")
except IOError as e:
return None
class BogusGetter(object):
"""
Fake language data.
This object pretends to retrieve data, and returns fixed language lists and percentages,
for testing purposes without hitting Transifex servers all the time.
"""
def __init__(self):
self.languages = dict()
for lang, completion in ( ("sq", 100), ("ar", 44), ("as", 28), ("de", 15), ("da", 4), ("ts", 82) ):
self.languages[lang] = dict(translated=dict(stringcount=686, percentage=(completion/100.0)))
class PrintOutputter(object):
"""
Output via print-statements.
"""
def __init__(self):
pass
def print(self, s):
print(s)
def __enter__(self):
return self
def __exit__(self, e, v, tb):
pass
class EditingOutputter(object):
"""
Edit CMakeLists in-place.
"""
def __init__(self):
with open("CMakeLists.txt", "r") as f:
lines = f.readlines()
mark = None
mark_text = None
for l in lines:
# Note that we didn't strip the lines, so need the \n here
if l.startswith("# Total ") and l.endswith(" languages\n"):
mark = lines.index(l)
mark_text = l
break
if mark is None:
raise TXError("No CMakeLists.txt lines for TX stats found")
self.pre_lines = lines[:mark]
nextmark = mark + 1
for l in lines[mark+1:]:
nextmark += 1
if l.startswith(mark_text):
break
if nextmark > mark + 150 or nextmark > len(lines) - 4:
# Try to catch runaway nextmarks: we know there should
# be four set-lines, which are unlikely to be 3 lines each;
# similarly the CMakeLists.txt is supposed to end with
# some boilerplate.
#
# However, gersemi will reformat to one-language-per-line,
# so we can get really long sections, that's why we use 150 as a limit.
raise TXError("Could not find end of TX settings in CMakeLists.txt")
self.post_lines = lines[nextmark:]
self.mid_lines = []
print("# Editing CMakeLists.txt in-place")
def print(self, s):
# Add the implicit \n from print()
self.mid_lines.append(s + "\n")
if s.startswith("#"):
print(s)
def __enter__(self):
return self
def __exit__(self, e, v, tb):
if e is None:
with open("CMakeLists.txt", "w") as f:
f.write("".join(self.pre_lines + self.mid_lines + self.post_lines))
print("# CMakeLists.txt updated")
def output_langs(all_langs, outputter, label, filterfunc):
"""
Output (via print) all of the languages in @p all_langs
that satisfy the translation-percentage filter @p filterfunc.
Prints a CMake set() command with the @p label as part
of the variable name.
Performs line-wrapping.
"""
these_langs = [l for s, l in all_langs if filterfunc(s)]
out = " ".join(["set( _tx_%s" % label, " ".join(sorted(these_langs)), ")"])
width = 68
prefix = ""
trailer = f" # {len(these_langs)} languages" # Comment at the end of the CMake line
while len(out) > width - len(prefix):
chunk = out[:out[:width].rfind(" ")]
outputter.print("%s%s" % (prefix, chunk))
out = out[len(chunk)+1:]
prefix = " "
outputter.print(f"{prefix}{out}{trailer}")
def get_tx_stats(languages, outputter, verbose):
"""
Does an API request to Transifex with the given API @p token, getting
the translation statistics for the main body of texts. Then prints
out CMake settings to replace the _tx_* variables in CMakeLists.txt
according to standard criteria.
If @p verbose is True, prints out language stats as well.
"""
# Some languages go into the "incomplete" list by definition,
# regardless of their completion status: this can have various reasons.
#
# - (Esperanto wasn't supported until Qt 5.12.2)
# - Interlingue still is not supported by the minimum Qt version
incomplete_languages = (
"ie", # Not supported by Qt at least through 5.15.0
)
all_langs = []
mark_text = "# Total %d languages" % len(languages)
outputter.print(mark_text)
for lang_name in languages:
stats = languages[lang_name]["translated"]["percentage"]
# Make the by-definition-incomplete languages have a percentage
# lower than zero; this way they end up sorted (in -v output)
# at the bottom but you can still determine the "actual" percentage.
if lang_name in incomplete_languages:
stats = -stats
all_langs.append((stats, lang_name))
if verbose:
for s, l in sorted(all_langs, reverse=True):
outputter.print("# %16s\t%6.2f" % (l, s * 100.0))
output_langs(all_langs, outputter, "complete", lambda s : s == 1.0)
output_langs(all_langs, outputter, "good", lambda s : 1.0 > s >= 0.75)
output_langs(all_langs, outputter, "ok", lambda s : 0.75 > s >= 0.05)
output_langs(all_langs, outputter, "incomplete", lambda s : 0.05 > s)
outputter.print(mark_text)
# Audit the languages that are in TX, mapped to git
for lang_name in languages:
if not os.path.exists("lang/calamares_{}.ts".format(lang_name)):
print("# !! Missing translation file for {}".format(lang_name))
if not os.path.isdir("lang/python/{}/LC_MESSAGES".format(lang_name)):
print("# !! Missing Python translation file for {}".format(lang_name))
# Audit the files that are in git, mapped to TX
special_cases = ("python.pot", "python", "CMakeLists.txt", "txload.cpp", "calamares_i18n.qrc.in")
for file_name in os.listdir("lang"):
if file_name in special_cases:
continue
elif file_name.startswith("calamares_") and file_name.endswith(".ts"):
key = file_name[10:-3]
if not key in languages and not key == "en":
print("# !! Translation file for {} not in TX".format(key))
elif file_name.startswith("tz_") and file_name.endswith(".ts"):
key = file_name[3:-3]
if not key in languages and not key == "en":
print("# !! Translation file for TZ {} not in TX".format(key))
elif file_name.startswith("kb_") and file_name.endswith(".ts"):
key = file_name[3:-3]
if not key in languages and not key == "en":
print("# !! Translation file for KB {} not in TX".format(key))
else:
print("# !! Weird translation file {} not in TX".format(file_name))
# Audit the python translation files that are in git, mapped to TX
for file_name in os.listdir("lang/python"):
if file_name not in languages:
print("# !! Translation file for Python {} not in TX".format(file_name))
return 0
def main():
parser = argparse.ArgumentParser(description="Update Transifex Statistics")
parser.add_argument("--verbose", "-v", help="Show statistics", action="store_true")
parser.add_argument("--bogus", "-n", help="Use bogus data (do not query Transifex)", action="store_true")
parser.add_argument("--edit", "-e", help="Edit CMakeLists.txt in-place", action="store_true")
args = parser.parse_args()
try:
if args.bogus:
getter = BogusGetter()
else:
getter = TransifexGetter()
if args.edit:
outputter = EditingOutputter()
else:
outputter = PrintOutputter()
with outputter:
return get_tx_stats(getter.languages, outputter, args.verbose)
except TXError as e:
print("! " + str(e))
return 1;
return 0
if __name__ == "__main__":
sys.exit(main())

29
calamares/ci/umount.sh Executable file
View File

@@ -0,0 +1,29 @@
#! /bin/sh
### LICENSE
# === This file is part of Calamares - <https://calamares.io> ===
#
# SPDX-FileCopyrightText: 2022 Adriaan de Groot <groot@kde.org>
# SPDX-License-Identifier: BSD-2-Clause
#
# This file is Free Software: you can redistribute it and/or modify
# it under the terms of the 2-clause BSD License.
#
### END LICENSE
#
# This is an "unmount" script that tries to unmount whatever
# filesystems Calamares might have left mounted (e.g. because of
# a crash, or ^C'ing the installer, or ..).
#
# Swap may have become enabled on the disks just used; assume
# we're in a live session where we don't want any.
sudo swapoff -a
# In Arch-based systems, there may be a gpg-agent started by
# pacman during installation, which lives in the chroot. Kill
# them all; again assume we're in a live session where it doesn't matter.
sudo pkill gpg-agent
# Unmount the filesystems in *reverse* order, since we need to ditch
# e.g. /run/udev before /run before the root filesystem.
sudo umount $( LC_ALL=C mount | awk '/calamares-root/{print $3}' | LC_ALL=C sort -r )