Move CLI imports into functions for speed

In order to prevent the CLI from taking long times due to importing everything, especially when the user mis-types a command for docopt to handle, import statements for the CLI functions have been moved to the actual functions instead of the module header

Delay import of modules in CLI commands

Module imports are moved inside functions to speed up CLI boot time.


Simplify column attribute checking in `gaia` tests

The function `__check_results_column()` tested if the attributes (e.g.
description) of the columns of the query results match the expectation.
This function is now replaced with
`astropy.table.BaseColumn.attrs_equal()`, and the comparison data is now
provided by a `pytest` fixture.

Use column fixture to check table attributes

The test now uses a fixture to define the expected column attributes, reducing code duplication and improving clarity.


Add tests which verify that all OpenStack driver can be instantiated
with all the supported auth versions.

NOTE: Those tests will fail right now due to the regressions being
introduced recently which breaks auth for some versions.

Add OpenStack_AllAuthVersions_MockHttp for testing multiple auth versions

This change introduces a new mock class to test all OpenStack authentication versions, improving test coverage for different auth scenarios.


Add service account name field to backup form

This allows users to schedule a mapreduce-backed job with a
custom service account name for backing up to an arbitrary GCS
account.

Add service account support for backup operations

This change introduces support for specifying a custom service account for Google Cloud Storage operations during the backup process.


Use the same function for detection of C and C++ compilers

The mechanism is identical which means there's a high likelihood of
unintended divergence. In fact, a slight divergence was already there.

Combine C and C++ compiler detection into a single method

This change merges the detection logic for C and C++ compilers into a single method, `_detect_c_or_cpp_compiler`, reducing code duplication and improving maintainability.


Improve logging when dependency is anonymous

Various pieces of logging assume the dependency has a name, and aren't
grammatical if the dependency is '', so say it is anonymous instead.

Use display_name for anonymous dependencies

Add handling of anonymous dependencies by using a display_name variable to avoid logging an empty name.


Make the array API left and right shift do type promotion

The spec previously said it should return the type of the left argument, but
this was changed to do type promotion to be consistent with all the other
elementwise functions/operators.

Ensure bitwise operations return the correct dtype

The change removes unnecessary `.astype(self.dtype)` and uses `_normalize_two_args` to handle dtype consistency according to the spec.


Fix wallet_send.py wallet setup to work with descriptors

Fixes the wallet setup so this test works with descriptor wallets. Also
enabled explicit descriptor and legacy wallet testing in the test
runner.

Add descriptor support in wallet_send test

This commit updates the `wallet_send.py` test to include descriptor-based wallets and adjusts the setup to handle both legacy and descriptor wallets appropriately.


Make loom process constraints for conditional mutual information.

Before, constraints (i.e. conditions were ignored) for mutual information
with loom.

Implement conditional mutual information with constraints

The commit adds methods to handle constraints for conditional mutual information calculations, including marginalization over constraints and converting constraints into Loom's required format.


Replace hashed meta with skill_gid as identifier

This also removes the notion of an owner skill and all skills may update settings on the server.

Simplify and streamline skill settings synchronization

Simplify the logic for synchronizing skill settings by removing unnecessary hash checks and direct meta handling, focusing on using `skill_gid` for synchronization.


Use alternate headers for HA auth requests

When sending a "Authorization: Bearer" header to the supervisor's /auth endpoint, the header gets used to authorize the addon and then _also_ authorize against HA which fails.  This changes requests made to the /auth enpoint so they use the "X-Supervisor-Token" header to avoid the conflict.

Update authentication header handling

Added support for X-Supervisor-Token and updated methods to use new header where necessary.


Update S3 storage driver so it suppots "region" constructor argument.

This way user can use this constructor class argument instead of using a
different driver class per region.

Also update code to return more user friendly error message if moved
permanently error is returned by the API.

Add region-to-host mapping and dynamic region handling

This commit introduces a region-to-host mapping for S3 and modifies the S3StorageDriver to dynamically handle different regions based on the provided region parameter.


Move to coredata some methods handling options

Those methods only use coredata object, so better just move them as a
coredata method.

Move option merging and setting logic to CoreData

The `merge_user_options` and `set_options` methods are moved from `Environment` and `Conf` classes to `CoreData` to centralize option management.


Use an enum instead of strings for method names.

If a non-string value is passed as a method, reject this explicitly
with a clear error message rather than trying to match with it and
failing.

Convert dependency methods to enum

The diff introduces an `Enum` called `DependencyMethods` to represent the different methods used for dependency detection, replacing string literals with enum members throughout the code.


Add RepoVersion class to make handling of many arguments easier

There are a number of arguments being passed around, nearly all of
which are duplicated between the old and new versions. Moving these
into a separate class should hopefully make it simpler to follow
what is being done.

Update AbiChecker to use RepoVersion class

The AbiChecker class now uses a RepoVersion class to encapsulate the details of each repository version, simplifying the initialization and management of repository and revision data. This change improves code readability and maintainability.


Remove `utils.tap.conn.TapConn.url_encode()`

The removed method simply called the standard library
`urllib.parse.urlencode()`.

Use `urlencode` directly instead of custom `url_encode` methods

This change simplifies the code by removing redundant `url_encode` methods and directly using Python's `urlencode` from `urllib.parse`.


Unify Polarion case searching

Save and load ids to/from results.yaml.
Add constant for extra identification keys.
Raise ReportError on TC not found in Polarion.

Store multiple IDs for test results in Polarion

Store multiple identifiers for test cases to enhance matching and reporting capabilities in Polarion. Update methods to handle multiple IDs and adjust Polarion reporting logic accordingly.


Remove an utility function for reading test data

`astroquery.utils.tap.xmlparser.utils.read_file_content()` has been
replaced with the standard library `pathlib.Path.read_text()`.
Furthermore, in the modules where multiple test functions previously
used `read_file_content()` to read the data files, they are now read
once at the module level instead.

Remove file reading utility and update test data handling

Read test data directly using Path and remove unused read_file_content function.


Use context managers to handle profiling

This avoids needing to remember to disable the profiler by every return statement, and reads more cleanly than using a `try` / `finally`

Add profiling context manager and nullcontext utility

This commit introduces a new `profiling_context` class to manage profiling within the scheduler and adds a `nullcontext` utility for use when a context manager is needed conditionally.


Move git helper out into mesonlib for reuse

Reuse the git helper for `meson wrap` and `meson subprojects` so we
don't need to maintain the same git-colors-on-windows workarounds in
multiple places.

Add git utility function in mesonlib

This commit introduces a utility function for executing git commands in `mesonlib.py` and updates other files to use this new function. It ensures consistent behavior and reduces code duplication.


Fix remove_fallback()

remove_fallback() was mainly created for use internally during shutdown
and required the wrapped callable to be able to remove a handler. This
makes it general, using a mapping to find the wrapper from a handler if
needed. The method now also returns the success/failure status

Add wrapper tracking and removal for fallback handlers

This commit introduces a `wrapper_map` to track original handlers and their wrapped versions, enhancing the ability to accurately remove specific fallback handlers. It also updates the `remove_fallback` method to correctly identify and remove handlers based on either the original or wrapped versions.


Handle translation of new computed virtual graph patterns

- uses new virtual graph function machinery for translation
- ensures we don't try to create indexes on virtual tables

Handle virtual graph pattern clauses in KGTKQuery

This change includes special-case handling for virtual graph pattern clauses in the `KgtkQuery` class and modifies the `TranslationState` class to support registration and lookup of virtual tables.


Allow tests to be run on demand

With this change, a string "[test]" can be added as a comment to a pull
request by a whitelisted user to initiate testing on a pull request

Expand PR handling to include comments and user whitelist checks

This commit enhances the Jenkinsfile and run_tests.py to handle pull request comments for triggering tests and includes user whitelist checks to ensure only authorized users can trigger tests via comments.


Make register_intent decorator register all intent types

This makes the register_intent decorator work for all types of intents,
both Adapt and Padatious.

Deprecate intent_file_handler and update register_intent

This commit deprecates the `intent_file_handler` decorator in favor of using `intent_handler`. It also updates the `register_intent` method to handle padatious intent files directly. Test cases are updated to include tests for padatious intents.


Use "import numpy as np" in the array_api submodule

This avoids importing everything inside the individual functions, but still is
preferred over importing the functions used explicitly, as most of them clash
with the wrapper function names.

Update array API functions to directly use numpy

This commit updates the array API functions to directly use numpy's functions instead of importing them from other modules within numpy. It also removes redundant import statements and simplifies the function calls.


Use named field for command_template when generating ninja command.

The command template become easier to read with named field.

Use f-string formatting for command templates

This change updates the command templates to use f-string formatting, improving readability and maintainability of the code.


Add ConfigToolDependency class

This class is meant abstract away some of the tedium of writing a config
tool wrapper dependency, and allow these instances to share some basic
code that they all need.

Add support for config-tool dependency method

This commit introduces a new dependency type, `ConfigToolDependency`, which allows finding dependencies using a "program"-config style tool. It includes methods to search for the tool, compare versions, and report the results.


Simplify datastore error handling

If an error code is defined, the response body does not need to be
specified. This prevents encoding issues in cases when a response
message has required fields.

Simplify error handling in transaction methods

This commit simplifies error handling in transaction methods by removing unnecessary protocol buffer instantiation and ensuring consistent return values across different error conditions.


Convert test protocol into an enum

This gives us better type safety, and will be important as we add more
test methods

Convert test protocol from string to enum

This commit introduces a `TestProtocol` enum to replace the string-based protocol in test configurations. It updates relevant parts of the codebase to use the new enum, ensuring consistent handling of test protocols across different modules.


Use _implementation on all functions that have it in the array API submodule

That way they only work on actual ndarray inputs, not array-like, which is
more inline with the spec.

Use `_implementation` for numpy function calls

This change updates the numpy function calls to use a hypothetical `_implementation` attribute, which may be used for custom implementations or extensions.


Add new storage API methods for downloading part of an object (range
download) and implement it for the S3 and local storage drivers.

Add methods for downloading object ranges

This change introduces methods `download_object_range` and `download_object_range_as_stream` to the StorageDriver base class and implements these methods in the LocalStorageDriver and S3StorageDriver. It also updates the `_save_object` method to support partial downloads and modifies the valid response codes in the S3 response handling.


Drop various guest `wake()` methods in favor of parent class

They all do the same things anyway: instantiate a guest class, with
given data, pass parent and name down the stream. A base class can do
all of this.

Move guest creation to base class

The `wake` method now handles guest creation in the base class `ProvisionPlugin`. This change simplifies the subclasses by removing duplicate code.


Fix resuming of SOBOL generator during ax search

This is done by fixing the seed and regenerating trials already performed.

Add logging for trial creation and resume in auto_search and ax_search

This commit introduces logging for when trials are created and resumed, providing clearer information about the trial parameters. Additionally, it ensures that Sobol models are resumed correctly in `ax_search` by fixing the seed and skipping previously generated trials.


Make skill ids use skill folder
This is necessary because in Python 3, hash(x) changes every single start of the application. Using the skill folder makes it consistent. In addition, the skill folder makes it easier to debug parts of the application in comparison to using something like an md5sum

Update skill_id handling to use alphanumeric conversion

This change updates the skill_id handling to use alphanumeric characters instead of converting numbers to letters. It also adjusts the tests to reflect these changes.


Tweak string escape approach

Use "" instead of {} to remove persistent backslashes in front of curly
braces in string.

Update string escaping for TCL compatibility

The commit changes how strings are escaped for TCL to ensure compatibility and correct handling of special characters within strings. It also updates a test comment for clarity.


Make regex errors/warnings additive, add reports

- Enables tools to extract additional errors/warnings from other logs
- Add logfiles under reports to reduce boilerplate, the logfiles always
  belong if regex is used

**Update error and warning handling in tool setup**

- Consolidated tool setup logic into a new `_setup_tool` method.
- Updated error and warning metrics to accumulate counts instead of overwriting them.
- Ensured logfiles are added to the report parameters for errors and warnings if not already present.


Use function attributes for intent decorators
This prevents needing to use a shared list which misbehaves when multiple skills initialize at once

Improve intent handler registration and cleanup

This commit updates the way intent handlers are registered, removing global lists and using function attributes instead. It simplifies the registration process and removes unnecessary parameters.


Move listen trigger to last chunk of sentence

If rendering a chunk of a sentence takes too long time, the audio queue
may run out and trigger the listening.

This moves the listening trigger to after the last chunk.

Add listen parameter to control microphone restart

This change adds a new parameter `listen` to control whether the microphone should be restarted after speech output, particularly useful when expecting a response from the user.


Add CLI option to install desktop qt automatically

This change also causes aqt to emit a warning when the option is not
turned on and the expected desktop Qt is not found.

Handle missing desktop Qt for mobile installations

This change adds a new `--autodesktop` option to automatically install the required desktop version of Qt for Android and iOS installations. It also includes a warning and the logic to check and handle missing desktop Qt installations.


Use test setups from the active (sub)project by default.

Replace the logic where a test setup with no project specifier defaults to
the main project with one that takes the test setup from the same
(sub)project from where the to-be-executed test has been read from.

Ensure `timeout_multiplier` has a default value and improve test environment handling

This commit sets a default value for `timeout_multiplier` and introduces a new method `get_test_env` to handle test environments more effectively. It also passes options and test details to relevant methods to ensure consistent environment setup and avoids copying options unnecessarily.


Support the service_account_name parameter

This allows GetAccessTokenRequests to specify a custom service
account for generating an authentication token.

Add JWT assertion generation for service accounts

The commit adds support for generating JWT assertions for service accounts and updates the access token generation process. It also introduces necessary imports and utility functions.


Add functionality to update listener/STT config

When a new configuration is discovered the producer and consumer is shutdown, the config is read and a new producer-consumer pair is launched.

### Add stop method and reload configuration

- Added `stop` method to `AudioProducer` and `RecognizerLoop` to gracefully stop threads.
- Added `reload` method to `RecognizerLoop` to handle configuration changes and restart threads.
- Added `stop` method to `ResponsiveRecognizer` to allow stopping the recognition process.


Add MonitOperator

This uses Monit's XML API and groups closely-timed Monit reloads
together.

Add MonitOperator class and status handling

This commit introduces a `MonitOperator` class to handle Monit operations asynchronously and adds functions to process Monit status responses.


Add 'install_mode' to all installable targets

This makes it possible to customize permissions of all installable
targets, such as executable(), libraries, man pages, header files and
custom or generated targets.

This is useful, for instance, to install setuid/setgid binaries, which
was hard to accomplish without access to this attribute.

Add `install_mode` support for various target types

This change adds support for `install_mode` in `BuildTarget`, `CustomTarget`, `Headers`, and `Man` classes, and updates the `meson_install.py` script to use this mode for setting permissions during installation.


Create a helper for checking if a string has a path component

This is used in a number of places, and in some places it is incomplete.
Use a helper to ensure it's used properly.

Replace path separator checks with `has_path_sep` function

This change introduces a new utility function `has_path_sep` in `mesonlib.py` to check for path separators in strings. It replaces direct checks for '/' and '\\' in multiple files with calls to this new function to improve code readability and maintainability.


Allow Python runtime to use an external API server

If given an external api port, the Python runtime will use it to
make App Identity calls.

Add external API server support

This change introduces support for an external API server by adding a new parameter to relevant functions and classes. It also updates the runtime configuration to handle the external API port.


Make execute_test less complex

Split the long function into several smaller ones just retaining the
main logic.

Add test case loading and input handling in SkillTest

This commit introduces a new function to load test case files and reorganizes the input handling logic within the SkillTest class to improve readability and maintainability.


Rename queries sp*/s*o/*po to sp_/s_o/_po

This makes them valid identifiers, which allows to use query types as
configuration keys (for example).

Update combine values in scoring methods

Replace "sp*" and "*po" with "sp_" and "_po" in scoring methods and relevant documentation for consistency.


Delete redirected wrap files in subprojects purge

We need to store the original filename as well as whether the wrap was
redirected in order to properly purge the redirected wrap.

Handle redirection cleanup in subprojects purge

The commit adds support for cleaning up redirected wraps during the subprojects purge operation. It updates the `PackageDefinition` class to track redirection information and modifies the `Runner` class to delete files associated with redirected wraps when purging. Corresponding test cases have been updated to reflect these changes.


Remove PDK-specific var defaults from openroad.py

We shouldn't be hardcoding tech-specific values in tool setup scripts,
so we should use the new ['pdk', 'variable'] parameter to drive these.

In order to support user overrides, we'll still drive them through the
['eda', ... 'variable'] keypath in the TCL script.

Add default OpenROAD tool variables in PDK setup

This commit introduces default OpenROAD tool variables for different PDKs (asap7, freepdk45, skywater130) and modifies the OpenROAD setup to read these defaults from the PDK configuration. If a default is not provided, the variable is marked as required for the user to set.


Make `-std=` fallback remapping more robust

* The current version matching logic is brittle
  with respect to Clang. LLVM and Apple Clang use
  slightly different but nowadays overlapping
  version ranges. Instead, we now just check whether
  the compiler supports the given `-std=` variant
  and try its respective fallback instead of
  testing version ranges.

Improve C++ standard detection and handling

The commit adds a new method `_test_cpp_std_arg` to test if the compiler supports a specific C++ standard flag. It also introduces `_find_best_cpp_std` to handle fallbacks for older compiler versions and refines the handling of `cpp_std` in `ClangCPPCompiler` and `GnuCPPCompiler` classes.


Fix for orphaned items that may develop from a failed watcher event.

- Also added optional (but on by default) silencing of verbose and useless botocore logs.

Handle orphaned deletions in security tasks

The commit includes a new function `fix_orphaned_deletions` to address orphaned items in the database by creating deletion records for them. This function is called in both `reporter_logic` and `find_changes` functions to ensure that orphaned items are handled before processing changes. Additionally, the commit adjusts the logging settings for `botocore` and adds a check for duplicate SQS items during the slurp process.


Add test to simulate GCE instance setup

This change adds a test that sets up hosts' addressing and routing as it
is on a GCE instance, and checks that we still get connectivity between
workloads on different hosts.

Enable simulate_gce_routing in DockerHost and tests

Add support for simulating GCE routing in DockerHost and update tests to use this option.


Move the Guest class from base to steps.provision

Here it makes more sense as it is related to provision classes.

Move Guest class to provision module

This commit relocates the Guest class from the base module to the provision module, updating import statements and references accordingly.


Reboot has to check for boot time

Previous attempt with closing connection was not succesful as shown by
/tests/execute/reboot/reuse_provision

btime in /proc/stat is boot time in seconds since epoch and reboot()
will make sure that it has changed

Adds disk requirement for tests/full which is helps with testing

Update disk size and improve reboot handling

The disk size is increased to 40GB in the test configuration. The reboot process is improved by checking the boot time and ensuring a proper connection after reboot without relying on a timed sleep.


Add macOS linker versioning information

This patch exploits the information residing in ltversion to set the
-compatibility_version and -current_version flags that are passed to the
linker on macOS.

Handle shared library version arguments consistently

This change updates methods to handle shared library version arguments consistently by introducing a `version` parameter and adjusting the arguments passed to linkers accordingly.


Update old style metaclasses

the old "__metaclass__" has been ignored since the switch to python 3
this restores the metaclass functionality by updating it to the new
class kwarg syntax

Use `metaclass=ABCMeta` in class definitions

This change updates several class definitions to use `metaclass=ABCMeta` directly in the class declaration, removing the legacy `__metaclass__ = ABCMeta` syntax.


Refactor TestResult object initialization

Just add or alter attributes of the TestResult object, rather than many,
many invocations of the constructor with very similar arguments.

Define helper methods for TestResult to add the results of a step and
set the failure reason.

Update TestResult to track build steps and times

The `TestResult` class now initializes with fewer arguments and includes methods to add steps and mark failures. The `_run_test` function has been updated to use these new methods for tracking different build steps and their outputs.


Add support for property metadata queries

This allows clients to request a list of property names and types
for all kinds in a given namespace.

Add property iteration and type range handling

This commit introduces PropertyIterator for iterating over indexed property names for a kind and extends SinglePropIndex with type_range to handle property types more effectively.


Enhance tmt.utils.retry_session with timeout support

`retry_session` is now usable as a context manager, and accepts a
timeout to honor for each HTTP request.

Current `retry_session` was overlapping with similar functionality in
`artemis` plugin, so merged both together and threw context manager in
too, because `retry_session` was used in that manner exclusively.

Use context manager for retry sessions

Changed the retry session creation to use a context manager to ensure resource management and consistency across the codebase. This replaces custom retry strategy installations with a more streamlined and reusable approach.


Add PiecewiseBrokenPowerLawSpectralModel

Add PiecewiseBrokenPowerLawSpectralModel, a generalised broken power law  defined by a set of energy and intensity values (nodes) that are piecewise connected by power laws.

Add PiecewiseBrokenPowerLawSpectralModel

This commit introduces a new spectral model, `PiecewiseBrokenPowerLawSpectralModel`, which represents a piecewise broken power-law at fixed energy nodes. The necessary changes include adding the class definition in `spectral.py`, updating the model registry, and adding tests in `test_spectral.py` and `test_io.py`.


Handle UpdateIndexes calls

This adds index definitions to a project's ZooKeeper node if they
aren't there yet.

Add UpdateIndexesHandler for managing datastore indexes

This commit introduces a new handler, UpdateIndexesHandler, which processes requests to update datastore indexes for a given project. It includes classes to represent index properties and indexes, and it handles the logic for adding new indexes to Zookeeper.


Create and destroy C sessions inside call_module

Removes the need to always do this when using the C API. It's what the
command line app does anyway so it's not wasteful.

Remove session parameter from call_module

The change removes the session parameter from the `call_module` function and instead creates and destroys the session within the function. This aligns the behavior of `call_module` with the GMT command-line interface, ensuring session management is handled internally. Adjustments have been made in `session_management.py` and the test files to accommodate this change.


Adjust UHV goals to new wonders

- second French goal now requires the Louvre and the Metropolitain instead of the Statue of Liberty
- second Mughal goal now requires Shalimar Gardens instead of Harmandir Sahib
- second American goal now also requires Brooklyn Bridge and Golden Gate Bridge

Update wonder goals for France, Mughals, and America

Additional details:
- Added Louvre and Metropolitain to France's wonder goals.
- Replaced Harmandir Sahib with Shalimar Gardens for Mughals.
- Added Brooklyn Bridge and Golden Gate Bridge to America's wonder goals.
- Updated relevant help texts and XML entries to reflect these changes.


Define SelectionPredicateComposition

Made a new class SelectionPredicateComposition to allow use of ~,&,| with certain PredicateCompositions.  Rewrote test_filter_transform_selection_predicates to match the new syntax.

Use `SelectionPredicateComposition` for selection operations

The code now uses `SelectionPredicateComposition` for handling selection predicates in `Parameter` class methods and updates the corresponding tests to reflect the changed output format.


Convert guest implementations to use serializable container for load/save

Instead of custom load/save methods, use those based on dataclasses. We
gain type annotations, automagic conversion from/to YAML, no need for
explicit moves of fields between instances and serialized data.

Convert guest data handling to use dataclasses

Convert the handling of guest data from dictionaries to dataclasses for better type safety and cleaner code. This change affects multiple provision plugins and the base guest class.


Implement Range downloads for the Azure Blobs driver and move common
method which can be re-used by S3 based and Azure drivers to the base
driver class.

Add support for downloading object ranges

Adds methods to download object ranges in Azure Blobs and S3 drivers. Also, adds a helper method to generate standard range strings used in range requests.


Fix RST parsing in schemagen

- Use Python standardized docstring "trim" function to preserve newlines
  and indentation in schelp strings.
- Fix schemagen parse_rst() to handle multiline text

Move `trim` function to `utils` module

The `trim` function is moved from multiple files to a centralized location in the `utils` module to avoid code duplication and improve maintainability.


Fix Player when playing audio source with silent audio driver.

When there is no video but only audio in the Source and the audio driver is silent,
the Player was never finished playing. Now it dispatches correctly the "on_eos"
event after the duration.

**Schedule EOS event for audio-only sources**

Added logic to schedule an end-of-stream (EOS) event for audio-only sources in the `Player` class. Also, improved formatting and consistency in the test files.


Support for dataset-specific configurations and metadata

preprocess now creates a dataset.yaml file with information about
the dataset.

**Update dataset preprocessing script and configuration**

The preprocess script now correctly accepts the folder name without the `--folder` flag. The script also generates a `dataset.yaml` file containing metadata about the dataset, and updates to the `Dataset` class allow for loading configurations from this file if it exists.


Add jobid and njobs argument to script command.

This allows to run multiple experiments from the same script in several parallel executions of YANK.

Add options to run a subset of experiments

The script now includes `--jobid` and `--njobs` options to allow running a subset of experiments in parallel from the same YAML script. The `ExperimentBuilder` class has been updated to handle these new parameters and distribute experiments accordingly.


Support short Fedora compose aliases in testcloud

Adds support for 'fedora' as the latest released Fedora, 'rawhide'
for the latest Rawhide plus 'fedora-X', 'f-X' and 'fX' shortcuts.

Update image URL guessing for Fedora Rawhide and specific releases

The commit enhances the image URL guessing mechanism to support specific Fedora releases and Rawhide, replacing hardcoded URLs with dynamic fetching of the latest compose IDs and image URLs.


Refactor sanitizer transform logic

Move logic to parse mapping of old to new column names and strip
prefixes into their own functions with tests. This refactoring
simplifies the code in the main body of the sanitizer script.

Add functions to parse renaming rules and strip prefixes

This commit introduces two new functions: `parse_new_column_names` to handle the parsing of column renaming rules and `strip_prefixes` to strip specified prefixes from strain names. These functions replace inline logic in the main execution block, improving code readability and reusability.


Improve reliablility of p2p-acceptblock.py

Numerous timing issues do with the the multi-threading and also
the requestmanager. Better to no rely on sync_with_ping() and also
we have to handle getheaders requests and subsequent multi hash
getdata requests.

Implement sync_getdata and update test_node usage

This commit adds a `sync_getdata` method to `TestNode` to wait for specific getdata responses and updates the test cases to use this method where appropriate. It also enhances debugging options and adds waiting mechanisms to ensure nodes are in the expected state before assertions.


Handle special translation needed for 'likelihood' and 'concat' functions

Also slightly refactored and cleaned up translation of function expressions.

Implement function_call_to_sql for better SQL expression handling

This commit introduces a new method `function_call_to_sql` to handle function calls in SQL expressions, improving support for functions like CAST and CONCAT. It also refactors the existing function call handling in `expression_to_sql` to use the new method.


Allow group locks to persist when commits fail

This prevents other clients from writing to a group before a large
batch is fully applied.

Handle lock acquisition failures explicitly

Implement explicit lock acquisition and release, and handle lock timeout errors by raising a `TimeoutError`. This change ensures that lock management is more robust and that timeout errors are properly caught and handled.


Remove unused keys parameter from wake() methods

This seems to be no longer used, and makes `wake()` more complicated
than necessary.

Remove `keys` parameter from `wake` method

Remove the `keys` parameter from the `wake` method in various plugin classes as it is not used. This simplifies the method signature and reduces unnecessary complexity.


Add new method: mlog.deprecation()

Instead of constructing it manually, use a helper.

Use `mlog.deprecation` for deprecation warnings

This change updates the logging method from `mlog.warning` and `mlog.log` to `mlog.deprecation` for all deprecation messages. It also adds the `deprecation` function to `mlog.py` to handle these messages consistently.


Add ability to construct RegressionManager with test/hook list

Standard constructor that uses discovery has been moved to the
`from_discovery` class method.

Update RegressionManager initialization to use discovered tests and hooks

The `RegressionManager` now accepts tests and hooks directly during initialization, with a class method `from_discovery` to handle automatic discovery. The logger is also centralized to avoid duplication.


Enable searching system crossfile locations on more platforms

There's no reason not to also look in these places on Cygwin or OSX.  Don't
do this on Windows, as these paths aren't meaningful there.

Move test_cross_file_system_paths from LinuxlikeTests to AllPlatformTests.

Update cross file system paths handling for Windows

The cross file system paths logic is updated to handle Windows separately. For Windows, an error is raised if system crossfile paths are not defined. The test for cross file system paths is also moved to ensure it skips on Windows and tests the Linux path handling correctly.


Refactor getting the host system of a cross compiler

Use mesonlib.for_windows or mesonlib.for_cygwin instead of
reimplementing them.

Add CrossBuildInfo.get_host_system to shorten the repeated the code in
the mesonlib.for_<platform> methods.

Use `get_host_system` for cross platform checks

The `get_host_system` method is introduced to simplify and centralize the retrieval of the host system in cross builds. This change updates the `for_windows`, `for_cygwin`, `for_linux`, `for_darwin`, `for_android`, and `for_haiku` functions in `mesonlib.py` to use `get_host_system` from `CrossBuildInfo` in `environment.py`.


Implement the array API result_type() manually

np.result_type() has too many behaviors that we want to avoid in the array API
namespace, like value-based casting and unwanted type promotions. Instead, we
implement the exact type promotion table from the spec.

Implement custom result_type function with type promotion table

This change introduces a custom `result_type` function in `_data_type_functions.py` that uses a promotion table defined in `_dtypes.py` to determine the result type according to the specified rules, avoiding NumPy's additional type promotions.


Add option to remove default output in models yaml file

Add option full_output (True by default) to model.to_yaml() and .to_dict(). Switching to False remove the entries for min, max, frozen and error if they are the same than the class defaults or nan.

Add `full_output` parameter to `to_dict` methods

This change introduces a `full_output` parameter to the `to_dict` methods across various classes in the Gammapy modeling framework. When set to `False`, it omits default parameter values from the output dictionary, simplifying the YAML serialization.


Rename clike_langs to clink_langs for clarity

D is not a 'c-like' language, but it can link to C libraries. The same
might be true of Rust in the future and Go when we add support for it.

This contains no functionality changes.

Rename `clike_` variables and functions to `clink_`

This change updates variable and function names from `clike_` to `clink_` to better reflect their use with languages that can be linked using C-style linkers.


Fix b_ndebug=if-release option

Provide get_{type}_options_for_target() methods that unite {type} and builtin options.

Allow multiple option sources in OptionOverrideProxy

This change modifies the `OptionOverrideProxy` to accept multiple sets of options and updates the `Backend` class to use this feature for different types of options, such as builtins, base, and compiler options. Additionally, it refines the logic for handling the `b_ndebug` option by ensuring correct syntax formatting.


Use the EntityLock in the groomer

The allows the groomer to make index modifications under a lock
without having to create transaction IDs.

Replace key locking with EntityLock in DatastoreGroomer

This commit replaces the custom lock acquisition and release methods with the EntityLock class for managing locks in the DatastoreGroomer class. It also introduces a new method to guess the entity group from a table key to support the new locking mechanism.


Move shared static functions to utils

This allows other modules like the Celery worker to use them.

Move queue utility functions to utils module

The utility functions related to queue management have been moved from `tq_config.py` to `utils.py` to improve modularity and reusability.


Adjust the minute provision debug messages wording

Just a couple of minor cosmetic adjustments to make the messages
consistent with the rest.

Update debug messages for clarity

Improve readability by making debug messages more concise and grammatically consistent.


Allow DeploymentConfig to take a KazooClient

This makes it easier to reuse the KazooClient for other things.

Update DeploymentConfig to accept KazooClient

This commit modifies the deployment configuration setup to accept a KazooClient instance directly, improving initialization and reducing redundant client creation across modules.


Reduce unnecessary "exists" watches

This allows managers to stop watches if it's clear they are no
longer needed.

Ensure watches are restarted if cancelled

This change adds functionality to restart watches if they have been cancelled, preventing the need for manual intervention. It also includes error handling for Zookeeper operations and ensures that watches are set up correctly when projects or configurations are updated.


Fix implementation of `helper.get_hash`

Causes `get_hash` to verify the hash length, and check that the hash can
be unhexlified properly.

Fixes the interface for `get_hash` so that the caller does not have to
run `binascii.unhexlify` or verify its output.

Remove redundant `binascii` usage

The `binascii.unhexlify` function is no longer used in the `get_hash` function, and its usage has been removed from other parts of the code where the hash was being unhexlified. The `get_hash` function now directly returns the hash in bytes format.


Drop phrasedml and sbml2matlab requirement.

phrasedml and sbml2matlab don't exist for 3.10 yet, so we need to adjust code and fail gracefully when trying to import them.

Remove sbml2matlab import and update error handling

The `sbml2matlab` import has been removed from the startup lines in `main.py`. Additionally, error handling has been improved in `convert_omex.py` and `convert_phrasedml.py` by wrapping imports in `try-except` blocks and updating error messages to include exceptions. The `phrasedmlImporter` class is now conditionally defined based on the existence of `phrasedml`.


Use pyglet.options['ffmpeg_libs_win'] to give the FFmpeg lib filenames.

This has to be used on Windows, as the dll names have a format like
avcode-57.dll, appending the version number at the end.

Add `ffmpeg_libs_win` option for specifying FFmpeg DLLs on Windows

This commit introduces a new option `ffmpeg_libs_win` to allow developers to specify the exact FFmpeg DLL filenames on Windows, accommodating version differences. It updates the initialization of FFmpeg libraries in `pyglet/media/sources/ffmpeg_lib` to use this option if available.


Reduce duplication in setting up `gaia` tests

The new module level fixtures allowed removing setup code from many
tests.

Improve test setup with mock queriers

The commit introduces mock queriers to simplify and standardize test cases in `test_gaiatap.py`. It removes repetitive setup code in each test method and replaces it with fixtures that provide pre-configured `GaiaClass` instances. This change enhances the readability and maintainability of the test suite.


Remove thread in DirectSound

We use instead pyglet.clock.schedule_interval_soft to regularly
refill the DirectSound buffer with new audio data.

Remove locking and use pyglet clock for refilling

This commit removes the threading lock throughout the `DirectSoundAudioPlayer` class and introduces a `pyglet.clock` scheduled interval to handle buffer refilling. This change simplifies the code and leverages pyglet's clock system for periodic tasks.


Add 'Compiler.get_display_language'

Use this when we print language-related information to the console and
via the Ninja backend.

Use `get_display_language` for consistent compiler language display

This change introduces a new method `get_display_language` in the `Compiler` class to provide a consistent and capitalized display name for each compiler. The method is then used in various parts of the codebase to replace direct access to `self.language` where a human-readable format is required.


Add a minValue option to nonNegativeDerivative and perSecond

It works in a way similar to maxValue: when the counter wraps, instead of
producing a null value, it computes the difference assuming the counter wrapped
to minValue.

Add `minValue` parameter to `perSecond` and `nonNegativeDerivative` functions

The `minValue` parameter is added to both `perSecond` and `nonNegativeDerivative` functions to handle cases where the counter resets to a value other than zero. Corresponding tests are updated to include scenarios with `minValue`.


Replace blank "type: ignore" with more specific waivers

Using mypy's error codes, waivers can target the actual erorr, leaving
space for other kind of violations to not be suppressed.

Improve type ignores in code

This commit adds more specific type ignores across the codebase, addressing various issues reported by mypy. It includes new static analysis hooks in the pre-commit configuration for Python code and updates type ignore comments in multiple files to be more precise.


Update setup.py so it doesn't rely on any functions from libcloud
packages.

This way we avoid having setup.py depend on typing, requests and other
modules libcloud depends on.

Inline package and data file functions in `setup.py`

The functions `get_packages` and `get_data_files` are moved into `setup.py` to avoid dependencies on Libcloud-specific code and external libraries. Adjustments are made to the `read_version_string` function to directly read the version from `libcloud/__init__.py`. The `skipdist` setting is corrected in `tox.ini` for the `py2.7-dist` and `py3.7-dist` environments.


Improve __repr__ for RunningTask objects.

Displays task name, status, and current coroutine.
If the task is pending on a trigger, displays the trigger.
If the task is finished, displays the outcome.

Add task ID and coroutine stack extraction

This commit introduces task IDs for better debugging and adds functionality to extract coroutine stacks.


Replace the global GMTSession with 'atexit'

No need for the class or the global instance. Register 'end' with
'atexit' to make sure end is called when exiting Python.

Remove GMTSession class and use atexit for cleanup

The commit removes the GMTSession class and replaces it with calls to `atexit.register` to ensure the session is properly ended when the Python interpreter shuts down. The global session management is simplified by directly using the `begin` and `end` functions and removing the need for the class-based session management.


Add subdirectories for aggregates

To enable multiple aggregates in a clean way in the output directory will now contain a sub-directory for each group for which the aggregate function is called

Create subdirectories for plot outputs

This change updates the scripts to create subdirectories for plot outputs, ensuring that plots are organized by group name. It also modifies the storage functions to handle default directories and absolute paths more gracefully.


Add a spectral cutout example

Also include support for numpy arrays for band and channel, and autocorrect the band and channel values to be in ascending order

Add numpy array support for band and channel

Details:
- Updated the `CasdaClass` to accept numpy arrays for `band` and `channel` parameters.
- Ensures that the `band` and `channel` values are always in ascending order.
- Added test cases to verify the functionality with numpy arrays.
- Updated the documentation to include examples for 3D cutouts.


Improve performance of path hint generation

Avoid instantiating tons of different logic instances and just reuse the same instance by resetting its state every time it's needed.

Prevent recreation of Logic instance for path checks

The change introduces logic to reuse a single Logic instance for multiple path checks, optimizing performance by saving and loading the simulated playthrough state.


Improve speed of rebalance script

This removes the call to `nodetool ring`, which can get
unreasonably slow as the amount of data in a BOP cluster increases.

It also adds a couple flags that allow the `nodetool status` call
to be skipped if the user is already sure the sanity checks will
pass.

Update rebalance script to use gossip and add command-line arguments

The script now fetches cluster gossip instead of ring information and includes command-line options to skip certain checks.


Support CUDA stream on memory pool

Now, memory pool will have an arena (bins) for each stream
to avoid concurrent streams touch the same memory block

Add support for concurrent memory management with streams

This change introduces support for managing CUDA memory with different streams. It includes adding a `stream_ptr` attribute to `Chunk`, modifying memory operations to accept an optional `stream` parameter, and updating the memory pool to handle different streams independently.


Improve combine metadata script

1. More detailed help message
2. Helpful errors printed if assertions fail
3. Encoding of metadata origin now uses explicit "yes"/"no" values
rather than using the empty string.

Improve metadata combination script with error handling and origin one-hot encoding

The script now includes error handling for input validation and implements one-hot encoding for origins using "yes" and "no" instead of booleans. Additional details were added to the argument description.


Add wait_for_message() method to messagebus client

- Refactor message waiting into a MessageWaiter class to be able to use the
  same code in both wait_for_message and wait_for_response.
- Add some basic unittests

Add `MessageWaiter` and `wait_for_message` method

The `MessageWaiter` class and `wait_for_message` method in `MessageBusClient` are added to encapsulate and simplify the logic for waiting on specific messages with a timeout. Unit tests for `MessageWaiter` are also included.


Add a nice provisioning progress to Artemis plugin

Artemis plugin now displays the sequence of of guest states
as its provisioning progresses. This has been implemented as
a reusable primitive context manager, coupled with a small
refactoring of code taking care of indentation.

**Introduce updatable_message for progress reporting**

Added `updatable_message` context manager in `tmt.utils` to facilitate progress reporting, updated `GuestArtemis` to use it, and added a TODO about potential use in `ExecuteInternal`.


Change to use argument name "geom" consistently.

Previously we had a mix of "geom" and "ref_geom" in Gammapy.

Rename `ref_geom` to `geom` for consistency

This change updates variable names from `ref_geom` to `geom` across multiple files to ensure consistency in naming conventions.


Convert resolve() to str.translate().

New translation table class _ResolveTable provides resolve functionality
and improves performance.

This preserves current behavior of COCOTB_RESOLVE_X = RANDOM, where
all instances of the character are translated to the same value.

Optimize binary string resolution using translation table

Implement a translation table for resolving binary strings, improving performance and readability of the `resolve` function and its usage within the `BinaryValue` class.


Support for audio seek

The audioservice can now jump forward and backward in the audio stream/file

The functionality is accessed via the audioservice class's seek_forward(),
seek_backward() and seek() methods

Add seek forward and backward functionality to audio service

Implement seek forward and backward methods in AudioService, AudioBackend, and VlcService. Update the message bus handlers to support these new commands.


Fix multiline strings in schema examples

Using '\' or multiline strings results in extra whitespace, so it seems
best to use Python's automatic concatenation of adjacent strings.

Fix inconsistent string formatting in schema examples

Corrected the string formatting in the examples to consistently use double quotes around the entire string and single quotes for inner string parameters.


Rewrite custom_target template string substitution

Factor it out into a function in mesonlib.py. This will allow us to
reuse it for generators and for configure_file(). The latter doesn't
implement this at all right now.

Also includes unit tests.

Simplify and modularize custom target command evaluation

The commit simplifies the evaluation of custom target commands by modularizing the process of generating input and output filenames and adding comprehensive template substitution and error checking functions.


Simplify and clean up curate_cache

- create helper functions for getting file stats and removing files in
order of age
- create wrapper function for turning MB into bytes

Add helper functions for cache management

The commit introduces two helper functions, `mb_to_bytes` and `_get_cache_entries`, and refactors the file deletion logic in `curate_cache` to use a new `_delete_oldest` function. This improves code readability and reusability.


Move protobuffer handling to different level

Parsing the commit request and populating the response should
happen at the protobuffer interface level rather than in
DatastoreDistributed.

Move transaction commit logic to request handler

The commit transaction logic is removed from `DatastoreDistributed` and moved directly into the `MainHandler` in `datastore.py`. Corresponding test for `commit_transaction` in `test_datastore_server.py` is also removed.


Hide away session management

Implement a GMTSession class to handle calling begin and end. Use it as
a global (package level) session. This way, each script/notebook uses
a single session. begin and end are no longer public API functions.

Replace `gmt.pscoas` with `gmt.pscoast` and manage modern sessions

Each GMT module name in the documentation and code was corrected from `gmt.pscoas` to `gmt.pscoast`. Additionally, introduced a `GMTSession` class to handle the beginning, management, and ending of modern mode sessions, replacing direct calls to `begin` and `end` in the public API. Updated tests to accommodate these changes.


Add a helper for importing a member from a module

This primitive appeared twice already, and follows the same
scenario: import a module, then locate a member (a class, usually)
in the module, and catch possible errors along the way.

Add import_member function to tmt.plugins

The `import_member` function is added to handle the import of specific members from modules, with error handling for missing modules or members. This function is used in `tmt.utils.SerializableContainer` and `_prenormalize_fmf_node` to dynamically import classes, replacing previous manual import logic.


Clear previously-loaded models in Antimony when getting a new one.

Repeated calls to 'loada' were causing steady increases in used memory because the Antimony library was storing every model.  This clears them out each time, since a tellurium user doesn't care about retrieving old models.

Also clean up a couple of warnings.

Remove unused import and improve model loading

- Removed the unused `sys` import.
- Added logging for `pySBOL` import failure.
- Updated model loading function to use `loadSBMLModel` where applicable.
- Added `antimony.clearPreviousLoads()` before conversions to prevent state pollution.


Adjust the default branch handling for libraries

Detect the default branch and store it after cloning the repo.
This seems to make the implementation a bit more readable.

Update beakerlib tests and library handling

The commit updates the `test_beakerlib.py` file to parameterize the `test_library_from_fmf` test, allowing it to handle different URLs, names, and default branches. Additionally, the `Library` class in `beakerlib.py` is modified to detect the default branch from the origin after cloning, removing previous hard-coded references, and improving branch handling logic.


Implement a generic `requires` for all plugins

Add support for gathering packages required on the guest to all
supported plugins to allow easier future extension if needed.

**Add `requires` method to steps and plugins**

Introduce a `requires` method to the `Plugin` and `Step` classes to list necessary packages for various steps (prepare, finish, report) and their plugins. This method will help in managing dependencies that need to be installed on the guest before executing these steps.


Allow custom service accounts for restore ops

This extends the datastore admin interface to allow custom
service accounts for restore operations.

Add service account selection to datastore admin

This commit adds functionality to select a service account for datastore operations, both for backups and restores. It includes changes to fetch service account names from the IAM API and integrate these names into the HTML templates for selection.


Simplify rollback_transaction method

This moves the protobuffer-related work to the proper layer.

Simplify rollback transaction handling

The `rollback_transaction` method now directly uses the transaction ID instead of handling the encoded request. This change updates the method signature and error handling to raise an `InternalError` if the rollback fails.


Support selecting objects under the current folder

Add a special single dot notation for filtering stories, plans and
tests under the current working directory. Update documentation
and usage messages accordingly.

Update documentation and code to support '.' for current directory selection

In order to select tests, plans, and stories under the current working directory, the single dot notation is now supported in the documentation and the relevant code has been updated to handle this notation.


Implement the datastore_v4.AllocateIds API

This allows clients to ensure that a list of entity IDs are never
re-allocated.

Add caching and min counter support for entity ID allocation

This commit introduces caching for the last reserved ID to avoid unnecessary Cassandra requests. It also adds support for setting a minimum counter value in the ID allocation process, ensuring that the allocated IDs meet or exceed a specified minimum value.


Reduce pressure on memory in stream tests

This change runs the python garbage collector before and after each
stream test.

The garbage collector is disabled in the CI since it has a significant
impact on the duration of the jobs (which are time constrained in the CI).

Ensure garbage collection in tests

This change adds a `run_gc` decorator to ensure garbage collection is run before and after certain tests, except when running on Travis or AppVeyor. It also introduces a `run_gc_param_store_comp_size` and `run_gc_param_data_buffer_size` decorators for parameterized tests that require garbage collection.


Use include_directories for D impdirs.

Change the code to store D properties as plain data. Only convert them
to compiler flags in the backend. This also means we can fully parse D
arguments without needing to know the compiler being used.

Handle D import directories with include_directories

The commit adjusts the handling of D import directories to use `include_directories` instead of manually constructing paths. It modifies the `BuildTarget`, `DCompiler`, `Interpreter`, and test cases to ensure compatibility and correct usage.


Use correct environment for REGEN in VS backend.

Try to guess which VS Command Prompt was used for the Meson call.
If one is chosen invoke it before calling Meson in REGEN command.

Add method to get vcvars command in Visual Studio backend

This commit adds a method `get_vcvars_command` in the Visual Studio backend to determine the appropriate command for setting up the Visual C++ environment variables. It also updates the command template to call this method before running the regeneration check. Additionally, it adjusts the architecture detection logic to account for the `VSCMD_ARG_TGT_ARCH` environment variable used in MSVC 2017.


Simplify setting DummyResponse data in TAP+ tests

`DummyResponse` is used in `esa/jwst` and `gaia` tests for mimicking
server responses. Now the response data can be specified by only setting
the parameters that have non-default values.

Simplify `set_data` calls by removing unnecessary parameters

The change simplifies calls to the `set_data` method by removing the `context`, `headers`, and other unnecessary parameters that were being set to `None`. This cleanup makes the code cleaner and easier to read.


Make handlers for single scheduled events one shot

- add_event() now accepts the parameter once, registring the event as a one shot event.
- remove_event for non-existing events is handled
- added a test for this

Add `once` parameter to `add_event` for single-run handlers

The `add_event` method now accepts an optional `once` parameter, allowing handlers to be registered as single-run events. This change also updates the `schedule_event` method to use this parameter appropriately and includes test cases to verify the new behavior.


Add support for relation prediction

Added a score_so method to KgeModel for this purpose and a default
implementation in RelationalScorer (which is currently used by all models).

Add support for `combine="s*o"` in scoring methods

This change updates the scoring methods in various model classes to handle the `combine="s*o"` case by delegating the operation to the superclass method. It also adds a new method `score_so` in `KgeModel` and ensures that `score_so` raises an exception in `ReciprocalRelationsModel` since it does not support scoring relations in that manner.


Add ignore_comments to documentation

- Also change data parsing functions to use 'ignore_data_comments'
  rather than 'ignore_comments'. This clarifies the difference between
  the two flags since 'ignore_data_comments' is used in parsing the data
  section and 'ignore_comments' is used in parsing header sections.

Add support for custom comment strings in LAS file headers

Details:
- Updated documentation to explain how to specify custom comment strings using the `ignore_comments` parameter.
- Renamed the `ignore_comments` parameter to `ignore_data_comments` in relevant functions to avoid confusion with the new `ignore_comments` parameter introduced for header items.
- Adjusted the `inspect_data_section`, `read_data_section_iterative_normal_engine`, and `parse_header_items_section` functions to accept the new parameters correctly.


Update AWS error response XML parsing code so we also handle situation
where response doesn't have a namespace.

It looks like that in some cases AWS returns error response without a
namespace which previous version of the code didn't handle correctly.

Add ignore namespace XML search functions

The commit introduces `findtext_ignore_namespace` and `findall_ignore_namespace` functions to handle XML responses that may or may not include a namespace. These functions first attempt to search using a provided namespace and fall back to a namespace-less search if no results are found.


Refactor FFmpegSource

Using 2 queues for video packets and audio packets. Whenever the
queues have space, more packets are read from the stream.

This work will allow to remove more easily the audio thread in
favor of a scheduled call to refill the audio player.

Add AudioPacket class and queue management

This commit introduces the `AudioPacket` class to manage audio packets similarly to video packets. It also refactors the queue management for both audio and video packets, adding methods to append, clear, and check the level of these queues. The `FFmpegSource` class now handles audio and video packet queues more efficiently.


Validate `svo_fps` query parameter names locally

`SvoFpsClass.data_from_svo()` now checks the names of the query
parameters and only connects with the server if all names are valid.

Validate query parameters in SVO FPS

Added validation for query parameters in `SvoFpsClass` to ensure they are within the allowed set, raising an `InvalidQueryError` if not. Updated tests to cover invalid query scenarios. Removed outdated and incorrect test for `IndexError`.


Stop routing deleted versions

This ensures that deleted versions are absent from the HAProxy
configuration file when HAProxy is reloaded.

Add exception for invalid HAProxy ports and update version routing manager

The commit introduces a new `InvalidConfig` exception to handle cases where multiple versions attempt to use the same port in HAProxy configurations. It also enhances the `VersionRoutingManager` class with a `stop` method to properly deactivate and reset the manager's state when routing is no longer needed.


Use better type signatures in the array API module

This includes returning custom dataclasses for finfo and iinfo that only
contain the properties required by the array API specification.

Add type hints and update function signatures

This commit includes adding type hints to the `__len__`, `dtype`, `device`, `ndim`, `shape`, `size`, and `T` methods in the `Array` class. It also updates the `asarray` function signature to accept `Array` objects, and modifies the `finfo` and `iinfo` functions to return custom data classes that align with the array API specification. Additionally, the `concat` and `stack` functions in `_manipulation_functions.py` are updated to accept both tuples and lists for their `arrays` parameter.


Support aliasing of input files and querying from stdin.

Because an input file might be referenced by its alias now, we
removed various 'os.path.realpath' normalizations and use
'normalize_file_path' more selectively now.

Normalize file paths and support file aliases

Add functions to normalize file paths and handle file aliases in SqliteStore. Update methods to use these functions where appropriate. Enhance `get_file_info`, `set_file_info`, `delete_file_info`, `has_graph`, and `add_graph` to support file aliases and standard input handling.


Check for available memory instead of its total amount in tests

This change prevents the python interpreter from triggering the
OOM-killer while trying to use/allocate more memory than available on
the system.

Check available memory for tests instead of total memory

This change updates the memory checks in multiple test files to use `psutil.virtual_memory().available` instead of `psutil.virtual_memory().total` to more accurately reflect the memory available for the tests.


Use enum instead of `int` for compiler variants

* Enums are strongly typed and make the whole
  `gcc_type`/`clang_type`/`icc_type` distinction
  redundant.
* Enums also allow extending via member functions,
  which makes the code more generalisable.

Replace compiler type constants with an enum and update usage

This change introduces an `enum` for compiler types, replacing individual constants like `GCC_STANDARD`, `CLANG_OSX`, etc. The commit updates all usage of these constants to use the new `CompilerType` enum, providing a more organized and maintainable approach to handling different compiler variants.


Use a single ArgumentParser for all subcommands

This has the adventage that "meson --help" shows a list of all commands,
making them discoverable. This also reduce the manual parsing of
arguments to the strict minimum needed for backward compatibility.

Simplify argument parsing in meson commands

Details:
- Removed individual argument parsers and replaced with a centralized `CommandLineParser` in `mesonmain.py`
- Updated `add_arguments` and `run` functions across commands (`mconf`, `minit`, `minstall`, `mintro`, `msetup`, `mtest`, `rewriter`, `wraptool`) to work with a single parser instance
- Added `run_with_args` in `mtest.py` to handle argument parsing and command execution
- Adjusted `run_project_tests.py` and `run_tests.py` to use `mtest.run_with_args` for test execution


Make tool regex set errors/warnings metrics

- "errors" and "warnings" suffix is special case
- Also update docs for relevant schema param and check_logfile()

Update check_logfile to handle logfile paths and return match counts

The `check_logfile` method now accepts an optional `logfile` parameter, allowing for direct specification of the log file path. If not provided, it defaults to using the step's log file. The method now returns a dictionary mapping regex suffixes to the number of matches found, which is used to set error and warning metrics in the `run` method. Additionally, the schema and example documentation for regex configurations have been updated to use "errors" and "warnings" as special suffixes.


Add support for saving embeddings at intermediate epochs

Returns a list of intermediate embeddings in UMAP.embedding_list

Allow `n_epochs` to accept a list to return intermediate embeddings

This change modifies the `n_epochs` parameter in `optimize_layout_euclidean` and `simplicial_set_embedding` functions to optionally accept a list of integers. When a list is provided, the functions will return a list of embeddings at the specified epochs. It also updates the `UMAP` class to handle the new `n_epochs` behavior, including validation and storage of intermediate embeddings.


Change schema for population.

    MODEL cols AS stattype has become SET STATTYPE OF cols TO st

Reasoning: The word MODEL is confusing here, we are setting the statistical
data type so we should name the command just that.

Update command syntax for setting statistical types

This commit changes the syntax for setting statistical types in the command grammar and parser, updating all relevant test cases to match the new syntax.


Add MPI parallelization on different MPI communicators.

This is necessary to parallelize functions that are also parallelized.

Add grouping for distributed tasks

Implement `group_nodes` parameter in `distribute` function to allow for splitting nodes into isolated groups. This is useful when tasks themselves call `distribute()`. Added corresponding tests to verify the grouping functionality.


Document & correct use of class conversion methods

Recently added `{to,from}_{serialized,raw,dict}` methods for various
conversions were poorly documented, and sometimes even wrongly used.
This patch adds a short summary on when to use them, and fixes some
sub par bits around the code.

Introduce `SpecBasedContainer` and update serialization methods

The commit introduces a new base class `SpecBasedContainer` for handling conversion to and from specification formats, alongside updating various classes to use `to_spec`, `from_spec`, and `SpecBasedContainer` where appropriate. This change clarifies the purpose of serialization methods and integrates a more structured approach to handling specification data.


Create separate set of constants for operations

This is to differentiate between transaction table values and
entity operations.

Replace TxnActions with Operations for datastore operations

This commit introduces a new class `Operations` in `dbconstants.py` to represent datastore operations (PUT and DELETE) and updates all references from `TxnActions` to `Operations` in `cassandra_interface.py` and the corresponding tests in `test_datastore_server.py`.


Use library instead of ad-hoc code

This replaces `pretty_print_combos` with `json.dumps`, and
`compare_combos` with `jsoncomparison.Compare`.

Optimize combination JSON writing and comparison

Switched to using `jsoncomparison.Compare` for comparing JSON structures and simplified JSON writing. Added a verbose flag for detailed output. Removed redundant pretty-printing functions.


Add 4 block attack scenarios to PV python testing

Also add the needed option -pvtest which slows down the checking
of inputs by putting a sleep time of 1 second for each input to
check.  This allows up to simulate very long to validate blocks
used in the attack scenarios.

Add extensive parallel test scenarios

This commit adds a new test script for extensive parallel test scenarios in the `rpc-tests` directory. It also modifies the `ParallelTest` class to handle extensive test cases when the `--extensive` flag is provided. Additionally, it introduces a new debug argument `pvtest` to slow down input checking for testing purposes.


Allow `MetadataFactory.fetch_http` to skip sha256

`MetadataFactory.fetch_http` must often download HTML pages, not
Updates.xml files. download.qt.io does not store checksums for these
files, so this particular function must be allowed to download these
pages without using a checksum.

Add `is_check_hash` parameter to `fetch_http`

The `fetch_http` method now accepts an `is_check_hash` parameter to conditionally check the hash of the fetched content. Updated all call sites to pass `False` where hash checking is not required. Adjusted test cases to accommodate the new parameter.


Add ProcessStatus class

ProcessStatus tracks the process status and allows callbacks on changes
and status queries over the messagebus.

StatusCallbackMap is used to setup the callbacks

ProcessState is an enum tracking the different states.

Introduce ProcessStatus for skill manager status tracking

This change introduces a new `ProcessStatus` class to manage and track the status of the skill manager, replacing direct calls to callback hooks and message bus emissions with a structured approach.


Remove `TestTap` class from `gaia` tests

Grouping the tests together in a test class served no purpose and
eliminating the class removed a level of indentation.

Flatten `TestTap` class into functions

The `TestTap` class in `test_gaiatap.py` has been flattened into individual test functions to improve readability and maintainability.


Add shorter tags for models

Redefine most of models tags with a list including shorter aliases to be used with model.create(tag) and in YAML serialization. By default the tag returned in model.to_dict() is the 0th tag which is also the class name.

Update model tag handling to support multiple tags

This change modifies the handling of model tags to support a list of tags instead of a single string. It updates the `Model.to_dict` method to handle cases where `self.tag` is a list, and adjusts assertions in tests to check for the presence of any tag in the list rather than an exact match. Additionally, it updates the `Registry` class to iterate through all tags in a list to find a matching class.


Add options to post metrics to queue

This commit adds an option to SM to post metrics to cloudwatch.
Metric data will be posted whenever scan queue items are added or
removed.

Enable sending scanning performance metrics

Configure Security Monkey to send performance metrics for queue operations.


Attach list of groups to transaction node

This allows the transaction groomer to track down which entity
groups might have an entity lock that needs to be cleared after
the transaction is resolved.

Track entity groups for transactions

This change introduces a method to associate entity groups with transactions, ensuring that the transaction manager keeps track of which groups are involved in a given transaction. This is important for managing cross-group transactions and maintaining consistency.


Make _print_tcl() more robust

- Use {} to escape strings, so that we can have spaces and newlines
- Dump tuples as lists to avoid parsing
- Make sure envvar syntax gets subbed in anywhere in string

Convert Python values to safe TCL values in `_print_tcl` method

The `_escape_val_tcl` helper function is added to handle the conversion of Python values to safe TCL values based on the SC type string. This function is used in `_print_tcl` to ensure proper formatting and escaping for various data types, including complex strings, tuples, booleans, and environment variables. Corresponding test cases are added to validate the functionality.


Add auto-generated unique names

- add auto-generated unique names for model and dataset on init and copy
- force unicity on datasets and skymodels

Ensure unique names for datasets and models by default

The commit introduces a mechanism to automatically generate unique names for datasets and models when no name is provided, using `make_name` which generates a short UUID. It also updates the import statements and method signatures accordingly.


Support dont-optimize switch on all or individual match clauses

This will use a CROSS JOIN operator where appropriate which in SQLite
will disable the query optimizer for those joins and process the joins
in the order listed.

Add `dont_optimize` parameter to disable query optimization

This change introduces a new `dont_optimize` parameter in the `KgtkQuery` class to disable query optimization, particularly useful for SQLite queries. It involves modifying the initialization method to accept the new parameter and updating several methods to respect this flag when building SQL queries.


Move the list of LLVM version suffixes to a common place

Both scan-build and llvm-config need the same list of LLVM version
suffixes. It is better to keep the list at a common place instead of
having several copies in different files, which is likely to become
out-of-sync when the list is updated.

Extract LLVM tool names into a utility function

The commit introduces a utility function `get_llvm_tool_names` in `environment.py` to generate a list of possible LLVM tool names with different version suffixes. This function is then used in `dev.py` to simplify the code for the `LLVMDependencyConfigTool` class.


Add processing options for SkyModel and SkyDiffuseCube

Add options to apply or not psf and edisp to SkyModel and SkyDiffuseCube

Improve code formatting and add conditional processing for PSF and EDisp

The commit includes improvements in code formatting, such as better line breaks and alignment. Additionally, it introduces conditional checks for PSF and EDisp processing based on the `processing` attribute of the model.


Refactor to update pull requests with pending status

This commit refactors code to update the pull request with a pending
status at the earliest possible time.

Additionally, this refactor includes fixes an issue where unexpected
characters in the pull request title or body caused issues parsing
github webhook payload json

Enhance Jenkins pipeline to handle pull request actions more effectively

The Jenkinsfile has been updated to include more detailed handling of pull request actions, including setting up the pull request data and triggering tests or builds accordingly. The README.md has been updated with instructions on deploying the Jenkins instance and configuring it for pull request automation. The run_tests.py script has been modified to use environment variables for pull request data and includes improvements for handling unexpected characters in the pull request title and body.


Complete overhaul of filter testing.

These tests are much more flexible now for when additional filters are added.

Reorganize and parameterize filter tests

The diff reorganizes the test cases for various ndimage filters into a more structured and parameterized format. It introduces a base class `FilterTestCaseBase` to handle common functionality and parameters, and then defines several subclasses to handle specific test scenarios, including special cases for different filter types, modes, and dimensions. This refactoring improves maintainability and readability of the tests.


Allow user to pass "timeout" argument to ScriptDeployment and
ScriptFileDeployment class.

With this argument, user can specify optional command run timeout for
those deployment steps.

Add timeout parameter to ScriptDeployment and run methods

This commit introduces a `timeout` parameter to the `ScriptDeployment` and `ScriptFileDeployment` constructors, as well as to the `run` method in the `BaseSSHClient` class and its implementations. The `MockClient` class also has been updated to handle the timeout parameter, raising a `ValueError` if a timeout is specified. The tests have been adjusted to include cases with and without the timeout parameter.