**Feature Request: Multiple iovec support in `spdk_dif_generate_stream`**

**Background**
The function `spdk_dif_generate_stream` currently only accepts a single `iovec` (i.e., `iovcnt == 1`). When more than one scatter-gather entry is provided, the call fails with `-EINVAL`. All other DIF generate/verify functions in the library already support multiple iovecs, making this limitation inconsistent and preventing use of `spdk_dif_generate_stream` in data paths that deal with non-contiguous buffers.

**Acceptance Criteria**

1. **Multi-iovec input** – `spdk_dif_generate_stream` shall accept `iovcnt >= 1`. Passing `iovcnt > 1` must no longer return `-EINVAL`.

2. **Correct DIF computation** – Given an array of iovecs that collectively represent a logical data buffer with interleaved metadata (block size – metadata size = data block size, and each data block immediately followed by its metadata), the function shall compute and write correct guard, application tag, and reference tag values for every block that is at least partially covered by the range `[offset, offset + read_len)`.

3. **Scatter-gather transparency** – The DIF values produced must be identical to those that would be produced if the same logical data stream were laid out in a single contiguous buffer (using the existing single-iovec path). This must hold even when data blocks or metadata slots cross iovec boundaries.

4. **Partial-block start and length** – The caller may specify an `offset` that is not a multiple of the data block size, and a `read_len` that does not end on a block boundary. The implementation shall update DIF for every block that contains at least one data byte of the range (i.e., the block containing `offset` through the block containing `offset + read_len - 1`).

5. **Buffer size validation** – If the total accessible data (sum of all iovec lengths, after accounting for the space occupied by interleaved metadata) is insufficient to cover the required number of blocks implied by `offset` and `read_len`, the function shall return `-ERANGE`.

6. **Preservation of single-iovec path** – The existing single-iovec code path must remain unchanged; behavior for `iovcnt == 1` shall not regress in any way.

**Non-Goals**
- DIF types that store metadata in separate buffers (DIX) are out of scope; only interleaved metadata is supported.
- The function’s signature (parameters, return values, and error codes other than the newly allowed multi-iovec usage and `-ERANGE`) stays the same.

dif: Support multiple iovecs for data buffer in spdk_dif_generate_stream


The band metadata version is currently stored in the VSS area; it must be moved into the fixed metadata structure itself. Introduce a new version field at offset 0 of `struct ftl_band_md`, increment the current version to `FTL_BAND_VERSION_2`, and adjust the `reserved2` array size accordingly. Existing on-disk band metadata at version 1 must be upgraded on load: the upgrade process reads the v1 data, shifts the remaining bytes to make room for the new `version` field, sets the version to `FTL_BAND_VERSION_2`, and persists the new layout. VSS for band metadata is no longer used.

The upgrade must be safe across power cycles and crashes:
- Before writing the new layout, create the v2 region entry so that interrupted upgrades can be resumed by detecting that the v2 region already exists.
- If the upgrade is interrupted after the v2 region is created but before all bands are persisted, the upgrade path will re-read the v1 data and retry the entire region rewrite, ensuring atomicity.

During startup, `ftl_bands_load_state` must validate each band’s `version` field against `FTL_BAND_VERSION_CURRENT` and fail the load if a mismatch is detected. The existing upgrade descriptor table for bands should be extended with a `v1_to_v2` entry that performs this conversion. The API `ftl_bands_load_state` must return an error code so that callers (e.g., `ftl_md_restore_region`) can propagate failures appropriately.


There is a bug in the iSCSI target task cleanup: when a connection exits or a LUN is hot-removed, tasks are not fully freed in all cases, leading to resource leaks.

Currently the connection-exit path (iscsi_conn_free_tasks) frees all tasks and PDUs from write_pdu_list, snack_pdu_list, and queued_datain_tasks without any LUN filtering. The LUN-removal path (_iscsi_conn_remove_lun) contains separate, similar logic that frees only tasks whose associated LUN matches the removed LUN (or where the task’s LUN is NULL). This duplicate logic is out of sync: some edge cases — such as a PDU in snack_pdu_list with a NULL task pointer, or a queued_datain_task that is already marked is_queued — are not handled consistently between the two paths. Over time this has caused secondary tasks to be orphaned, leaking memory and potentially keeping backing resources alive longer than expected.

We need to unify the task-freeing logic into a single helper that takes a LUN parameter:
- If LUN is NULL (connection exit), free *all* matching tasks/PDUs from those three lists.
- If LUN is non-NULL (LUN removal), free only tasks whose `pdu->task->scsi.lun` is either the removed LUN or NULL, and similarly for queued_datain_tasks.
- For snack_pdu_list, check `pdu->task` before dereferencing it, and only put the task and free the PDU when the LUN matches (or on full exit).
- For queued_datain_tasks, skip tasks that have `is_queued == true` when checking the LUN filter.

The fix must be accompanied by comprehensive unit tests that cover all combinations of connection exit / LUN removal, for each of the three lists, with tasks having various LUN assignments, tasks with NULL LUN, tasks with NULL parent, and tasks flagged as is_queued.

Without this cleanup, hot-removal of a LUN can leave stale tasks on a connection, causing use-after-free when the task’s deferred completion eventually runs. The refactoring will also serve as a clean foundation for a subsequent patch that prevents secondary tasks from leaking when primary tasks are freed.

lib/iscsi: Factor out freeing iSCSI tasks to a helper function


`spdk_nvme_probe_internal` unconditionally initializes every probed controller by calling `nvme_init_controllers`, even when the current process is a secondary process that should only attach to a PCIe controller already brought online by the primary.  This causes secondary processes to fail when probing local NVMe drives because the secondary attempts PCI device register access and controller initialization that it doesn’t own.

**Expected behavior**
- When `spdk_nvme_probe()` or `spdk_nvme_connect()` is called with a PCIe transport and the process is *not* the primary (`!spdk_process_is_primary()`), the probe path should only discover existing controllers and attach to them, without performing any controller-level initialization (no reset, no feature/config commands, no queue allocation).  The primary process already owns that hardware.
- For the *primary* process, or for all other transports (e.g. fabrics), the initialization step must still happen immediately after the probe/attach so the controller reaches the ready state.
- The same split must apply to both the probe (`spdk_nvme_probe`) and direct connect (`spdk_nvme_connect`) entry points.
- The return value and `ctrlr` retrieval for `spdk_nvme_connect()` must remain correct when initialization is skipped.

**Impact**
- Secondary processes cannot reliably use local NVMe PCIe controllers in multiprocess SPDK applications; they crash or error out because they try to perform hardware operations that only the primary is allowed to do. Fabric controllers and primary-process PCIe controllers work fine, highlighting the missing per-transport/per-process gating.

**Request**
Split the internal probe function so that controller discovery and initialization are separate, callable stages.  Move the initialization step (`nvme_init_controllers`) up into the public API functions (`spdk_nvme_probe`, `spdk_nvme_connect`), and gate it on `(primary || transport != PCIe)`.  Keep the existing error handling semantics so that a failure during the initialization phase propagates correctly.

nvme: broke up spdk_nvme_probe_internal() into two stages


After an iSCSI target is created, removing a portal group mapping from it does not clean up the corresponding SCSI port. The port remains on the SCSI device, making it impossible to re-add the same portal group later and leaving a stale port that is still reported even though the mapping no longer exists. Conversely, when a portal group mapping is added to an already-running target, no SCSI port is created at all, so the new mapping is not usable for serving LUNs.

This breaks any workflow that requires dynamic reconfiguration of portal groups on an existing target (e.g., moving a target to a different network interface or adding a new portal without tearing down the target). Expected behavior:

- Adding a portal group mapping to a target should atomically register a new SCSI port (using the portal group tag and a name derived from the target) so that the target is immediately reachable through that portal group.
- Removing a portal group mapping should unregister the matching SCSI port from the target’s SCSI device.
- The per-device limit on SCSI ports (SPDK_SCSI_DEV_MAX_PORTS) should be enforced when adding a new mapping; attempting to exceed it should fail cleanly without leaving partial state.
- Adding a mapping for a portal group that is already mapped should be a no‑op or return an error, but must not duplicate the port.

The fix should also avoid any regression for the existing static configuration path: targets created at startup with the initial set of portal groups must continue to work normally, with SCSI ports created for each initial mapping and cleaned up on target destruction.

iscsi: Support SCSI port removal when PG map is removed


**Bug Report: NVMe-oF Target Does Not Support Changed Namespace List Log Page**

**Summary:**
When a namespace attribute change occurs (e.g., namespace added or removed), the NVMe-oF target generates an Asynchronous Event Request (AER) completion with the “Namespace Attribute Changed” notice. However, the AER completion’s **Log Page Identifier** field is set to `0x00`, indicating no log page is associated with the event. Additionally, the corresponding **Changed Namespace List log page** (Log Identifier `0x04`) is not implemented. Consequently, connected hosts cannot efficiently discover *which* namespace(s) changed without scanning all namespaces.

**Expected behavior:**
Per the NVMe specification, when a namespace attribute change occurs:
1. The target should maintain a **Changed Namespace List** per controller, recording the NSIDs of namespaces that have changed attributes since the last time the list was retrieved.
2. The AER completion for “Namespace Attribute Changed” should set its **Log Page Identifier** field to `0x04` (Changed Namespace List), instructing the host to fetch that log page.
3. A host issuing a **Get Log Page** command with Log Identifier `0x04` should receive the list of changed NSIDs. Upon successful read, the target should clear the list so that subsequent reads only return new changes.
4. If more namespace changes occur than can fit in the log page (`struct spdk_nvme_ns_list` contains 1024 entries), the first entry should be set to `0xFFFFFFFF` and the rest zeroed, in accordance with the specification’s overflow semantics.

**Current behavior (observed):**
- The AER “NS Attribute Changed” event uses Log Page Identifier `0x00`, so the host is not directed to a log page.
- The `Get Log Page` command with Log Identifier `0x04` is not recognized by the target; the target returns a generic “invalid log page” error.
- There is no per-controller state tracking which namespaces changed.

**Impact:**
Drivers and host software cannot rely on the standard NVMe mechanism to identify changed namespaces after an attribute change notification. This forces workarounds (e.g., re-identifying all namespaces), which is inefficient and non‑compliant with the NVMe specification. This limitation affects correctness of hot-plug/hot-remove scenarios and dynamic namespace management in fabrics environments.

**Request:**
Implement support for the **Changed Namespace List** log page (Log Identifier `0x04`) in the NVMe-oF target, including:
- Per‑controller tracking of changed NSIDs.
- Populating the AER “Namespace Attribute Changed” event with Log Page Identifier `0x04`.
- Correct handling of the `Get Log Page` command for Log Identifier `0x04`, with clearing semantics and overflow indication.

nvmf: implement Get Log Page - Changed NS List


**Summary**: RAID5F volumes do not compute or write parity blocks, leaving all stripes unprotected and effectively acting as RAID0. This violates the RAID5 data redundancy guarantee.

**Observed Behavior**
When a full-stripe write is submitted to a RAID5F bdev, the module writes only the data chunks to the participating base bdevs. The parity chunk is neither computed nor submitted. The IO completion path assumes `base_bdev_io_remaining` is `N-1` (data chunk count only), so the stripe is considered fully written after data chunks complete.

**Expected Behavior**
A full-stripe write must:
1. Calculate XOR parity across all data chunks in the stripe, using the SPDK xor library.
2. Write the parity chunk to the parity base bdev.
3. Include the parity write in the outstanding IO counter (`base_bdev_io_remaining`) so that completion is only reported after all `N` base bdev writes (data + parity) are finished.
4. Handle alignment requirements: source and destination buffers must be aligned as required by the xor engine. If any data chunk’s iovecs are misaligned, the module must transparently bounce the data into aligned buffers before computing parity.
5. If parity calculation fails for any reason, the stripe write must be aborted with an IO failure status.

**Impact**
- No data protection; a single base bdev failure will cause total data loss.
- Application expectations of RAID5 are violated without any visible error.
- The missing parity writes also mean the durability guarantees described in documentation are not met.

**Additional Notes**
- The parity buffer must be allocated with sufficient alignment for both the xor engine and the parity base bdev, and freed when the stripe request is released.
- The module already defines a parity chunk (`parity_chunk`) inside the stripe request; write submission currently skips over it. That skip must be removed so that parity reaches the disk.
- The `raid5f_io_channel` should pre-allocate bounce buffers (one per data chunk) to avoid runtime allocation stalls during parity calculation when misaligned iovecs are encountered.


The AIO bdev module lacks a proper delete path for normal operation. The only existing way to remove an AIO disk is via the generic `delete_bdev` RPC, which is intended for debugging only and not for production lifecycle management. This causes problems for tools (such as `spdkcli`) that need to tear down AIO bdevs safely, and for configuration clearing scripts that rely on the correct per-module deletion method.

**Observed behavior**
- There is no `delete_aio_bdev` RPC method.
- The `spdkcli` UI uses a generic delete that bypasses the module-specific cleanup.
- The `clear_config.py` script maps AIO construction to `delete_bdev` instead of a dedicated deletion method.
- Attempting to remove an AIO bdev via the debug-only `delete_bdev` may lead to inconsistent state or is unsuitable for regular use.

**Expected behavior**
- A new `delete_aio_bdev` RPC method exists, accepting the bdev name as argument.
- The method validates that the named bdev exists and is an AIO bdev (returns `-ENODEV` when it is not).
- On success, the method calls a dedicated unregistration callback (analogous to `spdk_bdev_unregister`) and writes a JSON-RPC result (`true`/`false` based on success).
- A corresponding Python RPC client function (`delete_aio_bdev` in `rpc/bdev.py`) and CLI command (`delete_aio_bdev` in `rpc.py`) are provided.
- `spdkcli` uses the new per-module delete command for AIO bdevs.
- The configuration-clearing logic in `test/json_config/clear_config.py` maps `construct_aio_bdev` to `delete_aio_bdev` instead of the generic `delete_bdev`.

**Impact**
Without this change, there is no safe, official way to remove AIO bdevs at runtime. Management tools and automation scripts cannot correctly tear down AIO disks, risking stale resources or forcing users to rely on debug interfaces not designed for production use.


Extract the non-read task completion logic from `spdk_iscsi_task_cpl()` into a dedicated helper function to mirror the existing `process_read_task_completion()`. This extraction must preserve all existing behavior exactly, improve code clarity, and enable unit testing of the non-read completion path.

**Acceptance Criteria**

1. The non-read (write and other non-read) completion handling currently inline in `spdk_iscsi_task_cpl()` is moved to a separate static function with a signature comparable to `process_read_task_completion()`.
2. The extracted function correctly handles all observable outcomes:
   - Accumulates `bytes_completed` on the primary task.
   - Remembers the first failed subtask SCSI status (`CHECK_CONDITION` etc.) across subsequent subtasks, and propagates it to the primary when the full transfer completes.
   - Updates `scsi.data_transferred` on the primary only for subtask successes (excluding failed subtasks).
   - Removes the primary task from `active_r2t_tasks` and decrements its reference when the final subtask completes (including the tricky case where the primary itself is the final completion, i.e., initial task length < transfer length).
   - Releases the subtask reference with `spdk_iscsi_task_put()`.
3. No externally visible change in task completion sequences, error reporting, or connection lifecycle. Regression is not permitted.
4. Unit tests are added for the new function, covering at least:
   - A split write where an intermediate subtask fails, and later subtasks succeed, verifying correct status, bytes, and `data_transferred` propagation.
   - The scenario where the initial primary task is also the final completion (length != transfer_len), ensuring `active_r2t_tasks` cleanup and correct byte counts.
   - Verification that `active_r2t_tasks` is empty after the last subtask completion.
5. `spdk_iscsi_task_cpl()` delegates non-read completions to the new function instead of executing the extracted code directly. The read path continues to use `process_read_task_completion()` unchanged.
6. All existing unit tests for iSCSI task completion continue to pass without modification.

lib/iscsi: Factor out non-read task completion from spdk_iscsi_task_cpl()


**Title:** iSCSI LUN hot-removal may leave orphaned PDUs referencing a freed LUN, causing a crash

**Description:**
When a LUN is hot-removed from an iSCSI target node while there are active tasks in the connection’s `queued_datain_tasks` queue, the connection cleanup (`_iscsi_conn_free_tasks`) can leave behind PDUs in the `write_pdu_list` that still reference the freed LUN.  These orphaned PDUs are later freed and trigger a use‑after‑free of the LUN, resulting in a critical failure (assert/crash).

**Acceptance Criteria:**
1. After a LUN is removed, all PDUs associated with that LUN are freed before the LUN object itself is destroyed, even if some of those PDUs were temporarily in the `write_pdu_list` because they were moved from `queued_datain_tasks` during the cleanup procedure.
2. The cleanup routine must process the `queued_datain_tasks` queue before iterating the `write_pdu_list`, ensuring that any new PDUs added to `write_pdu_list` as a side effect are also considered for removal while the LUN reference is still valid.
3. No PDUs should remain in the `write_pdu_list` after `_iscsi_conn_free_tasks` completes when called with a specific LUN (non‑NULL argument), and all related task references are released cleanly.
4. The connection remains stable and no crash or assertion occurs during or after LUN hot-removal under the above scenario.

**Constraints:**
- The existing link/unlink order of PDUs and tasks in the connection’s internal queues must not be reversed arbitrarily; only the relative order of processing the queues in the cleanup function is adjusted to satisfy the criteria.
- The fix must not introduce new leaks or invalid memory accesses for any code path that frees tasks on connection teardown (e.g., when `lun == NULL`).

lib/iscsi: Fix orphaned PDUs when freeing PDUs and tasks for LUN hotplug


The crypto vbdev module does not properly drain outstanding crypto operations when a reset command is received. This can leave encryption/decryption requests in flight across the underlying crypto device, leading to use-after-completion, data corruption, or resource leaks.

**Acceptance criteria:**

1. When a `RESET` IO is submitted to a crypto bdev, the module must ensure that every crypto operation (both encryption and decryption) that was issued before the reset and is still queued or in-flight to the crypto device is completed **before** the reset IO itself is completed.
2. Any crypto operation that was still pending at the start of the reset must be failed with `SPDK_BDEV_IO_STATUS_FAILED` rather than being processed after the reset.
3. The reset IO must not be completed until all per-channel queues of pending crypto IOs are empty and no further crypto completions can occur for the affected bdev. This applies across all I/O channels associated with the bdev.
4. The bdev must be able to accept new I/O after the reset completes, with a clean crypto processing state (no stale operations from before the reset).
5. The quiesce mechanism must not introduce deadlocks: the module's poller should remain active to drain completions, and the reset completion path must be triggered only after the last channel finishes draining.
6. Edge cases to cover:
   - A reset arriving while crypto operations are actively being dequeued from the hardware.
   - Multiple resets on the same bdev (though a second reset could be handled gracefully).

The intended observable effect is that a reset on a crypto bdev behaves like a standard block layer reset that flushes all pending work, returns failure for any unfinished I/O, and leaves the device in a consistent, usable state.


When a namespace goes inactive (e.g., after detach or an AER), `nvme_ns_set_identify_data()` retains stale copies of `id_desc_list` and IOCS‑specific data, and its derived fields (sector_size, etc.) may still reflect the previous active state.  `nvme_ctrlr_identify_ns()` has its own error‑handling clear that returns success, bypassing any common inactive‑cleanup logic.

Make `nvme_ns_set_identify_data()` call `nvme_ns_clear()` whenever it detects the namespace is inactive, so that all per‑namespace dynamic data is freed and `nsdata` is zeroed, regardless of which code path triggered the update.  After this change, an inactive namespace will have a zeroed `nsdata` and no stale IOCS or descriptor data.

Simplify `nvme_ctrlr_identify_ns()` to log a warning on an identify failure and fall through to `nvme_ns_set_identify_data()`, which now handles the inactive case uniformly; remove the separate clear‑and‑return‑success path.  The function must not perform any derived‑field computation on data that belongs to a past active incarnation.

nvme/ns: clear stale namespace data when identify finds inactive NS


Update all iobuf initialization, channel create/destroy, stats, abort, and for_each_entry paths to operate over all active NUMA nodes (or only node 0 when NUMA is disabled), so the infrastructure supports multiple nodes in preparation for per-node allocation. The get/put functions must still allocate from node 0 only.

Introduce a single `IOBUF_FOREACH_NUMA_ID` iterator that, when `g_iobuf.opts.enable_numa` is true, walks nodes returned by `spdk_env_get_first_numa_id` / `spdk_env_get_next_numa_id`; otherwise it iterates only node 0. Use this iterator in:

- `spdk_iobuf_initialize` – allocate per-node pools; on failure, free all previously initialized nodes.
- `iobuf_channel_create_cb` / `iobuf_channel_destroy_cb` – initialize/assert-empty per-node queues.
- `iobuf_node_free` – guard against uninitialized nodes (small_pool == NULL) and skip them.
- `spdk_iobuf_channel_init` – call `iobuf_channel_node_init` and `iobuf_channel_node_populate` for each node; propagate errors to cleanup.
- `spdk_iobuf_channel_fini` – call `iobuf_channel_node_fini` for each node.
- `spdk_iobuf_for_each_entry` – iterate both small and large pool caches across all nodes.
- `spdk_iobuf_entry_abort` – attempt abort on each node via the helper `iobuf_entry_abort_node`.
- `iobuf_get_channel_stats` – accumulate stats from all nodes.

`iobuf_node_free` in `spdk_iobuf_initialize` error path and in `iobuf_unregister_cb` must use the iterator. The iterator must terminate by setting the loop variable to `INT32_MAX` when no more nodes exist.

thread: update all iobuf non-get/put functions for multiple NUMA nodes


# Rework crypto vbdev’s handling of cryptodev enqueue failures

**Background**
In `module/bdev/crypto/vbdev_crypto.c`, the submission path (`_crypto_operation`) currently calls `rte_cryptodev_enqueue_burst` and, if fewer operations are accepted than prepared, enters a tight loop that invokes the poller (`crypto_dev_poller`) to drain completions and then retries the enqueue until all ops have been accepted. This implementation assumes that the only reason for a partial enqueue is a full hardware queue.

**Problem**
An upcoming DPDK cryptodev API change will introduce situations where `rte_cryptodev_enqueue_burst` can fail for reasons *other* than “device full.” The current spinning retry would then become an infinite loop, blocking the submission thread indefinitely.

**Requested change**
Replace the spinning retry with a non-blocking deferred‑submission model, consistent with the pattern already used in the `compress` vbdev (`module/bdev/compress/vbdev_compress.c`). The goal is to handle temporary enqueue failures gracefully without assuming a specific failure cause.

Concrete requirements:

1. **Remove the spinning loop**
   In `_crypto_operation`, after the initial `rte_cryptodev_enqueue_burst` call, do **not** loop to retry. Instead, immediately handle any unsubmitted operations by queuing them for later.

2. **Queue for later resubmission**
   - Add a new `TAILQ_HEAD` named `queued_cry_ops` to `struct crypto_io_channel`.
   - Define a small structure (`vbdev_crypto_op`) to hold the necessary resubmission context per op: `cdev_id`, `qp`, a pointer to the `rte_crypto_op`, and a pointer to the owning `spdk_bdev_io`.
   - When `enqueue_burst` does not accept all ops, wrap each remaining op in this structure and insert it into `queued_cry_ops`.

3. **Reserve space inside the op for queue metadata**
   The crypto op pool already reserves per-op private data for the IV. Extend that private area by `sizeof(vbdev_crypto_op)` so the queuing structure can be stored directly inside each op without additional allocations. Update `QUEUED_OP_OFFSET` and the pool creation call accordingly (e.g., `AES_CBC_IV_LENGTH + QUEUED_OP_LENGTH`).

4. **Deferred submission from the poller**
   In `crypto_dev_poller`, after processing newly dequeued completions and before the channel-iterator check, walk the `queued_cry_ops` list. For each entry, attempt to enqueue exactly one op via `rte_cryptodev_enqueue_burst`. If successful, ensure the associated `spdk_bdev_io` is placed on the `pending_cry_ios` list **once** (introduce a flag like `on_pending_list` to prevent double-insertion), then remove the entry from the queued list. If an enqueue fails, stop the walk and leave the remaining entries for the next poller invocation.

5. **Maintain pending-list invariants**
   - A `bdev_io` must be added to `pending_cry_ios` only when *at least one* of its crypto ops successfully entered the device.
   - The poller must process the queued list even while tearing down a channel (i.e., before the `iter` check), so that queued operations are eventually submitted and completed during reset.

6. **Consistency with compression vbdev**
   The overall approach should mirror what `vbdev_compress` already does for its equivalent busy/retry scenario, making both modules’ interaction with the DPDK framework uniform and easier to maintain.

**Observable behavior**
- No change to I/O correctness; operations queued due to a transient full device complete as soon as space becomes available, without blocking the submission thread.
- No infinite loops when `rte_cryptodev_enqueue_burst` fails for non-“full” reasons (the exact conditions will be handled in a follow-up patch once the DPDK API change lands).

module/crypto: change how we handle crypto device full condition


Bug: `spdk_bs_init` does not validate the `spdk_bs_opts` configuration, allowing invalid values that cause crashes, data corruption, or confusing downstream failures.

**Observed Problem**
- Passing `0` for any of the options (`cluster_sz`, `num_md_pages`, `max_md_ops`, `max_channel_ops`) leads to division-by-zero, infinite loops, or assertion failures during initialization.
- Specifying a `cluster_sz` larger than the underlying block device size succeeds, but later blob operations fail silently or produce corrupt data.
- Using a `num_md_pages` value that results in metadata clusters exceeding the total number of clusters on the device causes the blobstore to claim invalid clusters, leading to out-of-bounds accesses.
- The check for “use default number of metadata pages” compares against the magic number `UINT32_MAX` instead of the defined constant `SPDK_BLOB_OPTS_NUM_MD_PAGES`.

**Expected Behavior**
`spdk_bs_init` should immediately reject invalid configurations and return an appropriate error:
- Return `-EINVAL` if any option is set to zero.
- Return an error (preferably `-EINVAL`) if `cluster_sz` exceeds the device size.
- Return `-ENOMEM` if the number of metadata pages would require more clusters than are available on the device.
- Log descriptive error messages for each failure so the caller can correct the configuration.
- Use the defined constant `SPDK_BLOB_OPTS_NUM_MD_PAGES` for the default metadata pages detection instead of the hardcoded `UINT32_MAX`.

**Impact**
Without these checks, users can unknowingly configure the blobstore in ways that lead to crashes or silent data corruption, making the API fragile and hard to debug.

blobstore: verify options passed to spdk_bs_init


The Get Log Page admin command is handled separately for regular NVM controllers and discovery controllers, leading to duplicated validation logic and the risk of diverging error responses. Unify the command processing so that both controller types exhibit identical input validation and error reporting for this command. The discovery log page must still be served for discovery controllers, while regular controllers return empty log pages for the currently supported log identifiers.
Specifically:
- Both controller types shall reject the command with a Generic Command Status / Invalid Field status when no data buffer is provided, when the log page offset is not dword-aligned, or when the requested log length exceeds the data buffer length.
- Discovery controllers shall continue to return the Discovery log page (LID 0x70) using the existing discovery log generation.
- Regular NVM controllers shall accept Error Information, Health Information, and Firmware Slot log pages, returning a zero-filled buffer (with a TODO for future log content), and shall reject any other log page identifier with Generic Command Status / Invalid Field.
- The separate admin command processing functions for regular and discovery controllers must use a single, shared implementation for Get Log Page rather than each containing its own copy of the logic.

nvmf: refactor get_log_page into a common function


**Title:** `comet_input_file_name` column not available for position-based schemas

**Component:** Ingestion (position file handler)

**Description**
Position‑based file ingestion does not expose the `comet_input_file_name` column in the parsed DataFrame. This means any script‑defined field that references `comet_input_file_name` (e.g. `script: "comet_input_file_name"`) fails during ingestion because the column is missing. DSV ingestion already provides this column automatically using Spark’s `input_file_name()` function, and the same behaviour should apply to position files.

**Expected behaviour**
- When a position file is ingested, the parsed DataFrame must contain a column named `comet_input_file_name` with the source file path.
- The column must be present before script‑processing runs, so that scripted fields can reference it.
- Attribute reordering logic (which aligns attribute definitions with the DataFrame’s columns) must include this extra column, analogous to how DSV ingestion handles it.

**Constraints / notes**
- The column should be added at dataset construction time (in `PositionIngestionUtil`) using `input_file_name()`.
- The `comet_input_file_name` column should not be part of the user‑defined schema attributes (`attributesWithoutScript`), but the ingestion flow must still account for it when computing `orderedAttributes`.
- The fix should avoid duplicating the DSV reordering logic; ideally the two formats share the approach.
- No change to the external schema definition YAML is required; this is entirely an internal plumbing fix.

**Acceptance criteria**
- Ingesting a position file with a script field `script: "comet_input_file_name"` succeeds and writes the file name into that field.
- The column is visible in the accepted/rejected DataFrames without manual schema changes.

bugfix : comet_input_file_name missing in position files


**Request: Spark BigQuery APPLICATION_DEFAULT auth should use JSON key file, not access token**

## Problem description
When running Starlake with a BigQuery sink and `authType` set to `APPLICATION_DEFAULT`, the Spark job currently obtains a short-lived access token from the application default credentials and passes it via `gcpAccessToken`. This can cause authentication failures (for example, in long-running jobs where the token expires) because the Spark BigQuery connector expects a base64-encoded service account JSON key under the `credentials` configuration key.

## Desired behavior
The `APPLICATION_DEFAULT` authentication flow should behave consistently with the existing `SERVICE_ACCOUNT_JSON_KEYFILE` flow: locate the application default credentials file (the JSON key file, typically specified by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable or a well-known gcloud location), read its content, and base64-encode it as the `credentials` Spark configuration. The old access-token path should be removed.

## Scope and constraints
- In the BigQuery Spark job, when `authType` is `APPLICATION_DEFAULT`, stop using `GoogleCredentials` and `gcpAccessToken`; instead, find the JSON key file, base64-encode its content, and set `session.conf.set("credentials", …)`.
- `GcpUtils` should expose a way to retrieve the path to the application default credentials file (checking `GOOGLE_APPLICATION_CREDENTIALS` or the well-known credentials file) without requiring a full credential object.
- No change to the `SERVICE_ACCOUNT_JSON_KEYFILE` path is needed; it already works correctly.
- The fix must not introduce new external dependencies beyond what is already in use.


**Symptom**
Running `DDL2Yml` against a JDBC database produces a domain YAML file whose table schemas have no primary key or foreign key information. Columns that are part of primary keys or foreign keys are indistinguishable from ordinary columns.

**Expected behavior**
- Each table schema should contain a `primaryKey` list with the column names that form the table’s primary key (an empty list or absent field if there is no primary key).
- For every column that participates in a foreign key constraint, the generated column definition should include a `foreignKey` field set to the fully qualified reference of the referenced primary key column, following the project’s existing convention: `schema.table.column` when the referenced table lives in a schema, otherwise `table.column`.

**Impact**
Without this metadata, schemas generated from live databases miss crucial constraints. Downstream processes that rely on schema definitions (merging, deduplication, relationship traversal, integrity checks) cannot leverage the foreign-key/primary-key information. Users are forced to manually annotate the generated YAML, which defeats the purpose of schema extraction.

**Acceptance criteria (high-level)**
- `primaryKey` is populated from `DatabaseMetaData.getPrimaryKeys`.
- Foreign key references are derived from `DatabaseMetaData.getImportedKeys`, mapping each local foreign-key column to the referenced primary-key column using the `<schema.>table.column` format.

DDL2Yml -> detect primary keys & foreign keys


**Title:** Add Quack connection state management and configuration models

**Background**
We are introducing a “Quack” subsystem (long-running connection proxies). We need the two core model components so downstream work can proceed: a typed configuration holder and a runtime-state tracker with on-disk persistence and automatic orphan cleanup.

**Request**

1. **Configuration model** — a case class in `ai.starlake.job.quack` that holds the CLI/configuration inputs for a Quack session: `action`, optional `connectionName`, optional `bind`, optional `port`, optional `token`, and optional `reportFormat`. It must extend the existing `ReportFormatConfig` trait.

2. **Runtime state model** — a case class that captures:
   - `connection` name
   - process `pid`
   - `bind` address
   - `port`
   - `logFile` path
   - `startedAt` timestamp

   A companion object must provide:
   - JSON round-trip via Jackson (with Scala module, ignoring unknown properties).
   - State files stored as `<connection>.json` under `<app-root>/.quack/` (the root comes from `settings.appConfig.root`).
   - `write` / `read` / `delete` helpers for individual connection states.
   - A `list` method that reads all `.json` files in the state directory and **drops any entry whose PID is no longer alive**, deleting the stale file on disk. Use `java.lang.ProcessHandle`.
   - Convenience accessors for `.quack/logs/` directory and per-connection log file paths.

**Additional context**
The “root” directory for state should be derived from `Settings.appConfig.root` (URI parsed to a local path) — the same way other job components locate their working directories. Any file I/O must be safe against missing directories (create as needed).


**Feature Request: Support `--from-json-schema` in `infer-schema`**

The `infer-schema` command currently only derives a Starlake table schema by inspecting a sample data file (CSV, JSON, XML, …). We often work with JSON Schema documents that already serve as the canonical definition of our data structures, and we would like to generate the corresponding Starlake YAML directly from them without providing a data sample.

**Expected behavior**

A new CLI flag (e.g., `--from-json-schema`) should tell `infer-schema` to treat the input file as a JSON Schema instead of a data file. The command must:

- Read the JSON Schema and extract the top-level `properties` and `required` arrays.
- Map the following JSON Schema types to Starlake types:
  - `"object"` → `struct` (recursively processing nested properties)
  - `"array"` → output an array. If the items are of type `"object"`, the array's element is a `struct` (again recursively); otherwise map primitive item types:
    - `"integer"` → `long`
    - `"number"` → `double`
    - `"boolean"` → `boolean`
    - any other item type → `string`
  - `"integer"` → `long`
  - `"number"` → `double`
  - `"boolean"` → `boolean`
  - any other type (or missing type declaration) → `string`
- For each property, set `required: true` if the property name appears in the top-level (or nested) `required` array, otherwise `required: false`.
- When a property’s type array contains multiple values (e.g. `["string","null"]`), the implementation should pick the first non-null type for mapping (commonly the first array element).
- The output must follow the existing Starlake YAML schema format for a `table`, using the usual domain name and table name parameters, the inferrable pattern, and a metadata block set appropriately for JSON input.

**Impact**

Today the only way to produce a Starlake schema from a JSON Schema is a manual, error-prone translation. This feature eliminates that manual step and ensures the generated schema stays aligned with the JSON Schema source of truth. It should be a natural extension of the existing `infer-schema` workflow, reusing all the standard scaffolding (domain output directory, `clean` flag, encoding, etc.) without affecting the other inference paths.


```
Title: Accept `--deltaColumn` CLI argument for extract script generation instead of schema merge timestamp

**Context**
The `comet script-gen` command produces extract SQL scripts from domain schemas.
Currently, for APPEND-mode exports it reads the delta column from each schema's `merge.timestamp` field. This couples the extract logic to ingestion merge settings.

**Desired behaviour**
- Add a new optional `--deltaColumn <column_name>` argument to `script-gen`.
- When supplied, use it as the `delta_column` Mustache template variable for every schema that is **not** a full export (i.e. write mode is APPEND).
- When not supplied, `delta_column` should be absent/None (no fallback to the schema's `merge.timestamp`).
- The generator must **stop** reading the delta column from `schema.merge.timestamp` entirely.
- The existing rule that full exports (OVERWRITE) never receive a `delta_column` must remain unchanged.
- Update the command’s usage/help text to document the new option.

**Why**
We need to decouple extract logic from ingestion configuration; the delta column for exports should be explicitly controlled by the job that invokes `script-gen`, not by the schema’s merge settings.
```

Using a main arg to get delta columns instead of timestamp excel col


# Improve KafkaJob load/offload flexibility and fix partition access

## Problem
`KafkaJob.run()` currently mixes load and offload logic into one method, with the offload path hardcoded to `/tmp/json` and no way to pass an in-memory DataFrame as input or retrieve the offloaded DataFrame programmatically. The `KafkaJobConfig` only holds a `path` string and a `store` flag, which does not express what kind of input is being used or whether the result should be saved to a file vs. returned.

Additionally, `KafkaTopicUtils.topicPartitions(topicName)` uses only the server‑side AdminClient options. When operating against a Kafka container with custom access options (e.g., a different bootstrap server passed to consumers/producers), partition discovery can fail because those access options are not used. This makes it impossible to correctly consume from or produce to topics configured with their own access‑options blocks in the Comet configuration.

## Required behavior

1. **Separate load and offload operations** – The `KafkaJob` must provide distinct entry points for loading data into Kafka and offloading data from Kafka, so that callers (including tests) can invoke each independently.

2. **Flexible input for load** – Loading should accept either a local file path (as before) or an existing Spark `DataFrame`. The `KafkaJobConfig` must be extended to carry this input specification.

3. **Flexible output for offload** – Offloading should support:
   * saving the consumed DataFrame to a configurable path (not hardcoded),
   * returning the DataFrame directly (e.g., for test assertions or further processing),
   * optionally merging with an existing DataFrame based on a `SaveMode` (`Append`, `Overwrite`, `Ignore`) – `ErrorIfExists` should be rejected with a clear exception since it has no meaningful semantics here.

4. **SaveMode on offload** – The config should allow specifying a Spark `SaveMode` (defaulting to `Append`) that controls how the offloaded data is written to the target path.

5. **Partition access respecting topic access options** – `KafkaTopicUtils.topicPartitions` and its callers must use the access options configured for the specific topic (e.g., from `settings.comet.kafka.topics(topic).accessOptions`) so that partition listing and offset retrieval work correctly with per‑topic bootstrap configurations.

## Constraints
* Keep backward compatibility for any existing scripts that pass `--path` and `--offload` flags via the CLI parser.
* The overall `run()` method should still dispatch to either `load()` or `offload()` based on the `offload` flag.
* In the `sinkToTopic` method, the fold on `accessOptions` currently names the accumulator `reader` – this should be `writer` to avoid confusion (minor cosmetic fix).
* The fix should make it feasible to write an integration test that loads 5000 JSON records into Kafka and then offloads them, verifying the count, without relying on hardcoded temporary paths.

## Affected areas (for orientation)
- `KafkaJobConfig` – add `mode`, replace `path`/`store` with a richer `input`
- `KafkaJob` – split into `load`/`offload`, handle the new config fields
- `KafkaTopicUtils` – change `topicPartitions` signature and usage; ensure `topicEndOffsets` and `topicCurrentOffsets` adapt
- Test spec `KafkaJobSpec` – after the refactoring it should be possible to write a meaningful load‑offload cycle using the embedded Kafka container


**Title:** Transform schema diff reports spurious changes when SQL query is unchanged

**Context**
When a transform task’s SQL has not changed, Starlake may still attempt a schema synchronization because the diff between the SQL-derived attributes and the YAML attributes incorrectly reports modifications or deletions. This leads to unnecessary updates and can cause failures or performance regressions in orchestrated pipelines.

**Observed issues**
1. `SQLUtils.extractTableNames` treats fully qualified column references (e.g., `SELECT domain.table.column`) as table names, producing false dependencies and potentially triggering unnecessary schema comparisons.
2. In `AutoTaskInfo.diffSqlAttributesWithYaml`, when every attribute is `UNCHANGED` and there are no `ADDED` or `REMOVED` attributes, the method still returns the list of `UNCHANGED` pairs. The caller should treat this as “no change” and avoid any synchronization.
3. `AutoTaskInfo.syncPreviewSqlWithYaml` calls `diffSqlAttributesWithYaml` even when the extracted SQL statement attributes are empty (e.g., due to parsing errors or unsupported SQL constructs). In such cases, the diff should be skipped entirely.
4. The attribute extraction from SQL currently does not handle deep schemas (tables containing columns of type `struct`). For those cases, the SQL-based diff should be ignored (return empty) to avoid incorrect behaviour.

**Expected behaviour**
- `SQLUtils.extractTableNames` must return only actual table references from `FROM`/`JOIN` clauses, not column qualifications.
- `diffSqlAttributesWithYaml` must return an empty list when there are no actual changes: no added, removed, modified, or changed-comment attributes.
- `syncPreviewSqlWithYaml` must skip the SQL-vs-YAML diff and return an empty list when the extracted SQL attributes are empty.
- If any source table in the dependency chain contains a `struct` column, the SQL-based attribute diff should be skipped (return empty) for that task.

**Acceptance criteria**
- Running a sync operation on a transform task whose SQL is unchanged should not attempt any schema update.
- Column references like `schema.table.column` in a `SELECT` should not be mistaken for table dependencies.
- No regressions for tasks with actual schema changes.


Bug: Column lineage fails to include column-level information for tables that are referenced in SQL but not defined in the Starlake domain schemas.

When running the `col-lineage` command (e.g., `starlake col-lineage --task domain.task_name`), the generated lineage output correctly shows table-level dependencies, but the column-level map is empty or missing for any table that is known only through an external connection (i.e., tables not part of the project's `load`/`transform` schema definitions). This means that for SQL queries joining or referencing such external tables, the resulting lineage lacks fine-grained column origins.

Expected behavior: If the task's connection already references an external system (as configured in the connection YAML), and the user provides an optional authentication token suitable for that system (via a new `--accessToken` flag on the CLI), then during lineage computation Starlake should attempt to retrieve the schema for any missing table from the external source and use the discovered column names and types to populate column-level lineage. If no token is provided, or the schema retrieval fails, the tool should still produce the best-effort lineage without error, but ideally log a warning.

This would bring external tables up to parity with native Starlake-managed tables, ensuring complete column lineage views for pipelines that span both internal and external data sources.


**Request (defect report / acceptance criteria)**

When a BigQuery dataset name is supplied with an explicit project ID (format `project-id:dataset`), several parts of Starlake’s BigQuery pipeline fail because the project is not extracted from the dataset identifier. Instead, the code treats the entire `project:dataset` string as the dataset name, which leads to invalid TableId construction and BigQuery API errors (e.g., “dataset not found”, or silently ignoring the project).

The visible symptoms are:

- Ingest or `bqload` jobs that specify `outputDataset = "my-project:my-dataset"` (with or without a separate `outputTable`) fail.
- Applying Row‑Level Security (RLS), Access Control Lists (ACL), or IAM policy tags against a project‑qualified dataset also fail or apply to the wrong project.
- The merge step that loads an existing BigQuery table during an ingestion update errors out if the target domain/schema produce a dataset name that includes a project prefix.

Expected behavior:
1. **Project‑qualified dataset names must be accepted** wherever a dataset is specified. The BigQuery pipeline shall extract the project and dataset correctly when the input is of the form `project:dataset` (and similarly accept `dataset` alone for backwards compatibility).
2. **When no project is given**, the pipeline shall fall back to the default project. The fallback logic should match the existing convention: use `fs.gs.project.id` from Spark/Hadoop configuration if present, otherwise use the GCP default project from the environment (via `ServiceOptions.getDefaultProjectId`).
3. **All BigQuery‑related paths must be consistent**: table creation, table lookup, ACL/RLS/policy‑tag application, and the ingestion merge path should all use the same resolution logic.
4. **The native BigQuery engine path** (`engine = NATIVE`) must exhibit the same behavior as the Spark‑based engine.

Non‑functional constraints:
- The logic must not require GCP credentials merely to parse a dataset string; any BigQuery client creation that requires real credentials (e.g., `PolicyTagManagerClient`) should remain lazily instantiated so that unit tests (which lack credentials) continue to work.
- Existing behaviour for pure dataset names (no project) must remain unchanged.

fix: allow dataset rename on bigquery job


Bug: Inconsistent jobId in audit logs and metrics across job types

Problem:
- In ingestion jobs, the `AuditLog` constructor receives `appName` (a string like `"ingest-job-20250315-143022.123"`) as the job identifier.
- In metrics jobs, the `jobId` column of the computed metrics DataFrame is seeded with `appName`.
- In assertion jobs, the `jobId` column previously used `session.sparkContext.applicationId`, which is the Spark application ID.
- This means that audit records and metrics from different job families carry different identifiers, making it impossible to reliably correlate logs across ingestion, metrics, and assertions that belong to the same overall execution.

Desired behavior:
- All audit logs and metrics DataFrames should record a consistent job identifier.
- The identifier should come from a single method on `JobBase` so that every job type uses the same value without being tempted to reach into Spark internals directly.
- The test helpers that verify audit log contents should adapt to the new consistent identifier without assuming a specific format.

Constraints / non-obvious details:
- The audit sink may need access to a SparkSession (`optionalAuditSession`). Ensure that the session is not eagerly initialized in a way that breaks reference order, particularly when the sink type is BigQuery (which does not require a session while other sinks do). Using `lazy` for that session is acceptable.
- The change must not break the existing contract of the `AuditLog` class — its first parameter stays a `String` identifier. Only the value passed changes.
- The existing `appName` field can stay as a simple diagnostic value but should not be the authority for the jobId written to log sinks. We can still generate it via timestamp if needed, but the logging path should rely on a new unified method.

Please fix so that:
1. A new method `applicationId()` exists on `JobBase`, initially returning the same string as `appName`.
2. All calls that previously wrote `appName` or `sparkContext.applicationId` as the job identifier use `applicationId()` instead.
3. `optionalAuditSession` is changed to `lazy` to avoid initialization-order pitfalls.
4. The JDBC-based audit assertions adjust accordingly so they expect the new consistent identifier.


Title: Inconsistent BigQuery project ID resolution leads to ingestion failures when dataset name lacks explicit project

## Problem
When ingesting data into BigQuery, if the `outputDataset` does not include an explicit project ID (e.g., `myproject:mydataset`), the actual GCP project used can be `null` or inconsistent across code paths, causing `NullPointerException` or BigQuery API errors.

### Observations
- **Native BigQuery jobs** resolve the project ID from `cliConfig.gcpProjectId` or fall back to `ServiceOptions.getDefaultProjectId()`.
- **Spark-based BigQuery jobs** resolve the project ID from `cliConfig.gcpProjectId` or fall back to `fs.gs.project.id` from the Spark configuration, ignoring the default GCP project ID entirely.
- **Dataset ID construction** in `BigQueryJobBase` attempts to extract the project from the parsed table ID, but when the dataset name has no project prefix, it falls back to a possibly-uninitialized `projectId` field that may have been resolved differently (or not at all) in the subclass.
- `BigQuerySparkJob.getTable` requires a `SparkSession` solely to obtain the default project ID from the Spark config, which is unnecessary.

### Expected Behavior
1. **Unified project ID resolution**: All BigQuery sink code paths should use the same resolution order:
   - `cliConfig.gcpProjectId` (from `BigQueryLoadConfig`), then
   - `ServiceOptions.getDefaultProjectId()`.
2. **Consistent dataset and table extractors**: The `extractProjectDataset` and `extractProjectDatasetAndTable` helpers should be used uniformly; no subclass should override or work around them.
3. **No unnecessary `SparkSession` dependency**: `BigQuerySparkJob.getTable` should not need a `SparkSession`—project ID resolution should happen centrally.
4. **Correct dataset ID**: `datasetId` should be derived directly from `cliConfig.outputDataset` using the same extraction logic (not via a table ID that may be missing the project component).

## Impact
- Ingestion to BigQuery fails when the dataset name does not contain a project prefix and the Spark config `fs.gs.project.id` is not set (or differs from the intended project).
- Developers adding new BigQuery functionality duplicate the resolution bug because no single place enforces the correct fallback.
- The API surface of `BigQuerySparkJob.getTable` is unnecessarily coupled to Spark.


**Ticket: bqSchema ignores attribute renames**

- **Component:** `Schema` → BigQuery schema generation
- **Affects:** `Schema.bqSchema(schemaHandler)`

**Observed behavior**
When a YAML domain schema defines attributes with a `rename`, the BigQuery schema returned by `Schema.bqSchema` uses the original attribute names rather than the renamed names. Consequently, BigQuery tables end up with the wrong column names.

**Expected behavior**
`bqSchema` should honour attribute renames: every BigQuery field name must be the renamed name (or the original name when no rename is defined), exactly as the YAML schema specifies.

**Background / hints**
- The `Schema` class already has a method `sparkTypeWithRenamedFields(schemaHandler)` that returns a Spark `StructType` reflecting renames.
- The BigQuery schema is ultimately derived from a Spark `StructType` via `BigQueryUtils.bqSchema`.
- The current implementation accidentally passes the Spark type *without* renames (`sparkType`) to that utility.

**Task**
Fix `Schema.bqSchema` so that it passes the renamed-field-aware Spark type (`sparkTypeWithRenamedFields`) to `BigQueryUtils.bqSchema`. No other behavioural change is required; the contract of `BigQueryUtils.bqSchema` and the way `SchemaHandler` loads domain schemas should remain unchanged.

**Verification criteria**
- For a schema with attributes that have `rename`, `bqSchema` must yield BigQuery field names matching the renamed names.
- Existing BigQuery schema generation for schemas without renames must still work identically.


As a maintainer, please rollback the recent support for a `kafka.properties` metadata-area properties file. That feature is unnecessary because Kafka connection settings are already available through the `server-options` section of the Kafka config.

**Required behavior change:**

1. **Remove `kafka.properties` loading** – The system must stop reading a `kafka.properties` file from the metadata area. Any code that loads and merges that file into Kafka options should be deleted.

2. **Derive Spark Kafka options from `server-options`** – For Spark DataFrame/stream `kafka` format readers and writers, the `server-options` map should be used, with the standard `kafka.` prefix applied to each entry so that Spark's Kafka connector recognizes them (e.g., `bootstrap.servers` → `kafka.bootstrap.servers`). This prefixing must be applied **only** to properties that are actual Kafka client properties. The following should **not** be prefixed (they are Spark-specific connector options): `assign`, `subscribe`, `subscribepattern`. Additionally, certain Kafka consumer/producer client properties should not be automatically forwarded to Spark with the prefix; the set of excluded properties should match what was previously ignored when loading `kafka.properties` (keys like `group.id`, `auto.offset.reset`, `key.deserializer`, `value.deserializer`, `enable.auto.commit`, `interceptor.classes`, `key.serializer`, `value.serializer`).

3. **Use `server-options` directly for native Kafka clients** – The `KafkaClient`'s `AdminClient` and any `KafkaConsumer`/`KafkaProducer` constructions that currently need a properties map should continue to receive the raw `server-options` map without the `kafka.` prefix applied.

4. **Clean up `Settings`** – After removal of the file-based loading, the `Settings` class must no longer carry a separate `kafkaProperties` field. All callers that currently use `settings.kafkaProperties` must be switched to the derived Spark-facing options obtained from the Kafka config (i.e., `settings.comet.kafka.sparkServerOptions`). The `allAccessOptions` helper on `KafkaTopicConfig` should still accept a map of base properties, but that argument will now be the derived Spark options rather than `kafkaProperties`.

**Constraints & invariants:**

- The semantics of existing Kafka ingestion jobs, sink jobs, and offset management must be preserved; they should observe effectively the same runtime Kafka properties as before (given equivalent `server-options` configuration instead of a `kafka.properties` file).
- The `comet_offsets` topic configuration must continue to work correctly, including offset retrieval via a raw `KafkaConsumer`.
- The change should not affect other configuration areas outside the Kafka feature.

rollbakc support for file kafka.properties.


The `import` command should optionally accept a comma‑ separated list of domain names via an `--include` flag. When the flag is provided, only the specified domains are loaded from the landing zone; when it is omitted, all domains are loaded (current behaviour). The command must not break when given an empty or missing `--include` argument, and it should silently ignore domain names that do not match any configured domain.


**Title:** DSV ingestion should preserve nulls as null, not coerce them to empty strings

**Description:**
When a column is missing (`null`) in a DSV row, the current ingestion pipeline converts it to an empty string (`""`) early in the processing. This empty string then flows through validation, privacy masking, and is ultimately written to the output as an empty string. The expected behavior is that a missing input value results in a true SQL `null` in the ingested datasets, particularly for optional (non-required) fields. Required columns that are null should still trigger a rejection as before.

**Context & edge cases:**
- **Nullable vs empty string:** A column whose value is absent (`null`) is semantically distinct from an explicitly empty string. The ingestion should reflect that distinction.
- **Defaults:** When a field is null and an explicit `default` is defined in the schema, the default value should be applied. If no default is defined, the field should remain `null`. Note: if the field is an empty string (not null), the default should still apply as it does today.
- **Privacy engines:** All privacy/encryption functions (MD5, SHA, Hide, etc.) should gracefully handle a null value without attempting to encrypt it – a null should remain null after privacy processing.
- **Validation logic:** The column-validation logic should treat null as “empty” for the purpose of optional/required checks, but not require it to satisfy a pattern match (a null optional field is valid; a null required field is invalid).
- **Rejection metadata:** Error-collection structures that capture the raw column value should carry `Option[String]` so that downstream consumers can distinguish between no-value and empty-string.

**Acceptance criteria (observable behavior):**
1. Ingest a DSV with missing values. In the resulting dataset (parquet/Hive), those columns should appear as `null`, not as empty strings.
2. Optional columns that are missing in the input do not produce empty-string values; they are `null` in the output.
3. Required columns that are missing still cause record rejection, no change to rejection behavior.
4. Privacy engines return `null` when given a `null` input (no exception).
5. Schema defaults are applied when the value is null (or empty) and a default is defined.
6. Existing tests continue to pass with updated expectations for null handling.

Handle null values as null not empty strings


We need a new CLI command that converts existing YAML domain/schema definitions back into Excel files, compatible with our current Excel import. Users should be able to edit schemas in a spreadsheet and re-import them without manual tweaks.

**Request**
Add a `yml2xls` command that reads domain YAML files and produces an Excel workbook with the same structure that `xls2yml` reads. The generated `.xlsx` must be directly usable as input to `xls2yml` without errors – that means column order, header rows, and cell types must match the expectations of the existing XLS reader.

**Behavior**
- Expose a `yml2xls` subcommand in the main comet CLI.
- Accept `--domain` (a comma-separated list of domain names; if absent, export all known domains).
- Accept `--xls` (output directory for the generated Excel files; one file per domain, named `<domain>.xlsx`).
- Each workbook must contain:
  - A `domain` sheet with the columns `_name`, `_path`, `_ack`, `_description`, in the exact order expected by the reader.
  - A `schemas` sheet listing every schema in the domain, with columns in the order expected by the reader (including merge-related columns, partitioning, clustering, etc.). Boolean properties (e.g., withHeader) must be written as boolean cells. Numeric properties (e.g., partition sampling) as numeric cells.
  - For each schema, an attributes sheet named after the schema, containing columns for name, rename, type, required, privacy, metric, default, script, description, position start, position end, trim in the order expected by the reader. Required must be boolean; position start/end must be numeric.

Ensure empty optional fields are represented as blank cells. The generated workbook must round-trip cleanly: reading it back with `xls2yml` should produce the original domain definition (or equivalent).

**Background**
We already have a fully working `xls2yml` path that parses Excel files and produces YAML domains. The reverse path is missing, making it hard for non-developers to fill in schemas. The Excel structure is defined by the headers currently used by `XlsReader` and the cell-type expectations (e.g., `CellType.BOOLEAN` for required, `CellType.NUMERIC` for positions). The implementation should reuse the domain and schema models already loaded by `SchemaHandler` and produce workbooks via the same Apache POI library already in the project.


```
CombinedDataSources currently reuses the same object instance for non-shared parameter data sources across all Cartesian-product combinations.

When you write:

  [Test]
  [CombinedDataSources]
  public async Task Foo(
      [Arguments(1, 2)] int x,
      [ClassDataSource<MyService>] MyService service) ...

the Cartesian expansion for the `MyService` parameter produces only one `MyService` instance, and that same instance is injected into every test case generated from the product.  This violates the `SharedType.None` contract that `[ClassDataSource]` defaults to, and it causes intermittent races during parallel test registration because multiple test cases simultaneously inject properties into a single mutable object.

Each Cartesian combination should invoke the per-parameter data-source row factory independently.  Non-shared sources must produce a fresh instance per test case; shared sources should continue to resolve to their shared instance (through their own caching).  Value-based sources such as `[Arguments]` can remain eager.
```

fix: create fresh non-shared instances per CombinedDataSources combination (#6179)


**Title:** Reduce per-test-row allocations in data-source factory construction

**Observation:**
When running large test suites that use `[Arguments]`, `[Matrix]`, or custom `DataSourceGenerator<T>` providers, profiling shows that TUnit allocates a `SelectArrayIterator` and a new `Func<Task<object?>>[]` closure for *every* data row. In projects with thousands of parameterised tests this leads to noticeable GC pressure and slower warm-up.

**Expected behaviour:**
The factory arrays should be built without per-row iterator allocations. The public API (returned `Func<Task<object?>>[]`) must remain identical — no observable change in test behaviour, just fewer allocations.

**Areas to investigate** (symptoms):
- `HandleTupleValue` currently uses `unwrapped.Select((_, index) => ...).ToArray()`, allocating a `SelectArrayIterator`.
- Inside those factory closures `UnwrapTupleAot(value)` is called again on every invocation, even though the tuple elements are stable and the already-unwrapped array (`unwrapped`) could be captured directly.
- Two similar patterns exist when processing tuple data and plain array data in the “expected parameter count” branches of `ProcessTestDataSource`.

**Request:**
Replace the LINQ `Select` + `ToArray` chains with manual loops that allocate the `Func<Task<object?>>[]` once and populate it directly. For `HandleTupleValue`, avoid the re-unwrapping call by capturing the already-available `unwrapped` array. The behaviour of the returned factories must stay exactly the same — only the allocation path changes.

**Acceptance criteria (to validate correct behaviour):**
- The number of factories returned for a given input does not change.
- Each factory returns the same value as before for any reasonable number of invocations.
- Handling of non-tuple values, single-element tuples, null, and `shouldUnwrap: false` remains consistent.

perf(core): replace Select+Func factory chain in DataSourceHelpers (#6081)


When a test is annotated with `[Timeout]` (or inherits a default test timeout), `CancellationToken` tokens obtained from `TestContext.Current.Execution.CancellationToken` **during the test body** become backed by a source that is disposed as soon as the body returns. This causes `ObjectDisposedException` in `[After(Test)]` hooks, `OnDispose` implementations, and any cleanup logic that touches `token.WaitHandle` (e.g. synchronous waits inside EF Core, Respawn, `SemaphoreSlim.Wait`, ASP.NET Core host shutdown). The failure surfaces sporadically because it only manifests when some captured copy of the timeout-scoped token is accessed after the body completes.

A prior change was made to avoid the exception by restoring `TestContext.Execution.CancellationToken` to the outer token before after-hooks, but that only fixes the context **property** — any code that captured a copy of the token into a local variable inside the body still holds a reference to the now-disposed source. The fix must address the **lifetime of the underlying source**, not just which token the property returns.

Requirements:
- Any `CancellationToken` obtained from `TestContext.Execution.CancellationToken` (or derived from it) while the test body runs must remain backed by a live (non-disposed) `CancellationTokenSource` until **after** all teardown phases have completed: `[After(Test)]` hooks, instance disposal/`OnDispose`, tracked object cleanup, and `[After(Class)]`/`[After(Assembly)]` hooks. A copy captured in the body and later touched in any after-hook must not throw `ObjectDisposedException`.
- The no-timeout fast path must incur no new allocations/overhead and must remain unchanged in behavior.
- Retry scenarios must be safe: a prior attempt's timeout-associated source must be released before a new one is allocated for the next attempt.
- Once all teardown is complete, the source must be properly disposed to avoid leaks.

fix(engine): keep per-test timeout token source alive through teardown (#6339) (#6349)


**Bug**: Keyed `[NotInParallel]` tests do not respect ordering and parallelism constraints correctly when multiple keys overlap or when the `Order` property is set.

**Expected behavior**:
- Tests annotated with `[NotInParallel("Key")]` that share the *same* key must never run concurrently.
- Tests with *disjoint* key sets should run in parallel (as much as the scheduler allows).
- The `Order` property must be honored within each key group.

**Observed issue**:
- Overlapping key sets cause excessive serialization: tests that only partially overlap keys block each other, often preventing legitimate parallel execution.
- When an `Order` is specified, keyed tests either execute out of order or miss batches altogether, leading to deadlocks or skipped tests.

**Constraints**:
- The fix must not break the existing semaphore-based concurrency control that already prevents same-key tests from running at the same time.
- The scheduling loop should be deterministic and avoid any reliance on key-set equality that is non-transitive.
- Batch execution should still wait for all started tests before removing them from the pending list, and one-time cleanup tracking must be preserved.


**Title:** Test display names should respect ArgumentDisplayFormatter, and DisplayName placeholders should resolve formatted values

**Description**
Right now, test display names (both auto-generated and those set via `[DisplayName("...")]`) do not apply registered `ArgumentDisplayFormatter<…>` implementations. When a parameterized test uses a custom argument type with a formatter, the display name still shows raw `ToString()` or generic type names. Similarly, a `[DisplayName("$paramName …")]` placeholder substitutes the argument’s raw value, not the formatted one.

We need display names to use the runtime `ArgumentDisplayFormatter<…>` that is in scope (assembly, class, or method). This affects every surface where a test name is shown: IDE test explorers, CLI output, JSON reports, and any other place that accesses the display name.

**Acceptance Criteria**

1. **Formatter-scoped argument formatting in default names**
   Given a test method with a `DataSource` supplying an argument of type `T`
   And a custom `ArgumentDisplayFormatter<T>` is registered via `[ArgumentDisplayFormatter<MyFormatter>]` at the appropriate level (assembly, class, or method)
   When the test appears in the test runner/explorer
   Then its display name (e.g. `TestMethod(arg1)`) uses the formatter’s output for the argument, not `ToString()` or the type name.

2. **Formatter-scoped interpolation in `[DisplayName]` placeholders**
   Given a test method decorated with `[DisplayName("Scenario with $paramValue")]` and a parameter `paramValue`
   And a formatter applicable to that parameter’s type
   When the test is displayed
   Then `$paramValue` is replaced by the formatter’s result.

3. **Attribute targets for `ArgumentDisplayFormatterAttribute<T>`**
   The attribute should be applicable to methods, classes, and assemblies (currently it appears limited to assembly). This lets users scope formatters to specific test classes or methods.

4. **Safe fallback for failed initializations**
   If a test fails during initialization (before arguments can be processed), the display name should fall back to the raw test method name, rather than crashing, showing `null`, or producing an empty string.

**Additional Context / Constraints**

- The display name must be determined at runtime, because `ArgumentDisplayFormatter` implementations may depend on runtime state that is unavailable at source-generation time.
- Any serialized test output (e.g. JSON model) must use the same runtime display name.
- The change must not break tests that do not use custom formatters; existing display names (both default and explicit) should remain stable for those cases.
- The `[DisplayName]` placeholder syntax uses `$parameterName` (not `{}`) as documented. Keep that syntax unchanged.

**Why this matters**
Without this, `ArgumentDisplayFormatter` is invisible in test reporting, making it useless for improving the readability of parameterized tests. Scoping it to smaller levels (method/class) also avoids forcing global formatter registrations for types used in specific test suites.


The XUnit attributes code fix currently translates `[Collection("Name")]` to `[ClassDataSource<T>(Shared = SharedType.Keyed, ...)]`, but it does not determine the correct generic type argument. Instead it leaves `T` as a placeholder or uses an incorrect type, because it only looks up the `CollectionDefinition` by name and not its `ICollectionFixture<T>` interface. Additionally, the fix ignores the `DisableParallelization` property on the collection definition.

Extend the code fix for `[Collection]` to:

- Resolve the correct T from the `CollectionDefinition` class’s `ICollectionFixture<T>` implementation. If `ICollectionFixture<T>` is present, emit the corresponding `[ClassDataSource<T>(Shared = SharedType.Keyed, Key = "...")]` using the extracted type.
- If the matching `CollectionDefinition` has `DisableParallelization = true`, also emit `[NotInParallel]` on the test class (in addition to the data-source attribute when applicable, or by itself if there is no fixture).
- Handle collection names passed as `nameof(SomeClass)` in both the `Collection` and `CollectionDefinition` attributes when matching them.
- When a collection definition has no `ICollectionFixture` interface but does set `DisableParallelization`, only `[NotInParallel]` should be applied. The collection class itself should still be marked `[System.Obsolete]`.

Ensure the fix still works correctly when a `CollectionDefinition` has both a fixture and parallelism disabled, producing both attributes simultaneously (e.g., `[NotInParallel, ClassDataSource<MyType>(Shared = SharedType.Keyed, Key = "MyCollection")]`).


**Feature: Shorten failure stack traces to user code (opt-out via `--detailed-stacktrace`)**

When a test fails, the reported stack trace includes many internal TUnit frames that are irrelevant to the user. These frames make failure output harder to read, especially for nested exceptions.

Desired behavior:
- By default, trim the stack trace of every reported exception (including inner exceptions) so that it only contains frames from the user's test code, stopping before any line that starts with `at TUnit`.
- Provide a new command-line option `--detailed-stacktrace` that restores the full stack trace (including internal TUnit frames).
- The option should follow the existing pattern for command-line providers (e.g., `--fail-fast`, `--disable-logo`). It should be a zero-argument flag that defaults to off.

Edge cases / safety:
- Trimming must not break the exception’s own stack trace accessors (e.g., `Exception.StackTrace` must return the cleaned trace, not throw or corrupt other data).
- Trimming must be applied before the exception is posted to the test framework message bus.
- If trimming fails for any reason, silently fall back to the original full trace (do not crash the test run).
- The behavior should apply to all failure paths (test timeout, assertion failure, unexpected exception, etc.).

Shorten Stack Traces to user defined code for easier reading (#1942)


**Title:** Fix one-time cleanup ordering and keyed-not-in-parallel synchronization

**Description:**
- One-time cleanup for a test class currently executes as soon as the last test of that class is *scheduled*, but it must not run until the test has actually finished.  The cleanup should be chained as a continuation on the test’s execution task, not invoked immediately.
- The check in the keyed-not-in-parallel test scheduler that prevents concurrent tests with overlapping constraint keys is not thread-safe.  Missing synchronization allows multiple tests sharing the same key to execute simultaneously.
- Both issues lead to observable race conditions during test runs.

**Expected behavior:**
- Class cleanup tasks do not start until the final test of the class has completed.
- Only one test per not-in-parallel key group runs at any time.

**Acceptance criteria:**
- No flaky test runs caused by early cleanup or parallel execution of constrained tests.
- Existing test suite passes reliably under load.


```
Performance: Test scheduler spends O(n²) time scanning all waiting tests on each key release

When many tests are waiting on constraint keys, the current scheduler drains the entire queue of waiting tests every time a running test completes and releases its keys. This leads to O(n²) total overhead for n constrained tests, making test execution unnecessarily slow.

We need the scheduler to only examine tests that are actually waiting on the keys being released. That is, after a test finishes and releases a set of constraint keys, only those waiting tests that depend on at least one of those released keys should be considered as candidates for unblocking. The full waiting set should not be scanned.

Acceptance criteria / invariants:
- Correctness: The set of tests that actually start after a key release must be exactly the same as before – no tests are lost, no deadlocks introduced, no spurious unblocking.
- Priority order must be preserved when multiple waiting tests become unblocked simultaneously (tests with lower priority value should be given preference to start).
- The fix must work correctly under the existing locking scheme (a single lock object is used for all shared state). The new data structures must be safe to access only under that lock.
- Logging for unblocking events should remain unchanged in content, only reflecting the test identity and constraint keys.
- No change in public API, no breaking behavioral change for test authors.
```

perf: index-based constraint key scheduler for O(k) lookups (#4678)


## Title
**Flow `TestContext` into Playwright teardown hooks so worker‑pooling is reliable**

## Context
`WorkerAwareTest` and `BrowserTest` are base classes used by consumers of the `TUnit.Playwright` library. Both define `[After(HookType.Test)]` teardown hooks that need to decide whether the just‑completed test passed (or was skipped) or failed.

Currently they read the test result from `TestContext.Current`.
That snapshot may already be null, may belong to a different test, or may not yet reflect the final outcome when the hook runs, leading to incorrect decisions about resetting vs. discarding worker services (and therefore pooling bugs—workers being reused after a failure, or being disposed after a pass).

TUnit hooks support receiving the correct `TestContext` as a parameter. We should switch to that mechanism in both teardown hooks and in the helper they call.

## Desired behaviour
1. `WorkerAwareTest.WorkerTeardown` and `BrowserTest.BrowserTearDown` **receive** a `TestContext` parameter from the framework.
2. Both hooks pass that instance into the existing `TestOk` method, which must accept a `TestContext` and use it instead of `TestContext.Current`.
3. The teardown logic itself does **not** change:
   – If the test passed or was skipped → services are reset and the worker is pushed back onto the pool.
   – Otherwise → services are disposed and the worker is abandoned.
4. While touching this code we should clean up the minor accidental visibility and usability issues that are already present:
   - Remove the unused `IPlaywright` parameter from `BrowserService.Register` (its only caller no longer needs it).
   - Make the nested `Worker` class `private` (it is currently `internal` despite being an implementation detail).
   - Mark its fields `readonly` and use `[]` for the collection initializers (bringing the code in line with our current code‑style guidelines).

## Steps for the implementer
- Find the hook signatures in `WorkerAwareTest` and `BrowserTest` and change them to accept `TestContext`.
- Adjust `TestOk` accordingly.
- Clean up `BrowserService.Register`.
- Apply the accessibility/readability fixes described above.
- No new tests need to be written—existing Playwright tests must keep passing.


**Acceptance criteria: Guarantee that hook execution order respects the class hierarchy, not just the `Order` property**

### Problem
When a test class has base classes that also define hooks (`[Before(Test)]`, `[After(Test)]`, `[Before(Class)]`, `[After(Class)]`), the current implementation orders all hooks across the entire hierarchy solely by the `Order` value. This means a derived class’s “Before” hook with a low (even negative) `Order` can fire before a base class’s “Before” hook with a higher `Order`. The same flattening happens for “After” hooks – a base class’s After hook with a low `Order` may run before a derived class’s After hook with a higher `Order`.

Expected standard behavior for inherited test classes is:
- **Before hooks:** base class hooks run first, then derived class hooks (base-to-derived / bottom-up).
- **After hooks:** derived class hooks run first, then base class hooks (derived-to-base / top-down).

Within a single class level, hooks should still be ordered by their `Order` property (ascending). Currently, the `Order` property dominates the hierarchy, which is incorrect.

### Desired behavior
1. For **every** test-level hook (`Before(Test)` / `After(Test)`) and **class-level** hook (`Before(Class)` / `After(Class)`), the execution order must respect the class hierarchy as described.
2. Hooks defined in base classes must execute **before** hooks defined in derived classes for “Before” hooks.
3. Hooks defined in derived classes must execute **before** hooks defined in base classes for “After” hooks.
4. Within the same class (at the same inheritance level), hooks are ordered by their `Order` value as before.
5. The fix must work correctly for arbitrary depths of inheritance, not just a single base class.
6. The `Order` property remains a first-class ordering mechanism for hooks *within* each type, but it must not cause hooks to cross class boundaries in the wrong direction.

### Non-goals
- This request does not ask for changes to the semantics of `Order` itself; only the interaction between `Order` and class hierarchy.
- No changes to method-level attributes or the way hooks are registered; only the order in which they are collected and invoked.

### Context
The TUnit documentation and design imply that hooks in base classes should serve as general setup/teardown that runs before/after more specific hooks in derived classes. This is the typical expectation from xUnit, NUnit, etc. The current global sort by `Order` breaks that and leads to surprising execution order when users combine inheritance with hook ordering.

Please fix the hook collection logic so that the lifecycle contract described above is honored for all hook types.

Guarantee before hooks execute bottom-up and after hooks execute top-down (#2742)


The xUnit-to-TUnit migration code fixer does not correctly handle `Xunit.Assert.All(...)` when the assertion lambda contains more than one inner assertion. The current conversion produces invalid TUnit code (e.g., an expression-based rewrite like `Assert.That(collection).All().Satisfy(...)` that does not exist), and the inner assertions inside the lambda are not properly converted into TUnit’s fluent assertion model.

**Expected behavior:**
When the code fixer encounters `Xunit.Assert.All(collection, element => { /* inner assertions */ })`:
1. It should replace the entire statement with an `Assert.Multiple()` block containing a `foreach` loop over the collection, using the same iteration variable name as the lambda parameter.
2. Every inner xUnit assertion (`NotNull`, `Null`, `True`, `False`, `Equal`, `NotEqual`, etc.) inside the lambda body must be converted to the corresponding TUnit fluent assertion (`Assert.That(...).IsNotNull()`, etc.), and each must be `await`ed.
3. If the containing method is not already async and returns `void`, the fixer should change its signature to `async Task` so that the `await` calls compile.
4. The fixer must work correctly when `Xunit.Assert.All` is the entire expression statement (i.e., it is not used as a sub-expression).

**Impact / Reproduction:**
Given a test method like:
```csharp
public void MyTest()
{
    var users = new[] { new User { Name = "Alice", Age = 25 }, new User { Name = "Bob", Age = 30 } };
    Xunit.Assert.All(users, user =>
    {
        Xunit.Assert.NotNull(user.Name);
        Xunit.Assert.True(user.Age > 18);
    });
}
```
Running the code fixer on `Xunit.Assert.All` currently either produces non-compilable code or ignores the inner assertions. The expected output should be something like:
```csharp
using (Assert.Multiple())
{
    foreach (var user in users)
    {
        await Assert.That(user.Name).IsNotNull();
        await Assert.That(user.Age > 18).IsTrue();
    }
}
```
and the method signature updated to `public async Task MyTest()`.

The fix should also keep other statements inside the lambda (non-assertion code) intact, only converting xUnit assertion invocations.

Note: The `IsInsideSatisfyLambda` handling in `GetNewExpression` is related to `Satisfy`/`Satisfies` patterns; this change should not break existing conversions for `.Satisfy()` lambdas, but should correctly handle the `All` case by emitting statement-level code rather than a single expression replacement.

feat(tests): fix code fixer for xUnit migration code fixer .All().Satisfy() (#3786)


```
Add support for a TUNIT_DISABLE_GITHUB_REPORTER environment variable.

The existing generic DISABLE_GITHUB_REPORTER variable can conflict with other tools. To avoid this, introduce a namespaced TUNIT_DISABLE_GITHUB_REPORTER variable that disables the GitHub reporter when set to any non-null value. Keep the old variable working for backward compatibility: if either variable is set, the reporter must be disabled.

Also, update the internal pipeline test modules that currently set DISABLE_GITHUB_REPORTER to use the new TUNIT_DISABLE_GITHUB_REPORTER variable instead.
```

Add support for TUNIT_DISABLE_GITHUB_REPORTER environment variable (#2716)


We need to make skipped tests more informative by attaching a reason to `SkipAttribute` and surfacing that reason through the test infrastructure.

**Request**
Change `SkipAttribute` so that it requires a mandatory reason string, e.g. `[Skip("Not implemented yet")]`. The reason should be exposed as a property (for example, `Reason`).

The test metadata (`Test` record) should capture this reason when a `SkipAttribute` is present on a test method or its containing class. The `IsSkipped` determination should then be based on whether a non-empty reason was captured.

When a test is skipped, the test adapter (`AsyncTestExecutor`) should include the reason in the recorded `TestResult` (for instance, in the `ErrorMessage` field), so that tooling and logs can show *why* the test was skipped.

**Edge cases / clarifications**
- If no `SkipAttribute` exists, the test is not skipped (`IsSkipped` should be `false`).
- When both a method-level and a class-level `SkipAttribute` are applied, the method-level reason should take precedence.
- Existing usages of `[Skip]` in the repo will need to be updated to supply a reason, since the constructor will now require one.


**Title:** `Is.Not.Null` does not narrow the type from nullable to non-nullable for chained assertions

**Description:**
When using the fluent assertion syntax with a nullable type, calling `.Is.Not.Null()` should confirm that the value is not null and then allow subsequent assertions to treat the value as its non-nullable type. Currently, however, the assertion chain either fails to compile or keeps the actual type as nullable, preventing further non-nullable assertions.

For example, the following code does not compile as expected:

```csharp
string? value = "hello";
await Assert.That(value)
    .Is.Not.Null()
    .And.Is.EqualTo("hello");
```

The issue is that after `.Is.Not.Null()`, the assertion chain still carries the nullable type `string?`, so `.And.Is.EqualTo(...)` (which expects a non-nullable `string`) is rejected by the compiler. This breaks the natural flow of fluent assertions and is inconsistent with TUnit’s goal of strict type checking.

**Expected behavior:**
- Calling `.Is.Not.Null()` on a nullable value should narrow the nullability and return an assertion condition on the non-nullable type.
- After `.Is.Not.Null()`, it should be possible to chain further assertions (e.g., `.And.Is.EqualTo(...)`) that accept the non-nullable type.
- The method should be accessible both on nullable and non-nullable actual types (for completeness) and should integrate correctly with `And`/`Or` connectors.

**Impact:**
Without this narrowing, users must resort to manual null checks or separate assertions, breaking the expressive, type-safe chaining that is a primary design goal of TUnit’s assertion library.


The generated test-invocation code emitted by the source generator is unnecessarily verbose:
it emits fully qualified `global::`-prefixed type names for many TUnit framework types
(e.g., `global::TUnit.Engine.Data.TestDataContainer`, `global::TUnit.Core.NotInParallelAttribute`,
`global::TUnit.Core.TimeoutAttribute`, `global::TUnit.Core.CategoryAttribute`, etc.).
These types live in namespaces that are already imported in the generated files, so the
qualified forms are not needed.

Change the generator so that it emits only the short type name (e.g., `TestDataContainer`,
`NotInParallelAttribute`) for TUnit framework types that are in imported namespaces.
Keep `global::` qualification only for types where ambiguity could otherwise arise.

Do this consistently across all generator components that construct emitted source strings
(hook registration, argument retrieval, test-invocation writing, etc.).


Allow abstract test methods with data-driven attributes in generic base classes, and support instance method data sources that can be overridden in derived classes.

Right now, defining an abstract `DataDrivenTest` or `DataSourceDrivenTest` method on a generic base class (with `[InheritsTests]` on the derived classes) causes misleading analyzer warnings, and the generated code doesn't handle instance method data sources correctly. Specifically:

- Analyzers should **not** flag abstract test methods as missing required arguments or data attributes; the derived class provides the concrete implementation that can supply them.
- When a `MethodDataSource` or `EnumerableMethodDataSource` refers to an instance method (non‑static), the generated invocation must call the method on an **instance** of the concrete test class, not try to call it as if it were static.
- For enumerable method data sources, the class instance must be created **before** the enumeration loop, and a **fresh instance** must be created for each iteration of the loop, so that the overridden methods are called on a properly constructed instance each time.

**Acceptance behavior (derived from the test patterns in `GenericTests.cs`):**
- A generic abstract class like `GenericTests<T>` can declare:
  - `[DataDrivenTest] public abstract Task DataDrivenTest(T t);`
  - `[DataSourceDrivenTest] [MethodDataSource(nameof(GetData))] [EnumerableMethodDataSource(nameof(GetEnumerableData))] public async Task DataSourceDrivenTest(T t) { … }`
  - `public abstract T GetData();`
  - `public abstract IEnumerable<T> GetEnumerableData();`
- Derived classes override these methods with concrete implementations and, for `DataDrivenTest`, add `[Arguments(…)]` attributes.
- The analyzers (`DataDrivenTestArgumentsAnalyzer`, `TestRequiresDataAttributesAnalyzer`, `DataSourceDrivenTestArgumentsAnalyzer`) must skip abstract methods entirely.
- The source generator must emit code that:
  - Creates the class instance (via `ResettableLazy`) **before** entering the enumerable method data loop.
  - Calls instance methods using `resettableClassFactory.Value.MethodName()`.
  - Re‑creates the class instance at the end of each iteration of an enumerable method data loop (so each test run gets a fresh instance).
- No false‑positive analyzer diagnostics and the tests run to completion with the overridden data methods supplying the correct values.

Generic MethodData that can be abstract and overridden


**Title:** UID-based test filtering incorrectly includes tests when class names are substrings of each other (e.g., ABCV vs ABCVC)

**Description:**
When running or discovering tests using a UID filter (e.g., via `dotnet test --filter`, the Microsoft Testing Platform test explorer, or Visual Studio), tests from classes whose names are substrings of the intended target class are falsely included.

For example:
- Classes `ABCV` and `ABCVC` both contain test methods like `B2`.
- Filtering with a UID that includes `ABCVC.B2` currently causes tests from `ABCV.B2` (and any declared dependencies like `ABCV.A1`) to also be discovered and executed.
- Conversely, filtering for `ABCV.*` incorrectly includes tests from `ABCVC.*`.

The root cause appears to be that the filter matching logic uses a simple substring check (`string.Contains`) against the full UID for namespace, class name, and method name separately, without verifying that the class name appears as a complete word/segment within the test node UID.

**Expected behavior:**
A UID filter like `Namespace.ABCVC.B2` should only match tests belonging to class `ABCVC` in namespace `Namespace` (and its dependency closure), not also tests from `ABCV` just because "ABCV" happens to be a substring of "ABCVC".

More generally, the UID matching must:
- Treat the class name as an atomic segment bounded by `.`, `<`, or end-of-string in the UID, preventing partial substring matches across segment boundaries.
- Consider the full `Namespace.ClassName` prefix to avoid cross-namespace false matches for classes with the same short name.
- Apply similar word-boundary logic to method names so that e.g. `B2` does not accidentally match `B20`.

**Impact:**
Developers unintentionally run large numbers of unrelated tests, especially in test suites with classes named following similar patterns (e.g., `FooService` vs `FooServiceIntegration`). This wastes execution time, pollutes test results, and makes debugging failures confusing because unrelated tests appear in the filtered run.

**Suggested investigation:**
Inspect the component responsible for evaluating `TestNodeUidListFilter` against `TestMetadata` (likely the `CouldMatchUidFilter` method or equivalent in the engine). The existing conditional that calls `Contains` for namespace, class, and method names should be replaced with a check that respects the UID grammar: segments are separated by `.`, `< `, and `(` characters, and the class name must appear as a full segment preceded by the namespace (if any) and followed by `.` or `<`.

fix: prevent substring matching in UID filter causing incorrect test inclusion (#4659)