We need to extend the resource activity log so that secret revisions are surfaced as distinct timeline events.

When a resource secret is updated and a new `SecretRevisions` record is created, the `/resources/{resourceId}/action-logs` endpoint must include an entry of type `SecretRevisions.created` in the parsed results. The entry must contain at least the creator’s user object (id, username, profile first/last name). The data payload itself can remain empty until the frontend provides requirements, but the entry must be produced and correctly associated with the resource.

**Requirements**
1. **Finder** (`ResourceActionLogsFinder`): add a subquery that joins `EntitiesHistory` → `SecretRevisions` for the target resource, and union it with the existing subqueries so that action logs referencing secret revisions are fetched alongside permission histories, resource updates, secret accesses, and secret histories.
2. **Parser** (`ActionLogResultsParser`):
   - Define a new type constant `TYPE_SECRET_REVISION_CREATED = 'SecretRevisions.created'`.
   - Process any `EntitiesHistory` row with `foreign_model === 'SecretRevisions'` and CRUD 'create'. For each such entry, emit a timeline entry of the new type with an empty data array and the action log’s creator information.
3. **Hybrid handling**: The implementation must work correctly when both legacy secret-history records and new secret-revision records exist for the same action log (e.g., during a gradual migration). Both types of entries should appear independently.

The change applies to the Pro edition AuditLog plugin. No changes are required for other resource types; only `SecretRevisions` mapping is needed.

PB-46084 [Pro] SH - WP3.6.7 Update action logging system to accommodate secret revisions


**Title**: Populate and use `users.last_logged_in` column directly in API responses (Part 2)

**Context**
In Part 1 we added the `last_logged_in` column to the `users` table and a real‑time listener that updates it on login. Part 2 covers the back‑population of existing users and the query‑side changes so that all endpoints that expose a user’s last login date read it from `users.last_logged_in` instead of computing it dynamically from `action_logs`.

**Ask**

1. **Back‑fill existing users**
   Write a migration and a service that sets `users.last_logged_in` for every *active, non‑guest, non‑deleted* user whose `last_logged_in` is currently `NULL`.
   The value must be the latest `created` timestamp from the `action_logs` where `action_id` is one of the recognised login actions (AuthLogin, JwtLogin, …).
   - Do **not** overwrite rows that already have a value in `last_logged_in`.
   - If a qualifying user has no matching action logs (e.g. never logged in), leave the field `NULL`.

2. **Switch reading paths to the new column**
   - In the users index endpoint (`GET /users.json`) and any other endpoint that includes user data with last log‑in information (e.g. metadata keys index), the `last_logged_in` field must now come straight from the `users` table.
   - Remove the dynamic sub‑query that was previously computing the value from `action_logs` and aliasing it as `Users__last_logged_in`.
   - Pagination sorting on `Users.last_logged_in` must use the column directly; drop the hack that remapped the sort key.
   - The `last_logged_in` virtual property of the `User` entity is no longer needed – remove it.

3. **Clean‑up / deprecation**
   - The existing `find('lastLoggedIn')` finder is only required for the back‑fill. Mark it as deprecated and adjust its implementation to simply expose the value as `action_logs_last_logged_in` (used by the migration service). No normal data‑reading flow should invoke it.
   - Ensure the `UserFactory` sets a sensible default for `last_logged_in` on created entities so tests don’t break.

**Expected observable behaviour**
- After running the migration, all existing users (except guests, inactive, deleted, and those who never logged in) have a non‑null `last_logged_in`.
- API responses that previously showed a dynamically computed `last_logged_in` continue to show the same value, now sourced from the row.
- No additional queries to `action_logs` are executed during user listings or related resource listings.
- Sorting and filtering on `last_logged_in` works as before.

PB-43187 Retrieve user last logged data from users table instead of the log to improve application performance (Part-2)


As an administrator, I want to configure what happens to Passbolt users when the directory sync detects that they are no longer present in the LDAP directory. Instead of always permanently deleting the user, I can choose to disable (suspend) the user.

## Acceptance Criteria

### Viewing the current behaviour setting
- When I retrieve the directory sync settings (e.g. via `GET /directorysync/settings.json`):
    - If no explicit choice has been saved, the property `delete_user_behavior` MUST NOT appear in the response.
    - If I have set the behaviour to `disable`, the response MUST contain `"delete_user_behavior": "disable"`.
    - If I have set the behaviour to `delete`, the response MUST contain `"delete_user_behavior": "delete"`.

### Updating the behaviour setting
- When I update the directory sync configuration (e.g. via `PUT /directorysync/settings.json`):
    - I can optionally include the field `delete_user_behavior`.
    - The only accepted values for this field are the strings `"delete"` and `"disable"`. Any other value (including an empty string, a boolean, or any arbitrary string) MUST cause a validation error and the configuration MUST NOT be saved.
    - If I do NOT include the field in the update payload, any previously saved behaviour choice MUST be cleared. The system will revert to the default behaviour, which is to delete users. Consequently, a subsequent GET MUST NOT contain the `delete_user_behavior` property.
    - If I include the field with a valid value, that value MUST be stored and returned on subsequent GETs.

### Effect on subsequent synchronizations
- When the directory sync process determines that a user should be removed because they are no longer in LDAP:
    - If the behaviour is set to `disable` (i.e. the stored configuration contains `deleteUserBehavior` with value `disable`), the user MUST be disabled (suspended) rather than permanently deleted. The existing method `DirectoryOrgSettings::isDeleteUserBehaviorDisable()` should report `true` in this case.
    - If the behaviour is set to `delete` or is absent (default), the user MUST be permanently deleted as before. `isDeleteUserBehaviorDisable()` should report `false`.

## Non-functional / Safety Constraints
- The default behaviour remains unchanged: if the administrator never touches this setting, users will be deleted.
- The API endpoints are only accessible to authenticated administrators.
- Validating the setting against the allowed values must happen both when reading the current configuration (to avoid serving corrupted data) and when applying an update.

PB-33828 [EE] As an administrator I can define the deletion behaviour of users to suspended instead of deleted


Add a `Message-ID` header to every email enqueued by `EmailSender`.

The `lorenzo/cakephp-email-queue` plugin disables automatic `Message-ID` generation, so the header must be set explicitly in the `headers` option passed to `EmailQueueTable::enqueue`. The header value must be a globally unique identifier following `RFC 2822`, shaped as `<unique-id@domain>`.

**Behavioral requirements**
- **Uniqueness**: generate the local part by removing dashes from a `Cake\Utility\Text::uuid()` (e.g., `e8c3db0628f41ea86a03dc53d16d11a`).
- **Domain**: use `HTTP_HOST` from the environment minus any trailing `:<port>`; if that is empty, fall back to `php_uname('n')`.
- **Thread-safety / idempotency**: each call to `sendEmail` must produce a distinct `Message-ID`.
- The header must coexist with any existing headers (e.g., `Auto-Submitted`).

**Constraints / edge cases**
- The generated `Message-ID` must not be cached or reused across requests.
- The header format must be exactly `<id@domain>`, with angle brackets.

PB-25863: Fix message-id header not set in emails


**Subject:** Sharing a v4 resource whose resource type has been deleted must be forbidden

**Actor:** Any user with permission to share the resource
**Impact:** Data integrity and API consistency

**Description:**
When sharing a v4 resource, the service currently does not validate whether the associated resource type still exists or has been soft-deleted. This allows shares to proceed on resources whose type no longer exists, which can lead to inconsistent behavior in clients and further operations (e.g., metadata handling, type-specific UI).

**Acceptance Criteria:**

1. If a share request targets a resource whose `ResourceType` record is either missing (null foreign key) or marked as soft-deleted (`deleted` is not null), the system must reject the operation with a `404 Not Found` response.
2. The same check applies to the share dry-run endpoint.
3. The error message returned must clearly indicate that the failure is due to the missing/deleted resource type (e.g., “The resource types associated to this resource does not exist.”).
4. The behavior must integrate with the existing `ResourcesShareService::getResource()` logic that already rejects soft-deleted resources; the new check should be part of that resource retrieval step.

**Technical Notes (non-prescriptive):**

- The existing `ResourcesShareService::share()` and `shareDryRun()` both call `getResource()` which currently only ensures the resource itself is not deleted.
- No change to the permission model or secret updates should be needed; the error must be raised before any further processing.
- The `ResourceTypes` fixture may need to be loaded in integration tests that exercise the share and dry-run endpoints if they previously relied on defaults without explicit resource type data.¹

**Out of scope:**
- Validation on creation/edition of resources (already covered elsewhere).
- Effects on v5 resources (the service already disallows sharing of v5 resources).

---
¹ This line references test infrastructure but does not prescribe changes to tests; it is a reminder for maintainers to verify fixture availability when implementing.

PB-44181 Fix as user I should not be allowed to share a v4 resource if v4 resource type is deleted


Add a `GET /metadata/setup/settings.json` endpoint that returns whether encrypted metadata should be enabled automatically when the first admin account is activated. The response body must be a JSON object with a single boolean key `enable_encrypted_metadata_on_first_admin_account_activation`.

Authentication requirement: only authenticated admins may access this endpoint.

Determining the value:
- Read the configuration flag `PASSBOLT_PLUGINS_METADATA_ENABLE_FOR_NEW_INSTANCES` (default `true`). If it is explicitly set to `false`, the flag must always be `false`.
- If the configuration flag is `true`, evaluate the current state of the instance:
  - Query all users (including disabled and deleted).
  - If the total number of users is **exactly 1**, and that sole user is **active** and has the **admin** role, the flag is `true`.
  - In any other scenario (more than one user, or the sole user is not active or not an admin), the flag is `false`.

This allows clients to safely determine that the instance represents a fresh installation eligible for automatic metadata enablement, without false positives caused by leftover users from earlier setup attempts.

PB-44285 Add endpoint to help clients enable e2ee by default for new instances


As a maintainer, I need a concrete implementation request for a missing email notification when an administrator upgrades the Passbolt instance from Community Edition (CE) to Pro Edition.

**Title**: Email notification to all other admins when an instance is upgraded to Pro Edition

**Background**
Currently, when an administrator performs an edition upgrade via `EditionUpgradeService`, no notification is sent to other administrators. We already have a symmetrical downgrade notification (`EditionDowngradeEmailRedactor`), but the upgrade path is missing.

**Requirements**

1. **Trigger**
   On the `EditionUpgradeService::EVENT_NAME` event, a new email redactor must send a notification to **all active administrator users** except the operator who performed the upgrade.

2. **Email content**
   - **Subject**: “{Operator full name} upgraded this Passbolt instance to Pro Edition” (localized to the recipient’s locale).
   - **Body**:
     - Avatar section showing the operator and the upgrade timestamp.
     - Text: “Your Passbolt instance was upgraded from Community Edition to Pro. Pro features are now available.”
     - Text: “Subscription seats: {seats}. Valid until {expiry date in yyyy-MM-dd}.”
     - A “Log in to Passbolt” button linking to the base URL.
   All strings must use the recipient’s locale.

3. **Data passed to the template**
   - `recipient` (the email receiver)
   - `operator` (the admin who upgraded) with profile and avatar
   - `upgradedAt` (the time of upgrade, a `DateTime`)
   - `seats` (int from the subscription key)
   - `expiry` (a `DateTime` or `Date` from the key)

4. **Recipients**
   - Only **active, non-disabled** administrators.
   - **Exclude** the operator himself.
   - If there are no other admins, no email should be sent.
   - The operator must exist; otherwise skip.

5. **Event data contract**
   The event must provide:
   - `uac` (a `UserAccessControl` of the operator)
   - `subscriptionKey` (a `SubscriptionKeyDto` with `users` and `expiry` properties).
   If either is missing or invalid, the redactor must throw an `InvalidArgumentException`.

6. **Registration**
   The new redactor must be included in `EditionRedactorPool::getSubscribedRedactors()` so it listens to the event.

7. **Template**
   Create the HTML email template at `Passbolt/Edition.AD/edition_upgrade` (the constant `EditionUpgradeEmailRedactor::TEMPLATE`). It should reuse existing email modules (avatar, text, button) consistent with the downgrade email.

8. **Non‑functional**
   - Emails must not be sent to disabled users.
   - The operator’s profile must have `full_name` available (virtual field).
   - The redactor must implement `SubscribedEmailRedactorInterface` and respect the subscribed event mechanism (no settings path, so `getNotificationSettingPath()` returns null).

**Acceptance criteria** (implicitly covered by the above, but for clarity)
- When an admin upgrades to Pro, every other active admin receives exactly one email.
- The email subject and body are properly localized.
- The upgrade timestamp is captured at the moment the redactor processes the event.
- The operator does not receive the notification.
- If the operator’s user record is deleted after the event, no email is sent (graceful skip).

PB-51957 MEP - WP 3.5 Create the EditionUpgradeEmailRedactor


**Bug: `has-parent` filter returns duplicate resources when a resource is associated with multiple folder relations**

### Environment
- Passbolt CE, Folders plugin enabled
- Resources endpoint: `GET /resources.json?filter[has-parent]=<folder_id>`

### Steps to Reproduce
1. Create a folder and grant Read access to a group that contains multiple users.
2. Add two resources to that folder.
3. As a group member, request resources with the folder parent filter:
   ```
   GET /resources.json?filter[has-parent]=<folder_id>
   ```

### Expected Behavior
The response should contain each resource exactly once, matching the number of resources actually located in the folder. In the scenario above, the response should contain exactly 2 entries.

### Observed Behavior
The response contains duplicate resource entries. In some cases the same resource appears multiple times in the array, inflating the result count beyond the true number of folder children.

### Impact
- Duplicate data breaks client-side pagination and directory listings.
- The UI may display the same resource multiple times.
- Calling code that relies on accurate resource counts (e.g., folder children count) will be incorrect.

### Cause Hypothesis
The filter internally joins the `folders_relations` table directly, leading to a row per matching relation (`foreign_model=resource` and `folder_parent_id`). When the authenticated user has multiple folder-relation records for the same resource (e.g., one per group member that grants access), the join produces multiple rows for that resource, and the resources are not deduplicated.

### Notes
- The filter should ensure that each resource appears only once per response, regardless of how many `folders_relations` entries match the criteria.
- The root case (`filter[has-parent]=null` or `filter[has-parent]=folder_id` with `includeRoot=true`) likely shares the same problem and should be addressed together.

PB-35980 Fix has-parent filter returning duplicate resources (GITHUB #523)


The `SmtpSettingsGetService::getSettings()` method returns a `source` field indicating where the SMTP settings originated. When settings come from the database, the source is `db`. However, when settings are not in the database, the service currently always reports `source` as `file`, even when the configuration is actually provided through environment variables (e.g., in Docker or cloud deployments). This prevents consumers of the API from accurately determining whether the configuration is file-based or environment-based.

Update the SMTP settings retrieval to correctly detect the source when settings are *not* in the database:

* If the Passbolt application config file (normally `config/passbolt.php`) exists, is readable, and explicitly defines *both* an `EmailTransport` and an `Email` key, then `source` must be `'file'`.
* If the config file is missing, not readable, or does not contain **both** of those keys (meaning the SMTP transport and sender email are coming from the default transport configuration, typically driven by environment variables), then `source` must be `'env'`.

No behavior should change when settings are already found in the database (source remains `'db'`). The detection must handle a missing or unreadable config file gracefully, defaulting to the environment source without raising errors. For testability, the constructor should accept an optional explicit path to the config file, with the default being the standard application config path.

PB-18646 Adds detection of SMTP settings in config/passbolt.php


**Title**: `./bin/cake passbolt healthcheck --application` incorrectly reports registration status when the Self Registration plugin is controlling registration

**Observed problem**
After enabling the Self Registration plugin and configuring the “Email domain safe list” provider, the application healthcheck still outputs:

```
[WARN] Registration is open to everyone.
```

That message is based solely on the old `passbolt.registration.public` boolean. It does not reflect the actual registration state and may convince administrators that the instance is wide‑open when, in fact, registration is restricted to a specific set of domains. Conversely, if the plugin is disabled but the old key is still `true`, the check may still claim registration is open even though no one can self‑register.

**Expected behaviour**
The healthcheck should correctly describe the self‑registration state by:

1. **Reporting whether the Self Registration plugin is enabled.**
2. **If the plugin is enabled and a provider is configured, showing the human‑readable provider name** (e.g. “Email domain safe list”) instead of a generic “open to everyone” message.
3. **If the plugin is disabled or no provider is configured, indicating that registration is closed** (only administrators can add users).
4. **Flagging the deprecated `passbolt.registration.public` configuration key** when it is still present, with a suggestion to remove it, because it is no longer honoured and can confuse operators.

In short, the healthcheck must become aware of the Self Registration plugin’s actual settings so that the reported registration status always matches the effective access control.

**Impact**
- Administrators relying on the healthcheck to audit their instance’s exposure get a false sense of security or unnecessary alarms, depending on their setup.
- The deprecated configuration key lingers in `config/passbolt.php` without any notice, leading to confusion during upgrades.

PB-19555 Adds self registration healthcheck


## Feature Request: Support Direct Subscription Key Input via Command Line (Not Just File Path)

**Title:** `subscription_import` command should accept subscription key text directly, not only a file path

**Severity:** Enhancement / Usability blocker for Docker deployments

**Component:** `passbolt subscription_import` command (`SubscriptionImportCommand`)

---

### Problem Statement

The `passbolt subscription_import` command currently only accepts a `--file` option pointing to a subscription key file on disk. In containerized (Docker) environments, requiring a file means operators must first mount or copy the subscription key into the container before running the import — adding unnecessary orchestration steps. There is no way to pass the subscription key content directly (e.g., from an environment variable or a secret injected at runtime).

### Expected Behavior

The command should accept the subscription key text (the Base64-encoded key data) directly via a new option, allowing operators to import a subscription without first placing it on the filesystem. Specifically:

1. A new `--text` / `-t` option should accept the subscription key as a string argument.
2. When `--text` is provided with valid subscription data, the import should succeed identically to the `--file` path.
3. When `--text` is provided but the value is not a valid subscription key string, the command should fail with a clear error message.
4. When `--text` is empty or missing, `--file` should be used as the fallback, preserving backward compatibility.
5. The success message should not hardcode the file path, since the source might not be a file.

### Impact

- Docker users and automation scripts can import subscription keys without staging files on disk.
- Reduces the deployment complexity for containerized Passbolt EE instances.
- No breaking change to existing `--file` workflows.

PB-44268 As a docker user I want to be able to load the subscription key directly and not use files


```
Feature: Logout all users on edition downgrade

When a Passbolt instance is downgraded from PRO to CE, every active JWT
refresh token must be invalidated. This ensures that users can no longer
use refresh tokens issued under the PRO licence, effectively forcing
them to re-authenticate.

Acceptance criteria:

1. After the `EditionDowngradeService::EVENT_EDITION_DOWNGRADED` event
   fires, no AuthenticationToken of type “refresh_token” remains active
   (`active = true`).

2. Tokens of other types (e.g., “login”) and refresh tokens that are
   already inactive are not modified.

3. The downgrade must not fail (no HTTP 500) if the token-update
   operation encounters an error. Instead, the error must be logged,
   and in debug mode the full exception message must be included in the
   log entry.

4. The behaviour is automatically registered by the JWT authentication
   plugin; no manual wiring is required.
```

PB-52072 MEP - WP 3.7 Create the JwtAuthenticationLogoutAllUsersOnEditionDowngradeListener


We need a way for administrators to delete the secret revisions organization settings so that the plugin falls back to its default behavior without having to manually manipulate the database. Currently there is no dedicated endpoint for this – settings can be created or updated via `POST`/`PUT` and read via `GET`, but never removed.

**Expected behavior**
A new `DELETE` endpoint at `/secret-revisions/settings.json` (following the existing route convention) that:
- is accessible only to authenticated administrators,
- requires a JSON request (otherwise an appropriate error is returned),
- removes the organization setting identified by the same property constant used by the `GET` settings service (`SecretRevisionsSettingsGetService::ORG_SETTING_PROPERTY`), and
- returns a success response even if no such settings exist (idempotent operation).

**Impact**
Administrators who want to revert secret revisions behavior to the plugin’s built-in defaults currently have no supported removal path. This missing functionality blocks operational hygiene and prevents automated reset in testing or recovery scenarios.

**Acceptance criteria**
1. `DELETE /secret-revisions/settings.json` is recognised and routed to the appropriate handler.
2. Unauthenticated requests receive an authentication error.
3. Non-admin authenticated users receive a forbidden error.
4. Requests not made with a JSON content type (or without the `.json` extension) receive a “not JSON” error.
5. When the settings exist, they are permanently deleted from the `organization_settings` table.
6. When the settings do not exist, the operation still returns a success response.
7. The response body adheres to the standard Passbolt success envelope.

**Notes**
The existing `GET` and `POST`/`PUT` endpoints are already in place; this request simply adds the missing `DELETE` verb to the same resource.

PB-46108 SH - WP4.3 Implement DELETE /secret-revisions/settings.json endpoint


**Feature Request: Optional SMTP Client Identifier**

**As an** administrator configuring outgoing email,
**I want** to supply an optional “Client” identifier (domain name or IP address) to the SMTP server,
**so that** the server’s HELO/EHLO greeting uses a custom client identity instead of the default.

---

### Acceptance Criteria

1. **New optional field**
   - The SMTP configuration form in the web installer and in the administration panel contains a new text field labeled **Client**.
   - The field is optional; submitting without it (or with an empty string) is accepted.

2. **Input validation**
   - When a value is provided, it must be either:
     - A valid domain name (e.g., `passbolt.com`) **or**
     - A valid IP address (IPv4 or IPv6).
   - Any other value must be rejected with the error message:
     *“The client should be a valid IP or a valid domain.”*
   - Non‑string values (e.g., integer `0`, arrays, `false`) are also rejected with the same message.
   - The empty string, `null`, or the complete absence of the field are treated as `null` and cause no validation error.

3. **Persistence**
   - The `client` value is stored as part of the SMTP settings (encrypted in the database, or in a file) alongside the existing fields (`host`, `port`, `tls`, `username`, `password`).
   - When `client` is `null`, it is stored as `null`; when it is a valid string, it is stored unchanged.

4. **Usage during test email**
   - When an administrator sends a test email, the debug SMTP transport configuration includes the `client` value.
   - If a `client` was provided, it is passed to the transport; if not, the transport receives `null`.

5. **Usage during normal email sending**
   - The production SMTP transport reads the `client` setting from the saved configuration.
   - When connecting to the SMTP server, the transport supplies the `client` value (if any) as the client identity.

6. **Backward compatibility**
   - Existing configurations that have no `client` key continue to work without errors, defaulting to `null`.
   - The list of allowed SMTP setting fields includes `client` so that it is accepted for storage and retrieval.

PB-21682 Adds the optional client settings to SMTP settings


As an admin running the passbolt cleanup command, I should be able to delete duplicate entries in the `resources_tags` table so that the data remains consistent and no orphaned or redundant tag associations persist.

**Acceptance criteria**

1. **Identifying duplicates**
   Two rows in `resources_tags` are considered duplicates when they share the same `resource_id`, `tag_id`, and `user_id` combination.

2. **Cleanup behaviour**
   When the cleanup is executed, all but one occurrence of each duplicate group must be removed. The specific surviving row is not prescribed, but the cleanup must guarantee that each unique combination of `resource_id`, `tag_id`, and `user_id` appears exactly once afterward.

3. **Idempotency**
   Running the cleanup multiple times must produce no further changes after the first run, as no duplicates remain.

4. **Dry‑run mode**
   The cleanup must support a dry‑run mode that reports the number of duplicate rows that *would* be deleted without actually modifying the database.

5. **Integration with existing cleanup command**
   The new de‑duplication step must be registered as a named cleanup task within the `Passbolt/Tags.ResourcesTags` model, so it appears automatically when an admin invokes the global cleanup command (e.g., `bin/cake passbolt cleanup`). The task name should clearly indicate it handles “Duplicated ResourcesTags”.

6. **Safety constraints**
   - The cleanup must never alter rows that are not duplicates.
   - It must leave all columns of the retained row unchanged.
   - No foreign key or other integrity constraints should be violated by the removal.
   - The operation must work correctly on large tables and not cause timeouts under normal circumstances.

**Out of scope**
- Preventing future duplicates at the application level (this is a data maintenance task, not an enforcement change).
- Cleaning up other tables or tag-related entities (only `resources_tags` duplicates are addressed).

PB-34080 As an admin running the passbolt cleanup, I should delete duplicate resources_tags entries


When the organization account recovery policy is changed from a non-disabled state (e.g., opt-out) to **mandatory**, all user-level account recovery settings with status `rejected` must be deleted.

Under a mandatory policy, rejection is not a valid choice – those stale entries should not persist. The cleanup should happen as part of the policy update operation, immediately after the policy transition is persisted but before any post-update events are dispatched.

PB-15271 Fix when account recovery org policy is set from !disabled to mandatory it should reset rejected account recovery user settings entries


**Title:** MEP WP 3.1 – Persist detected edition into organization settings

**Background**
Currently, the application detects whether it is CE or PRO by checking subscription key validity on the fly (via `SubscriptionKeyGetService`). Several components need a stable, persisted representation of the edition, without repeatedly re‑evaluating the subscription state or risking detection inconsistencies.

**Request**
Create a migration and a service that, during both new installations and upgrades, populate an immutable `EditionOrganization` setting in the database.

**Behavioral requirements**
1. **Detection logic:**
   - If a valid subscription key can be retrieved (file or database), the edition is `PRO`.
   - Otherwise the edition is `CE`.
   - Use the existing `SubscriptionKeyGetService::get()` to perform the check.
2. **Persistence constraints:**
   - The setting must only be inserted if no `EditionOrganization` row exists yet.
   - The row must be created with system identifiers: `created_by` and `modified_by` set to the zero UUID (`00000000-…`), because it is not user‑triggered.
3. **Invocation points:**
   - **Migration:** A v5.13.0 migration (`V5130PopulateEdition`) must invoke the population logic. The migration must **not** fail if an error occurs; log the error and continue.
   - **Installation:** The web installer must ensure the edition row is inserted during installation, so that the finished instance has exactly three organization settings: SMTP settings, subscription, and the new edition row.
4. **Immutability:**
   - An existing edition value must never be overwritten. The service must bail out early if the row already exists.

**Acceptance criteria (observable behavior)**
- After a fresh installation, querying the `EditionOrganizationSetting` table returns the correct edition (`CE` or `PRO`) based on whether a valid subscription key was provided.
- After upgrading an existing instance, the edition is populated exactly once; subsequent upgrades leave it unchanged.
- The migration never breaks an upgrade, even if the database connection or subscription key service is temporarily unavailable.
- All edition-dependent components (e.g. `EditionManager`) can rely on the persisted value as the single source of truth, rather than re-evaluating the subscription state.

**Repository scope**
- New service: `Passbolt/Edition/Service/PopulateEditionService` (under `plugins/PassboltCe/Edition`).
- New migration: `config/Migrations/20260528120318_V5130PopulateEdition.php`.
- The web installer expectations (`InstallationControllerTest`) must reflect the additional row.

PB-51953 MEP - WP 3.1 Create the V5130PopulateEdition migration


As an administrator, I need an optional security hardening to prevent attackers from discovering valid user emails through public endpoints.

## Acceptance Criteria

### Feature Flag
- A new configuration flag `passbolt.security.preventEmailEnumeration`, defaulting to `false`.
- Toggleable via the environment variable `PASSBOLT_SECURITY_PREVENT_EMAIL_ENUMERATION`.

### Account Recovery Endpoint (`POST /users/recover.json`)
- **When the flag is `false`** (default): existing behavior is unchanged — the endpoint returns an error response if the submitted email does not match an active user.
- **When the flag is `true`**: the endpoint must not reveal whether a given email exists. For any email (including non-existent or deleted users), the response is a `200 OK` with a generic success message (`"Recovery process started, check your email."`). No recovery email is actually sent when the user does not exist; the system simply replies as if it did.

### Public Registration Endpoint (`POST /users/register.json`)
- **When the flag is `false`** (default): existing self‑registration behavior is unchanged.
- **When the flag is `true`**: the entire registration endpoint must be disabled (always return a `404 Not Found`) with an appropriate message indicating that registration is unavailable to prevent email enumeration.

### Constraints & Safety
- Changes must be limited to the public-facing behavior of `/users/recover` and `/users/register` endpoints only.
- Internal operations (e.g., actual recovery-token generation for valid users) must not be altered.
- The configuration must be clearly documented in `config/default.php` to explain that:
  - It prevents information leakage via these endpoints.
  - It also disables legitimate user feedback (e.g., notifying a real user that their account may be disabled) and breaks open registration workflows by design.
- All existing functionality and test coverage must continue to pass when the flag is off.

PB-25030 Add feature flag to prevent user email enumeration


**Bug / Feature Request**

**Title:** PB-47893 – Email digest sender command does not log sending results and remaining queue counts

**Description**

As an administrator running `bin/cron passbolt email_digest send` (the email digest sender command), I cannot see in the logs how many emails were sent, how many failed, or how many emails are still waiting in the queue after each execution. The command completes silently with no log output about its outcomes, making it impossible to monitor the health of the email queue or to build automated alerts.

**Expected behaviour**

After the command processes a batch of emails from the queue, it should write a structured log entry (at info level) containing at minimum:

- `sent` – the number of emails successfully sent in this batch run.
- `failed` – the number of emails that failed to send in this batch run (e.g., due to socket errors).
- `pending` – the number of emails still waiting in the queue that are ready to be sent (not locked and not past retry limits).
- `locked` – the number of emails that are currently locked in the queue (e.g., because of previous send failures or in-progress sends).

The log entry should be machine-parseable (e.g., a JSON string) so that monitoring tools can easily extract the counts. This logging must not affect transactional integrity: if the batch fails part-way, the logged counts should reflect what actually happened (sent vs. failed) and the queue state after the attempt.

**Current behaviour**

`SenderCommand::execute()` calls `SendEmailBatchService::sendNextEmailsBatch()` but does not collect any result. No information about the send attempt or remaining queue state is logged. Only potential exceptions are propagated; the command exits silently on success.

**Impact**

Administrators and automation have no visibility into email queue processing. Without this information, it is impossible to answer basic operational questions:

- “Did the cron job send anything?”
- “How many emails are still queued?”
- “Are there stuck/locked emails that need manual attention?”

Adding this logging would allow administrators to monitor delivery progress and detect anomalies early.

**Additional context / constraints**

- Emails that are in a locked state (locked column = true) must be reported separately from ordinary pending emails. Locked emails typically indicate messages that are being processed or have encountered repeated errors and are no longer eligible for automatic retry.
- The “pending” count should consider only emails that have `sent = false`, `locked = false`, `send_tries ≤ 3`, and `send_at ≤ now`. This ensures the count reflects genuinely retryable messages.
- The log entry should be emitted even if no emails were sent (e.g., an empty batch), to maintain consistent log output across cron runs.
- The logging should not interfere with existing locking/unlocking logic or error handling in the service.

PB-47893 As an admin running the bin/cron command, I can see in the logs the number of emails left to send


**Title:** Add metadata healthcheck for server’s ability to decrypt the shared private key

**Description**
The metadata feature relies on a shared private key that the server must be able to decrypt. If decryption is broken, operations like resource sharing will fail without clear warnings. Administrators need an early warning when the server can no longer decrypt its own metadata private key.

**Acceptance criteria**
1. A new health check is available under the `metadata` domain and is classified as an **error**-level check.
2. The check automatically activates when the Metadata plugin is loaded (no manual wiring required).
3. When the latest **non-deleted** server metadata private key (i.e. where `user_id` is `NULL`) exists and can be decrypted using the configured server GPG key, the check shows a success message: “The server is able to decrypt the metadata private key.”
4. When no such server metadata private key is found, the check fails with the message: “No server metadata private key found.”
   - In debug mode, the failure message includes the underlying exception details.
5. When a key exists but decryption fails (e.g., corrupted data, wrong key, passphrase mismatch), the check fails with a message starting with: “Unable to decrypt the metadata private key data.” followed by the error details.
6. Soft-deleted metadata keys (where `MetadataKeys.deleted` is not null) are ignored when selecting the private key to decrypt.
7. The decryption attempt follows the same server‑key fallback logic used by existing GPG health checks: if the key is not already in the keyring, it should be imported before decrypting.

**External constraints**
- The check must behave identically whether run via CLI (`./bin/cake passbolt healthcheck`) or the status page.
- The legacy array key for this check should be `canDecryptMetadataPrivateKey`.
- No change to the existing healthcheck infrastructure beyond adding this new check and its domain constant.

PB-35120 Add healthcheck to try to decrypt the server metadata private key entry for the shared key


As an administrator, I want to be warned when the server cannot share metadata private keys due to a missing server-side key, so that I can fix the configuration before users encounter failures.

## Context
When zero-knowledge key sharing is **disabled**, the server must have access to the active metadata key(s) in order to share metadata private keys with users. If the server lacks a private key for any active metadata key, the sharing process will break silently or fail, preventing users from accessing resources.

## Acceptance criteria
1. A new healthcheck is added to the `metadata` domain, visible in the endpoint and CLI.
2. The healthcheck evaluates whether the server has the necessary private key(s) when zero-knowledge mode is **off**.
3. **Pass** when:
   - Zero-knowledge key sharing is **enabled** (no server access required).
   - Zero-knowledge key sharing is **disabled** and **every active (non-deleted) metadata key** has an associated server-owned private key (i.e., a key where `user_id` is null).
4. **Fail** when:
   - Zero-knowledge key sharing is disabled and **at least one active metadata key** lacks such a server-owned private key.
5. The check must **ignore deleted metadata keys** – only consider keys where `deleted` is null.
6. Severity: **error** level, as this situation directly impacts data availability.
7. Provide a human-readable failure message explaining the problem (e.g., “The server does not have access to metadata key”) and a help message outlining:
   - That zero-knowledge is off and the server must hold a copy of the metadata key.
   - That without this key, the server cannot distribute metadata private keys to users.
8. The check’s behavior must remain correct regardless of the number of active metadata keys, including scenarios with multiple keys and partially missing server keys.

PB-44406 Add new healthcheck to notify administrators when zero-knowledge disabled and the server does not have access to the shared metadata key


**PB-19553: Robust self‑registration settings validation and response contract**

**Background**
The `POST /self-registration/settings.json` endpoint allows admins to configure self‑registration (currently only the `email_domains` provider). The current implementation has a few rough edges:

* When the `provider` field is missing or set to an unsupported value, the service throws a `BadRequestException` (400) with a plain error message rather than returning a structured validation error response that follows the API conventions for form validation.
* The `SelfRegistrationBaseSettingsForm` is abstract and cannot be instantiated directly, preventing it from being used as a fallback for unknown providers.
* Validation messages are generic (“This field cannot be left empty.”) and do not clearly identify the field or the nature of the problem. For domain validation, the error message does not indicate which domain failed.
* The successful save response returns the internal `OrganizationSetting` entity, exposing implementation details rather than a consumer‑friendly JSON object with consistent keys (`provider`, `data`, `id`, timestamps, modifier ids).

**Intended behavior**

1. **Provider fallback & validation uniformity**
   - When the service receives a request, it must extract the `provider` from the payload (`$data['provider'] ?? null`) and use the appropriate form for validation.
   - If the provider is `email_domains`, use `SelfRegistrationEmailDomainsSettingsForm`.
   - For any other value (including `null` or an unsupported provider), the service shall instantiate `SelfRegistrationBaseSettingsForm` (which must become concrete) and rely on its provider‑list validation to produce an appropriate “provider should be part of the supported list” error. No `BadRequestException` should be thrown before validation.

2. **Improved error messages**
   - Base form:
     - “The provider should not be empty.” (instead of “This field cannot be left empty.”)
     - “The data should not be empty.”
   - Email domains form:
     - If `allowed_domains` is missing or empty: “The list of allowed domains should not be empty.”
     - If `allowed_domains` is not an array: “The list of allowed domains should be an array of strings.”
     - For each invalid domain: “The domain #{0} should be a valid domain.” where `{0}` is the zero‑based index of the invalid entry in the list.

3. **Cleaner save response**
   - On success, `saveSettings()` must return an associative array with the following keys:
     ```
     id, provider, data, created, modified, created_by, modified_by
     ```
     where `data` is the sanitized `email_domains` configuration object, and the timestamps/creator fields come from the persisted setting.

4. **Data safety**
   - The base form’s `execute()` method shall safely default missing `provider` or `data` keys to `''` and `[]` respectively before passing to `parent::execute()`, so that a total absence of input triggers validation errors rather than PHP errors.
   - The domain validation rule must accept `mixed` input and return a clear error when the value is not an array.

**Acceptance criteria (high‑level)**
- Sending a request with no `provider` returns a 400 validation error with the message “The provider should not be empty.”
- Sending an unknown provider returns “The provider should be part of the supported list: email_domains.”
- Sending an empty `allowed_domains` returns “The list of allowed domains should not be empty.”
- Sending a non‑array `allowed_domains` yields an error that it should be an array of strings.
- Invalid domain entries (e.g., `['passbolt.com', 'notadomain1234']`) produce “The domain #1 should be a valid domain.”
- A successful save returns a JSON body containing `id`, `provider`, `data`, `created`, `modified`, `created_by`, `modified_by`.
- No `BadRequestException` is thrown for missing/invalid `provider`; all such errors are surfaced via the standard form validation error format.


**Fix command injection in SQL export filename via shell redirect**

The `sql_export` and `mysql_export` dump helpers (`mysqlDump`, `mariaDbDump`, `postgresDump`) build shell commands that append ` > $dir/$file` without escaping the filename. A caller-supplied filename containing shell metacharacters (backticks, `$()`, `;`, `|`, etc.) can trigger arbitrary command execution.

**Required change**

1. Remove the shell redirect from all dump commands. Instead construct the process using an array of arguments and pipe the output directly to a PHP file handle.
2. Use `CommandRunner` (or Symfony `Process`) to execute the dump command; the output must be streamed incrementally to the target file so that large dumps do not load entirely into memory.
3. Mysql/MariaDB password switch `-p<password>` (no space) must be handled correctly.
4. PostgreSQL `PGPASSWORD` must be passed via the process environment, not in the command line.

**Additional requirement for `CommandRunner`**

`CommandRunner::run()` must accept an optional callback that receives stdout/stderr chunks. The callback will be used by the dump commands to write output to the file descriptor as it arrives.

**Acceptance criteria**

- A filename like `dump_$(id)_\`whoami\`.sql` causes no command execution and the file is created with that literal name.
- Dumps of large databases complete without memory exhaustion.
- All three database drivers (MySQL, MariaDB, PostgreSQL) are supported.

PB-50120 Use CommandRunner to run datatabase export command for sql_export (GITHUB #593)


We need to log secret revision creation in the action logs. Add support for `SecretRevisions` in the entities history system:

- When a secret revision is created (e.g., during resource updates that change the secret), an `entity_history` record must be persisted with `foreign_model = 'SecretRevisions'` and `crud = 'create'`.
- Update the configuration in `EntitiesHistoryCreateService` so that `SecretRevisions` appears alongside `Secrets` under the relevant action mapping (`ResourcesAdd.addWithSecrets` and any other sites that produce new secret revisions).
- Ensure `EntitiesHistory` has a `belongsTo` association to `SecretRevisions` so queries can join secret revision data.
- In the `ActionLogsModelListener`, add the same association for contexts where `modelName === 'SecretRevisions'`.
- On the `SecretRevisionsTable`, add a named `belongsTo` to `Resources` (as `SecretRevisionsResources`) to allow self-referencing conditions in deeply nested queries without ambiguity.
- Extend the `EntitiesHistoryFactory` with convenience methods (`secretRevisions()`, `withSecretRevisions()`) that set `foreign_model` and associate a secret revision fixture so tests can construct complete entities.

PB-46084 SH - WP3.6.7 Update action logging system to accommodate secret revisions


**Bug Report: SMTP Authentication Method Selection Missing in Web Installer**

**Summary**
When installing passbolt via the web installer, the email configuration page always assumes SMTP authentication uses a username and password. Admins cannot select other common authentication methods (e.g., no authentication, or username-only), which prevents completion of the installation for SMTP servers that use those methods.

**Observed Behavior**
- The web installer’s email step presents only the fields: *Host*, *TLS*, *Port*, *Username*, *Password*, *Client*, *Sender email*, and *Sender name*.
- There is no way to tell the system that the SMTP server does not require credentials, or that it requires only a username.
- Entering empty strings for username or password is not enough; the system still tries to authenticate with those empty values, leading to connection errors.
- No validation exists for an authentication method field.

**Expected Behavior**
1. The email configuration form includes a required dropdown (or radio buttons) labeled **Authentication method** with the following options:
   - `Username & password` (value: `username_and_password`) – default selection
   - `Username only` (value: `username_only`)
   - `None` (value: `none`)

2. When the method changes, the UI adapts accordingly:
   - **Username & password**: Both *Username* and *Password* fields are visible and editable.
   - **Username only**: Only the *Username* field is visible; the *Password* field is hidden and its value is cleared (set to empty string before submission).
   - **None**: Both *Username* and *Password* fields are hidden, and their values are cleared.

3. On the server side, the `EmailConfigurationForm` must:
   - Validate that `authentication_method` is present and in the allowed list.
   - During `execute()` (before validation rules run), sanitize the submitted data based on the chosen method:
     - If method is `none`: set `username` and `password` to `null` if they are empty strings.
     - If method is `username_only`: set `password` to `null` if it is empty.
     - If method is `username_and_password`: preserve both fields as-is.
   - This sanitization must happen before further processing so that empty strings are not mistakenly stored as credentials.

4. The web installer controller (`EmailController`) must use the new validation set (e.g., `validate => 'webInstaller'`) when processing the form, ensuring the authentication method field is required and constrained.

**Impact**
Administrators whose SMTP infrastructure uses no authentication (e.g., internal relays) or only a username without a password cannot complete the passbolt web installation. This leads to failed installs and unnecessary workarounds or manual configuration edits.

**Additional Notes**
- The JavaScript handling the UI visibility should be added to the existing `web_installer/email.js` file and triggered on page load and on change of the authentication method select element.
- The list of allowed methods must be defined as a constant in `EmailConfigurationForm` (`ALLOWED_AUTH_METHODS`).
- Any existing SMTP settings tests and web installer controller tests must be updated to include the new field and verify the hiding/clearing logic.

PB-25405: As AD installing passbolt through the web installer, I should be able to configure authentication method for SMTP


**Request: Isolate config permission changes during web installation to make tests safe**

As a developer running the web installer integration tests, I need those tests to stop modifying the file permissions of the application’s real `CONFIG` directory, because that can break my local development environment.

**Acceptance Criteria**

1. The permission-setting step that runs at the end of a web installation (changing the config directory to `0550` and files inside it to `0440`) must not be executed against the real `CONFIG` directory when tests run; instead, tests must be able to supply a fake implementation that operates on a disposable temporary directory.
2. The production behaviour remains unchanged: after a real installation through the installer controller, the permissions of the config directory and its contents must still be set as before (`0550` for directories, `0440` for files).
3. The permission-changing logic is extracted from the core installer utility into its own injectable service, with the target config path supplied at creation time. That service is invoked by the installer workflow.
4. The installation controller receives this service via dependency injection (not by creating it internally), so that a test can replace it with a double that points somewhere else.

**Safety Constraint**
No existing test (or future test) should ever touch the host’s real configuration files or directories. The injection mechanism must guarantee that the real permission service is not used in the test context unless explicitly opted in.

**External Visibility**
The installer controller’s `install` action and the internal `WebInstaller::install()` method no longer own the permission-changing step directly; they delegate it to the injected service. The overall installation flow – database connection, GPG key import, config file writing, user creation, temp file cleanup, permission change, settings flush – remains identical from an end-user perspective.

PB-19621 - Stop changing permissions in tests by  injecting a service in the InstallationController


**Title:** Support `Independent(Normal)` observation distributions in `GaussianHMM`

**Background**
`GaussianHMM` (in `pyro/distributions/hmm.py`) currently requires its `observation_dist` argument to be a `torch.distributions.MultivariateNormal`. In many models the observation noise is diagonal, and users naturally want to pass a `torch.distributions.Independent(Normal(loc, scale), 1)` for clarity and memory savings. Internally, the log-probability computation relies on the utility `matrix_and_mvn_to_gaussian` (in `pyro/ops/gaussian.py`) to convert the observation distribution into a `Gaussian` potential; that utility also only accepts `MultivariateNormal` today.

**Request**
1. Expand the accepted types for `observation_dist` in `GaussianHMM` to include `Independent(Normal, 1)` in addition to `MultivariateNormal`.
2. Update `matrix_and_mvn_to_gaussian` so that when it receives an `Independent(Normal)` it produces an equivalent `Gaussian` but avoids materializing the full dense precision or covariance matrix – the implementation must remain memory-efficient for diagonal noise.
3. The log-probability of a `GaussianHMM` with diagonal observations must be correct and numerically match the equivalent model constructed with a `MultivariateNormal` that has the same diagonal covariance.

**Constraints & edge cases**
- Only `Independent(Normal, 1)` (i.e., a single event dimension wrapping a `Normal` base distribution) needs to be supported as the new type. Other independent wrappers or non-normal bases are out of scope.
- The existing support for `MultivariateNormal` must remain unchanged.
- No behaviour change for the transition distribution (`transition_dist`), which should continue to require `MultivariateNormal`.
- The conversion helper `matrix_and_mvn_to_gaussian` should raise a clear error for unsupported distribution types, as it does today.

**Impact**
This will allow models with diagonal observation noise to be expressed directly and run without the overhead of full-covariance linear algebra, while preserving backward compatibility.

Support Independent(Normal) observations in GaussianHMM (#2005)


We need to correct the gradient estimates for `Trace_ELBO` when models use plates (`iarange`) and/or subsampling. Currently, `Trace_ELBO` uses a global log-ratio term in the score-function estimator, which is biased when plate dimensions are present. `TraceGraph_ELBO` already implements a partial Rao-Blackwellization that computes a site-specific log-ratio by summing over irrelevant plate dimensions; `Trace_ELBO` should adopt an analogous per-site `log_r`.

Additionally, the relevant test (`test_subsample_gradient_parallelized`) is currently skipped for `Trace_ELBO` with a TODO; it should be included and pass. Any infrastructure needed for this (e.g., making `iarange_info` available regardless of trace graph type, and removing the now-unused `MultiViewTensor`) should be done as part of this change.

add partial rao-blackwellization to trace_elbo (#865)


**Summary**
`SMCFilter` memory usage grows quadratically with the number of time steps, making it impractical for long sequences. The filter retains every historical latent variable, even for Markov models where only the current state is needed.

**Steps to reproduce**
1. Use `SMCFilter` with a state-space model that defines many per-timestep latent variables.
2. Run the filter for several hundred steps (e.g., 500).
3. Observe memory consumption via `get_values_and_log_weights()` or `get_empirical()`.

**Observed behavior**
The internal `_values` dictionary accumulates entries for every latent site at every time step (`z_1`, `z_2`, …), causing memory use to scale with `O(num_particles × total number of latent variables)`. For Markov models, this is unnecessary: the filter only needs the *current* state, not the full history.

**Expected behavior**
The user should be able to supply a state dictionary that is passed to `model.init()` and `model.step()` (and the corresponding guide methods). The dictionary should:

- Store only the tensors that are needed across time steps (e.g., the current latent state).
- Have its values resampled in-place by the filter after each step, so that the number of stored tensors stays constant (proportional to the state size, not to the number of time steps).
- Prevent the guide from writing to it (enforce read-only access), since the guide’s role is only to propose new values.

Specifically, the model and guide interfaces would change from
`model.init(*args)` / `model.step(*args)`
to
`model.init(state, ...)` / `model.step(state, ...)`, where `state` behaves like a dict of `torch.Tensor`s. Each tensor must have a leading dimension equal to the number of particles, and the filter should raise a clear error if a guide attempts to assign to the state or if a tensor shape is invalid.

`get_empirical()` should return empirical distributions for the *current* keys in the state dict, reflecting the filtered beliefs.

**Impact**
Without this, long sequences are infeasible. With a fixed-size state interface, memory usage becomes `O(num_particles × state_size)`, independent of the sequence length, enabling practical filtering on long time series.


Add a baseball batting example using NUTS and inference utilities. The example must:

- Implement three hierarchical models (complete pooling, no pooling, partial pooling) for the Efron–Morris dataset.
- Use `MCMC` with `NUTS` for inference.
- Use `TracePredictive` to generate posterior predictive samples and compute log predictive density.
- Output summary statistics for latent parameters and predictions.
- Be executable as a standalone script under `examples/`.
- Integrate into the existing example test suite so it runs with modest sample/warmup counts.

Expose a `get_samples_and_weights` method on `Empirical` that returns the finalized samples and log weights, enabling custom posterior summary computations.

Add `pandas` to extras_require since the example needs it for data handling.

Adding baseball hits example using NUTS and Inference utils (#1080)


**Title:** OED `EwmaLog` fails on GPU due to device mismatch and incorrect NaN/inf handling

**Description**
When using Optimal Experimental Design estimators (e.g., `donsker_varadhan_eig`) on a GPU, computations fail with device-mismatch errors. The `EwmaLog` class in `pyro.contrib.oed.eig` initialises its internal state `self.ewma` as a CPU scalar tensor (`torch.tensor(0.)`), and later operations that mix this with CUDA tensors raise errors.

Additionally, the NaN/inf guard `torch.isnan(self.ewma).any()` is called on a scalar tensor where `.any()` is invalid, and the moving-average state is not detached from the autograd graph, which retains gradients across steps and can cause unexpected behaviour.

**Expected behaviour**
`EwmaLog` should be device-agnostic: its internal state should follow the device of the incoming inputs. NaN/inf checks must work correctly for both scalar and tensor states, and the moving average should be stored as a detached value so that no history accumulates. The OED estimators (`donsker_varadhan_eig`, etc.) should work without errors on both CPU and GPU.

**Impact**
This blocks all GPU usage of the OED module, and the corresponding tests are marked as expected failures. Fixing `EwmaLog` will unblock GPU workflows and allow the xfailed test cases to pass.


Simplify the einsum shared intermediate backend to improve clarity and robustness.

Current issues:
- `shared.py` uses a `_Shared` wrapper object that is unnecessary; it complicates caching and forces all shared computation to go through intermediate objects.
- `_CURRENT_BACKEND` is a global list, which is awkward and error-prone.
- The `contract` function contains special-case logic for shared intermediates that is tangled with path caching.
- The `shared_intermediates` context manager does not guarantee cleanup of `_SHARING_STACK` on exception.
- Tensor id recycling can cause false cache hits because only ids are used as keys without keeping the tensors alive.
- Profiling variable `LAST_CACHE_SIZE` is embedded in `shared.py` and only reports the total number of cache entries, not a breakdown by operation type.
- The `_PATH_CACHE` for contraction path memoization is in `shared.py` but should logically reside in the public `contract` entry point in `__init__.py`.

Desired changes:
1. Remove the `_Shared` class. Shared functions (`transpose`, `tensordot`, `einsum`) should directly compute and cache backend results, returning normal tensors.
2. Introduce a `handle_sharing` context manager that sets `_CURRENT_BACKEND` appropriately and yields a backend name, removing the need for `_CURRENT_BACKEND` as a list. The `contract` function in `__init__.py` should use this context manager.
3. Move `_PATH_CACHE` to `pyro/ops/einsum/__init__.py` and keep path caching separate from intermediate caching.
4. Ensure `shared_intermediates` uses a try/finally to pop the stack reliably.
5. Add a `_save_tensors` helper that stores tensors in the cache under a 'tensor' key to prevent id recycling issues; use it before hashing tensor ids.
6. In `transpose`, `tensordot`, `einsum`, hash using tensor `id()` (kept alive by `_save_tensors`) and canonical equation forms, not the wrapper.
7. Move `LAST_CACHE_SIZE` to `pyro.infer.util` and change it to a `Counter()` so growth tests can report separate counts for 'einsum', 'tensordot', etc.
8. Update imports in tests accordingly, and adjust growth tests to use the new `LAST_CACHE_SIZE` from `pyro.infer.util` and print collated per-operation costs.

The external behavior of the `contract` and `shared_intermediates` APIs must remain the same; only internal simplification and improved profiling.


Add a `GaussianHMM` distribution in `pyro.distributions` that models a hidden Markov model with linear Gaussian dynamics:

```
z_0 ~ initial_dist
z_t = z_{t-1} @ transition_matrix + transition_dist
x_t = z_t @ observation_matrix + observation_dist
```

- Accept `initial_dist` (MultivariateNormal), `transition_matrix` (Tensor), `transition_dist` (MultivariateNormal), `observation_matrix` (Tensor), `observation_dist` (MultivariateNormal).
- All inputs should broadcast to a common batch shape and an optional leading time dimension; the distribution’s `batch_shape` excludes the time dimension, `event_shape` = `(num_steps,) + (obs_dim,)`.
- Support both homogeneous (num_steps=1 broadcasted) and heterogeneous (explicit time dimension) parameters.
- `log_prob(value)` must compute the exact marginal log-density of the observations, efficiently parallelized over time using `_sequential_gaussian_tensordot`, following the pattern of `DiscreteHMM` but with continuous Gaussians.
- The `log_prob` result must be differentiable (log-normalizer tracked).
- Provide an `.expand(batch_shape)` method that only expands the initial factor for efficiency.

Add a helper function `matrix_and_mvn_to_gaussian(matrix, mvn)` in `pyro.ops.gaussian` that converts the noisy affine map `y = x @ matrix + mvn.sample()` into a joint `Gaussian` over `(x, y)` with the correct precision, info_vec, and log_normalizer. The resulting `Gaussian` must have `dim() == x_dim + y_dim` and broadcasted batch shape.

Update `docs/source/distributions.rst` to include a `GaussianHMM` entry.

Implement GaussianHMM distribution (#1984)


**Title:** `sync_module` incorrectly merges all module-scoped parameters into the target module

When multiple `nn.Module`s are registered with `pyro.module()` (e.g., `"modA"` and `"modB"`), calling `sync_module("modA", target)` builds a `state_dict` that contains parameters belonging to _every_ registered module, not just `"modA"`. This causes `load_state_dict` to fail or silently assign wrong weights to `target`. The function should narrow the parameter set to those prefixed with the given `pyro_name` before stripping the namespace.

While fixing this, also address two smaller issues:
- `sync_module` currently returns `None`; it should return the module argument to allow chaining.
- Add a short docstring describing the purpose of `sync_module` (save/load round-trip support for nn modules).

Edge cases / invariants:
- Parameters that are not module-scoped (no namespace divider) must still be ignored.
- The module name `pyro_name` is validated to not contain the namespace divider itself (existing guard).
- After a save/clear/load cycle, calling `sync_module` for a specific module must restore exactly that module’s parameters without interference from other registered modules.

did suggested code review changes to paramstore


We need a new `ELBO`-like objective for variational inference that replaces the KL divergence with the Maximum Mean Discrepancy (MMD) between the variational posterior and the prior for latent variables. The class should be named `Trace_MMD` and be usable as the `loss` argument to `SVI`.

Requirements:
- Accept a `kernel` argument, either a single `pyro.contrib.gp.kernels.Kernel` (applied to all latent sites) or a dict mapping site names to individual kernels.
- Accept an `mmd_scale` argument (float or site→float dict) to weight each MMD term.
- Inherit the standard ELBO constructor parameters: `num_particles`, `max_plate_nesting`, `vectorize_particles`, `strict_enumeration_warning`, `ignore_jit_warnings`, `retain_graph`.
- Support both `vectorize_particles=True` and `False`. When vectorized, gather and reshape model/guide samples across particles appropriately before computing MMD. When not vectorized, flatten per-particle samples into a row.
- Only the particle dimension is treated as a batch dimension for MMD; all other dimensions are event dimensions. Document this limitation and recommend large `num_particles` and `vectorize_particles=True`.
- The loss formula should be: `-E_{q}[log p(x,z)] + sum_{latent sites} mmd_scale * MMD(q(z), p(z))`, where the expectation is over particles, MMD is computed via `_compute_mmd`, and the log-likelihood term only includes observed sites (identified by `model_site['is_observed']`).
- Both model and guide latent sites must be reparameterizable (`has_rsample`). Raise a `ValueError` otherwise.
- Expose standard methods: `loss` (returns a python float), `differentiable_loss` (returns a tensor that supports backward), and `loss_and_grads` (calls `differentiable_loss` and backward).
- Implement `_compute_mmd(X, Z, kernel)` that computes `MMD = E[k(X,X)] + E[k(Z,Z)] - 2 E[k(X,Z)]`.
- Use `get_importance_trace("flat", ...)` to obtain model and guide traces, consistent with other ELBO variants.
- Document references to the MMD-VAE / InfoVAE papers.

Edge cases:
- If an observed site has a corresponding site in the guide, it must still be treated as observed (i.e., contribute to log-likelihood, not MMD).
- No discrete enumeration support needed; only continuous reparameterizable latent variables.
- The `kernel` setter should store a `defaultdict` internally for convenience.
- The `mmd_scale` setter should similarly store a `defaultdict` that converts to float.

Add Trace_MMD class, add tests that MMD correctly fits distributions (#1818)


```
ubersum fails when called inside a @torch.jit.trace or pyro.ops.jit.trace context. The tracer cannot handle:

1. Iterating over tensor .shape directly — .shape returns a torch.Size whose elements are not plain ints, causing tracer errors.
2. Certain sumproduct calls inside Ring.global_local — the tracer raises warnings or fails on these code paths.

Make ubersum (and the ring operations it calls) traceable under the JIT without changing behavior. Use existing repo mechanisms to suppress known-safe JIT warnings where needed.
```

Fix ubersum to be compatible with pytorch jit (#1648)


Add `--tmc` and `--tmc-num-samples` flags to the capture-recapture (`cjs.py`) and hidden Markov model (`hmm.py`) examples.
When `--tmc` is set:
- Use `TraceTMC_ELBO` instead of `TraceEnum_ELBO`.
- Wrap the model with `poutine.infer_config` so that every site annotated with `infer={"enumerate": "parallel"}` receives `{"num_samples": args.tmc_num_samples, "expand": False}`.
- For `hmm.py`, if JIT is also requested (`--jit`), raise a `NotImplementedError` since JIT support is not yet available for `TraceTMC_ELBO`.
No other behavior should change; the existing enumeration path must continue to work exactly as before.

Add Tensor Monte Carlo support to more existing examples (#2145)


```markdown
**Title:** SVI with `PyroLRScheduler` does not update model parameters

**Description:**
When using a learning-rate scheduler wrapper (e.g., `pyro.optim.ExponentialLR` or `pyro.optim.LambdaLR`) as the optimizer for `SVI`, calling `svi.step()` does **not** change the model parameters.  The learning rate appears to be scheduled correctly (e.g., the `lr` attribute changes across steps), but the actual gradient-based parameter update never occurs.  This makes the scheduler wrappers unusable for training.

**Expected behavior:**
After each `svi.step()`, the model parameters should be updated using the current scheduled learning rate, just as they would be with a plain `PyroOptim` wrapper (e.g., `pyro.optim.SGD`).  The scheduler should still advance its internal state (learning rate, epoch count, etc.) on each step.

**Additional context:**
- The problem occurs because the underlying `torch.optim.lr_scheduler._LRScheduler` object is called, but only its `step()` method is invoked – this updates the learning rate but does **not** call the wrapped optimizer’s `step()`.  The optimizer that actually holds the parameter gradient step is `scheduler.optimizer`.
- Any special keyword arguments (e.g., `epoch`) should be forwarded to the scheduler’s `step()` as currently, but the optimizer’s step should receive the remaining arguments (excluding `epoch`) to avoid unexpected keyword errors.
- The fix should be compatible with all PyTorch LR schedulers that can be constructed through `PyroLRScheduler`.
```

fix optim lr scheduler bug, clarify docs (#1246)


## Bug report: `detach_iterable` does not detach tensors inside distributions, corrupting baseline gradients in `TraceGraph_ELBO`

### Problem
`TraceGraph_ELBO._construct_baseline()` uses `detach_iterable()` to stop gradients from flowing through the `nn_baseline_input` when constructing the baseline loss. However, `detach_iterable()` only detaches raw `torch.Tensor` objects or flat lists of tensors. If the guide site’s `nn_baseline_input` contains a `pyro.distribution.Distribution` (e.g., a `Normal`, `Beta`, or `TransformedDistribution`), all tensors inside that distribution are *not* detached. This causes gradients to leak into the baseline network through distribution parameters, leading to incorrect gradient estimates and unstable training.

### Expected behavior
There should be a utility that can recursively detach **all** tensors nested inside an arbitrary Python object, including distributions, transforms, tuples, lists, dicts, and any other objects that may contain tensors. This utility should:
- Accept any object.
- Return a deep copy of the object where every `torch.Tensor` is replaced with `tensor.detach()`.
- Not copy tensor data (i.e., avoid expensive data copies).
- Preserve the original object’s structure, including distribution types, batch shapes, and validation settings.
- Be usable in JIT‑traced functions without introducing graph breaks.

### Impact
Without this, users of `TraceGraph_ELBO` who use distribution‑valued baseline inputs see degraded or incorrect learning due to the gradient leak. Fixing this will require replacing the use of `detach_iterable` in `pyro/infer/tracegraph_elbo.py` with a general `detach()` that handles distributions and other structured objects correctly. The existing `detach_iterable` in `pyro/infer/util.py`, which is only used for this purpose, should be removed in favor of the new utility.

Implement a detach() helper that works on distributions (#2599)


```
Title: `pyro.module` should skip parameters with `requires_grad=False`

When I use `pyro.module(name, my_module)` to register a PyTorch module’s parameters in the Pyro parameter store, I expect only trainable parameters to be registered and managed by Pyro. However, the current implementation registers **every** parameter returned by `nn.Module.named_parameters()`, including those with `requires_grad=False` (for example, frozen embedding layers or fixed biases).

Why this is a problem:
- Non‑trainable parameters end up in the parameter store and can be picked up by inference algorithms or guides, leading to silent correctness issues.
- Pyro’s parameter machinery (`pyro.param`) assumes the registered entities require gradients; including frozen parameters can cause confusing runtime errors or wasted optimisation effort.
- There’s no indication to the user that non‑trainable parameters are being registered, so misbehaviour is hard to diagnose.

Desired behaviour:
- `pyro.module` should only register parameters where `param.requires_grad == True`.
  Parameters with `requires_grad=False` should be intentionally skipped.
- Optionally, a clear warning (e.g. `UserWarning`) should be emitted for each skipped parameter, helping users understand why a parameter is absent from the store.

This would align `pyro.module` with the expectation set by PyTorch optimisers and parameter handling, and prevent subtle bugs when modules contain frozen parts.
```

`module` will ignore params with requires_grad=False (#1779)


We need `kl_divergence` to cover more distribution combinations that appear in practical Pyro guides and models. Currently, several useful pairings either raise `NotImplementedError` or are simply not registered.

### Desired behavior

1. **Point-mass (Delta) against any distribution**
   `kl_divergence(Delta, q)` should be defined for any valid PyTorch distribution `q` as `-q.log_prob(Delta.v)`. This handles models that collapse to a point estimate.

2. **Independent vs Independent with differing event dimensions**
   When `p` and `q` are both `Independent` wrappers but have different `reinterpreted_batch_ndims`, the current code raises `NotImplementedError`. The corrected behavior should:
   - Identify the **shared** rightmost event dimensions `shared = min(p_ndims, q_ndims)`.
   - Treat the remaining dimensions on each side as part of the base distribution’s event shape.
   - Compute `kl_divergence` on those (possibly truncated) base distributions.
   - Sum the result over the shared event dimensions.
   Example: `kl_divergence(Independent(Normal, 1), Independent(MultivariateNormal, 2))` should succeed and match the analytical KL between a factorized normal and a full-covariance normal, properly summed over the appropriate dimensions.

3. **Factorized Normal to MultivariateNormal**
   `kl_divergence(Independent(Normal, 1), MultivariateNormal)` should be specially registered to avoid going through Monte Carlo. The result should be identical to the KL between the equivalent diagonal-covariance `MultivariateNormal` and the target `MultivariateNormal`. This allows efficient, closed-form computation when the guide uses a diagonal Gaussian and the model uses a full-covariance Gaussian.

4. **Factorized Delta to MultivariateNormal**
   Similarly, `kl_divergence(Independent(Delta, 1), MultivariateNormal)` should be defined as `-MultivariateNormal.log_prob(Delta.v)`.

### Non‑obvious edge cases / constraints
- All implementations must correctly handle arbitrary batch shapes; the returned tensor must have the proper batch dimensions (with event dimensions reduced).
- The shared‑ndims logic for `Independent` vs `Independent` must not assume that the base distributions are identical in type; the only requirement is that `kl_divergence` between the truncated base distributions is registered.
- The existing module `pyro/distributions/torch.py` currently contains a partial `kl_divergence` registration for `Independent`–`Independent` that only works when `p.reinterpreted_batch_ndims == q.reinterpreted_batch_ndims`. That registration (and any future ones) should be moved to a dedicated `pyro/distributions/kl.py` module that is imported automatically when `pyro.distributions` is imported, so that all KL registrations are loaded as a side effect.

### Acceptance criteria (observable)
- Calling `torch.distributions.kl_divergence(p, q)` succeeds for:
  - `Delta` with any distribution (e.g., `Normal`, `MultivariateNormal`, `Categorical`).
  - `Independent` wrapping `Normal`/`Delta` (event_dim=1) against another `Independent` with a different `reinterpreted_batch_ndims`.
  - `Independent(Normal, 1)` against `MultivariateNormal`.
- The results match manual computation or, where applicable, an alternative formulation that is mathematically equivalent.
- No extra user imports are needed; simply importing `pyro.distributions` activates the registrations.

Implement KL divergence for delta and independent distributions (#2043)


**Feature request: Default `.expand` for all Pyro distributions**

Currently, calling `.expand` or `.expand_by` on many Pyro distributions raises `NotImplementedError` because they rely on PyTorch’s base class that does not provide a generic implementation. Users need a way to broadcast batch dimensions of *any* `TorchDistribution` without every distribution having to implement its own expand logic.

## Desired behavior/acceptance criteria

1. **Universal `expand` support**
   For any Pyro distribution `d` (including custom ones derived from `TorchDistribution`), `d.expand(batch_shape)` must return a valid distribution `e` such that:
   - `e.batch_shape == broadcast_shape(d.batch_shape, batch_shape)` when the shapes are broadcastable.
   - `e` supports the same operations as the original distribution: `sample`, `rsample`, `log_prob`, `score_parts`, `enumerate_support`, `mean`, `variance`.
   - `e.log_prob(x)` yields the same result as `d.log_prob(x).expand(batch_shape + d.event_shape)` (after aligning events), preserving gradient flow appropriately.
   - `e.sample()` produces samples with shape `sample_shape + e.batch_shape + e.event_shape` and the correct statistical properties.
   - `e.score_parts(x)` returns a `ScoreParts` object where `log_prob`, `score_function`, and `entropy_term` are correctly broadcast to `e.batch_shape`.

2. **`expand_by` never raises `NotImplementedError`**
   `d.expand_by(sample_shape)` should use the default expand mechanism when the distribution’s own `.expand` is not implemented, so that `expand_by` works for all distributions.

3. **Correctness under composition**
   Expanding a distribution that has already been reshaped (via `.to_event` or `.independent`) must produce a distribution whose batch and event shapes follow the usual broadcasting rules. For example, reshaping `OneHotCategorical` then expanding should not produce shape mismatches.

4. **Proper error handling**
   - Attempting to expand to a shape that is not broadcastable (e.g., shrinking an existing batch dim from 5 to 3, or mixing incompatible sizes) must raise a `ValueError` or `RuntimeError`.
   - Expanding an already expanded distribution must still allow further expansion only in directions compatible with the original base distribution’s shape; unauthorized shrinking must fail.

5. **No regression for existing distributions**
   Distributions that already define their own `.expand` (like `MaskedDistribution`) must keep working identically; their `expand` is preferred when it does *not* raise `NotImplementedError`.

6. **Log-prob and score parts broadcasting**
   The expanded distribution’s `log_prob` and `score_parts` should return tensors whose batch shape matches the expanded batch shape, even when the base distribution’s batch shape is smaller.

7. **Enumerate support**
   `enumerate_support(expand=True)` on the expanded distribution must return an enumerable tensor that, when expanded, has shape `(enum_size,) + batch_shape`.

## Non-observable constraints (for implementation reference)
- The implementation should introduce a generic `ExpandedDistribution` wrapper (similar in spirit to `MaskedDistribution`) that delegates to a base distribution and handles batch broadcasting at sample/log-prob time.
- `TorchDistribution` must override `expand` to use this default instead of relying on the PyTorch base class’s `NotImplementedError`.

This feature would remove a long-standing friction point and make working with batch dimensions in Pyro models more predictable and consistent.

Add default TorchDistribution.expand method (#2209)


Add an optional `clip_args` dict to `PyroOptim` and `PyroLRScheduler` to support gradient clipping.

The existing `optim_args` parameter can be a dict or a callable returning per‑parameter dicts. The new `clip_args` should work the same way, accepting keys `"clip_norm"` and/or `"clip_value"` (matching `torch.nn.utils.clip_grad_norm_` / `clip_grad_value_`). When `clip_args` is not `None`, and contains at least one of these keys, gradients of each parameter are clipped immediately before the optimizer step in `__call__`. This must work for both standard optimizers and for `PyroLRScheduler` (which wraps a scheduler around an optimizer). The auto‑generated wrappers in `pyro.optim.pytorch_optimizers` should also forward `clip_args`. No clipping should be performed if `clip_args` is omitted or empty.

First pass at adding `clip_args` to PyroOptim (#2111)


The `batch_log_pdf` method of `MultivariateNormal` raises a `NotImplementedError` whenever `normalized=False` and `sigma_cholesky.requires_grad` is `True`. This prevents using the distribution in any model that requires differentiating through the unnormalized log-density, even though the value of the normalization constant itself is not needed.

**Steps to reproduce**
```python
import torch
from torch.autograd import Variable
from pyro.distributions import MultivariateNormal

mu = Variable(torch.randn(5))
L = Variable(torch.tril(torch.randn(5, 5)), requires_grad=True)   # scale_tril
dist = MultivariateNormal(mu, scale_tril=L, normalized=False)
x = Variable(torch.randn(5))
log_p = dist.batch_log_pdf(x)
log_p.backward()   # currently raises NotImplementedError
```

**Observed behavior**
```
NotImplementedError: Differentiation is not supported (yet) if normalized=False.
```
The error appears both when the distribution is parameterised by a full `covariance_matrix` and when it is parameterised by `scale_tril`.

**Expected behavior**
- Calling `batch_log_pdf` (or `log_pdf`) with `normalized=False` should be differentiable with respect to the covariance parameters (or `scale_tril`).
- The gradient should be the same as when `normalized=True` so that parameter updates are correct, while the forward value can remain the unnormalised quadratic form. This allows users to avoid computing the normalisation constant when only the shape of the density matters, yet still obtain proper parameter gradients for optimisation or MCMC.

**Additional context**
The `normalized=False` option is useful for variance reduction or when the normalising constant is cancelled in ratios (e.g., in Metropolis-Hastings). Without gradient support it cannot be used with algorithms like SVI or HMC that rely on automatic differentiation through the log density. Lifting this restriction would make `MultivariateNormal` uniformly usable in those contexts.

(If gradients of the cholesky factor `scale_tril` have additional complications, please treat them consistently with the full covariance case.)

Add Gradients of MultivariateNormal.batch_log_pdf  (#663)


```
Title: Simplify sum_dims parameter in tensor contraction from a per‑tensor dict to a flat set

Background
In `pyro.ops.contract`, the functions `contract_to_tensor`, `contract_tensor_tree`
and their helper `_contract_component` currently require a `sum_dims` dictionary that
maps each individual tensor to the set of dimensions that should be summed out.
This mapping is redundant because the relevant dimensions can always be inferred
from the global set of dimensions that are candidates for summation and the shape
of each tensor. Moreover, the dict forces extra bookkeeping inside the contraction
logic (`_contract_component` mutates it) and makes the API more cumbersome than
necessary for callers.

Goal
Refactor `sum_dims` so that it becomes a plain `set` of integer dimensions
(indexed from the right, negative). The set represents all dimensions that
may be summed out anywhere in the contraction problem. The contraction
implementation should internally determine the actual dimensions to sum for a
given tensor by intersecting the global `sum_dims` with the tensor’s own
dimensions (those with size > 1).

Scope
- Public / internal functions that accept `sum_dims`:
  * `contract_to_tensor`
  * `contract_tensor_tree`
  * `_contract_component`
  * `ubersum` (currently builds a per‑tensor dict)
- Downstream callers in `pyro/infer` that construct or consume `sum_dims`:
  * `_compute_model_factors`, `_compute_marginals` in `pyro.infer.traceenum_elbo`
  * `BackwardSampleMessenger` (also in `traceenum_elbo`)
  * `TraceEinsumEvaluator.log_prob` in `pyro.infer.mcmc.util`
- Tests under `tests/ops/test_contract.py` must continue to pass after adjusting
  only the construction of `sum_dims` to the new type.

Behavioural invariants
- The contraction result must be identical to the current implementation.
- `sum_dims` is a set of negative integer dimensions; tiles that do not contain
  those dimensions (or where the size equals 1) simply ignore them.
- `_contract_component` must no longer mutate or populate a per‑term sum_dims
  dictionary. Instead it should derive necessary dims from tensor shapes
  (e.g., using `ring.dims(term)`) and check membership in the global set.
- `ubersum` should produce a single global `sum_dims` set from its equation
  instead of a per‑operand dict.
- Any caching in `UnpackedLogRing.dims` must remain correct.

Acceptance criteria
- All existing tests pass without changes to their logic (only the shape of
  the `sum_dims` argument may change from a dict to a set).
- The new code is clean, well-documented, and avoids residual dead bookkeeping.
```

Refactor sum_dims to be a set rather than dict (#1500)


Add support for HMC/NUTS on distributions with constrained support.
Currently, these kernels only work when all latent sites have unconstrained (real) support and are reparameterized.
They should accept models with constrained latent variables (e.g., Beta, Gamma, Dirichlet, etc.) by:

- Automatically transforming each constrained variable to an unconstrained space for Hamiltonian simulation, using `torch.distributions.biject_to` and the support of the site’s distribution.
- Applying the corresponding Jacobian log-determinant correction in the potential energy.
- Optionally allowing the user to supply custom transformations via a `transforms` dict.
- Dropping the assertion that every stochastic node must be reparameterized (the transform handles the constraint).

The NUTS kernel (which inherits from HMC) should gain the same behavior without duplicating logic beyond what’s needed for its tree building.

Edge cases:
- If no bijection is registered for a support in `constraint_registry`, the standard error from `biject_to` is acceptable.
- For the `Dirichlet` support specifically, the `StickBreakingTransform` currently lacks a `log_abs_det_jacobian`, which will cause a failure; that can be addressed later but should be noted as a known limitation.

Using torch transforms to run HMC on distributions with constrained support (#740)


We need the `VonMises` distribution to support sampling.

Currently, the only available operation is `log_prob`.
Users can evaluate the log-density of a given angle, but cannot generate random samples.

The required behavior:

1. **Sampling method**
   - `sample(sample_shape)` must draw angles from the von Mises distribution.
   - The sampling algorithm must be numerically stable and efficient across a wide range of concentration values (very small ~0.01 up to very large ~100).
   - The implementation must handle batched `loc` and `concentration` tensors, and support arbitrary `sample_shape` leading to correct output shapes.
   - Sampling must work on both CPU and CUDA devices.
   - Reparameterized sampling is not required (`has_rsample` remains `False`).

2. **Distribution properties**
   - `mean` property should return the location parameter (the circular mean).
   - `variance` property should return the circular variance, computed from the concentration.

3. **Internal helper**
   - The existing `_log_modified_bessel_fn` (which currently only handles order 0) must be extended to support order 1 as well, for use in variance calculation and the sampling algorithm.

The sampling method should follow the Best–Fisher algorithm (rejection sampling) as described in the reference paper.

Any solution must not change the existing `log_prob` interface or its correctness.

Implement sampling for von Mises distribution based on Best-Fisher algorithm (#1729)


We need a `MaskedMixture` distribution that elementwise selects between two component distributions based on a binary mask tensor. This will be used when the mask is a latent variable that can be marginalized via parallel enumeration.

Requirements:
- Constructor: `MaskedMixture(mask, component0, component1)`. `mask` must be a `ByteTensor`. The two components must have identical `event_shape`. The constructor should broadcast `mask`, `component0`, and `component1` to a common batch shape.
- `log_prob(value)` should evaluate `component0.log_prob` where `mask == 0` and `component1.log_prob` where `mask == 1`, broadcasting appropriately over sample and batch dims. Validation of inputs must respect the per-element constraint: implement a `MaskedConstraint` that checks each element against the support of the component from which it was drawn. Disable per‑component internal validation to avoid false positives on cross‑component samples.
- `sample` / `rsample` should similarly draw from the appropriate component per element.
- `mean` and `variance` should be the elementwise mixture of the component means/variances.
- `expand(batch_shape)` must return a new `MaskedMixture` with expanded mask and components.
- Support multivariate event shapes. To enable correct support handling for multivariate distributions, introduce an `IndependentConstraint` that wraps a base constraint and aggregates over `reinterpreted_batch_ndims` event dimensions during `check`. Existing multivariate distributions (`MultivariateNormal`, `LowRankMultivariateNormal`) and `ReshapedDistribution` should be updated to use `IndependentConstraint` for their support so that `_validate_sample` works with multivariate events. `ReshapedDistribution` must also propagate `_validate_args` to its base distribution.
- Update the EM tracking example (`test_em.py`) to replace the use of `poutine.scale` with a single `MaskedMixture` for real vs. fake detections.
- Add the new distribution to the docs and the package `__init__`.

No breaking changes to existing public APIs.

Implement MaskedMixture distribution (#1264)


**Title:** Missing Cauchy distribution

**Description:**
Pyro does not include the Cauchy (Lorentz) distribution. This omission prevents users from building models that require a heavy-tailed continuous distribution, such as regression with Cauchy noise or prior distributions where the mean and variance are not defined. I need to use the Cauchy distribution in my model, but no implementation exists.

**Expected behavior:**
A `Cauchy` distribution should be added to `pyro.distributions` with:

- Parameters `mu` (location) and `gamma` (scale), both tensors.
- A `sample()` method that uses PyTorch’s underlying Cauchy sampler.
- `log_pdf()` and `batch_log_pdf()` methods that compute the log-probability density correctly.
- `analytic_mean()` and `analytic_var()` that raise a `ValueError` because the Cauchy distribution has no defined mean or variance.
- The same stateful/stateless calling convention as other distributions (e.g., `Cauchy(mu, gamma)` and `cauchy()` for the default singleton).
- Registration in `pyro/distributions/__init__.py` and documentation entry in the docs so that the distribution is discoverable.

**Impact:**
Without this distribution, models that require Cauchy priors or likelihoods cannot be expressed natively in Pyro, forcing users to implement workarounds or abandon the framework for these use cases.


We need a `mask(mask)` method on `TorchDistribution` that returns a new distribution whose `log_prob` (and `score_parts`) are element-wise multiplied by a broadcastable zero-one mask. This replaces the use of `poutine.scale(None, mask)` for per-time-step masking in the DMM example.

Requirements:
- `mask` must be a zero-one float tensor broadcastable to the distribution’s `batch_shape`; raise `ValueError` otherwise.
- The masked distribution should delegate `sample`, `rsample`, `enumerate_support`, `mean`, `variance` transparently to the base distribution.
- Update the DMM example (model and guide) to use `dist.Normal(...).mask(mini_batch_mask[:, t - 1:t])` and `dist.Bernoulli(...).mask(mini_batch_mask[:, t - 1:t])`, removing the `poutine.scale(None, ...)` wrapping for the per-step mask (but keep the `poutine.scale(None, annealing_factor)` for annealing).

Implement TorchDistribution.mask() for dependent masks (#821)


We need a `PermuteTransform` bijection to use between `InverseAutoregressiveFlow` layers. Currently we only provide `InverseAutoregressiveFlow`, but chaining multiple IAFs without permuting the input ordering limits the flexibility of the induced distribution. A permutation transform that reorders the event dimensions will allow shuffling the variable ordering between autoregressive steps, which is standard practice for normalizing flows.

**Request**
1. Implement a `PermuteTransform` class inheriting from `torch.distributions.transforms.Transform`:
   - Constructor accepts a `torch.LongTensor` permutation vector specifying the new ordering.
   - `_call(x)` applies the permutation: `x[..., self.permutation]`.
   - `_inverse(y)` applies the inverse permutation. Compute the inverse lazily (e.g., via `lazy_property` so it’s cached on the tensor device).
   - `log_abs_det_jacobian` should return zeros shape like `x`, since the determinant magnitude of a permutation matrix is always 1.
   - Mark `bijective = True` and `codomain = constraints.real`.

2. Ensure the permutation and its inverse are placed on the correct device (e.g., using `self.permutation.device` when constructing the inverse).

3. Add `PermuteTransform` to `pyro/distributions/__init__.py` and export it.

4. Add documentation entry in `docs/source/distributions.rst` under the “Transformed Distributions” section (similar to `InverseAutoregressiveFlow`).

5. While we’re modifying the flow tests, the current test file `tests/distributions/test_iaf.py` mixes flow tests with pure `AutoRegressiveNN` mask/jacobian tests. Please split them:
   - Move the `AutoRegressiveNNTests` class into a new file `tests/nn/test_autoregressive.py`.
   - Keep only the flow-specific tests (IAF jacobian, shapes, and the new permutation tests) in a new file `tests/distributions/test_flows.py`. This should include tests for the `PermuteTransform` that verify inversion correctness and shape handling.

Edge cases:
- The permutation vector must be a 1D `LongTensor` of unique indices covering `0..dim-1`.
- The identity permutation should be allowed and trivial.
- The transform should work with any number of batch dimensions (i.e., index along the last axis).
- The inverse should remain correct after moving tensors between devices.