**Title:** Plugin option validation should warn on unknown options rather than failing the build.

**Description:**
When a user configures a Gatsby plugin with an option that is not recognized by that plugin’s schema, the current behavior halts the build with an error. This is too aggressive—unknown options should only produce a warning, allowing the site to build successfully. Additionally, the warning message must provide a usable link to the plugin’s repository for reporting a potentially valid but undocumented option.

**Acceptance Criteria:**

1. **Build outcome:** A site build must not fail (i.e., must not call `process.exit`) when any plugin receives one or more options that are absent from its `pluginOptionsSchema`. The build should succeed and the unknown options should produce a **warning** logging each unknown key and a link to `https://ghub.io/<plugin-name>`.
2. **Warning vs. error separation:** The bootstrap validation logic (`validatePluginsOptions`) must clearly distinguish **unknown-option warnings** from **actual schema errors** (e.g., wrong type, missing required field). Only schema errors increment the error count that leads to a build failure; unknown options count as warnings and must not affect the exit status.
3. **Validation function return value:** The `validateOptionsSchema` function (from `gatsby-plugin-utils`) must **return** an object containing `value` (the processed options, with defaults applied) and `warning` (with a `details` array of unknown-option info) instead of throwing an error when unknown keys are present. The function must still **throw** for genuine validation errors so existing error-handling flow works.
4. **Test utility output:** The `testPluginOptionsSchema` test helper must reflect the new behavior:
   - Passing unknown options must result in `isValid: true`, `hasWarnings: true`, and the `warnings` array containing the unknown key messages.
   - Passing invalid known options (e.g., incorrect types) must still result in `isValid: false` with the corresponding `errors` array.
5. **Deprecated option handling:** Plugins that previously used `Joi.forbidden().messages(...)` to emit a custom message for a renamed/removed option must instead rely on the standard “not allowed” warning produced by the unknown-option flow. This ensures a consistent warning format and removes the need for one-off forbidden messages.
6. **Telemetry:** The telemetry event `UNKNOWN_PLUGIN_OPTION` (tracked via `trackCli`) must be associated with the **warning** path, not an error path, and must not contribute to error counts.
7. **Link protocol:** The warning message URL must use the `https` protocol (`https://ghub.io/…`); the previous bare `ghub.io` link must be corrected.

**Edge Cases / Safety Constraints:**

- Invalid known options (type mismatches, missing required fields, etc.) must continue to cause a build failure, and `validateOptionsSchema` must throw in those cases.
- Plugins with no schema or a non‑object schema must still log an appropriate warning without a crash.
- The warning message must include the **relative config directory** when the plugin is configured by a theme, as before.
- When a plugin has sub‑plugins, validation after applying the main options must still function correctly and not mask warnings or errors.

fix(gatsby): don't throw on warnings in `pluginOptionsSchema` (#34182)


**Title:** Development redirect triggers erroneous “matches both a page and a redirect” warning for routes without a user-defined page

**Type:** Bug report

**Affected scope:** Development runtime – client-side redirect handling

**Observed problem:**
In development mode, creating a client-side redirect via `createRedirect` (with `redirectInBrowser: true`) for a path that does **not** have a corresponding page still logs the console error:

> The route "/some-path" matches both a page and a redirect; this is probably not intentional.

This happens even though no page has been explicitly created for that path. The warning is misleading and clutters the developer console.

**Steps to reproduce:**
1. In a Gatsby project, use `createRedirect` in `gatsby-node.js` to define a redirect from `/from-path` to another path (with `redirectInBrowser: true`), but **do not** create a page at `/from-path`.
2. Run `gatsby develop` and navigate to `/from-path`.
3. Observe the browser console. The error about a matching page appears, despite there being no user-defined page.

**Expected behavior:**
No error should be logged. The redirect should work silently when no explicit page exists at that route. The warning should only appear when both a real page (a page created by the user, not the `404` fallback) and a redirect target the same route.

**Impact:**
- Misleading error message for valid redirects.
- Makes it harder to spot genuine conflicts.
- Obscures the fact that the development runtime is treating the 404 fallback as a “page” for conflict detection.

**Suggested approach:**
The redirect conflict detection in `maybeRedirect` should distinguish between a route that is truly backed by a user-defined page and one that is only served by the `404` page fallback. A route that resolves to the `404` page should be considered “not found” for the purpose of redirect warnings. This could be achieved by exposing a capability on the loader (like `isPageNotFound`) so the check can safely skip the warning when no real page exists.

fix(gatsby): don't show error for proper redirects (#19789)


**Title:** Slices do not render in pages served via Development SSR (`gatsby develop`)

---

In `gatsby develop`, when a page uses `<Slice alias="..." />` and is rendered on the server (SSR), the slice content is missing or the page fails. Slices work correctly in static generation (SSG) and in production SSR engines, but the development SSR code path does not set up the necessary slice data and context providers.

The root cause is in the dev SSR static entry (`ssr-develop-static-entry.js`):
- It never reads slice data from `public/slice-data`.
- It never wraps the page component tree with `<SlicesContext.Provider>`, `<SlicesMapContext.Provider>`, `<SlicesPropsContext.Provider>`, or `<SlicesResultsContext.Provider>`.
- The `Slice` component’s own render environment check (`slice.js`) does not recognize `"dev-ssr"` as a valid environment for inline rendering; it only handles `"browser"` and `"engines"`.

Additionally, the SSR component filter in `requires-writer.ts` incorrectly prunes slice components (those whose chunk names start with `slice---`) from the set of visited page components, leading to a missing component chunk error when a development SSR page includes a slice.

**Expected behavior:**

- Visiting an SSR page in `gatsby develop` that uses `<Slice>` should produce the same rendered output (slice content, correct data, etc.) as in the production SSR engine build.
- The page must not crash with “cannot find module” errors for slice chunks.
- Slice data should be read from the slice data JSON files available in the development public directory, just as the engine renderer does.

**Acceptance criteria:**

1. Dev SSR pages that reference slices render all slice content without errors.
2. The `<Slice>` component resolves correctly in dev SSR by recognizing `"dev-ssr"` as an inline rendering environment.
3. Slice components remain included in the dev SSR component bundle and are not discarded during the SSR visited component cleanup.

fix(gatsby): support slices in DEV_SSR (#37542)


Extend the tag linking feature (controlled by plugin option `enableTags`) from Contentful entries to assets. When `enableTags` is true, asset nodes (`ContentfulAsset`) must expose `metadata.tags` as references to `ContentfulTag` nodes, just like entry nodes do. This allows filtering asset queries by tag contentful_id (e.g., `allContentfulAsset(filter: { metadata: { tags: { elemMatch: { contentful_id: { eq: "…" } } } } })`) and accessing tag names on asset nodes.

Implementation constraints:
- Modify `createAssetNodes` to map the asset’s `metadata.tags` array to Gatsby node IDs using `createNodeId` (same pattern as entries).
- Pass `pluginConfig` through from `sourceNodes` into `createAssetNodes`.
- No tag linking should occur for assets when `enableTags` is false (backward compatibility).

fix(contentful): enable tag support for assets (#34480)


**Title:** GraphQL query parsing and validation performed for every query execution, causing unnecessary slowdowns

**Environment:** Gatsby development/build

**Problem:**
Every time a GraphQL query is executed through the `query` method on the `GraphQLRunner` (which happens for every page’s data query and during development tooling), the raw query string is re-parsed and re-validated against the current schema. This repeats for identical query strings across many pages and in repeated dev server cycles, leading to significant cumulative overhead—especially on sites with large schemas or hundreds of pages.

**Expected behavior:**
- Parsing of a query string into a GraphQL document AST should be cached once per unique query string.
- Validation of that document against a given schema should be cached, so repeated queries with the same string and same schema skip full validation.
- The validation cache must be invalidated whenever the underlying schema changes (e.g., during schema customization or source data changes) to avoid serving stale validation results that could hide real errors.
- To prevent unbounded memory growth during long-running processes (like `gatsby develop`), caches should be pruned periodically or after a batch of queries completes.

**Impact:**
Without these caches, builds and dev server startup are measurably slower than necessary. Caching is expected to reduce repeated work and improve performance in both development and production builds.

perf(gatsby): cache parsing and validation results in graphql-runner (#20477)


Fix `downloadLocal` in `gatsby-source-contentful` so it works with LMDB stores.

The `downloadLocal` option currently relies on the deprecated `localFile___NODE` convention to link asset nodes to their downloaded file nodes. This silently breaks when using the LMDB data layer; the `localFile` field is always null.

Switch to using `createNodeField` to attach the file node ID, then define the `localFile` field on `ContentfulAsset` in `createSchemaCustomization` via a `@link(from: "fields.localFile")` to a `File` node.

Also remove the `onPreBootstrap` hook that manually creates a cache directory – it’s no longer necessary and can cause interference. Ensure the download still works and that a query like `contentfulAsset { localFile { ... } }` returns the expected file node.

fix(gatsby-source-contentful): fix downloadLocal for LMDB (#33715)


In `packages/gatsby/cache-dir/static-entry.js`, the head component reordering (introduced for #22206) moves all `<meta>` elements to the front so that crawlers see them before any large inline styles or scripts. However, the current logic does not guarantee that the original order among the `<meta>` elements themselves is preserved; they can get shuffled. This breaks scenarios where multiple meta tags rely on their document order (for example, a series of Open Graph tags or duplicate meta names).

Fix the reordering so that:
- All `<meta>` elements appear before any non-`<meta>` head components.
- The relative order of the `<meta>` elements is kept exactly as provided.
- The relative order of non-`<meta>` elements is also kept exactly as provided.

fix(gatsby): Keep order in moved meta tags (#36158)


### Bug Report: Interface field filtering fails when some implementing types have zero nodes

**Summary**
When querying an interface that has a custom field resolver (e.g., a `slug` resolved from the node id), filtering on that field returns no results if *any* implementing type contains zero nodes — even if other implementing types do have matching nodes.

**Steps to reproduce**
1. Define a GraphQL interface (e.g., `TestInterface`) with a field that uses a custom resolver (e.g., `slug: { type: 'String', resolve: source => source.id }`).
2. Create multiple object types implementing that interface (e.g., `Test`, `Test2`, `Test3`), all with the same custom `slug` resolver.
3. Populate nodes only for some of the implementing types (e.g., create a node of type `Test3` with a known id, but create zero nodes of type `Test2`).
4. Run a query like:
   ```graphql
   { allTestInterface(filter: { slug: { eq: "id3" } }) { id } }
   ```

**Expected behavior**
The query should return the node from `Test3` because its `slug` resolves to `"id3"`. The presence of an implementing type with no nodes (`Test2`) should not affect the result.

**Actual behavior**
The query returns `null` or an empty result set. The filtering seems to stop processing when it encounters a type with no nodes, causing the resolved values for subsequent implementing types to be skipped.

**Impact**
This breaks any interface-based filtering when at least one implementation type has no nodes in the current data layer. This is a common pattern in dynamic schemas (e.g., generated from CMS content models) where some types may temporarily have no entries. The workaround of removing empty types is fragile and not always possible in multi-source setups.

**Possible cause**
During the internal node resolution phase (`saveResolvedNodes`), the loop over implementing types appears to abort with an early `return` when a type has no nodes, instead of skipping that type and continuing. This prevents the resolution of nodes for the remaining types, so their filtered values are never cached and the query returns nothing.

fix(gatsby): fix error from ts conversion (#26681)


The site’s search form (the Algolia DocSearch integration on www) only initializes when the user clicks or hovers over the input. Keyboard-only users who Tab into the search field never trigger the lazy-loaded DocSearch script, so the search dropdown and suggestions never appear. The input remains inert, making search completely inaccessible via keyboard.

**Steps to reproduce**
1. Navigate to a page that contains the search form (e.g., gatsbyjs.org/docs).
2. Using the keyboard, press Tab repeatedly until focus lands on the search input (identified by id `doc-search` or the visible search icon).
3. Observe that no DocSearch script loads, no dropdown menu is injected, and typing yields no search suggestions.

**Expected behavior**
Whenever a user indicates intent to search—including by focusing the search input via keyboard (Tab)—the DocSearch library should be initialized. After initialization, the input should retain focus so the user can immediately start typing a query without additional manual clicks.

**Impact**
This regression breaks keyboard accessibility for the primary documentation search, affecting users who rely on keyboards, screen readers, or other assistive technology. Search is a critical navigation tool, so the broken tab flow significantly degrades the overall site accessibility and user experience.

fix(www): tweak docsearch to init algolia when tabbed into (#23040)


Simplify WhistleblowerTip to remove the separate `internaltip_id` foreign key and `access_counter` field. Make the `WhistleblowerTip.id` itself double as a reference to `InternalTip`, so a whistleblower tip’s primary key is exactly the corresponding internal tip’s ID. Move the access counter into `InternalTip` as a new field `wb_access_counter` on that model (alongside the existing `wb_last_access`).

Requirements:
- Remove `whistleblowertip.internaltip_id` and `whistleblowertip.access_counter` from the database schema.
- Add `internal_tip.wb_access_counter` column to the schema.
- The `WhistleblowerTip.id` column becomes a foreign key referencing `internaltip(id)`.
- All code that currently reads or writes `wbtip.internaltip_id` or `wbtip.internaltip_id == something` (e.g., query conditions for receiver tips, comments, files) must use `wbtip.id` instead, since the IDs are now the same.
- All code that increments or reads `wbtip.access_counter` must instead use `wbtip.internaltip.wb_access_counter`.
- Provide a migration script that converts existing data: for every old whistleblower tip row, the new whistleblower tip’s id should be set to the old `internaltip_id`, the access counter value should be moved to the corresponding internal tip’s `wb_access_counter`, and the old columns are dropped.
- The serialized form of a whistleblower tip should no longer include the access counter (it is an internal concern now).

Goal: a one-to-one identity link where `WhistleblowerTip` is effectively a token with `receipt_hash` keyed directly off the `InternalTip` record, eliminating an unnecessary indirection.

Simplify implementation of WhistleblowerTip removing a reference and moving data inside InternalTip


```
Fix statistics collection and decouple Logger from GLSettings.

The hourly statistics collection is producing wrong intervals because the
collection start time lives as an instance variable on the job, while the
other hourly counters (event queue, anomaly dict, exceptions, mail counters)
live on the global settings singleton. Resets of these two groups are not
coordinated, so the collection window becomes inconsistent.

Consolidate all hourly state and resets: move the collection start time into
the settings singleton and provide a single method that resets every hourly
counter together, including the start time. Call that method from the
`StatisticsSchedule.operation` handler instead of resetting pieces in
multiple places.

The `Logger` class currently imports and reads `GLSettings.loglevel`, creating
an undesirable coupling. Give `Logger` its own loglevel attribute and a setter;
remove `loglevel` from `GLSettings`. The default log level for the `--loglevel`
command-line option must become hardcoded `"ERROR"` since the settings
attribute no longer exists.

Update any test setup/teardown that used the old job-level reset to use the
new consolidated reset on the settings singleton. Remove the old classmethod
`reset` from `StatisticsSchedule`.
```

Fix statistics collection and improve packaging of related variables


**Request: Expose session expiration in the authentication API and fix timezone inconsistency**

The `/authentication` endpoint currently returns a session ID and user ID on login (POST) and identity verification (GET). Clients need to know when their session will expire so they can proactively refresh or re-authenticate instead of waiting for a 401.

**Acceptance criteria:**

1. Both the POST (login) and GET (check session) responses must include a new field `session_expiration` containing a Unix epoch integer. This value must represent the absolute time at which the session will become invalid.
2. The expiration must be set according to the configured session lifetime for the user’s role (the values already stored in `GLSetting.defaults.lifetimes`). When a session is created, its expiry should be `current_time + role_lifetime`; when a session is verified or refreshed, its expiry should be extended to `now + role_lifetime` so active users are not logged out.
3. The session’s internal state must be updated on each successful authentication check, just as the `refreshdate` is already bumped. This ensures that the expiration reported by the API always reflects the true remaining time.
4. Any utility functions used to calculate future timestamps must be consistent with the rest of the system’s time handling. Specifically, code that previously used `datetime.utcnow()` without adjusting for the local timezone offset must be corrected to match the behavior of the existing `datetime_now()` helper, so that all time arithmetic yields the same frame of reference.

**Non-goals:**
- Changing the session storage mechanism.
- Altering the lifetime configuration semantics.

**Background:**
Currently the client cannot determine session validity without receiving a 401 error. Adding a machine-readable expiration allows better user experience and reduces unnecessary failed requests.

update auth API https://github.com/globaleaks/GlobaLeaks/issues/603


**Ticket: Use IPv4 loopback address 127.0.0.1 for local-only platform access**

### Context
The platform currently treats the hostname `localhost` as the special local-loopback identifier for directing requests to the default tenant and for self-signed TLS certificate generation. In some environments (e.g., containerised deployments), `localhost` may fail to resolve or may resolve to an IPv6 address, which prevents the platform from correctly serving local requests. We need to guarantee the platform always listens on, and can be accessed via, the IPv4 loopback address `127.0.0.1`.

### Required behavior
- Requests arriving with `Host: 127.0.0.1` must be routed to the default tenant (tid=1) in the same way `localhost` previously was.
- The self-signed certificate generated for local use must use `127.0.0.1` as the hostname (common name / SAN) instead of `localhost`.
- The `local_hosts` list in `SettingsClass` ('127.0.0.1', 'localhost') appears unused and should be removed to avoid confusion.
- All internal test fixtures that rely on a local hostname must use `127.0.0.1` so the tests remain in sync with the new default.

### Constraints / edge cases
- The wizard-complete check and the IP-address fallback that also routes to tenant 1 must still work.
- Self-signed certificate generation must also still set the IP SAN to `127.0.0.1` (already the case).
- The change must not break HTTPS configuration workflows that previously set the hostname to `localhost`; those workflows will now use `127.0.0.1`.

Ensure the platform could always listen on 127.0.0.1


Currently, when a context or user picture is uploaded via the file upload API, the `picture` field in the serialized response remains `false` (or empty) and the picture cannot be retrieved through the file download endpoint.

The file upload endpoint accepts an ID that identifies the upload target (e.g., a context UUID, a user UUID, or a special name like `logo`). For context and user pictures, the front-end passes the entity’s UUID as the upload ID. The expected behaviour is that the uploaded file becomes permanently associated with that entity and is later served under that same identifier.

Acceptance criteria:

1. **Upload association**: when a file is uploaded with a UUID matching a context or user ID, the file record must store that UUID in its `name` column so that the file is unambiguously linked to the entity. Uploads using special names (`css`, `favicon`, `logo`, `script`) should continue to work as before.

2. **Picture presence in serialised output**:
   - In admin context serialisation, the `picture` field must return `true` when a `File` record exists whose `name` equals the context ID.
   - In user serialisation, the `picture` field must return `true` when a `File` record exists whose `name` equals the user ID.
   - In public context and receiver listings, the `imgs` dictionary must reflect the presence of a picture by keying on the entity’s ID (the `name` column of the file record), so consumers can construct correct picture URLs.

3. **File download endpoint (`GET /files/<name>`):**
   - The handler must interpret the URL segment as a `name` (which may be a context/user UUID, or a special app-file name like `logo`), not as a raw file ID.
   - It must look up the file by `name` for the current tenant.
   - If no file is found for the current tenant, fall back to tenant **1** (global tenant).
   - The response must include a `Content-Type` header: for known app files (`favicon`, `logo`, `css`, `script`) use the predefined MIME type; for all other files use `application/octet-stream`.

4. **File deletion**: deleting a file by its upload ID (UUID or special name) must remove both the database record and the on-disk file, correctly resolving the file’s internal identifier from the provided name.

No other behaviour (e.g. handling of submission attachments, non-picture files, or special file uploads) should be affected.

Revise files handlers in relation to proper management of contexts and users images


The `new_fieldgroup` helper in the application’s model layer is supposed to create a new `FieldGroup` instance, attach any child field groups passed by ID, and hand back the constructed object to its caller. At the moment the function runs correctly but returns `None`, making it impossible for callers to obtain or further configure the freshly created group.

**Acceptance criteria**

1. After a successful call to `new_fieldgroup`, the caller must receive the `FieldGroup` object that was added to the store, regardless of whether children are specified.
2. Any direct construction or manipulation of `FieldGroup` objects must respect the model’s field types. The `label`, `description`, and `hint` attributes are declared as `Unicode` columns; code that assigns a bare Python dictionary to them is incorrect and must be adjusted—the original intent was to store a stringified dictionary representation. Fix such assignments so they provide a `str` (e.g. the `repr` of a dict) without changing the intended content.
3. The existing test for composing field groups (`TestComposingFields.test_dataset`) contains a few structural issues that prevent it from exercising the intended logic:
   - The test is not decorated with `@inlineCallbacks` despite using asynchronous operations, which leads to the deferred result not being properly handled.
   - A `transact`-based lookup is invoked directly instead of being `yield`-ed, so the test continues before the database query completes.
   - The returned value is stored in a variable named `gen`, but the subsequent assertion checks a variable named `generalities`, causing a `NameError`. Unify the variable naming so the assertion checks the actual result of the lookup.
   - Once the lookup succeeds, verify that the retrieved `FieldGroup` is not `None` (as other field-group tests do), and then proceed to call `add_children` on it.
4. All other existing tests for field groups and fields must continue to pass without modification. No changes to the `FieldGroup` table schema, `FieldGroup` class definition (beyond usage fixes), or `new_field` helper are needed.

**Expected impact**

The API consumer who calls `new_fieldgroup` will be able to capture and use the resulting `FieldGroup`, aligning the function’s contract with the pattern already established by the sibling `new_field` factory. The corrected test will properly exercise the composition of field groups, ensuring that `add_children` can be called on a previously persisted parent group.


**Summary**
A submission can be created with receivers that are not associated with the chosen context, potentially exposing information to unintended recipients.

**Steps to reproduce**
1. Create two contexts (e.g., `ctx-a` and `ctx-b`) and two receivers; assign `recv-1` only to `ctx-a`, and `recv-2` only to `ctx-b`.
2. Submit a new whistleblower tip via the normal submission API, specifying context `ctx-a` but including `recv-2` (the receiver that belongs to `ctx-b`) in the `receivers` list.

**Observed behavior**
The server accepts the submission and associates it with `recv-2`, even though that receiver is not part of the submitted context. The tip becomes visible to a receiver in a completely different context than the one the whistleblower selected.

**Expected behavior**
The server should reject the submission (e.g., with a validation error) when any of the requested receivers does not belong to the given context. A receiver must only be usable within the contexts it is actually assigned to.

**Additional context**
This bypasses the context–receiver association model and could be exploited by a malicious client to force submissions into arbitrary receiver queues. Even a partially forged list (mixing valid and invalid receiver IDs) should be refused entirely. The check should be applied during the submission creation and update paths to prevent an attacker from injecting unauthorized receivers after initial creation.


Title: Drop unrecognized answer keys silently instead of rejecting submissions
Labels: enhancement, security

The answer validation in `db_validate_submission_answers` currently rejects the entire submission whenever the answers dictionary contains extra keys that don’t belong to the questionnaire schema – for example the client’s transient `required_status` flag, field IDs that the schema does not define at that position, or a non‑boolean value stored under a valid checkbox option id.

A modified client can exploit this to smuggle arbitrarily long strings past the per‑field length / format checks: it simply puts the oversized payload under an extra key that the current validation skips but that later consumers (like `index_answers` or the export path) may still traverse. The same modified client can also craft deeply nested answer payloads (a field‑id list recursively referencing an arbitrary id) that would blow the recursion limit when recipients open, export, or redact the report.

### Desired behavior

1. **Prune, don’t reject** – `db_validate_submission_answers` should reduce the `answers` dict *in place*, dropping every key that does not carry recognised answer data for its field, so that only canonical schema‑conformant data is persisted. Submissions, identity‑form fills, and additional‑questionnaire fills must all succeed, carrying only the cleaned‑up data.

2. **What gets dropped**
   - The transient `required_status` flag (an internal UI key that the client sometimes sends).
   - Any key whose shape is not a UUID (non‑field‑id keys inside an entry).
   - A field‑id key nested under a leaf field (e.g. an `inputbox`, `textarea`, `selectbox`, `multichoice`, `date`, `daterange`, `tos`) that has no `children` – previously this was rejected as “unexpected nested field”, now it should be silently removed.
   - Any field‑id key that is not a child of the containing `fieldgroup` according to the schema.
   - Checkbox option keys whose value is not a boolean (`True`/`False`). A non‑boolean value (e.g. a string, even under a valid option id) must be stripped because it does not represent a selection.
   - Any key not recognised as a field id by the top‑level schema (top‑level keys that don’t correspond to any step’s children).

3. **Deep‑nesting DoS resilience** – The traversal must remain driven by the schema: when an answer entry contains a field‑id key that is not a child of the field the entry belongs to, that entry is simply pruned away *without recursing* into the attacker‑controlled depth. In particular, a payload that nests the same fieldgroup id inside itself thousands of times should result in an empty `fieldgroup` entry (or an empty top‑level list) rather than an error, because the forged key is not a recognised child at that level.

4. **Valid answers still validated** – Data under the keys that *are* kept must still run through the existing per‑field constraints (`db_validate_field_entry`). Invalid text lengths, unknown option ids, malformed dates, etc. should still be rejected with `InputValidationError`.

5. **Checkbox helper consistency** – The test helper `fill_random_field_recursively` currently sets checkbox option values to the string `'True'`. Since the real client submits booleans, that helper should store the boolean `True` instead, so that existing tests continue to produce valid answer payloads after the pruning logic is introduced.

### Affected endpoints

- Submission creation (`POST .../submission`).
- Whistleblower identity form (`POST .../wbtip/identity`).
- Additional questionnaire fill (`POST .../wbtip/additional`).

### Implementation notes

- The existing internal `validate_entries` helper is the natural place to perform the key‑based pruning.
- A new utility `parse_ISO8601` extracted from the existing date‑parsing logic may help avoid duplicating the string‑slicing in `db_validate_field_entry`; this is acceptable as a small refactor.
- The change must not alter the scoring, trigger evaluation, or receiver‑set computation – those algorithms already operate on schema‑conformant answers and will simply see the cleaned values.

Prune submission answers to the questionnaire schema instead of rejecting


**Feature request: Recovery keys for user private‑key backup**

## Summary
When a user first sets their password (and tenant encryption is enabled), the system should automatically create a *recovery key* so the user can later regain access to their private key. This requires two new binary columns on the `User` model and a new static method on the `GCE` helper.

## Acceptance criteria

1.  **Database columns**
    Add `crypto_rec_key` (length 80 bytes) and `crypto_bkp_key` (length 72 bytes) to the `User` table. Both default to empty bytes and are non‑nullable.

2.  **Recovery‑key generation**
    Introduce `GCE.generate_recovery_key(prv_key)` that:
    - Takes the raw user private key (as returned by `GCE.generate_keypair`).
    - Generates a random symmetric key.
    - Symmetrically encrypts `prv_key` with that random key to produce the *backup key*.
    - Derives the public key from `prv_key` and asymmetrically encrypts the random symmetric key with that public key to produce the *recovery key*.
    - Returns `(backup_key, recovery_key)`.

3.  **First password‑change trigger**
    In the user‑update flow, when encryption is active and `user_session.cc` is not yet set (i.e. the very first password change that generates the user’s keypair), call the new method and store the returned backup key in `user.crypto_bkp_key` and the recovery key in `user.crypto_rec_key`.

4.  **Recovery invariant**
    The stored values must enable recovery: possessing the user’s private key allows decrypting `crypto_rec_key` to obtain the symmetric key, which then decrypts `crypto_bkp_key` to yield the original private key.

5.  **No regressions**
    Subsequent password changes must not regenerate or alter the recovery key. All existing encryption and user‑update behaviour must remain unchanged.

Implement database support and encryption primitives for Recovery Keys (#2649)


**Title:** Fix duplicate filename detection in whistleblower file upload and add file‑upload support to `TestHandler.request`

**Description:**

When a receiver uploads a file to a whistleblower tip via the `WhistleblowerFileHandler` POST handler, the handler is expected to reject the upload if a file with the same name already exists on that tip. Currently this duplicate check is broken:

1. The existing file names are read directly from the ORM objects in `rtip['wbfiles']` instead of from the serialised representation of the tip. The objects do not expose a `.name` attribute in the expected way, causing an `AttributeError` or a meaningless comparison.
2. The uploaded filename is not properly extracted from the `request.files` structure; the handler accesses the file data incorrectly, so the comparison never receives the correct new filename.

As a result, either the handler crashes or it never detects duplicates, allowing identically named files to be uploaded.

**Expected behaviour:**

- The `can_perform_action` method of `WhistleblowerFileHandler` should obtain the list of existing whistleblower file names from the *serialised* tip data (the same form as returned by the tip detail API).
- The handler must extract the newly uploaded filename from `self.request.files['file'][0]['filename']`. If the name matches an existing file, the handler should return a `FailedSanityCheck` error.
- The mock request factory (`TestHandler.request` in `tests/helpers.py`) should be extended to accept an optional `attached_file` argument. When provided, it should construct the appropriate `files` dictionary on the request object so that file‑uploading handlers can be exercised through the same mock infrastructure. The dictionary should place the file description under the `'file'` key as a list.

**Non‑obvious edge cases & safety constraints:**

- The fix relies on the fact that the tip’s serialised `wbfiles` list contains dictionaries with a `'name'` key. Ensure that the serialisation is performed with the correct language context (available as `self.request.language`).
- The `request.files` may not contain a `'file'` key if no file was sent. The extraction `self.request.files['file'][0]['filename']` will throw a `KeyError` in that case, which is acceptable (it will result in a 500 error rather than a silent bypass). If a more graceful handling is desired, it should fall back to rejecting the request, but the current contract assumes a file is always present when the handler is invoked.
- The mock helper’s `attached_file` parameter must support being `None` (meaning no file), in which case `request.files` should be empty. When a file dict is provided, it should be placed inside `{'file': [attached_file]}` to match the format expected by the handler.

Add mock files for upload to tests/helpers to


Title: Cleaning scheduler fire-and-forgets deletion transactions; finalized-tip tests query wrong marker

Body:
`APSCleaning.operation()` calls the transactional helper `itip_cleaning()` (a `@transact` function) but does **not** `yield` the returned Deferred for the expired-submission and expired-tip loops. This means the scheduled cleaning can return before the actual database deletions have completed, causing expired records to be left behind.

Additionally, the `FinalizedTipCleaning` test cases in `test_simple_cleaning.py` assume that a delivered `InternalTip` is still in the `'finalize'` mark state (`_marker[1]`). After the delivery schedule runs, delivered tips are moved to the `'first'` mark state (`_marker[2]`). The tests are therefore querying by the wrong marker and failing.

**What to do**
1. In `APSCleaning.operation()`, `yield` each `itip_cleaning(...)` call so the job properly waits for the deletion transaction to finish.
2. Fix the `FinalizedTipCleaning` tests so they fetch delivered tips using `InternalTip._marker[2]` (`'first'`) instead of `_marker[1]`.
3. Add a test that forces a delivered tip to be considered expired (e.g. by zeroing its remaining life) and asserts the expiration check passes, to cover the path for forced expiration of delivered tips.

No external API or configuration changes are required; the scheduler’s public promise of “cleaning expired data” must remain the same, but it must actually finish the work before the job tick ends.

completed unitTest in scheduled cleaning


**Title:** Fix localized handling of context fields, text attributes, notification templates, and receiver description

**Background**
Several model attributes have been moved to localized dictionaries (stored as a dict mapping language → value), but not all handlers and serializers were updated to match. This leads to incorrect serialized output, validation failures, and broken notification template rendering.

**What needs to work correctly**

1. **Context fields storage and serialization**
   - Fields are now stored as a dict `{ language: [field_def, …] }` (not a plain list).
   - `admin_serialize_context` must serialize fields using `l10n` like other localized attributes.
   - The fields-validation step must accept the dict shape (`{ lang: [fields...] }`) and iterate over the inner lists to check field types.
   - In `create_context`, the key-generation loop must walk the new nested structure to set `'key'` per field per language.

2. **Context text attributes no longer hardcoded**
   - `receipt_description`, `submission_introduction`, `submission_disclaimer` are stored as localized strings.
   - Currently they are serialized to the string `u'NYI'` regardless of the stored value. They must be included in the `l10n` loop so the true translations appear in the admin response.

3. **Submission field importing**
   - `import_fields` receives the configured fields (from the context) as a language-keyed dict, but treats it as a plain list. It must flatten across languages to collect required/optional keys, or iterate the dict properly.

4. **Notification template substitution**
   - Notification templates (e.g. `tip_template`, `file_template`) are stored as localized dicts (`{ "en": "...", ... }`).
   - `_iterkeywords` currently expects a plain string and will break when given a dict. It should detect when `template` is a dict and extract the template for the default language (`GLSetting.default_language`).

5. **Receiver description in requests**
   - When creating a receiver, the client now sends `description` as a plain string (for the default language), not a full localized dict. The backend must handle this and store it internally as a localized dict.
   - When updating a receiver, the existing logic that re-assigns `request['description']` to the full stored dict (overwriting any partial update) must be removed so that only the incoming language entry is set without wiping out other translations.

**Constraints**
- Do not break backward compatibility for already‑persisted localized dicts.
- Keep the serialization order consistent with the existing API contract where appropriate.
- Error messages from field‑validation should still carry the offending field’s name.

**Affected areas** (for reference)
- Admin context serialization / creation / update
- Submission field import
- Notification plugin mail formatting
- Receiver create / update handlers

fixed some bugs related on fields validation https://github.com/globaleaks/GlobaLeaks/issues/438


The date/time helpers in `globaleaks.utils.utility` have several behavioural issues that need to be fixed so they compose correctly and handle edge cases as expected.

Required changes:

- `datetime_now()` must return only the current time (adjusted for timezone and any debug time‑travel offset). It must *not* add a separate future delta — that belongs to `utc_future_date()`.
- `utc_future_date(seconds, minutes, hours)` must be implemented in terms of `datetime_now()` and `utc_dynamic_date()`. This removes the duplicated debug‑offset logic and makes the delta application consistent.
- `is_expired(check_date, seconds, minutes, hours, day)` must use `datetime_now()` to obtain the current time, so that debug time‑travel applies uniformly across the utility. The expiry comparison is `now > (check_date + lifetime)`. If `check_date` is `None`, return `False`.
- `pretty_date_time(date)` – when `date` is `None` it must return an ISO‑8601 string for the epoch time (UTC 1970‑01‑01T00:00:00/ local equivalent), **not** the literal `'Never'`.
- `very_pretty_date_time(isodate)` – when `isodate` is `None` it must also use the epoch time as the fallback, formatted consistently (e.g. `"Thursday 01 January 1970 01:00"` in the local timezone). The format order should be day name, day, month name, year, hour:minute.
- `acquire_bool(boolvalue)` must **not** raise an assertion error for unrecognised values. Return `True` only for `'true'` (bytes or unicode) and the boolean `True`; return `False` for everything else (including `'false'`, `False`, `None`, and any other input).


Add questionnaire export/import endpoints with deterministic ID replacement.

- **Export**: Allow `GET` on the questionnaire instance resource to return the full nested questionnaire object (steps, fields, etc.). For now, return the internal representation unchanged.
- **Import**: Add a `POST` endpoint that accepts a questionnaire JSON body. The import must:
  1. Use the questionnaire’s `name` field as its stable database identifier (instead of autogenerated UUID).
  2. For each step: set `questionnaire_id` to the questionnaire name, and generate a step ID of the form `{questionnaire_name}_step_{index}`.
  3. For each field that is a direct child of a step: generate a field ID of the form `{parent_step_id}_field_{index}`, set `instance` to `'instance'`, and set `step_id` to the parent step’s ID.
  4. For each field nested inside another field (fieldgroup): generate a field ID of the form `{parent_field_id}_field_{index}`, set `instance` to `'reference'`, and set `fieldgroup_id` to the parent field’s ID.
  5. Apply this replacement recursively to the entire field tree so that every field and step receives a predictable, collision-free ID derived from its position.
- **Out of scope**: Field options and field attributes retain whatever IDs they carry in the imported JSON; they will be addressed later.

The import must accept the output of the export, meaning any existing step/field IDs present in the JSON must be replaced by the deterministic ones described above.

Replace questionnaire, step and field ids for API import


**Title:** Hostname validation allows empty string and bare "onion" / "localhost" values; test coverage missing

**Component:** Admin configuration – hostname management (config handler)

**Description:**
The admin hostname configuration endpoint currently has several gaps in input validation and its test coverage is incomplete:

1. **Empty hostname not rejected** – Providing an empty string as a hostname is silently accepted. This should be treated as invalid input and result in a validation error.
2. **Weak forbidden‑endings check** – The validation only blocks hostnames ending with `".onion"` (with a preceding dot). As a result, the bare value `"onion"` (no dot) can be set without complaint. The same applies to `"localhost"`. Extend the forbidden‑endings logic so that any hostname whose string **ends with** `"onion"` or `"localhost"` is rejected, regardless of whether there is a leading dot.
3. **Verify‑hostname error path not exercised** – When `verify_hostname` is called with a domain that cannot be resolved (e.g. a non‑existent public domain), the endpoint returns an `ExternalResourceError`. This behaviour must be covered by tests.
4. **Missing test cases for set_hostname** – Add tests that attempt to set hostname to the following prohibited values and assert that each raises an appropriate `InputValidationError`:
   - Empty string
   - `"onion"`
   - `"localhost"`
   - `"antani.onion"`
   - `"antani.localhost"`

Also ensure that a valid hostname (e.g. `"antani.gov"`) can still be set successfully through `set_hostname`.

**Acceptance criteria:**
- Empty hostname input causes `InputValidationError`.
- Hostnames ending with `"onion"` or `"localhost"` (with or without leading dot) are rejected with a validation error.
- The test suite covers the `ExternalResourceError` path from `verify_hostname` and the new `set_hostname` rejection scenarios.
- Existing valid‑hostname flows continue to work unchanged.

Improve unit testing of handler admin/config.py


The handler execution model and API have been updated. Update the test suite to align with these changes:

- Handlers now return the response body directly (e.g., a string or a list) from their `get`/`post`/etc. methods instead of writing to the request object. Adjust tests that previously inspected HTTP status codes to instead examine the return value of the handler method.
- The anomaly collection endpoint no longer supports `DELETE`. Remove any test that exercises that behavior.
- The `decorate_method` function uses a class-level flag to avoid re-decorating methods. Ensure the test helper that calls `decorate_method` properly sets and checks this flag (e.g., `decorated` attribute on the handler class) so methods are decorated only once.
- Cache tests now need an explicit import of the `public` handler module because the import path changed.

Fix some unit tests to conform to new API


**Title:** `ObjectDict` raises `AttributeError` for missing keys, causing crashes in tenant cache refresh and notification generation

**Description:**
`ObjectDict` is used extensively in the application state (e.g., `State.tenant_cache` items) and is expected to support safe attribute access for optional configuration fields. Currently, `ObjectDict.__getattr__` raises `AttributeError` when accessing a key that does not exist. This design causes hard crashes in at least two production flows:

1. **Tenant cache refresh** – during `db_refresh_memory_variables` the code checks `State.tenant_cache[tid].hostname != ''` and `State.tenant_cache[tid].onionservice != ''`. If a tenant has no configured hostname or onion service, the key is absent from the underlying dict and a `KeyError` (wrapped as `AttributeError`) propagates, aborting the refresh and potentially preventing the service from starting or reconfigured correctly.
2. **Notification generation** – `MailGenerator.generate` iterates over `self.state.tenant_cache` and accesses `cache_item.notification.disable_receiver_notification_emails`. When a tenant’s cache object does not contain a `notification` entry, a chained `AttributeError` is raised and the whole notification job fails.

The core problem is that callers need to be able to probe for optional attributes without wrapping every access in `try`/`except` or using `get` repeatedly, but the current behavior makes such probes impossible.

**Expected behavior**
`ObjectDict.__getattr__` should return `None` instead of raising `AttributeError` when a key is missing. This makes missing values falsy, which aligns with how callers already intend to use these attributes (they treat absent values as “not present” or “disabled”).

**Impact & required adjustments**
Once the base behavior is changed, existing call sites must be updated to handle the new semantics safely:

- In `db_refresh_memory_variables`, the explicit empty‑string comparisons (`!= ''`) must be replaced with truthiness checks (`if hostname:`, `if onionservice:`) so that `None` is not passed to `.encode()`, which would cause a new `AttributeError`.
- In `MailGenerator.generate`, access to `cache_item.notification.disable_receiver_notification_emails` must guard against `cache_item.notification` being `None` before dereferencing its attribute.

All other usages of `ObjectDict` should be audited to ensure they are safe with the new `None`‑returning semantics (most callers already use truthiness tests or `.get()` and should be unaffected).


**Title:** Rate-limit job monitor alerts and fix submission token lifecycle

**Background**
- The job monitor (`GLJobsMonitor`) currently sends an exception email every time it detects a job that has exceeded its configured monitor time. If a job remains slow, this results in repeated duplicate emails, flooding operators.
- Submission creation (`db_create_submission`) deletes the submission token inside the database transaction, which can cause problems if the transaction is retried or if downstream code relies on the token still existing. It also accesses the token’s `uploaded_files` attribute directly, coupling it to the token object.

**Requirements**

1. **Job monitor alert rate‑limiting**
   - When a job exceeds its monitoring period, send at most one alert email per monitoring period for that job.
   - If the job remains in a long‑running state, do not send another alert until a full monitor period has elapsed since the last alert was triggered.
   - Accumulate alert messages for all currently problematic jobs and send them in a single combined email (instead of one email per job).

2. **Submission token handling**
   - Move token deletion out of the database transaction (`db_create_submission`) and into the caller after the submission has been successfully persisted.
   - Pass the list of uploaded files as a separate argument rather than pulling it from the token object, so that the submission creation path does not depend on the token instance at that point.

3. **Minor corrections**
   - Fix the grammar in the job failure message (`"is died"` → `"died"`).
   - Fix the spelling of `"cicles"` → `"cycles"` in a comment.
   - Fix the misspelled function name `extact_exception_traceback_and_send_email` → `extract_exception_traceback_and_send_email`.

**Affected areas**
- Job monitor logic in the `GLJob` base class and the `GLJobsMonitor` operation.
- Submission handler and support functions (`create_submission` wrapper, `db_create_submission`, and the endpoint in `SubmissionInstance.put`).
- Job schedule subclasses that override the monitoring period (`DeliverySchedule`, `NotificationSchedule`).

**Non‑obvious constraints**
- The rate‑limiting state must persist across monitor cycles even if the job is still running, so that a fresh alert can be sent only after the job finishes (or after another full period).
- Token deletion after submission must happen after the transactional write completes, not within it, to avoid premature removal in retry scenarios.


Finish implementation of feature #1375: expose the questionnaire hash in the receiver tip list response and ensure deleting a tip properly cleans up the corresponding archived schema when no other tip references it.


Improve the security of how the operator escrow key is handled in user sessions.

Currently the session's `ek` field holds the encrypted escrow private key, which is serialized (still encrypted) and transmitted to the client. Instead, the session should only store a boolean flag indicating whether the operator has escrow access. When an operation needs the escrow key it must be loaded on demand from the operator's `crypto_escrow_prv_key` field in the database and decrypted using the session's `cc` key.

Acceptance criteria:
- After login (`login`) and after a password reset (`validate_password_reset`), `UserSession.ek` is a boolean (`True` if the user has a non-empty `crypto_escrow_prv_key`, `False` otherwise).
- `Sessions.new` accepts a boolean for `ek`. All callers pass a boolean derived from `user.crypto_escrow_prv_key != ''`.
- Any code path that previously read the escrow key from `user_session.ek` and decrypted it with `user_session.cc` (e.g. inside `toggle_user_escrow`, `db_set_user_password`, `set_tmp_key`) now loads the key from the database and decrypts it. The key is never stored in the session itself.
- `toggle_escrow` updates the session's `ek` flag to `True` when escrow is enabled for the operator's own tenant.
- `OperatorAuthSwitchHandler.get` creates the whistleblower session with an escrow flag of `False`.
- The session `encrypt` and `decrypt` methods no longer include `ek` in the encrypted payload; only `cc` is encrypted.

At the same time, prevent password-reset token abuse by adding a rate limit inside `generate_password_reset_token_by_username_or_mail`. When `State.settings.enable_rate_limiting` is true, check `State.RateLimit.check(b"password_resets_per_hour_per_user:" + user.id.encode(), 5, 3600)`; if the limit is exceeded for a user, skip generating a token for that user.

Store only an escrow-access flag in the session and load the escrow key on demand


```markdown
**Title:** Bind sessions to tenants to prevent cross-tenant session reuse

**Background**
GlobaLeaks supports multiple tenants, each with its own isolated configuration, users, and data. User sessions are currently stored in a global in‑memory store (`Sessions`) without any tenant scoping. As a result, a session cookie (the `x-session` header) obtained on one tenant can be used to access resources on a different tenant with the same user ID, leading to potential data leakage and privilege escalation.

**Requested change**
- Each session must be bound to the tenant for which it was issued.
- When a request is authenticated via `x-session`, the session’s tenant must match the tenant identified in the current request. If they do not match, the session must be treated as invalid (no authenticated user).
- All session creation paths must be updated to record the tenant ID at session creation time:
  * Login via username/password (`AuthenticationHandler`)
  * Login via whistleblower receipt (`ReceiptAuthHandler`)
  * The pre‑built API‑token session that is created during tenant cache refresh (currently in `db_refresh_tenant_cache`) must also be bound to a tenant. As the admin API token is always used on the root tenant, bind it to tenant ID `1`.
- The centralized session validation in `BaseHandler.get_current_user` must enforce the tenant check.

**Acceptance criteria / expected behavior**
- A session created on tenant A cannot be used to authenticate requests to tenant B.
- All existing session‑based flows (admin API token, user login, whistleblower receipt login) continue to work correctly on the tenant where the session was created.
- The change must not introduce any degradation in session creation performance or storage.

**Non‑goals**
- No changes to session expiration, revocation, or storage backend (remain in‑memory `TempDict`‑based).
- No changes to the `Sessions` global itself, other than the objects it holds carrying a tenant ID.

**Relevant internal references (for discoverability)**
- Session class and `new_session()` helper
- `BaseHandler.get_current_user()` where the `x-session` header is validated
- `AuthenticationHandler.post()` and `ReceiptAuthHandler.post()`
- `db_refresh_tenant_cache()` that creates the admin API token session
- `XTIDX = 1` as the root tenant constant
```


```
The ZIP archive downloaded by receivers currently includes a hardcoded COLLECTION_INFO.txt with file names, sizes, and SHA256 hashes. The admin should be able to customize this info file via the existing notification templating system, using the `zip_description` template from the notification settings. The ZIP handler should render that template with all relevant placeholders (including `%FilesNumber%`, `%TotalSize%`, `%FileList%`, and the usual tip/context/receiver/node placeholders), and include the result in the archive in place of the hardcoded text. The ETag header for the ZIP response should also change to reflect the collection info content rather than a flattened SHA256 of filenames. Ensure that the new keyword resolution logic is shared with the email notification rendering so that both paths stay consistent.
```

added Zip collection template usage and generation


**Title:** All file-bound log writes should pass through `escape_string`, not just those from `Logger.print`

**Description:**
The current logging setup applies `log_remove_escapes` only inside `Logger.print`. However, logs emitted by Twisted’s log infrastructure and written directly to a file using `LogFile` bypass this sanitization. This means that control characters and other potentially dangerous or confusing byte sequences end up in the on-disk log file, which could be used for log injection or simply produce garbled output in log viewers.

Additionally, the name `log_remove_escapes` is misleading: the function escapes strings (produces a repr-like representation) rather than removing escape sequences.

**Expected Behavior:**
1. Rename the escaping function from `log_remove_escapes` to `escape_string` so that its purpose is clear.
2. Ensure that **any** data written to a log file — regardless of the code path — is routed through `escape_string` before it hits the file. The invariant should be: every line persisted in the log file has been sanitized by this function.

**Acceptance Criteria:**
- The escaping function is named `escape_string` and applied to all file-level writes performed by the log infrastructure.
- The fallback error message inside the escaping function is updated to reflect the new name (e.g., no longer contains “log_remove_escapes”).
- Existing log behavior for `Logger.print` remains unchanged, except for the rename of the helper.

Revise implementation of escape_string function


**Title:** File upload size check truncates to whole megabytes, allowing oversized files

**Description:**

The `maximum_filesize` tenant configuration specifies the maximum allowed upload size in MiB (megabytes, as an integer). However, the upload handler enforces this limit by dividing the incoming chunk size, the total file size, and the accumulated size by `1024*1024` using *integer division* (floor division) before comparing with the configured value. This effectively truncates sizes to whole MiB, so a file whose byte size is larger than `maximum_filesize` MiB but smaller than `(maximum_filesize + 1)` MiB minus one byte passes the check. For example, with `maximum_filesize = 1` (1 MiB), a file of 2 MiB – 1 byte is accepted instead of being rejected.

A similar issue exists in the cumulative size check during chunked uploads: only the previously accumulated size (`f.size`) is floor‑divided, and the new chunk size is not considered before the comparison, which can allow the total to exceed the limit.

**Steps to reproduce:**
1. Set a tenant’s `maximum_filesize` to 1 (MiB).
2. Upload a file that is 2 MiB – 1 byte in total size.
3. The upload succeeds and the `FileTooBig` error is not raised.

**Expected behavior:**
Any upload whose chunk size, total file size, or accumulated size (including the current chunk) exceeds the exact byte limit corresponding to `maximum_filesize` should be rejected with a `FileTooBig` error. The comparison must be performed against the byte‑exact limit (i.e., `maximum_filesize` × 1024 × 1024 bytes), not against the floor‑divided MiB values.

**Impact:**
This is a security‑relevant bug: it allows whistleblowers or malicious actors to upload files substantially larger than the configured limit, potentially exhausting disk space, affecting system performance, or bypassing administrative policies.

Compare the file upload size limit in bytes


The `Notification` job processes reminder emails on every tick (every 10 seconds). Reminders should instead be evaluated and sent at most once per day. All other notification processing (immediate tip notifications) must still run on every tick. On process start, the first tick should be allowed to evaluate reminders so that reminders are not delayed by a full day.


The `WorkManager.run()` method returns a `DeferredList` that fires when the initial batch of jobs completes. However, if a job fails and is rescheduled (up to `retries`), the caller's deferred fires early, before the rescheduled work is actually done. This makes it impossible to know when all work has truly finished.

Change `run()` to return a deferred that fires only when the work queue is empty after all jobs (including retries) have either succeeded or exhausted their retries. A job that fails and is rescheduled should not be considered done; it must stay in the queue, and the final completion deferred must not fire until the queue has been fully drained.

Also clean up the debugging `print` statements peppered through `_failed`, `_timedOut`, `_done`, and elsewhere. Fix the labeling in `showState` so that the Work Queue and Failed Queue sections are printed correctly (currently the headers are swapped). Adjust the `retries` delay configuration to something reasonable for development and testing — currently it's `[3, 6, 10]`, which is unnecessarily long for immediate feedback. A single short retry, like `[1]`, is more practical while iterating.

Note: `_done` should trigger the new completion deferred after checking that `workQueue` is empty.

Polish scheduler and implement basic unittests


**Bug: Update notification on admin home page fails to display the latest version and may misidentify the newest version due to incorrect ordering**

### Observed behavior
- On the admin home page, when a newer GlobaLeaks version is available, the warning banner shows the text *“An update is available:”* but the version number is missing (or shows an empty value).
- In some deployments, the system falsely reports that no update is available, or it identifies an older version as the “latest” – for example, preferring `2.0.666` over `2.0.1337` even though the latter is a higher release.

### Expected behavior
1. The admin home page should display the **latest available version** next to the update warning (e.g., *“An update is available: 2.0.1337”*).
2. The latest version must be determined by **semantic version comparison** (numeric ordering of major.minor.patch components), not by lexicographic string sorting.
3. The update notification should only appear when a genuinely newer version exists on the repository.

### Context / possible root causes
- The GlobaLeaks backend periodically fetches the Debian `Packages` file from the repository and extracts version entries. The logic that selects the highest version seems to rely on string ordering, which fails for numeric subversions like `666` vs `1337`.
- The Angular frontend service that holds the latest version uses a property name that does not match the template, preventing the value from being displayed. This might also cause the update-warning visibility to behave incorrectly.

### Steps to reproduce (approximate)
1. Deploy an instance with the built-in update check enabled (and with outgoing connections allowed).
2. Arrange for the repository to contain multiple versions where a “smaller” numeric patch segment appears later in lexical order (e.g., `2.0.666` and `2.0.1337`).
3. Observe the admin **Changelog** tab – the update warning is either missing or shows no version number.

Fix visualization and ordering of latest version


We need to change the shape of field attributes across the application from a list of objects to a dictionary keyed by the attribute's `name`. This affects the admin field create/update API, the public questionnaire schema served to whistleblowers, and the archived questionnaire schema stored with submissions.

Currently field attributes are represented as a list:
```json
"attrs": [
  { "name": "some_attr", "type": "unicode", "value": "hello" }
]
```
This forces consumers to scan the list to find a particular attribute. The desired representation is:
```json
"attrs": {
  "some_attr": { "type": "unicode", "value": "hello" }
}
```
The key must be the attribute's `name` string, and the value must be an object containing at least `type` and `value`. Localized attributes must still be handled correctly: when stored/edited through the admin interface their `value` is a multi-language dictionary; when served to unauthenticated clients (e.g., the whistleblower questionnaire) they must be resolved to a single language by picking the appropriate translation.

Every producer and consumer of field attributes must be updated to use the new shape:
- The admin handlers that persist field attributes (create/update) must accept and store a dict keyed by name.
- The public-facing serialization of fields (the node handler that builds the questionnaire schema for whistleblowers) must emit attrs as a dict instead of a list, with localized values resolved to the requested language.
- The archived schema used during submission processing (stored in `ArchivedSchema` and read back when displaying submitted content) must be able to traverse the dict and resolve localized values to the correct language for display.
- Field validation rules that enforce the shape of incoming admin requests should accept `dict` instead of `list` for the `attrs` field.

Edge cases:
- When attrs are empty the expected representation is an empty dict `{}`, not an empty list.
- Localized `value` fields must continue to work seamlessly. Admin-side code should keep the multi-language dict intact; public/archived code must extract the single-language string using the existing localization helpers.
- The commit message says "Index field attributes by attribute.name", so the attribute's `name` property must be present and become the key in the dictionary. The name must not be duplicated inside the value object when serialized; the key replaces it.


```
Title: Expand templating unit tests to cover all notification event types and templates

Our templating tests currently only exercise the encrypted tip notification path with a small set of hardcoded template names. We need comprehensive coverage that loops over all template files used in notifications (including mail titles, message, comment, file, and zip collection templates) and verifies keyword substitution for each corresponding event type.

What to do:
- Load each actual template file (from `client/app/data/txt`) and append all keywords that should appear in it – generic ones (%NodeName%, %HiddenService%, %PublicSite%, %ReceiverName%, %ContextName%) plus the specific keywords for that template’s event type (tip, file, comment, message, etc.).
- For each template, construct an appropriate `Event` object with the correct `type` and `trigger` fields, and with `trigger_info`/`trigger_parent` populated as a mock dict containing the fields needed by the keyword converters (e.g., file templates need `name`, `size`, `content_type`; comment templates need the comment source, etc.). The same mock can be used for both `trigger_info` and `trigger_parent` for simplicity.
- Call `Templating().format_template()` with the assembled template and event, then verify that the output contains the expected context, node, and receiver values – confirming the keywords were replaced and not left as raw `%KEYWORD%` literals.

While working on this, please also check the production keyword converter classes. At minimum, add an explicit assertion in the `TipKeyword` (and related) constructors that the provided trigger/event info is a `dict`, so we get clear failures if a test or production caller passes the wrong shape. Other minor cleanups in `format_template` are fine if they improve readability or error messages, but keep them minimal.

The goal is to ensure we can confidently change the templating machinery without fear of breaking notification formatting for event types we never exercise.
```

changed approach and loop inside of templating UT #1024


**Bug: Test suite and production code not aligned with latest field/id serialization and missing robustness checks**

**Summary**
After recent API changes, several areas are broken: context/receiver serialization now uses a generic `id` key instead of the old `context_id`/`receiver_id`, the static `sample_context_fields` fixture is gone and replaced by a dynamic default-field generator, the temporary file utility may fail silently on non-writable directories, and some test setup lacks robust directory creation. This leads to multiple failures in the test suite and potential cryptic production errors.

**Observed problems**
1. **Context and receiver objects are now serialized with an `id` field** (likely returned by the admin API). Many test helpers still read or set `context_id` and `receiver_id`, causing `KeyError` or wrong data when building submission requests or updating contexts.
2. **The old static `sample_context_fields` list is no longer available; instead there is a function `default_context_fields()` that dynamically builds default fields using the `Fields` class.** Many tests still reference the old variable and break because the field structure is missing or wrong.
3. **`GLSecureTemporaryFile` in `globaleaks/security.py` can fail at runtime if the key directory is not writable.** The constructor tries to open a key file without checking permissions, resulting in hard-to-diagnose IO errors.
4. **The test setup in `TestGL._setUp` does not check for the `ramdisk` directory and uses `os.path.exists` (which returns `True` for a file) instead of `os.path.isdir`.** Missing directories cause tests to fail confusingly; errors are not logged.
5. **A leftover `import pdb; pdb.set_trace()` is present in `TestGL.fill_data`**, causing tests to hang when run.
6. **Assumptions about the tip-preview structure** (e.g., that `preview[0]` always has keys `label` and `text`) are no longer valid after the dynamic field changes, leading to assertion failures.

**Expected behavior**
- All code that reads serialized context/receiver dicts should use the `id` key (matching the current API output).
- Every place that previously used `sample_context_fields` should instead call `default_context_fields()` to obtain the correct, up-to-date field definitions.
- `GLSecureTemporaryFile.__init__` must check that the key directory is writable before opening the key file and, if not, log a clear error and raise an `OSError`.
- Test directory creation in `_setUp` must ensure all required directories (`files`, `files/submission`, `ramdisk`) exist, using `os.path.isdir`, and raise a meaningful error if creation fails.
- The debug `pdb.set_trace()` must be removed.
- Assertions about the exact shape of `preview[0]` in tip lists should be removed or weakened because the content depends on which fields a context defines.

aligned other UT - still remain issue with ramdisk


**Title:** Improve authentication debug logging and standardize login failure returns

**Background**
Debug logs in the authentication path currently lack important context and the lower-level login helpers signal errors inconsistently, making both debugging and caller-side error handling harder than necessary.

**What needs to change**

1. **Add handler information to authentication debug logs**
   The `authenticated` decorator logs success and session expiry for a given role, but doesn’t include *which handler* is being accessed. Each log line should also include the handler (e.g., `method_handler`) so operators can understand which endpoint triggered the message.

2. **Standardize login helper error returns**
   The three login helpers – the one for whistleblowers, the one for receivers, and the one for the admin – use inconsistent failure signals:
   - Whistleblower login currently raises an `InvalidAuthRequest` exception on an invalid receipt.
   - Receiver login returns `None` in some failure paths.
   - Admin login already returns `False` on failure.

   **All three helpers must return `False` on any normal authentication failure** (bad receipt, unknown username, wrong password, expired credential, etc.). They should not raise exceptions for expected authentication failures, nor return `None`. The caller in `AuthenticationHandler.post` should be able to check a uniform boolean result and decide how to respond.

3. **Make log messages more descriptive**
   - Whistleblower receipt failures should log something like `"Whistleblower: Invalid receipt"` rather than a generic `"Fail auth"`.
   - Success should log `"Whistleblower: Valid receipt"`.
   - Admin failure/success messages should clearly distinguish password validation (e.g., `"Admin login: Invalid password"` / `"Admin login: Valid password"`).

**Acceptance criteria**
- Debug log lines from the `authenticated` decorator include the name of the handler being called.
- `login_wb`, `login_receiver`, and `login_admin` all return `False` (and only `False`) on authentication failure.
- An invalid receipt for a whistleblower no longer raises `InvalidAuthRequest`; it returns `False`.
- Updated log messages are present and clearly identify the failure/success scenario.
- Existing tests that exercise these paths must pass after the changes (the test for bad whistleblower receipts must be updated to match the new return semantics).

verifiy debug logs https://github.com/globaleaks/GlobaLeaks/issues/318


## Request: admin/field consistency and serialization fixes

**Background**
The admin field endpoints (`/admin/fields` and `/admin/field/<id>`) are responsible for CRUD operations on form fields. Their implementation currently deviates from patterns used in other admin handlers in several ways, and the serialized field representation has two known correctness gaps.

**Acceptance criteria**

1. **`options` must always be a dictionary**
   When a field is serialized (returned via any of the GET/POST/PUT endpoints), the `options` key shall **always** be a dictionary. If a field’s stored `options` value is `None` or otherwise falsy, the serialized result must return an empty dictionary `{}`. Clients already depend on `options` being a dict, and a null value causes errors.

2. **`children` must always be present in the serialized representation**
   The serialized field object must **always** include a `children` key containing a list of child field IDs. Currently `children` is omitted when a field has no children; it must instead be present as an empty list `[]` in that case. This ensures clients can unconditionally iterate over the list.

3. **Consistent naming for store transaction functions**
   The transaction helper functions used by the field handler currently have a `transact_` prefix (e.g., `transact_create_field`). All other admin handlers use unprefixed names for their equivalent helpers (e.g., `create_field`). Remove the `transact_` prefix from field transaction functions so that naming is uniform across the admin area.

4. **Use standard model helpers for field retrieval and deletion**
   The `Field` model provides `Field.get()` and `Field.delete()` static methods, which are the canonical way to locate and remove field records. The admin field handler should use these helpers instead of performing manual `store.find(Field, ...)` queries and `store.remove`. `Field.get`/`Field.delete` correctly raise `FieldIdNotFound` when an identifier is missing, aligning with error behavior used by other model classes.

5. **No redundant intermediate functions**
   The implementation currently defines separate `db_create_field`/`db_update_field` wrappers that add no value; the transaction functions should call `Field.new`/`Field.update` directly and return the serialized result.

**Non-goals**
- Changing the request/response formats beyond the two serialization fixes above.
- Modifying any other handler.

**Verification**
After the changes, every admin field endpoint (list, create, get, update, delete) should return consistent serialization with `options` as a dict and `children` as a list, and the code should use the `Field` model helpers plus the conventional unprefixed transaction function names.


Subject: Migration crash on NULL `exception_email_pgp_key_expiration` and unnecessary memory‑variable refresh in test helpers

When migrating an installation from database version 33 to 34, the migration script crashes if the old notification record has a NULL value for the column `exception_email_pgp_key_expiration`. The migration tries to call `iso_strf_time(None)`, but the field is now unused and should tolerate a missing value.

Separately, `Config.set_v()` does not handle a `None` argument gracefully: passing `None` eventually causes a type‑check failure because the value does not match the descriptor’s type. An explicit `None` should be replaced with the default zero‑value for the type (empty string for Unicode, 0 for Int, False for Bool) to avoid runtime errors when a config variable has no stored value.

In the test helpers, the function `update_node_setting` automatically calls `db_refresh_memory_variables` after every setting change. This is often unnecessary and makes it awkward to batch several updates before refreshing the in‑memory state. Please remove that implicit refresh so that tests can control when the memory copy is updated.

As a consequence of removing the automatic refresh, the existing tests that rely on the side effect must be adjusted to keep working. For example, the Ahmia description handler tests and the robots.txt handler tests expected the in‑memory variables to be refreshed after calling `update_node_setting`. Those tests should instead either set `GLSettings.memory_copy` directly or explicitly refresh the memory after the relevant database updates. The `TestGL.setUp` method should also be updated to explicitly refresh memory after setting `allow_unencrypted` so that the plaintext/mixed‑encryption scenarios behave correctly.


**Improve error handling in the submission flow**

The submission finalize handler currently catches a broad `Exception` when calling `create_tips`, prints the error, and does not consistently return a proper error status or log useful details. The model layer similarly catches generic exceptions and prints messages without cleaning up the Storm store (rollback/close) or raising distinct error types.

The submission model should raise specific, typed exceptions for the failures it can detect:

- submission not found
- collision/duplicate (`NotOneError`)
- no receivers selected (context not chosen)
- context not found or collision when looking up the selected context

These exceptions should extend a common model error base class so the handler can catch them uniformly. In every error path inside the model, ensure the store is rolled back and closed before re-raising.

In the handler, catch only the model-level error base class, set status 412, log the error properly (e.g., using `log.err`), and finish the request without proceeding to normal success.

Get some decent error handling in submission


When processing whistleblower submissions, the server requires an `answer_order` key inside each `wb_fields` entry. The client no longer sends that key (only `value`), so all submissions fail validation. Remove the `answer_order` requirement from field validation and accept the standalone `value`.

Additionally, the submission creation and update handlers receive a `language` parameter indicating the request’s locale, but they ignore it when fetching the context’s field definitions – they always query using the default language. As a result, required‑field checks and field serialisation operate on the wrong language. Ensure the language is forwarded to the context‑field lookup.

Finally, the recursive helper that gathers context fields (used by both the admin API and the submission path) drops the `language` argument when recursing into child fields, causing child fields to be serialised in English regardless of the requested language. Fix the recursive call so the language is propagated correctly throughout the field tree.

twin commit of https://github.com/globaleaks/GLClient/commit/5634f88693a4e8c1a15b031679b05c1560b70868


We need to ensure the whistleblower submission flow handles file uploads containing arbitrary Unicode characters in filenames and body content without crashing. Several places in the backend implicitly assume byte strings or ASCII-safe data, which leads to internal errors when non-ASCII text is encountered.

**Acceptance criteria**
- File uploads with non-ASCII filenames (e.g., Greek, Cyrillic, symbols) are accepted, stored on disk, and linked correctly to the internal tip. The relationship between the original filename and the stored file must remain reliable regardless of characters.
- File bodies containing non-ASCII text are correctly written to disk via the existing `fdesc.writeToFD` helper. Encodings must be explicit so that raw Unicode strings do not reach low-level write operations.
- During file registration and storage, any logging that involves filenames or file content (debug/info/error messages) must not raise `UnicodeEncodeError` or print garbled output. Use safe encoding (UTF-8) when preparing log strings.
- Validation of incoming data (e.g., integer fields in JSON messages) should not break when the value is a Unicode string that represents a valid integer. The existing `validate_python_type` logic should be lenient enough to convert such values without treating them as invalid.
- The `is_whistleblower` and `is_receiver` property methods should be simplified and not rely on verbose conditionals, but behave identically.
- Email address extraction (`acquire_mail_address`) must handle Unicode email strings without crashing.
- Logging utilities must be resilient to Unicode strings, converting them to UTF-8 bytes for console output, so that any internal log statement with non-ASCII data does not crash the process.
- Static file dumping (`dump_static_files`) must encode file bodies to UTF-8 before passing them to `writeToFD`.
- Debug logging of HTTP request/response data (headers, URL, body, write buffer) must avoid direct `str()` calls that can fail with Unicode; use safe conversion or let the logging framework handle encoding.


Add a `reset_css` boolean to the node update API.
When `true`, delete the custom stylesheet file (`custom_stylesheet.css`) from the static files directory.
Also ensure that the `/custom_stylesheet.css` endpoint never returns 404: if the file is missing, respond with HTTP 200 and an empty body.


**Bug Report: Receiver brute-force lockout never activates (failed_login counter not persisted)**

When a receiver login attempt uses an incorrect password, the `failed_login` counter on the receiver’s record does not increase (it stays at 0). As a result, repeated incorrect attempts can continue indefinitely, and the account lockout mechanism (controlled by the alarm threshold) never triggers. This is a critical security flaw—it allows unrestricted brute‑force attacks against receiver credentials.

**Expected behavior**
- Each failed receiver login attempt increments the receiver’s `failed_login` counter, and the increment is durable (persisted to the database).
- Once the counter reaches the configured alarm threshold, the account should be locked out (or at least the alarm should be raised reliably).
- Successful authentication resets the counter to 0.

**Current behavior**
- The `login_receiver` function raises an exception on certain failure paths. Because the transaction is rolled back on exception, the in‑memory increment of `failed_login` is lost and never committed. The counter therefore stays 0, and lockout never occurs.

**Acceptance criteria**
1. After *N* consecutive invalid password submissions for a receiver, the persisted `failed_login` counter must equal *N*.
2. The receiver login flow must still return an appropriate failure indicator (so the caller can react accordingly), but must not raise an exception that prevents the counter update from being committed.
3. The successful login path must continue to reset the counter and return the receiver’s user ID as before.

**Impact**
This is a security‑sensitive issue. Any deployment is currently vulnerable to unlimited receiver password guessing. A fix should be prioritised and must not introduce regressions in the normal login/logout flow.

solved critical security flaw in Receiver auth


**Title:** Prevent plaintext fallback when file encryption fails; mark file as unavailable

**Summary:**
When a submission file cannot be encrypted for a specific receiver, the delivery process currently falls back to delivering the plaintext file. This undermines confidentiality guarantees. Instead, encryption failures should mark the file as unavailable, provide no download link, and continue delivery for other receivers.

**Acceptance criteria:**

1. **Encryption failure handling during delivery**
   - If GPG encryption of a file for a receiver fails (e.g., invalid key, missing public key, encryption error), the system does **not** keep or deliver the plaintext file.
   - The receiver file record is instead given the status `unavailable` (the existing fourth status in the `ReceiverFile` status list).
   - The delivery job continues normally for other receivers whose files are successfully encrypted or are handled without encryption.

2. **Behaviour when listing files for a receiver tip**
   - The file list response must include files with status `unavailable`.
   - For an unavailable file:
     - The `href` field is empty (no download token/link).
     - The `name` field uses the original submission filename (do not append `.pgp`).
     - The `size` field reflects the original file size (not a zero or placeholder).
     - The `sha2sum`, `content_type`, and `creation_date` fields come from the original internal file metadata.
     - A new `status` field is present, set to `"unavailable"`.
   - For successfully encrypted files, the `status` field is also included (e.g., `"encrypted"` or `"reference"`) so that clients can consistently rely on this field.

3. **No regression in existing encryption/delivery paths**
   - Successfully encrypted files continue to be delivered with the `.pgp` suffix, correct download link, and appropriate metadata.
   - Files that do not need encryption (reference copies) remain unchanged and include their download links.
   - The delivery job remains idempotent and does not abort the entire relay when one receiver’s encryption fails.

**Security constraint:**
Under no circumstances should a file that was meant to be encrypted be silently delivered in plaintext. Any unrecoverable encryption error must result in an unavailable file that cannot be downloaded.


After completing the initial setup wizard for a new GlobaLeaks instance, activation email links are broken: the hostname part is either missing (rendered as `[UNDEFINED]`) or the protocol is wrong when the instance is accessed by IP address. This prevents users from completing activation.

**Steps to reproduce:**
1. Access a fresh GlobaLeaks instance and go through the setup wizard.
2. Create the initial user; an activation email is sent.
3. The activation URL in the email either shows `[UNDEFINED]` as the host, or uses `https://` even when the setup was performed via a raw IP address (common in development/testing).

**Expected behavior:**
- The wizard should capture the hostname from the request that was used to reach the setup interface and persist it in the instance configuration.
- Activation links and other generated URLs should use this hostname.
- When the hostname is an IP address, the generated URL should use `http://` (since TLS certificates are not practical for IPs); for domain names it should continue to use `https://`.

**Impact:** Account activation fails immediately after the initial setup, making the platform unusable until the configuration is manually fixed.

Register hostname during wizard and use it for activation links


**Title:** Missing backend support for tip delete, global delete, and postpone operations

**Description**

The receiver-facing UI needs to expose three actions on tips: personal delete, global delete (removes the tip for all receivers), and postpone (extends expiration). The current backend does not adequately support these workflows, making it impossible for the client to implement the UI correctly.

**Observed problems**

1. **Authentication response**
   When a receiver logs in via `POST /auth/login` (role=receiver) or retrieves the current session via `GET /auth`, the response does not include any permissions object. The client cannot determine whether to show a “global delete” button without knowing `can_delete_submission`.

2. **Tip data for receivers**
   When a receiver fetches a tip (`GET /tip/{id}`), the response omits the `can_delete_submission` field. The UI requires this to conditionally enable global delete controls.

3. **DELETE endpoint**
   `DELETE /tip/{id}` currently always performs a *personal* deletion (removes the tip only for that receiver). There is no way to trigger a *global* deletion through the API, even for receivers with the appropriate permission.

4. **PUT endpoint**
   `PUT /tip/{id}` contains commented-out code for global delete and pertinence voting, and does not properly expose the *postpone* (extend) operation that the UI needs.

5. **Schema naming**
   The `actorsTipOpsDesc` request schema defines the key `total_delete`, while the client and the rest of the system refer to this operation as “global delete”. The mismatch prevents correct message validation.

6. **Serializer consistency**
   The `anon_serialize_node` function does not accept a `user` parameter, unlike other serialization helpers that may need to gate fields based on the requesting user. For consistency and to allow future access-control decisions, it should accept this parameter.

**Expected behavior and constraints**

- **Login / session endpoint**
  The response for receiver authentication must include a `permissions` object with a boolean `can_delete_submission` that reflects the receiver’s permission to delete submissions globally. The same object must be present in the `GET /auth` response and be stored on the session so it can be retrieved later.

- **Tip retrieval**
  When a receiver fetches a tip, the returned tip data must contain a `can_delete_submission` field, matching the value from the receiver’s record.

- **DELETE /tip/{id}**
  The endpoint must validate the incoming request body against `actorsTipOpsDesc`. If the body contains `global_delete: true` and the authenticated receiver has `can_delete_submission == true`, the endpoint must perform a global deletion (i.e., remove the entire internal tip). If the receiver lacks the permission and requests a global delete, the endpoint must return a **403 Forbidden** error. If `global_delete` is false or absent, the endpoint must perform only a personal deletion.

- **PUT /tip/{id}**
  The endpoint must support an `extend` action (postpone) when the request body contains an `extend` flag. The obsolete placeholder for global_delete should be removed from this method, because global deletion is handled via the DELETE endpoint as described above.

- **actorsTipOpsDesc schema**
  Rename the existing `total_delete` key to `global_delete`. The schema must still accept a boolean value.

- **anon_serialize_node**
  Update the function signature to accept a `user` parameter. The parameter does not need to be used at this point; the change is for interface consistency.

All changes must not alter the behavior of admin login, whistleblower tip submission, or the public node information endpoint (beyond the harmless signature change).

finished implementing UI for tip delete/tip global delete/tip postpone


**Task: Harden the general-settings update path and tighten receiver-facing field surface**

### Background / observable problem
- `GeneralSettingsDesc` currently validates many fields that are really admin-only or localized config keys (`maximum_filesize`, `languages_enabled`, `default_language`, all the `header_title_*` variants, `disclaimer_title`, `disclaimer_text`, `custom_privacy_badge_text`, `contexts_clarification`, etc.).
- When a receiver (who has `can_edit_general_settings`) submits a general‑settings update, those extra fields are exposed in the validated request, and the update logic tries to persist them — some as node config and some via `NodeL10NFactory` — creating confusion about what a receiver can actually modify.

### What needs to change

1. **Trim `GeneralSettingsDesc`** in `backend/globaleaks/rest/requests.py` down to the fields that really are receiver‑editable general settings: `name`, `header_title_homepage`, `presentation` (localized), `description` (localized), `whistleblowing_question` (localized), `whistleblowing_button` (localized), and `footer` (localized). Remove the other fields that should only be in the admin‑node or full‑node descriptors.

2. **Make the L10N update unconditional on the request payload.**
   The localized pieces of the general‑settings request should be persisted through `NodeL10NFactory.update(...)` always, not only when `'languages_enabled'` is present in the request. When languages‑enabled is absent, the L10N update must still run so that `presentation`, `description`, etc. get saved.
   — *Edge case*: do not gate this on the incoming request’s `languages_enabled` list; instead use the system’s *persisted* enabled languages for the tenant (so the update works even when the caller isn’t touching language configuration).

3. **Guard the config-update loops against missing keys.**
   `ConfigFactory.update` currently iterates over `request.keys()` and updates matching `self.res` entries; `ConfigL10NFactory.update` iterates over *all* `self.keys` and reads `request[key]`. Both should only touch keys that are actually present in the request, so a sparse request (e.g. only `header_title_homepage`) does not cause `KeyError` or try to write `None`/missing values into the config tables.

4. **Remove the `enable_signup` force‑disable for tenants other than 1.**
   The `db_update_node` path currently hard‑sets `enable_signup = False` whenever `tid != 1`. That policy should be removed and managed by the usual request‑filtering / ACL machinery.

5. **Fix a stale `session.merge` in the test helper `set_receiver_acl_flag_true`.**
   The helper in the node‑admin test suite calls `session.merge(rcvr)` on an already‑queried object — that is unnecessary and should be dropped.

### Acceptance criteria (behavioral invariants)
- A PUT to the node‑instance endpoint from a receiver with `can_edit_general_settings` only accepts and persists the fields in the reduced `GeneralSettingsDesc`.
- Privileged fields (e.g. `smtp_server`, `maximum_filesize`, `enable_signup`, `languages_enabled`, non‑homepage headers) are silently ignored or rejected for that role.
- `NodeL10NFactory` persists locale‑specific fields even when `languages_enabled` is not present in the request, using the tenant’s current enabled‑language set.
- Sparse updates work: omitting optional keys from the request does not cause errors.
- The existing admin‑node update path (full `AdminNodeDesc` or notification desc) continues to work as before.


We need to implement the GPG key expiration warning functionality. Currently, the daily job `GPGEXpireCheck` in `globaleaks/jobs/gpgexpire_sched.py` does not actually check key expiration dates or send any notifications. Receivers who upload PGP keys with an expiration date never receive a warning, which means encrypted notifications can silently stop working when the key expires.

The check should:
- Run daily (already scheduled, but the operation is empty).
- Query all receivers that have a GPG key configured.
- Compare each key's expiration date to the current date.
- Categorize keys into three buckets:
  - Expiring within 3 days (high urgency).
  - Expiring within 2 weeks (low urgency).
  - Already expired.
- For each receiver with a key in any of these buckets, send a warning email using the node’s configured SMTP notification settings. The email should indicate which category applies (e.g., "expire in three days", "expire in two weeks", "it's already expired"). If multiple receivers share the same email address, only one email should be sent (using the most urgent category for that address).

Additionally, the `sendmail` function in `globaleaks/utils.py` currently expects a file-like object (`StringIO`) for the message content. The scheduler will generate the email body as a plain string, so `sendmail` should be updated to also accept a `str` or `unicode` directly and convert it to a `StringIO` internally. Invalid types should raise an error.

The job class itself was originally defined as a function (`def GPGEXpireCheck(GLJob):`) which is incorrect for the GLJob framework; it must be a proper `GLJob` subclass so that its `operation` method is invoked by the scheduler.

completed and tested key expiration scheduler, part of https://github.com/globaleaks/GlobaLeaks/issues/187