# Bug: Compiler crash with nested vectorization and store predicates

**Summary**
When a Halide pipeline is scheduled with *nested vectorization* (vectorizing multiple dimensions) and store/input tail strategies like `TailStrategy::Predicate` or `TailStrategy::PredicateStores`, the compiler can crash with an internal assertion failure (`"IfThenElse with vector condition"`) or generate incorrect code. The root cause is that the simplifier and loop-partitioning passes assume a store’s predicate is always scalar, but nested vectorization can produce a broadcast of a vector condition, which is not being correctly reduced to a scalar predicate.

**Observed behavior**
- Compiler assertion failure: `internal_assert(condition.type().is_scalar())` inside `IfThenElse::make`.
- Possible miscompilation where store predicates are mishandled, leading to wrong output values.

**Expected behavior**
- The pipeline compiles without an internal error.
- The produced output is identical to an equivalent, unscheduled pipeline (bit‑identical).
- Store predicates resulting from nested vectorization are correctly unwrapped/scalarized before being used in further lowering.

**Steps that trigger the issue**
A schedule that:
1. Splits and fuses loops to create an outer block dimension and an inner vectorized dimension.
2. Applies `vectorize` to the inner dimension.
3. Also vectorizes the outer block dimension (causing nested vectorization).
4. Uses `TailStrategy::Predicate` or `PredicateStores` on a producer split.
5. Computes the producer at the innermost vectorized loop level (`compute_at`).

The problem is not tied to a single fixed schedule but arises from the interaction of multiple tail strategies and vectorization levels.

**Acceptance criteria**
1. The compiler must not crash when encountering store predicates that are broadcasts of vector conditions (nested vectorization).
2. The lowering must correctly treat such predicates as scalars after unwrapping nested broadcasts.
3. The pipeline must produce correct pixel values (matching an unscheduled reference) for all combinations of tail strategies and vectorized loops.

Handle nested vectorization in store predicates (#7864)


As a user tuning Halide’s blur pipeline for GPU backends (e.g., an NVIDIA GTX 970), I need to be able to select among several GPU scheduling strategies directly from the generator parameters. The blur generator should accept:

- A `schedule` parameter (string or enum) with allowed values `"inline"`, `"cache"`, `"slide"`, and `"slide_vector"`. The default must be `"slide_vector"`.
- `tile_x` and `tile_y` integer parameters (default 32 and 8, respectively) controlling GPU tile sizes.

When the target includes a GPU feature:

- `"inline"`: tile `blur_y` directly on the GPU using the given tile dimensions.
- `"cache"`: additionally compute the intermediate `blur_x` at the GPU block level.
- `"slide"`: schedule `blur_y` so that each work-item processes a sliding window of `tile_y` rows, reusing `blur_x` implicitly (split, reorder, unroll, then tile across x only).
- `"slide_vector"`: same as `"slide"`, but also vectorize across x using a factor derived from `sizeof(int)/sizeof(short)`.

For non‑GPU targets, the existing CPU schedule (parallelisation, vectorisation, `store_at`/`compute_at`) must remain exactly as it was.

Additionally, any test or benchmarking harness that calls the generated pipeline must correctly synchronise the output buffer with the host: after each execution, if the buffer is a device buffer, copy the result back to host memory and issue a device sync before timing or reading results. This ensures timing measurements and validation logic see the final data regardless of scheduling strategy.


Add a public API to retrieve Halide's current target string (including HL_TARGET environment override and native platform detection), then use it in all correctness tests to remove duplicated environment reads, and fix the SIMD op check test so it correctly enables AVX/SSE4.1 when the target contains "avx" or "avx2" without requiring an explicit "sse41" substring.

Expose Halide's target selection and make use of it in the tests.


We need the OpenGL backend to produce correct results for Halide pipelines that use images with a single color channel (for example, a grayscale image or a mask).  Currently, the OpenGL codegen and runtime assume at least three channels and either crash or generate wrong output.

Acceptance criteria / observable behavior:

1. **Single‑channel input images** – Provide a `buffer_t` with `extent[2] == 1` (or `extent[2] == 0`, which should be treated as a single channel) and use it in any Halide computation (e.g., a simple pass‑through or a dependent lookup). The pipeline must execute without errors and yield the same pixel values as the CPU reference.

2. **Kernel argument generation** – In the generated GLSL, image calls for a single‑channel image must not attempt to use a varying channel coordinate.  The missing dimension should be effectively pinned to 0 with an extent of 1, so that the texture read only uses the spatial coordinates and does not break the compiler’s expectation of a fixed set of coordinates.

3. **OpenGL texture format selection** – The runtime (especially `get_texture_format`) must select an appropriate texture format for 1‑channel buffers.  The chosen format must be widely supported on both OpenGL ES 2.0 and desktop OpenGL 2.1/3.x (e.g., `GL_LUMINANCE`).  The internal format for floating‑point data must be a single‑component renderable format (e.g., `GL_RGBA32F` with appropriate swizzling, or mapping to a format that the hardware can handle – the exact mechanism may be internal but the buffer contents must be preserved exactly).

4. **Zero‑extent tolerance** – When a dimension of the buffer has `extent == 0`, the runtime must treat that dimension as having size 1 rather than failing or producing invalid texture dimensions.  This applies especially to the height and channel count when they are zero.

5. **Vectorized loops on single‑channel images** – Vectorizing a loop that processes a single‑channel image must not cause a crash or produce incorrect values, even when the vectorized dimension involves a `shuffle_vector` where the channel expression is constant but the shuffled expression references the vectorized variable.  The vectorizer should correctly scalarize the entire node in such a scenario.

6. **No regression for multi‑channel cases** – The changes must not affect the correctness of existing 3‑channel or 4‑channel pipelines.  All existing OpenGL tests (including the dependent lookup test) must continue to pass.

In short: the OpenGL backend must safely handle buffers with 1, 2, 3, or 4 channels, using hardware‑accelerated texture reads/writes, and produce identical results to the CPU path.  The implementation should respect the OpenGL version and profile constraints that already exist in the runtime (e.g., fall back gracefully if a required format is not available).

Add support for images with one color channel to GLSL runtime and codegen. Added test.


We need a set of higher-level scheduling primitives in `Func` to reduce the repetitive, error-prone boilerplate required for CUDA schedules and common tiling patterns. Currently, a typical CUDA schedule involves manually splitting a dimension with block and thread indices, marking them `parallel`, optionally transposing for coalescing, and sometimes renaming dimensions to match GPU runtime conventions. This leads to verbose, hard-to-read code in both user applications and internal tests.

### Desired Additions

1. **`tile` overload with explicit outer/inner variables**
   A `tile` scheduling method that accepts distinct `xo, yo` (outer tiles) and `xi, yi` (inner tiles) variable names, in addition to the original `x, y`. Internally it should `split` each dimension into the given outer/inner pair and perform the standard `transpose(xo, yi)` for memory coalescing. The existing `tile` overload that reuses `x` and `y` as the outer variables should remain available.

2. **`rename`**
   A method that renames a dimension variable, so that subsequent schedule transforms see the new name. This is needed internally by CUDA-specific helpers but also useful for general schedules.

3. **CUDA helpers**
   - **`cudaTile`** (1D and 2D): A shortcut to tile a dimension (or pair) using standard block/thread index variable names, with the tile factors specified as integers, and automatically mark the resulting block and thread dimensions as `parallel`. This should produce the conventional GPU launch decomposition in a single call.
   - **`cuda`** (1D and 2D): A helper that assumes the user already has variables named like block/thread indices and renames them to the concrete ``blockidx``/``threadidx`` (and similarly for y), marks them `parallel`, and transposes appropriately so the schedule matches CUDA expectations.

4. **All new methods must be chainable** (return `Func &`) and push the corresponding transform metadata onto the internal schedule vector, consistent with existing scheduling methods.

### Motivation & Constraints

- Eliminate the need to write long sequences of `split`/`parallel`/`transpose` in CUDA paths, which are a maintenance burden and a source of scheduling mistakes.
- The `rename` primitive must work as an identity split (factor 1) under the hood; no observable semantic side-effects beyond the name change.
- The CUDA helpers should use the conventional ``threadidx``/``blockidx`` naming and not introduce new runtime concepts—they are pure scheduling sugar.
- All methods should be declared in the public header `cpp_bindings/Func.h` and implemented in `cpp_bindings/Func.cpp` alongside the existing transform methods.

This request focuses purely on the C++ user-facing scheduling API. Internal plumbing and test cleanups are out of scope; the core deliverable is the set of new scheduling methods.

Cleaning up tests. Needs retesting on cuda!!


Improve compile-time error reporting when Halide cannot infer finite bounds for image accesses.

* In `src/Bounds.cpp`, the `merge_boxes` function should propagate undefined intervals correctly. When merging two `Box` regions, if an interval’s min or max is not defined (i.e., the Expr is empty/undefined), the merged interval must also become undefined instead of silently producing a bogus min/max. This prevents later passes from using incomplete bounds as if they were valid.

* In `src/Lower.cpp`, the `add_image_checks` pass should examine each dimension’s required bounds after inference. If any dimension has an undefined min or max (i.e., the access is unbounded), it must halt compilation immediately with a clear diagnostic message that identifies the buffer name and the dimension index. This replaces latent assertion failures or invalid runtime checks with an upfront, actionable error.

No runtime error path should be generated for unbounded dimensions; the failure must occur at Halide compile time. The message format should be explicit enough for users to locate the offending Func/dimension in their pipeline.

Better errors for unbounded access to input/output images.


# Bug Report: `atomic()` should require provably associative updates for parallel/vectorized scheduling

**Observed Problem**

When a Halide reduction `Stage` is scheduled with both `atomic()` and a parallel or vectorized loop for an `RVar`, the current implementation unconditionally accepts the scheduling without verifying whether the update operation is actually associative. This means that non-associative updates (e.g. conditional writes, maximum, or custom operator compositions) can be scheduled atomically and in parallel, leading to silently incorrect outputs due to race conditions that atomics cannot resolve safely.

**Steps to Reproduce**

1. Define a reduction update with a non-associative operator, e.g. a compare-and-swap-like overwrite or a `max` (via `Halide::max`).
2. Call `atomic().parallel(r)` or a GPU atomic split on the update definition.
3. Compile and run; the program may run without errors but produce wrong results compared to sequential execution.

**Expected Behavior**

Calling `atomic()` on an update stage should be an assertion that the reduction operator is associative and therefore safe to parallelize. Halide already includes an associativity prover. The schedule should be rejected with a clear error message if:

- The associativity prover cannot prove the operation is associative.
- For Tuple updates, each output value must be provably associative.

If proof is impossible (e.g., because the type is `float16`, which the prover does not support), the error should suggest using `allow_race_conditions()` as an explicit escape hatch.

**Actual Behavior**

Currently, `atomic()` simply silences the race-condition safety check entirely, allowing any reduction to be scheduled parallel/vectorized without verifying associativity. This silently accepts non-associative operations, leading to correctness holes.

**Impact**

Reduction operations that are not associative produce non-deterministic, incorrect results when parallelized. Developers might rely on `atomic()` for performance without realising that their operator isn’t associative, introducing subtle bugs that are hard to diagnose. Requiring associativity proof (or an explicit `allow_race_conditions()` override) prevents such correctness issues at schedule time.

**Suggested Fix**

- In `Stage::set_dim_type`, when an RVar is marked parallel/vectorized and `atomic()` is set, invoke the existing `prove_associativity` mechanism with the update arguments and values.
- If associativity cannot be proved, raise a `user_assert` with a message like:
  `"Failed to call atomic() on … since it can't prove associativity of the operator. Use allow_race_conditions() to override this check."`
- For Tuple updates, ensure each output value is checked individually.
- The existing check for `allow_race_conditions()` should still bypass the associativity requirement entirely, as a deliberate opt-out.


In `detect_clamped_load` (CodeGen_X86.cpp), the bounds-check condition for vectorized clamped loads is always computed as an intersection of both upper-bound and lower-bound checks on both the first and last ramp elements, regardless of the stride’s sign. When the stride is a known constant (positive or negative), only one end of the ramp actually matters for each bound; the other check is redundant. This leads to unnecessarily complex conditions that can degrade runtime performance of the generated vector code.

**Observed behavior**:
For a ramp with a constant positive stride, the generated condition checks both `(base >= lower)` and `(end >= lower)`, even though the latter is automatically satisfied when the stride is positive (since `end > base`). Similarly, for a constant negative stride the condition still includes `(end <= upper)` which is already implied by the base being ≤ upper. In the unknown-stride case both checks are needed, but the current code never exploits stride-sign knowledge.

**Expected behavior**:
When the stride is detected as a positive constant, the condition should only enforce `end <= upper` (if an upper bound exists) and `base >= lower` (if a lower bound exists). For a negative constant stride, it should enforce `base <= upper` and `end >= lower`. When both bounds exist, the sign determines which element is tested against which bound. When the stride sign is unknown, the full set of checks remains. The resulting condition should also be simplified to avoid trivial `true && ...` patterns.

**Impact**:
This affects all uses of clamped vector loads (e.g., `clamp(x, min, max)` followed by a vectorized load) where the ramp stride is compile-time constant. Performance-sensitive imaging pipelines see a measurable slowdown from the larger condition, and the JIT-generated assembly includes unnecessary comparison and branch instructions.

**Request**:
Please optimize `detect_clamped_load` so that it emits the minimal bounds-check condition when the stride is a known constant, using the sign information to prune redundant checks. The existing pattern-matching for the ramp (cases 1–4) should be retained, and the output condition should be simplified.

Performance improvement for clamped vector load.


Add Python bindings for the following `Func` scheduling methods: `tile`, `unroll`, `reorder`, `reorder_storage`, and `bound`. Also fix the existing `vectorize` binding so that it returns the `Func` itself, enabling method chaining (all scheduling methods should return a reference to `self`).

For `tile`, provide both signatures: the six-argument form (`x, y, xo, yo, xi, yi, xfactor, yfactor`) and the shorter four-argument form that reuses the old variable names as the new outer dimensions (`x, y, xi, yi, xfactor, yfactor`).

For `unroll`, provide both `(var, factor)` and `(var)` (unroll full extent).

For `reorder` and `reorder_storage`, accept a variable-length list of `Var`/`VarOrRVar` arguments via a Python tuple. Validate the contents and raise an informative `ValueError` if any element cannot be converted to the required type.

Each new method should include a Python docstring that briefly describes its purpose. Ensure that code like `blur_y.tile(x, y, xi, yi, 8, 4).parallel(y).vectorize(xi, 8)` can be written from Python without type errors.

test_basics.py passes. Func cpu methods coverage is quite extensive now. Time to get a real example running.


We need to remove an overzealous internal stride assertion in lowered Halide pipelines. Currently, when interleaved storage layouts and tiled `compute_at` schedules are combined, the generated code may hit a fatal assertion:

> "Applying the constraints to the required stride made it smaller"

This assertion is too strict. Strides are treated as a suggestion—the compiler may legitimately adjust them during bounds inference. It is acceptable for the final required stride to end up smaller than the originally computed required stride, as long as the region extent remains valid.

**Acceptance criteria:**
- Any Halide pipeline that uses channel-interleaved storage (e.g., `reorder_storage` with `set_stride` on the inner dimension) and tiled computation across that pipeline (e.g., `compute_at` inside a tiled loop) shall run without hitting the stride assertion above.
- The region-extent check ("Applying the constraints to the required region made it smaller") must continue to fire when the required region actually shrinks.
- The pipeline must produce correct output values, identical to a non-tiled execution of the same algorithm.

**Implementation note:**
The faulty check lives in `add_image_checks` inside `src/Lower.cpp`. The assertion that compares `stride_proposed >= stride_required` should be removed entirely, leaving only the region-extent check in place.


```
Title: Remove test-only PoW-skip flag from production PoW validators

Background:
The production `EthashBlockHeaderValidator` and the `EthashValidators` factory method currently accept a `shouldSkipPoW` boolean parameter. This flag is set to `true` only when the Ethereum Test Suite (ETS) scenarios declare `sealEngine: "NoProof"`. The production code should never skip Proof-of-Work validation; that behavior belongs exclusively in the test harness.

The flag was introduced to work around upstream test vectors that don't provide valid PoW for blocks (see ethereum/tests#480). As a result, the production binary carries a dead code path and an API that implies PoW can be disabled at runtime — which is both misleading and unsafe.

Request:
- Remove the `shouldSkipPoW` parameter from `EthashBlockHeaderValidator` and the corresponding `EthashValidators.apply` overload. After the change, the production validators must unconditionally perform PoW validation.
- In the ETS blockchain test setup (which reads `sealEngine` from `BlockchainScenario` and decides whether to skip PoW), provide a test-specific block header validator that always reports PoW as valid. The replacement must be confined to the ETS test sources and must never leak into production compilation.
- The test setup must still respect `sealEngine: "NoProof"` by using this test-only validator when the engine is NoProof, and the standard production validators otherwise.
- Ensure that the ommers validator continues to use the same block header validator instance as the block validator (they share a common `BlockHeaderValidator` interface). The test-specific validator must satisfy that interface so that ommers headers are validated consistently with the main block headers in both test and production paths.

Acceptance criteria:
- Production codebase contains zero references to a `shouldSkipPoW` flag.
- All existing ETS blockchain tests pass without modification to the test vector files.
- The final artifact does not include any test-only validator classes.
```

Create ValidatorsWithSkippedPoW and EthashTestBlockHeaderValidator to prevent test code from changeing production code


**Title:** Add `decompressPublicKey` for secp256k1 compressed keys and make `compressPublicKey` idempotent

**Description**
The discovery layer’s `Secp256k1SigAlg` only supports one-way compression: `compressPublicKey` turns an uncompressed 65‑byte public key into a 33‑byte compressed form. There is no way to go in the opposite direction. If a peer sends us a compressed key (33 bytes) we cannot reconstruct the full 65‑byte key for signature verification, effectively making those messages unverifiable.

Additionally, calling `compressPublicKey` on an already‑compressed 33‑byte key currently throws an `IllegalArgumentException`. It should instead recognise that the key is already compressed and return it unchanged (the same way decompression should return an uncompressed key unchanged). The error messages for unexpected sizes ought to distinguish between “unexpected uncompressed public key size” and “unexpected compressed public key size” to help debugging.

**Expected behavior**
- Add a `decompressPublicKey` method that:
  - Accepts a `PublicKey` (a `BitVector`).
  - If the input is already a 65‑byte uncompressed key, returns it as‑is.
  - If the input is a 33‑byte compressed key, recovers the full uncompressed key using the secp256k1 curve equation:
    - `y² = x³ + 7` over the field prime
      `0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffefffffc2f`
    - The first byte of the compressed form is the parity indicator; it tells whether the recovered `y` is even or odd. The existing constant `ECDSASignature.CompressedEvenIndicator` can be used as a reference for the “even” indicator byte.
    - The result must be the standard 0x04‑prefixed uncompressed point (X concatenated with Y, both 32 bytes).
  - Throws a descriptive `IllegalArgumentException` if the input size is anything other than 33 or 65 bytes.
- Modify `compressPublicKey` so that it returns a 33‑byte compressed key unchanged instead of throwing.
- Use `ByteUtils.bigIntToBytes` for converting the recovered `y` coordinate to a fixed‑width byte array, consistent with how other key‑handling code works in this object.

**Impact**
Without decompression, the discovery protocol cannot verify signatures from nodes that only provide compressed public keys. This blocks interoperability with other Ethereum clients that routinely send compressed keys.

ETCM-167: Decompress public key using formulas.


As a maintainer ticket:

---

**ETCM-167: Enable EIP-8 test vectors for discovery RLP payload decoding**

Our RLPx discovery protocol currently fails on the EIP-8 test vectors because the RLP payload decoders are too strict. The EIP-8 specification allows discovery packets to contain additional list elements beyond those expected by a particular message type, as well as trailing random data after the RLP-encoded content. Decoders for `Ping`, `Pong`, `FindNode`, `Neighbors`, `ENRRequest`, and `ENRResponse` must be robust enough to decode such packets and silently ignore the superfluous entries and any trailing bytes.

Additionally, the `payloadCodec` inside `RLPCodecs` must correctly pass the full packet data (after the single-byte packet type) to the RLP decoder, tolerating extra data that may follow the RLP structure.

While making these changes, improve the error messages emitted during RLP decoding failures. Include the target type name so that it's clear which payload type failed to decode. For per-field errors, use the exception message rather than serializing the entire exception object. This will make future diagnosis of malformed packets significantly easier.

The test suite currently has several ignored test cases based on the EIP-8 vectors. Once the payload decoders are updated to accept extra list elements and trailing data, those tests should be un-ignored and extended to validate each packet type independently, including a separate check that the full decoder stack (packet decoding + RLP decoding) works correctly end-to-end.

ETCM-167: Testing the RLP codecs on the EIP8 test vectors.


Improve discovery message sending to eliminate double-encoding and ensure token consistency.

Currently, `PeerDiscoveryManager` needs to know the exact bytes that will be sent on the wire for a ping message (to compute the MDC and data hash used for pong validation). It achieves this by calling `DisoveryListener.encodePacket` directly, while the actual outbound message is sent via `DiscoveryListener.SendMessage`, causing the listener to encode the message a second time. This double-encoding risks producing different packet bytes due to non-deterministic signing, leading to validation mismatches with other clients.

Fix this by making `PeerDiscoveryManager` encode messages itself before handing them off to the listener. The listener should accept only already-encoded packets and forward them to UDP without further processing. Move the `Packet` case class and the associated encode/decode/utilities from the `DiscoveryListener` companion object into the `discovery` package object so they are accessible from both the manager and the listener. Ensure that all message sends from the manager (including pings, pongs, find-node, and neighbours) go through this single encoding path, and that the token validation uses the same bytes that are actually sent.

EC-596 - Improve our handling of sending discovery message


```markdown
Title: Implement EIP-1234 — Constantinople Difficulty Bomb Delay and Block Reward Adjustment

Background:
EIP-1234 delays the Ethereum Classic difficulty bomb (ice age) by increasing the fake‑block‑number offset from 3,000,000 to 5,000,000 blocks at the Constantinople fork block number. It also reduces the miner block reward from 3 ETC to 2 ETC at the same fork.

Current state:
- The client already applies a difficulty bomb delay for Byzantium (offset 3,000,000) and a reduced block reward (3 ETC) when `blockNumber >= byzantiumBlockNumber`.
- Monetary policy is configured via `mantis.monetary-policy` in `application.conf` and is parsed into `MonetaryPolicyConfig`.
- Difficulty calculation is in `EthashDifficultyCalculator`; block reward calculation is in `BlockRewardCalculator`.

What needs to happen:
1. **Difficulty bomb delay for Constantinople.**
   When `blockNumber >= constantinopleBlockNumber`, the fake block number used for the difficulty bomb exponent must be computed as
   `(blockNumber - 5_000_000).max(0)`.
   For blocks between Byzantium and Constantinople (exclusive), the existing Byzantium delay of 3,000,000 must still apply.
   The `difficultyBombRemovalBlockNumber` guard must continue to work as before (only the offset changes).

2. **Block reward reduction for Constantinople.**
   When `blockNumber >= constantinopleBlockNumber`, the base miner reward must be 2 ETC (20 e17  wei).
   The reward for ommer inclusion must scale accordingly from that base.
   The existing Byzantium reward of 3 ETC must remain for blocks `>= byzantiumBlockNumber` and `< constantinopleBlockNumber`.

3. **Configuration.**
   `application.conf` needs a new entry for the Constantinople reward (e.g., `first-era-constantinople-reduced-block-reward = "2000000000000000000"`).
   `MonetaryPolicyConfig` must carry this value, and all callers that read the config must be kept in sync.

4. **Testing helper configurations.**
   The test blockchain configurations (e.g. in `BlockchainTestConfig`) that define monetary policy for forks must include the new reward parameter so that reward calculations are correct when Constantinople is active. They also need to provide a `constantinopleBlockNumber` where required.

Acceptance criteria:
- On Constantinople fork block (and above), the difficulty bomb delay is 5,000,000 and miner reward is 2 ETC; below it, Byzantine rules apply unmodified.
- No existing Byzantium‑era tests break, and the new behavior can be verified with typical blockchain test configurations.
- The configuration is self‑explanatory and backward‑compatible (a default should be a no‑harm filler, e.g., `"0"`, if the value is missing from config).
```

EIP-1234 Constantinople Difficulty Bomb Delay and Block Reward Adjustment


**Title**: Mining on single‑node private networks fails – mined blocks silently discarded

**Observed Behavior**
When running a Mantis node as the sole participant on a private network (no peers, CPU mining enabled), mined blocks are never accepted or added to the chain. The miner produces blocks, but they are silently dropped by the sync layer, and the chain never progresses.

**Expected Behavior**
Mined blocks should be imported and appended to the local chain whenever the block importer is idle, regardless of whether the node is synchronized to the “network top.” A single‑node network has no external top to sync to, so mining must still produce a valid, growing chain.

**Impact**
CPU mining on private/test networks with a single node is completely broken. Developers cannot test mining or consensus changes on isolated networks without artificially adding remote peers.

**Root Cause / Change Needed**
The guard that decides whether to import a mined block currently requires both that the importer is not already busy (`!state.importing`) **and** that the node is considered to be at the chain tip (`state.isOnTop`). The `isOnTop` condition is problematic because in a network with no peers the node never receives updates that would mark it as “on top,” causing all mined blocks to be rejected.

The fix should relax this guard so that a mined block is imported whenever the importer is idle, without also requiring the node to be on top. The existing safety mechanism that prevents concurrent imports (`!state.importing`) is sufficient; the extra `isOnTop` check should be removed for the mined‑block path.

No other control flows (e.g., handling of blocks received from peers) should be affected; they may continue to use the `isOnTop` condition if appropriate.

Change guards for importing mined blocks to allow mining on single-node private networks


**Ticket: Make fast sync reject non-chain-forming block header responses**

**Component:** `FastSync` (block header download stage)

**Problem:**
When a peer sends a `BlockHeaders` response, the handler currently only checks whether the number of headers is less than or equal to the requested count. It does not verify that the headers actually form a valid chain continuation from the node’s current best-known header. This allows a faulty or malicious peer to:

- send an *empty* header list, which can stall or confuse the sync logic, and
- send headers that start at a block number far in the future (i.e., not `bestBlockHeaderNumber + 1`), which may advance the sync state to a disconnected point, potentially corrupting the sync and wasting resources processing unchainable headers.

These issues reduce the resilience of fast sync and can be exploited to harm the node.

**Request:**
Improve the block header response validation inside the fast sync `SyncingHandler` so that, in addition to the size check, it also verifies that:

1. The response is *non-empty*.
2. The `number` of the *first* header in the response equals `syncState.bestBlockHeaderNumber + 1` (i.e., it starts exactly where the local chain left off).

If either check fails, the peer should be blacklisted with an updated reason message (e.g., indicating “empty or non-chain-forming headers”).

**Desired outcome:**
Fast sync no longer accepts block header responses that break the chain contiguity; such responses simply trigger a blacklisting and the headers are ignored, keeping the local sync state safe.

Make fastSync more resisilent to failures


**Bug Report: Block Body Storage -- Insert/Delete Round-Trip Failures**

After creating a `BlockBodiesStorage` instance, putting a `BlockBody` associated with a `BlockBodyHash` and then immediately trying to get it back returns `None` (i.e. the stored body is not found or is corrupted). This prevents the node from maintaining a correct mapping from block hashes to block bodies, breaking everything that depends on block body lookups (block propagation, sync, chain validation).

**Steps to reproduce**
1. Create an empty `BlockBodiesStorage` backed by an ephemeral data source.
2. Put a `BlockBody` (obtained from a valid `Block`) using its header hash as key.
3. Call `get` with the same hash.

**Expected behavior**
`get` returns `Some(...)` containing the identical `BlockBody` that was stored. A subsequent `remove` of that hash followed by `get` should return `None`.

**Observed behavior**
`get` returns `None` after a successful put, indicating the body was not persisted correctly or cannot be deserialized. Deletion behaviour may also be broken because the stored data is inconsistent.

**Impact**
- The block body mapping is unusable.
- Fast sync, bootstrap sync, and regular block processing rely on this store; they will silently fail to find required block bodies, causing stalled sync or incorrect chain state.
- Peer-serving of block bodies may be affected.

**Likely causes (from initial investigation)**
- **Key serialization**: `keySerializer` currently uses `identity`, which does not convert a `ByteString`-like `BlockBodyHash` into a proper `IndexedSeq[Byte]`. Some `ByteString` implementations may not directly implement `IndexedSeq[Byte]`, leading to broken keys. It should instead call `_.toIndexedSeq`.
- **Value deserialization**: The deserializer calls `Unpickle[BlockBodiesStorage].fromBytes(...)` but should unpickle a `BlockBody` itself since the value stored is a serialized `BlockBody`, not the companion object.
- **Picklers for nested types**:
  - `byteStringPickler` is typed as `Pickler[BlockBodyHash]`, but it should be `Pickler[ByteString]` because it is used to serialize `ByteString` fields elsewhere (e.g., in `Address` and signatures) and `BlockBodyHash` is just an alias for `ByteString`.
  - `addressPickler` is currently auto-generated (`generatePickler[Address]`), which may not correctly serialize/deserialize an `Address` (which internally holds raw bytes). The fix should use `transformPickler` to map through `ByteString`, similar to how other ByteString-based types are handled.
  - The `BlockBody` constructor in the pickler should use the companion-object factory (`BlockBody(stx, nodes)`) rather than `new BlockBody(...)` to ensure normalised construction.

**Definition of done**
- Basic insert/get round-trip works: after inserting multiple distinct block bodies, every stored body is retrievable by its hash and matches the original.
- Delete round-trip works: removing a subset leaves only the remaining bodies retrievable, and the removed ones are gone.
- The tests should run against real generated block data (e.g., using `ObjectGenerators.newBlockGen`) to cover realistic, nested structures including signed transactions, headers, and addresses.


The RegularSync test suite (RegularSyncSpec) currently uses `Thread.sleep(...)` in several places to wait for asynchronous actor behavior. This makes the tests unnecessarily slow and introduces occasional non-deterministic failures in CI.

Acceptance criteria:

- All real-time waits in RegularSyncSpec must be replaced with deterministic time progression using the test scheduler (`time.advance`) and/or polling with `eventually` from ScalaTest.
- No test should rely on a fixed sleep duration; the suite should complete in near-zero wall-clock time.
- All existing behavioral assertions must be preserved, including checks for blacklisting of a peer and for resumption of regular sync after branch resolution.
- An assertion that was previously missing must be added: when branch resolution succeeds and normal syncing resumes, the `ommersPool` actor must receive an `AddOmmers` message.
- The timeout for `eventually` should be long enough to allow the scheduler to process all pending actor messages, but the test must not wait for real clock time.


## EIP-1283 – Net gas metering for SSTORE without dirty maps (Constantinople)

Implement EIP-1283 gas metering for the `SSTORE` opcode for blocks at or after the Constantinople fork block. The goal is to replace the previous dirty-map-based accounting with the “net gas metering without dirty maps” scheme described in the EIP.

### Behavioural acceptance criteria

1. **Fork activation**
   The new gas metering logic must only apply when `isAfterConstantinopleFork` is true (i.e. block number ≥ the configured Constantinople block).  Pre-Constantinople blocks must continue to use the existing dirty-map logic with no change in behaviour.

2. **Original-value snapshot**
   For each contract execution triggered by a transaction, the storage state of the account must be captured **once** before any `SSTORE` operations are applied. This snapshot represents the “original value” of every slot for the lifetime of the transaction. Subsequent writes must not alter this snapshot, and all `SSTORE` cost/refund calculations within the same execution must reference it.

3. **New gas metering algorithm**
   After Constantinople, the gas used and the gas refund for an `SSTORE` must be derived from the tuple `(original_value, current_value, new_value)` according to the rules in EIP-1283.  In particular:
   * No separate dirty-map tracking state should be used; the net refund is determined directly from the relationship among the three values.
   * The implementation must correctly handle all cases enumerated in the EIP, including (but not limited to):
     - storing zero over a non-zero original fresh slot,
     - storing a non-zero value that matches the original value in a dirty slot,
     - storing a non-zero value different from the original in a dirty slot,
     - sequences of multiple `SSTORE`s that eventually restore the original value and the corresponding net refund.

4. **Backward-compatibility of non-CP tests**
   Any existing tests that exercise `SSTORE` (or other opcodes) without testing Constantinople-specific features must continue to operate correctly. They must not inadvertently switch to the new logic; use a pre-Constantinople block number for those test fixtures.

5. **Testability of the EVM execution loop**
   The VM’s main execution method (currently `VM.exec`) must be accessible from test code within the same package so that direct unit-testing of opcode behaviour (e.g., executing a specific sequence of opcodes and observing the resulting state) is possible. Adjust its visibility accordingly (e.g. `private[vm]`).

6. **Correctness with multiple writes**
   Ensure that the net gas refund across several `SSTORE` operations in the same contract execution matches the EIP’s expected values, especially for sequences that return a slot to its original value (where the refund should reflect the full net change).

### Non-goals

* Modifying gas metering for any opcode other than `SSTORE`.
* Changing storage trie logic or persistence — this is purely a gas accounting change.

EIP 1283 - Net gas metering for SSTORE without dirty maps - add test cases from spec and update implementation


When the sync fetcher receives a checkpoint block that is older than the local best block (i.e. `block.number <= lastBlock`), it currently dispatches the block directly to the importer via a `NewCheckpointBlock` message. This bypasses the ready-blocks queue and the normal block-picking flow, which can cause blocks to be imported out of order and potentially lead to sync stalls or chain anomalies.

**Expected behavior**
For a checkpoint block older than the current best block, the fetcher should:

1. Clear any in-flight header/body fetching queues.
2. Place the block into the ready-blocks queue (so it will be picked up and imported in order with other ready blocks).
3. Continue normal fetching without handing the block directly to the importer as a special case.

This mirrors the handling that is already done when a checkpoint block does not fit into the current fetching queues but is not older than the last block (the `knownTop` case): queues are cleared and the block is placed into the ready blocks. The older-than-last-block case should behave similarly, just without updating the known-top tracker.

**Impact**
Without this change, a checkpoint that is behind the local chain can short-circuit the ready-blocks queue, potentially importing a block while earlier blocks are still pending, breaking the expected import sequence and confusing the sync state.

ETCM-670: handling checkpoint older than local best block


When a node receives a checkpoint that triggers branch resolution near genesis, the regular-sync block importer can request block fetch starting from a negative block number. This causes invalid fetch requests and prevents the node from correctly re-syncing to the checkpointed branch. Ensure that the `unknown-branch` fallback in the branch resolution logic never requests a block number below 0; clamp the computed "going back to" block to at least 0. The fix must allow the new checkpoint-driven reorganisation tests to pass, where short chains with checkpoints cause reorganisations that previously triggered negative fetch.


Title: Replace Option/Either wrappers in MPT node representation with explicit NullNode/HashNode types

The current `BranchNode` and `ExtensionNode` classes use nested `Option[Either[...]]` wrappers to encode three states for branch children (empty, hash-reference, or direct child) and two states for extension next (hash-reference or direct child). This adds unnecessary case-analysis boilerplate across the trie implementation, the trie serialization, and all callers (e.g., fast-sync handlers). The representation should be flattened into proper `MptNode` subtypes so that empty slots are represented by a `NullNode` (singleton) and hash references by a `HashNode(ByteString)`.

**What must change:**

- `BranchNode.children` must become `Seq[MptNode]` (length 16) with `NullNode` for empty positions and `HashNode` for hashed children.
- `ExtensionNode.next` must become `MptNode`, using `HashNode` when the next node is referenced by its hash.
- The RLP encoder/decoder (`MerklePatriciaTrie.nodeEnc`/`nodeDec`) must handle the new types while remaining binary-compatible with the existing format:
  - empty child → `RLPValue(empty array)`
  - hash child → `RLPValue(32-byte hash)`
  - inlined child → recursive encoding of the inner `MptNode`
  - `HashNode` and `NullNode` must appear in the `nodeEnc` match so that encoding `BranchNode.children` is straightforward.
- All MPT traversal helpers (`getNextNode`, `getChild`, and the internal `get`/`put`/`remove`/`fix` functions) must be updated to use the new representation, and the `Option`-based null checks must be replaced with `NullNode` checks (e.g., an `isNull` method on `MptNode`).
- The `FastSync` code that collects hashes from branch children and extension next pointers must pattern-match on `HashNode` rather than `Some(Left(...))`.
- Remove the `Option`/`Either`-based overloads in companion objects (`BranchNode.withSingleChild` accepting `Either`, `ExtensionNode.apply` that accepts `Either`) so that only the new clean constructors remain.
- Existing logic for node compaction (`fix`) and updating a branch child (`updateChild`) must be adjusted to maintain identical trie behavior; `fix` must still compact branches with a single non-null child into an extension or leaf when the terminator is absent.

**Non-goals / constraints:**
- The on-disk/wire RLP encoding must not change.
- Trie state-hash computation must remain identical.
- The change is an internal refactoring; all public API surface that returns `MptNode` types must still compile and behave as before after callers are adjusted.


```
TimeSlotStats currently keeps each slot's timestamp and statistics map in two separate
IndexedSeq fields (timeSlots and statSlots), maintained as parallel ring buffers. This
makes every operation that changes or reads a slot index over both structures separately,
and opens the door to alignment bugs.

Refactor the internal representation so a single ring buffer holds an entry per slot,
combining the slot timestamp and stats. Keep the public API and semantics identical, but
ensure the new structure is used consistently for addition, removal, folding, and range
checks. No external callers should notice a difference beyond the changed internal
field.
```


We need to add support for the Byzantium → Constantinople transition test network to the blockchain test harness. The Ethereum test suite now includes transition tests under `ets/BlockchainTests/TransitionTests`, and those tests reference a network named `"ByzantiumToConstantinopleAt5"`.

The harness currently loads scenarios from the parent `BlockchainTests` directory and does not recognize the new network name. As a result, transition tests are either skipped or misconfigured, causing failures.

Acceptance criteria / required behavior:

- The harness should load test scenarios from `ets/BlockchainTests/TransitionTests`.
- A new fork configuration must be registered for the network `"ByzantiumToConstantinopleAt5"`. This configuration should activate Byzantium at block 0, Constantinople at block 5, and treat all fork block numbers before Byzantium (Frontier, Homestead, EIP-150, EIP-155, EIP-160, EIP-161) as disabled (set to –1). The monetary policy must reflect the Constantinople-era parameters (including the post-Constantinople block reward reduction).
- Validator instances must be provided for both the regular validation path and the alternate path that skips proof-of-work validation (follow the same pattern used for `"EIP158ToByzantiumAt5"`).
- The scenario setup logic that maps network names to (configuration, validators) must include the new network so that a test scenario of type `"ByzantiumToConstantinopleAt5"` resolves to the correct configuration and validators.


**Issue: `hold_trait_notifications()` emits notifications with incorrect old values when the same trait is set multiple times inside the block.**

When a `HasTraits` object’s attributes are modified inside a `hold_trait_notifications()` context, the delayed change notifications should reflect the **net change** across the entire block:
- The **old value** in the notification should be the value of the attribute **immediately before entering the context**.
- The **new value** should be the **final value** assigned within the block.

Currently, if an attribute is changed several times inside the block, the old value reported after the block is the value just before the first assignment inside the block, not the pre-block value. For example, with `t.a = 0` before the block:

```python
with t.hold_trait_notifications():
    t.a = 3
    t.a = 4
# current behavior: notification reports old=3, new=4
# expected behavior: notification reports old=0, new=4
```

Intermediate assignments to the same trait should be merged so that the notification preserves the original pre-block old value and the final new value. The fix should be internal to `hold_trait_notifications` and must not break:

- cross-validation of traits that occurs after the block
- proper rollback of all changes when a validation error is raised inside the block
- the existing `_notify_trait` signature and its invocation for the merged events

No new public API is needed; only the merging logic inside the context manager should change.

Merging notifications when changing the same attribute multiple times


```
**Title**: `ExecuteReply` rich display repr methods discard output metadata (e.g., image width)

---

When an IPython parallel engine executes code that produces a rich display object (like an `IPython.display.Image` with explicit dimensions), the `ExecuteReply` wrapper returned by `client[...].execute(...).get()` exposes convenience repr methods (`_repr_png_()`, `_repr_html_()`, etc.). These methods currently only extract the raw mime data from the execution result’s `pyout` dict and ignore any per-mime-type metadata that was attached to the rich output.

For example, if an engine runs:

```python
from IPython.display import Image
Image(data=b'garbage', format='png', width=10)
```

the resulting `ExecuteReply`’s `_repr_png_()` returns a plain base-64 string, but it should return a `(data, metadata)` tuple – specifically `(b64data, {'width': 10})` – according to IPython’s rich-display protocol.

The practical consequence is that clients (e.g., the notebook frontend) that rely on these repr methods cannot faithfully reconstruct the rich output with its original metadata (width, height, etc.). The engine’s rich output metadata is completely lost for parallel results.

**Desired behavior**:

- `_repr_*_` methods on `ExecuteReply` should return `(data, metadata)` tuples whenever per-mime metadata is present in the engine’s pyout.
- There should be a single `_repr_mime_()` method that follows the standard protocol: for a given mime key, if the data contains that key and there is associated metadata, return `(data, metadata)`; otherwise return just the data.
- The change should remain backwards-compatible for callers that only expect data without metadata (i.e., returning the raw string alone when no metadata exists should still be acceptable, though the protocol allows tuple returns).
- Any existing manual `_repr_*_` methods should be replaced by the unified `_repr_mime_` implementation to reduce duplication and ensure all mime types – including those added in the future – automatically support metadata.

**Suggestion / context**:

IPython already has a robust canonical implementation for this pattern (used, for instance, by output capture machinery). The parallel client’s `ExecuteReply` is currently managing rich representations in an ad-hoc way that is inconsistent with the rest of the codebase; adopting the existing shared abstraction would fix the metadata bug and bring the parallel client in line with IPython’s display contract.

---

**Impact**: parallel execution results that include rich output with metadata (images with sizes, HTML with extra attributes, etc.) are missing that metadata, causing degraded rendering in downstream consumers. This makes the parallel client less reliable for notebooks and other IPython frontends when displaying computed rich objects.


Move `ConnectionFileMixin` and `write_connection_file` from `ipython_kernel.connect` to `jupyter_client.connect`, so they can be used without importing IPython. The IPython-specific helpers (`get_connection_file`, `get_connection_info`, `connect_qtconsole`) must remain in `ipython_kernel.connect` and should delegate to the shared `jupyter_client` functions where possible. Update all internal imports accordingly.

Specific requirements:

- `jupyter_client.connect` must provide:
  - `write_connection_file`
  - `find_connection_file` (rework its signature to accept a `path` parameter instead of an IPython profile name; searching defaults to the current directory)
  - `ConnectionFileMixin` (the whole class, including its traits, methods, and socket helpers)
  - `tunnel_to_kernel`
- `ipython_kernel.connect` must provide:
  - `get_connection_file` (unchanged behavior)
  - `find_connection_file` (still respects profile names and security directories, but calls `jupyter_client.find_connection_file` with the resolved paths)
  - `get_connection_info` (unchanged)
  - `connect_qtconsole` (unchanged)
  - `write_connection_file` (imported from `jupyter_client`)
- `ipython_kernel.kernelapp` must import `ConnectionFileMixin` from `jupyter_client.connect` instead of the old location.
- Remove any leftover general utility code and imports from `ipython_kernel.connect` that are now in `jupyter_client`.
- No externally visible behavior should change; all existing functionality must be preserved through the delegation.

split kernel/client pieces of kernel.connect


We need a way for users and extensions to register AST node transformers that are automatically applied to every piece of user input executed in an IPython cell.  The transformers should be `ast.NodeTransformer` instances that can inspect and modify the parsed AST before any code runs.

**Acceptance Criteria**

1. The interactive shell exposes a new configurable **list attribute** that holds `ast.NodeTransformer` instances.
   - The default value is an empty list.
   - Users can populate it via configuration (e.g., `c.InteractiveShell.<name> = [...]`).
   - Transformers are kept in insertion order.

2. **When a user submits a cell** for execution, the shell must:
   - Parse the raw input into an AST (as it already does).
   - Before executing that AST, **apply every registered transformer** in order: each transformer’s `.visit()` method is called on the root AST node, and the result is used as the AST that subsequent transformers see.
   - Finally, **execute the (possibly modified) AST** normally.

3. **Fault tolerance** – If any transformer **raises an exception** during its visit:
   - The shell must **warn** the user that the transformer failed and will be unregistered.
   - The failing transformer is **removed** from the list so that future cell executions proceed without it.
   - The execution of the *current cell* should not be aborted; the AST that was present before that transformer attempted its change should be used for the rest of the pipeline (the behavior should be as if the failed transformer were skipped).

4. **Observable behavior** – Transformers must be able to modify the AST in ways that are reflected in execution output.  For example, a transformer that reverses the sign of numeric literals would cause `print(12 + 22)` to output a different number than 34.

5. **Scope** – Only user-entered cell input is transformed.  The transformation must not affect internal Python execution, magic commands, or code that IPython itself runs for bookkeeping.

**Additional notes**
- The transformation step must happen **after** the input is split and parsed into an AST, but **before** the AST is handed off to the interactive execution machinery.
- Do *not* change the existing interactive execution modes (`all`, `last`, `last_expr`, `none`) — they should continue to work identically on the possibly transformed AST.
- Any helper methods introduced for this feature should be private implementation details of the shell and not part of the public API beyond the configurable list.

Add framework for AST transformations of input code.


```
Feature request: Discard terminal autosuggestion on `Esc`

In the IPython terminal, when an autosuggestion hint (grey completion text from history) is visible after the cursor, there is currently no way to dismiss it without either accepting at least part of the suggestion or altering the buffer. We would like pressing the `Escape` key to discard the suggestion entirely.

Acceptance criteria

- If a suggestion is visible, pressing `Esc` should clear the hint immediately, leaving the current buffer content and cursor position exactly as they were.
- If no suggestion is present, `Esc` must behave normally (e.g., dismiss completion menus, trigger other documented bindings). It must not affect the buffer.
- The discard must happen on the `Esc` key press itself, without waiting for possible multi-key sequences that begin with `Esc`. This avoids introducing a delay for other chords and ensures the user sees the hint disappear instantly when they press the key.
- The behavior should be consistent regardless of what the buffer already contains (empty, partial match, or full match against the suggestion). No text from the suggestion should appear in the buffer or inserted as side effect.

This applies to IPythonʼs terminal interface, not the notebook.
```


```
Autosuggest (NavigableAutoSuggestFromHistory) in multiline terminal input has two issues:

1. up/down always swap suggestions, overriding line movement. In multiline inputs, up/down should only cycle through suggestions when the cursor is at the very top or bottom edge of the document; otherwise they should move the cursor normally between lines.

2. Suggestions are only displayed on the last line. When the cursor is on an earlier line, the suggestion for that line should be shown at the end of that line. Use a prompt_toolkit `Processor` to append the suggestion text because the toolkit’s built‑in suggestion rendering only covers the last line.

Also, the suggestion query must use the text of the line where the cursor sits, not the whole input’s last line.

These behaviors should activate only when the autosuggest provider is the default `NavigableAutoSuggestFromHistory`, leaving other providers unaffected.
```

Autosuggest: only navigate on edges of doc, show in multi-line doc


Change `install_nbextension` and the `ipython install-nbextension` CLI to support installing only a single extension at a time – a file, directory, zip/tarball archive, or URL. Remove the ability to pass lists or dicts of multiple paths; passing a list/tuple now should raise a `TypeError`.

Add an explicit `destination` argument to control the installed name independently from the source basename. When a `destination` is given for a non-archive source, install to that name instead of deriving it from the source path. Archives must not accept a `destination` – raise a `ValueError` when both are specified, because the archive determines its own contents.

For URLs, download the resource and then install the local copy using the same single-install logic, respecting `destination` if provided.

Update the `NBExtensionApp` class to expose the new `destination` option (e.g., `--destination`), adjust its description and usage examples to reflect the single-extension API, and wire it through to `install_nbextension`.

Existing error cases must still hold: symlinking from a URL or archive remains disallowed. The rest of the function’s parameters (`overwrite`, `symlink`, `user`, `prefix`, `nbextensions_dir`, `verbose`) should continue to work as before, just operating on a single source.

Change install_nbextension to take install only a single nbextension (file, folder, archive, url), with an optional destination argument


Add a `TransformerManager.check_complete(cell)` method that determines whether a code cell is syntactically ready to run, needs more input, or is invalid.

Return a `(status, indent_spaces)` tuple:

- `'complete'` — the cell is a valid, self-contained unit that can be executed.
- `'incomplete'` — more lines are needed to form a valid construct.
- `'invalid'` — the cell contains a syntax error.

When `status` is `'incomplete'`, `indent_spaces` should give the indentation (in space-equivalent characters) that the next line should use. After a colon that starts a block, use the block’s requested indentation. For other incomplete constructs (backslash continuation, open brackets/parens/braces, unterminated multi-line strings), provide the indentation level found on the last line, with tabs treated as 4 spaces.

The method must account for IPython-specific input processing:
- Apply the usual prompt stripping and other cleanup transforms so that the check works on raw interactive input.
- Handle cell magics (lines beginning with `%%`): the magic is considered complete when followed by a blank line, otherwise return `'incomplete'`.
- Reuse the existing input-transformation logic where possible, but do not permanently modify the cell.

For the final validity check, use a real Python parser fallback after all transformations have been applied.

Start adding code for checking when input is complete


**Ticket: Truncate CompositeError output when many engines fail**

When a parallel task fails across a large number of engines (e.g., a trivial `1/0` on 100 engines), the resulting `CompositeError` prints all engine exceptions in full, flooding the terminal. The output should be truncated to a reasonable number of tracebacks with a message indicating how many more exceptions exist.

**Desired behaviour**

- `CompositeError` should expose a configurable limit on how many tracebacks are shown (e.g., an attribute `tb_limit`). A sensible default would be `4`.
- The `__str__` method and the `render_traceback` method (when displaying *all* exceptions, i.e., `excid=None`) should honour this limit: only the first `tb_limit` engine exceptions are included in the output, followed by a line like `… X more exceptions …` to indicate how many were omitted.
- The `__repr__` may continue to show the total count and does not need to change.

Acceptance criteria (for a contributor implementing the change):
- Printing a `CompositeError` that wraps, say, 10 engine exceptions should show only 4 tracebacks plus a “more exceptions” message.
- The limit must be adjustable at runtime (e.g., `e.tb_limit = 10`) without requiring a code change.
- The existing `render_traceback(excid=N)` behaviour (showing a single engine’s traceback) must remain unaffected.

truncate potentially long CompositeErrors


**Title**: Formatter warnings should use a specific warning class instead of generic logging

**Description**

Currently, when a formatter’s `_repr_*_` method raises an exception (other than `NotImplementedError`) or returns an invalid type, IPython emits a warning. However, the warning is produced via the general-purpose logging/warning mechanism, which means it appears as a plain `WARNING` message with no dedicated warning category.

The practical problem is that users cannot easily suppress or filter these formatter warnings using Python’s built-in warning control system (`warnings.filterwarnings`, `warnings.simplefilter`, `-W` flag, etc.). The only way to silence them is to completely turn off the logging module’s output, which is too coarse and loses other important messages.

**Expected behavior**

1. Introduce a custom warning category specifically for formatter errors (for example, a subclass of `UserWarning`). The class should be defined somewhere accessible in `IPython.core.formatters` so it can be imported and filtered.

2. All warnings about formatter failures—both when a formatting method raises an unexpected exception and when it returns an invalid type—should be issued via `warnings.warn(..., category=NewCategory)` rather than via the generic logging-based warning.

3. The existing rule for `NotImplementedError` must be preserved: if a formatter raises `NotImplementedError`, no warning should be emitted at all. This is an explicit “not supported” signal, not an error.

4. The warning class should be a subclass of `UserWarning` (the standard base for user-visible warnings that are not `DeprecationWarning`, etc.) so that it appears by default in the warning stream and can be filtered using the usual patterns.

**Impact**

After this change, users who want to suppress all formatter warnings can do something like:

```python
import warnings
import IPython.core.formatters as fmt
warnings.simplefilter('ignore', fmt.FormatterWarning)
```

This is a targeted, standard Python idiom that gives users far more control than the current situation.

**Additional context**

The affected code is the decorator `warn_format_error` that wraps `__call__` on formatters, and possibly the base formatter classes. No behavioral changes are requested beyond the categorization of the emitted warnings.


**Title:** Autocall incorrectly processes Python string literal prefixes (raw, bytes, f‑string, unicode)

**Description:**
IPython’s autocall feature transforms a bare name followed by its arguments into a function call (e.g., `f x y` → `f(x,y)`). This causes a problem when the “name” is actually a Python string prefix such as `r`, `b`, `f`, `u`, `br`, `rb`, `fr`, or `rf`. For example, if no variable `r` exists, typing `r'a string'` triggers an error because IPython tries to autocall `r(''a string'')`. Even when a callable named `r` exists, autocall should **not** activate when the line starts with the prefix immediately followed by a single or double quote—it should be treated as a plain string literal.

**Expected behavior:**
- Lines that start with a valid Python 3 string prefix (`r`, `b`, `f`, `u` and their two‑character combinations) followed directly by `'` or `"` must **never** be rewritten by autocall. The line should be passed through as a normal expression, producing the expected literal string value.
- Normal autocall for a callable named `r` (or any other prefix name) must still work when the invoking syntax uses an explicit escape character (e.g., `,r arg`, `/r arg`, `;r arg`), because those escapes explicitly request autocall behavior.

**Reproduction (example):**
```python
In [1]: r'a string'
# Currently may raise NameError or attempt to call r('a string')
# Should simply return a raw string 'a string'
```

**Acceptance criteria:**
- `b'bytes'`, `f'{expr}'`, `u"unicode"`, `rb'data'`, `br'data'`, `fr'{x}'`, `rf'{x}'` are all left untouched by autocall.
- If a user defines `def r(x): ...`, then `r'hello'` still evaluates to the string `'hello'` (not a call to `r`), while `,r 1 2 3` still correctly autocalls `r(1,2,3)`.

**Note:**
This only affects lines where the prefix character(s) **immediately** precede a quote character; whitespace or any other character between the prefix and quote should not be treated specially.

Fixed #10322 - Made autocall to ignore raw & string literals


As a user of the notebook format conversion (`IPython.nbformat.v4.convert.upgrade` and `downgrade`), I need the v3 ↔ v4 conversion to be robust and schema‑compliant. Currently the conversion is incomplete and can produce invalid notebooks or silently drop data. Please harden the conversion so that:

1. **Validation**
   Before converting, validate the input notebook against its declared format. After converting, validate the output notebook against the target format. The conversion must never produce an invalid notebook.

2. **Metadata handling**
   - In the upgrade, store the original format version (`orig_nbformat`) inside the notebook’s `metadata` dictionary, not as a bare top‑level attribute, so the v4 schema is satisfied.
   - Remove the v3‑specific `name` metadata key during upgrade, as it does not belong in v4.
   - During downgrade, clean up any `orig_nbformat` and `orig_nbformat_minor` keys from metadata so the resulting v3 notebook remains valid.

3. **Code cell conversion**
   Upgrading a v3 code cell must:
   - Move `collapsed` into `metadata.collapsed` (v3 had it at the cell top level).
   - Remove the `language` key (v3 stored it on the cell; v4 does not).
   - Rename `input` → `source`.
   - Preserve `prompt_number` if missing by setting it to `None` so the cell structure remains consistent.

   Downgrading a v4 code cell must symmetrically:
   - Restore `language` (with a sensible default, for example `'python'`).
   - Rename `source` → `input`.
   - Move `collapsed` back from metadata to a top‑level `collapsed` key.

4. **HTML cells**
   If a v3 notebook happens to contain an `html` cell type, upgrade it to a `markdown` cell type, as v4 does not have a raw html cell type.

5. **Output conversion**
   Upgrading v3 outputs:
   - Rename `pyout` → `execute_result`.
   - For stream outputs, rename the `stream` key to `name`.
   - Convert mime‑type keys (e.g., `'png'` → `'image/png'`) on the output itself **and** on its metadata. Ensure output metadata is a proper notebook node (not a plain dict) to avoid key errors.
   - When `application/json` is present, parse it from a JSON string into a Python object (since v3 stored it as a string).
   - For binary image data (`image/png`, `image/jpeg`), promote `bytes` to unicode (ascii‑decode) to match v4 expectations.
   - Leave `error` outputs (`pyerr`) mapped to `error` unchanged, but ensure no other output type is mishandled (e.g., `display_data` should also go through mime‑key remapping).

   Downgrading v4 outputs must symmetrically:
   - Rename `execute_result` → `pyout`, `error` → `pyerr`.
   - For stream outputs, rename `name` → `stream` and drop the `metadata` key (v3 streams have no metadata).
   - Demux mime‑type keys back to short keys.
   - Serialize `application/json` objects back to a JSON string.

6. **Round‑trip safety**
   The final goal is that `downgrade(upgrade(v3_nb))` and `upgrade(downgrade(v4_nb))` produce notebooks that are valid according to the respective format schemas, with no structural data lost beyond intentional v3→v4 simplifications. Any conversion that cannot be performed should raise a clear error (e.g., converting from formats older than v3).

The conversion must not rely on callers pre‑cleaning notebooks; it should handle the full set of notebook features documented for v3 and v4, including the various output types and metadata structures.

test and update conversion between v3 <-> v4


Feature Request: Add metadata support to `display.JSON`

Currently, `display.JSON` only returns raw JSON data from `_repr_json_()`. There is no way to attach metadata such as an `expanded` flag, which Jupyter frontends (e.g., JupyterLab, Jupyter Notebook) can use to control whether a JSON output is expanded by default. This limits interactive exploration of structured data.

By contrast, `display.Image` already supports metadata (e.g., `width`, `height`, `unconfined`, and a generic `metadata` dict) by returning a `(data, metadata)` tuple from its `_repr_*_` methods. The `JSONFormatter` (`application/json`) already expects this tuple format and knows how to process it.

Please extend `display.JSON` so that users can:

- Pass an `expanded` boolean parameter (defaulting to `False`) in the constructor.
- Pass an optional `metadata` dictionary, which can be merged with the automatically generated metadata.

The `_repr_json_()` method should return a `(data, metadata)` tuple where `metadata` contains at least `'expanded': True/False`, and also includes any extra keys from the user-supplied `metadata` dict.

The existing behavior that warns when a serialized JSON string is passed (and then parses it) must be preserved.

feat(JSON): Add expanded metadata to JSON display class


The /files/ endpoint currently sets `Content-Disposition: attachment` unconditionally, forcing the browser to always download a file instead of displaying it inline. Change the behaviour so that:

- By default (no extra query parameter) file contents are served without a `Content-Disposition` header, allowing the browser to render them inline.
- When the request includes `?download=1`, set `Content-Disposition: attachment; filename="<original_filename>"` and preserve the download behaviour.

The notebook’s “Download as” menu action (the one that opens a URL for the current notebook file) must append `?download=1` so that downloading still works from that UI path.

use ?download=1 to trigger download in /files/


When I exit IPython, the following exception is raised during shutdown:

```
AttributeError: 'str' object has no attribute 'unlink'
```

The traceback points to `atexit_operations`, and the issue seems to stem from the history manager’s handling of the history file path. Some cleanup code expects the history file to be a `pathlib.Path` object, but in certain code paths it remains a plain string, so calling `.unlink()` on it fails.

**Expected behavior:** IPython should shut down cleanly without throwing an error, and any temporary or corrupt history files should be handled properly regardless of whether the `hist_file` configuration value was originally provided as a string.

**Context:**
- Users can configure the SQLite history file location via `ipython --HistoryManager.hist_file=/path/to/history.sqlite` or in their config file.
- The history manager and its helper routines perform filesystem operations on that path, including renaming corrupt databases and cleaning up during exit.
- The codebase is a mix of `os` functions (which work with strings) and `pathlib` methods (which require `Path` objects). This mismatch leads to the error when a string is passed to code that now expects a `Path`.

**Request:** Update the history subsystem so that:
1. The internal representation of the history file path is consistently a `pathlib.Path` object, while still allowing users and config to supply a string.
2. All filesystem operations on the history file (renames, stat checks, existence tests, etc.) use `pathlib` methods instead of `os`/`os.path` calls.
3. The special in-memory database marker `':memory:'` is still supported and does not cause path-related errors.
4. The fix should cover both `HistoryAccessor` and `HistoryManager`, and any downstream test infrastructure that sets `hist_file` programmatically.

This should eliminate the `AttributeError` on exit and ensure that history file maintenance is robust against type inconsistencies.


Currently, `guarded_eval` in `IPython.core.guarded_eval` enforces evaluation policies on item access (`__getitem__`) and attribute access (`__getattr__`), but does **not** guard binary, unary, or comparison operations. In a restricted evaluation mode (e.g., `"limited"`), expressions like `a + b`, `-x`, or `x == y` are executed directly without checking whether the objects involved and their dunder methods are permitted by the active policy.

This is a security gap: an attacker could supply an expression that dispatches to a custom `__add__`, `__eq__`, etc. on an object that should not be reachable in a limited context.

We need to add operation guards that mirror the existing `__getitem__`/`__getattr__` guards:

1. **Extend `EvaluationPolicy`**
   - Add an `allow_all_operations` flag (default `False`) and a `can_operate(dunders, a, b=None)` method.
   - When `allow_all_operations` is `True`, all operations are permitted (this should be the case for the `"unsafe"` policy).
   - Otherwise operations should be rejected by default.

2. **Extend `SelectivePolicy`**
   - Introduce `allowed_operations` (a set of types) and `allowed_operations_external` (for external libraries like pandas, same pattern as used for item/attr access).
   - Override `can_operate` to allow an operation only if **every** relevant dunder method on the operand is the original method from an allowed type (using the existing `_has_original_dunder` logic).
   - Handle the case where multiple dunders need to be checked (e.g., for `__eq__` and `__ne__` when type is not hashable, etc.).

3. **Update `eval_node`**
   - Replace the direct-execution branches for `ast.BinOp`, `ast.UnaryOp`, and `ast.Compare` with calls to `policy.can_operate(...)` before performing the operation.
   - For unary ops, call the appropriate dunder method on the operand.
   - For binary ops, call the dunder method on the left operand with the right operand.
   - For comparisons, handle chained comparisons correctly, and treat `is`/`is not` as identity checks (no guard needed) while `in`/`not in` should use `__contains__` with its own guard.

4. **Define allowed operations for “limited” mode**
   - The `"limited"` policy should allow operations on built‑in types that are already trusted for attribute access: `int`, `float`, `complex`, plus the existing `BUILTIN_GETATTR` types (like `list`, `dict`, `str`, `tuple`, etc.). This keeps basic arithmetic and comparisons on literals and built‑in containers working.
   - The `"minimal"` policy should keep `allow_all_operations=False` (i.e., reject all operations).
   - The `"unsafe"` policy should set `allow_all_operations=True`.

The change must not break any existing tests for guarded evaluation, and should guard against malicious custom dunder methods while allowing normal usage on standard built‑in types.

Add guards for binary, unary operators and comparators


When `FileLinks` (or `LocalDirectory`, which delegates to it) is instantiated with a directory path that ends in a trailing slash, the resulting HTML (notebook) and plain-text representations may contain double slashes (e.g., `mydir//` instead of `mydir/`).
Normalise the input path by stripping any trailing slash before it is used for formatting, and ensure that the output formatting always appends exactly one trailing slash to directory names so that the display is consistent regardless of whether the original path had a trailing slash. This should affect both the notebook `_repr_html_` output and the terminal `__repr__` output.

noew removes trailing slashes if they are provided in input so the output can be formatted consistently by always appending a trailing slash (which previously could result in two trailing slashes if the input path included a trailing slash)


**Title:** Speed up initial population of unicode name completions

**Description:**
IPython’s completion engine supports completing unicode names (e.g., `\N{...}` and backslash-latex-to-unicode) by building an internal list of all named Unicode characters the first time it is needed. Currently that list is built by naïvely iterating over *every* possible code point from 0 to 0x110000, calling `unicodedata.name()` on each. Because only about 10% of those code points actually have names, this takes far longer than necessary and causes a noticeable delay the first time a user triggers unicode completion in a session.

We want to make the initial population much faster by iterating only over the contiguous blocks of code points where names actually exist (skipping the large gaps with no named characters). The final list of names produced must be **identical** to what the exhaustive scan yields – not a single valid name may be lost.

The change should be limited to the internal caching of unicode names; no user-visible completion behavior or public API should change except for reduced first-use latency.


```
Fix HistoryAccessor.get_tail to be session-agnostic.

HistoryAccessor.get_tail raises AttributeError because it references
self.session_number, which only exists on the HistoryManager subclass.
The accessor should work without any current-session context.

Expected behavior:
- HistoryAccessor.get_tail returns the last n entries from the full
  database, regardless of session. Order: latest entry last.
- HistoryManager.get_tail should override this with the existing
  session-prioritizing behavior: entries from the current session
  appear last, giving a view that feels natural to the active shell.

Restore the previous session-agnostic implementation in the accessor
and keep the session-aware logic only in the manager.
```

Fix HistoryAccessor.get_tail bug (#13666)


**Prompt stripping fails when first line lacks initial prompt, and risks stripping valid `...` literals**

When pasting code with interactive prompts (classic `>>>`/`...` or IPython `In [n]:`/`...:`) into IPython, the prompt-stripping logic has two defects:

1. **Missing initial prompt on the first line** – If the copied block does _not_ contain an initial prompt on its first line (e.g. only continuation lines were copied), the continuation prompts (`...` or `...:`) are not removed from subsequent lines. This causes false syntax errors because the prompts remain in the pasted code.

2. **Ellipsis literal on the first line** – On the first line of a cell, the continuation-prompt pattern `...` is a valid Python expression (the Ellipsis literal in Python 3). The current logic may strip it, silently changing the meaning of the code.

**Expected behavior**

- **First line:** Only the *initial* prompt pattern (`>>>`, `In [n]:`) should be stripped. Continuation-prompt patterns (`...`, `...:`) must be preserved here because they may represent valid Python.
- **Second line:** *Any* prompt pattern (initial or continuation) should be stripped. This covers the copy-paste scenario where the initial prompt was omitted.
- **Once a prompt is detected** on either of the first two lines, both initial and continuation prompts should be stripped from all remaining lines of the block.

For the IPython prompt mode (`In [n]:` / `...:`) the continuation prompt is never valid Python, so a single combined regex is sufficient for all lines beyond the first; there is no need for a separate “initial-only” regex.

The overall goal is that pasting code snippets from interactive sessions (even when the first `>>>` or `In [1]:` is missing) reliably removes all prompts, while a standalone `...` on the first line of a cell is left untouched.

adjust strip_prompts logic a bit further


**Title:** `dir2` should not crash on objects whose attribute access raises non-`AttributeError` exceptions

**Description**
When introspecting objects with `dir2()`, the function currently uses standard `hasattr` checks and direct `getattr` calls on attributes such as `__class__`, `__bases__`, `trait_names`, and `_getAttributeNames`. Python's `hasattr` only suppresses `AttributeError`; any other exception raised by the object’s `__getattr__`, a property, or similar mechanism will propagate out of `dir2` and crash the caller. This makes `dir2` unsafe for general-purpose introspection where objects may be poorly behaved (for example, a property that raises `KeyboardInterrupt`, or a `__getattr__` that raises `KeyError`).

Expected behavior:
- Calling `dir2(obj)` must never propagate an exception raised by attribute access on `obj`.
- When an attribute used for introspection (e.g., `__class__`, `__bases__`, `trait_names`, `_getAttributeNames`) cannot be safely accessed, the function should silently skip that attribute and continue collecting whatever attribute names it can.
- The `get_class_members` helper, which also uses `hasattr`, must be similarly robust.
- All existing behavior for well-behaved objects must be preserved; `dir2` must still return the same set of attribute strings it does today for normal objects.

**Affected code area**
The issue is in `IPython/utils/dir2.py`, particularly the `dir2` function and `get_class_members`. Any other internal callers that rely on `dir2`’s exception safety are also affected if they expect it to never raise.


**Title:** Add checkpoint support for non-notebook files in the contents API

**Background**
The IPython notebook server’s contents API provides checkpoint endpoints (`/api/contents/{path}/checkpoints`) for creating, listing, restoring, and deleting checkpoints. Currently these endpoints only function correctly for notebook files (type `notebook`); they fail or produce unusable results when the target path is a generic file (type `file`), such as a plain text or binary file. There is no way to checkpoint arbitrary files through the API.

**Request**
Extend the contents API’s checkpoint machinery so that every file type supported by the contents manager—specifically `notebook` *and* `file`—can be check-pointed and restored via the same set of endpoints.

**Concrete behavior**

1. **Create a checkpoint**
   `POST /api/contents/{path}/checkpoints` should inspect the file type, save a snapshot of the file’s current content, and return a checkpoint info object (id, last_modified).
   - For a notebook, behavior should be unchanged.
   - For a file, the checkpoint must capture the exact content and the encoding format (`text` or `base64`) that the file was served with at the moment the checkpoint was created.

2. **List checkpoints**
   `GET /api/contents/{path}/checkpoints` should return all available checkpoints for the given path regardless of file type. The response format stays the same.

3. **Restore a checkpoint**
   `POST /api/contents/{path}/checkpoints/{checkpoint_id}/restore` should restore the file to the state captured in the checkpoint.
   - After restore, reading the file should return the checkpointed content with the same content type and encoding format that were captured.
   - Restoring a notebook checkpoint must continue to work identically to the current implementation (including validation if applicable).

4. **Delete a checkpoint**
   `DELETE /api/contents/{path}/checkpoints/{checkpoint_id}` should work identically for any file type.

**Safety / edge cases**
- The existing checkpoint format and storage location for notebooks must not be broken by this change.
- Checkpointing a binary file (base64) must correctly store and restore the raw bytes without corruption.
- If a file’s content cannot be decoded as UTF-8 when the format is forced to `text`, appropriate HTTP errors should be raised (as already done for normal file reads).
- The implementation should gracefully handle the situation where a checkpoint exists for a file that has since changed type (e.g., a notebook was replaced by a plain file on disk), by using the current file type when deciding how to read the checkpoint back.
- The checkpoint directory naming (`.ipynb_checkpoints`) should remain unchanged for all file types to keep the on-disk layout predictable.

**Out of scope**
- Changing the public API surface or adding new endpoints.
- Checkpointing directories (`type: "directory"`).

The goal is that a user or frontend client can call the same checkpoint API for any writable file and get consistent, reliable behavior.

DEV: Add full support for non-notebook checkpoints.


```
Title: Remove usage of deprecated `InteractiveShell.magic()` method

Background
----------
IPython’s `InteractiveShell.magic()` method has been deprecated for years (accepts a single
string representing the full magic command line). It should no longer be used internally.
The supported replacement for line magics is `InteractiveShell.run_line_magic(magic_name, line)`,
where the magic name and its arguments are passed as two separate strings.

Task
----
Update **all** internal callers across the entire IPython codebase (production code and
test suites) to stop using `.magic()` and instead use `run_line_magic()` (or another
appropriate, non-deprecated method when the magic is not a line magic – though all
current internal uses appear to be line magics).

Examples of required transformations
------------------------------------
- `shell.magic('xmode ' + dstore.xmode)`  →  `shell.run_line_magic("xmode", dstore.xmode)`
- `_ip.magic("prun -q x = '\\t'")`        →  `_ip.run_line_magic("prun", "-q x = '\\t'")`
- `ip.magic("run " + fname)`              →  `ip.run_line_magic("run", fname)`
- `_ip.magic('logstart -to %s' % logfname)` → `_ip.run_line_magic("logstart", "-to %s" % logfname)`

Special care
------------
1. **When `.magic()` is passed as a callable** (e.g., `pytest.raises(..., _ip.magic, ...)`),
   the callable must become `_ip.run_line_magic` and the arguments must be adjusted so that
   the magic name and the line are two separate arguments.
   For example:
   `pytest.raises(ImportError, _ip.magic, "load_ext daft_extension")`
   → `pytest.raises(ImportError, _ip.run_line_magic, "load_ext", "daft_extension")`.

2. **String formatting / concatenation** that used to build a single magic line must be
   separated: extract the magic name as a literal first argument and pass everything else
   as the line argument. Do not keep the old concatenation inside a single string.

3. When a magic has no arguments, pass an empty string for the line:
   `run_line_magic("magic_name", "")`.

4. The change must be **exactly behavioural-preserving**. No functional changes are
   intended, only the removal of the deprecated API.

5. The fix should cover every occurrence in the repository, including inside test files,
   so that the test suite continues to pass without any uses of the deprecated method.

Reference
---------
See the original deprecation of `InteractiveShell.magic()` and the introduction of
`run_line_magic` / `run_cell_magic` as the supported public API.
```

do not use .magic(...) that has been deprecated for a decade


**Feature request: Support notebook URLs without .ipynb extension**

Currently the notebook server requires that URLs under `/notebooks/` and `/nbconvert/<format>/` explicitly end with `.ipynb`. This is unnecessarily strict. Some deployments use custom filesystem layouts (via `ContentsManager`) where notebook files may have other extensions or no extension at all. The server should determine whether a path represents a notebook by asking the `ContentsManager`, not by matching a file extension in the URL router.

**Acceptance criteria**

1. **General path handling**
   The URL patterns for notebook, file, and nbconvert handlers should use the same generic path regex for all filesystem paths. There should be no separate “notebook path” regex that enforces an `.ipynb` extension. Any valid file path that the contents manager can resolve should be reachable under `/notebooks/` and `/nbconvert/<format>/`.

2. **Notebook view (`/notebooks/...`)**
   - If the contents manager reports `type: "notebook"` for the given path, the notebook UI is served.
   - If the path exists but is **not** a notebook (e.g., a CSV or plain text file), the server redirects the request to the file‑view endpoint.
   - If the path does not exist, a 404 is returned (unchanged behavior).

3. **Nbconvert `GET` (`/nbconvert/<format>/...`)**
   - If the contents manager reports `type: "notebook"`, the conversion proceeds as before.
   - If the path exists but is **not** a notebook, the server returns an HTTP 400 (Bad Request) with a message indicating the path is not a notebook.
   - Non‑existent paths continue to result in a 404.

4. **Backward compatibility**
   Existing URLs that include the `.ipynb` extension must continue to work normally, since they are a subset of the valid paths.

The goal is to let the `ContentsManager` be the single source of truth for what is a notebook, rather than hard‑coding that decision into the URL routing layer.


The pretty printer currently has redundant special-case logic inside its printer factories for sequences, sets, and dicts: each factory checks whether a subclass of the built‑in type overrides `__repr__` and, if so, uses that `__repr__` output. Because a recent change to the dispatch mechanism already ensures that a type’s custom `__repr__` is respected automatically, these manual checks are no longer needed and can be removed.

Additionally, when pretty-printing an empty set or frozenset, the output should use the actual type name (for example, `MySet()` when `MySet` is a subclass of `set`) rather than hard‑coding the base type name.

Simplify the three factory functions (`_seq_pprinter_factory`, `_set_pprinter_factory`, and `_dict_pprinter_factory`) by removing the `basetype` parameter and the associated `__repr__`‑comparison branch. Update the registration table (`_type_pprinters`) to drop the `basetype` argument, and adjust the empty‑set special case to use `type(obj).__name__` instead of `basetype.__name__`. No other behaviour changes are expected, and the existing pretty‑printing output for all built‑in collections should remain unchanged except for the empty‑set naming improvement.

Remove unnecessary code checking for __repr__ on a subclass of builtin types


```
Title: line_at_cursor returns wrong line offset when cursor is at end of cell without trailing newline

The `line_at_cursor()` utility in `IPython/utils/tokenutil.py` is supposed to map a cursor position to the corresponding line and its start offset within the cell.
However, when the cursor is **at the end of the cell** (i.e., position equal to `len(cell)`) and the last line does **not** end with a trailing newline, the function currently advances past that last line and returns the end-of-cell offset instead of the start offset of the final line.

**Expected behavior:**
A cursor at the very end of the cell should still be considered to be on the last line (just after its final character). The returned offset must be the start of that last line, not the offset that lies beyond it.
This mirrors how a cursor positioned right after a newline character is treated as the beginning of the next line – the end of the cell should act like an implicit newline for the purpose of determining line membership.

**Affected scenario:**
Any cell where the last line does not end with `\n` and the cursor is placed at position `len(cell)`. Calls with cursor positions inside the cell (including on trailing-newline boundaries) should remain unchanged.

**What to do:**
Adjust `line_at_cursor` so that a line without a trailing newline is treated as if it extended one character beyond its content, thereby keeping the cursor at the cell’s end on the final line and returning its correct starting offset.

No other public API or behavior should change. Ensure the fix handles empty cells gracefully and does not alter results for cells whose last line ends with `\n`.
```

Fix line_at_cursor for end of last line without trailing newline


**Description**
When IPython’s completion engine evaluates a method call to offer attribute completions on the result, assignments to `self.<attr>` inside the method body are not taken into account. This means that after a method returns `self.<attr>` (or a chain of such accesses), tab completion fails to suggest the correct attributes/methods of the returned object.

**Steps to Reproduce**
1. Start IPython with default settings (`evaluation = 'limited'`).
2. Define a simple class:
   ```python
   class Container:
       def build(self):
           self.items = []
           return self.items
   c = Container()
   ```
3. Type `c.build().` and press Tab.
4. Alternatively, use a method that returns a plain `list` without involving `self` and notice completions appear normally for that case.

**Actual Behavior**
After typing `c.build().`, no completions are offered (e.g., `append`, `extend`, etc. are missing). The completion behaves as if the return type is unknown.

**Expected Behavior**
The completer should recognise that `self.items` has been assigned a `list` inside `build` and therefore provide list-attribute completions on the result of `c.build().`.

**Context**
- This missing behaviour limits the usefulness of greedy/limited evaluation for completions in object-oriented code, where returning instance attributes is a common pattern.
- The problem should also be fixed for unpacking assignments to `self` attributes (e.g., `self.a, self.b = …`) and for starred assignments, so that any case where an instance attribute is definitely assigned inside the method is reflected when that method’s return value is completed.
- The fix should not suddenly cause spurious completions for attributes that are only conditionally assigned, but it is acceptable to be conservative: if an instance attribute is assigned along *any* path, it should contribute to the available completions for the return expression.
- Debugging artifacts (e.g., `print` statements) should not appear during completion evaluation.

**Impact**
Users writing object-oriented code with methods that return instance attributes will get a degraded completion experience, forcing them to manually check the method’s return type instead of relying on tab completion.


### Feature Request: Syntax highlighting for cells using language magics (e.g. `%%R`, `%%bash`) in exported HTML and LaTeX

#### Problem
When exporting a notebook that uses cell-level magic extensions to write code in other languages (e.g. `%%R`, `%%octave`, `%%bash`, `%%perl`, `%%ruby`), the generated HTML and LaTeX output highlights those cells as Python, ignoring the actual language. This makes the output confusing to read and does not match what a user expects from an enriched notebook.

#### Desired solution
Add a new **preprocessor** to `nbconvert` that automatically detects code cells that start with a supported language-magic directive, and tags each such cell’s metadata with the corresponding Pygments lexer identifier (e.g. `"r"` for `%%R`, `"bash"` for `%%bash`, etc.). The built-in highlighting filter (used by HTML and LaTeX templates) should then be able to pick up that metadata and apply the correct syntax highlighting.

#### Detailed requirements

1. **Default enablement** – The preprocessor must be **enabled by default for the HTML and LaTeX exporters** (it may remain available but disabled for other exporters). Users should not need to modify configuration to get correct highlighting in those two output formats.

2. **Detection rules** – A cell is considered to be using a foreign language only when the magic directive (e.g. `%%R`) appears **at the very beginning of the cell content, optionally preceded by whitespace**. The same text appearing inside a string literal, comment, or elsewhere deeper in the cell must **not** trigger tagging (false positives must be avoided).

3. **Configurable language mapping** – The mapping from magic directives to Pygments lexer names should be **configurable** (e.g. through `traitlets` / config). A sensible default set of common magic languages must be provided (at minimum `%%R`, `%%bash`, `%%octave`, `%%perl`, `%%ruby`).

4. **Metadata key** – The preprocessor must write the lexer identifier into the cell’s metadata under the key that the highlight filter already uses (or under a well-defined key that the templates’ highlight filter expects), so that no template changes are required.

#### Context
The `nbconvert` tool already has a highlight filter that uses Pygments to apply syntax highlighting based on a cell’s metadata. Other preprocessors have been added for similar annotation purposes (e.g. `LatexPreprocessor`, `CSSHTMLHeaderPreprocessor`). This request extends that pattern to make language magics work transparently in exported documents.