**Feature request: Add first/last seen timestamps to peer statistics**

The peer statistics tracked by the `PeerStatisticsActor` should expose two new per‑peer timestamps:

- `firstSeenTimeMillis` – the wall‑clock time (in milliseconds since epoch) when the node first observed a message from that peer.
- `lastSeenTimeMillis` – the wall‑clock time of the most recent message observed from that peer.

Both times should be represented as `Option[Long]` and be absent (`None`) for a peer that has never sent a message.

**Behaviour requirements**

1. **Recording**
   - Whenever a `MessageFromPeer` event is processed, the current system time must be captured and used to set both timestamps **for that individual observation** (first and last equal to the same instant).
   - The `empty` stat for a peer (before any messages) must have both timestamps absent.

2. **Aggregation across time windows**
   - Statistics are aggregated over sliding time windows. The merging logic must combine the timestamps correctly:
     - The merged `firstSeenTimeMillis` is the **earliest** of all observations across the merged windows.
     - The merged `lastSeenTimeMillis` is the **latest** of all observations across the merged windows.
     - If either side has a timestamp absent, the other side’s value is used unchanged.

3. **Stability**
   - Once a peer has been observed, its `firstSeenTimeMillis` must never increase and its `lastSeenTimeMillis` must never decrease as more windows are merged into the overall statistics.

4. **Peer removal**
   - When a peer disconnect event is processed, the timestamps should no longer be returned for that peer (consistent with the current removal behaviour for other stats).

**External view**
The change is visible through the `StatsForAll` response, which should include the new fields in the `Stat` values for each peer.

This will allow monitoring tools to see both how long ago a peer was first discovered and when it was last active.

ETCM-463: Add first/last seen timestamps.


**Description:**
The `debug_listPeersInfo` JSON-RPC method currently fails or hangs (produces an error response or times out) when any connected peer does not yet have associated peer info. This can happen for peers that are still handshaking or have not completed the protocol negotiation.

**Acceptance Criteria:**

1. When no peers are connected, the method must return a successful response with an empty list of peer information.
2. When one or more peers are connected but some of them have no peer info available, the method must return a successful response containing only the peer info entries that *are* available, omitting those without info.
3. The method must never throw an exception or return an error solely because peer info is absent for a subset of connected peers.
4. The response format (list of peer info objects) must remain unchanged; consumers should simply receive a filtered list.

**Additional context:**
The method `debug_listPeersInfo` is part of the debug namespace. The fix should ensure that the implementation gracefully handles missing peer info without changing the public response contract.

Add DebugServiceSpec, change return type for getPeerInfo


**Request: Be strict by default about the number of items in RLP lists for discovery protocol messages**

The current RLP decoding for Ethereum discovery protocol structures (`Node`, `Node.Address`, etc.) is too lenient: it accepts lists with extra trailing items and silently ignores them. This can hide protocol mismatches and data corruption. We want decoding to fail early when a list has too many elements.

**Desired behavior**

1. **Node.Address** – An RLP list representing a `Node.Address` has exactly three items (IP, UDP port, TCP port). Decoding a list with more than three items should throw an `RLPException`.
2. **Node** – An RLP list representing a `Node` has exactly four items (three address fields + public key). Decoding a list with more than four items should throw an `RLPException`.
3. **Discovery payloads** (`Ping`, `Pong`, `FindNode`, `Neighbors`, `ENRRequest`, `ENRResponse`) – These must remain **permissive**: extra trailing items are ignored to preserve forward-compatibility and pass test vectors with extra data (e.g., EIP‑8).
4. **Default derivation policy** – The generic case-class RLP list decoder, used outside the payload scope, should reject unexpected trailing items by default.

**Implementation notes**

- The existing `DerivationPolicy.omitTrailingOptionals` default is `true` (which causes the permissive behaviour). Change the default to `false`.
- The payload codecs in the discovery module should explicitly override the derivation policy to retain `omitTrailingOptionals = true`.
- The custom `Node` and `Node.Address` decoders in `RLPCodecs` must be tightened from lenient item-count checks (e.g., `>= 4`) to exact item-count checks.

**Acceptance criteria**

- Decoding a `Node.Address` from a 4-item RLP list raises an error.
- Decoding a `Node` from a 5-item RLP list raises an error.
- Decoding any discovery payload from a list with extra trailing items still succeeds.
- No regressions in round-trip encode/decode of well-formed discovery messages.

ETCM-168: Be strict by default with the RLPList item numbers.


```
Fix eth_getStorageAt to respect ethCompatibleStorage configuration

The eth_getStorageAt RPC endpoint currently ignores the
ethCompatibleStorage blockchain config setting when reading contract
storage positions. This leads to incorrect results on chains that use
ArbitraryIntegerMpt (ethCompatibleStorage = false) because the hardcoded
EthereumUInt256Mpt serializes keys differently, causing lookup failures
or wrong values for storage slots that were written correctly by EVM
execution (which already branches on this flag in
InMemoryWorldStateProxy).

Make Blockchain.getAccountStorageAt accept an ethCompatibleStorage
parameter and select the appropriate MPT implementation
(EthereumUInt256Mpt vs ArbitraryIntegerMpt), then thread
blockchainConfig.ethCompatibleStorage from EthService.getStorageAt into
that call so the read path matches the write path.
```


We need to clean up block validation and import safety:

- Block validation logic (`validateBlockBeforeExecution`, `validateBlockAfterExecution`, along with helpers `getBlockHeaderFromChainOrQueue`, `getNBlocksBackFromChainOrQueue`) is currently split between `LedgerImpl` and `BlockExecution`. Move all validation routines into a single, dedicated validation class. `LedgerImpl` and `BlockExecution` should delegate to it.

- In `BlockImport`, validation futures currently use the general `blockExecutionContext`. Use a separate validation context (passed explicitly) for any future that performs block validation, keeping it distinct from execution futures.

- The debug message in `importBlockToTop` (the one that logs imported blocks) eagerly builds the string even when debug logging is disabled. Make it lazy so that the string concatenation only happens when debug is enabled.

- Remove all unsafe `.get` calls from block import and validation paths (especially in `reorganiseChainFromQueue`, `removeBlocksUntil`, and the caller of `getTotalDifficultyByHash` in `importBlock`/reorganisation). Handle `Option` results safely without risking `NoSuchElementException` crashes. If a required block or receipt is unexpectedly missing, propagate the failure as a suitable import error rather than throwing.


**Title:** Faucet RPC methods must return proper JSON-RPC errors when the faucet handler actor cannot be resolved

**Context**
The application exposes two JSON‑RPC methods backed by `FaucetRpcService`:
- `faucet_sendFunds`
- `faucet_status`

Under the hood, each method obtains a reference to a “faucet handler” actor via `faucetHandler()` (a `Task` that resolves to an `ActorRef`). That task includes retry logic and can fail if the actor system has not (yet) started the handler, the actor path is unreachable, or the actor resolution times out.

**Observed problem**
When `faucetHandler()` fails for any reason, the RPC call does **not** return a valid JSON‑RPC error. Instead the caller receives an unhandled task failure (e.g. a raw exception, a timeout, or a generic crash), which breaks the JSON‑RPC contract.

**Desired behaviour**
1. **Always a JSON‑RPC error** – If `faucetHandler()` fails (any exception), both `faucet_sendFunds` and `faucet_status` must recover and return a JSON‑RPC error with code `-32603` (Internal error). The error body can be generic, but the service must log the original failure for diagnostics.
2. **Existing application‑level errors untouched** – The known error responses that arise **after** the handler actor is successfully obtained (`FaucetIsUnavailable`, `WalletRpcClientError`) must remain mapped exactly as they are today, with their current JSON‑RPC error codes and messages.
3. **Success path unchanged** – When the handler actor is available and the operation succeeds, the success response must be returned unaltered.
4. **Consistent recovery** – The same recovery logic must apply to both `faucet_sendFunds` and `faucet_status`.

**Acceptance criteria (user‑facing)**
- Call `faucet_sendFunds` (valid address) while the faucet handler actor is intentionally unavailable or the resolution times out.
  → The response is a JSON‑RPC error with code `-32603`. No raw exception or timeout reaches the client.
- Call `faucet_status` under the same condition.
  → Same guarantee: JSON‑RPC error `-32603`.
- Call either method when the handler actor is available but replies with `FaucetIsUnavailable` or `WalletRpcClientError`.
  → The original error codes (“Faucet is unavailable…” / “Faucet error: …”) are still returned, not an internal error.
- Call either method when everything works correctly.
  → The expected success response is returned, no regressions.

**Notes**
- The `faucetHandler()` task is defined in the `FaucetHandlerBuilder` trait and may throw/timeout after retries are exhausted. The solution must handle that failure without changing the retry/actor‑selection logic itself.
- Logging the raw failure before returning the internal error is advisable so operators can diagnose why the handler was unreachable.


Fix touched account handling to match ETC‑chain consensus after the EIP‑161 workaround

When a contract‑to‑contract call fails (e.g. revert, out‑of‑gas), the touched accounts from the failed subcall must not be merged into the caller’s world state.  The only exception is the RIPEMD precompile at address `0x0000000000000000000000000000000000000003`: if that address was touched during a call that then fails, it must be preserved in the parent’s touched set.  This is required to avoid a chain‑split on Ethereum Classic (see EIP‑716 and the handling of tx `0xcf416c…` in geth/parity).

Separately, during transaction gas payment after execution:
- The gas refund to the sender **must not** touch the sender’s address (to prevent an empty sender account from being deleted by EIP‑161).
- The gas payment to the beneficiary/miner **must** touch the miner’s address.

Currently the client unconditionally merges all touched accounts from failed subcalls and unconditionally touches the sender on refund, which can lead to unintended deletion of accounts.  Please update the VM call‑opcode logic and the transaction‑execution gas‑payment logic to implement the rules above.


```
Improve state sync downloader latency and add processing statistics.

The current downloader uses a 50 ms timer (CheckForPeers) to re‑attempt
assigning work after a peer response or failure, causing unnecessary
delay.  Instead, after handling a response/failure, if pending nodes
remain, immediately trigger re‑assignment of work to free peers without
waiting.  This can be accomplished by sending the existing GetMissingNodes
message with an empty list when work is queued.

Also enforce that peer download requests always contain at least one
node hash – empty requests should never be created.

On the scheduler side, collect counts of duplicate responses, unsolicited
(not‑requested) responses, and total nodes saved.  Log a summary of these
counters every 30 seconds.  Reduce the log level of frequent sync progress
messages (e.g. node reception, batch persistence) from INFO to DEBUG to
reduce noise.
```


Update the external VM handshake to use the new fields from the bumped protobuf definition.

The protobuf `EthereumConfig` message now carries `byzantiumBlockNumber`, `constantinopleBlockNumber`, `petersburgBlockNumber`, `istanbulBlockNumber`, and `chainId`.

Previously the server hardcoded placeholder fork block numbers and a fixed chainId (`0x3d`) with TODO comments indicating these should eventually come from the protobuf message. The client did not include these fields in the handshake.

After this change:
- The client must populate and send these fields from the local `BlockchainConfig` when building the `Hello` message.
- The server must read these fields from the received `EthereumConfig` and use them when constructing its `BlockchainConfig`, replacing the hardcoded values.
- The `chainId` should be handled according to its protobuf type (`bytes`); for the handshake, map `BlockchainConfig.chainId` (a single `Byte`) to/from a `ByteString`.


**Title:** MerklePatriciaTrie: persistent-storage regression and RLP round-trip inconsistency in node encoding

**Background**
`MerklePatriciaTrie` is our Merkle-Patricia trie used for Ethereum state and storage.  MPT nodes are decoded from RLP and later re-encoded when the trie is persisted.  The current code has two problems:

1. **Shared mutable storage.**  `put` and `remove` call `updateNodesInStorage` which modifies the `nodeStorage` in place, but the new `MerklePatriciaTrie` instance is constructed with the *old* storage handle.  This means further operations on the returned trie see unexpected state, and objects that appear to be independent share the same underlying storage, breaking immutability.

2. **Inconsistent re-encoding of already-decoded nodes.**  Node decoding, encoding, and the “cap large nodes with a hash” logic are duplicated between `MerklePatriciaTrie` (implicit RLP encoder/decoder, private `encodeNode`) and `MptNode`.  Once a node is decoded from RLP, the original RLP structure is lost; re-encoding can produce a byte-for-byte different representation (e.g. because of children that were represented as an embedded-node vs a hash, or different field ordering), causing hash mismatches.  This also makes the code fragile for future RLP changes.

**Desired behaviour**
- `put` and `remove` must return a new trie that owns a *post-update* snapshot of the storage.  `updateNodesInStorage` (or its replacement) should return both the new root hash and the new storage, and the callers must use that returned storage.
- RLP encoding of a node that was created by decoding an RLP representation should re-produce the identical bytes whenever possible, by preserving the original RLP structure on the node object.  This applies to `LeafNode`, `ExtensionNode`, `BranchNode`, and `HashNode` (though `HashNode` is always a 32-byte value).  Encoding of nodes that were constructed in memory (e.g. during trie updates) should still be correct and consistent.
- The encoding logic (including the decision to replace a large node encoding with its keccak hash) should be centralized, eliminating duplication between the trie and the networking layer (`PV63` message encoders/decoders).

**Edge cases / constraints**
- Do not break the Ethereum wire protocol: `PV63.GetNodeData`/`NodeData` messages must send/receive correctly encoded MPT nodes.
- The `updateNodesInStorage`/`encodeNode` contract must handle the empty trie (root hash `EmptyRootHash`) correctly.
- The change must not affect the on-disk format for nodes already stored by a previous version (they are pure RLP and should be decoded and re-encoded identically).
- `BranchNode`, `ExtensionNode`, `LeafNode` equality semantics should be based on the node hash, not array content, to avoid broken set/map behaviour; this behaviour should be lifted to `MptNode` so it is uniform.


**Issue:** `getBestBlock()` occasionally throws a `NoSuchElementException` after a chain reorganization that replaces a longer chain with a shorter but heavier branch. The in-memory best‑block cache becomes inconsistent with the persistent store; the cache still refers to a block that has been removed, while the database only contains the new chain.

**Requirement:** Fix the ordering of best‑block metadata updates relative to database commits during block persistence (both in the normal `Blockchain.save` path and in the fast‑sync block‑saving logic) so that the cached best‑block number always points to a block that exists in the database.


**Title:** Split the Consensus miner API into fire-and-forget and request-response operations

**Context**

The `Consensus` trait currently exposes a single `sendMiner` method that returns `Task[MinerResponse]`. This method is used for two different kinds of interactions with the internal miner:

1. **Fire-and-forget** – messages like `StartMining` and `StopMining`, where the caller only needs to deliver a command and does not expect a reply.
2. **Request-response** – messages like `MineBlocks` (from `MockedMinerProtocol`), where the caller (`QAService.mineBlocks`) needs the resulting `MinerResponse` to produce an RPC result.

**Problem**

Conflating these two patterns causes several issues:
- Every caller gets a `Task[MinerResponse]` even when no response is needed, forcing the creation and eventual discard of a `Task` that may never complete.
- The intent is unclear: from the signature alone it is impossible to know whether the caller expects a reply.
- The implementation in `EthashConsensus` uses `askFor` internally, which always expects a response, making the fire-and-forget usage (especially for `StartMining`) semantically awkward.

**Request**

Separate the two concerns by introducing a clear distinction in the `Consensus` interface and all its implementations (`EthashConsensus`, `TestmodeConsensus`). Specifically:

- Keep a method that **sends** a miner protocol message without waiting for a response. It should return `Unit` and be suitable for commands like `StartMining` and `StopMining`. In `EthashConsensus` this would typically deliver the message using actor `!` (fire-and-forget), and in `TestmodeConsensus` it can be a no-op.
- Add a method that **asks** the miner and returns a `Task[MinerResponse]`. This should be used for messages like `MineBlocks` where a reply is required. Its `EthashConsensus` implementation would use `askFor`, and `TestmodeConsensus` would return `Task.now(MinerNotExist)` as before.
- Update `QAService.mineBlocks` to call the **ask** variant instead of the current `sendMiner`, because it depends on the response.
- Update all other existing callers of `sendMiner` that only send commands to use the fire-and-forget variant.

Make sure the behaviour for the request-response path remains identical to the current `sendMiner` flow for callers that need a response. The fire-and-forget path must not introduce blocking or result in unobserved `Task` failures.

[FIX] divided sendMiner into askMiner and sendMiner


When exporting a notebook with the `%notebook` magic, outputs that have multiple MIME representations (e.g., a `display_data` output containing both `text/plain` and `text/html`) are being split into separate notebook output cells. Per the notebook format, a single `display_data` or `execute_result` should appear as one output cell whose data dictionary holds all MIME types. Instead, each MIME key becomes its own output, producing invalid notebooks. Similarly, stream outputs (`stdout`/`stderr`) that arrived as multiple text chunks are emitted as multiple stream outputs rather than merged into one.

Observed behavior:
- A `display_data` with `{"text/plain": ..., "text/html": ...}` yields two separate display_data cells, which breaks notebook validation and rendering.
- Stream outputs produce duplicate entries when the stored history output bundles multiple pieces of text.

Desired behavior:
- All MIME types from a single `execute_result` or `display_data` should be collected into one output cell with a `data` dictionary mapping each MIME to its content.
- All text chunks for a single stream output should be combined into one `v4.new_output("stream", text=...)` call, preserving the original order and channel.
- The fix should handle all output types stored in `HistoryManager.outputs` without failing on unrecognized output types.

As part of ensuring correctness, the existing test `test_notebook_export_single_display` leaks history state into subsequent tests. Its cleanup currently calls `_ip.history_manager.reset()`, but this does not properly restore the `outputs` dictionary and `execution_count` attribute. The test must save and restore the original `outputs` and `execution_count` after execution so that it does not pollute the shared test shell's state.


```
The autoreload extension's deduper reloader reads module source files to perform hot-patching. Currently it assumes all sources are UTF-8 encoded. However, the import system respects PEP 263 coding cookies (e.g., `# -*- coding: latin-1 -*-`). When a module uses a non-UTF-8 encoding, reading the source with an explicit `encoding='utf-8'` raises a UnicodeDecodeError, which is caught and logged, leaving the source empty and disabling hot-patching for that module, falling back to a full reload. Additionally, the logged traceback is noisy on the first `%load_ext autoreload`.

Fix the source-file reading in the deduper reloader so that it honors PEP 263 encoding declarations, matching how the import system reads the file. This will allow modules with non-UTF-8 encoding cookies to be hot-patched and eliminate the noisy error logs.
```

Use tokenize.open to read source files in deduperreload


```
Title: IPython applies non-Python transforms inside incomplete multi-line Python statements

Report
------
While typing a multi-line Python statement (list, dict, or bracketed expression) at an IPython prompt, I see that magic commands, system commands, help shortcuts, and custom TokenInputTransformer-based transforms fire on partial lines before the statement is complete.  For example:

```python
In [1]: a = [1,
   ...: %quickref   # ← this gets executed as a magic, but it's inside the list!
   ...: 2]
```

Here, `%quickref` is evaluated as a magic command even though it appears inside an unfinished list literal.  The same happens with `!ls`, `?help`, and user-defined `TokenInputTransformer` wrappers (like a decimal-number transformer).  This makes it impossible to write a multi-line statement that contains a `%`, `!`, or `?` character in a string or comment, or to use token-level transformers on an expression that spans multiple lines.

Another symptom: when using a `TokenInputTransformer`-based transformer (as documented), the transform function receives tokens from only one line at a time, so it cannot see the full token stream of a multi-line statement.  For a transformer that replaces decimal literals, a line like `a = [1.2,` triggers the replacement immediately, leaving `a =[Decimal ('1.2'),` and then the closing `3]` arrives separately.  The expected behaviour is that the transformer should receive the entire statement's tokens at once: `a =[Decimal ('1.2'),\n3 ]`.

Additionally, explicit line-continuation (`\`) does not work correctly with magic and system assignments.  Lines joined by a backslash should be treated as a single logical line before IPython-specific transforms run.  Currently, `%magic arg1 \` followed by `arg2` fails to recognise the continuation, and `a = !cmd arg1 \` followed by `arg2` is also broken.

Expected behaviour
------------------
IPython should first assemble complete Python statements:

1.  Join lines ending with `\` into a single logical line.
2.  Accumulate input until a complete, balanced, top-level Python statement is formed (i.e. all brackets are closed, multiline strings terminated).
3.  Only then apply IPython-specific transforms (magic, shell, help, `assign_from_magic`, `assign_from_system`, and user `TokenInputTransformer`s) to the entire statement.

Individual physical-line transforms like prompt stripping and leading-indent removal should still run as soon as the line arrives, because they do not depend on the syntactic completeness of the statement.

Acceptance criteria (observable)
---------------------------------
- `%`, `!`, `?` inside an incomplete list/dict/expression are left untouched until the brackets close.
- `TokenInputTransformer` wrappers receive the complete token stream of a full Python statement.
- `\` line continuations work inside magic commands, system calls, and help escapes.
- All existing single-line interactivity is preserved (backward-compatible).
- `leading_indent`, `classic_prompt`, `ipy_prompt` still operate per physical line as before.
- Whitespace-only or comment-only lines are handled gracefully; they should not cause buffered state to be emitted prematurely.

The fix should be implemented in IPython's input-splitting / transformation machinery.  Please add tests that cover multi-line statements with magic/shell/help characters, custom TokenInputTransformer usage, and backslash continuations in those contexts.
```


**Title:** Tab completion fails to infer return type when function body assigns to a local variable before returning

**Severity:** medium
**Component:** completer, guarded_eval

###

**Description**

When IPython tries to provide attribute completions after a function call like `foo().`, it evaluates the function body to infer the return type. The current logic appears to walk `return` statements inside the function, but it does not account for assignments that happen earlier in the body.

A minimal example that should work but currently does not:

```python
def foo():
    l = []      # assignment to local
    return l    # return that local

foo().          # pressing <tab> here should show list methods (e.g. "append")
```

The user expects completions for the return type of `foo()` — in this case, `list` methods such as `append`. Today the completer provides no suggestions.

**Template**

```python
def foo():
    <assignment>
    return <variable>
foo().
```

with any empty-list/empty-dict/class-constructor-style assignment should produce the corresponding completions on the dotted expression.

**Additional context**

- This affects the guarded evaluation path used by the completer when `evaluation` is set to `"limited"` or above.
- The return-inference logic already handles simple literals in `return` statements (e.g. `return []` works). The gap is specifically when the value is first bound to a local name.
- AnnAssign (e.g. `l: list = []`) in the function body should also be handled.
- The inference should use a function-local namespace for these assignments so that they do not leak into the calling context’s evaluation state.
- There is no requirement to handle control-flow (conditionals, loops) — a straight-line assignment-before-return is sufficient for now.


Fix: LaTeX converter inserts a stray newline after multi-line math spans, and mishandles text that contains a single `$`

The LaTeX conversion pipeline uses a helper that strips whitespace from inside `$...$` math delimiters. It currently has two issues:

1. When a math span is broken across lines (e.g., a lone `$` at the end of a line, then content, then a closing `$`), the helper appends a newline `\n` at the point where the multi-line span ends. That extra newline corrupts the `.tex` output. Please remove this spurious newline.

2. The helper does not correctly handle input that contains an isolated, unmatched `$` character (a single dollar sign that is not part of a math pair). Such text should be left entirely unchanged—no whitespace stripping and no newline injection. Currently the logic can mangle that text.

Additionally, make the handler safe for empty input (it should just return the empty string without error).

Existing tests live in the transformer test suite; please make sure the fix is covered by a few extra examples that exercise multi-line math closure and unchanged single-dollar-sign text.


We need to extend the `%load` magic to support loading only specific top‑level function or class definitions from a Python source file.

**Syntax**
`%load -s <symbols> <source>`

`<symbols>` is a comma‑separated list of names, e.g. `-s MyClass,helper_func`.

**Acceptance criteria**

1. **Top‑level definitions only** – When the `-s` option is present, `%load` extracts the source of the given **function** and **class** definitions from the target Python file. Top‑level variable assignments (e.g. `a = 10`) are **never** extracted; such symbols are silently skipped.

2. **Source fidelity** – The extracted code block for each requested symbol must be copied verbatim from the original file, preserving all whitespace, blank lines, decorators, and docstrings exactly as they appear.

3. **Non‑Python files** – If the target file is not valid Python (e.g. a Ruby script or plain text), `%load -s` must not raise an error; it should simply load nothing for the requested symbols. The rest of the magic should still behave as usual (e.g. honour `-y` and other options).

4. **Missing symbols** – Symbols that do not exist in the file are silently omitted; no warning or error should be produced.

5. **Compatibility** – The new option must work together with the existing `-r` (line‑range) and `-y` options, and should not change the behaviour of `%load` when `-s` is not supplied. If both `-s` and `-r` are given the order of processing should be: first extract the requested symbols, then apply the line‑range filter to the joined result (or the other way around that yields a reasonable, documented result—decide whichever makes more sense and document it in the help string).

6. **Help text** – Update the `%load` docstring to include an example with `%load -s MyClass,wonder_function myscript.py` and describe the new option.

**Edge cases to consider**
- A symbol that appears multiple times (e.g. a function `f` that is also a local variable inside another function) should only refer to the top‑level definition.
- The target file may contain only a class definition with no functions; the class should still be extracted correctly.
- The comma‑separated list may contain spaces around commas (e.g. `-s A, B , C`). At minimum the implementation should handle the standard non‑space version; whitespace trimming is a nice‑to‑have but not required.

adds the optional parameter -s to the magic %load that loads specific function or classes from the python source given. forcing the patch ignoring whitespace changes


Title: Disallow session numbers without trailing slash in history range syntax

**Observed problem**
The history range parser (`extract_hist_ranges` used by `%history` and other commands) accepts bare session identifiers like `~5` (without a trailing slash) to mean “all lines of session 5 before the current session”. The documented syntax requires a trailing slash – `~5/` for the whole session and `~5/1-10` for a range within the session. The bare form is not documented and introduces an unnecessary variation that can cause confusion and ambiguity.

**Expected behavior**
- `~5` should **no longer** be treated as a valid session identifier. Only the slash‑ended form (`~5/`) should be recognized when referring to an entire session.
- Likewise, `5/` (current session) should remain valid, while `5` alone continues to mean just line 5 of the current session.
- Session + line‑range specifications like `~5/1-10` or `~5/4-` must keep working unchanged.
- The docstring of `extract_hist_ranges` and the help text of `%history` should be updated to remove any mention of the no‑slash shorthand.

**Impact**
This is a small behavioural change that aligns the parser with the documented usage and removes an ambiguous shortcut. It avoids scenarios where `~5` might be misinterpreted in future extensions of the range syntax.


The `%notebook` magic should export a trusted notebook so that users don't see the "not trusted" warning when opening it in Jupyter. Currently the generated `.ipynb` file is not trusted, which means outputs and JavaScript are blocked until the user explicitly trusts the notebook. Ensure that any notebook exported via `%notebook` is automatically treated as trusted in Jupyter by applying the standard nbformat trust/signing mechanism before writing the file.


**Title:** Autoreload infinite recursion when reloading classes with complex object graphs

**Background**
The `%autoreload` extension can reload modules and update live class objects. When a class definition changes, autoload attempts to find all existing instances of the old class and replace their `__class__` with the new class. The current instance search does a recursive traversal of live objects, but it does not guard against cycles or repeated visits, which can lead to infinite recursion when an object graph contains circular references or when the same object appears through multiple paths.

**Desired behavior**
Implement a cycle-safe, bounded graph exploration that:
- Starts from the user namespace (`shell.user_ns`) and the hidden user namespace (`shell.user_ns_hidden`), filtering out private names (those starting with `_`) and module objects.
- Walks through objects by inspecting their `__dict__` (skipping functions and methods) and, for containers, by iterating over items (skipping private keys and modules).
- Uses a visited set (or equivalent) to ensure each object is processed at most once.
- Includes a maximum iteration count to prevent unreasonably long loops; if exceeded, raise an error rather than hanging.

This exploration should replace the current recursive instance search in the class-update path.

**Shell attribute dependency**
The instance search now depends on `shell.user_ns` and `shell.user_ns_hidden` being present on the shell object that holds the autoreload magics. Any shell-like object passed to `AutoreloadMagics` must expose these two namespace dictionaries.

Try to fix updating classes in Autoreload.


**BUG: `raw_input` broken in in-process kernel and pollutes global builtins**

### Steps to reproduce
1. Use IPython's in-process kernel (for example, by calling `embed_kernel` or launching an in-process terminal frontend).
2. Execute code that calls the builtin `raw_input()` (Python 2) or `input()` (Python 3) with a prompt, e.g.:
   ```python
   x = raw_input('Enter value: ')
   ```
3. Observe the prompt and input reading behavior.
4. After the kernel cell finishes, attempt to call `raw_input()` (or `input()`) again from ordinary Python code outside the kernel.

### Actual behavior
- No prompt string is displayed to the user.
- The call does **not** read from `sys.stdin`; it either blocks indefinitely, raises an error, or returns unexpected data.
- After the kernel execution completes, the global builtin `raw_input`/`input` remains permanently replaced with an IPython internal version that raises an error, breaking any subsequent use of `raw_input`/`input` in the same process.

### Expected behavior
1. When user code inside the in-process kernel calls `raw_input(prompt)`, the `prompt` is printed to the process's normal standard output (without requiring a ZMQ frontend).
2. The call then reads a line from `sys.stdin` and returns it, exactly as the original builtin would.
3. After the kernel finishes executing the cell, the **original** builtin `raw_input`/`input` function is restored so that later Python code outside the kernel continues to work correctly.
4. The in-process kernel's stdin channel must handle `input_request` messages locally instead of forwarding them over ZMQ, because no external frontend is connected.

The fix must respect Python 2 vs Python 3 (the builtin name differs) and ensure that the saved original function is used for input, not the IPython replacement that normally talks to a ZMQ frontend.

BUG: raw_input logic incorrect for in-process terminal frontend.


The notebook server's REST API returns `Location` headers in 201 responses when creating notebooks, kernels, sessions, and checkpoints. These headers currently contain the raw, unescaped path – which is invalid for URLs that include characters not allowed in URIs, such as spaces or non-ASCII characters (e.g., accented letters). HTTP requires that `Location` headers contain a URI reference, so all special characters must be properly percent-encoded.

We need all resource creation endpoints to return a fully percent-encoded URL in the `Location` header. Additionally, any internal helper that builds a notebook's location URL should produce an already-escaped string so that callers do not accidentally send unescaped headers.

The fix should cover at least the following scenarios:
- Creating a new kernel (POST to kernels API)
- Creating a new session (POST to sessions API)
- Creating a new notebook (POST to notebooks API, including untitled and upload/copy flows)
- Creating a new checkpoint (POST to notebook checkpoints API)

The behavior should be consistent: after the change, creating a notebook in a directory with a space or Unicode character in its name (e.g., `å b`) must produce a `Location` header like `/api/notebooks/%C3%A5%20b/Untitled0.ipynb` rather than the raw path.

Existing tests that verify the `Location` header should be updated to validate the full escaped URL, not just the last segment of the path.


As a maintainer, I would write:

---

**Title:** Add `--url` option to iptest to run JS tests against an existing notebook server

**Description:**

Currently, when you run the notebook JavaScript test suite through `iptest`, the runner automatically launches a local notebook server, selects a free port, and passes that port to CasperJS via `--port`. The CasperJS helper then constructs the base URL as `http://127.0.0.1:<port>` and hits that server.

This is inconvenient when you already have a notebook server running (for example, in a development session, a custom configuration, or a CI setup where the server is managed outside the test runner). You have to kill the existing server, let the test runner start its own, and you cannot easily target a different host, port, or base path.

**Requested behavior:**

- Accept a new `--url` option on the `iptest` command (and the underlying `iptestall`/controller machinery) that takes a string URL.
- When `--url` is provided, the JavaScript test controller must **not** start a new server. Instead, it should forward the URL to CasperJS (e.g., `--url=<value>`).
- The CasperJS test utility script (the one that provides `get_notebook_server` and similar helpers) should use the supplied `--url` if it is present. If not present, it should fall back to the existing behaviour: construct `http://127.0.0.1:<port>` from the `--port` argument (defaulting to `8888`).
- The existing behaviour when no `--url` is given (auto-launch a server, use its port) must continue to work exactly as before.

**Non-goals:**
- No changes to Python test execution or other test types.

This will let developers run `iptest --url=http://localhost:9000/tree` or similar against any pre-running notebook instance, making debugging and custom CI workflows much easier.


```
Restore valid value when invalid selection is set

When a `DropdownWidget` (or any `_SelectionWidget` subclass) is assigned a value that is not a key in its `values` dictionary, it raises `KeyError` but leaves its internal `value` attribute holding the invalid input. The widget must instead reset the value to the previous valid value before the error is raised.

Additionally, the `interact`/`annotate` helpers should still handle the case where a default is out of bounds by creating a widget whose value is a valid fallback (e.g., the first available option).
```


The `Connect` class in `IPython.utils.traitlets` does not initially synchronize the values of connected traits, and it accepts fewer than two traits without complaint.

When a new `Connect` is created with two or more `(obj, attr_name)` pairs, all the target traits should be driven to the same value immediately – the value of the first trait at connection time must be propagated to every other trait. Only after that initial sync should ongoing changes propagate through the normal listener flow.

Also, constructing `Connect` with fewer than two traits is an error: raise a `TypeError` for zero or one argument.


**Title:** Replace insecure `tempfile.mktemp` calls with safe temporary‑file creation

**Description:**
`tempfile.mktemp` only generates a filename but does not create the file. Between the time the name is returned and the time our code opens it, another process (or attacker) can create a file with that same name, leading to data leakage, corruption, or privilege escalation. This affects several places in the codebase:

- The text pager fallback: when `page()` uses a custom pager command and writes the content to a temporary file, it calls `mktemp('.txt')`, opens the file, writes, closes, and then redirects it to the pager via `pager_cmd < tmpfile`.
- Kernel connection file generation: `write_connection_file()` generates a temporary `.json` connection file via `mktemp('.json')` when no explicit filename is given.
- Test‑support utilities: `default_config()` creates a temporary history database file via `mktemp(u'test_hist.sqlite')`.
- A few test modules (notably in the parallel test suite) use `mktemp` to create temporary files for running code snippets.

**Expected behavior / Requirements:**
- Replace every `tempfile.mktemp` call with a safe alternative that atomically creates the file (avoiding the TOCTOU race).
- The pager case is the most subtle: the temporary file **must be closed** before the system command (`pager_cmd < tmpfile`) is executed, otherwise some platforms (especially Windows) will fail with a sharing violation. The old code explicitly closed the file first; the new implementation must preserve that ordering.
- The temporary files must still be cleaned up after use (explicit `os.remove` or a context manager/`delete` flag), unless the file is intended to outlive the process (which none of the current usages require).
- The fix must work across Python 2.7 and 3.3+ (the versions IPython supports).

**Impact:**
Currently, a race‑condition attacker on a shared system could interfere with IPython’s temporary files, potentially exposing sensitive kernel connection information or corrupting pager output. This is a security hardening request; no user‑visible behavior change is expected beyond the elimination of the race window.


Title: `Video` display object does not handle raw bytes or pre-base64-encoded unicode data, and ignores explicit `mimetype` when embedding from a file

**Description**

When constructing an `IPython.display.Video` object with raw video data supplied as `bytes`, or with a unicode string that already contains base64-encoded video data, the resulting HTML embed is broken. Additionally, the explicit `mimetype` argument is ignored when embedding a video from a file.

**Observed problems**

1. **Raw bytes input** – Passing `data=b'...'` to `Video` fails or produces incorrect HTML. The constructor calls `data.startswith('http')` unconditionally, which will fail for `bytes` on Python 3 (bytes objects have no `startswith` accepting string arguments) or may interpret arbitrary bytes incorrectly as a URL. Even if that check is bypassed, raw bytes are not base64-encoded before being placed in the data URI.

2. **Base64-encoded unicode input** – Passing `data=u'...'` where the string is already a valid base64-encoded video gives a double-encoded output. The string is passed through plain text, likely re-encoded again, rather than being used directly.

3. **URL detection on non‑string data** – The current URL detection treats *any* object whose string representation starts with `http` as a URL. This should only be done for real string types (`str` / `unicode`), not for `bytes` or other types.

4. **Mimetype override ignored for file embedding** – When embedding a video from a file with `embed=True` and a specific `mimetype`, the provided `mimetype` is discarded; the guessed mimetype from the filename is always used. This prevents embedding video from a file with a non-standard extension or enforcing a specific format.

**Expected behavior**

- `Video(data=<bytes>)` should treat the bytes as raw video data, base64-encode them, and embed them using either the supplied `mimetype` parameter or a guessed mimetype.
- `Video(data=<unicode string that is already base64>, mimetype=...)` should embed that string directly as the base64 payload without any re-encoding.
- URL detection should only occur when `data` is a string type (`str` or `unicode`) and starts with `http:` or `https:`.
- When `filename` is given together with `embed=True` and an explicit `mimetype`, the explicit mimetype should be used in the data URI instead of being automatically guessed and silently overridden.

**Impact**

Users cannot display video content from memory (e.g., bytes loaded from a database, generated frames, or already base64-encoded strings) without resorting to temporary files. The mimetype override bug also prevents embedding of video files where the file extension does not match the actual video format.


**Title:** Container traitlets (List, Tuple, Set) should accept compatible sequence types and coerce them.

**Description:**

In `IPython.utils.traitlets`, the container-based trait types `List`, `Tuple`, and `Set` enforce exact type matching on assignment: a `List` trait only accepts a `list`, a `Tuple` trait only accepts a `tuple`, and a `Set` trait only accepts a `set`. This is overly strict and leads to `TraitError` when users naturally pass a tuple to a list field, or a list to a tuple field, which is common in Python code.

The desired behavior is:
- A `List` trait should accept a `tuple` and automatically convert it to a `list`.
- A `Tuple` trait should accept a `list` and automatically convert it to a `tuple`.
- A `Set` trait should accept a `list` or a `tuple` and automatically convert it to a `set`.

The coercion must happen **before** element validation, so that element-level constraints (e.g., `List(Int)`) are applied to the converted collection, not to the original sequence type. The existing handling of `None` values (controlled by `allow_none`) and default values must remain unchanged. This change aligns container traits with the general traitlets pattern where compatible types are accepted and coerced (e.g., `Int` accepting `float` and truncating).


**Title:** `Exporter` should not accept `template` as an alias for `template_file`

**Description:**

The `Exporter` class (and subclasses like `HTMLExporter`, `LatexExporter`, `SlidesExporter`) currently accepts a keyword argument `template` that silently sets the `template_file` attribute. For example, `HTMLExporter(template='basic')` works, but the documented and configurable trait is `template_file`. This undocumented alias causes confusion and can lead to broken template resolution when a `config` object is also passed, because the config initialisation order does not handle the alias properly.

**Steps to reproduce:**

```python
from IPython.nbconvert import HTMLExporter
exporter = HTMLExporter(template='basic')          # accepted but unclear
exporter = HTMLExporter(template_file='basic')     # correct, documented
```

**Expected behaviour:**

The `template` keyword argument should **not** be accepted as a shorthand for `template_file`. Passing `template=` should either be ignored (or raise an error), forcing callers to use `template_file=` explicitly.

**Additional notes:**

While removing this alias, the following related config-initialisation bugs should also be fixed, as they affect template handling and were uncovered during investigation:

1. When the `config` argument is `None` (the default), the exporter does not initialise `self.config` with `self.default_config` before calling `super().__init__`. This means some trait defaults may not be properly available.
2. The `_config_changed` method does **not** call `super()._config_changed(...)`, which breaks the config-change notification chain.
3. Inside `_config_changed`, the comparison that decides whether to update `self.config` uses `c != new` but should use `c != old` to correctly detect when the computed config has actually changed.

All existing uses of the `template=` shorthand in the codebase (including in exporter constructors and test helpers) should be updated to use `template_file=` to match the new rule.

don't allow 'template' to specify 'template_file'


KernelManager should provide a way to obtain a fully configured client for the kernel it manages. Currently, consumers that start a kernel via `KernelManager` must manually instantiate a `KernelClient` subclass, pass the connection file, and call `load_connection_file`, duplicating logic the manager already owns.

Acceptance criteria:

1. **`client()` method** – `KernelManager` gains a public method that returns a client instance ready to connect to the running kernel. The returned client must already have all connection information (transport, IP, ports, session key, etc.) populated from the manager's state, so that the caller only needs to invoke `start_channels()` (or equivalent) on the client afterward – no manual file loading or parameter passing.

2. **Configurable client class** – The manager must allow users to specify which client class is instantiated. The default should be the standard `KernelClient`. Switching to another client (e.g., a blocking variant) should be possible by setting a configurable attribute on the manager, following the same pattern used by other IPython traitlets.

3. **Override support** – Any keyword arguments passed to `client()` should override the default connection parameters, allowing fine-tuning without breaking the convenience.

4. **Control socket linger timing fix** – While working in this area, fix a subtle behavioral issue: the control socket’s `linger` option is currently set only on closure. This can cause hangs during shutdown if the socket is never explicitly closed. Move the linger setup to occur when the control socket is first connected, so that even implicit destruction respects the linger timeout.


**Title:** Feature request: Add "Trust Notebook" option to the File menu

**Description:**
Currently there is no way to explicitly trust a notebook from the IPython notebook web interface. Users who want to mark a notebook as trusted (so its cells can display rich output without re-execution) must use the command-line `ipython trust` tool. We need an in-UI "Trust Notebook" action available from the File menu.

**Expected behavior:**
- A new "Trust Notebook" item should appear in the File menu, separated from the existing items by dividers.
- Clicking "Trust Notebook" should display a modal confirmation dialog that:
  - Warns the user that a trusted notebook may execute malicious code when opened.
  - Includes a link to the IPython security documentation.
  - Offers "Cancel" and "Trust" buttons (the "Trust" button should be visually styled as a danger action).
- When the user confirms by clicking "Trust":
  1. The frontend should send a POST request to the notebook server's API (a dedicated trust endpoint for the current notebook).
  2. The server should use the notebook notary to sign the notebook, mark all cells as trusted, and save the notebook.
  3. On success, the browser should reload the page to reflect the new trust state.
  4. On failure, an appropriate error event should be triggered.

**Additional context / constraints:**
- The notebook trust mechanism already exists (notebook notary signing, `mark_trusted_cells`, `check_and_sign`). The new backend endpoint should reuse these existing facilities correctly.
- The implementation must ensure the argument ordering for path/name across the manager methods is consistent with the rest of the notebook API to avoid signature mismatches when trusting.
- The frontend trust function should follow the same patterns as other notebook actions (e.g., save, checkpoint) with proper event triggers (`notebook_trusting.Notebook`, `notebook_trusted.Notebook`, `notebook_trust_failed.Notebook`).


```
On Windows, the IPython notebook displays broken links when a `FileLinks` object is rendered. The generated HTML anchor tags (`<a href="...">`) use backslashes (`\`) as path separators in the `href` attribute, because Windows filesystem paths use backslashes. Browsers cannot interpret those backslashes as valid URI path separators, resulting in broken links.

We need the notebook representation of `FileLinks` (i.e., the output of `_repr_html_()`) to always produce forward slashes (`/`) in URIs, regardless of the operating system's native path separator. The terminal/plain-text representation (`__repr__`) can and should continue to use the OS-native separators (backslashes on Windows, forward slashes on Linux/macOS) – only the HTML/notebook output must be sanitized for web use.

This change should be applied only to the notebook display pathway; no other representations or internal uses of the file path should be modified unless they also serve as a URI in an HTML context.

Related: issue #2732.
```

this should address the failure in #2732


Title: Adjust filename increment behavior for untitled items and copies

Currently, the notebook server applies a zero-based increment to all newly created untitled notebooks, files, and directories, and to copies that require deduplication:

- Untitled notebooks start as `Untitled0.ipynb`, plain files as `untitled0`, and directories as `Untitled Folder0`.
- When a file with a `-Copy` suffix (e.g., `Foo-Copy1.ipynb`) is copied into the same directory, the result keeps the old suffix and adds another `-Copy` layer, producing `Foo-Copy1-Copy1.ipynb` instead of advancing to `Foo-Copy2.ipynb`.
- When a file is copied into a different directory, the result unconditionally receives a `-Copy0` suffix even if no name collision exists, rather than preserving the original basename.

We want three adjustments to make the naming feel more natural:

1. **Untitled items start without a number.** The first untitled creation in a directory should have no numeric suffix (e.g., `Untitled.ipynb`, `untitled`, `Untitled Folder`). The first collision should then introduce a suffix – `Untitled1.ipynb`, `Untitled Folder 1`, and so on. (For directories, a space should appear before the number: `Untitled Folder`, `Untitled Folder 1`, ….)

2. **Copy of a copy increments the existing `-CopyN` suffix.** When a filename already ends with `-Copy` or `-CopyN`, copying it should strip that suffix and replace it with the next number rather than stacking another `-Copy`. For example, copying `Foo-Copy1.ipynb` in the same directory should yield `Foo-Copy2.ipynb`. The first copy of a file without a `-Copy` suffix should produce `Foo-Copy1.ipynb`.

3. **Copy to a different directory honors the original basename.** When a file is copied into a target directory that does not yet contain a file with the same name, the copy should use the exact original filename, with no `-Copy` suffix. Only if a collision occurs should the usual `-CopyN` increment logic kick in.

These changes should apply everywhere the server generates a name for an untitled item or a duplicate-safe copy – including the REST API endpoints used by the frontend (e.g., creating untitled notebooks/text files/directories and copying files).


When connecting to a kernel, the client discovers the kernel’s protocol version by sending a `kernel_info` request and inspecting the reply. If the reply indicates a version different from the client’s own, the client must adapt its session’s protocol version automatically. This adaptation is currently implemented in a way that only works for the blocking shell channel (`BlockingShellChannel`), and only when the reply is read through that channel’s `_recv` method.

As a result:

- `QtKernelClient` never performs protocol adaptation. The Qt shell channel emits the `kernel_info_reply` as a signal but nothing connects it to the session adaptation logic.
- Code that creates kernels via `start_new_kernel` must manually request `kernel_info` and wait for the reply, then flush channels of stale messages. This is fragile and leaks the responsibility to callers.
- `BlockingKernelClient` does not expose any “wait until kernel info received” mechanism, forcing users to manually listen for the reply.

The expected behavior is:

1. Sending a `kernel_info` request should be an automatic part of starting the shell channel in `KernelClient`. After the shell channel is started, the base client should send the request, and the reply should be processed to set `session.adapt_version` when needed.
2. `BlockingKernelClient` should provide a way (e.g., a `wait_for_ready()` method) that blocks until the kernel info reply is received and any queued IOPub messages from startup are drained. This method should be used by `start_new_kernel` and by any blocking code that needs a fully initialized kernel client.
3. `QtKernelClient` should connect the `kernel_info_reply` signal from its shell channel to the adaptation handler, process the reply exactly once, and then disconnect the signal so that the client is ready.

Additionally, some dead code left over from the old blocking-only approach should be cleaned up: the `_handle_kernel_info_reply` method and the `_recv` override in `BlockingShellChannel`, and the unrelated `msg_queue`/`proxy_methods` attributes that were incorrectly attached to `BlockingStdInChannel`.

Make sure that all existing callers that relied on manual kernel_info handling (the `start_new_kernel` function and the embed kernel test setup) no longer perform this work themselves and instead rely on the new client-built-in readiness mechanism.

Move kernel_info for adaptation onto KernelClient


We need an in-process way to drive the terminal’s main interaction loop during tests. Today, any terminal test that cares about the input-code loop has to spawn a real IPython subprocess and pipe commands into it. That’s slow, fragile, and hard to debug.

Provide a reusable decorator/helper that lets a test act as a generator of prompt responses, runs `interact()` in-process, and captures any exception. It should stop the loop cleanly when the generator is exhausted. Make sure that if the generator itself raises an exception that is trapped and re-raised after the loop exits, so the test properly fails.

As a concrete motivation and first use case, fix a bug in the terminal shell: a `SyntaxError` raised inside an `InputTransformer.push()` currently causes the whole interactive session to abort. After the fix, the shell should print the traceback (as it would for any normal code error) and stay alive so the user can keep entering commands.

Write a test for this behavior using the new in-process mechanism. Remove the old subprocess-based test for transformer syntax errors (`IPython/terminal/tests/test_terminal.py`) and its `ipexec` dependency. If necessary, also add a small `__repr__` to `IOStream` so that test output capture works reliably in this setting.

Mechanism for testing terminal interact loop in-process.


**Issue**: Prompt alignment breaks when escape sequences other than color codes appear in the prompt template

**Background**
IPython’s terminal frontend uses `readline` (or `pyreadline` on Windows). `readline` requires that any zero‑width sequences in the prompt be marked with `\001` and `\002` delimiters. Our prompt machinery already embeds color codes this way, but it currently determines “invisible” characters by subtracting the length of the rendered prompt *without* color from the length *with* color. That approach only captures the known color escapes and ignores other zero‑width sequences that a user or extension might embed in a prompt template — for example, OSC‑style terminal title updates (`\033]133;A\007`), hyperlink escapes, or any custom readline‑marked sequence.

**Problem**
When a prompt template contains such non‑color invisible sequences, the prompt width is calculated incorrectly, causing the input line to be misaligned with the prompt text. The prompt may appear to “jump” or overlap with the input.

**Request**
Change the invisible‑character counting so that it recognises *any* sequence wrapped in `\001` … `\002` as invisible, matching `readline`’s own semantics. The number of invisible characters on the last line of a rendered prompt should be the total length of all substrings that lie between a `\001` and a `\002`, not just those that happen to be colour codes.

**Constraints**
- Colour escape sequences that are already correctly wrapped in `\001`/`\002` must still be treated as zero‑width (no regression).
- Sequences that are *not* wrapped in `\001`/`\002` should **not** be counted as invisible. The presence or absence of the delimiters is the sole criterion.
- The change must be confined to the prompt‑width calculation (the prompt rendering itself does not need to change).
- The overall prompt length (visible characters) must be correct after the fix, so that the `width` attribute on prompt instances matches the number of displayable columns.

**Expected behaviour**
After the fix, a prompt containing e.g.
```
\x01\x1b]133;A\x07\x02In [\#]: \x01\x1b]133;B\x07\x02
```
should have a display width equal to the visible text `"In [1]: "` (plus dynamic count), with the bracketed escape sequences contributing zero width.

Be a little smarter about invisible characters in terminal prompts


**Title:** Display function signatures in the shell’s pretty-printed output

**Description:**
When IPython displays a function object – whether in the interactive prompt, via `%pprint`, or when inspecting a property’s underlying getter – it currently shows something like `<function mymodule.myfunc>`. This omits the function’s parameters, so users must run `?` or `help()` separately to see its signature.

We want to enhance that display so that the function signature is included directly in the representation, e.g. `<function mymodule.myfunc(a, b, c=42)>`. This should work for any callable whose signature can be reliably determined, including pure-Python functions and built-in functions (where `inspect` can extract the information).

**Acceptance criteria:**
- In the standard pretty-printing pathway (the `_function_pprint` family of formatters), the displayed text for a function includes its full parameter signature after the qualified name, separated by no space (i.e. `qualified_name(…)`).
- The change applies to all function object types that currently use the `<function …>` format, such as regular functions, built-in functions, and any other callable that enters the function pretty-printer.
- If retrieving the signature raises an exception (for example, a `ValueError` from `inspect` for certain built-ins or objects with no signature), the display must **fall back gracefully** to the current name-only format (`<function name>`) without raising an error.
- The new format carries over to the object inspector: when a property wraps a function (e.g., `fget = <function …>` in source output), the function part should now show the full signature in the same way.
- The implementation must be compatible with IPython 6+ (Python 3 only). Avoid dependencies on Python-2 compatibility modules; the existing signature utility (`IPython.utils.signatures`) is available and forward-compatible with Python 3’s `inspect.signature`.
- Existing tests for pretty-printing and object inspection must be updated to match the new output; new tests should exercise a typical function, a built-in (or a well-known standard-library function whose signature is stable), and a case where signature retrieval fails and the fallback is used.

Add signature to the display of functions in the shell


```
Refactor `generate_tokens` in `IPython.utils.tokenutil` to only accept a readline callable, matching the interface of the standard library's `tokenize.generate_tokens`. Currently the function accepts either a str or a callable, which is inconsistent and can cause confusion at call sites. Update all internal callers (including `token_at_cursor` in the same module, and `accept_token` in `IPython.terminal.shortcuts.auto_suggest`) to pass `StringIO(text).readline` instead of raw strings. Ensure that the function signature and docstring clearly reflect the new contract.
```


Title: Add `spread` option to `columnize` so columns fill available display width

The `IPython.utils.text.columnize` function formats a list of strings into columns using a fixed inter-column separator (default `'  '`). When the total column widths plus separators are smaller than `displaywidth`, unused whitespace remains on the right side. For tabular displays it is often desirable to distribute the columns evenly across the full display width instead.

We need a new boolean parameter `spread` (default `False` to preserve current behavior) that, when set to `True`, expands the separator width to consume the remaining space within `displaywidth`. The individual column widths (as computed by the existing layout algorithm) must not change; only the separator width is increased.

The layout information already returned by `compute_item_matrix` includes the optimal separator width that would result in full-width rows. When `spread=True`, the separator should be padded with spaces to that optimal width, using the caller-supplied `separator` as the minimum separator string. When `spread=False` or when items cannot fit a single row, the original fixed separator is used.

Edge cases / constraints:
- If `displaywidth` is too narrow to fit all columns in one row, the layout still produces multiple rows; `spread` should apply per-row, using the same optimal separator width computed for the overall layout (as currently done, the layout determines a single column structure and separator width for all rows).
- The implementation must be fully backward-compatible: all existing callers who do not pass `spread` must see identical output.
- The argument `separator` should continue to work as before when `spread=False`; when `spread=True`, it serves as the base separator string that gets right-padded with spaces.

Acceptance criteria:
- Adding `spread=True` to `columnize` calls yields rows whose total length equals `min(displaywidth, sum of column widths + (ncols-1) * optimal_separator_width)`, modulo rounding.
- The test suite (existing and any new cases) passes.
- No public API breakage.

This request concerns only the public function `columnize` in `IPython/utils/text.py`. The necessary layout metadata is already available from the existing helper; the change is purely about using it to expand the separator when requested.

Add spread option, to allow columns to fill space.


**Request: Rename `skip_exceptions` to `stop_on_error` in kernel execute API**

**Background**
The `KernelClient.execute` method and the kernel’s `execute_request` handler currently accept a `skip_exceptions` flag (default `False`). The flag controls whether the execution queue aborts after an unhandled exception. The name is a double negative: `False` means “do not skip exceptions” and therefore stop on error. This is confusing for both users and maintainers.

**Acceptance Criteria**

1. Rename the parameter to `stop_on_error` with a default of `True` throughout the entire stack, so that the default behavior (aborting the queue on error) is retained without a double negative.
2. Supplying `stop_on_error=False` must reproduce the previous `skip_exceptions=True` semantics: the queue continues executing subsequent requests even after an error.
3. The rename must be applied consistently in all relevant layers:
   - The Python `KernelClient.execute` method signature and its docstring.
   - The kernel’s `execute_request` handler, where the content dict is inspected and `_abort_queues` is conditionally called.
   - The JavaScript notebook front-end’s `CodeCell.prototype.execute` method, which forwards the flag when sending an execute request.
   - The messaging protocol documentation (`messaging.rst`) that describes the `execute_request` content format.
4. The underlying ZeroMQ message content key must change from `'skip_exceptions'` to `'stop_on_error'`.
5. External behavior must be unchanged: omitting the flag still causes the queue to abort when an error is encountered; providing `stop_on_error=False` lets execution continue past errors.

**Why this helps**
A positive flag name (`stop_on_error=True`) directly expresses the intended outcome, matching the style of other flags such as `silent`.

Use stop_on_error instead of skip_exceptions


We need a separate, truly interruptible IPython debugger variant.
Currently, sending a `KeyboardInterrupt` (Ctrl‑C) to the standard `Pdb` debugger is unreliable: depending on context, the interrupt may be caught and the debugger re‑prompts, or it prints a confusing traceback, or it leaves the trace hook installed and the process in a broken state. In ipykernel, for example, this can cause hangs when a second interrupt is needed to finally exit.

Requirements:
* Introduce a new debugger class that cleanly stops on `KeyboardInterrupt`.
  On interrupt, the debugger must quit immediately — disable any stop‑on‑next‑event, clean up the trace function, and re‑raise the interrupt so the calling code can handle it or exit.
  No further prompts should appear after the interrupt; the debugger must not loop back to `input()`.
* The existing `Pdb` class must keep its current behavior so that users who depend on the current Ctrl‑C handling are not affected. The new interrupt‑quit semantics should only apply when explicitly using the new debugger.
* While we’re at it, in `Pdb.interaction` the `KeyboardInterrupt` handler should display the exception summary (via the IPython shell) instead of re‑raising a bare interrupt.

Acceptance criteria (observable behavior, not implementation details):
1. When the new debugger is running and a `KeyboardInterrupt` arrives while waiting for input, the debugger exits, the trace function is removed, and the interrupt propagates up. The input function is called at most once.
2. The original `Pdb` still handles `KeyboardInterrupt` exactly as before (or nearly‑as) when interacting with a user, but `interaction` now shows the shell’s exception information on interrupt.
3. Setting a trace with the new debugger works the same as with the old one, except for the interrupt‑quit guarantee.

A maybe better-working interruptible Pdb. Also make separate class so we don't


```
Bug report: nbconvert stdout writer fails on unicode output
---

The `StdoutWriter` in `IPython.nbconvert.writers.stdout` currently
uses a plain `print` call to output converted notebook content.
This causes failures or mangled output when the content contains
non-ASCII unicode characters, especially on Python 2 where `sys.stdout`
is typically a byte stream with an ASCII encoding.

A user who runs `jupyter nbconvert --to notebook --stdout`
(or uses the `StdoutWriter` programmatically) with input that
includes e.g. accented or special characters will encounter
either a `UnicodeEncodeError` or incorrectly encoded output.

The writer should be able to output the full unicode content of
a notebook to stdout without surprises, regardless of whether the
environment is Python 2 or Python 3.  IPython already has
well-tested internal utilities for unicode-safe stdout
writing that are used elsewhere; the fix should leverage
those to avoid reinventing encoding logic.
```

Use unicode_std_stream in nbconvert stdout writer


We need to adjust the kernelspec listing API and its consumers to make the data easier to use and to prepare for future client-driven customization. Currently `GET /api/kernelspecs` returns a JSON array of kernel spec objects, with ordering determined on the server via the `_pythonfirst` heuristic. This makes it awkward to look up a specific kernel by name and to know which kernel is the system default. Change the API to return a JSON object with two keys:

- `"default"`: a string containing the kernel manager's default kernel name (as currently exposed by the kernel manager).
- `"kernelspecs"`: an object whose keys are kernel names and whose values are the corresponding kernel spec dictionaries (the same spec data as before, including a `"name"` field for redundancy).

Remove the `_pythonfirst` import and sorting from the handler; server-side ordering is no longer needed. Instead, both the notebook kernel selector menu (in the running notebook) and the "new notebook" menu on the tree/dashboard page must sort the kernels by their `display_name` property client-side, in ascending alphabetical order. When populating the menus, iterate over the kernels in that sorted order.

On the notebook kernel selector, simply use the new `data.kernelspecs` object directly rather than converting an array. On the tree page's new-notebook widget, similarly use `data.kernelspecs`, and ensure that if the user has not previously saved a preferred kernel name in `localStorage`, the widget falls back to the `"default"` name from the response (instead of leaving the selection unset).

All existing API consumers (which currently expect a top-level array) must be updated to match the new response shape. The API tests should be updated accordingly, without breaking the other kernelspec tests (e.g., fetching a single kernel spec and kernel resources remain unchanged). The change must not break the handling of invalid kernel directories; the API handler should still skip specs that fail to load and log an error, while still producing the `{"default": ..., "kernelspecs": {...}}` envelope.


```
**Title:** Single-line cells with a closing bracket before its opening counterpart should be rejected as invalid

**Problem:**
When a user types a single line that contains a closing bracket character that appears before its corresponding opening bracket (e.g., `]]`, `)(`, `}{`), IPython incorrectly treats the input as incomplete and opens a continuation prompt. This is confusing because the line is clearly invalid Python, yet the user is prompted for more input as if the cell could be completed.

**Desired behavior:**
For a **single-line** input cell where the bracket depth ever becomes negative (i.e., a closing bracket is encountered before any matching opening bracket), `TransformerManager.check_complete` should return `('invalid', None)` immediately, without attempting to treat the cell as incomplete or continuable. This mirrors the behavior of a syntax error and avoids the false continuation prompt.

**Acceptance criteria:**
1. A single-line input consisting only of `)` or `]` or `}` is marked invalid.
2. A single-line input like `)(`, `][`, `}{`, `())`, `)[])(`, or `()]` is marked invalid.
3. Legitimate single-line expressions that are syntactically incomplete but do **not** exhibit a negative bracket depth (e.g., `(1+2`, `[3, `) are **not** affected and continue to be reported as incomplete.
4. Multiline cells are **not** affected by this check; the overall bracket balancing logic takes precedence.

**Constraints:**
- The change must apply only when the cell consists of exactly one line.
- The determination of invalidity must be based solely on bracket depth tracking (round, square, curly) and not on general syntax analysis. A depth dropping below zero at any point in the line is sufficient to treat it as invalid.
- The existing behavior for all other inputs (valid/incomplete/multiline) must be preserved.
```

Bail on singleline cells with sunken brackets


```
Title: Completion does not work for loop variables in for-loops over iterables

IPython currently fails to provide completions for loop variables inside `for`-loops when iterating over built-in types such as `range`, `list`, `dict`, `str`, `enumerate`, etc. For example, after typing:

    for i in range(10):
        i.<TAB>

no completions are shown (e.g. `bit_length`), because the guarded evaluator cannot infer the type of `i`. The same issue occurs with tuple unpacking (`for key, value in ...`) and nested loops.

The root cause is that the evaluator's handling of `ast.For` only injected the loop variable into the transient locals if the target was a simple name, and it did so by calling `iterable.__iter__()` and `next()` without proper policy checks or support for complex assignment targets. Moreover, the allowed-calls whitelist does not include `__iter__` for most built-in types, so the policy check fails even when the iterable is safe.

The desired behavior:
1. Inside any `for` loop where the iterable is a safe, known type (including `range`, `list`, `dict`, `dict_keys`, `dict_values`, `dict_items`, `str`, `tuple`, `enumerate`, and other container-like collections from `collections`), get one sample element and bind it to the loop variable(s) in the transient locals so that attribute completions work.
2. Support tuple/unpacking targets, including nested loops.
3. Add the necessary `__iter__` methods (and for dict views, their `__iter__` methods, plus `enumerate` itself) to the `ALLOWED_CALLS` set so that the policy allows iterating over these types.

Acceptance criteria: In a completion context with evaluation policy `"limited"` (the default used for completions), typing `for i in range(10): i.` should offer `bit_length`; `for item in ['a', 'b']: item.` should offer string methods; `for key, value in {'a':1}.items(): key.` and `value.` should offer appropriate completions; nested loops and `enumerate` should also work.
```


Fix IPython's backslash+unicode completions: when completing a backslash followed by a non-ASCII character (e.g. `\β`, `\Ⅴ`), the completer should return exactly one reverse‑mapping suggestion — the LaTeX name (`\beta`) or the Unicode name (`\ROMAN NUMERAL FIVE`).  Currently it sometimes returns multiple or wrong matches because it seems to be merging results from the forward and reverse matchers instead of returning the first match it finds.

Requirements:
- Forward unicode‑name completion (e.g. `\snowman` → ☃) must still work.
- After the forward matcher fails, the two reverse matchers should run in order: first the LaTeX reverse matcher, then the unicode‑name reverse matcher.
- Only the first successful reverse matcher’s result should be used; do not combine results from multiple matchers.
- Single ASCII letters after a backslash (like `\f`, `\Y`) should still return zero completions (no false positives).


```
Bug: Callbacks fire twice when traitlets are bound with `bind`.

**Context**
`IPython.utils.traitlets` provides a `bind` utility to keep two traits synchronised. When two traits are bound and `on_trait_change` handlers are registered on the objects, changing one trait causes the callback on the other trait to be invoked twice instead of once.

**Expected behaviour**
- For any bound pair, a single change to one trait must trigger the change notification on the other trait **exactly once**.
- The initial synchronisation that `bind` performs (when the two traits start with different values) must also produce **exactly one** change notification on the trait that gets updated.
- After `unbind()` is called, further changes must no longer propagate, and no duplicate or stale callbacks may fire.

**Additional constraints**
- Bidirectional synchronisation must remain intact: setting either trait updates the other, and a sequence of changes must keep the values in sync.
- The fix must properly clean up listeners when `unbind()` is invoked, leaving no leftover callbacks that could cause notifications after the bind is undone.
- Existing tests that verify the value‑synchronisation behaviour of `bind` must keep passing.

Please fix the `bind` implementation so that every bound change results in a single callback invocation per affected trait.
```

Use closure to prevent traitlet callbacks from firing twice.


We'd like to add support for requesting contents metadata without the full body, so that API consumers—including the file browser—can efficiently fetch only the name, path, type, format, and other metadata for a directory, notebook, or file.

The GET endpoint `/api/contents/<path>` should accept a new optional query parameter `content`, taking values `0` or `1`, with a default of `1` (backward-compatible). When `content=0` is set, the returned model must contain `"content": null` regardless of type, and the server must not attempt to load the actual content (no file body, no directory listing). For directories this means the content list is omitted instead of being sorted and included. Any value other than `0` or `1` should be rejected with a 400 error.

Internally, `ContentsManager.get` already accepts a boolean `content` argument; the REST handler should simply extract the query parameter, validate it, and forward it to the manager. Similarly, the `validate_model` helper should be called with the appropriate `expect_content` flag so that model validation does not reject the `null` content.

On the JavaScript side, the `Contents` service's `get` method should accept an options field `content` set to `true` or `false`. When `false`, it should add `content=0` to the AJAX request parameters, so frontend code can opt into metadata-only fetching.

The expected behavior is:

- `GET /api/contents/some/path?content=0` → 200 with `"content": null` and correct `path`, `type`, `name`, `format` (for files).
- `GET /api/contents/some/path?content=1` or without the parameter → same as before.
- `GET /api/contents/some/path?content=2` → 400 Bad Request.

Ensure that existing directory listing and notebook/file content retrieval continue to work unchanged when `content` is not supplied or is `1`.


The kernel message protocol adapter (in `IPython/kernel/adapter.py`) translates messages between v4 (old) and v5 (current) wire formats. Several conversion bugs and missing transformations have been identified. Fix the adapter so that both directions (v4→v5 and v5→v4) produce correct messages.

**Required fixes:**

1. **Do not transform error/aborted replies.** If a message's content has `"status": "error"` or `"status": "aborted"`, the adapter should pass it through unchanged. Currently only missing msg_type handlers are skipped, but error replies are being corrupted by transformation.

2. **Fix `_version_str_to_list`.** The helper currently uses an undefined `content[key]` instead of the `version` parameter, causing a runtime error.

3. **`complete_request` v5→v4:** The output field should be `'block'`, not `'blob'`.

4. **`object_info_request` → `inspect_request` (v4→v5):** The input key is `'oname'`, not `'name'`. Set `cursor_pos` to `len(name)` (not `len(name)-1`). Discard the obsolete `line` return from `code_to_line`.

5. **`complete_reply` v4→v5:** The v4 reply includes a `matched_text` field; the v5 matches should be the suffixes that follow the matched text, not the full completions. Set `cursor_start` and `cursor_end` to 0, and include an empty `metadata` dict.

6. **`inspect_reply` (v4→v5):** Currently the adapter blindly discards all data. Instead, populate `data['text/plain']` from the available definition and docstring fields (`definition`/`call_def`/`init_definition`, and `docstring`/`call_docstring`/`init_docstring`). Use an `<empty docstring>` fallback.

7. **`display_data` (both directions):**
   - v4→v5: `application/json` values are strings in v4. If present, `json.loads` them into objects for v5.
   - v5→v4: `application/json` values are objects in v5. If present, `json.dumps` them into strings for v4.
   - Both conversions must be wrapped in a try/except to avoid crashing on bad data.

8. **Miscellaneous:** Adapter output messages that have a status field (like inspection and completion replies) should include `"status": "ok"` where appropriate.

All fixes should be symmetric and respect the field naming conventions of each protocol version. Edge cases like missing docstrings or missing JSON data must be handled gracefully.

further adapter tests, and associated fixes


```
Title: %timeit should gracefully fall back to ASCII when ± is unsupported

The %timeit magic includes the Unicode plus-minus sign (±) in its output string (e.g. "mean ± std"). In environments where sys.stdout.encoding cannot represent this character (like pure ASCII or certain legacy encodings), printing the result raises a UnicodeEncodeError.

To fix this, the output should use the ASCII fallback "+-" instead of ± whenever the stdout encoding cannot encode the ± character. Concretely:
- If sys.stdout is not available or has no encoding, use "+-".
- Otherwise, attempt to encode '±' using stdout.encoding; if the encoding succeeds, use "±"; if it raises an error, fall back to "+-".

The behavior when encoding is UTF-8 or other Unicode-aware encodings should remain unchanged.
```

Use +/- if stdout encoding does not support ±


**Summary**: Remove the dependency on the third-party `simplegeneric` package and migrate to Python's built-in `functools.singledispatch`, because IPython 7+ targets Python 3.4+ where this standard library solution is available.

**Problem**:
IPython currently uses `simplegeneric` to define the generic functions `inspect_object` and `complete_object` in `IPython.utils.generics`. These are public extension points that allow users and internal code to register type-specific implementations for custom completion and object inspection (e.g., via `complete_object.when_type(...)`). The `simplegeneric` package is an extra dependency in `setup.py` and is also listed in the `IPython.external` package.

Python 3.4 added `functools.singledispatch`, which provides equivalent single-dispatch functionality. Moving to the standard library eliminates a non-essential dependency and reduces potential maintenance burden.

**Expected changes**:
- Replace usage of `simplegeneric.generic` with `functools.singledispatch`.
- The registration API changes: `generic.when_type(T)` becomes `singledispatch.register(T)`. All call sites that register implementations for `inspect_object` and `complete_object` must be updated accordingly, including any in-code comments or examples that show patterns like `.when_type(...)`.
- Remove `simplegeneric` from the project's `install_requires` in `setup.py`.
- Clean up `IPython/external/__init__.py` to no longer reference `simplegeneric`.
- Ensure that existing custom completers and inspectors continue to work, and that the migration does not alter observable behavior.

**Non-obvious edge cases / safety constraints**:
- The `singledispatch` decorator returns a regular function rather than a class; the registration API is `@func.register(type)`, not `@func.when_type(type)`. Make sure that any decorated function that calls `.register` directly (not just via decorator syntax) also uses the new name.
- The generic functions `inspect_object` and `complete_object` are expected to raise `TryNext` when no implementation matches. This fallback behavior must be preserved exactly.
- There may be third-party code that registers implementations using `.when_type`. While a deprecation period could be considered, this task focuses on updating the core repository’s internal usage and dependency declarations; external code would need to adapt to `singledispatch`’s `.register` method independently, as the Python standard library symbol is the only one available after the change.

Replace simplegeneric.generic with functools.singledispatch


When running the IPython test suite, a small test extension module (`daft_extension`, located under `IPython/core/tests/daft_extension/`) is installed into the current Python environment using `pip install`. This creates two serious problems:

1. **Privilege issues** – the installation often requires administrator/sudo rights, which are not available in many development, CI, or container setups, causing test failures unrelated to the actual IPython code.
2. **Environment pollution** – after the tests finish, the extension remains installed globally. This can contaminate the environment and cause subsequent test runs (or even normal IPython usage) to behave incorrectly or fail when they try to load the now-persistent test extension.

The test that exercises the `%load_ext` magic should not modify the system Python environment at all. Instead, it should make the extension importable without a permanent installation (for example, by temporarily placing the extension’s directory on `sys.path` so that `import daft_extension` succeeds).

Additionally, to ensure the corrected test works when running from an IPython source distribution (where the extension files may not be present by default), the packaging configuration must be updated so that the `.py` files inside `IPython/core/tests/daft_extension/` are included as package data for `IPython.core.tests`.


```
Interact: `Output` widget causes regressions in output clearing and container structure

After the recent change that introduced `Output` widget into `interact`/`interactive`, the following regressions are observed:

1. Output clearing no longer affects the cell’s main output area.
   `clear_output(wait=True)` inside the callback now clears only the contents of the `Output` widget, so previous outputs can remain visible or behave unexpectedly. This breaks the common pattern of refreshing the entire visual output block.

2. The interactive container now includes an extra child widget (the `Output` instance) that was not present before.
   - Code that inspects `container.children` (e.g. to enumerate the auto-generated controls) sees one extra child, which can break layouts and custom scripting that relies on the previous child count.
   - The extra widget also adds unwanted visual space.

3. The `Output` context manager unconditionally calls `get_ipython().kernel`.
   This fails when no active IPython kernel is available (e.g. in terminal-only IPython sessions, certain headless or testing configurations), making `interact` unusable in those environments.

Expected behaviour: `interact`/`interactive` should not embed an `Output` widget. Output must be displayed directly in the cell’s standard output area, and clearing should behave as it did before the introduction of the `Output` widget. The container’s direct children should consist only of the parameter widgets (and, for manual mode, the manual button).
```

Revert "Use Output widget with Interact"


**Title:** Tab completion broken when kernel speaks protocol v4

**Description**
When the IPython notebook is connected to a kernel that uses message-spec version 4 (or anyone relying on the V4→V5 adapter), tab completion misbehaves. The old adapter:

- Strips the common prefix (`matched_text`) from each match sent to the frontend.
- Sets `cursor_start` and `cursor_end` both to 0, regardless of the actual completion range.

The frontend completer expects full match strings and absolute cursor positions (or sentinel values that let it compute positions relative to the current cursor). The mismatch causes the wrong text to be inserted, or the completion to fail silently.

**Additional issues found in the same adapter layer:**

1. **Spurious `status` field in `object_info_request`** – the V5→V4 adapter includes `"status": "ok"` in the adapted request content, which is not part of the V5 specification and may confuse kernels that strictly validate the message.
2. **Error replies are silently passed through** – when a reply has `status` equal to `error` or `aborted`, the adapter returns the message unchanged. There is no separate extension point for a future handler to process error messages, which limits the ability to adapt error payloads in the same pipeline.

**Expected behavior**

- Tab completion should work identically regardless of the kernel’s protocol version. The V4→V5 adapter should preserve the full match list and encode the completion range in a way that allows the notebook’s completer to calculate the replacement region using its own cursor position. A reasonable approach would be to pass a negative `cursor_start` representing the offset of the matched text from the cursor, and a `cursor_end` of `null`, so that the frontend can resolve the absolute positions.
- The V5→V4 adapter’s `object_info_request` must not inject a `status` field; its content should only contain the fields defined by the V4 spec.
- The adapter dispatch in `__call__` should route error-status replies through a dedicated (possibly default no-op) handler, so that subclasses can override it without changing the normal message-unwinding logic.

**Impact**
Users working with older kernels or custom kernel implementations that speak v4 will see broken autocompletion in the notebook. The incorrect `status` field may cause some kernels to reject legitimate `object_info_request` messages.


We need a repository-specific implementation request that captures the bug described in the commit. The key observable issue: IPython’s tab-completion for magic commands incorrectly hides completions when the magic name matches a global variable, even if the user has explicitly typed a leading `%` (or `%%`). The fix should ensure that explicit magic invocation via `%`/`%%` always proposes magic completions, while preserving the existing shadowing behavior when no percent prefix is present.

Here's a maintainer-style ticket:

**Title:** Explicit magic completion should not be shadowed by global names

**Description:**

In IPython, when a magic command (e.g., `%matplotlib`) has the same name as a Python variable in the user’s namespace, completion for that magic is intentionally suppressed to avoid confusion. For example:

```python
In [1]: matplotlib = 1
In [2]: mat<tab>        # does not show %matplotlib
```

This behavior is correct. However, the same suppression happens even when the user has **already typed the leading `%`** (or `%%` for cell magics):

```python
In [3]: %mat<tab>       # %matplotlib should appear, but currently does not
```

If the user explicitly types `%`, they are clearly requesting a magic command, and the global variable should not block those completions.

**Expected behavior:**

- A completion like `%mag<tab>` (or `%%mag<tab>`) should always list matching line/cell magics, regardless of whether a global name in the namespace coincides with the magic.
- For completions **without** a leading `%` (e.g., `mag<tab>`), the existing shadowing rule must remain unchanged: magics that match global names should still be excluded to prevent ambiguity.

**Additional context / constraints:**

- The magic escape character is `%`, and cell magics use `%%`. Both cases should behave identically with respect to shadowing (i.e., prefixes `%` and `%%` signal explicit magic intent).
- This applies to both line magics and cell magics.
- The existing ordering and presentation of completions should remain as they are; the only change is that the explicit percent cases bypass the global-name filter.

**Reproduction steps:**

1. Start IPython.
2. Define a variable with the same name as a magic (e.g., `matplotlib = 1`).
3. Type `%mat` and press Tab. No completions are offered. After the fix, `%matplotlib` should be suggested.
4. Without the percent (`mat<tab>`), the magic should still be hidden, confirming that the original shadowing rule is preserved.

Do not shadow explicit magic completion.


When using the `%%script` cell magic (including all derived magics like `%%bash`, `%%ruby`, `%%python`, etc.), the stdout and stderr of the subprocess are completely buffered. Nothing appears on the terminal until the subprocess exits. This makes long-running scripts tedious because you cannot see progress or incremental output.

**Desired behavior**
- Stdout and stderr produced by the subprocess must be written to the terminal **line‑by‑line**, without waiting for the process to finish. Each line (or small chunk) should be flushed immediately so the user sees incremental output.
- When `--out` or `--err` flags are used to capture output into Python variables, the *terminal* display should remain suppressed (the current behavior), but the captured variable should still be populated. The existing semantics for those flags must not regress.
- Background execution (`--bg`) must continue to work; output streams should still be accessible as file-like objects.
- The `--raise-error` flag must still raise an exception when the subprocess exits with a non‑zero return code.
- The streaming behavior must work on all supported platforms including Windows, and must correctly handle both stdout and stderr, as well as any mix of `--out`/`--err` options.

Line-buffered output for %%magic scripts


**Title:** Allow clients to select a kernel spec when creating a kernel via the REST API

**Background**
The notebook’s kernel infrastructure already supports running different kernel specs (e.g., different language kernels). The `KernelManager` accepts a `kernel_name` parameter to select the spec. However, the kernels REST API (`POST /api/kernels`) does not expose this capability – it always starts a kernel with the default Python spec, and the returned kernel model does not report which spec is active.

**Request**
Extend the kernels REST API so that clients can choose which kernel spec to launch when creating a new kernel.

- **Request format**
  `POST /api/kernels` should accept an optional JSON body containing a `"name"` field whose value is the kernel spec name (i.e., the same `kernel_name` that would be passed to `KernelManager.start_kernel()`).
  - If no JSON body is provided, the endpoint should return a `400` error.
  - If the JSON body is present but does not contain the `"name"` key, return a `400` error with a descriptive message.

- **Response changes**
  Both the response to `POST /api/kernels` (the created kernel model) and the kernel model returned by `GET /api/kernels` and `GET /api/kernels/{kernel_id}` should include a `"name"` field that reflects the `kernel_name` of the running kernel. This allows clients to know which spec a kernel is using without extra round-trips.

- **Propagation**
  The kernel name must be threaded from the API handler through the `MappingKernelManager` into `MultiKernelManager.start_kernel()`, and ultimately to the new `KernelManager` instance, preserving the existing behavior of passing along extra arguments and defaults.

**Acceptance criteria**
- Sending a valid `{"name": "<kernel_spec_name>"}` creates a kernel with that spec, and the response model contains `"name": "<kernel_spec_name>"`.
- Listing kernels includes the `"name"` field for each kernel.
- Missing or invalid JSON body yields a `400` error with a clear message.
- Existing functionality (default Python kernel when no spec is explicitly requested) remains unchanged from the API consumer’s perspective, except that the `"name"` field now appears in models.
- The notebook frontend and existing tests continue to work; the API’s new requirement for a JSON body is publicly documented for API consumers.

Add support for different kernel specs to kernels REST API