@@ -336,6 +337,106 @@ func TestSlowWatcherResync(t *testing.T) {
 	}
 }

cache: Add on-demand progress notification from cache

Don't special function per each type of event. Just have function to collect N responses and return them. Use the response to compare with expected response.


@@ -101,6 +101,14 @@ func New(client *clientv3.Client, prefix string, opts ...Option) (*Cache, error)
 	return cache, nil
 }

cache: Add on-demand progress notification from cache

> `RequestProgress` method can be shifted to a new `Watcher` interface which will be returned on `cache.Watch() ` call. Does that make sense?

No, I said that this is current interface, not that we want to fix it now.

> `the revision returned is specific to watcher.` -> As far as I know, etcd sends progress notification to all the watchers on the single grpc stream when a ProgressRequest call is made and that's why I was broadcasting it to all the watchers irrespective of who asked for the revision.

Just remove `cache's current revision to all active local watchers` as this is inaccurate.


@@ -152,6 +157,7 @@ func TestConfigFileFeatureGates(t *testing.T) {
 			expectedFeatures: map[featuregate.Feature]bool{
 				features.StopGRPCServiceOnDefrag: false,
 				features.DistributedTracing:      false,

KEP-4578: migrate experimental-initial-corrupt-check flag to feature gate.

Can we add a test for the enablement of the flag?


grpcproxy: return io.EOF at end of server-streaming RPCs

This seems a little misleading. This callback is supposed to create a stream instead of sending or recving any messages. In this test, you don't have to create a real stream, but suggest to send msg  in a separate goroutine.


@@ -101,6 +101,14 @@ func New(client *clientv3.Client, prefix string, opts ...Option) (*Cache, error)
 	return cache, nil
 }

cache: Add on-demand progress notification from cache

>This is an implementation detail that can change and should not be documented.

But should the lagging watchers actually get the progress revision based on the lag? For example:
A lagging watcher is still receiving events with revision 4 and the cache is at revision 6. Now, the progress notification sent to it should be 4 and not 6, correct?


@@ -721,147 +746,223 @@ var (
 )

 type getTestCase struct {
 }

 var getTestCases = []getTestCase{
 	{
 	},
 	{
 	},
 	{
 	},
 	{
 	},
 	{

Please add serializable read at rev=latest


@@ -101,6 +101,14 @@ func New(client *clientv3.Client, prefix string, opts ...Option) (*Cache, error)
 	return cache, nil
 }

cache: Add on-demand progress notification from cache

Also, I believe the expectation is not that the lagging watchers should receive progress revision based on their lag? That's why I am not sending them progress notification until they become active.


@@ -130,6 +130,30 @@ type AuthConfig struct {
 	Password string `json:"password"`
 }

Update `epHealthCommandFunc` to reuse `clientConfigFromCmd`

Maybe add a test that ensures that Clone doesn't get outdated if any new field is added?


@@ -135,12 +135,23 @@ func (c *Cache) Watch(ctx context.Context, key string, opts ...clientv3.OpOption
 		defer c.waitGroup.Done()
 		defer close(responseChan)
 		defer c.demux.Unregister(w)
 		for {
 			select {
 			case <-ctx.Done():
 				return
 			case <-c.internalCtx.Done():
 				return

cache: Support WithProgressNotify() for cache watch

Progress should only be sent if there was no other response sent.


@@ -67,7 +67,7 @@ func Repair(lg *zap.Logger, dirpath string) bool {

 		case errors.Is(err, io.ErrUnexpectedEOF):
 			brokenName := f.Name() + ".broken"

Ensure consistent file permissions on broken WAL files

Don't think it's the same, the LockFile function not only creates file but takes a OS lock on it. Maybe what I proposed doesn't make sense here as we need a function that would would need different options for
* Creating a normal file or a locked file (fileutil.LockFile vs os.OpenFile)
* Create a new file or truncating existing file. (os.O_WRONLY|os.O_CREATE vs os.O_CREATE|os.O_WRONLY|os.O_TRUNC)

It might be useful to have a single function to create WAL files to show all the options, however I'm not sure that it would be beneficial. cc @ahrtr


grpcproxy: return io.EOF at end of server-streaming RPCs

the change looks good. could you please add this case as well?

Just want to verify it in e2e. Thanks


@@ -26,7 +26,13 @@ import (
 type RangeOptions struct {
 	Limit int64
 	Rev   int64
 	Count bool

mvcc: add WithTotalCount optimization for Range

`Count` is not the best name here, adding comment doesn't help a lot. Better name would be `CountOnly` like we have in etcd client options.

Before making your change, can you send a PR that will align the `RangeOptions` field names with client?


@@ -135,12 +135,23 @@ func (c *Cache) Watch(ctx context.Context, key string, opts ...clientv3.OpOption
 		defer c.waitGroup.Done()
 		defer close(responseChan)
 		defer c.demux.Unregister(w)

cache: Support WithProgressNotify() for cache watch

Ticker doesn't work as you think. Please read https://pkg.go.dev/time#NewTicker `The ticker will adjust the time interval or drop ticks to make up for slow receivers`


@@ -135,12 +135,29 @@ func (c *Cache) Watch(ctx context.Context, key string, opts ...clientv3.OpOption
 		defer c.waitGroup.Done()
 		defer close(responseChan)
 		defer c.demux.Unregister(w)
 		for {
 			select {
 			case <-ctx.Done():
 				return
 			case <-c.internalCtx.Done():
 				return

cache: Support WithProgressNotify() for cache watch

Do we need to check it? I think we can just depend on reseting timer.

My understanding of the flow:

T0 Watch is created, let's say a watch event returned reseting timer and setting `needsProgress = false`
T1 timer sends event to `progressChan` , no event is send as `needsProgress` is set to false. Timer is reset and `needsProgress = true`
T2 timer sends event to `progressChan`. Event is sent. Timer is reset and `needsProgress = true`
T3 repeat previous step.

So basically first progress notify is delayed to 2*`ProgressNotifyInterval`, while all following are sent with `ProgressNotifyInterval`.

The only question is whether we can depend on just timer, basically what happens if we call "Reset" on timer that already emitted a value. Based on Go documentation we should be good as of Go 1.23 https://pkg.go.dev/time#NewTimer:

```
Before Go 1.23, the channel associated with a Timer was asynchronous (buffered, capacity 1), which meant that stale time values could be received even after [Timer.Stop](https://pkg.go.dev/time#Timer.Stop) or [Timer.Reset](https://pkg.go.dev/time#Timer.Reset) returned. As of Go 1.23, the channel is synchronous (unbuffered, capacity 0), eliminating the possibility of those stale values.
```

Please note this flows should be tested with fake timer like progressNotifier to ensure this behavior doesn't change.


@@ -193,7 +193,7 @@ func (ep *ExpectProcess) ExpectFunc(ctx context.Context, f func(string) bool) (s

 		select {
 		case <-ctx.Done():

fix: enable error-is-as rule from testifylint

I'm not entirely sure if this change is ok. Previously the `context.DeadlineExceeded` error was "swallowed" here.

This was not discovered until now because the test for this wrongly used `require.ErrorAs`, which is not appropriate for a sentinel error value like `context.DeadlineExceeded`. Instead `require.ErrorIs` must be used to test for the appropriate error but that didn't work because the error was swallowed.


@@ -546,6 +546,71 @@ func TestCacheWatchPrefixProgressNotify(t *testing.T) {
 	}
 }

cache: Add on-demand progress notification from cache

Makes sense. This would require sending progress request to the etcd server. Should we just implement that instead of this? But this would also mean that each progress request will now make a network call.


@@ -1011,8 +1011,15 @@ def test_get_pipeline_class_from_flax(self):

 class CustomPipelineTests(unittest.TestCase):
     def test_load_custom_pipeline(self):

Should we investigate the `ValueError` messaging as well (it should have something related to the use of `trust_remote_code` or not something else)?


@@ -433,7 +437,7 @@ def get_ltx2_video_vae_config(version: str) -> Tuple[Dict[str, Any], Dict[str, A
                 "downsample_type": ("spatial", "temporal", "spatiotemporal", "spatiotemporal"),
                 "upsample_residual": (True, True, True),
                 "upsample_factor": (2, 2, 2),

Where is `timestep_conditioning` used in the corresponding Video VAE class?


@@ -1045,8 +1058,15 @@ def test_load_custom_github(self):
         assert pipeline.__class__.__name__ == "UnetSchedulerOneForwardPipeline"

     def test_run_custom_pipeline(self):

Could we add a check for the error message content

e.g.

```py
 with self.assertRaises(ValueError) as cm:
      DiffusionPipeline.from_pretrained(
          "google/ddpm-cifar10-32", custom_pipeline="hf-internal-testing/diffusers-dummy-pipeline"
      )
  self.assertIn(
      "Please pass the argument `trust_remote_code=True` to allow custom code to be run.",
      str(cm.exception),
  )
```


@@ -353,15 +355,6 @@ def check_inputs(
                 f" {negative_prompt_embeds}. Please make sure to only forward one of the two."
             )

fix(qwen): fix CFG failing when passing neg prompt embeds with none mask

What is the reasoning behind this?

What happens when users pass the embeds and not the masks? Maybe we should warn them?


@@ -1674,21 +1678,6 @@ def download(cls, pretrained_model_name, **kwargs) -> str | os.PathLike:
                 custom_class_name = config_dict["_class_name"][1]

             load_pipe_from_hub = custom_pipeline is not None and f"{custom_pipeline}.py" in filenames

Well just to be clear, using the same local-dir as Hub repo is a possible trick to hide the attack as anyone who didn't pre-download wouldn't be affected but any local directory is affected, and the local directory could be from other sources like  `snapshot_download` or `git clone`.


@@ -584,9 +584,7 @@ def __call__(

         device = self._execution_device

fix(qwen): fix CFG failing when passing neg prompt embeds with none mask

How do we handle `negative_prompt_embeds_mask`? 👀 Is this block affected?

https://github.com/huggingface/diffusers/blob/71a6fd9f0df04d3764dfa999268a05d87903a85a/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py#L261-L267


@@ -353,6 +353,15 @@ def _get_llama_prompt_embeds(
             text_crop_start = crop_start - 1 + image_emb_len
             batch_indices, last_double_return_token_indices = torch.where(text_input_ids == double_return_token_id)

Fix IndexError in HunyuanVideo I2V pipeline

should we just replace the hard coded numbers with this logic instead? regardless the transformer version?


@@ -234,3 +234,29 @@ def test_vae_tiling(self, expected_diff_max: float = 0.2):
             expected_diff_max,
             "VAE tiling should not affect the inference results",
         )

fix(qwen): fix CFG failing when passing neg prompt embeds with none mask

Should we not be using it because it's the core thing that we're fixing here?


@@ -2323,17 +2325,45 @@ def load_components(self, names: list[str] | str | None = None, **kwargs):
                     elif "default" in value:
                         # check if the default is specified
                         component_load_kwargs[key] = value["default"]
             try:
                 components_to_register[name] = spec.load(**component_load_kwargs)
             except Exception:
                     f"\nFailed to create component {name}:\n"
                     f"- Component spec: {spec}\n"
                     f"- load() called with kwargs: {component_load_kwargs}\n"
                     "If this component is not required for your workflow you can safely ignore this message.\n\n"
                     "Traceback:\n"
                 )

[modular] not pass trust_remote_code to external repos

Of course, we cannot generalize to all possible combinations but what if the underlying component is a scheduler or a non-modeling component?

For schedulers, we could have `AutoScheduler` or something so that users won't have to remember a specific class name. For others (such as tokenizers, image processors), we already have "auto" classes.

I guess it's not that big a use case yet, but just wanted to flag.


@@ -1342,8 +1429,11 @@ def convert_and_load_state_dict_in_model(
             # `cancel_futures=True` in case the program was interrupted, to avoid wasting time on exit
             thread_pool.shutdown(wait=False, cancel_futures=True)

[loading] Clean way to add/remove full parts in checkpoint names

Does it not create another loophole if some conversions aren't used by model? But prob if the test checks full list of conversions, we can consciously decide to add task-specific patterns like `lm_head`


@@ -600,9 +617,9 @@ def __init__(
         self.block_size = block_size
         self.hidden_dim = config.hidden_size
         self.activation_scheme = activation_scheme

Fix configuration reading and error handling for kernels

We could use `hasarttr` but it's not very nice to read.

vLLM has a very similar utility called `getattr_iter` for checking multiple names in configs.


@@ -989,13 +1016,86 @@ def setup(self, args, state, model, **kwargs):
                 logger.info("Could not log the number of model parameters in Trackio due to an AttributeError.")
         self._initialized = True

     def on_train_begin(self, args, state, control, model=None, **kwargs):
         if not self._initialized:
             self.setup(args, state, model, **kwargs)

     def on_train_end(self, args: TrainingArguments, state, control, model=None, processing_class=None, **kwargs):

Update `trackio` integration to use Buckets and "freeze" Space after training

from the code in _resolve_trackio_space_id, it feels like gradio_space_id is never None no or  `SQLiteStorage.get_space_id(args.project) can return None` ? Also maybe we can rename a bit the method to call it `get_trackio_space_id` maybe ?


@@ -1324,21 +1324,15 @@ def is_base_mistral(model_id: str) -> bool:
                 # Detect if we can skip the mistral fix by
                 #   a) having a non-mistral tokenizer
                 #   b) fixed version of transformers

[`fix`] Always early return for non-Mistral models in _patch_mistral_regex

Can repro now, but isn't this a case where the versioning boundaries were wrong? Version `4.57.3` was missing in between; the last comparison should be `version.parse(transformers_version) > version.parse("4.57.2")` instead.

This current version assumes model type to exist which can also fail (None). And honestly, I don't want to accept a case where the version is None - that should indicate that we should still check --> maybe add a warning in that case for more user information.


@@ -4783,6 +4783,11 @@ def test_reverse_loading_mapping(self, check_keys_were_modified=True, skip_base_
                 conversions = get_model_conversion_mapping(model, add_legacy=False)
                 if len(conversions) == 0:
                     self.skipTest(f"No conversion found for {model_class}")

[loading] Clean way to add/remove full parts in checkpoint names

same q as in `save dummy model test`, are we sure this works when model without prefix is saved and loaded back?


@@ -765,7 +765,8 @@ def create_accelerator_and_postprocess(self) -> None:
             else:
                 self.args.gradient_accumulation_steps = grad_acc_kwargs["num_steps"]
         else:

I meant in the issue that this should always be the case. We don't even need the else here.


@@ -73,6 +80,11 @@ def _load_triton_kernel():
     _triton_available = False  # mark attempted before any early exit

     kernel = lazy_load_kernel("finegrained-fp8")

Fix configuration reading and error handling for kernels

We never get to that error because if kernels is not installed then `kernel` is `None` and `getattr(None, some_kernel)` fails with an unhelpful error


@@ -1041,13 +1141,22 @@ def on_push_begin(self, args, state, control, model, **kwargs):

         space_id = self._trackio.context_vars.current_space_id.get()
         if space_id is None:

Update `trackio` integration to use Buckets and "freeze" Space after training

maybe we can have a seperate method to get this space_id ? it will be more clearer. Not sure how this relates to _resolve_trackio_space_id above


@@ -192,6 +191,22 @@ def _build_checkpoint_conversion_mapping():
             WeightRenaming("attention_layer_norm", "input_layernorm"),
             WeightRenaming("feedforward_layer_norm", "post_attention_layernorm"),
         ],

Fix configuration reading and error handling for kernels

Can we instead make this copy from qwen 3.5 text and qwen2 moe


@@ -778,6 +781,245 @@ def test_register_checkpoint_conversion_mapping_overwrites(self):

         self.assertEqual(len(get_checkpoint_conversion_mapping("foobarbaz")), 2)

[loading] Clean way to add/remove full parts in checkpoint names

to be clear, let's take a llava example. User creates random llava arch, trains it and saves without `prefix`. Then if the model was loaded `from_pretrained`, we still match keys right?


@@ -111,13 +112,19 @@ async def handle_request(self, body: dict, request_id: str) -> StreamingResponse
         gen_manager = self.generation_state.get_manager(model_id, use_cb=use_cb)
         processor_inputs = self.get_processor_inputs_from_messages(body["messages"], modality)

         inputs = processor.apply_chat_template(
             processor_inputs,
             add_generation_prompt=True,
             tools=body.get("tools"),
             return_tensors=None if use_cb else "pt",
             return_dict=True,
             tokenize=True,

managed to use it but `torchcodec` is required for that. Otherwise, it fails back to the other lib and it fails. Also `torchcodec` + `ffmpeg` was a bit of a pain to install correctly.
Also, we should maybe should force the user to install `torchcodec` when using this no cc @eustlb ?


@@ -270,6 +270,12 @@ def test_resize_tokens_embeddings(self):
     def test_tied_weights_keys(self):
         pass

skip 1 invalid test case for higgs_audio_v2

Wouldn't the other FA tests also fail if that were the case? Just not sure about the reasoning here


fix transformers + torchao nvfp4 serialization

We are always using the latest version of torchao in our tests so maybe it is fine to not do the try except ?


@@ -885,3 +886,87 @@ def test_resource_warning(tmp_path):
         else []
     )
     result.stdout.fnmatch_lines([*expected_extra, "*1 passed*"])

Add `--max-warnings` option to fail test runs

Could you please also add some tests with configurations that filter warnings, together with the new --max-warnings option? This will capture the current behavior and ensure we do not regress in the future.


@@ -42,13 +42,22 @@ where
     type Output = io::Result<usize>;

     fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<io::Result<usize>> {
         let me = self.project();

         if !me.buf.has_remaining() {
             return Poll::Ready(Ok(0));
         }

Implement vectored writes for `write_buf`

I have a project in which I checked if `Buf::chunk().len()` < `Buf::remaining()` before using vectored writes and it resulted into a small performance boost. Could be useful here too.


@@ -133,8 +133,12 @@ impl<T: Cancellable + Completable + Send> Future for Op<T> {
                 let entry = entry_opt.take().expect("Entry must be present");
                 let waker = cx.waker().clone();
                 // SAFETY: entry is valid for the entire duration of the operation

io-uring: Change `Completable` to not return io::Result

When `register_op` returns an error, we should fail.


sync: return `TryRecvError::Disconnected` from `Receiver::try_recv` after `Receiver::close`


@@ -136,6 +137,44 @@ impl<T> sealed::FromStreamPriv<T> for Vec<T> {
     }
 }

impl `FromStream` for `std::collections::*`

For ordered collections, can you add a test that they appear in the same order as when using normal `Iterator::collect`?


@@ -205,6 +205,17 @@
 //! This is done with [`Builder::enable_io`] and [`Builder::enable_time`]. As a
 //! shorthand, [`Builder::enable_all`] enables both resource drivers.
 //!

rt: clarify the documentation of `Runtime::spawn`

You placed this under "Runtime Configurations", but it's not a way you can configure the runtime. Please make it into its own section.

Also, a different section name such as "Driving the runtime" would make more sense to me.


@@ -241,7 +241,7 @@ doc! {macro_rules! try_join {
             if is_pending {
                 $crate::macros::support::Poll::Pending
             } else {

Can you re-export it from `$crate::macros::support` instead? That's more reliable than using `::std`.


Please also add a test for LocalRuntime.


@@ -407,6 +407,24 @@ async fn try_join_next_with_id() {
 mod spawn_local {
     use super::*;

task: add tests for `spawn_local` in panic scenarios

I think these tests are also missing for `JoinQueue`


@@ -6,16 +6,17 @@ use std::future::Future;

 cfg_rt! {
     /// Spawns a new asynchronous task, returning a

rt: clarify the documentation of `Runtime::spawn`

Rustdoc vs. spellcheck - the fight of the tools :-)
Maybe this is the reason why backticks are in use here!
You could try with ``[`JoinHandle`]``. If spellcheck still complains then you will need to update https://github.com/tokio-rs/tokio/blob/4bc2a15d28554426a500ce73be969ecd9ed6dd89/spellcheck.dic:
1) increment the number on the first line (e.g. 311 -> 312)
2) Insert `JoinHandle` (between `iteratively` and `Kotlin's`)


@@ -200,15 +200,14 @@ impl Runtime {
         &self.handle
     }

     ///
     /// This spawns the given future onto the runtime's executor, usually a
     /// thread pool. The thread pool is then responsible for polling the future
     /// until it completes.
     ///

rt: clarify the documentation of `Runtime::spawn`

It is recommended to add two sentences explaining the behavior of `Runtime::spawn`: one for current thread runtime and another for multi-threaded runtime.

Please keep it concise—verbose documentation makes it harder for downstream users to understand.


@@ -164,9 +164,8 @@ impl Handle {
     /// thread pool. The thread pool is then responsible for polling the future
     /// until it completes.
     ///

rt: clarify the documentation of `Runtime::spawn`

> IMO it will be clearer if the documentation explain when it will be executed immediately and when it needs blocking. I.e. be more specific instead of letting the reader guess and experiment.

@Its-Just-Nans Could you explain the correct behavior in docs, and then we can evaluate weather it is too long.


@@ -112,6 +112,28 @@ describe('reactivity/reactive', () => {
     expect(dummy).toBe(false)
   })

fix(reactivity): handle `Set` with initial reactive values edge case

The test case for `Map.set` here is incorrect because you only modified `add` without changing `set`. I think this part of the test would pass even without this PR. Additionally, the `observedMap` should have an initial value like `observedSet`