Fixed seed 42？
Hello,
I noticed that the seed is fixed to 42 when generating the mask for each CSI sample. As a result, the masked positions remain the same across all samples. I'm wondering if this might be detrimental to training — or perhaps even a bug?
My understanding is that the masked positions are typically expected to be randomized.
Based on my tests, setting "seed=None" does not seem to significantly affect the training results.
I have another question I’d like to discuss with you. During the pretraining process, I noticed that the training_loss recorded in the CSV file is consistently higher than the validation_loss. Could this be due to the use of dropout?
masked_pos = np.random.choice(range(1, tokens_size), size=n_masks, replace=False)
We would like to point out another issue to the authors: in the above statement (within the lwm_tokenizer.make_sample function), the lower bound starts from the first channel patch excluding the CLS patch, and the upper bound is n_patches - 1. This implies that the last channel patch is never masked.