- focus on the non-convexity problem
- bring the solutions proposed by Gemini
- pass those to the claude project
- redo the tests using only the small model with maximum weight decay
- testing with
- try lowering the embedding dimension of each job (maybe the minimum is the number of machines ... and the neural architecture should have the inductive bias grouping the embeddings 2 by 2 ... like a 1D conv ...) -let's test this with Sm, Wd=10, metropolized filling ... try all the tops ... if possible get an overview of the number of tops there exist in each instance, and their volume ...
==========================
the current code is creating the training and validation split just by randomlu shuffling the data and splitting into two splits (train and validation). I'd like to improve this logic to make the train and validation split reflect the same data distribution in terms of makespan, which I'd like to accomplish through the following logic: take the data (the schedules and their associated makespans), "group" together the schedules that have the same makespan, then take a fixed percentage train_ratio from each group to compose the final training split, while the remaining (schedule, makespan) pairs will go to the validation split. If a group is so small that there will be no schedules from it left for the validation split, it's ok just give everything to the training split
==========================
In the function recover_schedules, I'd like to implement the "metropolized filling" algorithm to escape local optima. The idea of this metropolized filling is to "mimic tabu search" by the following: when a local optima is detected during optimization over X2 (by some heuristic like gradients being near zero), we add a penalty term to the optimized function (the optimized function being total_loss in the code) which penalizes the "proximity" to that local optima. For example, if we detect that we are achieving a local optimum at x_0, we would add the term "1/||x-x_0||" (x here symbolizing X2 in the code). When we detect additional local optimum (like x_1, x_2, and so on), we similarly add their penalty term to the objective function, resultion in the objective function looking like f(x) + 1/||x-x_0|| + 1/||x-x_1|| + 1/||x-x_2|| ... We could use some FIFO mechanism to limit the "penalized local optimas" to the N most recent ones. Can you update the code to implement this. Make sure to add a boolean parameter that allows to activate/desactivate this metropolized filling mechanism. Also remove everything related to the "langevin" mechanism that is currently implemented. Also regarding the detection mechanism of local optima, i suggested earlier to check for the gradients but if you know a better way feel free to propose it.
================================= uniform sampling over schedules local optima avoidance schedules enrichment in recursive mode