Difficulty in optimizing machine learning models using hybrid approaches

Signals over time

2 observed signals across 1 sources, tracked for 1 days. Confidence: low.

What people are saying Observed

• “Interesting, I had been wondering if you could cycle distillation and then splitting weights to turn a saturated model into a non-saturated with the same parameter count for further training. Train until you stop getting sidnificant improvements. distill to a quarter size model. Expand back up, and continue training. Split the weights so W1+W2 = W with W1 = (W + Randoffset)/2, W2 = (W - RandOffset)/2. Double the width of the layers using the split weights, you get the same result from a 4x size network. My hypothesis is that this has way more scope to train that the model you distilled from.”

  Hacker News · question

• “Even though this is way more expensive than backprop, could a hybrid approach where you fine tune an existing checkpoint that's been backpropped unlock further gains? It would be cool to apply this to different stages and see if that affects the learning trajectory”

  Hacker News · question