Discussion about this post

User's avatar
Doug's avatar

I agree that most of the time the problem is that your model is bad and you should feel bad, but I think 'overfitting' is still a useful concept to have in your mind as an explanation for what just happened when your kitchen-sink ML model fits absolutely beautifully in training and just doesn't on the validation/test data. The moral high ground says not to train models by throwing the kitchen sink in to the neural net blender with no thought about what features you think ought to fit, but if you went with that you'd have missed most of the huge gains in AI over the last decade or two. We are still struggling to make any sense whatsoever of what features LLMs have ended up learning about language (and maybe indirectly the world?!) but they sure can generate very plausible text.

Expand full comment
John Quiggin's avatar

Isn't the absence of explicit theory in machine learning (roughly, discriminant analysis on steroids) central to the problem. There's always an implicit theory but if it is just "all these variables must be related in some way", you're committed to overfitting from the start.

Expand full comment
15 more comments...

No posts