<rant>
Generally, the current thought is to train a model on a ton of data- broad data, usually.
I think the idea is to turn it into a human.
But is this really what we need.
I see modular models
A model that knows all Maths
A model that knows all Art
and etc
and you query based on the need to pull that data.
A single giant model is like a monolithic database
It knows trillions of things
But I just want to know how to tie a tie.
Speed, cost, time. Machine learning is expensive. It seems magnificently inefficient to ask a one trillion parameter model how to fry an egg.
Why not fork the question to lightweight edge models? Use proxy caching for most commonly requested embeddings and leave the GPU compute to breathe a little?
Don’t forget the complexity of added RAG to this and still keeping it all organized. Wow.
</rant>