They aren’t comparable. GPS is deterministic and simple so as you said it could be reduced to an FPGA or ASIC.
LLMs are partially matrix math and probabilities but they’re also more complex than that and require a large amount of RAM to run even the first time. Each query added to the context increases the RAM needed further.
I dont believe the memory, both ram and storage combine, are what’s causing data centres to gobble up power and associated resources (electricity, cooling,…).
Its the GPUs.
I’m not counting on a future with LLMs by any means, but there’s tons of efficiencies we can do to make the cost go down. Were just not doing it.
We saw the same thing with crypto mining. It started on GPUs but then speficially designed silicon (asyc miners) were vastly more efficient than a GPU rig.
It still needed memory yes, but again, thats not the constraint.
For crypto mining, the speed of computation before the next block was mined was the constraint so GPUs were being used for parallel computation before ASIC miners were built. This is a great example of where ASICs shine: deterministic simple algorithms that don’t require much if any I/O.
LLMs have a minimum RAM requirement in order to even load the model so even if you built a dedicated LLM card with an ASIC, you would still need at least that amount of RAM on the card. You’d probably want more RAM in order to store context because that takes RAM as well. At that point you pretty much have a GPU with an ASIC instead of more general compute units and no video outputs.
You might get some improvements by using an ASIC for LLMs but not nearly as much as when ASICs started being used for mining.
They’re actually not because they have randomness built in. Since we are working with probabilities, it won’t always pick the next token that has the highest probability and the randomness can be tuned via a “temperature” setting to make it more or less likely that it will choose the most probable token.
The weights for the model could be stored in a firmware chip but you still need RAM because it pulls all the weights into RAM in order to perform the calculations.
The LLM itself is deterministic. It outputs a vector that we interpret as a probability distribution over the set of tokens. It’s the program using the output of the LLM, such as a chatbot program, that selects an individual token using (or not using) these vector elements as weights.
The weights do not change once the model is trained. This is why I am suggesting they could be incorporated directly into the structure of an ASIC for a specific model, rather than storing them in memory.
Edit: Of course another major factor could be that the models are just to big to be wholly implemented in a single IC by any currently existing manufacturer.
They aren’t comparable. GPS is deterministic and simple so as you said it could be reduced to an FPGA or ASIC.
LLMs are partially matrix math and probabilities but they’re also more complex than that and require a large amount of RAM to run even the first time. Each query added to the context increases the RAM needed further.
I dont believe the memory, both ram and storage combine, are what’s causing data centres to gobble up power and associated resources (electricity, cooling,…).
Its the GPUs.
I’m not counting on a future with LLMs by any means, but there’s tons of efficiencies we can do to make the cost go down. Were just not doing it.
We saw the same thing with crypto mining. It started on GPUs but then speficially designed silicon (asyc miners) were vastly more efficient than a GPU rig.
It still needed memory yes, but again, thats not the constraint.
For crypto mining, the speed of computation before the next block was mined was the constraint so GPUs were being used for parallel computation before ASIC miners were built. This is a great example of where ASICs shine: deterministic simple algorithms that don’t require much if any I/O.
LLMs have a minimum RAM requirement in order to even load the model so even if you built a dedicated LLM card with an ASIC, you would still need at least that amount of RAM on the card. You’d probably want more RAM in order to store context because that takes RAM as well. At that point you pretty much have a GPU with an ASIC instead of more general compute units and no video outputs.
You might get some improvements by using an ASIC for LLMs but not nearly as much as when ASICs started being used for mining.
LLMs are also deterministic and the weights for a model could be ‘hard-coded’ into a chip, no?
They’re actually not because they have randomness built in. Since we are working with probabilities, it won’t always pick the next token that has the highest probability and the randomness can be tuned via a “temperature” setting to make it more or less likely that it will choose the most probable token.
The weights for the model could be stored in a firmware chip but you still need RAM because it pulls all the weights into RAM in order to perform the calculations.
The LLM itself is deterministic. It outputs a vector that we interpret as a probability distribution over the set of tokens. It’s the program using the output of the LLM, such as a chatbot program, that selects an individual token using (or not using) these vector elements as weights.
The weights do not change once the model is trained. This is why I am suggesting they could be incorporated directly into the structure of an ASIC for a specific model, rather than storing them in memory.
Edit: Of course another major factor could be that the models are just to big to be wholly implemented in a single IC by any currently existing manufacturer.