Greg Clarke

Mastodon: @greg@clar.ke

  • 34 Posts
  • 378 Comments
Joined 4 years ago
cake
Cake day: November 9th, 2022

help-circle



  • I have a RTX4090 (24GB VRAM) plus 128GB DDR4 system RAM on Ubuntu 24.04 and I am running these models locally

    Model Speed Context window
    Qwen3.8-Flash-Next UD-Q4_K_XL — 125B/6B-active hybrid MoE, 104 GB ~16.7 tok/s 128k tokens
    GLM-5.3-Flash UD-IQ3_XXS — 320B/18B-active MoE, 120 GB ~9 tok/s 128k tokens
    Qwen3.8-27B UD-Q4_K_M — dense hybrid, 16 GB, fully on the 4090 ~49 tok/s 262k tokens

    The larger models don’t fit on the GPU alone but they’re mixture-of-experts and their active weights easily fit on the 24GB. The bottleneck is my system RAM as llama.cpp has to constantly move the experts between RAM and VRAM. I’m running older hardware, AM4 CPU, DDR4 RAM, etc. so 128GB is my limit on system memory and it’s relatively slow compared to DDR5. With DDR5 I would expect a good bump in speed for Qwen 3.8 Flash Next and GLM 5.3 Flash.

    I also have a Kimi subscription which I use as my daily agentic driver and use DeepSeek for random tasks (DeepSeek Flash 4.1 is really fast and cheap).