• 11 posts
  • 7 comments
Joined 4 months ago
Cake day: July 29th, 2026
  • https://piefed.social/comment/13343280

    PieFed has the concept of silencing another instance. A silenced instance is not defederated from but their posts do not show in the Popular or All feeds and their communities are not shown during onboarding. Their communities can still be found in the communities list (if you unblock the instance first) and joined in the normal way. Once joined, posts in there show up in the Subscribed feed of the joiner as usual.

    Since I added silencing functionality I feel like there’s almost no need for (more) defederation. I can offer protection to newbies who don’t yet know which places to avoid and they can explore the dark corners when they’re ready. It’s a low-drama way to put a bit of distance between groups that rub each other the wrong way.

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This might actually work in 8GB VRAM? 74B parameters at 24GB means about 3GB for the 9b active parameters.

Anyone tested this yet?

Enable with --moe-cache-mib N. In experiments it was found 10% of the total expert size is good.

Baseline is master with the default --fit. With --moe-cache-mib, --fit counts the cache and keeps fewer whole expert layers on the GPU.

I’m still using

commit 38976372452722cf5e6846429aa33ee892c9d89a
Author: Miltos22

To run that commit, do this:

git remote add https://github.com/miltos22/llama.cpp-wackMall-merge-request.git miltos22
git fetch miltos22
git fetch miltos22 38976372452722cf5e6846429aa33ee892c9d89a:refs/tags/miltos22-expert-caching-1
git checkout miltos22-expert-caching-1

When I have to use a low expert-hot-s value because I don’t have enough VRAM for the model, I use like:

expert-hot-s = 13
expert-hyst = 1.35
expert-heat-decay = 0.9988
expert-dwell = 2

Higher hysteresis, lower decay, and a little bit of expert-dwell reduces the PCI-e back-and-forth bandwidth usage.

There were a bunch of competing PRs for expert caching.

  1. https://github.com/ggml-org/llama.cpp/pull/17044
  2. https://github.com/ggml-org/llama.cpp/pull/21609
  3. https://github.com/ggml-org/llama.cpp/pull/21614
  4. https://github.com/ggml-org/llama.cpp/pull/23170
  5. https://github.com/ggml-org/llama.cpp/pull/24524
  6. https://github.com/ggml-org/llama.cpp/discussions/24528
  7. https://github.com/ggml-org/llama.cpp/pull/25294
  8. https://github.com/ggml-org/llama.cpp/discussions/25469
  9. https://github.com/ggml-org/llama.cpp/pull/25932
  10. https://github.com/ggml-org/llama.cpp/pull/26414
  11. https://github.com/ggml-org/llama.cpp/pull/26563
  12. https://github.com/ggml-org/llama.cpp/pull/26824
  13. https://github.com/ggml-org/llama.cpp/pull/27861
  14. https://github.com/ggml-org/llama.cpp/discussions/28248

llama.cpp v0.6.0 introduces the new llama_batch_ext extended batch API (with llama_process) for mixed token/embedding inputs and MTP/deepstack state embeddings, adds support for the GLM-5.3-Flash (GLM5-Next) 320B hybrid model, the Clef decision model (text and vision) and MTP speculative decoding for Qwen4Exp, ships a new /v1/systemone server API for decision models (laya, julia-1, lev, openjev, kev, nimble), overhauls the Web UI with a Hugging Face Hub data layer and model download pipeline, adds a Metal tensor-API flash attention kernel for F16 KV, sparse flash attention for quantized K/V on Vulkan, and updates ggml to v0.26.0.