"so I ran some more math, and newer Xeon machines (Grand Rapids etc) with 1.5TB of RAM (MRDIMM - more throughput than DDR5) can deal with more concurrent users on MoE models like Kimi 2.6 and GLM 5.1 with longer context windows (>200k) though with reduced tok/sec/user for about 1/4-1/5 of the price of B200x8 or 2x(H100x8).

That means that some of my "batching" theories, assuming I can prove dedicated kernels on AMX and AVX512, means that headless coding agents that don't have a human following them, that runs in the background can be executed on said models in a MUCH cheaper way.

running a first test on Grand Rapids CPUs (only managed to get 16 CPUs - GCP won't give me quota).

You would think with the amount of models being downloaded, most clouds should benefit from a proxy for those the same way they do for linux packages..."

(http://www.autoadmit.com/thread.php?thread_id=5861511&forum_id=2Reputation#49850253)

wtf is Grand Rapids Michigan CPU

(http://www.autoadmit.com/thread.php?thread_id=5861511&forum_id=2Reputation#49850260)

Provide updated screenshots or gtfo

(http://www.autoadmit.com/thread.php?thread_id=5861511&forum_id=2Reputation#49850254)

https://i.imgur.com/F6Nwlsl.png

(http://www.autoadmit.com/thread.php?thread_id=5861511&forum_id=2Reputation#49850321)

roughly the idea is that if you have a AST parser + compiler/interpreter in the loop, you can do the following things:

- flag syntax errors (obv) + easily gaslight the model into thinking it wrote them correctly

- autocomplete a set of patterns to save on inference compute by just prefilling the completion

- rewind the cache / inference pointer (not sure what the proper name is) to semantically meaningful points (say, function starts)

- insert // docstrings that help the model along for trickier APIs

- for certain types of functions / unit tests, run the unit tests right after the code was generated, parallelizing with inference, and you can easily rewind

- on CPU, where you are memory constrained (I guess it's the same on GPUs but I haven't done GPGPU stuff since 2012), you can run speculative loops / split the code writing across multiple cores and then merge

All of these things mostly benefit from it being tailored to your own codebase / being RL-optimized to your codebase through transcript analysis, so model providers who are more reliant on batching similar jobloads and often are "too fast" for this kind of trickery won't really benefit from it.

(http://www.autoadmit.com/thread.php?thread_id=5861511&forum_id=2Reputation#49850311)