Using the 260,000 yen mining GPU "CMP 170HX" to run LLM in production.
#SRG (Service Reliability Group) is a group that primarily provides cross-functional support for the infrastructure surrounding our media services, and is involved in improving existing services, launching new ones, and contributing to open-source software.
This article documents the process of unlocking an NVIDIA CMP 170HX mining-dedicated GPU purchased from Alibaba for 260,000 yen (including customs duties and import taxes), and then putting it into production use with Qwen3.8-27B with 64GB of VRAM.
In this context, "production operation" refers to an individual using it as a local LLM (Live Language Management System).
The CMP 170HX, an unconventional GPU with 64GB of VRAM, is available for 260,000 yen.
Mining-specific card
When running LLM on your own host, the biggest bottleneck is always the GPU's VRAM capacity.
In particular, when trying to run 20B-30B class models in parallel with a practical context length, consumer-grade 24GB cards (GeForce RTX 3090 or RTX 4090) quickly run out of VRAM.
Therefore, the focus of this article is on the "NVIDIA CMP 170HX," a dedicated accelerator manufactured for cryptocurrency mining.
The CMP 170HX features the same GA100 die (Ampere generation, Compute Capability 8.0, sm_80) as the NVIDIA A100, NVIDIA's flagship GPU for data centers.
The memory uses high-bandwidth HBM2e and has a physical capacity of 64GB.
However, since this card was sold specifically for mining, it does not have a video output terminal. In a standard driver environment, graphics functions and some calculations are limited, and depending on the distribution method, it may only show 8GB of memory.
"cmpunlocker" is the tool that bypassed this restriction.
This guide will not cover the specific usage or operation methods of cmpunlocker.
I purchased this CMP 170HX from an Alibaba vendor through personal import, and it cost me approximately 260,000 yen, including all customs duties and import consumption tax.
To achieve 64GB of VRAM with current standard products, you would need enterprise-grade cards such as the NVIDIA A100 80GB, H100, or RTX 6000 Ada, requiring a budget of several hundred thousand to several million yen, whether new or used.
There is no official support or warranty. If you understand that, then there is no other option to get a 64GB HBM2e for 260,000 yen.
From assembly to actual operation — software unlocking and physical constraints
The biggest hurdle: the bandwidth limitation of PCIe Gen2 x16.
If you're going to use this card, you need to understand the bandwidth of the connecting bus.
gen2.serviceFurthermore, this retraining only works if the slot advertises a Gen2 speed.
This CMP170HX, which I purchased from Alibaba, had already been modified at the hardware level (to x16), so the process was straightforward.
Compared to modern, common GPU environments such as PCIe Gen4 x16 (approximately 32 GB/s) and PCIe Gen5 x16 (approximately 64 GB/s), the bus bandwidth is limited to 1/4 to 1/8 of that.
This narrowness limits the number of possible inference architectures.
- Not suitable for distributed inference using multiple GPUs.When using Tensor Parallelism (TP) or Pipeline Parallelism (PP) to distribute a model across multiple GPUs, a large amount of tensor synchronization and communication occurs for each layer. With PCIe Gen2 bandwidth, bus communication immediately becomes a bottleneck, and even when multiple GPUs are combined, throughput is significantly reduced.
- Not suitable for offloading to host RAM.The approach of saving model weights overflowing from VRAM, Experts from MoE (Mixture of Experts), or KV cache to the host's main memory (RAM) and transferring them via PCIe with each inference iteration is completely broken due to PCIe Gen2 latency and transfer delays.
- The initial loading of the model is slow.Transferring approximately 18-20GB of model weights to VRAM takes about 45 seconds.
In short, the CMP 170HX is completely unsuitable for applications such as "distributing 70B-class loads across multiple cards" or "offloading some of the load weights to host memory to run large models."
Conversely,The weights, KV cache, and activations are all contained within a single 64GB VRAM module.With this design, this drawback is almost completely eliminated.
This is because all calculations performed during inference are completed between the HBM2e memory on the card and the GA100 core.
Only the input token sequence and output token of the request pass through the PCIe bus, and the narrowness of PCIe Gen2 has no impact on actual inference throughput.
Benchmark — llama.cpp vs SGLang
Comparison environment and points to note
For this verification, we adopted "Qwen3.8-27B" as an open model with practical coding and inference capabilities.
Qwen3.8-27B is a 27B class model with a hybrid architecture consisting of 48 Gated DeltaNet (linear attention) layers and 16 Full Gated Attention layers out of a total of 64 layers, and boasts a native context length of 262,144 tokens.
The inference host previously used llama.cpp, but we have migrated to SGLang to modernize the inference engine and fully support long-text contexts.
The comparison conditions before and after the migration are as follows:
| item | llama.cpp (before migration) | SGLang (after migration) |
|---|---|---|
| version | v0.4.0 | 0.5.19 |
| Model | unsloth/Qwen3.8-27B-GGUF（17.6 GB） | philbert440/Qwen3.8-27B-W4A16-AWQ （19.56 GB） |
| KV Cache | q8_0 | bf16 |
| Containment settings | ||
| Context window | Total CTX 873,472 (per slot)218,368） | 262,144 |
| Speculative Decoding | FastMTP depth 2 | none(Reasons will be explained later) |
| Main execution arguments | triton attn + mamba, pytorch sampling | |
| Power restrictions | PL 250W | PL 250W |
For a fair comparison, please keep the following points in mind.
- This is not a comparison using identical models (it's a comparison of configurations).As described below, the GGUF format cannot be read by SGLang, so we are using the AWQ quantization model in safetensors format. Both use Qwen3.8-27B, but the recipes are different. This benchmark is not a pure engine comparison, but rather a comparison of the "performance of production configurations" that can be built with each runtime.
prompt_n
- Disabling prefix cacheThis measures pure cold execution performance by adding a unique prefix to the beginning of the prompt and eliminating the influence of caching.
1. Prefill throughput and TTFT (Time to First Token)
--max-tokens 128| Prompt length (tokens) | llama.cpp prefill (t/s) | SGLang prefill (t/s) | Throughput difference | llama.cpp TTFT (ms) | SGLang TTFT (ms) | TTFT improvement rate |
|---|---|---|---|---|---|---|
| 512 | 511.4 | 989.8 | +94% | 1,463 | 478 | -67% |
| 2,048 | 764.6 | 1,649.0 | +116% | 3,457 | 1,220 | -65% |
| 8,192 | 843.6 | 1,883.8 | +123% | 10,762 | 4,328 | -60% |
| 32,768 | 821.6 | 1,528.0 | +86% | 40,649 | 21,427 | -47% |
| 131,072 | 626.3 | 785.7 | +26% | 209,987 | 166,783 | -21% |
| 245,760 | Cannot be inserted | 499.1 | — | — | 492,337 | — |
SGLang is superior in overall prompt length.
Around 8,192 tokens, a number frequently encountered in practical trading, the trading rate increased from 843.6 t/s to 1,883.8 t/s, a rise of approximately 2.2 times (+123%).
The time to generate the first token (TTFT) was reduced by approximately 20% to 67% across the board.
The waiting time for inputting 8,192 tokens, which used to be approximately 10.8 seconds, has now been reduced to approximately 4.3 seconds.
For conversational purposes, this difference directly translates to a noticeable difference in user experience.
Up until now, in the operation of llama.cpp, we had determined that "the limit of the GA100's computing power is around 730-860 t/s with cold prefill."
However, that limitation wasn't due to the GPU hardware itself, but rather to the runtime kernel implementation and batching processes.
It has been demonstrated that even with the same CMP 170HX card and the same power limit (250W), scaling up to approximately 1,884 t/s is possible by using the Triton-optimized kernel of SGLang.
2. Verification of Decode throughput
Next, let's compare the throughput during generation (decoding).
| Prompt length (tokens) | llama.cpp decode (t/s) | SGLang decode (t/s) | difference |
|---|---|---|---|
| 512 | 55.63 | 70.23 | +26% |
| 2,048 | 55.06 | 69.07 | +25% |
| 8,192 | 52.94 | 65.28 | +23% |
| 32,768 | 48.83 | 54.26 | +11% |
| 131,072 | 41.84 | 31.14 | -26% |
SGLang is superior for tokens under 32K, but llama.cpp has overtaken it for 131,072 tokens.
Up to 32K tokens, SGLang traded at 54-70 t/s, outperforming llama.cpp by 10-26%.
With 131,072 tokens, llama.cpp achieved 41.84 t/s compared to SGLang's 31.14 t/s, resulting in a change of ranking.
This reversal is due to FastMTP (speculative decoding, depth 2) being enabled on the llama.cpp side, which was generating an average of 2.3 tokens per iteration in a batch.
The reason why SGLang is running without speculative decoding will be explained in detail in the verification section below.
3. Concurrency Performance and Head-of-Line Blocking
conc| Number of parallelisms (concurrency) | llama.cpp Single Stream (t/s) | llama.cpp total generation (sum_tg) | SGLang stream only (t/s) | SGLang total generation (sum_tg) |
|---|---|---|---|---|
| 1 | 52.77 | 52.77 | 23.61 | 23.61 |
| 2 | 18.31 | 36.61 | 14.76 | 29.52 |
| 4 | 9.79 | 58.62 | 8.37 | 33.57 |
llama.cpp, with a parallelism of 2, has a total throughput lower than that of parallelism of 1. SGLang does not experience this drop.
While there is a common perception that "SGLang is robust for concurrent execution," when compared to llama.cpp with speculative decoding enabled, llama.cpp outperformed in total generation throughput (sum_tg).
As is evident from the difference in single-parallel processing (52.77 t/s vs 23.61 t/s), this difference is mainly due to the presence or absence of FastMTP.
On the other hand, llama.cpp contained a serious "Head-of-Line Blocking" issue.
-bTherefore, when a massive prefill is initiated in one slot, the iteration cycle itself is stretched to several seconds, dragging down the decoding of other slots and causing extreme delays (down to about 1 t/s in actual measurements).
The reason why the total throughput of parallelism level 2 (36.61 t/s) is lower than that of parallelism level 1 (52.77 t/s) is due to this interference.
On the other hand, while the total throughput on the SGLang side was lower, it increased monotonically with respect to the degree of parallelism, and this discontinuous drop was not observed.
It should be noted that this comparison includes the significant difference of whether or not speculative decoding is performed.
The conditions are not yet met to conclude that "SGLang's scheduler is superior."
4. Release the effective context window.
The most important and decisive motivation in this inference engine comparison was the "physical release of the context window."
ctx_sizen_ctx_slot = ctx_size / --parallelctx_size 873472This reduced the window per slot to a maximum of 218,368 tokens (effective input limit after deducting safety margin was 200,704 tokens).
Increasing the parallelism to 5 will cause the VRAM to exceed 64GB due to the increased compute buffer per slot (approximately 900 MiB) and KV cache, resulting in a boot-out-of-memory error (OOM).
In other words, in llama.cpp, there was a perfect zero-sum trade-off relationship between "capacity (degree of parallelism)" and "context window length".
--max-running-requestsThe KV cache area (Radix Cache) is dynamically shared across all slots, eliminating the need to divide the window by the number of parallel processes.
The llama.cpp file will experience a window break as the degree of parallelism increases. SGLang, however, can receive 245,724 tokens in a single request while maintaining 4 parallelisms.
In actual device testing245,724 tokensThe prompt was handled in a single request, and the process was completed with a prefill of 499.1 t/s.
This window directly relates to practical operational challenges.
An analysis of LiteLLM's actual traffic (27,343 requests in total) revealed thatLong requests exceeding 64,000 tokens account for 23.3% of the total.It occupied a significant portion of the market.
--parallel 1After migrating to SGLang, the CMP 170HX can maintain 4 parallel connections while receiving this 23.3% without restriction.
5. Verification of MTP speculative decoding and reasons for rejection
--speculative-algorithm NEXTN--mem-fraction-static 0.80| setting | 512 prefill (t/s) | difference | 8,192 prefill (t/s) | difference | 512 decode (t/s) | difference | 8,192 decode (t/s) | difference | 4 parallel total generation (sum_tg) |
|---|---|---|---|---|---|---|---|---|---|
| No speculation | 1,049.9 | — | 1,893.0 | — | 69.6 | — | 64.9 | — | 33.0 |
| s1 k1 d2 | 874.3 | -16.7% | 1,672.1 | -11.7% | 108.8 | +56% | 52.3 | -19% | 28.2 |
| s2 k1 d3 | 812.8 | -22.6% | 1,646.6 | -13.0% | 159.2 | +129% | 72.4 | +12% | 28.1 |
| s3 k1 d4 | 752.6 | -28.3% | 1,640.4 | -13.3% | 209.4 | +201% | 94.7 | +46% | 33.2 |
| s5 k1 d6 | 378.5 | -63.9% | 1,532.6 | -19.0% | 254.9 | +266% | 110.9 | +71% | 29.5 |
| s3 k2 d4 | 752.6 | -28.3% | 1,656.8 | -12.5% | 167.7 | +141% | 94.0 | +45% | 32.7 |
Enabling MTP increases the decoding speed of short sentences by up to approximately three times (209.4 t/s), but conversely, the prefill speed decreases depending on the number of paragraphs.
For short texts, the decoding speed for s3 k1 d4 (3 steps, topk 1, draft 4) is 69.6 t/s.209.4 t/s（+201%）It increased to approximately three times that amount.
Breaking down the prefill performance degradation, the culprit wasn't a decrease in computational throughput. It was the increase in the fixed overhead of draft invocation from 214 ms to 363 ms.
The pure prefill rate plateaus at 1,992 t/s to 1,770 t/s (-11%) and does not worsen even when the number of draft stages is increased.
Since the fixed cost is charged with every request regardless of the prompt length, the ratio appears worse for shorter messages.
This -11% is acceptable as a trade where the decode rate triples.
The problem was a long passage.
| Input length (tokens) | llama.cpp decode (t/s) | SGLang no speculation decode (t/s) | SGLang s3 decode (t/s) | SGLang s3 + ReplaySSM decode (t/s) |
|---|---|---|---|---|
| 512 | 55.6 | 69.6 | 209.4 | 199.1 |
| 8,192 | 52.9 | 64.9 | 94.7 | 93.6 |
| 32,768 | 48.8 | 54.3 | 38.9 | 32.6 |
| 131,072 | 41.8 | 31.1 | 10.1 | 11.4 |
MTP (s3) falls below the level of no speculation above 32,000 tokens, and at 131,072 tokens, it drops from 31.1 t/s to 10.1 t/s, a decrease of approximately one-third.
What's strange here is that the accept len (length of accepted draft tokens) remained at 3.0-3.4 even for longer tokens (acceptance rate of 73-80%).
The cause lies in the architecture of Qwen3.8-27B.
This model uses Gated DeltaNet (linear attention) for 48 out of 64 layers.
SGLang's speculative validation logic takes a snapshot of the recurrent state for each draft.
Since the amount of state is proportional to the context length, longer texts require more copying and calculation for verification.
--enable-linear-replayssm-specIn a production environment where 23.3% of traffic exceeds 64K, a reduction of 1/3 in long text decoding is unacceptable.
The current operation isIt will operate without speculation, without employing MTP speculative decoding." is what it says.
If you were to set up a separate host specifically for short texts, S3 would be a strong option, but it's not suitable for this machine, which is primarily used for long texts.
Stepped into a trap
Before we could get it running smoothly, we encountered numerous pitfalls that weren't mentioned anywhere.
I'm recording this for anyone who might be building the same setup.
1. GGUF models cannot be read by SGLang.
Initially, the plan was to simply load existing GGUF files into SGLang and compare them.
qwen35RuntimeErrorFurthermore, SGLang lacks the GGUF-derived mmproj (multimodal projector) loader, resulting in the loss of vision functionality.
Therefore, we abandoned the direct use of GGUF and switched to a safetensors-based AWQ quantization model.
2. MTP weights are silently discarded in SGLang 0.5.19.
SGLang 0.5.19 has a bug that causes it to skip MTP heads in compressed-tensors format due to a module mapping error, resulting in it running with random number initialization (upstream issue #35797).
Load weight endaccept lenI patched the Ansible deployment task to enable weighting.
3. The pitfalls of loading draft models and VRAM allocation
--mem-fraction-static 0.9Because the draft model is launched as a ModelRunner instance separate from the main model, it consumes not only weights (approximately 0.85GB) but also separate compute buffers and CUDA Graph space.
During speculative testing, we lowered this ratio to 0.80 to address the issue.
Even after cutting speculation, the same issue occurred in the actual production environment. In version 0.9, only about 2GB of free VRAM remained after CUDA Graph capture, resulting in 118 Out-of-Memory (OOM) errors in 24 hours due to failure to secure activation under real traffic.
0.85raise_exception
raise_exceptionminimalIf you use the raw template, the backend will return a 500 Internal Server Error to the client's request.
medium5. Hopper-based kernel conflicts with sm_80
CMP 170HX is GA100 (sm_80).
The default Attention backend and Mamba backends (such as flashinfer and fa3) in the latest release of SGLang may encounter kernel implementations that assume the Hopper generation (sm_90+), causing crashes during initialization in sm_80 environments.
--mamba-backend triton6. Incompatibility between CUDA 12.0 and FlashInfer JIT
SGLang's default sampling backend (flashinfer) compiles the sampling kernel in a Just-In-Time (JIT) manner when the process starts.
--compress-mode=sizeIf the host environment's CUDA Toolkit is version 12.0, the nvcc compiler outputs an unknown option error, causing the build to fail and the process to die immediately after the model loads successfully.
This crash is a trap because it occurs after the health check endpoint briefly returns a 200 status, making it easy to mistakenly believe that the system has started up normally.
--sampling-backend pytorch-tool-call-parser
The tool invocation format output by the Qwen3.8-27B template is an XML-like notation, as shown below.
JSONDecodeErrortool_calls--tool-call-parser qwen3_coderOptions for 260,000 yen
The fact that 64GB of memory made this possible in reality
There is only one achievement this time.We were able to run a 27B-class model with just one 64GB VRAM card, while simultaneously handling long text contexts and 4 parallel processes.
Even with the model weight (approximately 19.6GB for W4A16), there's still over 44GB remaining.
This entire free space can be allocated to the KV cache, CUDA Graph, and dynamic activation, so the context window won't break even if the number of parallel processes is increased. 245,724 token requests will be accepted without any issues.
Power is limited to 250W. Increasing from 170W to 250W only increases throughput by +9.2%, so if you are in an environment with limited heat dissipation or power supply capacity, you can consider limiting it to 170W-200W.
For those who are undecided about buying it
Based on what we've discussed so far, let's summarize who is suited and who is not.
Should buy:
- For those who want to run 20B-30B class LLMs on a single GPU with long contexts (tens of thousands to hundreds of thousands of tokens) and multiple parallel processes.
- Someone who can handle kernel module patch builds, Proxmox/LXC device node control, and driver version pinning themselves.
- People who want to get a 64GB HBM2e with a budget of around 200,000 yen
You shouldn't buy it:
- For those who want to perform distributed inference of over 70B using Tensor Parallelism by bundling multiple graphics cards (PCIe Gen2 x16 will be the bottleneck).
- This is for those who want to offload weights that don't fit in VRAM or KV cache to host RAM (which will also cause problems with PCIe bandwidth).
- People who want to install via a package manager and rely on manufacturer support.
In conclusion
A card originally designed for mining and nearly lost its purpose has been transformed into a production inference node with the help of cmpunlocker and a carefully designed infrastructure.
The key was designing it to avoid the weaknesses of PCIe Gen2. In a configuration where everything is on a single card, this weakness doesn't affect throughput.
I hope this will be helpful for those who are running their own local LLM (Language Learning Module).
If you are interested in SRG, please contact us here.
Recruitment information - CyberAgent SRG #ca_srg