Which one should you install on your PC (Mac)? Comparing oMLX and TensorFold

I'm Hasegawa (@rarirureluis) from the Service Reliability Group (SRG) of the Media Division Headquarters.
#SRG (Service Reliability Group) is a group that primarily provides cross-cutting support for the infrastructure of our media services, improves existing services, launches new services, and contributes to OSS.
We compared oMLX and TensorFold as inference servers for handling concurrent requests to LLMs on Mac.
A third-party review introducing TensorFold 0.6.2 reported that while oMLX processed concurrent requests one by one, TensorFold was faster, returning the first token for all 8 requests within 1.6 seconds.
This time, we conducted measurements using NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit, a model supported by TensorFold, on a Mac (M4 Max, 64 GiB memory), with both servers fully configured to load the same MTP weights.
The measurement results showed that TensorFold was approximately 1.16 times faster (+16.4%) for 8 concurrent short inputs, but oMLX was also progressing the generation of all requests in parallel, so it wasn't a simple case of "oMLX is serial and TensorFold is parallel, hence faster." Additionally, the number of MTP draft acceptances during multi-request operation remained a small fraction of the total, making it difficult to explain this speed difference solely by MTP.
In this article, we organize the measurement results and examine the structure of inference generation costs, as well as implementation designs to reduce costs for both short and long inputs, by looking at the code.

Overview of Measurement Results


全件の生成トークン数 / (全件で最初の送信から最後のテキストを受け取るまでの時間)
Concurrency / Input LengthItemoMLX 0.7.0TensorFold 0.6.4
1 Short InputTime to First Token (TTFT)0.48 seconds0.30 seconds
1 Short InputCompletion Time2.56 seconds2.51 seconds
1 Short InputOverall Processing Speed100.1 tok/s101.8 tok/s
3 Short InputsTime to First Token (TTFT)0.94–1.62 seconds0.36–1.33 seconds
3 Short InputsCompletion Time for All5.64 seconds4.93 seconds
3 Short InputsOverall Processing Speed136.2 tok/s155.7 tok/s
8 Short InputsTime to First Token (TTFT)0.90–4.56 seconds0.34–3.40 seconds
8 Short InputsCompletion Time for All13.00 seconds11.16 seconds
8 Short InputsOverall Processing Speed157.6 tok/s183.4 tok/s
1 Long InputTime to First Token (TTFT)5.57 seconds4.72 seconds
1 Long InputCompletion Time7.48 seconds6.56 seconds
1 Long InputOverall Processing Speed34.2 tok/s39.0 tok/s
3 Long InputsTime to First Token (TTFT)6.23–19.96 seconds4.85–18.45 seconds
3 Long InputsCompletion Time for All23.32 seconds20.61 seconds
3 Long InputsOverall Processing Speed32.9 tok/s37.3 tok/s
*Input length: Short inputs are 270–276 tokens, long inputs are 5,337–5,340 tokens. Each request generates 256 tokens.

Timeline of Concurrent Batch Processing


For the trial where the completion time for all requests was the median on each server, we compare the timeline for each request (time to first token response and completion time), with the first submission of all requests as the origin.
OrderoMLX First TokenoMLX CompletionTensorFold First TokenTensorFold Completion
10.92 seconds11.63 seconds0.34 seconds10.39 seconds
20.93 seconds11.63 seconds0.82 seconds10.67 seconds
31.54 seconds12.11 seconds1.29 seconds10.93 seconds
42.14 seconds12.48 seconds1.75 seconds10.97 seconds
52.74 seconds12.67 seconds2.15 seconds11.01 seconds
63.33 seconds12.84 seconds3.31 seconds11.16 seconds
73.94 seconds12.94 seconds3.33 seconds11.12 seconds
84.56 seconds13.00 seconds3.40 seconds11.13 seconds
In oMLX, the first tokens for all 8 requests (up to 4.56 seconds) begin responding before the first request completes (11.63 seconds). Therefore, oMLX also performs batch processing that groups multiple requests, and generated tokens are output in parallel.
However, the time until each request joined the batch was 4.56 seconds for oMLX compared to TensorFold's maximum of 3.40 seconds (a difference of 1.16 seconds). Since TensorFold completed the join earlier, and the number of generated tokens remaining for each request at the final join point also differed, we cannot directly compare the time after the first token as pure decode speed.

Measurement Environment and Method


To ensure reproducibility of the measurements, we organized the hardware, model, and operational conditions.

Hardware and Software Configuration

The measurement date was October 5, 2026, and the verification machine was a MacBook Pro with an Apple M4 Max, 40-core GPU, 64 GiB unified memory, and macOS 26.7.
4d4f5a280bc1739ba2cf39c1cee44fd5cc89cb40
6ea5ade26c4335491c50275af0be32b81f75f525

Model Used and Architecture

d9d758fb83953437f7263256b0d96157e2a348b8
nemotron_h

Preparing MTP Weights for oMLX

mtp.
mtp.
7f6894ffa08c2a7f0cec79d7baefec80830cd50b29884020503bdce6bab2e510

Conceptual Cost per Confirmed Token


We organize the cost structure of the inference process.

Two Stages: Prefill and Decode

Request processing for an LLM is broadly divided into two stages.
The first stage, Prefill (prompt processing phase), computes the entire token sequence of the input prompt in parallel, constructs the KV cache (Attention layers) and the internal state holding past information, known as the SSM state (Mamba layers), and outputs the first token. Because it includes the computational cost of Self-Attention, the computational complexity increases non-linearly with the input length.
In the second stage, Decode (token generation phase), the process of sequentially generating the next token using the immediately preceding token as input is repeated. This stage has sequentiality dependent on the previous token, and involves reading and writing model weights (selected Experts and shared Experts in MoE, Attention, and Mamba parameters) and states (KV cache and SSM state), as well as dispatch overhead from the host CPU (launch overhead) for each step.

Cost Formula in Speculative Decoding

temperature=0
The conceptual time cost consumed per confirmed token can be organized as follows.
  • : Computation and verification time by the main model (target model)
  • : Draft generation time by the MTP head
  • : Time for updating and rolling back caches and states
  • : Dispatch and control overhead from the host CPU
  • : The total number of tokens confirmed by the main model in that step (including accepted draft tokens, correction tokens, and bonus tokens)
speculative.accepted
In batch processing that bundles multiple requests, the processing rate per computation unit during decoding is as follows.
This round rate represents the processing speed during decoding and differs from the end-to-end overall processing speed, which includes Prefill time and join wait time.
Points to note include the fact that loading MTP weights does not guarantee functionality under concurrent processing (oMLX halted with two or more concurrent requests in this Nemotron path), and that throughput will not improve if the overhead of drafting and verification () exceeds the time saved.

What was happening with short inputs


In 8-way concurrent processing with short inputs (270–276 tokens), TensorFold recorded 183.4 tok/s, approximately 1.16 times faster than oMLX (157.6 tok/s). We examine the implementation factors that caused this difference.

Results with drafting disabled

Upon checking the logs for oMLX 0.7.0, MTP was functioning in the three main measurements for single requests, with 1.77–1.80 tokens per cycle and a draft acceptance rate of 75.2–78.9% (※ aggregated only from the main measurements, distinct from the 1.75 and 1.78 recorded during warm-up trials). However, when accepting two or more concurrent requests with Nemotron, it output the following log, immediately stopped MTP, and switched to standard batch generation.
draft=false
Input length / ConcurrencyTotal completion time with drafting enabledTotal completion time with drafting disabled (no-drafts)Speedup factor from enabling drafting
1 Short Input2.51 seconds2.49 seconds0.99x (no difference)
3 Short Inputs4.93 seconds5.58 secondsApprox. 1.13x
8 Short Inputs11.16 seconds12.48 secondsApprox. 1.12x
1 Long Input6.56 seconds7.11 secondsApprox. 1.08x
3 Long Inputs20.61 seconds22.24 secondsApprox. 1.08x
As a result of the control measurement, the completion time for TensorFold with drafting disabled in 8-way concurrent processing of short inputs was 12.48 seconds. This is approximately 1.04 times faster than oMLX's 13.00 seconds.
Looking at the actual draft acceptance counts recorded in the measurement logs, across three trials of 8-way concurrent processing with short inputs (totaling 24 requests and 6,144 generated tokens), the total number of draft tokens accepted by TensorFold was 66.
The acceptance rate for the 182 presented draft tokens was 36.3% (66/182), but the accepted draft tokens accounted for only 1.07% of the total generated tokens (6,144 tokens) (tokens per round were 1.004–1.045). The acceptance count per request ranged from 0 to 10.
Even in 3-way concurrent processing with short inputs, the accepted portion accounted for 32 / 2,304 tokens (1.39%) of the total generation.
In single-request processing, TensorFold accepted 70–72 draft tokens out of 256 (totaling 213 / 768 tokens, with tokens per round at 1.384–1.399), progressing by approximately 1.4 tokens per round, whereas the acceptance count dropped significantly in 8-way concurrent processing. Note that oMLX's 1.77–1.80 tok/cycle during single processing refers to the per-cycle value for the MTP-active interval only, while TensorFold's tokens per round uses the entire request, including standard generation rounds, as the denominator; therefore, the two values cannot be directly compared.
draft=false
Given that the accepted portion accounts for only 1.07% of total generation, it is difficult to attribute the primary cause of the 16.4% difference solely to MTP. However, the overlap of main model steps and verification time associated with drafting is complex, and we have not conducted measurements to fully isolate the contribution of each factor, so the strict breakdown remains unmeasured.

Determining Draft Depth and Shared Row Allocation

Why was the draft presentation itself only 182 tokens when TensorFold processed 8 items concurrently?
DraftDepth
expected / cost
drafts = []
_allocate
hidden_rows

TensorFold: Design to Reduce Cost per Round

If the contribution of MTP is limited, what caused the speed difference?
We organize the design features identifiable from TensorFold's source code.
OUT
qmv
💡
The Apple M4 does not have the Tensor Unit introduced in the M5. Therefore, in M1–M4 environments, tensor unit-dependent processing is avoided, and row kernels that maintain row independence at the thread group level are selected.
GROUP_ROWS = 8
This is a design that creates room to reduce memory bandwidth consumption by reusing weights when the same Expert is selected for tokens from multiple streams. However, while pair reading for each row maintains the model-specific strict bit sequences, the effective reuse rate—specifically, what proportion of weights were actually reused within a single shared round (8 streams)—was not measured in this benchmark.
keep_rows_streams
The backbone's decode cache and the MTP head's cache are managed separately, maintaining a distinction between the backbone and head states.
Note that not all 52 layers run in a single kernel, nor is CPU overhead zero; it is a design intended to reduce the number of kernel calls and unnecessary re-reading of intermediate tensors.

What Happened with Long Inputs


Measurement results for long inputs reaching approximately 5,340 tokens.
  • 1 long input: oMLX 7.48 seconds vs TensorFold 6.56 seconds (approx. 1.14x)
  • 3 long inputs: oMLX 23.32 seconds (32.9 tok/s) vs TensorFold 20.61 seconds (37.3 tok/s, approx. 1.13x)
In concurrent processing of 3 items, TensorFold completed approximately 2.7 seconds faster than oMLX, but this is not because it generated the 3 requests in parallel simultaneously.

Prefill Chunk Batch Processing within a Single Request

First, there is the efficiency of Prefill within a single request for long prompts.
hidden_pass
prefill_mm.gather_sorted
prefill_raised=[True, False]
hidden_pass

The observed output intervals did not overlap

The second characteristic is the memory admission control in TensorFold and the resulting sequential completion behavior.
Checking the timeline of each request during the concurrent processing of three long inputs reveals the progress of the output.
First orderoMLX First / CompletionTensorFold First / Completion
16.41 seconds / 17.03 seconds4.57 seconds / 6.51 seconds
219.39 seconds / 22.96 seconds11.40 seconds / 13.50 seconds
319.99 seconds / 23.32 seconds18.31 seconds / 20.61 seconds
In TensorFold, after the first request completed at 6.51 seconds, the first token of the second request was returned at 11.40 seconds, and after the second request completed at 13.50 seconds, the third request began responding. In other words, the observed output intervals did not overlap.。
Why did this behavior occur? The TensorFold startup logs contained the following record.
_chunk_fits
Since the workspace for the 8-stream shared round occupies up to 7.22 GiB, and there is a risk of exceeding the budget when multiple requests holding caches of the 5,300-token scale overlap, it is designed to make them wait.
However, since individual waiting reasons are not recorded in the logs, it is not possible to identify the specific branching.
In contrast, oMLX started the Prefill of subsequent prompts in parallel while generating the first one. Since prefill and decode use the same GPU, if loading a new prompt occurs between decodes, the time allocated to generation decreases. In actual measurements, the completion of the first request was delayed from 7.48 seconds when run alone to 17.03 seconds.
Note that the sequential completion of outputs is not direct proof that Prefill calculations were completely sequential. However, it is clear from the measured timeline that the factor allowing TensorFold to complete all requests faster than oMLX for long inputs was not due to "parallel decode of 3 requests."

Verification of output consistency


temperature=0
The results comparing output consistency between single execution and concurrent processing using 8 types of 4-sentence English inputs (max_tokens=128) are as follows.
Comparison TargetoMLX 0.7.0TensorFold 0.6.4
Responses sent twice individually8/8 match8/8 match
Comparison of 3 concurrent vs. single (5 trials × 3 inputs = 15 times)8/15 match (7 mismatches)15/15 match
Comparison of 8 concurrent vs. single (3 trials × 8 inputs = 24 times)9/24 match (15 mismatches)24/24 match
Total match rate for concurrent processing17/39 match (22 mismatches)39/39 match
Since oMLX does not include token ID sequences in API responses, we compared output strings and generated token counts.
TensorFold compares the hash values of token ID sequences in addition to strings and token counts.
In oMLX, output mismatches occurred in 22 out of 39 concurrent processing runs (for example, the output changed from "宽大窗户" when run alone to "宽敞窗户" during concurrent processing). We believe that changes in batch shape due to concurrent processing, combined with the toggling of MTP enabled/disabled states in Nemotron, caused these output fluctuations.
In contrast, for all 39 requests processed concurrently by TensorFold, the strings, number of generated tokens, and token ID sequence hashes matched those from standalone execution.
draft=false

Mechanism for Output Consistency

When batching multiple requests on a GPU, thread assignments and the order of reductions (summation calculations) can change subtly. Since floating-point arithmetic is not associative, differences in addition order can introduce rounding errors, potentially flipping the selected token.
TensorFold fixes the order of reduction operations within the row kernel (row invariant). Even when rows from other requests are added to the batch, the sequence of floating-point operations and their bit representations for each row remain identical to those during standalone processing. Furthermore, inference is executed using only the row widths (verification width 16, shared width 128) that were confirmed during startup self-checks to reproduce standalone processing results bit-for-bit.

Similarities, Differences, and Which to Choose


We summarize the characteristics of both inference servers as revealed by measurement results and implementation investigations.

Similarities

Both inference servers are OpenAI-compatible inference servers specialized for the Apple Silicon (MLX) environment and are released under the Apache-2.0 license. For standalone requests with short inputs (with MTP loaded), completion times overlapped in the measurement range, with oMLX at 2.56 seconds and TensorFold at 2.51 seconds, yielding nearly equivalent results.
In the scope of this verification (8 inputs × 2 runs), both servers achieved 8/8 matches during standalone execution. Both also feature cache mechanisms (prefix cache / prompt cache) that reuse KV caches.

Summary of Differences

AspectoMLX 0.7.0TensorFold 0.6.4
Overall speed for 8 short inputs157.6 tok/s (completion 13.00 seconds)183.4 tok/s (completion 11.16 seconds, approx. 1.16x)
Completion time for 1 short input2.56 seconds2.51 seconds (measurement ranges overlap, nearly equivalent)
Output consistency between concurrent and standalone processing17 out of 39 matched (22 mismatches)39 out of 39 matched
Nemotron multi-request MTPAutomatically stops for 2 or more concurrent requestsContinues drafting even with multiple streams (draft adoption accounts for 1.07% of total generation)
Behavior with 3 long inputsPrefill interrupts generation; total 23.32 secondsObserved output intervals did not overlap; total 20.61 seconds
Breadth of supported modelsDiverse models supporting mlx-lm / mlx-vlmLimited to specific families with dedicated kernels implemented
Operation and management interfaceMenu bar resident, Web management UI, dynamic model switchingCLI, launchd service registration, TUI, CUDA support
Queue control503 error when queue limit of 32 is exceededAdmission control and waiting based on memory budget

Which Should You Choose?

TensorFold is suitable for operating models explicitly listed as optimization-supported, such as Nemotron 3.5 Lightning, when you want to maximize decode throughput in environments with concurrent requests from multiple clients, or when you prioritize output consistency with standalone execution regardless of the number of concurrent batch processes. It is well-suited for configurations where it is operated as a backend inference infrastructure via launchd services, CLI scripts, or TUI.
oMLX is suitable when you want to try a wide variety of models on Hugging Face, or when you want to easily manage model downloads, pinning, and swapping through a menu bar resident app and Web management UI. It is well-suited for configurations centered on interactive use with one request at a time, prioritizing GUI convenience.
In this measurement, where both were loaded with the same MTP weights, standalone execution with short inputs yielded nearly equivalent results, with measurement ranges overlapping.

Reproduction Information and Code References


The verification results presented in this article are based on the following repository, commits, and measurement logs.

Referenced Source Code and Implementation Locations

SoftwareTarget CodeRole / Implementation Points
TensorFold v0.6.4<br>(commit: )Calculation of memory acceptance budget () and startup estimation logs
TensorFold v0.6.4Determination of draft depth and shared row allocation (, , )
TensorFold v0.6.4, In the fused kernel for residual addition and normalization (, )
TensorFold v0.6.4 class, Mamba scan (), multi-stream execution ()
TensorFold v0.6.4MoE routing and row aggregation ()
TensorFold v0.6.44-bit row kernel for M4 ()
TensorFold v0.6.4Grouped execution of SwitchMLP (、)
TensorFold v0.6.4Prefill determination for M1–M4 (), chunk submission (), draft generation ()
TensorFold v0.6.4MoE Expert aggregation during Prefill (, ), MoE block ()
oMLX v0.7.0MTP stop determination processing in Nemotron multi-row batches
oMLX v0.7.0External Prefill and Decode Fairness scheduling logic

Conclusion


We compared the performance differences and behavioral variations between oMLX 0.7.0 and TensorFold 0.6.4 using the same MacBook Pro (M4 Max) and NVIDIA Nemotron 3.5 Lightning.
For standalone processing of short inputs, completion times were nearly identical, while TensorFold was faster for concurrent processing of multiple requests.
The focus of cost reduction in the implementations differed, and we could not explain the speed differences by a single factor such as "only draft acceptance via MTP" or "only the difference between serial and parallel execution."
Although not included in this comparison, in an MLX environment,
is also a candidate for comparison.
In Splash, the qwen35, qwen35moe, and qwen4_exp models run quite fast.
Although the contribution of each optimization has not been isolated, it is important to determine how hardware characteristics and framework implementation logic align with your specific use case.
If you are interested in SRG, please contact us here.