Performance of llama.cpp on Apple Silicon M-series #4167

ggerganov · 2023-11-22T09:46:54Z

ggerganov
Nov 22, 2023
Maintainer

Summary

LLaMA 7B

	BW [GB/s]	GPU Cores	F16 PP [t/s]	F16 TG [t/s]	Q8_0 PP [t/s]	Q8_0 TG [t/s]	Q4_0 PP [t/s]	Q4_0 TG [t/s]	commit
✅ M1 ¹	68	7			108.21	7.92	107.81	14.19	`8e672ef`
✅ M1 ¹	68	8			117.25	7.91	117.96	14.15	`8e672ef`
✅ M1 Pro ¹	200	14	262.65	12.75	235.16	21.95	232.55	35.52	`8e672ef`
✅ M1 Pro ¹	200	16	302.14	12.75	270.37	22.34	266.25	36.41	`8e672ef`
✅ M1 Max ¹	400	24	453.03	22.55	405.87	37.81	400.26	54.61	`8e672ef`
✅ M1 Max ¹	400	32	599.53	23.03	537.37	40.2	530.06	61.19	`8e672ef`
✅ M1 Ultra ¹	800	48	875.81	33.92	783.45	55.69	772.24	74.93	`8e672ef`
✅ M1 Ultra ¹	800	64	1168.89	37.01	1042.95	59.87	1030.04	83.73	`8e672ef`

✅ M2 ²	100	8			147.27	12.18	145.91	21.7	`8e672ef`
✅ M2 ²	100	10	201.34	6.72	181.4	12.21	179.57	21.91	`8e672ef`
✅ M2 Pro ²	200	16	312.65	12.47	288.46	22.7	294.24	37.87	`8e672ef`
✅ M2 Pro ²	200	19	384.38	13.06	344.5	23.01	341.19	38.86	`8e672ef`
✅ M2 Max ²	400	30	600.46	24.16	540.15	39.97	537.6	60.99	`8e672ef`
✅ M2 Max ²	400	38	755.67	24.65	677.91	41.83	671.31	65.95	`8e672ef`
✅ M2 Ultra ²	800	60	1128.59	39.86	1003.16	62.14	1013.81	88.64	`8e672ef`
✅ M2 Ultra ²	800	76	1401.85	41.02	1248.59	66.64	1238.48	94.27	`8e672ef`

🟥 M3 ³	100	8
🟨 M3 ³	100	10			187.52	12.27	186.75	21.34	`8e672ef`
🟨 M3 Pro ³	150	14			272.11	17.44	269.49	30.65	`8e672ef`
✅ M3 Pro ³	150	18	357.45	9.89	344.66	17.53	341.67	30.74	`8e672ef`
✅ M3 Max ³	300	30	589.41	19.54	566.4	34.3	567.59	56.58	`8e672ef`
✅ M3 Max ³	400	40	779.17	25.09	757.64	42.75	759.7	66.31	`8e672ef`
✅ M3 Ultra ³	800	60	1121.80	42.24	1085.76	63.55	1073.09	88.40	`8e672ef`
✅ M3 Ultra ³	800	80	1538.34	39.78	1487.51	63.93	1471.24	92.14	`8e672ef`

🟥 M4 ⁴	120	8
✅ M4 ⁴	120	10	230.18	7.43	223.64	13.54	221.29	24.11	`8e672ef`
✅ M4 Pro ⁴	273	16	381.14	17.19	367.13	30.54	364.06	49.64	`8e672ef`
✅ M4 Pro ⁴	273	20	464.48	17.18	449.62	30.69	439.78	50.74	`8e672ef`
✅ M4 Max ⁴	410	32	736.25	24.29	718.56	43.87	713.93	69.95	`8e672ef`
✅ M4 Max ⁴	546	40	922.83	31.64	891.94	54.05	885.68	83.06	`8e672ef`
🟥 M4 Ultra	820	64
🟥 M4 Ultra	1092	80

🟥 M5 ⁵	154	8
🟨 M5 ⁵	154	10			715.42	18.42	722.79	31.88	`c1d0e7a`
✅ M5 Pro ⁵	307	16	1310.78	21.39	1302.59	38.94	1340.44	67.59	`60081bb`
✅ M5 Pro ⁵	307	20	1588.78	21.55	1553.86	38.92	1620.64	66.33	`c1d0e7a`
🟥 M5 Max ⁵	460	32
✅ M5 Max ⁵	614	40	3158.49	37.11	3143.81	72.42	3219.99	119.92	`c1d0e7a`
🟥 M5 Ultra ⁵	1228	64
✅ M5 Ultra ⁵	1228	80	5207.14	73.60	4910.55	119.01	4944.66	179.10	`c1d0e7a`

🟥 M6 ⁶	154	8
✅ M6 ⁶	170	8	952.53	11.59	970.46	21.01	988.93	35.74	`c1d0e7a`

plot.py

# Plot: PP vs GPU Cores and TG vs Bandwidth for F16, Q8_0 and Q4_0 (Llama 2 7B)
# Data: https://github.com/ggml-org/llama.cpp/discussions/4167 (M1..M4, commit 8e672ef)
# Style follows the original plot.py in the discussion post.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# (Chip, BW [GB/s], GPU Cores, F16 PP, F16 TG, Q8_0 PP, Q8_0 TG, Q4_0 PP, Q4_0 TG)
rows = [
    ("M1",       68,  7,    None,   None,  108.21,   7.92,  107.81,  14.19),
    ("M1",       68,  8,    None,   None,  117.25,   7.91,  117.96,  14.15),
    ("M1 Pro",  200, 14,  262.65,  12.75,  235.16,  21.95,  232.55,  35.52),
    ("M1 Pro",  200, 16,  302.14,  12.75,  270.37,  22.34,  266.25,  36.41),
    ("M1 Max",  400, 24,  453.03,  22.55,  405.87,  37.81,  400.26,  54.61),
    ("M1 Max",  400, 32,  599.53,  23.03,  537.37,  40.20,  530.06,  61.19),
    ("M1 Ultra", 800, 48,  875.81,  33.92,  783.45,  55.69,  772.24,  74.93),
    ("M1 Ultra", 800, 64, 1168.89,  37.01, 1042.95,  59.87, 1030.04,  83.73),
    ("M2",      100,  8,    None,   None,  147.27,  12.18,  145.91,  21.70),
    ("M2",      100, 10,  201.34,   6.72,  181.40,  12.21,  179.57,  21.91),
    ("M2 Pro",  200, 16,  312.65,  12.47,  288.46,  22.70,  294.24,  37.87),
    ("M2 Pro",  200, 19,  384.38,  13.06,  344.50,  23.01,  341.19,  38.86),
    ("M2 Max",  400, 30,  600.46,  24.16,  540.15,  39.97,  537.60,  60.99),
    ("M2 Max",  400, 38,  755.67,  24.65,  677.91,  41.83,  671.31,  65.95),
    ("M2 Ultra", 800, 60, 1128.59,  39.86, 1003.16,  62.14, 1013.81,  88.64),
    ("M2 Ultra", 800, 76, 1401.85,  41.02, 1248.59,  66.64, 1238.48,  94.27),
    ("M3",      100, 10,    None,   None,  187.52,  12.27,  186.75,  21.34),
    ("M3 Pro",  150, 14,    None,   None,  272.11,  17.44,  269.49,  30.65),
    ("M3 Pro",  150, 18,  357.45,   9.89,  344.66,  17.53,  341.67,  30.74),
    ("M3 Max",  300, 30,  589.41,  19.54,  566.40,  34.30,  567.59,  56.58),
    ("M3 Max",  400, 40,  779.17,  25.09,  757.64,  42.75,  759.70,  66.31),
    ("M3 Ultra", 800, 60, 1121.80,  42.24, 1085.76,  63.55, 1073.09,  88.40),
    ("M3 Ultra", 800, 80, 1538.34,  39.78, 1487.51,  63.93, 1471.24,  92.14),
    ("M4",      120, 10,  230.18,   7.43,  223.64,  13.54,  221.29,  24.11),
    ("M4 Pro",  273, 16,  381.14,  17.19,  367.13,  30.54,  364.06,  49.64),
    ("M4 Pro",  273, 20,  464.48,  17.18,  449.62,  30.69,  439.78,  50.74),
    ("M4 Max",  410, 32,  736.25,  24.29,  718.56,  43.87,  713.93,  69.95),
    ("M4 Max",  546, 40,  922.83,  31.64,  891.94,  54.05,  885.68,  83.06),
]

cols = ["Chip", "BW (GB/s)", "GPU Cores",
        "F16 PP (t/s)", "F16 TG (t/s)",
        "Q8_0 PP (t/s)", "Q8_0 TG (t/s)",
        "Q4_0 PP (t/s)", "Q4_0 TG (t/s)"]
df = pd.DataFrame(rows, columns=cols)


def plot_multi_series(ax, x, y_series, labels, xlabel, ylabel, title, poly_power=1):
    colors = ['r', 'g', 'b']  # Colors for different series
    for i, y in enumerate(y_series):
        # Sorting data for regression
        sorted_indices = np.argsort(x)
        x_sorted = np.array(x[sorted_indices], dtype=float)
        y_sorted = np.array(y[sorted_indices], dtype=float)

        # Masking NaN values
        mask = ~np.isnan(y_sorted)
        x_sorted = x_sorted[mask]
        y_sorted = y_sorted[mask]

        # Fitting a polynomial regression model
        coefficients = np.polyfit(x_sorted, y_sorted, poly_power)
        polynomial = np.poly1d(coefficients)

        # Creating a range of x-values for a smoother trendline
        x_range = np.linspace(x_sorted.min(), x_sorted.max(), 500)
        trendline = polynomial(x_range)

        # Plotting
        ax.scatter(x, y, color=colors[i], label=labels[i], s=20)
        ax.plot(x_range, trendline, f"{colors[i]}-", linewidth=1)  # Trendline in the same color

    ax.set_xlabel(xlabel)
    ax.set_ylabel(ylabel)
    ax.set_title(title)
    ax.legend()

    # Annotating points (at the F16 series position) with the number of GPU cores and Bandwidth
    for i in range(len(df)):
        y0 = y_series[0][i]
        if np.isnan(y0):
            continue
        ax.annotate(f"{df['GPU Cores'][i]} Cores, {df['BW (GB/s)'][i]} GB/s",
                    (x[i], y0), fontsize=8)


# Creating plots for PP vs Cores and TG vs Bandwidth
fig, axs = plt.subplots(1, 2, figsize=(15, 6))
fig.suptitle('PP vs GPU Cores and TG vs Bandwidth for F16, Q8_0, and Q4_0')

# PP vs GPU Cores
y_series_cores_pp = [df["F16 PP (t/s)"], df["Q8_0 PP (t/s)"], df["Q4_0 PP (t/s)"]]
plot_multi_series(axs[0], df["GPU Cores"], y_series_cores_pp,
                  ['F16 PP', 'Q8_0 PP', 'Q4_0 PP'], 'GPU Cores', 'Performance (t/s)',
                  'PP Performance vs GPU Cores', 1)

# TG vs Bandwidth
y_series_bw_tg = [df["F16 TG (t/s)"], df["Q8_0 TG (t/s)"], df["Q4_0 TG (t/s)"]]
plot_multi_series(axs[1], df["BW (GB/s)"], y_series_bw_tg,
                  ['F16 TG', 'Q8_0 TG', 'Q4_0 TG'], 'Bandwidth (GB/s)', 'Performance (t/s)',
                  'TG Performance vs Bandwidth', 2)

plt.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.savefig("m_series_perf.png", dpi=150)
print("saved m_series_perf.png")

Description

This is a collection of short llama.cpp benchmarks on various Apple Silicon hardware. It can be useful to compare the performance that llama.cpp achieves across the M-series chips and hopefully answer questions of people wondering if they should upgrade or not. Collecting info here just for Apple Silicon for simplicity. Similar collection for A-series chips is available here: #4508

If you are a collaborator to the project and have an Apple Silicon device, please add your device, results and optionally username for the following command directly into this post (requires LLaMA 7B v2):

Note

These instructions have been updated as of 2026 Aug 25 to support the new M5+ chips

git checkout c1d0e7a00
cmake -B build
cmake --build build --target llama-bench -j

./build/bin/llama-bench \
  -hf ggml-org/Llama-2-7B-GGUF:F16  \
  -hf ggml-org/Llama-2-7B-GGUF:Q8_0 \
  -hf ggml-org/Llama-2-7B-GGUF:Q4_0 \
  -p 512 -n 128 -ngl 99 --delay 30 2> /dev/null

Make sure to run the benchmark on commit c1d0e7a
Please also include the F16 model as shown, not just the quantum models
Contributors can post the same results in the comments below
If a device is already benchmarked and your results are comparable, there is no need to add it again
PP means "prompt processing" (bs = 512), TG means "text-generation" (bs = 1), t/s means "tokens per second"
✅ means the data has been added to the summary

Note that in this benchmark we are evaluating the performance against the same build 8e672ef (2023 Nov 21) in order to keep all performance factors even. Since then, there have been multiple improvements resulting in better absolute performance. As an example, here is how the same test compares over time on M2 Ultra:

	BW [GB/s]	GPU Cores	F16 PP [t/s]	F16 TG [t/s]	Q8_0 PP [t/s]	Q8_0 TG [t/s]	Q4_0 PP [t/s]	Q4_0 TG [t/s]
2023 Nov 21
M2 Ultra `8e672ef`	800	76	1401.85	41.02	1248.59	66.64	1238.48	94.27
2024 Nov 12
M2 Ultra `86ed72d` + FA	800	76	1525.95	43.15	1368.18	73.11	1391.78	108.80
2025 Aug 02
M2 Ultra `5c0eb5e` + FA	800	76	1561.35	43.24	1386.97	73.35	1412.42	109.41
2026 Aug 25
M2 Ultra `c1d0e7a` + FA	800	76	1627.82	49.55	1486.51	82.83	1488.81	125.21

M1 Pro, 8+2 CPU, 16 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	302.14 ± 0.07
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.75 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	270.37 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.34 ± 0.00
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	266.25 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	36.41 ± 0.01

build: 8e672ef (1550)

M2 Ultra, 16+8 CPU, 76 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1401.85 ± 1.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	41.02 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1248.59 ± 0.73
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	66.64 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1238.48 ± 0.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	94.27 ± 0.05

build: 8e672ef (1550)

M3 Max (MBP 14), 12+4 CPU, 40 GPU (@slaren) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	794.26 ± 3.16
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.27 ± 0.07
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	749.37 ± 8.35
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	43.00 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	690.99 ± 33.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.85 ± 0.22

build: d103d93 (1553)

QueryType · 2023-11-22T17:17:06Z

QueryType
Nov 22, 2023

M2 Mac Mini, 4+4 CPU, 10 GPU, 24 GB Memory (@QueryType) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	201.34 ± 0.21
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	6.72 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	181.40 ± 0.05
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	12.21 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	179.57 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	21.91 ± 0.02

build: 8e672ef (1550)

0 replies

brozkrut · 2023-11-23T15:50:17Z

brozkrut
Nov 23, 2023

M2 Max Studio, 8+4 CPU, 38 GPU ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	755.67 ± 0.11
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.65 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	677.91 ± 0.26
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.83 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	671.31 ± 0.20
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.95 ± 0.08

build: 8e672ef (1550)

8 replies

maver1ck Dec 16, 2023

Wow. I wasn't aware that 4090 is so fast.

vitali-fridman Dec 26, 2023

This is from one/two generations old hardware but it's for 70B model which might be of interest.

CPU: AMD 3995WX, GPU: 2x Nvidia 3090, Ubuntu 23.10, Kernel 6.5.0-14, NV Driver: 545.23.08, CUDA: 12.3.1

model	size	params	backend	ngl	test	t/s
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	pp 512	179.29 ± 2.83
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	tg 128	21.17 ± 0.04

For comparison, 7B model on the same hardware

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1178.60 ± 88.08
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	87.34 ± 0.89

zotona Dec 27, 2023

could you try at 7b model for correct comparation? Thanks!

pukhrajvansh Feb 16, 2025

what the hell this is lower than m4 max, i mean 2x 3090 whatt..??

atlas5301 Feb 17, 2025

what the hell this is lower than m4 max, i mean 2x 3090 whatt..??

Probably because llama.cpp is not well optimized on gpus. You can expect significantly better throughput with sglang and vllm.

crasm · 2023-11-23T19:02:44Z

crasm
Nov 23, 2023

M2 Ultra, 16+8 CPU, 60 GPU (@crasm) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1128.59 ± 0.82
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	39.86 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1003.16 ± 0.39
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	62.14 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1013.81 ± 0.92
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	88.64 ± 0.06

build: 8e672ef (1550)

0 replies

ymcui · 2023-11-24T03:17:39Z

ymcui
Nov 24, 2023

M3 Max (MBP 16), 12+4 CPU, 40 GPU (@ymcui) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	779.17 ± 0.49
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.09 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	757.64 ± 1.03
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.75 ± 0.06
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	759.70 ± 2.26
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.31 ± 0.12

build: 55978ce (1555)

Short Note: mostly similar to the one reported by @slaren . But for Q4_0 pp 512, my result is 759.70 ± 2.26, while the one in the main post is 690.99 ± 33.76. Not sure about the source of the difference.

2 replies

slaren Nov 24, 2023
Maintainer

I am not sure why, but the results that I get are not very consistent. I suspect that it may due to the cooling limitations of the smaller laptop. I repeated the test now and the results are very similar to yours.

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	787.24 ± 0.84
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.15 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	755.88 ± 1.56
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.64 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	760.65 ± 0.77
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.35 ± 0.24

TeeAaTeeUu Aug 29, 2026

Almost 3 years later, at c1d0e7a00 per updated instructions, tiny bit faster:

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	12	pp512	934.98 ± 0.39
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	12	tg128	27.82 ± 0.04
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	12	pp512	873.59 ± 8.04
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	12	tg128	48.94 ± 0.06
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	12	pp512	889.62 ± 7.96
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	12	tg128	81.57 ± 0.06

Low Power Mode

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	12	pp512	678.18 ± 1.43
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	12	tg128	19.89 ± 0.15
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	12	pp512	659.98 ± 0.55
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	12	tg128	32.11 ± 0.15
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	12	pp512	691.12 ± 1.23
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	12	tg128	49.95 ± 0.45

Azirine · 2023-11-24T09:08:33Z

Azirine
Nov 24, 2023

In the graph, why is PP t/s plotted against bandwidth and TG t/s plotted against GPU cores? Seems like GPU cores have more effect on PP t/s.

0 replies

Azirine · 2023-11-24T15:08:41Z

Azirine
Nov 24, 2023

How about also sharing the largest model sizes and context lengths people can run with their amount of RAM? It's important to get the amount of RAM right when buying Apple computers because you can't upgrade later.

1 reply

ggerganov Nov 24, 2023
Maintainer Author

You can compute these. By default, you can use ~75% of the total RAM with the GPU. You can use more if you do some tricks

minosvasilias · 2023-11-24T20:36:12Z

minosvasilias
Nov 24, 2023

M2 Pro, 6+4 CPU, 16 GPU (@minosvasilias) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	312.65 ± 15.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.47 ± 0.71
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	288.46 ± 0.06
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.70 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	294.24 ± 0.10
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	37.87 ± 0.10

build: e9c13ff (1560)

0 replies

to3d · 2023-11-24T22:06:23Z

to3d
Nov 24, 2023

Would love to see how M1 Max and M1 Ultra fare given their high memory bandwidth.

0 replies

MrSparc · 2023-11-25T00:11:27Z

MrSparc
Nov 25, 2023

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 96 GB RAM (@MrSparc) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.50 ± 0.58
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.79 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	669.51 ± 1.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	64.55 ± 1.36

build: e9c13ff (1560)

2 replies

rlippmann Nov 26, 2023

I'm also using a MBP16 M2Max with the same CPU/GPU specs, but only 32 gb ram and my results are roughly the same:

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 32 GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	747.99 ± 0.28
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.54 ± 0.22
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.37 ± 0.63
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.67 ± 0.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	668.28 ± 0.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	62.98 ± 0.06

build: 22da055 (1566)

MrSparc Nov 26, 2023

Yes, it is expected that the same cpu/gpu spec will have similar performance values for same models to be compared regardless of RAM, as long as the size of the model to be used can be loaded into memory.
The amount of RAM is a limiting factor in the size of the model that can be loaded, as only 75% (by default) of the unified memory can be used as VRAM on the GPU
https://github.com/ggerganov/llama.cpp#memorydisk-requirements

CedricYauLBD · 2023-11-25T00:16:50Z

CedricYauLBD
Nov 25, 2023

M1 Max (MBP 16) 8+2 CPU, 32 GPU, 64GB RAM (@CedricYauLBD) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	599.53 ± 0.86
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	23.03 ± 0.09
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	537.37 ± 0.19
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.20 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	530.06 ± 0.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	61.19 ± 0.15

build: e9c13ff (1560)

Note: M1 Max RAM Bandwidth is 400GB/s

0 replies

philipturner · 2023-11-25T03:32:09Z

philipturner
Nov 25, 2023

Look at what I started

1 reply

yxzwayne Nov 25, 2023

off topic, but your benchmark output is my desktop rn :D

paramaggarwal · 2023-11-25T03:47:44Z

paramaggarwal
Nov 25, 2023

M3 Pro (MBP 14), 5+6 CPU, 14 GPU (@paramaggarwal) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	272.11 ± 1.40
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	17.44 ± 0.42
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	269.49 ± 1.14
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	30.65 ± 0.20

build: e9c13ff (1560)

5 replies

ggerganov Nov 25, 2023
Maintainer Author

This one has 150 GB/s memory bandwidth, correct?

paramaggarwal Nov 25, 2023

Yes, that's correct. (source)

Kaszebe May 30, 2024

Could it run a Q5 quant of llama3 70b Instruct at ~2 tokens per second?

mladencucakSYN Mar 22, 2025

I'm also interested to see if it can run a bit bigger model with some kind of reasonable outcome. Just don't want to spend MCB Max money

bagobones Mar 22, 2025

The old models give excellent comparative numbers but I wonder if the benchmark needs to be re-based around the current most popular models at some point.

Not just bigger ones for finding the biggest but popular sets / distillations that go from small to very large.

It looks like 96-128 ish gigs of shared memory will be practical on Apple / AMD / nvidia digits going forward.

brozkrut · 2023-11-25T14:50:23Z

brozkrut
Nov 25, 2023

Chip (vs. Predecessor)	F16 PP	F16 TG	Q8_0 PP	Q8_0 TG	Q4_0 PP	Q4_0 TG
M2 Pro (16) vs. M1 Pro (16)	312.65 302.14	12.47 12.75	288.46 270.37	22.7 22.34	294.24 266.25	37.87 36.41
	+3.48%	-2.20%	+6.69%	+1.61%	+10.51%	+4.01%
M2 Max (38) vs. M1 Max (32)	755.67 599.53	24.65 23.03	677.91 537.37	41.83 40.2	671.31 530.06	65.95 61.19
	+26.04%	+7.03%	+26.15%	+4.05%	+26.65%	+7.78%
M2 Ultra (60) vs. M2 Max (38)	1128.59 755.67	39.86 24.65	1003.16 677.91	62.14 41.83	1013.81 671.31	88.64 65.95
	+49.34%	+61.90%	+48.04%	+48.48%	+51.03%	+34.41%
M2 Ultra (76) vs. M2 Max (38)	1401.85 755.67	41.02 24.65	1248.59 677.91	66.64 41.83	1238.48 671.31	94.27 65.95
	+85.67%	+66.45%	+84.24%	+59.47%	+84.53%	+43.06%
M2 Ultra (76) vs. M2 Ultra (60)	1401.85 1128.59	41.02 39.86	1248.59 1003.16	66.64 62.14	1238.48 1013.81	94.27 88.64
	+24.25%	+2.91%	+24.43%	+7.23%	+22.19%	+6.33%
M3 Pro (14) vs. M2 Pro (16)			272.11 288.46	17.44 22.7	269.49 294.24	30.65 37.87
			-5.67%	-23.17%	-8.41%	-19.07%
M3 Max (40) vs. M2 Max (38)	779.17 755.67	25.09 24.65	757.64 677.91	42.75 41.83	759.7 671.31	66.31 65.95
	+3.11%	+1.78%	+11.76%	+2.20%	+13.17%	+0.55%

0 replies

pudepiedj · 2023-11-25T17:33:00Z

pudepiedj
Nov 25, 2023

### M2 MAX (MBP 16) 38 Core 32GB ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	754.39 ± 0.36
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.31 ± 0.38
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	671.33 ± 2.65
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.85 ± 0.32
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	664.07 ± 9.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	63.29 ± 0.15

build: 795cd5a (1493)

0 replies

MrSparc · 2023-11-25T21:49:00Z

MrSparc
Nov 25, 2023

I'm looking at the summary plot about "PP performance vs GPU cores" and evidence that original unquantised fp16 model always delivers more performance than quantized models.
Sorry if my question is silly, I'm new to this area, but can someone explain to me why original model delivers more performance than quantized models? Thanks

1 reply

ggerganov Nov 26, 2023
Maintainer Author

The question is not silly - the observation is expected. At large batch size (PP means batch size of 512) the computation is compute bound. I.e. the speed depends on how many FLOPS you can utilize. For quantum models, the existing kernels require extra compute to dequantize the data compared to F16 models where the data is already in F16 format.

d-shehu · 2026-03-03T21:35:47Z

d-shehu
Mar 3, 2026

I'm holding out for a benchmark of M5 Max and assuming there is a Mac Studio launching soon. I'd like to see some numbers comparing it to a Spark and Strix Halo respectively.

Anyone have $5K laying around for a new Macbook Pro ... ;)

https://www.apple.com/shop/buy-mac/macbook-pro/14-inch

2 replies

olegshulyakov Mar 9, 2026

There is Strix Halo result.

Hassan-A-K Mar 28, 2026

Done sorry for the delay also doing M5 10C soon

JJJ-M · 2026-03-08T11:20:49Z

JJJ-M Mar 8, 2026

This is a great project you created, I am contributing.
It would be great to add the quantization and MLX or GGUF, so we can make exact comparisons.

uncSoft · 2026-03-09T07:22:58Z

uncSoft Mar 9, 2026

Added! there was a bug in sparkle, please manually update to the latest release: https://github.com/uncSoft/anubis-oss/releases/latest

Hassan-A-K · 2026-03-28T04:05:13Z

Hassan-A-K
Mar 28, 2026

M5 Max (MBP 16"), 6+12 CPU, 40 GPU, 64 GB

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1018.30 ± 0.89
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	37.58 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1051.59 ± 0.67
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	64.61 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	987.10 ± 1.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	102.93 ± 0.17

build: 8e672ef (1550)
M5 Max 18C 40C 64GB 1TB ✅

11 replies

ggerganov Mar 29, 2026
Maintainer Author

Hm, interesting observation. I suppose the tensor API does a better job at interleaving memory reads with the computation.

arthurhjorth Mar 30, 2026

Ah, thanks! If this holds for all models (and I don't see why it wouldn't) then that would really help alleviate the M-chipsets' Achilles heel.

CSendner Mar 31, 2026

just to extend/confirm the initial data:

M5 Max (MBP 16"), 6+12 CPU, 40 GPU, 128 GB

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1032.18 ± 1.34
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	38.06 ± 0.04
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1054.83 ± 0.70
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	65.11 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	990.53 ± 0.44
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	104.03 ± 0.09

build: 8e672ef (1550)

Razvan-H-Andrei Apr 4, 2026

What about M5 Pro? does anyone have any data?

bnggge Apr 14, 2026

What about M5 Pro? does anyone have any data?

M5 Pro (MBP 14“), 10+5 CPU, 16 GPU, 24 GB

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	pp512	1310.28 ± 8.21
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	tg128	21.11 ± 0.18
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	pp512	1304.66 ± 1.03
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	tg128	37.78 ± 0.01
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	pp512	1350.43 ± 1.97
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	tg128	63.86 ± 0.06

build: c3a1128 (8509)

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	pp512	695.51 ± 0.74
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	tg128	21.37 ± 0.01
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	pp512	964.18 ± 0.87
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	tg128	37.70 ± 0.14
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	pp512	1135.71 ± 9.26
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	tg128	63.74 ± 0.06

build: 073bb2c (8762)

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	423.30 ± 0.14
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	20.22 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	431.14 ± 0.10
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	35.86 ± 0.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	403.19 ± 0.22
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	60.04 ± 0.21

build: 8e672ef (1550)

SpyderMYK · 2026-05-12T01:56:59Z

SpyderMYK
May 12, 2026

Apple M-series benchmark — mac-mini-m4

Hardware: Apple M4, 10 cores, 24 GB RAM, macOS 26.4
llama-bench:
Settings: -p 512 -n 128 -ngl 99 -r 3 (Metal backend, full GPU offload)

llama3.2:3b

GGUF: sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff

model	size	params	backend	threads	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	BLAS,MTL	4	pp512	530.01 ± 0.86
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	BLAS,MTL	4	tg128	45.95 ± 0.02

build: 2e97c5f (9100)

qwen2.5-coder:7b

GGUF: sha256-60e05f2100071479f596b964f89f510f057ce397ea22f2833a0cfe029bfc2463

model	size	params	backend	threads	test	t/s
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	BLAS,MTL	4	pp512	239.73 ± 2.07
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	BLAS,MTL	4	tg128	22.60 ± 0.01

build: 2e97c5f (9100)

0 replies

michaelasper · 2026-06-22T03:50:14Z

michaelasper
Jun 22, 2026

Apple M5 Max - 128gb - 16"
Total Number of Cores: 18 (6 Super and 12 Performance)
Total Number of GPU Cores: 40

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	pp512	3189.90 ± 8.62
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	tg128	35.39 ± 0.10
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	pp512	3219.94 ± 2.06
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	tg128	72.52 ± 0.11
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	6	pp512	3347.44 ± 1.53
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	6	tg128	119.08 ± 0.69

build: 52b3df0 (9754)

./bin/llama-bench \
  -m ../models/llama-7b-v2/ggml-model-f16.gguf  \
  -m ../models/llama-7b-v2/ggml-model-q8_0.gguf \
  -m ../models/llama-7b-v2/ggml-model-q4_0.gguf \
  -p 512 -n 128 -ngl 99

0 replies

bluemoehre · 2026-06-26T13:51:03Z

bluemoehre
Jun 26, 2026

Holy cow ... we're gonna get milked 😅

Apple plans a new Mac Studio by the end of the year, codenamed J775, which will feature an M5 Ultra for the first time (codename “Sotra D” or H17D). It is planned to have up to 36 CPU cores and 80 GPU cores. Apple is said to have planned to deliver the computers with up to 768 GByte of RAM [...]

Source: Heise

1 reply

bluemoehre Aug 25, 2026

There we go:

M5 Ultra
with 36-core CPU, 80-core GPU, and 32-core Neural Engine (1.2TB/s memory bandwidth) up to 512GB RAM.

M6
with 12‑core CPU (2 super cores, 4 performance cores, 6 efficiency cores), 12-core GPU, and dual 16-core Neural Engine (170GB/s memory bandwidth) up to 32GB RAM.

ggerganov · 2026-08-25T18:55:51Z

ggerganov
Aug 25, 2026
Maintainer Author

Instructions have been updated for the M5+ series. Let's collect the rest of the data points.

4 replies

ggerganov Aug 25, 2026
Maintainer Author

M5 Max, 6+12 CPU, 40 GPU, 128 GB Memory (@ggerganov) ✅

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	pp512	3158.49 ± 41.07
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	tg128	37.11 ± 0.17
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	pp512	3143.81 ± 29.60
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	tg128	72.42 ± 0.10
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	6	pp512	3219.99 ± 36.79
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	6	tg128	119.92 ± 0.17

build: c1d0e7a (10621)

marcingomulkiewicz Aug 28, 2026

M1 Ultra, 16+4 CPU, 64 GPU, 128 GB Memory

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	16	pp512	1294.31 ± 0.51
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	16	tg128	47.82 ± 0.04
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	16	pp512	1167.67 ± 2.97
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	16	tg128	79.13 ± 0.06
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	16	pp512	1179.89 ± 1.71
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	16	tg128	113.79 ± 0.06

build: c1d0e7a (10621)

ggerganov Aug 29, 2026
Maintainer Author

@marcingomulkiewicz This has already been benchmarked. The missing data points are for the M5 family.

marcingomulkiewicz Aug 30, 2026

Yes, but the results are incomparable - different version of llama.cpp, it's comparing apples to oranges. It probably makes sense to either benchmark M5 on the old version (not fun, given the new features present in M5 and not in the old llama) or re-benchmark old machines on the new llama.

brunodantas · 2026-08-30T21:37:29Z

brunodantas
Aug 30, 2026

M5 (base, 10-core GPU), MacBook Air 16 GB, macOS 26.6.1 ✅

model	size	params	backend	threads	test	t/s
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	4	pp512	715.42 ± 4.90
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	4	tg128	18.42 ± 0.06
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	4	pp512	722.79 ± 3.30
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	4	tg128	31.88 ± 0.03

build: c1d0e7a (10621)

F16 does not run on this config: model weights (13.5 GiB) exceed recommendedMaxWorkingSetSize (12.7 GB), matching the blank F16 cells on other base chips.

0 replies

hnfong · 2026-08-31T05:15:18Z

hnfong
Aug 31, 2026

Was supposed to do this a couple months ago but I forgot...

MacBook Pro, M5 Pro, 64GB ✅

Total Number of Cores: 20

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	pp512	1588.78 ± 3.16
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	tg128	21.55 ± 0.02
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	pp512	1553.86 ± 2.18
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	tg128	38.92 ± 0.06
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	6	pp512	1620.64 ± 1.16
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	6	tg128	66.33 ± 0.05

0 replies

oakleygolf · 2026-09-24T15:11:34Z

oakleygolf
Sep 24, 2026

Mac mini, M5 Pro, 64 GB RAM, macOS 27.0 ✅

15-core CPU (5 super + 10 performance), 16-core GPU
Command: llama bench -hf ggml-org/Llama-2-7B-GGUF:F16 -hf ggml-org/Llama-2-7B-GGUF:Q8_0 -hf ggml-org/Llama-2-7B-GGUF:Q4_0 -p 512 -n 128 -ngl 99 -r 5 --delay 30
Build: 60081bb2b (11046)

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	pp512	1310.78 ± 1.33
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	tg128	21.39 ± 0.53
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	pp512	1302.59 ± 2.05
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	tg128	38.94 ± 0.04
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	5	pp512	1340.44 ± 1.69
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	5	tg128	67.59 ± 0.05

0 replies

ddtsm · 2026-09-30T17:22:31Z

ddtsm
Sep 30, 2026

Mac Studio M5 Ultra 36/80 (256GB) ✅

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	12	pp512	5207.14 ± 11.61
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	12	tg128	73.60 ± 0.04
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	12	pp512	4910.55 ± 14.90
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	12	tg128	119.01 ± 0.14
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	12	pp512	4944.66 ± 17.61
llama 7B Q4_0	3.53 GiB	6.74 B	MTL,BLAS	12	tg128	179.10 ± 2.45

➜  llama.cpp git:(v0.3.0) system_profiler SPHardwareDataType SPDisplaysDataType | grep -E "Chip|Total Number of Cores|Memory|Chipset"
      Chip: Apple M5 Ultra
      Total Number of Cores: 36 (12 Super and 24 Performance)
      Memory: 256 GB
      Chipset Model: Apple M5 Ultra
      Total Number of Cores: 80
➜  llama.cpp git:(v0.3.0) sw_vers
ProductName:		macOS
ProductVersion:		27.0.1
BuildVersion:		26A434
➜  llama.cpp git:(v0.3.0) git log -1 --oneline | cat
c1d0e7a00 llama.cpp : bump version to 0.3.0 (#27696)

1 reply

ggerganov Sep 30, 2026
Maintainer Author

Thanks and congrats on the new machine 😄

jhen0409 · 2026-10-01T08:37:10Z

jhen0409
Oct 1, 2026
Collaborator

Mac mini M6, 12c GPU, 32 GB RAM (BW 170GB/s), macOS 27.0.1 ✅

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	BLAS,MTL	2	pp512	952.53 ± 1.53
llama 7B F16	12.55 GiB	6.74 B	BLAS,MTL	2	tg128	11.59 ± 0.02
llama 7B Q8_0	6.67 GiB	6.74 B	BLAS,MTL	2	pp512	970.46 ± 2.67
llama 7B Q8_0	6.67 GiB	6.74 B	BLAS,MTL	2	tg128	21.01 ± 0.05
llama 7B Q4_0	3.53 GiB	6.74 B	BLAS,MTL	2	pp512	988.93 ± 5.22
llama 7B Q4_0	3.53 GiB	6.74 B	BLAS,MTL	2	tg128	35.74 ± 0.04

build: 0c1e570

0 replies

This comment was marked as spam.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Performance of llama.cpp on Apple Silicon M-series #4167

Uh oh!

Uh oh!

ggerganov Nov 22, 2023 Maintainer

Summary

Description

M1 Pro, 8+2 CPU, 16 GPU (@ggerganov) ✅

M2 Ultra, 16+8 CPU, 76 GPU (@ggerganov) ✅

M3 Max (MBP 14), 12+4 CPU, 40 GPU (@slaren) ✅

Footnotes

Replies: 97 comments · 182 replies

Uh oh!

Uh oh!

M2 Mac Mini, 4+4 CPU, 10 GPU, 24 GB Memory (@QueryType) ✅

Uh oh!

Uh oh!

M2 Max Studio, 8+4 CPU, 38 GPU ✅

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

M2 Ultra, 16+8 CPU, 60 GPU (@crasm) ✅

Uh oh!

Uh oh!

M3 Max (MBP 16), 12+4 CPU, 40 GPU (@ymcui) ✅

Uh oh!

slaren Nov 24, 2023 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov Nov 24, 2023 Maintainer Author

Uh oh!

Uh oh!

M2 Pro, 6+4 CPU, 16 GPU (@minosvasilias) ✅

Uh oh!

Uh oh!

Uh oh!

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 96 GB RAM (@MrSparc) ✅

Uh oh!

Uh oh!

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 32 GB RAM ✅

Uh oh!

Uh oh!

Uh oh!

M1 Max (MBP 16) 8+2 CPU, 32 GPU, 64GB RAM (@CedricYauLBD) ✅

Uh oh!

Uh oh!

Uh oh!

ggerganov
Nov 22, 2023
Maintainer

Replies: 97 comments 182 replies

slaren Nov 24, 2023
Maintainer

ggerganov Nov 24, 2023
Maintainer Author