Question 1

Can AMD Instinct MI100 32GB run mistral small 3.1 24b instruct 2503 hf?

Accepted Answer

Yes, AMD Instinct MI100 32GB can run mistral small 3.1 24b instruct 2503 hf with a B grade (Runs well). Expected decode speed: 54.5 tok/s.

Question 2

How much VRAM does mistral small 3.1 24b instruct 2503 hf need?

Accepted Answer

mistral small 3.1 24b instruct 2503 hf (24B parameters) requires approximately 22.8 GB of memory with Q4_K_M quantization.

Question 3

What is the best quantization for mistral small 3.1 24b instruct 2503 hf?

Accepted Answer

The recommended quantization for mistral small 3.1 24b instruct 2503 hf is Q4_K_M, which balances quality and memory efficiency.

Workload	Grade	Fit	Decode	TTFT	Context
Agentic Coding	C	Tight fit	54.5 tok/s	5165 ms	39K
Chat	C	Runs well	54.5 tok/s	1937 ms	12K
Coding	B	Runs well	54.5 tok/s	3551 ms	22K
RAG	C	Tight fit	54.5 tok/s	6456 ms	39K
Reasoning	B	Runs well	54.5 tok/s	4196 ms	22K

Quant	Bits	VRAM	Quality	Fit
Q2_K	2	9.4 GB	Low	D35
Q3_K_S	3	11.8 GB	Low	D37
NVFP4	4

Can AMD Instinct MI100 32GB run mistral small 3.1 24b instruct 2503 hf?

Memory breakdown