Question 1

Can NVIDIA V100 32GB run mistral small 3.1 24b instruct 2503 hf?

Accepted Answer

Yes, NVIDIA V100 32GB can run mistral small 3.1 24b instruct 2503 hf with a C grade (Runs well). Expected decode speed: 41.2 tok/s.

Question 2

How much VRAM does mistral small 3.1 24b instruct 2503 hf need?

Accepted Answer

mistral small 3.1 24b instruct 2503 hf (24B parameters) requires approximately 22.8 GB of memory with Q4_K_M quantization.

Question 3

What is the best quantization for mistral small 3.1 24b instruct 2503 hf?

Accepted Answer

The recommended quantization for mistral small 3.1 24b instruct 2503 hf is Q4_K_M, which balances quality and memory efficiency.

Workload	Grade	Fit	Decode	TTFT	Context
Agentic Coding	C	Tight fit	41.2 tok/s	6837 ms	39K
Chat	C	Runs well	41.2 tok/s	2564 ms	12K
Coding	C	Runs well	41.2 tok/s	4700 ms	22K
RAG	C	Tight fit	41.2 tok/s	8546 ms	39K
Reasoning	C	Runs well	41.2 tok/s	5555 ms	22K

Quant	Bits	VRAM	Quality	Fit
Q2_K	2	9.4 GB	Low	D35
Q3_K_S	3	11.8 GB	Low	D37
NVFP4	4

Can NVIDIA V100 32GB run mistral small 3.1 24b instruct 2503 hf?

Memory breakdown