Phi-3.5-mini-instruct

Unknown Developer

Phi-3.5-mini-instruct is a compact 3.8 billion-parameter language model designed for efficiency. This model excels in handling long contexts of up to 128,000 tokens and demonstrates enhanced proficiency in more than 20 languages. Fine-tuned with extra training and safety protocols, it exhibits superior performance in instruction adherence, logical reasoning, mathematical problem-solving, and code creation. Released under the permissive MIT license, it is particularly well-suited for applications with limited memory or strict latency requirements.

Model Specifications

Technical details and capabilities of Phi-3.5-mini-instruct

Core Specifications

3.8B Parameters

Model size and complexity

3400.0B Training Tokens

Amount of data used in training

128.0K / 128.0K

Input / Output tokens

September 30, 2023

Knowledge cutoff date

August 22, 2024

Release date

Capabilities & License

Multimodal Support
Not Supported
Web Hydrated
No
License
MIT

Resources

Research Paper
https://arxiv.org/abs/2404.14219
API Reference
https://huggingface.co/microsoft/Phi-3.5-mini-instruct

Performance Insights

Check out how Phi-3.5-mini-instruct handles various AI tasks through comprehensive benchmark results.

90
68
45
23
0
86.2
GSM8K
86.2
(96%)
84.6
ARC Challenge
84.6
(94%)
84.1
RULER
84.1
(93%)
81
PIQA
81
(90%)
79.2
OpenBookQA
79.2
(88%)
78
BoolQ
78
(87%)
77
RepoQA
77
(86%)
74.7
Social IQA
74.7
(83%)
73.5
MEGA XStoryCloze
73.5
(82%)
69.6
MBPP
69.6
(77%)
69.4
HellaSwag
69.4
(77%)
69
BigBench Hard CoT
69
(77%)
69
MMLU
69
(77%)
68.5
WinoGrande
68.5
(76%)
64
TruthfulQA
64
(71%)
63.1
MEGA XCOPA
63.1
(70%)
62.8
HumanEval
62.8
(70%)
62.2
MEGA TyDi QA
62.2
(69%)
61.7
MEGA MLQA
61.7
(69%)
55.4
Multilingual MMLU
55.4
(62%)
48.5
MATH
48.5
(54%)
47.9
MGSM
47.9
(53%)
47.4
MMLU-Pro
47.4
(53%)
46.5
MEGA UDPOS
46.5
(52%)
41.9
Qasper
41.9
(47%)
37
Arena Hard
37
(41%)
30.9
Multilingual MMLU-Pro
30.9
(34%)
30.4
GPQA
30.4
(34%)
25.9
GovReport
25.9
(29%)
24.3
SQuALITY
24.3
(27%)
21.3
QMSum
21.3
(24%)
16
SummScreenFD
16
(18%)
GSM8K
ARC Challenge
RULER
PIQA
OpenBookQA
BoolQ
RepoQA
Social IQA
MEGA XStoryCloze
MBPP
HellaSwag
BigBench Hard CoT
MMLU
WinoGrande
TruthfulQA
MEGA XCOPA
HumanEval
MEGA TyDi QA
MEGA MLQA
Multilingual MMLU
MATH
MGSM
MMLU-Pro
MEGA UDPOS
Qasper
Arena Hard
Multilingual MMLU-Pro
GPQA
GovReport
SQuALITY
QMSum
SummScreenFD

Model Comparison

See how Phi-3.5-mini-instruct stacks up against other leading models across key performance metrics.

90
72
54
36
18
0
69
MMLU - Phi-3.5-mini-instruct
69
(77%)
69.8
MMLU - GPT-3.5 Turbo
69.8
(78%)
75.2
MMLU - Claude 3 Haiku
75.2
(84%)
81.3
MMLU - Grok-1.5
81.3
(90%)
78.9
MMLU - Phi-3.5-MoE-instruct
78.9
(88%)
79
MMLU - Claude 3 Sonnet
79
(88%)
30.4
GPQA - Phi-3.5-mini-instruct
30.4
(34%)
30.8
GPQA - GPT-3.5 Turbo
30.8
(34%)
33.3
GPQA - Claude 3 Haiku
33.3
(37%)
35.9
GPQA - Grok-1.5
35.9
(40%)
36.8
GPQA - Phi-3.5-MoE-instruct
36.8
(41%)
40.4
GPQA - Claude 3 Sonnet
40.4
(45%)
48.5
MATH - Phi-3.5-mini-instruct
48.5
(54%)
43.1
MATH - GPT-3.5 Turbo
43.1
(48%)
38.9
MATH - Claude 3 Haiku
38.9
(43%)
50.6
MATH - Grok-1.5
50.6
(56%)
59.5
MATH - Phi-3.5-MoE-instruct
59.5
(66%)
43.1
MATH - Claude 3 Sonnet
43.1
(48%)
62.8
HumanEval - Phi-3.5-mini-instruct
62.8
(70%)
68
HumanEval - GPT-3.5 Turbo
68
(76%)
75.9
HumanEval - Claude 3 Haiku
75.9
(84%)
74.1
HumanEval - Grok-1.5
74.1
(82%)
70.7
HumanEval - Phi-3.5-MoE-instruct
70.7
(79%)
73
HumanEval - Claude 3 Sonnet
73
(81%)
MMLU
GPQA
MATH
HumanEval
Phi-3.5-mini-instruct
GPT-3.5 Turbo
Claude 3 Haiku
Grok-1.5
Phi-3.5-MoE-instruct
Claude 3 Sonnet

Detailed Benchmarks

Dive deeper into Phi-3.5-mini-instruct's performance across specific task categories. Expand each section to see detailed metrics and comparisons.

Math

GSM8K

o1
97.1%
Grok-1.5
Grok-1.5
90.0%
Claude 3 Haiku
Claude 3 Haiku
88.9%
Phi-3.5-MoE-instruct
88.7%
Gemini 1.5 Flash
Gemini 1.5 Flash
86.2%
Phi-3.5-mini-instruct
86.2%
Qwen2.5-Coder 7B Instruct
Qwen2.5-Coder 7B Instruct
83.9%
Qwen2 7B Instruct
Qwen2 7B Instruct
82.3%
Llama 3.2 3B Instruct
Llama 3.2 3B Instruct
77.7%
Gemma 2 9B
Gemma 2 9B
68.6%
Current model
Other models
Avg (85.0%)

Coding

HumanEval

Claude 3.5 Sonnet
Claude 3.5 Sonnet
93.7%
Pixtral-12B
72.0%
Phi-3.5-MoE-instruct
70.7%
GPT-3.5 Turbo
68.0%
GPT-4
67.0%
Phi-3.5-mini-instruct
62.8%
Gemma 2 27B
Gemma 2 27B
51.8%
Gemma 2 9B
Gemma 2 9B
40.2%
Ministral 8B Instruct
Ministral 8B Instruct
34.8%
Current model
Other models
Avg (62.3%)

MBPP

Qwen2.5-Coder 32B Instruct
Qwen2.5-Coder 32B Instruct
90.2%
Qwen2 72B Instruct
Qwen2 72B Instruct
80.2%
Qwen2.5 7B Instruct
Qwen2.5 7B Instruct
79.2%
Codestral-22B
78.2%
Gemma 3 27B
Gemma 3 27B
74.4%
Phi-3.5-mini-instruct
69.6%
Qwen2 7B Instruct
Qwen2 7B Instruct
67.2%
Gemma 2 27B
Gemma 2 27B
62.6%
Gemma 2 9B
Gemma 2 9B
52.4%
Current model
Other models
Avg (72.7%)

Reasoning

HellaSwag

Qwen2.5 32B Instruct
Qwen2.5 32B Instruct
85.2%
Phi-3.5-MoE-instruct
83.8%
Mistral NeMo Instruct
Mistral NeMo Instruct
83.5%
Qwen2.5-Coder 32B Instruct
Qwen2.5-Coder 32B Instruct
83.0%
Gemma 2 9B
Gemma 2 9B
81.9%
Qwen2.5-Coder 7B Instruct
Qwen2.5-Coder 7B Instruct
76.8%
Llama 3.2 3B Instruct
Llama 3.2 3B Instruct
69.8%
Phi-3.5-mini-instruct
69.4%
Current model
Other models
Avg (79.2%)

Knowledge

MMLU

o1
91.8%
GPT-3.5 Turbo
69.8%
Jamba 1.5 Mini
69.7%
Llama 3.1 8B Instruct
Llama 3.1 8B Instruct
69.4%
Pixtral-12B
69.2%
Phi-3.5-mini-instruct
69.0%
Mistral NeMo Instruct
Mistral NeMo Instruct
68.0%
Qwen2.5-Coder 7B Instruct
Qwen2.5-Coder 7B Instruct
67.6%
Ministral 8B Instruct
Ministral 8B Instruct
65.0%
Llama 3.2 3B Instruct
Llama 3.2 3B Instruct
63.4%
Current model
Other models
Avg (70.3%)

GPQA

o3
87.7%
Llama 3.2 3B Instruct
Llama 3.2 3B Instruct
32.8%
Jamba 1.5 Mini
32.3%
GPT-3.5 Turbo
30.8%
Llama 3.1 8B Instruct
Llama 3.1 8B Instruct
30.4%
Phi-3.5-mini-instruct
30.4%
Gemini 1.0 Pro
Gemini 1.0 Pro
27.9%
Qwen2 7B Instruct
Qwen2 7B Instruct
25.3%
Current model
Other models
Avg (37.2%)

MATH

o3-mini
97.9%
Llama 3.2 11B Instruct
Llama 3.2 11B Instruct
51.9%
Grok-1.5
Grok-1.5
50.6%
Olmo 2 32B
Olmo 2 32B
49.7%
Qwen2 7B Instruct
Qwen2 7B Instruct
49.6%
Phi-3.5-mini-instruct
48.5%
Llama 3.2 3B Instruct
Llama 3.2 3B Instruct
48.0%
Qwen2.5-Coder 7B Instruct
Qwen2.5-Coder 7B Instruct
46.6%
Claude 3 Sonnet
Claude 3 Sonnet
43.1%
Gemini 1.0 Pro
Gemini 1.0 Pro
32.6%
Current model
Other models
Avg (51.8%)

Hallucination

TruthfulQA

Phi-3.5-MoE-instruct
77.5%
Phi-3.5-mini-instruct
64.0%
Llama 3.1 Nemotron 70B Instruct
Llama 3.1 Nemotron 70B Instruct
58.6%
Qwen2.5 14B Instruct
Qwen2.5 14B Instruct
58.4%
Jamba 1.5 Large
58.3%
Qwen2.5 32B Instruct
Qwen2.5 32B Instruct
57.8%
Qwen2 72B Instruct
Qwen2 72B Instruct
54.8%
Qwen2.5-Coder 32B Instruct
Qwen2.5-Coder 32B Instruct
54.2%
Current model
Other models
Avg (60.5%)

Non categorized

Multilingual MMLU

Phi-3.5-MoE-instruct
69.9%
Phi-3.5-mini-instruct
55.4%
Current model
Other models
Avg (62.7%)

Multilingual MMLU-Pro

Phi-3.5-MoE-instruct
45.3%
Phi-3.5-mini-instruct
30.9%
Current model
Other models
Avg (38.1%)

MGSM

Phi-4
80.6%
Claude 3 Haiku
Claude 3 Haiku
75.1%
GPT-4
74.5%
Llama 3.2 11B Instruct
Llama 3.2 11B Instruct
68.9%
Phi-3.5-MoE-instruct
58.7%
Llama 3.2 3B Instruct
Llama 3.2 3B Instruct
58.2%
GPT-3.5 Turbo
56.3%
Phi-3.5-mini-instruct
47.9%
Current model
Other models
Avg (65.0%)

MEGA MLQA

Phi-3.5-MoE-instruct
65.3%
Phi-3.5-mini-instruct
61.7%
Current model
Other models
Avg (63.5%)

MEGA TyDi QA

Phi-3.5-MoE-instruct
67.1%
Phi-3.5-mini-instruct
62.2%
Current model
Other models
Avg (64.7%)

MEGA UDPOS

Phi-3.5-MoE-instruct
60.4%
Phi-3.5-mini-instruct
46.5%
Current model
Other models
Avg (53.4%)

MEGA XCOPA

Phi-3.5-MoE-instruct
76.6%
Phi-3.5-mini-instruct
63.1%
Current model
Other models
Avg (69.8%)

MEGA XStoryCloze

Phi-3.5-MoE-instruct
82.8%
Phi-3.5-mini-instruct
73.5%
Current model
Other models
Avg (78.1%)

GovReport

Phi-3.5-MoE-instruct
26.4%
Phi-3.5-mini-instruct
25.9%
Current model
Other models
Avg (26.2%)

QMSum

Phi-3.5-mini-instruct
21.3%
Phi-3.5-MoE-instruct
19.9%
Current model
Other models
Avg (20.6%)

Qasper

Phi-3.5-mini-instruct
41.9%
Phi-3.5-MoE-instruct
40.0%
Current model
Other models
Avg (40.9%)

SQuALITY

Phi-3.5-mini-instruct
24.3%
Phi-3.5-MoE-instruct
24.1%
Nova Pro
19.8%
Nova Lite
19.2%
Nova Micro
18.8%
Current model
Other models
Avg (21.2%)

SummScreenFD

Phi-3.5-MoE-instruct
16.9%
Phi-3.5-mini-instruct
16.0%
Current model
Other models
Avg (16.4%)

Arena Hard

Mistral Small 3
Mistral Small 3
87.6%
Jamba 1.5 Large
65.4%
Jamba 1.6 Mini
Jamba 1.6 Mini
51.2%
Jamba 1.5 Mini
46.1%
Phi-3.5-MoE-instruct
37.9%
Phi-3.5-mini-instruct
37.0%
Command R+
33.1%
Llama 3.1 8B Instruct
Llama 3.1 8B Instruct
28.2%
Current model
Other models
Avg (48.3%)

BigBench Hard CoT

Phi-3.5-MoE-instruct
79.1%
Phi-3.5-mini-instruct
69.0%
Current model
Other models
Avg (74.0%)

MMLU-Pro

DeepSeek-R1
DeepSeek-R1
84.0%
Jamba 1.5 Large
53.5%
Grok-1.5
Grok-1.5
51.0%
Qwen2.5-Coder 32B Instruct
Qwen2.5-Coder 32B Instruct
50.4%
Llama 3.1 8B Instruct
Llama 3.1 8B Instruct
48.3%
Phi-3.5-mini-instruct
47.4%
Qwen2 7B Instruct
Qwen2 7B Instruct
44.1%
Jamba 1.5 Mini
42.5%
Qwen2.5-Coder 7B Instruct
Qwen2.5-Coder 7B Instruct
40.1%
Current model
Other models
Avg (51.3%)

ARC Challenge

Llama 3.1 405B Instruct
Llama 3.1 405B Instruct
96.9%
Jamba 1.5 Large
93.0%
Phi-3.5-MoE-instruct
91.0%
Claude 3 Haiku
Claude 3 Haiku
89.2%
Jamba 1.5 Mini
85.7%
Phi-3.5-mini-instruct
84.6%
Llama 3.1 8B Instruct
Llama 3.1 8B Instruct
83.4%
Ministral 8B Instruct
Ministral 8B Instruct
71.9%
Command R+
71.0%
Qwen2 72B Instruct
Qwen2 72B Instruct
68.9%
Current model
Other models
Avg (83.6%)

BoolQ

Gemma 2 27B
Gemma 2 27B
84.8%
Phi-3.5-MoE-instruct
84.6%
Gemma 2 9B
Gemma 2 9B
84.2%
Phi-3.5-mini-instruct
78.0%
Current model
Other models
Avg (82.9%)

OpenBookQA

Phi-3.5-MoE-instruct
89.6%
Phi-3.5-mini-instruct
79.2%
Mistral NeMo Instruct
Mistral NeMo Instruct
60.6%
Current model
Other models
Avg (76.5%)

PIQA

Phi-3.5-MoE-instruct
88.6%
Gemma 2 27B
Gemma 2 27B
83.2%
Gemma 2 9B
Gemma 2 9B
81.7%
Phi-3.5-mini-instruct
81.0%
Current model
Other models
Avg (83.6%)

Social IQA

Phi-3.5-MoE-instruct
78.0%
Phi-3.5-mini-instruct
74.7%
Current model
Other models
Avg (76.4%)

WinoGrande

Gemma 2 27B
Gemma 2 27B
83.7%
Phi-3.5-MoE-instruct
81.3%
Qwen2.5-Coder 32B Instruct
Qwen2.5-Coder 32B Instruct
80.8%
Qwen2.5-Coder 7B Instruct
Qwen2.5-Coder 7B Instruct
72.9%
Phi-3.5-mini-instruct
68.5%
Current model
Other models
Avg (77.4%)

RULER

Phi-3.5-MoE-instruct
87.1%
Phi-3.5-mini-instruct
84.1%
Current model
Other models
Avg (85.6%)

RepoQA

Phi-3.5-MoE-instruct
85.0%
Phi-3.5-mini-instruct
77.0%
Current model
Other models
Avg (81.0%)

Providers Pricing Coming Soon

We're working on gathering comprehensive pricing data from all major providers for Phi-3.5-mini-instruct. Compare costs across platforms to find the best pricing for your use case.

OpenAI
Anthropic
Google
Mistral AI
Cohere

Share your feedback

Hi, I'm Charlie Palars, the founder of Deepranking.ai. I'm always looking for ways to improve the site and make it more useful for you. You can write me through this form or directly through X at @palarsio.

Your feedback helps us improve our service

Stay Ahead with AI Updates

Get insights on Gemini Pro 2.5, Sonnet 3.7 and more top AI models