Version:

15B-Q4_K_M CPU: Q4_K_M

MAX Model

Install our magic package manager:
```
curl -ssL https://magic.modular.com/ | bash
```
Then run the source command that's printed in your terminal.

Install Max Pipelines in order to run this model.

magic global install max-pipelines && magic global update

Start a local endpoint for phi-4-llama-distill/15B-Q4_K_M:

max-pipelines serve --huggingface-repo-id=unsloth/phi-4-GGUF \
--weight-path=kholiavko/test-ph4/unsloth.Q4_K_M.gguf

The endpoint is ready when you see the URI printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Now open another terminal to send a request using curl:

curl -N http://0.0.0.0:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
    "model": "unsloth/phi-4-GGUF",
    "stream": true,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the World Series in 2020?"}
    ]
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '
' | sed 's/\n//g'

🎉 Hooray! You’re running Generative AI. Our goal is to make this as easy as possible.

Deploy this model to cloud

Phi-4

Phi-4 is the latest scalable language model from Microsoft, meticulously developed for robust performance across natural language processing tasks. Enhanced for accuracy and versatility, Phi-4 is built on an extensive dataset that ensures small yet potent models excel in quality reasoning and diverse applications.

Model Features and Architecture

Phi-4 is a sophisticated language model equipped with a 14 billion parameter dense decoder-only Transformer architecture. Designed for text generation in chat formats, it provides a rich context length of 16K tokens for responsive outputs tailored to user inputs.

Training and Development

The model underwent a rigorous training period, sustaining 21 days of complex computations across 1920 GPUs with a staggering 9.8 trillion tokens from various datasets. Phi-4's creation focused on integrating high-quality educational content, augmented with synthetic data to bolster reasoning abilities.

Performance and Benchmarking

Phi-4 achieves superior results in multiple benchmarks such as MMLU for multitask language understanding and complex problem-solving datasets like MATH and GPQA. Its competency also extends to human-like code generation, factuality, and logical reasoning across diverse data sets.

Safety and Reliability

Safety is paramount in the design of Phi-4. Employing a robust safety post-training approach, it combines supervised fine-tuning with iterative preference optimization to adhere to high safety and ethical standards.

Responsible AI Considerations

Phi-4 aligns with responsible AI guidelines, minimizing biases and improving service quality for diverse linguistic inputs. Notably, it attempts to mitigate the risks associated with perpetuating stereotypes or inaccuracies, though developers are urged to apply additional safeguards for high-risk applications.

Limitations

The model's proficiency is primarily in English, with limited multi-language support. Developers should employ extra measures to ensure accurate, unbiased outputs.

Intended Use Cases

Phi-4 aids in advancing AI research, suitable for general-purpose applications where quick and logical text processing is essential. Its design accommodates memory-constrained and latency-sensitive environments, ready to enhance AI-powered functionalities.

Conclusion

Phi-4 represents a leap in AI-driven language processing, offering superior accuracy and safety. Aware of its limitations, developers should implement responsible AI principles and consider specific application needs when deploying this innovative language model.

Citation

Phi-4 Technical Report
Model License: MIT

Metadata

model_type	llama
version	3
tensor_count	363
kv_count	32
general.architecture	llama
general.type	model
general.name	Phi 4
general.version	4
general.organization	Unsloth
general.basename	phi
general.size_label	15B
llama.block_count	40
llama.context_length	16384
llama.embedding_length	5120
llama.feed_forward_length	17920
llama.attention.head_count	40
llama.attention.head_count_kv	10
llama.rope.freq_base	250000
llama.attention.layer_norm_rms_epsilon	0.000009999999747378752
llama.attention.key_length	128
llama.attention.value_length	128
general.file_type	15
llama.vocab_size	100352
llama.rope.dimension_count	128
general.quantization_version	2

Version: 15B CPU Q4_K_M

This code works on compatible Linux machines.
We are actively working on enabling MAX Serve for MacOS ARM64 as well.

You can quickly deploy phi-4-llama-distill-15B to an endpoint using our MAX container. It includes the latest version of MAX with GPU support and our Python-based inference server called MAX Serve.

With the following Docker command, you’ll get an OpenAI-compatible endpoint running phi-4-llama-distill-15B:

docker run --gpus 1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_HUB_ENABLE_HF_TRANSFER=1" \
    --env "HF_TOKEN=" \
    -p 8000:8000 \
    docker.modular.com/modular/max-openai-api:nightly \
    --huggingface-repo-id unsloth/phi-4-GGUF \
    --weight-path=kholiavko/test-ph4/unsloth.Q4_K_M.gguf

In order to download the model from Hugging Face, you just need to fill in the HF_TOKEN value with your access token, unless the model is from https://huggingface.co/modularai.

Learn more

For more information about the container image, see the MAX container documentation.

To learn more about how to deploy MAX to the cloud, check out our MAX Serve tutorials.

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

DETAILS

ChatMODEL CLASS

MAX Model

MAX Models are popular open-source models converted to MAX’s native graph format. Anything with the label is either SOTA or being worked on. Learn more about MAX Models.

Browse all MAX Models

HARDWARE

CPU

QUANTIZATION

Q4_K_M

ARCHITECTURE

MAX Model

MAX GITHUB

Modular / MAX

BASE MODEL

unsloth

unsloth/phi-4-GGUF

QUANTIZED BY

kholiavko

kholiavko/test-ph4

QUESTIONS ABOUT THIS MODEL?

Resources & support for
running phi-4-llama-distill-15B

Browse 27+ Tutorials

View Tutorials

Get help using MAX

Modular Forum

Read Documentation

Go to Docs

phi-4-llama-distill-15B

Phi-4

Model Features and Architecture

Training and Development

Performance and Benchmarking

Safety and Reliability

Responsible AI Considerations

Limitations

Intended Use Cases

Conclusion

Citation

Metadata

Learn more

Resources & support for running phi-4-llama-distill-15B

Resources & support for
running phi-4-llama-distill-15B