Version:

24B GPU: BF16

MAX Model

This version is not quantized and a GPU is recommended.

Install our magic package manager:
```
curl -ssL https://magic.modular.com/ | bash
```
Then run the source command that's printed in your terminal.

Install Max Pipelines in order to run this model.

magic global install max-pipelines && magic global update

Start a local endpoint for Dolphin3.0-Mistral/24B:

max-pipelines serve --huggingface-repo-id=cognitivecomputations/Dolphin3.0-R1-Mistral-24B

The endpoint is ready when you see the URI printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Now open another terminal to send a request using curl:

curl -N http://0.0.0.0:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
    "model": "cognitivecomputations/Dolphin3.0-R1-Mistral-24B",
    "stream": true,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the World Series in 2020?"}
    ]
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '
' | sed 's/\n//g'

🎉 Hooray! You’re running Generative AI. Our goal is to make this as easy as possible.

Deploy this model to cloud

Dolphin 3.0 Mistral 24B 🐬

Part of the Dolphin 3.0 Collection

Curated and trained by Eric Hartford, Ben Gitter, BlouseJury, and Cognitive Computations.

What is Dolphin?

Dolphin 3.0 represents the next generation of the Dolphin series of instruct-tuned models. It aims to be a versatile, local model for applications including coding, math, agentic tasks, function calling, and general use cases. Unlike other models, Dolphin offers businesses more control over prompts, model versions, alignment, and data use. It’s designed to be adaptable, putting the control in the hands of the user, making it an extension of the user's will. More about uncensored models can be found here.

Quants

Find more details at: https://huggingface.co/bartowski/cognitivecomputations_Dolphin3.0-Mistral-24B-GGUF

Chat Template

Dolphin uses ChatML for the chat template to assist interactions.

System Prompt

The system prompt in Dolphin allows for setting the tone and alignment of responses. Users can define characters, moods, or rules, ensuring that the model follows the desired behavior. Without a custom system prompt, the model defaults to standard behavior.

Sample Outputs

TBD

How to Use

Dolphin can be used through various platforms and tools including:

LM Studio
Transformers library
vllm
sglang
tgi

Evals

TBD

Appreciation

Thanks to the creators of open source datasets including:

OpenCoder-LLM (opc-sft-stage1, opc-sft-stage2)
Microsoft (orca-agentinstruct-1M-v1, orca-math-word-problems-200k)
NousResearch (hermes-function-calling-v1)
AI-MO (NuminaMath-CoT, NuminaMath-TIR)
Allenai (tulu-3-sft-mixture)
HuggingFaceTB (smoltalk)
m-a-p (CodeFeedback-Filtered-Instruction, Code-Feedback)

Special thanks to:

Meta, Qwen, and OpenCoder for their foundational contributions.
RLHFlow for their reward model used in dataset filtering.
Deepseek for the fast Deepseek-V3 used in data augmentation.

Metadata

architectures.0	MistralForCausalLM
model_type	mistral

Version: 24B GPU BF16

This code works on compatible Linux machines.
We are actively working on enabling MAX Serve for MacOS ARM64 as well.

You can quickly deploy Dolphin3.0-Mistral-24B to an endpoint using our MAX container. It includes the latest version of MAX with GPU support and our Python-based inference server called MAX Serve.

With the following Docker command, you’ll get an OpenAI-compatible endpoint running Dolphin3.0-Mistral-24B:

docker run --gpus 1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_HUB_ENABLE_HF_TRANSFER=1" \
    --env "HF_TOKEN=" \
    -p 8000:8000 \
    docker.modular.com/modular/max-openai-api:nightly \
    --huggingface-repo-id cognitivecomputations/Dolphin3.0-R1-Mistral-24B

In order to download the model from Hugging Face, you just need to fill in the HF_TOKEN value with your access token, unless the model is from https://huggingface.co/modularai.

Learn more

For more information about the container image, see the MAX container documentation.

To learn more about how to deploy MAX to the cloud, check out our MAX Serve tutorials.

Dolphin belongs to YOU, it is your tool, an extension of your will. Just as you are personally responsible for what you do with a knife, gun, fire, car, or the internet, you are the creator and originator of any content you generate with Dolphin.

DETAILS

ChatMODEL CLASS

MAX Model

MAX Models are popular open-source models converted to MAX’s native graph format. Anything with the label is either SOTA or being worked on. Learn more about MAX Models.

Browse all MAX Models

HARDWARE

GPU

QUANTIZATION

BF16

ARCHITECTURE

MAX Model

MAX GITHUB

Modular / MAX

MODEL

cognitivecomputations

cognitivecomputations/Dolphin3.0-R1-Mistral-24B

QUESTIONS ABOUT THIS MODEL?

Resources & support for
running Dolphin3.0-Mistral-24B

Browse 27+ Tutorials

View Tutorials

Get help using MAX

Modular Forum

Read Documentation

Go to Docs

Dolphin3.0-Mistral-24B

Dolphin 3.0 Mistral 24B 🐬

What is Dolphin?

Quants

Chat Template

System Prompt

Sample Outputs

How to Use

Evals

Appreciation

Metadata

Learn more

Resources & support for running Dolphin3.0-Mistral-24B

Resources & support for
running Dolphin3.0-Mistral-24B