Version:

4B GPU:

PyTorch

This version is not quantized and a GPU is recommended.

Install our magic package manager:
```
curl -ssL https://magic.modular.com/ | bash
```
Then run the source command that's printed in your terminal.

Install Max Pipelines in order to run this model.

magic global install max-pipelines && magic global update

Start a local endpoint for Nemotron-Mini-Instruct/4B:

max-pipelines serve --huggingface-repo-id=nvidia/Nemotron-Mini-4B-Instruct

The endpoint is ready when you see the URI printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Now open another terminal to send a request using curl:

curl -N http://0.0.0.0:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
    "model": "nvidia/Nemotron-Mini-4B-Instruct",
    "stream": true,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the World Series in 2020?"}
    ]
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '
' | sed 's/\n//g'

🎉 Hooray! You’re running Generative AI. Our goal is to make this as easy as possible.

Deploy this model to cloud

Nemotron-Mini-4B-Instruct

Model Overview

Nemotron-Mini-4B-Instruct is built for generating responses tailored for roleplaying, retrieval augmented generation, and function calling. It's a streamlined language model (SLM) enhanced for speed and on-device deployment through distillation, pruning, and quantization. The model, a derivative of the pruned and distilled Nemotron-4 15B and fine-tuned version of Minitron-4B-Base, can handle a context length of 4,096 tokens. Released for commercial applications, it's explicitly optimized for English language tasks.

Model applications, integration details, and demonstrations for NVIDIA AI Inference Manager are available through various resources including their official blog, demos, and downloadable model checkpoints.

Model Developer: NVIDIA
Training Period: February 2024 - August 2024

Model License

The model operates under the NVIDIA Community Model License.

Model Architecture

Designed with efficiency in mind, Nemotron-Mini-4B-Instruct utilizes:

Model embedding size: 3072
Attention heads: 32
MLP intermediate dimension: 9216
Techniques like Grouped-Query Attention (GQA) and Rotary Position Embeddings (RoPE).

Architecture Type: Transformer Decoder
Network Architecture: Nemotron-4

Prompt Format

Single Turn

System
{system prompt}

User
{prompt}
Assistant\n

Tool Use

System
{system prompt}

 ... 
 ... 

User
{prompt}
Assistant
 ... 
Tool
{tool response}
Assistant\n

Usage

The model can be utilized using programming libraries for text generation and requires manual association between tokenizer objects and text generation functions. Here's a sample structure for using AutoTokenizer and text generation functions.

AI Safety Efforts

Nemotron-Mini-4B-Instruct went through rigorous AI safety evaluations, which included:

Garak: Automated vulnerability scanning for weaknesses like prompt injection and data leakage.
AEGIS: A dataset and LLM classifier model evaluating content safety across 13 critical risk categories.
Human Content Red Teaming: Direct human interactions assessing the model's response quality.

Limitations

The model draws from internet-sourced data, which includes potential biases and toxic language. This may lead to biased or harmful outputs, particularly with inappropriate prompts. The model's performance can degrade without using a recommended prompt template and may provide inaccurate or socially inappropriate responses.

Ethical Considerations

NVIDIA champions Trustworthy AI and provides strategic policies to ensure AI applications align with ethical standards. Developers using this model must ensure compliance with pertinent industry ethics and misuse prevention guidelines. Ethical concerns or security vulnerabilities can be reported directly to NVIDIA through their support channels.

Citations

Metadata

architectures.0	NemotronForCausalLM
model_type	nemotron

Version: 4B GPU undefined

This code works on compatible Linux machines.
We are actively working on enabling MAX Serve for MacOS ARM64 as well.

You can quickly deploy Nemotron-Mini-Instruct-4B to an endpoint using our MAX container. It includes the latest version of MAX with GPU support and our Python-based inference server called MAX Serve.

With the following Docker command, you’ll get an OpenAI-compatible endpoint running Nemotron-Mini-Instruct-4B:

docker run --gpus 1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_HUB_ENABLE_HF_TRANSFER=1" \
    --env "HF_TOKEN=" \
    -p 8000:8000 \
    docker.modular.com/modular/max-openai-api:nightly \
    --huggingface-repo-id nvidia/Nemotron-Mini-4B-Instruct

In order to download the model from Hugging Face, you just need to fill in the HF_TOKEN value with your access token, unless the model is from https://huggingface.co/modularai.

Learn more

For more information about the container image, see the MAX container documentation.

To learn more about how to deploy MAX to the cloud, check out our MAX Serve tutorials.

LICENSE: nvidia-community-model-license

DETAILS

ChatMODEL CLASS

PyTorch

HARDWARE

GPU

QUANTIZATION

ARCHITECTURE

PyTorch

MAX GITHUB

Modular / MAX

MODEL

nvidia

nvidia/Nemotron-Mini-4B-Instruct

QUESTIONS ABOUT THIS MODEL?

Resources & support for
running Nemotron-Mini-Instruct-4B

Browse 27+ Tutorials

View Tutorials

Get help using MAX

Modular Forum

Read Documentation

Go to Docs

Nemotron-Mini-Instruct-4B

Nemotron-Mini-4B-Instruct

Model Overview

Model License

Model Architecture

Prompt Format

Single Turn

Tool Use

Usage

AI Safety Efforts

Limitations

Ethical Considerations

Citations

Metadata

Learn more

Resources & support for running Nemotron-Mini-Instruct-4B

Resources & support for
running Nemotron-Mini-Instruct-4B