mixtral-8x7b

PyTorch

2 versions

A set of Mixture of Experts (MoE) model with open weights by Mistral AI in 8x7b and 8x22b parameter sizes.

Run this model

  1. Install our magic package manager:

    curl -ssL https://magic.modular.com/ | bash

    Then run the source command that's printed in your terminal.

  2. Install Max Pipelines in order to run this model.

    magic global install max-pipelines
  3. Start a local endpoint for mixtral/8x7b:

    max-serve serve --huggingface-repo-id mistralai/Mixtral-8x7B-Instruct-v0.1

    The endpoint is ready when you see the URI printed in your terminal:

    Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
  4. Now open another terminal to send a request using curl:

    curl -N http://0.0.0.0:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
        "model": "mixtral/8x7b",
        "stream": true,
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the World Series in 2020?"}
        ]
    }' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '
    ' | sed 's/\n/
    /g'
  5. 🎉 Hooray! You’re running Generative AI. Our goal is to make this as easy as possible.

About

The Mixtral large language models (LLMs) are generative Sparse Mixture-of-Experts (SMoE) models designed for high performance and cost efficiency. They are available in two sizes: mixtral-8x22b and mixtral-8x7b.

Mixtral 8x22B

Mixtral 8x22B features 141 billion parameters, with only 39 billion active at a time. This innovative sparsity offers significant computational efficiency while maintaining state-of-the-art performance.

Key capabilities include:

  • Multilingual fluency in English, French, Italian, German, and Spanish.
  • Strong mathematical and coding skills.
  • Native support for function calling.
  • A 64K token context window for effective retrieval of information from large documents.

References

Announcement
HuggingFace

DETAILS

MODEL CLASS
PyTorch

MODULAR GITHUB

Modular

CREATED BY

mistralai

MODEL

mistralai/Mixtral-8x7B-Instruct-v0.1

TAGS

autotrain_compatible
base_model:finetune:mistralai/Mixtral-8x7B-v0.1
base_model:mistralai/Mixtral-8x7B-v0.1
conversational
de
en
endpoints_compatible
es
fr
it
license:apache-2.0
mixtral
region:us
safetensors
text-generation
text-generation-inference
transformers

@ Copyright - Modular Inc - 2024