models / EXAONE-3.5-Instruct-2.4B-Q4_K_M

Deploy this at scale

MAX can save up to 80% on your compute costs in the cloud. Get in touch with us to discuss running this model for your company.

Choose Version

(9 versions)

MAX Model

Q4_K_M

CPU

CPU platforms supported: Mac, Linux, and WSL for Windows - More Details
RAM required for this model: 1.95GB

Install our magic package manager:

curl -ssL https://magic.modular.com/fafa18b5-4293-4500-b5ce-16545f8c88d6 | bash

Then run the source command that's printed in your terminal.

Install max-pipelines in order to run this model.

magic global install max-pipelines && magic global update

Start a local endpoint for EXAONE-3.5-Instruct/2.4B-Q4_K_M:

max-pipelines serve --model-path=LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct \
  --weight-path=bartowski/EXAONE-3.5-2.4B-Instruct-GGUF/EXAONE-3.5-2.4B-Instruct-Q4_K_M.gguf \
  --trust-remote-code

The endpoint is ready when you see the URI printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Now open another terminal to send a request using curl:

curl -N http://0.0.0.0:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
    "model": "LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the World Series in 2020?"}
    ]
}'

🎉 Hooray! You’re running Generative AI. Our goal is to make this as easy as possible.

Deploy this model to cloud

EXAONE-3.5-2.4B-Instruct

Introduction

EXAONE 3.5 by LG AI Research is a suite of instruction-tuned bilingual (English and Korean) generative models. Ranging from 2.4 billion to 32 billion parameters, they excel in performance across real-world applications and long-context understanding. The suite includes a 2.4B model for small devices, a 7.8B version with enhanced performance, and a robust 32B model. The models accommodate long context processing with up to 32K tokens, showcasing state-of-the-art capabilities for innovative solutions.

For detailed information, see our technical report, blog, and GitHub.

This repository highlights the 2.4B model featuring:

Parameters: 2.14B (excluding embeddings)
Layers: 30
Attention Heads: GQA with 32 Q-heads, 8 KV-heads
Vocabulary Size: 102,400
Context Length: 32,768 tokens
Word Embeddings: Constructed with tying enabled (distinct from larger models)

Evaluation

Below is a table displaying evaluation results for EXAONE 3.5’s real-world performance with various benchmarks:

Models	MT-Bench	LiveBench	Arena-Hard	AlpacaEval	IFEval	KoMT-Bench[1]	LogicKor
EXAONE 3.5 2.4B	7.81	33.0	48.2	37.1	73.6	7.24	8.51
Qwen 2.5 3B	7.21	25.7	26.4	17.4	60.8	5.68	5.21
Qwen 2.5 1.5B	5.72	19.2	10.6	8.4	40.7	3.87	3.60
Llama 3.2 3B	6.94	24.0	14.2	18.7	70.1	3.16	2.86
Gemma 2 2B	7.20	20.0	19.1	29.1	50.5	4.83	5.29

[1] KoMT-Bench is a Korean translation of MT-Bench; see README for details.

Deployment

EXAONE 3.5 models can be integrated into various frameworks including:

TensorRT-LLM
vLLM
SGLang
llama.cpp
Ollama

For further details, visit the EXAONE 3.5 GitHub.

Quantization

Pre-quantized models are available in AWQ and GGUF formats for effective deployment. Access the quantized models in the EXAONE 3.5 collection.

Limitations

While the EXAONE language models strive to deliver accurate and contextually relevant outputs, they have limitations and may occasionally produce inappropriate or biased content. The limitations stem from training data and model design, affecting output reliability and relevance. LG AI Research is committed to refining model performance to mitigate risks related to inappropriate outputs.

License

The model follows the EXAONE AI Model License Agreement 1.1 - NC.

Citation

@article{exaone-3.5,
  title={EXAONE 3.5: Series of Large Language Models for Real-world Use Cases},
  author={LG AI Research},
  journal={arXiv preprint arXiv:https://arxiv.org/abs/2412.04862},
  year={2024}
}

Contact

For technical support, reach out to LG AI Research at contact_us@lgresearch.ai.

Metadata

version	3
tensor_count	274
kv_count	36
general.architecture	exaone
general.type	model
general.name	EXAONE 3.5 2.4B Instruct
general.finetune	Instruct
general.basename	EXAONE-3.5
general.size_label	2.4B
general.license	other
general.license.name	exaone
general.license.link	LICENSE
general.tags.0	lg-ai
general.tags.1	exaone
general.tags.2	exaone-3.5
general.tags.3	text-generation
general.languages.0	en
general.languages.1	ko
exaone.embedding_length	2560
exaone.attention.head_count	32
exaone.attention.head_count_kv	8
exaone.context_length	32768
exaone.attention.layer_norm_rms_epsilon	0.000009999999747378752
exaone.feed_forward_length	7168
exaone.block_count	30
general.file_type	15
exaone.rope.freq_base	1000000
exaone.rope.dimension_count	80
general.quantization_version	2
quantize.imatrix.file	/models_out/EXAONE-3.5-2.4B-Instruct-GGUF/EXAONE-3.5-2.4B-Instruct.imatrix
quantize.imatrix.dataset	/training_dir/calibration_datav3.txt
quantize.imatrix.entries_count	210
quantize.imatrix.chunks_count	137

Version: 2.4B CPU Q4_K_M

This code works on compatible Linux machines.
We are actively working on enabling MAX Serve for MacOS ARM64 as well.

You can quickly deploy EXAONE-3.5-Instruct-2.4B to an endpoint using our MAX container. It includes the latest version of MAX with GPU support and our Python-based inference server called MAX Serve.

With the following Docker command, you’ll get an OpenAI-compatible endpoint running EXAONE-3.5-Instruct-2.4B:

docker run --gpus 1 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_HUB_ENABLE_HF_TRANSFER=1" \
    --env "HF_TOKEN=" \
    -p 8000:8000 \
    docker.modular.com/modular/max-nvidia-base:nightly \
    --model-path LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct \
    --weight-path=bartowski/EXAONE-3.5-2.4B-Instruct-GGUF/EXAONE-3.5-2.4B-Instruct-Q4_K_M.gguf

In order to download the model from Hugging Face, you just need to fill in the HF_TOKEN value with your access token, unless the model is from https://huggingface.co/modularai.

Learn more

For more information about the container image, see the MAX container documentation.

To learn more about how to deploy MAX to the cloud, check out our MAX pipelines tutorials.

EXAONE AI Model License Agreement 1.1 - NC

This License Agreement (“Agreement”) is entered into between you (“Licensee”) and LG Management Development Institute Co., Ltd. (“Licensor”), governing the use of the EXAONE AI Model (“Model”). By downloading, installing, copying, or using the Model, you agree to comply with and be bound by the terms of this Agreement. If you do not agree to all the terms, you must not download, install, copy, or use the Model. This Agreement constitutes a binding legal agreement between the Licensee and Licensor.

1. Definitions 1.1 Model: The artificial intelligence model provided by Licensor, which includes any software, algorithms, machine learning models, or related components supplied by Licensor. This definition extends to encompass all updates, enhancements, improvements, bug fixes, patches, or other modifications that may be provided by Licensor from time to time, whether automatically or manually implemented. 1.2 Derivatives: Any modifications, alterations, enhancements, improvements, adaptations, or derivative works of the Model created by Licensee or any third party. This includes changes made to the Model's architecture, parameters, data processing methods, or any other aspect of the Model that results in a modification of its functionality or output. 1.3 Output: Any data, results, content, predictions, analyses, insights, or other materials generated by the Model or Derivatives, regardless of whether they are in their original form or have been further processed or modified by the Licensee. This includes, but is not limited to, textual or numerical produced directly or indirectly through the use of the Model. 1.4 Licensor: LG Management Development Institute Co., Ltd., the owner, developer, and provider of the EXAONE AI Model. The Licensor holds all rights, title, and interest in the Model and is responsible for granting licenses to use the Model under the terms specified in this Agreement. 1.5 Licensee: The individual, organization, corporation, academic institution, government agency, or other entity using or intending to use the Model under the terms and conditions of this Agreement. The Licensee is responsible for ensuring compliance with the Agreement by all authorized users who access or utilize the Model on behalf of the Licensee.

2. License Grant 2.1 Grant of License: Subject to the terms and conditions outlined in this Agreement, the Licensor hereby grants the Licensee a limited, non-exclusive, non-transferable, worldwide, and revocable license to: a. Access, download, install, and use the Model solely for research purposes. This includes evaluation, testing, academic research, experimentation, and participation in competitions, provided that such participation is in a non-commercial context. Notwithstanding Section 3.1, the Licensee may only provide the Model or Derivatives for a competition if no commercial license is granted to the competition organizer or any third party. b. Publicly disclose research results and findings derived from the use of the Model or Derivatives, including publishing papers or presentations. c. Modify the Model and create Derivatives based on the Model, provided that such modifications and Derivatives are used exclusively for research purposes. The Licensee may conduct experiments, perform analyses, and apply custom modifications to the Model to explore its capabilities and performance under various scenarios. If the Model is modified, the modified Model must include “EXAONE” at the beginning of its name. d. Distribute the Model and Derivatives in each case with a copy of this Agreement. 2.2 Scope of License: The license granted herein does not authorize the Licensee to use the Model for any purpose not explicitly permitted under this Agreement. Any use beyond the scope of this license, including any commercial application or external distribution, is strictly prohibited unless explicitly agreed upon in writing by the Licensor.

3. Restrictions 3.1 Commercial Use: The Licensee is expressly prohibited from using the Model, Derivatives, or Output for any commercial purposes, including but not limited to, developing or deploying products, services, or applications that generate revenue, whether directly or indirectly. Any commercial exploitation of the Model or its derivatives requires a separate commercial license agreement with the Licensor. Furthermore, the Licensee shall not use the Model, Derivatives or Output to develop or improve other models. 3.2 Reverse Engineering: The Licensee shall not decompile, disassemble, reverse engineer, or attempt to derive the source code, underlying ideas, algorithms, or structure of the Model, except to the extent that such activities are expressly permitted by applicable law. Any attempt to bypass or circumvent technological protection measures applied to the Model is strictly prohibited. 3.3 Unlawful Use: The Licensee shall not use the Model and Derivatives for any illegal, fraudulent, or unauthorized activities, nor for any purpose that violates applicable laws or regulations. This includes but is not limited to the creation, distribution, or dissemination of malicious, deceptive, or unlawful content. 3.4 Ethical Use: The Licensee shall ensure that the Model or Derivatives is used in an ethical and responsible manner, adhering to the following guidelines: a. The Model and Derivatives shall not be used to generate, propagate, or amplify false, misleading, or harmful information, including fake news, misinformation, or disinformation. b. The Model and Derivatives shall not be employed to create, distribute, or promote content that is discriminatory, harassing, defamatory, abusive, or otherwise offensive to individuals or groups based on race, gender, sexual orientation, religion, nationality, or other protected characteristics. c. The Model and Derivatives shall not infringe on the rights of others, including intellectual property rights, privacy rights, or any other rights recognized by law. The Licensee shall obtain all necessary permissions and consents before using the Model and Derivatives in a manner that may impact the rights of third parties. d. The Model and Derivatives shall not be used in a way that causes harm, whether physical, mental, emotional, or financial, to individuals, organizations, or communities. The Licensee shall take all reasonable measures to prevent misuse or abuse of the Model and Derivatives that could result in harm or injury.

4. Ownership 4.1 Intellectual Property: All rights, title, and interest in and to the Model, including any modifications, Derivatives, and associated documentation, are and shall remain the exclusive property of the Licensor. The Licensee acknowledges that this Agreement does not transfer any ownership rights to the Licensee. All trademarks, service marks, and logos associated with the Model are the property of the Licensor. 4.2 Output: All rights, title, and interest in and to the Output generated by the Model and Derivatives whether in its original form or modified, are and shall remain the exclusive property of the Licensor. Licensee may use, modify, and distribute the Output and its derivatives for research purpose. The Licensee shall not claim ownership of the Output except as expressly provided in this Agreement. The Licensee may use the Output solely for the purposes permitted under this Agreement and shall not exploit the Output for unauthorized or commercial purposes. 4.3 Attribution: In any publication or presentation of results obtained using the Model, the Licensee shall provide appropriate attribution to the Licensor, citing the Model's name and version, along with any relevant documentation or references specified by the Licensor.

5. No Warranty 5.1 “As-Is” Basis: The Model, Derivatives, and Output are provided on an “as-is” and “as-available” basis, without any warranties or representations of any kind, whether express, implied, or statutory. The Licensor disclaims all warranties, including but not limited to, implied warranties of merchantability, fitness for a particular purpose, accuracy, reliability, non-infringement, or any warranty arising from the course of dealing or usage of trade. 5.2 Performance and Reliability: The Licensor does not warrant or guarantee that the Model, Derivatives or Output will meet the Licensee’s requirements, that the operation of the Model, Derivatives or Output will be uninterrupted or error-free, or that defects in the Model will be corrected. The Licensee acknowledges that the use of the Model, Derivatives or Output is at its own risk and that the Model, Derivatives or Output may contain bugs, errors, or other limitations. 5.3 No Endorsement: The Licensor does not endorse, approve, or certify any results, conclusions, or recommendations derived from the use of the Model. The Licensee is solely responsible for evaluating the accuracy, reliability, and suitability of the Model for its intended purposes.

6. Limitation of Liability 6.1 No Liability for Damages: To the fullest extent permitted by applicable law, in no event shall the Licensor be liable for any special, incidental, indirect, consequential, exemplary, or punitive damages, including but not limited to, damages for loss of business profits, business interruption, loss of business information, loss of data, or any other pecuniary or non-pecuniary loss arising out of or in connection with the use or inability to use the Model, Derivatives or any Output, even if the Licensor has been advised of the possibility of such damages. 6.2 Indemnification: The Licensee agrees to indemnify, defend, and hold harmless the Licensor, its affiliates, officers, directors, employees, and agents from and against any claims, liabilities, damages, losses, costs, or expenses (including reasonable attorneys' fees) arising out of or related to the Licensee's use of the Model, any Derivatives, or any Output, including any violation of this Agreement or applicable laws.

7. Termination 7.1 Termination by Licensor: The Licensor reserves the right to terminate this Agreement and revoke the Licensee’s rights to use the Model at any time, with or without cause, and without prior notice if the Licensee breaches any of the terms or conditions of this Agreement. Termination shall be effective immediately upon notice. 7.2 Effect of Termination: Upon termination of this Agreement, the Licensee must immediately cease all use of the Model, Derivatives, and Output and destroy all copies of the Model, Derivatives, and Output in its possession or control, including any backup or archival copies. The Licensee shall certify in writing to the Licensor that such destruction has been completed. 7.3 Survival: The provisions of this Agreement that by their nature should survive termination, including but not limited to, Sections 4 (Ownership), 5 (No Warranty), 6 (Limitation of Liability), and this Section 7 (Termination), shall continue to apply after termination.

8. Governing Law 8.1 Governing Law: This Agreement shall be governed by and construed in accordance with the laws of the Republic of Korea, without regard to its conflict of laws principles. 8.2 Arbitration: Any disputes, controversies, or claims arising out of or relating to this Agreement, including its existence, validity, interpretation, performance, breach, or termination, shall be referred to and finally resolved by arbitration administered by the Korean Commercial Arbitration Board (KCAB) in accordance with the International Arbitration Rules of the Korean Commercial Arbitration Board in force at the time of the commencement of the arbitration. The seat of arbitration shall be Seoul, Republic of Korea. The tribunal shall consist of one arbitrator. The language of the arbitration shall be English.

9. Alterations 9.1 Modifications: The Licensor reserves the right to modify or amend this Agreement at any time, in its sole discretion. Any modifications will be effective upon posting the updated Agreement on the Licensor’s website or through other means of communication. The Licensee is responsible for reviewing the Agreement periodically for changes. Continued use of the Model after any modifications have been made constitutes acceptance of the revised Agreement. 9.2 Entire Agreement: This Agreement constitutes the entire agreement between the Licensee and Licensor concerning the subject matter hereof and supersedes all prior or contemporaneous oral or written agreements, representations, or understandings. Any terms or conditions of any purchase order or other document submitted by the Licensee in connection with the Model that are in addition to, different from, or inconsistent with the terms and conditions of this Agreement are not binding on the Licensor and are void.

By downloading, installing, or using the EXAONE AI Model, the Licensee acknowledges that it has read, understood, and agrees to be bound by the terms and conditions of this Agreement.

DETAILS

ChatMODEL CLASS

MAX Model

MAX Models are popular open-source models converted to MAX’s native graph format. Anything with the label is either SOTA or being worked on. Learn more about MAX Models.

Browse all MAX Models

QUANTIZATION SPECS

HARDWARE

CPU

QUANTIZATION

Q4_K_M

CREATED BY

bartowski

LINK

bartowski/EXAONE-3.5-2.4B-Instruct-GGUF

MODEL SPECS

ARCHITECTURE

ExaoneForCausalLM

MODALITY

Code

MAX GITHUB

Modular / MAX

CREATED BY

LGAI-EXAONE

MODEL

LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct

Resources & support for
running EXAONE-3.5-Instruct-2.4B

Browse 27+ Tutorials

View Tutorials

Get help using MAX

Modular Forum

Read Documentation

Go to Docs

EXAONE-3.5-Instruct-2.4B