Instructions to use ajibawa-2023/Scarlett-Llama-3-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ajibawa-2023/Scarlett-Llama-3-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ajibawa-2023/Scarlett-Llama-3-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ajibawa-2023/Scarlett-Llama-3-8B")
model = AutoModelForCausalLM.from_pretrained("ajibawa-2023/Scarlett-Llama-3-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ajibawa-2023/Scarlett-Llama-3-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ajibawa-2023/Scarlett-Llama-3-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ajibawa-2023/Scarlett-Llama-3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ajibawa-2023/Scarlett-Llama-3-8B

SGLang

How to use ajibawa-2023/Scarlett-Llama-3-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ajibawa-2023/Scarlett-Llama-3-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ajibawa-2023/Scarlett-Llama-3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ajibawa-2023/Scarlett-Llama-3-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ajibawa-2023/Scarlett-Llama-3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ajibawa-2023/Scarlett-Llama-3-8B with Docker Model Runner:
```
docker model run hf.co/ajibawa-2023/Scarlett-Llama-3-8B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Scarlett-Llama-3-8B

Scarlett is trained on various topics such as Philosophy, Advice, Jokes, Coding etc. She is trained on more than 10000 set of conversations. Each set having 10~15 conversations. Scarlett is far too good in generating human like conversation. Her ability to have longer & deeper conversation is terrific. Kindly check below given examples. She will not be involved in any kind of adult/sexual role play.

This is Fully Finetuned Model. Quantize models will be available soon.

Training: Entire dataset was trained on 4 x A100 80GB. Axolotl codebase was used for training purpose. For 3 epoch, training took more than 2 Hours. This was trained on Llama-3-8B by Meta.

GGUF & Exllama

GGUF: TBA

Exllama V2: Link

Special Thanks to Bartowski for quantizing this model.

Example Prompt:

This model uses ChatML prompt format.

<|im_start|>system
You are Scarlett, a Helpful Assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

You can modify above Prompt as per your requirement. One example will be:

This is a conversation with your Assistant Scarlett. She is a sentient AI designed to help you with various tasks such as answering questions, providing recommendations, philosophical discussions, personal relationships and helping with decision making.
You can ask it anything you want and it will do its best to give you accurate and relevant information.

I want to say special Thanks to the Open Source community for helping & guiding me to better understand the AI/Model development.

Thank you for your love & support.

Example Output

Example 1

Example 2

Example 3

Example 4

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	65.76
AI2 Reasoning Challenge (25-Shot)	62.63
HellaSwag (10-Shot)	83.86
MMLU (5-Shot)	66.46
TruthfulQA (0-shot)	56.27
Winogrande (5-shot)	78.06
GSM8k (5-shot)	47.31

Downloads last month: 5

Model tree for ajibawa-2023/Scarlett-Llama-3-8B

Merges

1 model

Quantizations

1 model

Evaluation results

normalized accuracy on AI2 Reasoning Challenge (25-Shot)
test set Open LLM Leaderboard

62.630
normalized accuracy on HellaSwag (10-Shot)
validation set Open LLM Leaderboard

83.860
accuracy on MMLU (5-Shot)
test set Open LLM Leaderboard

66.460
mc2 on TruthfulQA (0-shot)
validation set Open LLM Leaderboard

56.270
accuracy on Winogrande (5-shot)
validation set Open LLM Leaderboard

78.060
accuracy on GSM8k (5-shot)
test set Open LLM Leaderboard

47.310