Text Generation
Transformers
Safetensors
PyTorch
llama
facebook
meta
llama-3
text-generation-inference
Instructions to use meta-llama/Llama-3.1-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use meta-llama/Llama-3.1-8B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-8B")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B") model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B") - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use meta-llama/Llama-3.1-8B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "meta-llama/Llama-3.1-8B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "meta-llama/Llama-3.1-8B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/meta-llama/Llama-3.1-8B
- SGLang
How to use meta-llama/Llama-3.1-8B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "meta-llama/Llama-3.1-8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "meta-llama/Llama-3.1-8B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "meta-llama/Llama-3.1-8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "meta-llama/Llama-3.1-8B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use meta-llama/Llama-3.1-8B with Docker Model Runner:
docker model run hf.co/meta-llama/Llama-3.1-8B
Access request FAQ
pinned#21 opened almost 2 years ago
by
samuelselvan
Request for clarification on repeated rejection of access request
#198 opened 13 days ago
by
zhanhuatao
Need access
#197 opened 16 days ago
by
hjt15574089453
need access
#196 opened about 1 month ago
by
blue-blue
Request: DOI
#195 opened about 1 month ago
by
Daniyalatta
Request: DOI
#194 opened about 1 month ago
by
Skowek
Request: DOI
#193 opened about 1 month ago
by
AshishPatel4886
Request: DOI
#192 opened about 1 month ago
by
Himanshu6692
Access Request
#191 opened about 2 months ago
by deleted
Reset my access request
#190 opened about 2 months ago
by deleted
Update README.md
#189 opened about 2 months ago
by
JoeFunny30
Access Request
#188 opened about 2 months ago
by
Xotiic-Official
Request for Access: Academic Research Purpose
#186 opened 2 months ago
by
evantsao
Request Wanted
#185 opened 2 months ago
by
yixuzh
Request : DOI
#184 opened 2 months ago
by
tuandebu
Request : DOI
#183 opened 2 months ago
by
tuandebu
fix: set `clean_up_tokenization_spaces` to `false`
#182 opened 2 months ago
by
maxsloef
Request: DOI
#181 opened 2 months ago
by
maka350
Request to reopen access request for Llama 3.1-8B
#180 opened 3 months ago
by deleted
Install & run meta-llama/Llama-3.1-8B easily using llmpm
#179 opened 3 months ago
by
sarthak-saxena
Access Request
#178 opened 3 months ago
by deleted
PetAI
#177 opened 3 months ago
by
chipkkang9
Access request
#174 opened 3 months ago
by
Zocotroco12
Request Access
#172 opened 4 months ago
by
Rebecca0876
Request: DOI
#171 opened 4 months ago
by
phil089
Re-evaluation for model access.
#170 opened 4 months ago
by
Qz07
Request for Re-evaluation: Llama 3.1 Access
#169 opened 4 months ago
by
Shotaro7
Request: DOI
#167 opened 5 months ago
by
nrusso18
Delay Giving Permission
#162 opened 6 months ago
by
ahmedjameel
Request: DOI
#161 opened 7 months ago
by
Angular27
sdff
#160 opened 7 months ago
by
Vitalya1604
my request to access LLama 3.1 model has been rejected , i want to re-apply
8
#159 opened 8 months ago
by
Michel-George
Need model for Learning
#157 opened 8 months ago
by
wasiqmahmood93
Request: DOI
#156 opened 8 months ago
by
darshanAnghan
Access request denied?
#155 opened 8 months ago
by
hetbell
Request: DOI
#154 opened 8 months ago
by
Madhuks
VLLM curl call error
#153 opened 9 months ago
by
VikramanHF
Rejected
#152 opened 9 months ago
by
GordenFree
Request: DOI - Request for Access (Access Denied)
#151 opened 9 months ago
by
amd-pratmish
Request: DOI
#150 opened 9 months ago
by
Vemo96
Request to access repo denied
❤️ 1
2
#149 opened 9 months ago
by
nizzyhuss
Request: DOI
#148 opened 9 months ago
by
tprashanth94
One possible reason getting rejected
#147 opened 10 months ago
by
kyle-amd-dev
Access to repo rejected.
#146 opened 10 months ago
by
preferablehuman
Access to Meta LLaMA Models — Request Rejected, No Reapply Option
1
#145 opened 10 months ago
by
TharunMahendra
Model Request for Llama 3.1 Family of Models Rejected
#144 opened 10 months ago
by
Reubensuju