RedHatAI
/

Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Text Generation

text-generation-inference

compressed-tensors

Model card Files Files and versions

robgreenberg3 commited on 27 days ago

Commit

dc7738d

·

verified ·

1 Parent(s): d450271

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -87,7 +87,7 @@ vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://do
   <summary>Deploy on <strong>Red Hat AI Inference Server</strong></summary>
 ```bash
-$ podman run --rm -it --device nvidia.com/gpu=all -p 8000:8000 \
  --ipc=host \
 --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
 --env "HF_HUB_OFFLINE=0" -v ~/.cache/vllm:/home/vllm/.cache \

   <summary>Deploy on <strong>Red Hat AI Inference Server</strong></summary>
 ```bash
+podman run --rm -it --device nvidia.com/gpu=all -p 8000:8000 \
  --ipc=host \
 --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
 --env "HF_HUB_OFFLINE=0" -v ~/.cache/vllm:/home/vllm/.cache \