Add llama.cpp ARM64 (Graviton) CPU-inference container - #6434
Open
Jyothirmaikottu wants to merge 1 commit into
Open
Add llama.cpp ARM64 (Graviton) CPU-inference container#6434Jyothirmaikottu wants to merge 1 commit into
Jyothirmaikottu wants to merge 1 commit into
Conversation
Introduces a new llama.cpp framework: an ARM64 (aarch64) CPU-only
inference container for AWS Graviton, serving quantized GGUF models via
llama-server's OpenAI-compatible HTTP API. This is the first ARM64 image
in the current pipeline and the first CPU-only LLM-serving image.
llama.cpp is compiled from a pinned upstream ref for aarch64 with
-mcpu=neoverse-v1 (Graviton3 baseline). A single image is portable across
Graviton3/4/5 because llama.cpp repacks quantized weights to the
CPU-optimal int8 layout at load time, so a Graviton3-baseline build still
gets the newer-generation int8/bf16 kernels at runtime.
Contents:
- docker/llama_cpp/Dockerfile.amzn2023 - multi-stage AL2023 aarch64 build
(builder compiles llama-server; slim CPU-only runtime), telemetry + OSS
compliance wired in like the other frameworks.
- scripts/docker/llama_cpp/dockerd_entrypoint.sh - EC2 serve entrypoint.
- .github/config/image/llama_cpp/ec2-amzn2023.yml - image config
(arch_type: arm64, device_type: cpu, job_type: inference; release gated
off until an ARM64 build fleet is available).
- .github/config/model-tests/llama-cpp-model-tests.yml - CPU smoke test on
an ARM64 fleet with a small Q4_0 GGUF.
- .github/workflows/llama-cpp.{pipeline,tests-model,pr-amzn2023,
autorelease-ec2-amzn2023}.yml - build/test/release pipeline scoped to the
CPU/EC2 path.
- test/llama_cpp/scripts/llama_cpp_model_smoke_test.sh - serves a GGUF and
asserts /health + a chat completion.
- test/security/data/ecr_scan_allowlist/llama_cpp/framework_allowlist.json.
The Dockerfile builds and serves a model end-to-end on a Graviton4
instance (image ~305 MB; /health ready in ~5s). CI build requires an
ARM64 CodeBuild fleet (fleet:arm64-llama-cpp-build-runner); until that is
provisioned the pipeline build job cannot run, so release is disabled in
the config.
Jyothirmaikottu
commented
Jul 21, 2026
| codebuild-fleet: | ||
| - name: "qwen2.5-0.5b-q4_0" | ||
| s3_model: "qwen2.5-0.5b-instruct-q4_0.gguf.tar.gz" | ||
| fleet: "arm64-c8g4xl-runner" |
Contributor
Author
There was a problem hiding this comment.
tbd - both model and fleet
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduces a new llama.cpp framework: an ARM64 (aarch64) CPU-only inference container for AWS Graviton, serving quantized GGUF models via
llama-server's OpenAI-compatible HTTP API. This is the first ARM64 image in the current pipeline and the first CPU-only LLM-serving image.llama.cpp is compiled from a pinned upstream ref for aarch64 with
-mcpu=neoverse-v1(Graviton3 baseline). A single image is portable across Graviton3/4/5 because llama.cpp repacks quantized weights to the CPU-optimal int8 layout at load time, so a Graviton3-baseline build still receives the newer-generation int8/bf16 kernels at runtime.What's included
docker/llama_cpp/Dockerfile.amzn2023— multi-stage AL2023 aarch64 build (builder compilesllama-server; slim CPU-only runtime), with telemetry + OSS-compliance wired in like the other frameworks.scripts/docker/llama_cpp/dockerd_entrypoint.sh— EC2serveentrypoint..github/config/image/llama_cpp/ec2-amzn2023.yml— image config (arch_type: arm64,device_type: cpu,job_type: inference; release gated off for now — see note)..github/config/model-tests/llama-cpp-model-tests.yml— CPU smoke test on an ARM64 fleet with a smallQ4_0GGUF..github/workflows/llama-cpp.{pipeline,tests-model,pr-amzn2023,autorelease-ec2-amzn2023}.yml— build/test/release pipeline scoped to the CPU/EC2 path.test/llama_cpp/scripts/llama_cpp_model_smoke_test.sh— serves a GGUF and asserts/health+ a chat completion.test/security/data/ecr_scan_allowlist/llama_cpp/framework_allowlist.json.Testing
The Dockerfile builds and serves a model end-to-end on a Graviton4 instance:
docker buildsucceeds (image ~305 MB).Q4_0;/healthready in ~5s;/v1/chat/completionsreturns a correct response.Note for reviewers
The CI build job targets an ARM64 CodeBuild fleet (
fleet:arm64-llama-cpp-build-runner). Until that fleet is provisioned the build job cannot run, soreleaseis set tofalsein the image config. This PR is intended for review of the framework structure; enabling the build fleet is a follow-up.