This book is for software developers who are new to AI and want to understand how AI models are served in real products.
Most beginner AI material focuses on training models. Inference engineering is different. It is about taking a trained model and making it answer users reliably, quickly, safely, and at a cost the business can afford.
You do not need machine learning experience to read this book. The chapters explain terms from the ground up and use software engineering language wherever possible.
The book follows one practical product idea: SupportBot, an AI assistant that answers customer questions from a company's documentation.
SupportBot is useful because it touches almost every inference engineering topic:
- It receives user questions through an API.
- It searches company documents using embeddings.
- It builds prompts for an LLM.
- It streams answers back to users.
- It logs latency, token usage, quality signals, and errors.
- It must be fast, reliable, safe, and affordable.
The full request flow looks like this:
flowchart LR
User[User asks a question] --> API[Application API]
API --> Auth[Auth and rate limit]
Auth --> Retrieve[Retrieve relevant documents]
Retrieve --> Prompt[Build prompt]
Prompt --> Tokens[Tokenize input]
Tokens --> Server[Inference server]
Server --> GPU[GPU runs model]
GPU --> Stream[Stream tokens]
Stream --> Validate[Validate and post-process]
Validate --> User
Server --> Metrics[Logs and metrics]
By the end of the book, you should be able to explain each box in that diagram and know what can go wrong inside it.
Read the book in order if you are new to AI. Later chapters depend on the vocabulary from earlier chapters.
If you are already a developer working near production systems, pay special attention to:
- Part 2: Inference
- Part 4: Software
- Part 5: Optimization
- Part 7: Production
Each chapter now uses this structure:
- Plain-language explanation
- Concrete example
- Diagram or flow where useful
- Production notes
- Common mistakes
- Developer checklist
This part explains the basic ideas behind AI, machine learning, deep learning, neural networks, transformers, large language models, and common terminology.
- 01. What Is Artificial Intelligence?
- 02. Machine Learning
- 03. Deep Learning
- 04. Neural Networks
- 05. Transformers
- 06. Large Language Models
- 07. AI Terminology
This part explains what happens when a model receives a request and generates an answer.
- 08. Training vs Inference
- 09. How LLMs Generate Text
- 10. Tokenization
- 11. Prefill And Decode
- 12. KV Cache
- 13. Sampling
- 14. Performance Metrics
This part explains GPUs and the hardware choices that affect inference speed, cost, and reliability.
This part introduces the software stack commonly used to run models efficiently.
This part explains techniques used to make inference faster, cheaper, and more scalable.
- 26. Quantization
- 27. Speculative Decoding
- 28. KV Caching
- 29. Model Parallelism
- 30. Disaggregation
- 31. Attention Optimizations
This part explains text, vision, speech, image, and video systems from an inference point of view.
This part uses one realistic case study: a bank with 100 developers wants an on-prem LLM assistant for daily engineering work and cannot send prompts or code to a public cloud provider.
- 38. Bank On-Prem LLM Requirements
- 39. On-Prem Deployment Options
- 40. Kimi K2 Sizing Math
- 41. Hosting And Deployment Plan
- 42. Security, Governance, And Audit
- 43. Operating The On-Prem LLM
- 44. Cost, Capacity, And Rollout Decision
Inference engineering is the discipline of making trained AI models useful in production.
It combines:
- Software engineering
- Distributed systems
- Hardware awareness
- Model behavior understanding
- Performance measurement
- Production operations
The goal is not only to make a model work. The goal is to make it work for real users under real traffic, real latency targets, real budgets, and real failure conditions.