The short answer: AI engineer interviews test whether you can build AI features that work reliably in production, not just call a model API. Expect machine learning fundamentals (overfitting, metrics, data leakage), building with large language models (retrieval-augmented generation, prompting, tool use and agents), evaluation (how you know it works), deployment and monitoring, a design question (“Build an AI assistant for our support team”), coding, and behavioral questions. If you don’t code, there are AI roles that suit you, such as AI product manager or AI trainer, and their interviews focus on judgment, evaluation and communication instead (see the section at the end).
Interviews have moved quickly toward agents, retrieval details and security, so this guide covers those too. Answers are checked against primary sources, listed at the end, and the coding samples were run on Python 3.12.
What the AI engineer loop looks like
- Coding: usually Python, often practical (process data, call an API, write a small evaluation script) as well as data structures.
- ML and LLM fundamentals: concepts and trade-offs.
- System design: an AI feature end to end, including data, evaluation and cost.
- Project deep dive: something you built, what didn’t work, and what you measured.
- Behavioral: working with product teams, handling failures, and responsible AI judgment.
Machine learning fundamentals
“What is overfitting, and how do you prevent it?”
Google’s Machine Learning Crash Course defines overfitting as creating a model that matches the training set so closely that it fails to make correct predictions on new data. Prevent it with more or better data, simpler models, regularization, early stopping, and evaluation on a held-out validation and test set.
“Precision or recall?”
Precision is the share of positive predictions that are correct, TP / (TP + FP); recall is the share of actual positives the model finds, TP / (TP + FN). Choose by the cost of errors: for a fraud alert that a human reviews, you might favor recall; for an automated account ban, precision. Google’s course notes that these metrics change with the classification threshold, which is often tuned to the business need. Accuracy alone misleads on imbalanced data.
“What is data leakage?”
Information from outside the training data, often from the future or from the target itself, sneaks into the features, so the model looks great in testing and fails in production. Split data by time when predicting the future, and build features only from what’s available at prediction time.
“How do you handle imbalanced data?”
Use metrics such as precision, recall and the precision-recall curve, resample or weight classes, and tune the threshold.
Model internals
Companies that train or serve their own models go a level deeper than the API.
“What is attention, and why does it matter?”
Attention lets a model weigh how much each token matters to every other token. Google’s Machine Learning Crash Course describes self-attention as asking, for each input token, how much each other token affects its interpretation. The 2017 paper “Attention Is All You Need” introduced the Transformer, an architecture based solely on attention mechanisms, and that’s the design today’s large language models build on.
“What is the KV cache?”
A model generates one token at a time, and each new token depends on all the earlier ones. A key-value (KV) cache stores the keys and values from the attention layers for tokens already processed, so they’re reused instead of recomputed. Hugging Face’s documentation gives the trade-off: attention cost per step goes from quadratic to linear in sequence length, while memory grows linearly. So long sequences and big batches cost you memory. Caching is for inference, not training.
“What is LoRA, and when would you use it?”
LoRA (low-rank adaptation), from Hu and colleagues in 2021, freezes the pretrained weights and trains small added matrices instead. The paper reports it can cut the number of trainable parameters by 10,000 times compared with fine-tuning GPT-3 175B. Reach for it when you need fine-tuning on a budget, or several task-specific adapters on one base model.
“Why quantize a model, and what can go wrong?”
Quantization stores weights in lower-precision types such as 8-bit integers. Hugging Face notes that this cuts memory and compute, lets larger models fit, and speeds up inference. The risk is lost quality, and it varies by model and task, so measure each level (8-bit, 4-bit) on your evaluation set instead of assuming.
Building with LLMs
“When would you use RAG, fine-tuning or just prompting?”
Prompting first: it’s fastest and cheapest to iterate. Retrieval-augmented generation (RAG), introduced by Lewis and colleagues in 2020, combines a model with retrieved documents, which suits knowledge that changes or is private, and makes answers traceable to sources. Fine-tuning suits consistent style or format, specialized tasks, or making a smaller model good enough to cut cost and latency. Many systems combine RAG and light fine-tuning. Our LLM and RAG interview questions go deeper.
“How do tool use and agents work?”
The model is given descriptions of tools (search, a database query, an API), decides when to call them, and the application executes the calls and returns results to the model. Agents chain these steps toward a goal. Key engineering concerns: limiting which tools can be called and with what permissions, validating inputs, setting step and cost limits, and logging every action for review.
“How do you reduce hallucinations?”
Ground answers in retrieved sources and ask the model to cite them, instruct it to say when it doesn’t know, constrain outputs (structured formats, allowed values), check answers against sources automatically, and keep a human in the loop for high-stakes actions.
Retrieval beyond the basics
Our LLM and RAG guide covers pipelines, chunk sizes and hybrid search. These are the follow-ups that come next.
“The retrieved context doesn’t contain the answer. What should the system do?”
Say so. Set a minimum relevance score below which nothing reaches the model, and calibrate it on your evaluation set, because scores differ between embedding models. Tell the model to answer only from the context provided and to say when it can’t. Return a clear fallback or hand off to a person, and log those queries: they show you what your knowledge base is missing.
“What does a reranker do?”
Retrieval casts a wide net cheaply. A reranker is a slower, more accurate second stage that reorders the top results, so only the best few reach the prompt. It adds latency and cost, so keep it only if your evaluation shows better answers.
“How do you build permission-aware retrieval?”
Store who can see each document as metadata, and filter on the identity of the person asking at query time. Microsoft’s Azure AI Search documentation calls this the security filter pattern: a field of allowed groups on each document, and a filter applied to every query. Never leave that decision to the model. OWASP’s 2025 Top 10 for LLM applications warns about context leaking between users in shared vector databases, and recommends permission-aware stores and strict logical partitioning of data.
Agentic AI interview questions
“What’s the difference between a workflow and an agent?”
Anthropic’s engineering guide on building agents draws the line this way: workflows orchestrate models and tools through predefined code paths, while agents let the model dynamically direct its own process and tool use. The deciding question is who picks the next step, your code or the model. The same guide recommends the simplest solution that works, so a strong answer includes when you wouldn’t use an agent.
“What is the ReAct pattern?”
ReAct, from Yao and colleagues in 2022, has the model interleave reasoning with actions: think, call a tool, read the result, think again. Compared with chain-of-thought alone, the model can look things up and correct course. The paper reports that this overcame the hallucination and error-propagation problems of chain-of-thought on its question-answering and fact-verification tasks.
“What is MCP, and how is it different from function calling?”
The Model Context Protocol is an open protocol that standardizes how AI applications connect to outside data and tools. It uses JSON-RPC 2.0 messages between hosts (the AI application), clients (connectors inside the host) and servers (which provide the capabilities), and servers can offer resources, prompts and tools. Function calling is how a model asks your application to run a function in one API. MCP is how tools get exposed to any application that speaks the protocol, so one integration can serve many clients. The specification treats tools as arbitrary code execution, and says hosts must obtain explicit user consent before invoking one.
“How do you stop an agent from looping or doing damage?”
- Limit steps and spending. Anthropic’s guide notes it’s common to include stopping conditions such as a maximum number of iterations. Add timeouts and a budget cap.
- Return tool errors to the model. Let it recover from a bad call, and retry only transient failures, with backoff.
- Make actions safe to repeat. Use idempotency keys so a retry doesn’t send the same email or payment twice.
- Shrink what it can do. OWASP calls this risk excessive agency. Its mitigations include offering only the tools the task needs, granting the least permission (read-only where possible), enforcing authorization in the downstream system rather than trusting the model, and requiring human approval for high-impact actions.
“How do you evaluate an agent?”
Check the outcome on realistic tasks in a test environment, then the path it took: the right tools, the right arguments, how many steps, what it cost. Agents vary from run to run, so run each case several times, and read the transcripts of failures yourself.
Evaluation
“How do you know your AI feature works?”
Build an evaluation set of real, representative examples with expected outcomes before tuning anything. Measure task-specific metrics (accuracy of extracted fields, answer correctness and faithfulness to sources, refusal rates), use automated checks and LLM-based graders validated against human judgment, and track results for every change to prompts, models or retrieval. Then confirm with online metrics such as task completion, user feedback and escalations.
“How do you evaluate a RAG system?”
Score retrieval and generation separately, so you know which one failed. The Ragas paper (Es and colleagues, 2023) separates three dimensions: whether retrieval finds relevant, focused passages, whether the model uses them faithfully, and the quality of the generated answer.
Deployment and monitoring
- Latency and cost: cache repeated requests, stream responses, use smaller models for simpler steps, and limit context size.
- Monitoring: track quality signals, errors, latency, cost per request and drift in inputs over time.
- Safety and privacy: filter inputs and outputs, protect against prompt injection (especially from retrieved documents and tool results), and don’t send sensitive data where policy doesn’t allow it.
- Rollouts: test new prompts or models against the evaluation set, then release gradually.
Security questions
“What is prompt injection, and how do you defend against it?”
It’s the first entry in OWASP’s 2025 Top 10 for LLM applications. A direct injection comes from the user’s own input. An indirect one arrives in content the model reads, such as a website or file. There’s no single fix, so layer defenses: constrain the model’s role, validate outputs against an expected format with deterministic code, filter inputs and outputs, give the application least privilege, require human approval for high-risk actions, clearly separate untrusted content, and test adversarially.
“What else is on the OWASP Top 10 for LLM applications?”
Sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. Be ready with an example of two or three, such as running model-generated SQL without validation (improper output handling) or a user running up your bill (unbounded consumption).
Coding questions
Expect practical Python more than puzzles. Our Python interview questions cover the language itself.
“Write cosine similarity without a vector library.”
from math import hypot, sumprod
def cosine(a, b):
norms = hypot(*a) * hypot(*b)
if norms == 0:
return 0.0
return sumprod(a, b) / normsmath.sumprod needs Python 3.12 or later. Mention the zero-vector case before you’re asked.
“Write the core loop of an agent, with a step limit.”
MAX_STEPS = 5
def run_tool(tools, call):
fn = tools.get(call["name"])
if fn is None:
return "error: unknown tool"
try:
return str(fn(**call["args"]))
except Exception as exc:
return f"error: {exc}"
def run_agent(model, tools, msgs):
for _ in range(MAX_STEPS):
reply = model(msgs)
call = reply.get("tool")
if call is None:
return reply["text"]
result = run_tool(tools, call)
msgs.append({
"role": "tool",
"content": result,
})
raise RuntimeError("step limit")Errors go back to the model as text so it can recover, and the loop can’t run forever. Then say what you’d add for production: timeouts, retries for transient errors, permission checks and logging.
Design question: “Build an AI assistant for our support team”
- Clarify: who uses it (agents or customers), which questions, accuracy needs, languages, data sources, and what happens when it’s wrong.
- Data: help-center articles and past resolved tickets, cleaned, chunked and indexed with metadata.
- Flow: retrieve relevant content, generate a draft answer with citations, and show it to the agent to edit and send (human in the loop first).
- Evaluation: 200 to 500 real past tickets with known good answers; measure correctness, faithfulness and time saved.
- Rollout and monitoring: a pilot team, feedback buttons, weekly review of failures, cost per ticket.
- Risks: outdated articles, private customer data, and prompt injection in incoming messages.
Behavioral questions
- “Tell me about an AI project that didn’t work. What did you learn?”
- “How did you decide your model was good enough to ship?”
- “Tell me about a time you pushed back on using AI for something.”
- “How do you explain model limitations to non-technical stakeholders?”
If you’re interviewing for a junior AI engineer role
Expect more fundamentals and coding, and less system design depth. The best preparation is one small project you can defend end to end: a retrieval app over real documents, with an evaluation set, the numbers it produced, and what you changed because of them.
AI roles for non-coders
Not every AI job requires heavy coding. Roles where interviews focus on judgment, evaluation and communication include:
- AI product manager: defining what to build, success metrics and evaluation. See our product manager interview questions.
- AI trainer or data annotator: writing and rating model responses, often requiring subject expertise. See our guides to AI trainer jobs and data annotation jobs.
- AI evaluation or quality specialist: designing test sets and reviewing model behavior.
- AI solutions or implementation consultant: helping customers adopt AI tools.
- AI governance and policy: risk, compliance and responsible use.
Expect questions such as “How would you evaluate whether this AI feature is ready?”, “Give an example of a model response you’d rate poorly, and why,” and “How would you explain this AI tool’s limitations to a customer?” Basic literacy helps everyone: what models can and can’t do, prompting, and why evaluation matters.
Frequently asked questions
What questions are asked in an AI engineer interview?
What is the difference between precision and recall?
When should you use RAG instead of fine-tuning?
How do you evaluate an AI feature?
Can I get an AI job without coding?
What is the difference between a workflow and an agent?
What is MCP?
What is prompt injection?
What coding is asked in an AI engineer interview?
Sources
- Google for Developers, Machine Learning Crash Course: Accuracy, precision and recall and Overfitting. Read on 1 October 2026.
- Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. Read on 1 October 2026.
- Google for Developers, Machine Learning Crash Course: LLMs: What’s a large language model?. Read on 9 November 2026.
- Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762. Read on 9 November 2026.
- Hugging Face Transformers documentation: How caching works and Quantization. Read on 9 November 2026.
- Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. Read on 9 November 2026.
- Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. Read on 9 November 2026.
- Es, S., et al. (2023). Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217. Read on 9 November 2026.
- Anthropic, Building effective agents, 19 December 2024. Read on 9 November 2026.
- Model Context Protocol, Specification, version 2026-07-28. Read on 9 November 2026.
- OWASP, Top 10 for LLM Applications 2025, including Prompt Injection, Excessive Agency and Vector and Embedding Weaknesses. Read on 9 November 2026.
- Microsoft Learn, Security filter pattern in Azure AI Search. Read on 9 November 2026.
