How I Interview AI Engineers: The Projects That Actually Impress Me

Applied AI roles, especially at agent startups, are starting to converge on a fairly predictable set of problems. Everyone knows the basic definition of an agent by now. The more interesting question is whether you understand how to make one reliable, scalable, measurable, and useful when it runs for a long time in the real world.

If you are targeting GenAI startups in India or the US, do not stop at wrapping an LLM API in a nice UI. Build something that forces you to deal with sandboxing, persistence, context management, evals, integrations, and observability. That is the gap between a fun side project and a project that makes a strong interview conversation.

The Applied AI Topics Worth Knowing First

Before getting into projects, there are five broad areas that keep showing up when building serious agents.

These five areas cover much of the engineering work behind production-grade AI agents.

• Memory: How does an agent remember who a person is, what it has done before, and what it learned from previous sessions?

• Firecracker and sandboxing: Firecracker is a virtualization technology also used under the hood by AWS Lambda and E2B. Understanding how to create and operate isolated execution environments is extremely useful for coding agents.

• Context engineering: Long-running tasks create huge contexts. You need ways to compact, summarize, retrieve, and manage that context without making the system expensive or causing context rot.

• Evals and RL environments: How do you test whether an agent is actually improving? How do you compare one model or harness against another objectively?

• Cloud agents: Products such as Devin involve far more than an LLM call. They are full-stack systems with remote execution, infrastructure, orchestration, and user-facing timelines.

For instance, a model may have an extremely large context window, but that does not mean you should keep blindly appending everything forever. When a long task reaches hundreds of thousands of tokens, cost and quality can both become problems. A strong engineer should be able to talk through what gets retained, what gets summarized, and what must remain directly accessible.

The AI Agent Interview Questions That Matter

Imagine the task is to build a coding agent similar to Devin. A user submits a GitHub issue, the system opens an isolated environment, reads and edits code, runs commands and tests, and eventually makes a commit. The interface shows a timeline of the agent opening terminals, reading files, writing files, and taking actions.

That sounds like a straightforward product prompt. It is also a very good way to find out whether someone understands applied AI engineering beyond prompts.

A coding agent interview should probe architecture, durability, context, evaluation, and observability.

1. What services does the system need?

Start with the architecture. How many frontends and backends exist? What runs in the infrastructure layer? How does the system work when 100 people are using the agent simultaneously?

The key insight is that a coding agent generally needs a separate sandbox per active task. If 100 users are simultaneously asking agents to fix repositories, you may need roughly 100 isolated environments running at once. Those environments need to be started, secured, monitored, and shut down when the work is complete.

The frontend and backend themselves may be relatively standard. LLM provider capacity is also generally external if you are using an API. The infrastructure-heavy part is managing the sandboxes:

• How are sandboxes provisioned and torn down?

• How are repositories, credentials, and task state isolated?

• What happens when demand suddenly increases?

• How does the system prevent one task from affecting another?

• Where does the final GitHub commit originate?

If you can reason clearly about sandbox lifecycle and horizontal scaling, you are already discussing the real difficulty of this class of product.

2. What happens when the agent server crashes?

Normal full-stack apps do not always have jobs that run for 30 minutes or several hours. Agents do. A coding agent may be stuck on a difficult issue, exploring a repository, running tests, retrying commands, and iterating for a long time.

So if the backend or sandbox crashes halfway through, how does the task resume?

You need durability across sessions. Persist enough state that another worker or restarted service can pick up from a checkpoint. In an agent system, the exchange between the agent and the model gives you a useful recovery point. If you have stored the message history and task state, another part of the infrastructure can reconstruct the conversation and continue.

This is where workflow durability systems such as Temporal become relevant. You do not need to treat one server process as the single source of truth for a multi-hour workflow.

3. How do you manage context?

Short conversations are usually not the problem. Models tend to perform reasonably well when the task is small and the context is controlled. The difficult cases are large features, difficult bugs, and gnarly repository issues that create lengthy tool traces and many rounds of back-and-forth.

A long-running coding agent needs a context strategy. Common directions include:

• Periodically compacting the working history.

• Summarizing completed investigations and decisions.

• Preserving critical state, such as current files, test failures, and user constraints.

• Retrieving relevant repository information rather than keeping every detail in the prompt.

• Switching techniques when simple compaction loses too much important context.

The goal is not merely to fit under a context limit. It is to make sure the agent continues reasoning effectively without burning unnecessary tokens.

4. How do you evaluate whether an agent improved?

This is the part many side projects miss entirely. It is easy to ship a new prompt, harness, or model and feel that it looks better. That is subjective. A serious team needs an objective answer before a release: did the agent get better, worse, or simply different?

Coding-agent tasks are particularly suitable for deterministic evaluation. Benchmarks such as SWE-bench and Terminal-Bench provide a way to check whether a model or agent harness solves a defined task successfully. Eventually, every company will need its own evals built around its own product data and workflows.

The point of an eval suite is to answer questions like these:

• Does model A outperform model B on our important tasks?

• Did the new agent loop improve issue resolution rates?

• Did a prompt or tooling change cause regressions?

• Are engineers becoming more effective because of the AI component?

Writing good evals is a niche skill today, but it will become much more common. A lot of future engineering work will involve building tests, human tasks, and RL environments that teach us whether agents are improving.

5. What do you observe as the application grows?

We already had observability in traditional software. Tools such as Datadog, New Relic, Prometheus, and Grafana help teams understand whether services are working. Agents introduce another set of questions.

• Where does the agent fail?

• What tool call, context state, or infrastructure event caused the failure?

• Did the agent reach a final answer?

• Are people happy with the output?

• What trace led to the final result?

• Did a model provider or infrastructure provider become unavailable?

At scale, an AI application needs to detect failures and respond. If one infrastructure provider is down or capacity is unavailable, the system may need to switch providers. If the agent suddenly becomes less reliable, the team should be able to identify why rather than guess.

These questions apply beyond coding agents. Any company building agentic workflows eventually has to improve the agent, evaluate the agent, observe the agent, and scale its execution environment.

Seven Applied AI Projects That Will Teach You the Right Things

The best portfolio projects force you to solve a real systems problem, not just make an LLM call.

1. Build a terminal agent, then benchmark it

A terminal agent is the simplest starting point. Open-source agents such as Pi have readable TypeScript codebases that are small enough to understand if you are comfortable with TypeScript.

But building the initial agent is not the impressive part. The hard part is getting it to perform competitively with agents such as Claude Code while using the same underlying model.

Do not just build the loop and call it done. Benchmark it against other agents. If it performs worse, investigate why. Try changes and measure whether they help.

There may be shortcuts that push benchmark performance in the right direction, including handling specific edge cases. That is not the final product solution, but it can help reveal which capabilities matter. The real goal is to understand what changes actually improve task completion.

For example, Pi does not include sub-agent orchestration by default, while Claude Code and Codex do. Yet Pi can still perform almost as well on Terminal-Bench. That tells you sub-agent orchestration is one possible lever, not a magical requirement. Learn to form hypotheses, test them, and improve the harness.

2. Build an autonomous multi-platform bot

A Hermes-style cloud bot is more involved than a basic terminal agent. It needs memory and integrations with platforms such as WhatsApp, Telegram, and Slack. At that point, the work becomes partly full-stack engineering.

The core challenge is connecting several services while letting the AI take autonomous actions over time. This forces you to think about:

• Persistent memory and user state.

• Webhooks and event-driven workflows.

• Permissions and integration boundaries.

• Action logging and recovery.

• How much autonomy the system should have.

Many strong projects in this area are open source. Use an LLM to help summarize an unfamiliar architecture, then verify the parts that matter by tracing the code and the data flow yourself.

3. Create an AI-native Slack workspace

There is a potentially large category of products that looks like Slack but treats agents as first-class workspace members. In a normal Slack workspace, the members are humans. In an AI-native workspace, you can also tag specialized agents.

One agent might be focused on frontend work. Another could handle writing. Another could investigate data or respond to customer questions. You interact with them in a collaborative channel rather than orchestrating every action from a terminal.

Projects such as Buzz and PromptQL are useful references for this direction. Buzz is open source and written in Rust, making it an interesting project for someone who wants to contribute to an AI product while building Rust skills.

The important design question is not merely, “Can I add a chatbot to Slack?” It is, “What does collaboration look like when agents participate in the workspace with distinct roles and capabilities?”

4. Build a multi-agent coding workspace

A multi-agent coding workspace gives one interface for multiple coding harnesses. For example, a person could assign one issue to Claude Code, another to Codex, and a third to Devin.

Two products can look similar at the UI level but have very different architectures.

• CLI-oriented architecture: Spawn the existing CLIs in separate tabs and surface their terminal output. This is mainly a full-stack integration challenge.

• Agent-native architecture: Start or integrate with the underlying agent directly, intercept its messages, and render a custom interface around the actual agent loop.

The second approach can teach you much more about agents, including the messages an agent produces and the back-and-forth between the agent and the model. The first approach can still be useful, but it is closer to a polished orchestration layer around existing tools.

Both Superset and T3 Code are worth studying as architectural references. Building your own version requires you to make real product and systems choices instead of merely embedding one coding agent.

5. Build a generative UI learning platform

Generative UI is a very interesting technical direction. Rather than presenting a static course catalog, build a platform that generates a structured learning experience based on the topic someone wants to learn.

For example, a request to learn transformers or the paper “Attention Is All You Need” could result in a course structure, slides, explanatory material, and an agent that guides the learner through the material.

Generative UI can turn a topic request into a structured course with navigable lessons and agent assistance.

In the past, creating a course like that required manually gathering material, writing lessons, and assembling the structure. A generative system can create much of that initial interface in one go.

Whether a fully AI-driven learning product achieves product-market fit is still open to debate. Personal learning often needs accountability and human guidance. But as a technical project, it is excellent because it combines content generation, stateful interaction, course structure, and dynamic UI creation.

A variation is an AI-powered slides and Mentimeter-style platform. An admin could select a topic such as transformers and attention, generate a slide deck, add a 10 or 20 question quiz, and distribute it to a class. That is a very practical generative UI project.

6. Build a smart AI model router

A model router is not just an API aggregator. The interesting version is a system that understands the incoming request and decides which model should handle it.

AI usage can get expensive very quickly when developers have access to effectively unlimited credits. At some point, companies may give everyone a coding-agent interface but choose the underlying model based on task complexity, cost limits, and policy.

A useful router might decide:

• Does this request require the most capable model?

• Would a cheaper general model be good enough?

• Would an open-source model work better for this task?

• Does this person have sufficient budget remaining?

• Should the request be blocked because it is unrelated to company work?

This is a nuanced project because there is no universally correct routing strategy yet. It sits in the middle of full-stack engineering and AI engineering. Whoever solves it well can help companies save significant money while still delivering useful output.

7. Build benchmarks and eval frameworks for a real repository

This is probably the most valuable and difficult project on the list. Pick an open-source codebase and build an evaluation framework that determines which agents or models are better at solving issues in that repository.

Do not underestimate how hard this is. Even setting up a real codebase locally can take significant effort. Once it runs, you still need to create tasks, tests, and environments that correctly measure model capability.

The challenge is not writing a thousand tests just for the sake of it. The challenge is ensuring those tests actually represent what it means for an agent to perform well on that project.

The work has shifted. It used to be impressive to set up a complex repository locally and solve an issue yourself. Increasingly, agents will solve many individual issues. The more valuable question is whether you can build an evaluation suite that reveals which agent is genuinely better at solving those issues.

Over time, companies will likely spend more engineering effort on evals that both measure capability and create useful training data for their own codebases. That may sound a little meta, but it is a reasonable direction for AI-native software teams.

How to Make Any of These Projects Interview-Worthy

Pick one project and spend a week going deeper than the obvious implementation. The implementation alone is rarely the differentiator. The differentiator is whether you can explain the difficult tradeoffs.

• Show the architecture and why each service exists.

• Document failure modes and how the system recovers.

• Measure quality with an eval suite instead of anecdotes.

• Track cost, latency, success rate, and failure traces.

• Explain what you changed after benchmarking the first version.

• Be honest about the parts that are still speculative or unsolved.

If you can do that, you are not merely saying that you know what an agent is. You are showing that you know how to build, operate, improve, and reason about one.

For more structured applied AI learning, you can explore the 100xDevs cohort.