By Jay Krivanek, AI Engineering Manager
The most common question I get about DJ, our internal AI assistant at JWX, is some version of "so you trained it on our documentation?" It's a fair question. DJ answers questions about our deployments, our incidents, our docs, and our code like it's been here for years. It might surprise some but we have never trained anything. Not a model, not a fine-tune, we don’t even build an embedding index. Understanding why is probably the single most useful piece of AI literacy I can offer, because it explains how we use and develop with AI across the whole company.
First let me give you some terminology. Training is any process that durably modifies a model's weights. Context engineering is everything you do to a frozen model at inference time by controlling what enters its context window. That's it. The mental model I give everyone is that the weights are long-term knowledge frozen at deployment, and the context window is a scratchpad rebuilt on every single request. Training is like a student doing practice problems with an answer key, changing what they know. Inference is that same student taking the exam. They can read anything you put on the desk, but they leave the room knowing exactly what they knew walking in. DJ is the student and we hand a very well-organized desk.
I am not saying training doesn't matter. Model builds do an enormous amount of it to produce the Claude models DJ calls. Understanding the scaling differences matters though. Training a frontier model burns on the order of ten billion times the compute of answering one question. Nobody accidentally trains a model. Every answer DJ gives sits at the cheap end of that scale, running against weights that never change.
So how does DJ know anything about JWX? DJ is a harness, a loop of code wrapped around frozen Claude models running on AWS Bedrock. Around that loop sit roughly twenty-five live tool modules: GitLab, Datadog, JIRA, Buildkite, Sentry, Notion, Google Drive, Slack, Snowflake, our deployment tooling, and more. When you ask DJ how a service is deployed, it doesn't recall a trained-in fact. It goes and looks, through the same APIs an engineer would use, at that moment. There is no knowledge base to build and no vector index to refresh. DJ is stateless enough that its pods can restart and nothing is lost, because its "memory" is literally Slack threads and pinned messages re-read into the context window.
This is what the industry now calls context engineering, a term that emerged in mid-2025 when people building agents realized the job was no longer writing clever prompts. The best definition I've seen: providing the right information and tools, in the right format, at the right time. That is the discipline we practice at JWX, whether it's DJ, our CLAUDE.md files and skills in Claude Code, or anything else we build.
Now the misconceptions, because they're persistent. DJ does not learn from your chats in real time. Uploading a document to an AI does not train it; the document enters the context for that request and then it's gone. And no, fine-tuning would not teach DJ our facts better. Research has shown that fine-tuning a model on new knowledge actually increases hallucination, and in one benchmark commercial fine-tuning APIs averaged 37% accuracy on new facts. Fine-tuning teaches behavior and format, not knowledge. Our runbooks change weekly. A fine-tune would be stale before it finished.
Finally, when DJ gives a wrong answer because a doc is outdated, the fix is to edit the doc. The very next question gets the right answer. No retraining, no deploy, no waiting. When DJ hallucinates, the fix is almost never "more training." It's better grounding, pointing it at a live tool call instead of letting it guess. When a new model is made available, we can switch models with a configuration change and lose nothing, because nothing was ever trained in.
Everything we put in front of a model is transient, inspectable, and reversible. So, when someone says "we trained DJ on our documentation", that's wrong and it is our biggest strength. Models may come and go, but DJ gets to benefit from the latest and greatest by simply changing which model applies to the context engine we have built. Knowing which side of the line you're on changes how you build, what you budget, and what you promise. We chose the context side deliberately, and DJ is the proof it works.
The most common question I get about DJ, our internal AI assistant at JWX, is some version of "so you trained it on our documentation?" It's a fair question. DJ answers questions about our deployments, our incidents, our docs, and our code like it's been here for years. It might surprise some but we have never trained anything. Not a model, not a fine-tune, we don’t even build an embedding index. Understanding why is probably the single most useful piece of AI literacy I can offer, because it explains how we use and develop with AI across the whole company.
First let me give you some terminology. Training is any process that durably modifies a model's weights. Context engineering is everything you do to a frozen model at inference time by controlling what enters its context window. That's it. The mental model I give everyone is that the weights are long-term knowledge frozen at deployment, and the context window is a scratchpad rebuilt on every single request. Training is like a student doing practice problems with an answer key, changing what they know. Inference is that same student taking the exam. They can read anything you put on the desk, but they leave the room knowing exactly what they knew walking in. DJ is the student and we hand a very well-organized desk.
I am not saying training doesn't matter. Model builds do an enormous amount of it to produce the Claude models DJ calls. Understanding the scaling differences matters though. Training a frontier model burns on the order of ten billion times the compute of answering one question. Nobody accidentally trains a model. Every answer DJ gives sits at the cheap end of that scale, running against weights that never change.
So how does DJ know anything about JWX? DJ is a harness, a loop of code wrapped around frozen Claude models running on AWS Bedrock. Around that loop sit roughly twenty-five live tool modules: GitLab, Datadog, JIRA, Buildkite, Sentry, Notion, Google Drive, Slack, Snowflake, our deployment tooling, and more. When you ask DJ how a service is deployed, it doesn't recall a trained-in fact. It goes and looks, through the same APIs an engineer would use, at that moment. There is no knowledge base to build and no vector index to refresh. DJ is stateless enough that its pods can restart and nothing is lost, because its "memory" is literally Slack threads and pinned messages re-read into the context window.
This is what the industry now calls context engineering, a term that emerged in mid-2025 when people building agents realized the job was no longer writing clever prompts. The best definition I've seen: providing the right information and tools, in the right format, at the right time. That is the discipline we practice at JWX, whether it's DJ, our CLAUDE.md files and skills in Claude Code, or anything else we build.
Now the misconceptions, because they're persistent. DJ does not learn from your chats in real time. Uploading a document to an AI does not train it; the document enters the context for that request and then it's gone. And no, fine-tuning would not teach DJ our facts better. Research has shown that fine-tuning a model on new knowledge actually increases hallucination, and in one benchmark commercial fine-tuning APIs averaged 37% accuracy on new facts. Fine-tuning teaches behavior and format, not knowledge. Our runbooks change weekly. A fine-tune would be stale before it finished.
Finally, when DJ gives a wrong answer because a doc is outdated, the fix is to edit the doc. The very next question gets the right answer. No retraining, no deploy, no waiting. When DJ hallucinates, the fix is almost never "more training." It's better grounding, pointing it at a live tool call instead of letting it guess. When a new model is made available, we can switch models with a configuration change and lose nothing, because nothing was ever trained in.
Everything we put in front of a model is transient, inspectable, and reversible. So, when someone says "we trained DJ on our documentation", that's wrong and it is our biggest strength. Models may come and go, but DJ gets to benefit from the latest and greatest by simply changing which model applies to the context engine we have built. Knowing which side of the line you're on changes how you build, what you budget, and what you promise. We chose the context side deliberately, and DJ is the proof it works.