By Ken Rona, Chief AI Officer, JWX
Priority one of JWX's AI mandate was accelerating software development. This post covers how that started with a hiring philosophy that ran against conventional wisdom, and how it produced a growing family of internally-built tools with names like DocBot, TestBot, and BugBot.
If you're deciding whether to build your own AI tooling or wait for the model providers to catch up, this is a real answer to that question, not a hypothetical one.
How we turned a hiring bet into 3x engineering output
In terms of execution, the first thing I did was propose a “Center of Excellence" (CoE) staffing model. My goal for the CoE was that they would build software, using the latest techniques, and then we could train the rest of the product and engineering organization in how to use AI in their own work. To start, I recruited a team of developers that were actively using AI in their work, demonstrating interest and commitment. I also hired three junior developers who were actively using AI, but were not expert software engineers. But they were very engaged and enthusiastic. This is a critical point: I would much rather hire junior developers who are all in on AI than a more senior developer who is reluctant to use the latest tools.
In terms of initial projects, I identified a series of small to medium sized internal efforts that would let us experiment with different software development processes, all AI-supported. The projects all had one thing in common: they were to automate tasks that software engineers don't enjoy. Here's what came out of it:
- DocBot: scans a code base and generates developer documentation, so the rest of the team can leverage it for harder work.
- TestBot: builds unit and integration tests.
- ReviewBot: does code review, now triggered automatically as part of our CI workflow.
- BugBot: handles bug-related software development.
- Gofer: writes whole features.
We also built a PM workflow that writes drafts of all of the documentation product managers need to communicate with engineering.
We have been pretty careful about rolling these tools out for the engineering staff to use; we want to make sure that they land cleanly and the teams using them are enthusiastic supporters. You don't get many chances to make a good impression and it's important that these tools are not treated as some throwaway internal CLI tooling that performs a function so specific, no one cares how easy it is to use. We treat every tool rollout as one would release software to a customer. We have training and support and, based on client feedback, we evolve our internal tooling.
As we were developing our internal tools, Claude Code (CC) came out with what they call slash commands that replicate some of the functionality of our internal tools, and we asked ourselves a question: should we continue to develop our internal tooling or let CC catch up? We decided to push ahead. We found that our tooling was easier to use and gave us a level of control that the slash commands did not. So, while there is a /code-review command in CC that developers are free to use, we tend to rely on ReviewBot instead.
We learned a lot in creating these tools. Building tools using agents isn't exactly like software development; it's a bit like onboarding a new staff member. They are going to get some stuff wrong and you need to coach it and put processes in place to ensure that they do the things you want; learning-as-you-go is part of the process you can't avoid. Some of what the team builds isn't directly useful. But over a fairly short period of time they learn how to use the AI tools to build, and then they become super-productive. In our case, an experienced dev team sees about 3x productivity compared to pre-AI tools usage.
Our current challenge is to manage costs, which is a good problem to have: it means the CoE's tools got adopted widely enough that spend became something to actively manage. As part of our cost management efforts, we built a tool that sits in the system tray and shows a user how much of their monthly AI budget they have left. We found that selecting the right model for a given task is critical to managing costs, but users didn't know how much anything was costing until they could see it. Managers now have a dashboard where they can see their team's budget, and each user can see both their burn rate and how much budget remains. Still early days, but we think this small tool is going to have a major impact on people's ability to manage their own spend.
The Bot family solved the productivity half of the equation. But partway through building these tools, one of them stopped staying in its lane. It started as a way for developers to search their own codebase. It ended up running half the company.
Key Takeaway: If you're weighing whether an internal AI tool can actually spread past its original team, that's exactly what happened next, and it's worth reading before you scope your own pilot too narrowly.
Next up: “The Trojan Horse" blog is coming up on the 6th October.