Most technical interviews still ask a candidate to write a function on a whiteboard. Adam Witanowski says that question is already obsolete, and what should replace it says everything about how AI is changing who gets hired.
In this episode of Building Tech Teams, James MacDonald sits down with Adam Witanowski, AI Architect at nib, where he has spent the past several years building the insurer’s AI engineering practice from the ground up. They get into why most enterprise AI projects underperform (nobody is measuring the right things), what “back pressure” means for AI-generated code, why Australian salaries can’t compete with the US for senior AI talent, and why the technical interview needs a full rebuild.
It’s a hands-on practitioner’s take on what is actually happening inside enterprise AI teams, not the conference-keynote version. They cover the real dollar gap between hiring an AI engineer in Australia versus the US, why “the whiteboard should come back” as an interview tool, how Adam built his own AI agent to interview for jobs on his behalf, and the testing harness his team uses to check whether AI-generated work is any good.
About the guest Adam Witanowski is an AI Architect at nib in Sydney, where he has built Prometheus, an integrated suite of agentic AI tools now running in production across the business. He works at the intersection of senior engineering and AI leadership, and has spoken publicly on enterprise AI engineering discipline, including harness design, testing and evals, and AI governance.
About the show Building Tech Teams is hosted by James MacDonald, Managing Director of NTP Talent, and produced by Day One®, the team that helped build Blackbird Ventures’ Wild Hearts. Sister shows include First Cheque, Oversubscribed and Pick My Brain.
Chapters
0:00 Cold open: the line that explains why most AI investments don't work ▶ 0:44 Who Adam Witanowski is, and why James wanted him on ▶ 2:06 Welcome, and building an AI SDLC from scratch inside a regulated insurer ▶ 5:12 What “back pressure” actually means, and why solo AI coding doesn't scale ▶ 10:52 Startups vs enterprises: re-engineer whole processes, not thin slices ▶ 13:24 Why most companies can't even measure the problem they're trying to solve ▶ 17:10 Software eats the world, and why that makes engineers apex predators ▶ 19:44 Build in-house or buy: the blind spots consulting companies won't tell you ▶ 23:57 Forward deployed engineers, and the rise of the hybrid product-engineer ▶ 30:43 Build, upskill or outsource: the renters, owners and caretakers test ▶ 34:18 What AI talent actually wants beyond a bigger title ▶ 38:30 The real dollar gap: $220K in Australia versus $500K in the US ▶ 46:37 What to actually pay your first AI hire ▶ 49:01 “What you don't measure, you can't change” ▶ 51:01 Redesigning the interview: show me your harness ▶ 56:45 Why the whiteboard should come back ▶ 58:38 Building a harness to test his harness ▶ 1:03:46 He built his own AI agent to interview for him ▶ 1:07:28 Why motivation and taste still beat any technical test ▶ 1:12:19 Apex predator, Jedi, stallion, unicorn: Adam's closing line ▶ 1:13:00 James's takeaway: it's a measurement problem, not a technology problem ▶ Transcript Search or select a timestamp to jump ▾ 0:00 Adam Witanowski: AI agents are good at generating code. They'll do that happily all day long. But is it good code for your environment, for your context? Back pressure is the only thing which is going to shape that. You know, we've taken a bunch of learners, learner drivers, and then given them an F1 car and said, drive around the city, have fun. No, you need to start to think about Jan's whole role. And that's what the startups are doing, and that's what enterprises need to do, is start to re-engineer whole processes, not thin slices. And until they do that, they're going to be really hamstringing themselves. And so I think, you know, the first thing that any organization that wants to go down this route of increasing their AI capability is they need to figure out how to measure, because what you don't measure, you can't change. 0:41 James MacDonald: I think that's the best line you've said today. That's Adam Winternowski. In his previous role, he was AI architect at NIB, where he helped build their AI delivery capability. It was 18 systems, a knowledge graph across 3,600 repos. It was AI agents shipping production to code inside a regulated health insurer. There's not many people in Australia that have done that. AI in production at scale, highly regulated. But more recently, and this was the reason I wanted him on the podcast, he's been on the job market and he's in the process with 9 different companies. And at the time of recording, he had 7 potential offers. That's what I wanted to quiz him on. What does a good interview look like for this senior AI talent? How do companies actually interview people that are more technical than them, further along their AI journey than them? As well as that, we discuss what back pressure is, why your agents are writing code that doesn't fit your business without it, why most companies see no ROI from AI, and why Adam says that's a measurement problem, not a technology problem, and what senior AI talent actually costs in Australia at the moment. Real numbers on the offers that he's been receiving. I interview AI engineers for a living, And I still learned a lot from Adam. If you lead a tech team, you're planning on your first serious AI hire, or looking to upskill your interview game looking for this senior AI talent, this is the episode to steal some tips from. I'm James McDonald. This is Building Tech Teams. Here's Adam. Welcome to another episode of Building Tech Teams. On today's episode, we have Adam Wuntanowski. Did I get that? 2:12 Adam Witanowski: Wuntanowski. I got it wrong. Wuntanowski is fine. 2:15 James MacDonald: I was close enough. Mate, most recently AI architect at NIB. You're one of the few people I know who's actually built AI in production in enterprise. Talk me through what that journey was like. 2:28 Adam Witanowski: Yeah, so that started probably a little over 12 months ago and I was brought in to build sort of the AI SDLC capability within a health insurer, NIB. And obviously a highly regulated environment, FSI. And FSI loves to talk about how hard it is to do things in their space because of the regulation. And 14 months ago, there really wasn't, and there still isn't in a lot of respects, the tools you need to be able to effectively build an AI SDLC, like really genuinely build it for a team of 300 engineers, Plus the tools just don't exist. So it was really looking at it from scratch and thinking, what is a— re-engineering what it looked like to build software at scale in enterprise or really anywhere. And so it started with dismantling the key components of the SDLC, came up with what are the— what are the— what's the appropriate back pressure that you would need to enable you know, AI agents to build effectively, uh, and the various tools that you would need throughout that full lifecycle, including cyber architecture, QA, actually building the thing, shipping the thing. And then it started to morph later on into a conversation of what does it look like to enable org slop, where you're going to have non-engineers building and shipping as well. So it was a fairly wide scope. And that's something that I think NIB has done well in the, in the past, is really think 3 or 4 steps ahead. You know, they were first on the cloud for insurers in Australia and really had quite a mature data space. So they were ready, really quite well set up to move into this sort of AI journey. So that started with 11 systems that ended up bloating out to about 18. each fairly large in their own right, giving everything from the ability to ship, which was the smallest piece, through to giving the AI agents a 360-degree view of the environment around them through, uh, through a, um, an application called Skynet, which ingested all 3,600 repos, created a knowledge graph and a vectorization and a, uh, and a code intelligence system that could be queried and giving, giving the AI agents a view of upstream and downstream impact of changes. So from the really complex down to the really quite basic, you had to think of it all the way through. 5:12 James MacDonald: And you mentioned back pressure before, and I know you've mentioned that online before. Talk to me about that, because I think it's come up in some of your interviews as well. But just talk to me about the back pressure part of what you just mentioned. 5:23 Adam Witanowski: Right. So back pressure comes out of fluid dynamics. So, you know, you go to the pump, fill your car with petrol, and it knows to stop because of the back pressure of the fluid. Engineering has a similar concept, and engineers would be familiar with this even if they're not familiar with the term. So this is things like testing harnesses, gold standards in data. And so, you know, you think things like Um, is the agent able to test it, test its own work? Is it able to, uh, get signal from CI which triggers, uh, which triggers a piece of rework within the system? Now most people when they build a harness or they use just like, you know, most companies are probably just getting GitHub Copilot if I'm honest, which is functionally useless, or going to Claude or Codex, which is way better. They're just prompting and then it's writing a bunch of things, and then they're shocked that it's not fit for purpose in their environment. AI agents are good at generating code. They'll do that happily all day long. But is it good code for your environment, for your context? Back pressure is the only thing which is going to shape that. And so this is about creating as much good steer as possible. The prompt that you put in is part of good steer, but then it's the rest of the context of your organization, how you deploy, the environment you're deploying into, its peer repos, upstream and downstream consumers. The, you know, is it taking too much bandwidth within your server environment? Like, it might run, but backpressure's the only thing which is gonna tell it whether it's actually fit for purpose in your context. And I think we need to be thinking about when we build these systems, how do we re-inject backpressure into the agent so it understands when it's failed or succeeded or needs to rework something. 7:20 James MacDonald: That's the difference between building software. Everyone can build now to an extent, but that's the difference between everyone can build, everyone should build, or putting software AI into enterprise. It's these things where we're going that next level down. It's not just, as you said before, creating an environment for people like myself to start to vibe code solutions or build things. I need to have wrappers around me. I need to have these testing environments set up. I need to have the right environment for me to be able to build. Is that a big challenge that you're seeing, I guess, for organizations to allow their employees to move forward, allow them to use these tools, but in a safe and secure environment? 7:59 Adam Witanowski: Yeah, so I think we've taken a bunch of learners, learner drivers, and then given them an F1 car and said, You know, just go for a couple of— drive around the city, have fun. Whereas I think, you know, if you look at taking F1 as an example, not that I'm any fan of F1, but that's a team race. It's a team sport. It's not a single driver. And so I think, you know, when you get to enterprise, you need to be thinking about these from a multiplayer standpoint, not just a single engineer working alone at his desk in his hoodie. You need to allow, say, cyber to inject cyber policies into that. engineer's coding harness. He shouldn't have to think about it like those, those sort of policies— legal, risk compliance, cyber, cloud, etc.— those shouldn't live in a document somewhere and then we hope an AI agent finds it and maybe potentially incorporates that into what you're building. Now that's an engineer. This is someone who knows better. This is someone who, you know, they, if they open up that code and do a code review, which is also not really sensible in this day and age. But they open up this code review, they know what to look for, but they shouldn't have to. Like, the agent should have forward pressure of knowing that this is the policy. We always ship on Lambdas, for example. That shouldn't be something which the engineer has to think about. That should just be intrinsic knowledge within a context, and that should be provided by a multiplayer system where cloud is injecting that into— so you as an engineer injecting that into your coding harness. So I think we need to start thinking about these things much more on a, uh, rather than an individual scale, and start thinking of it more as an enterprise scale. You know, we've spent the last 2 years teaching agents how to work collaboratively, uh, and they can work in swarms of thousands now. And we've stuck all of that in front of a single user and said, steer this, steer this whole ship on your own. And we've done nothing to think about the, the multiplayer, multiplayer aspect of that harness, which lets really— that's where enterprises are, an organism of multiple intelligences, multiple people working together in this sort of weird harmony. And we've kept that out of the, out of the coding harness. We need to find ways to inject that in. And so everything I was building at NIB was— not everything, probably not 80% of it, say— had multiplayer aspects to it, which allowed multiple teams to have opinions. And they didn't have to be there next to the engineer to inject those opinions. Those could come from various systems and various back pressure points throughout the— throughout the sort of— I call it a meta harness or harness harness. 10:52 James MacDonald: You mentioned the start, re-engineer. It's like taking what's existing, pulling it apart, re-engineering. There's a talk of AI-native startups building from the ground up with nothing existing, significantly easier to build it AI-native. Sure. Going into an environment that's currently built, working, highly regulated environment, significantly harder. If you were to go into a brand new company right now, and as simple as we can break this down, They're not using AI at all. Where do we start? How do we break this down? 11:23 Adam Witanowski: The advantage of startups is they don't come with a bunch of preconceived ideas. All right, so startups are able to start from zero and reimagine a process from the ground up. They're not treading on anyone's toes. They're not like— if they make a decision, Jan from accounting isn't going to come and throw hands in the air and say, you know, you're taking my job away. Enterprises really need to start thinking in the same way. And they probably always needed to be thinking in this way, but now just the speed of change is so, is so rapid, um, that unless they're, unless they're eating their own market, um, they're gonna, they're gonna find these startups are, um, uh, you know, are taking, are taking all, you know, their business away. And we're going to see that, I think, more and more over the next 5 years. I think the, they need to, enterprises need to start reimagining what the system looks like, not just injecting AI into a process, finding a, you know, human thing that we do, and then finding a way for AI to do that one sort of sliver of work. We need to think about the whole process as a unit and start to replace, you know, not just go, okay, well, how does Jan from accounting manage these— this set of spreadsheets? Let's stick an AI to manage those spreadsheets. No, you need to start to think about Jan's whole role. And that's what the startups are doing. And that's what enterprises need to do is start to re-engineer whole processes, not thin slices. And until they do that, they're going to be really hamstringing themselves and they're not going to see the sort of ROI that they want. They're going to start to see 5%, 10% or zero. 12:59 James MacDonald: There are significant challenges with this, right? 13:02 Adam Witanowski: Of course. 13:03 James MacDonald: Leaders potentially fighting against the board, fighting against the CEO, hate to do this. we're actually gonna have to invest a significant amount of money, right? Yeah. This is a big decision on are we changing the nature of our business? Any advice for companies to do that? Because a lot of the advice out there at the moment is people tackle it from a big perspective and it takes a while and they're not seeing any form of ROI. So the alternative view is take a process, throw some AI automation at it, actually see some ROI, and then take another process and iterate and just continue to chip away. 13:39 Adam Witanowski: Yeah, 13:39 James MacDonald: completely different approach to what you're mentioning here, which I agree with, but comes with a significantly bigger challenge. 13:45 Adam Witanowski: Yeah. So this is— I mean, I think this is what everyone's wrestling with. It's why they bring in consulting companies, because they struggle to see— you look at engineering, engineering, we know what engineers do. Engineers are measured by the ticket. which could be an hour, it could be half a day, a day, whatever. Call centers are the— is the other part of the, the organization that's being, you know, largely eaten by AI. Uh, and we know what they do by the minute because we have every recording. What we don't know is what accounting does, marketing does. We don't measure them. And so part of— big part of it is the measurement problem. We don't know how to measure other parts of the enterprise beyond engineering and call centers. And so that's why we you know, those are the— those are in the spotlight at the moment, engineering call centers, because they're, they're relatively easy to understand because we know how to measure them. We don't know how to measure all of the other parts. And so we bring in consulting companies and we do time and motion studies and, and, and all that kind of thing. Um, but you're, you're asking people who really don't know your business to come in and solve a problem, which is problematic. Um, they might understand abstractions, and so they might be able to come in and suggest this thin slice, this whole process. But I think you're going to get a redefining of a lot of industries where, you know, previously winners and losers might have been decided over 10, 15, 20, 30 years. The same compression and compaction that's happening in engineering and in AI, you know, where AI is being heavily applied, you're going to start to see those compactions happening for any player that throws down heavily in this space. And so winners aren't gonna be decided over 15 years, 20 years. They're gonna be decided over 15 months, 20 months. You're gonna see a completely different scoreboard for this if companies invest, right? If they don't, and this is the, you know, there was a study done or, you know, someone wrote a blog which became a book, The 1,000-Day Problem, where we have 1,000 days and we're currently at day 2. 600 or something. We have 1,000 days before, you know, AI kills us all, or, you know, there's an AI meltdown. I don't buy it. But the main saving grace of that is enterprise's relatively slow AI adoption, right? So there is a, there is a sense to which enterprise has always been relatively slow to adopt, you know, new technologies. And, you know, in the insurance space, you have a look around and Um, you got— I used to go to conferences in, uh, in insurance, and they're still talking about their cloud migration strategy. I mean, that was— that, that's very early 2000s, you know. You go a little bit further back and you got Marc Andreessen from Andreessen Horowitz saying software is going to eat the world, 1996. Software's going to eat the world. That's true now more than ever. Um, and so, you know, I think the more businesses are able to cannibalize themselves and really understand the processes that they have internally, which a lot of companies aren't doing, is, you know, having that sort of deep introspection. The more they can understand their processes, understand what actually adds value now, the more they're going to find value out of AI adoption. 17:10 James MacDonald: I think that what you mentioned right there as well, it's— and it's the roles like marketing and these other roles that aren't as process-driven. Do this, then do this, then do this. Breaking down that process is having that standard operating procedure because once you've got that, you can actually throw AI at it, right? Those ones that aren't as well documented, that's where a lot of companies struggle. The other part you mentioned just there, software eats the world, the SaaS apocalypse, everyone's talking about it, 17:36 Adam Witanowski: right? 17:36 James MacDonald: If software is going to eat the world and we're continuing down that route, is it all companies become software companies and all companies are building their own software or Or is it there will be a rise again of more AI-native SaaS companies? 17:51 Adam Witanowski: Right. So there's a couple of things there. Is software going to eat the world? Yes. Right. And we've been saying for— not we as in engineers, we as in people in enterprises, business, business owners, etc. Every company is a software company. And, you know, Mark from NIB, Mark Fitzy, I think he truly believed that it was a software company that sold insurance. And I think the companies that have done that well have been able to operate at lower cost and, you know, higher margin, etc., over time. And so, yeah, I think ultimately software does eat the world and that makes engineers apex predators, you know, in this space at the moment. You know, it's sort of weird to think about because I know engineers, they don't look like apex predators, but Um, you know, you're not going to have, you're not going to have, uh, you know, accounting eating their own, um, eating their own portion of the business because it's still hard. Like, building with AI is still hard. You still need engineering discipline. Otherwise you're going to be escaping all sorts of data out into the world that shouldn't be escaping, etc. Is software going to eat the world? Yeah. Is the SaaS-pocalypse a thing? Yeah. There's going to be some companies which don't survive. there's going to be some practices that we used to use which we don't need anymore, um, because you can— because a new competitor comes along that does it faster, cheaper, better, or some companies are going to build those tools in-house and just don't need, don't need the SaaS anymore. However, that, that creates a high maintenance cost and something which isn't your core business. 19:35 James MacDonald: It's not your core business, but every business is a software business, 19:39 Adam Witanowski: right? 19:40 James MacDonald: Does that mean every business starts to build out internal engineering capability? 19:44 Adam Witanowski: On, on the parts that add value outside. So this is on the parts that add value, right? So there's not a lot of, not a lot of great, as an example, just off-the-shelf fraud engines. And so that's a core capability of an insurer or an FSI company. And so they're probably going to need to build out their own fraud detection mechanisms that fit their system and are able to be understood, tweaked, dials moved. Now, do they need to build their own ticket management system? Probably not, right? Do they need— there's going to be some things which are just generic. And those should be bought off the shelf. And there's going to be things which are unique to your business or unique to your industry, and those should be built, not bought. So, and that can happen faster than ever now and better than ever now and create better customer experiences now. And that becomes— I mean, that is the customer experience that you're able to create has always been the hedge, right? Health insurance, there's 1,000 of them. There's, you know, 20 banks, there's whatever. But the experience that you get, the functionality that they provide, that's the unique offering. And price is also a way to compete. But if you can do it cheaper and you can create better customer experiences and AI enables that, which it does, and you know, you win. That's just at the end of the day, you win. And so I think that's where it becomes everyone's a software company. They need to start thinking at the speed of a software company, not at the speed of an insurance company or a bank, bank, you know, that have typically operated over 5-year timelines. Those don't exist anymore. 21:39 James MacDonald: No. You mentioned earlier about consultancy companies coming in. We just talked about building internal dev capability, engineering capability. The other one is if you've already got that engineering capability, upskilling the team internally. They're the 3 options I look at. They're the 3 options I talk with other companies about that are looking to start some form of AI journey. Talk to me about, is there a one-size-fits-all? Is there, in a particular situation, I would bring a consultancy in? In this particular situation, I've got to build an AI capability because that's not cheap. 22:10 Adam Witanowski: Every company's got blind spots. Every person has blind spots. Every company has blind spots. You have a certain way of operating in companies. And you just think this is the way we do it, this is the way it's done. So having an extra set of eyes come in and reimagine what that could look like is all I think always helpful, as long as you're willing to listen to it, right? And not all consulting companies are equal. So some consulting companies are going to come in, they're going to sell you, you know, unicorn ground-up unicorn horn magic dust that you're never going to be able to build out yourself, and they're probably not capable of building out either if you're really honest, because the talent pool is actually surprisingly small. And so they may come in with all sorts of promises that they're not able to fulfill on internally. 22:55 James MacDonald: Well, you have a look at the big rise of the forward-deployed engineer. Oh yeah. It's not a big percentage of them are working for consultancies. Yeah. Consultancies are trying to hire them when they tried to hire so many of these same type of people that don't exist. 23:12 Adam Witanowski: Right. 23:13 James MacDonald: They don't all exist, right? At the same time, we're going to get a bunch of underskilled people being presented as forward-deployed engineers. 23:19 Adam Witanowski: Oh yeah, for sure. And, you know, the forward-deployed engineer obviously comes out of Palantir. And this is it. This is a handful of unicorns that most engineers don't like spending a lot of time with people. Most engineers like solving problems and working, really working through those problems. Um, they love mastering their craft and honing skill and being opinionated around which database is best and all this sort of stuff, right? So, um, engineers are, um, are a sort of beautiful and unique stallion, uh, that, you know, and, uh, they, they, they don't— but what happened with AI was over the last probably 18 months, 2 years, it got everyone thinking, oh, I, you know, Our engineers could be product owners, our engineers could be engineers, our engineers could talk to customers, our engineers like— and we started to blur about 5 different roles into one. We imagined that we could because AI enabled certain things or gave time back, etc. I think what we're realizing, or hopefully what we're realizing, is there's some things that a product person does that an engineer is just not good at. And so we're going to end up, as you say, with a bunch of forward-deploy engineers that actually aren't fit for purpose. There are, there are a group that, that totally exists that are, you know, extroverted and quite happy to go and talk to customers and they're also able to build things. But that's not every engineer that I've met. 24:46 James MacDonald: My opinion is actually, yeah, quite the opposite. I think you're easier to turn a traditional business analyst or product manager, product owner, upskill them from an engineering capacity and turn them into that role as opposed to the vast majority of engineers who, as you said, really like to get technical. Those people definitely have their limitations as well. The ex-PM, the ex-BA have their limitations, but I feel like the opportunity for those people to now upskill technically has changed significantly. 25:16 Adam Witanowski: I think that's true. I think the hard wall they hit is as soon as they want to ship something. 25:22 James MacDonald: Correct. But I feel like that initial scope, where can we actually throw AI at this problem, is probably better suited to that type of role. backed up by proper engineering. 25:34 Adam Witanowski: Yeah, product people with product thinking. I mean, the people, the people who are going to win over the next couple of years are people who can think technically and can think with a product human, human-centric mindset or a product mindset where they can imagine tool applications. They can imagine how, how does, how does this solve a customer problem? They can see pain and understand the pain empathetically for an organization, and then quickly come up with what's the salve for that pain. And that's been the tension between engineers and product people, and it's a healthy tension. And it's definitely something, if you're talking just up to POC stage or solution engineering, or solution imagining. Yeah, definitely taking a product or a BA into that space and building the first pass, it's something they can definitely do now. I mean, that role significantly, 26:39 James MacDonald: probably 26:40 Adam Witanowski: a 13-year-old shipping software. Yeah. 26:42 James MacDonald: All right, I'm going to go down that route real quick. It was a tangent, but I like tangents. I've got 8, 10, and 13-year-old children. We're looking at the future of what they want to be when they grow up. Obviously life's going to change. Who knows? Just enjoy your life. You're a kid at the moment. I know you've spoken about your child is actually building at the moment and actually building some, you know, pretty significant things. Yeah. Now talk to me about that journey. Like, was that something that they just showed an interest in? Obviously seeing you and what you're interested in and just it's evolved from there. Is it something you'd recommend for others? 27:17 Adam Witanowski: Look, We, you know, we got, we got 2 kids, my wife and I, a boy, 13, a girl, 12. My daughter has basically zero interest in, in AI at all. The only interest she's ever shown in software engineering is they had a mufti day to dress as a future profession, and she wanted to wear a hoodie and jeans, so she went as a software engineer. 27:38 James MacDonald: I like it. 27:39 Adam Witanowski: She will not be a software engineer. So maybe she will blow me away. My son, however, He's homeschooled. He's very gifted. He likes solving problems and tinkering. I think he wants to work in space and engineering or something along those lines in the future. And he just sees this as an opportunity to make some money. So, he's picked up the tools and started playing around, and that's led him to do talks and other things. 28:09 James MacDonald: Real quick one then, if you would give advice for other parents, children is showing some form of interest, what are the guardrails you put around them? What are the tools that you provide them with, you know, to get started? 28:20 Adam Witanowski: So for my son, who, you know, he showed an interest, wanted to build some stuff, he, I mean, started off with he wanted to build some things to help him win in video games. So like an auto shooter, for example, that was able to identify the identify opponents and auto-target their heads and shoot them. Uh, you know, great little starter project. Um, and so it was just from there I just walked him through how I would build, walked him through some best practice around things like, um, you know, spec-driven engineering, uh, test-driven development, and a few concepts that started to balloon out too. Because then he built it, he built a thing. it didn't work that well. Like, it was sloppy and didn't work. Went through and did a little bit more of the background, so getting into some architectural principles of engineering, really the same thing you would do for a junior. Which, I mean, juniors now, it probably looks more like an apprenticeship, you know, where you're walking them through some of the things which, which I think unis have historically done and are typically now skipping perhaps a little bit more. some of that real foundational engineering thinking. 29:36 James MacDonald: I actually think that's the most important part of the university degree for software engineers that I've met is the understanding of the foundations. Yeah, to start with, because now with the tooling, it's like if you've got that, then the juniors with that solid understanding of fundamentals with the new tooling essentially become mids, right, when they come in as opposed to juniors. 29:57 Adam Witanowski: Yeah. 29:58 James MacDonald: People without that self-taught don't have those fundamentals are, as you said, more apprentices. 30:04 Adam Witanowski: It's vibey. They just— it's 100% vibes versus the, you know, like someone who is self-taught, they may be able to ship code really quickly. And look, I know, I know some self-taught developers that are fantastic, but they're fantastic because they've taken the time to actually understand fundamentals. They learned them in reverse, the hard way, through scar tissue as opposed to through education. And scar tissue is always going to be the best way to learn anyway. But if you're not willing to take the time to build the scar tissue, then you're, you're, you're going to struggle to ship anything meaningful that endures. 30:43 James MacDonald: Go back to the last question for a real quick one. Companies sitting there at the moment wants to start their AI journey, upskill internally, hire externally, bring them in, the talent, or use an agency? Do you have an answer? Is there a best practice? 30:58 Adam Witanowski: I think building internal capabilities is always going to be the win. Yeah, like, you know, for sure, hire in some people who are good at— good at— good with AI ahead of your team, but use them as a resource to train. I think if you can upskill your team, it's a gift to you and it's a gift to them. You know, the, the future in this space is people who know how to use AI and leverage it, um, for, uh, for volume. Like, we don't— I haven't written a line of code in probably 11 months, 12 months. Um, and more and more, that's going to be the common experience. Uh, and, and so you've got people in organizations all the way from AI can't possibly do my job through to 400, 500x, and then this sort of mushy middle. I think the more you get those 400, 500x to be able to decompose, train— and this is something that engineers have historically done really well— brown bag sessions, you know, taking time out to peer reviews, PRs, and pair programming. We have a— engineers have a long history of training and coaching one another. The more we can leverage that within companies and upskill everyone at somewhat of an equal bar or equal bench, anyone who doesn't want to use it will opt out and they'll find somewhere else that, you know, Luddite-ish or something. But raise the tide so all the boats rise together, I think, is a really important idea. And these people come in with just a, just a huge amount of legacy knowledge that if you can if you can codify that, and then eventually they get an offer and walk out the door, I think, you know, A, you make your people more poachable, which makes them actually, I think, counterintuitively more loyal because they've been invested in. They know they get the, the opportunity to experiment. Um, uh, you, you will also get this, this enduring benefit of they know the industry, they know your systems, they know how to change them, modify them, and uplift them. Bringing people in We brought in contracting consulting companies before, body shopped them in. Yeah. Now there might be, you know, there are definitely situations where consulting companies coming in, they solve genuine problems, they do it and they do it in an enduring way. But I'd say most engagements from most consulting companies, it's a, it's a matter of them coming in. They don't— I think of it like renters, owners, and caretakers. they come in as renters. Renters aren't going to uplift the kitchen. They're not going to, you know, tear out the tile. They're not going to look for the termites. They're just going to, you know, slowly trash the place over time. Um, owners are going to genuinely care for it. They're going to make sure that its value increases over time. That's— I mean, there's a massive difference between bringing in fly, you know, fly-by-night teams that come in once, quick hit, quick spend, Versus the people who have to live with it long-term. So yeah, I mean, you got these, they're 3 categories. I would rather invest in owners than renters or at least caretakers who are going to lovingly nurture what exists. 34:13 James MacDonald: So the recommendation is to hire. 34:16 Adam Witanowski: Oh yeah, you're going to love that, NTP. 34:19 James MacDonald: Well, there's some challenges on my side. There's some challenges for internal recruitment, right? I think I'm reasonably technical. I've interviewed a bunch of AI engineers at varying levels of quality. There's a real challenge for someone like myself or internal talent teams to actually know what good looks like. Mm-hmm. I think the standard software engineering interview has changed. I don't think what used to work for traditional software engineering roles work for the new AI engineers, or we're talking the senior, the principals that are going to come in, as you said, to change a company's trajectory. You mentioned make sure it's somebody who's good that comes in. There's a lot of people that are talking about good. They've built a POC. They've never put something into production. I know recently you've interviewed for 9 different companies. 35:07 Adam Witanowski: Yeah. 35:08 James MacDonald: I want to break down what a good interview practice looks like for someone at your level, or even if you want to expand it a little bit down. What does good look like? Because I imagine you've seen the best and the worst of it. If you're going to take all the knowledge from your 9 interviews you've just been through, start it from the start. What does a good ad look like? Reach out— what does that look like? How do I engage somebody like you? And I want to break it down step by step. 35:39 Adam Witanowski: Yeah, yeah. So I think top talent in AI want to know that they're playing with AI, number one. They're probably going to want something which is a title which reflects that. Um, titles at the moment are all over the place, like they're just a red-hot mess. And, uh, and title used to be representative of comp in some way. That doesn't seem to be true anymore. Um, you know, you've got FDE roles that range from basically what you would have paid a senior, uh, previously all the way up to what would have been considered ridiculous. you know, 2 years ago. Um, so there's, there's not an indicator anymore. I think the— in terms of, yeah, people, people who are in this space, I think they recognize— I mean, they've been in a company before. They recognize that they're probably in the, in the 5% of their organization, um, and that they're, they're relatively rare. Um, and so they know they're probably worth money. Rare things are worth money. Um, and so, you know, an understanding of, um, well, for someone, for someone who's relatively new, so they've, they know how to drive Claude or Codex or Cursor, um, they want to know that there's opportunity to continue to flex that skill. There's going to be opportunities to learn and opportunities around people who are, uh, be around people who are better than them or more advanced than them in AI. For someone who's been around a little bit longer and is, uh, you know, they're not just using Claude the way it's built and, and they're starting to build their own harness and that sort of thing, they want to know that there's some room and flexibility to experiment and make a mark, a dent within there. That they're still engineers, they want to solve problems, and they want to understand the problems are meaty. They're not going to be shoehorned into one particular thing because they can probably solve that particular thing in a couple of months now as opposed to a couple of years. Um, and so, you know, and the people who are really pushing against the edges of this, of this stuff, um, they don't probably fit all that well into, um, sitting with one team for a really long time because their, their, their mindset is fringe, right? They push against the edge. And if you put them into just one little box all the time, and that's where they live for 38 hours a week, then they're probably not going to have that much fun. 37:59 James MacDonald: So I'm saying hearing the vision, sell the vision, obviously the tooling, making sure that they're going to have the full access to actually continue to grow and lean into this and then actually fixing some big problems, right? And not just being put in a box. But, but mention one other third, then we're going to go there right now. Comp. If we're talking about comp, I'm going to have to pay you a significant amount of money, right? Putting you in a box doesn't make sense. 38:24 Adam Witanowski: That's right. 38:24 James MacDonald: I want to throw you at the biggest problems across the company and have you fix them and continue to chip away at them, right? Comp in Australia. 9 companies you've interviewed with, some of them international, some of them Australian. Where does comp currently sit for you? Is it— can Australia actually compete for talent like yourself? 38:42 Adam Witanowski: Yeah, so I've applied, or I've interviewed for and applied for half the roles I didn't apply for, right? So there were people reaching out. The other half that I did, And the, the comp and the ones that were low self-filtered, right? So these aren't included in that 9 because it's one of the first questions I asked because the roles make no sense. So I'm like, what's the comp? And because that's an indicator of seniority and effort now, apparently. So, so yeah, the roles, roles here versus here, very senior, more senior roles are the ones that I was looking for, things like distinguished engineer, you know, big grand title. Feels like I should wear a top hat. That was paying less than a— this— an engineer role, not even a senior role in a startup in the US. And the startup in the US was offering, you know, $400,000, $450,000 to $500,000 base 39:47 James MacDonald: USD. 39:47 Adam Witanowski: That's converted to Australian. And there was other ones that offered more, obviously. And but there was a lot of that was stock and, and that sort of thing. So, you know, I think offers around the $450,000 to $500,000 USD mark were, you know, in the mix, which for me just didn't move the needle for triggering a move to the US and moving my family and all this sort of thing. Because you can get, you can get those, that kind of money here, but it's a much more senior role, very different role, probably less close to the tools. And so that's really an appetite. How close do you want to be to the tools? Those same roles here are probably paying around $200,000, $220,000-ish. So $220,000 versus $500,000, which one are you going to choose? Particularly if you're, you know, I'm married, got kids, got a life here. If you're an engineer in their, in their 30s, not married or married with no kids, I mean, why would you? Why would, why would you stay for $220,000? I mean, that makes no sense at all. That's working against yourself. Um, I, I've found salaries here to be discounted probably between 60% and, uh, and, uh, 200% to 300% depending on the role. Like, there was roles, uh, with the same title that I had previously at NIB that were paying well over $1 million and into the $2 million space. Um, and I don't know, those companies 41:19 James MacDonald: were based in the US? 41:19 Adam Witanowski: They're based in the US. And those— I don't think those are, you know, all that common, um, but they, they are, uh, they are around. And, and I didn't see any AI roles for less than, um, less than double of what was being offered here. 41:35 James MacDonald: So for Australia to keep talent like yourself in Australia, to build that talent here, What do we need to do to compete? Obviously you mentioned the vision, you mentioned the tools and all that, but from a pure comp perspective, where do companies need to be to be having the conversations? Are we talking minimum $300s? Are we talking higher than that? 41:57 Adam Witanowski: It's going to depend on the role. Obviously it's got to be reflective of the value that the person brings. And I think in the US, they're anticipating at that pay grade, they're anticipating that you bring a lot of value. But typically they're also solving more interesting problems. Like Australia, there's a couple of drivers. So one is, um, you know, we in America's got big tech, um, companies like Google and, uh, and, you know, DeepMind, Anthropic, OpenAI, etc., which are paying just ridiculous salaries, like just, you know, astronomical. And, um, and they do that to attract the best talent, and they're in a talent war with one another because they're in an arms race with one another and with China. Right. So of course they're paying huge amounts of money, $10 million, $20 million, billions, you know, to acquire really top lab talent. And that's, that's one thing that puts upward pressure just generally across the market. Right. Because if you've got these companies which are actually making things at a large scale to roll out to billions of people, that drives salaries up. We don't make stuff here, not like that. So Google here is largely sales, right? So Google doesn't— we don't have a Google driving salaries up. On the other end, you've got startups. Startup funding here is typically very low as compared to the US by probably a factor of 10x or larger. And so they can't— they just don't have the money to offer the same sort of ridiculous salaries that are being offered. And again, that's an arms race. Like you've got 10 different, uh, voice companies competing there for, for talent that have worked in, um, you know, voice models, uh, whereas here you might have one and that one is funded at a couple of million dollars versus their 20 or 200. So they're often able to offer way more money. That drives salary up because now you've got 2 competes for salary. One is well-funded startups, the other is Um, is, uh, the hyperscalers that are competing in an arms race. Um, now that means that if any, like, Bank of America wants to get talent, they're competing against the drain out of 2 spaces. We're competing really against— we've got enterprises, um, which are— we've never had the sort of inflation, um, of salaries that, uh, that the US got anyway over the last 10 years. Um, and a lot of enterprises will still complain that engineers are too expensive, um, because they don't see themselves as a software company in competition with, you know, whoever the hell is going to come and eat their lunch. So we don't have the startup pressure, we don't have the hyperscaler pressure, and we've got enterprises that think engineers are too expensive. To fix it, I mean, salary is one thing when you've got low investment. Um, and Australia is, I would say, risk-averse in terms of its investment, uh, in startups. And so you don't get the big swings and the super exciting problems to solve typically. You might get one or two of those, but typically they're probably going to be something derivative of another successful startup in America released for an Australian context. And so engineers— an engineer sitting there He's a, you know, a 5xer, a 10xer, um, using AI, knows he's a, knows he's a bit of a unicorn in his own context. Um, and he looks around the market and he sees, I can go into sales, which is functionally what an FDE role is, right? I can go into, uh, sales and maybe make a little bit more, uh, maybe, you know, bump up my salary by 50 grand. Um, or I can go to a startup and take a massive haircut. And work on a problem that's really not all that compelling. Or I can go to a brand name like Atlassian whose stock comp is going down. I can go to Google and work again in functionally sales. I can go to Canva and get shoehorned into one thin slice of a product. Like, so what are their choices? They're really— it's going to be a, it's going to be a lateral move into something which pays slightly more. Or I'll go to the US, I'll go to Europe, I'll go somewhere else, which is going to double my comp. 46:14 James MacDonald: Are you seeing many of those roles in Australia fully remote? 46:19 Adam Witanowski: From the US to here? No, they— all the roles that I, that I got offers for and there might be one or two, but they're typically, you know, more 4 or 5 days in office. Yeah, they want you there, they want you on the ground, they want you with the team. 46:34 James MacDonald: Especially at that salary. Yeah. So if I'm a company at the moment and I want to hire my first AI architect, senior AI engineer, somebody at your level, what do I need to pay to make things interesting? Where's my, where's my starting point? 46:50 Adam Witanowski: Yeah, I mean, you're probably looking at the, you're probably looking at the $300 to $500 range, and that might feel ridiculous for a lot of companies. But, you know, what do you— 47:07 James MacDonald: It comes down to the mentality though, right? If it comes down— if your mentality is it's a, it's a same as I'm looking at, same as a normal software engineer, it's expensive. If you have a look at it, we're going to re-architect our solution, we're going to really aggressively lean into AI and change the nature of our business, provide better customer service like you said before, take market share from competitors, then it's not a ridiculous number. 47:31 Adam Witanowski: For most enterprises, if you know, even, even the most modest numbers. So, you know, they say AI 15% uplift is, you know, sort of a range. You've got individual engineers that are capable of, of, and, you know, even at NIB, I saw this, my own experience, other engineers I talked to were seeing way higher than this, but a 200% increase in productivity. is relatively easy to accomplish on a thin slice of their week. Now, end to end, if you want to see that, you need to engineer the Meta Harness. And otherwise your productivity, you know, they'll have engineers typically historically have spent about 16% of their time actually writing code. That's the bit that Claude Code replaces, is that 16%. Now, if you want to see that flow end to end, you need more infrastructure, more, more thinking, and an AI architect to help you actually map that out. And, you know, if you don't, you're going to see between a 0% and 40% increase in productivity. If you do, you can translate almost 100% of that into throughput now. So if you've got that, if you've got even a 1% uplift, if you hire this person and they bring a 1% uplift in your company, I mean, they've paid for their salary multiple times over. 48:57 James MacDonald: Do you think the big challenge at the moment is most companies aren't seeing that ROI on AI? 49:02 Adam Witanowski: I think most companies aren't measuring. They don't know how to measure it. And, you know, we've had DORA stats, for example, DORA metrics in engineering for years. I would say there's probably only 5 to 10% of companies that actually look at that. And, you know, I think this is why Atlassian bought DX developer experience platform. Um, which helps measure both engineer sentiment but also, uh, their engineering, engineering capability within an organization. And it, you know, plugs pretty well into Jira, and Jira does a pretty good job of pulling out a lot of stats. You probably need to look wider than just Jira, but, um, you know, I think most companies aren't even measuring it, and then they're claiming, well, we see no benefit. Um, and I think that's probably because they're looking at the wrong things. I think they've done a real— we've done a relatively good job at measuring how much a token costs, and we've done a relatively poor job of understanding what value that token has added. And so I think, you know, the first thing that any organization that wants to go down this route of increasing their AI capability is they need to figure out how to measure, because what you don't measure, you can't change. 50:11 James MacDonald: I think that's the best line you've said today. I genuinely do, because I think, I think that changes the mentality, because if you can actually see the measurement, you can actually see the uplift. It changes the mentality about what you might pay for your next hires and all the rest of it. 50:24 Adam Witanowski: Well, you can also, you can also only make change at the level of constraint. So if you don't, number one, you don't understand what the company, what, what portions of a company, of your company does, you don't actually understand what the constraints are. So you can make a change that makes no difference. And then the other side of it is if you don't measure the productivity, you know, that you currently have, you can't measure the delta of what the spend has given you. 50:48 James MacDonald: Yeah. 50:48 Adam Witanowski: So I mean, yeah, constraints and measurement are the 2 key things to do before anything happens. 50:54 James MacDonald: All right, I'm going back to this interview. How do I get this person in the door? We've got the comp sorted. Let's put that to the side. You come in for your first interview. How are you designing that if you're sitting in my seat? If you're taking all your best learnings over these 9 different companies and what you've done previously when you've hired into your team, What does it look like when I get you in the door for the first interview? 51:18 Adam Witanowski: There's some things which are always going to be true. You don't want to hire impressive jerks, right? So you want to make sure that people are a good culture fit. They're there. And you have to recognize, depending on where this person is in terms of their level of stallionhood, you know, or rareness, you— they're interviewing you as much as you're interviewing them. They have 51:42 James MacDonald: I agree. 51:43 Adam Witanowski: Right. And so there, I think, go back 5 years, it was much more the upper hand was with the organization that was hiring. I think there's, there's that shifted for a certain pool of talent. The power is definitely in their hands at the moment. And, you know, that's why, you know, I've got, like I said, I've got 7 offers. I've got another one that'll, that'll come this week. And most of them are willing to wait. And one of them even said, we need you more than you need us, right? So the power dynamic has shifted a little bit. So they're interviewing you as much as you're interviewing them. And the interview processes that I've had have been everything from, we've heard about your reputation and we just want you, here's an offer, meet some people so you can be comfortable with, you know, us as an organization. And then here's an offer all the way through to here's 9 interviews of various things. Some have been, um, you know, old-school coding interviews and, uh, write, write a function which does this. And I'm like, I haven't written a function in 12 months, man. Um, so I think, you know, a much more compelling thing, particularly at that level, uh, if you're hiring high-capability engineers, they're not building— they're not writing code anymore, they're building harnesses. And I, I think if you just sit down with them and say, show me how you use AI to write code— don't, you don't have to build anything, just show me your, show me your harness— and some of them are going to open up a Claude window and they're going to go, and here's how I prompt and here's how I give it context. And great, they're, you know, they're a level 1, and, you know, uh, AI engineer, and you get a good idea of what their throughput is going to be. Some of them, you're gonna, they're gonna open up their monitor and they're going to have 15 Clouds open, still running. They've got cloud handoff. You're hearing their phone ding. Okay, this person's more like, you know, level 3. Some of them are going to be showing you a dark factory producing software while they sleep, right? And those are very different AI engineers through that journey. 53:51 James MacDonald: Real quick one for a non-technical internal TA team or a recruiter who's trying their best to stay up with it. I've asked you to come in and show me how you do it. For the good engineers, they're going to be that couple of levels above. What are the things you look for for somebody that's good versus somebody who's maybe very thin in their expertise but can look flashy from the outside? 54:17 Adam Witanowski: Yeah. 54:18 James MacDonald: What are the questions you ask? 54:19 Adam Witanowski: I'm going to be asking them things like, have, have you ever shipped to production and what did that look look like. Because, you know, building— I, I have 1,000 apps. My wife's a teacher and she says, I want to go on this excursion to the art gallery, um, and I want a thing which does this with AI, um, and I'll spin that out. And she'll ask me that at, you know, 7 o'clock in the morning and it'll be live by 8:30, right? Um, and I could never sell that as a product. because it would just be, uh, you know, for a very small group of people, easy to lock down, easy to put in a server, all those things. Um, now if I— if that has to ship to, you know, 100,000 users in a regulated environment, that's a completely different animal. Uh, in fact, to the point which I would say, um, a month spent building an AI-centric product in an enterprise, a month spent building probably means 6 months deploying it, right? It's a, it's a, it's a factor of magnitude higher. Now that still, you're probably still saving 2 years, right? Because it's going to be more complete, more feature-rich, and all that sort of thing. Um, but it's a, it's a much slower, uh, process actually productionizing something. So walk me through how you would productionize this. What hurdles are you going to hit? What are the, uh, you know, how did you, um, you know, uh, how did— and, and I would look for standard language that an engineer would otherwise, uh, otherwise use. Things like, you know, I use test-driven development, look for back pressure markers, you know. So how do you ensure that you have test coverage? How do you ensure that you, um, uh, give your agent the ability to test what it's built? You know, all these sort of standard things that you would do as an engineer anyway. Are they using those practices? Or are they just throwing slop out into the world? And, and look, there's nothing wrong with a little bit of slop here and there. The agent can always rewrite it. And, uh, and it's a— it's where everyone starts, you know, build a— just build a sloppy, awful thing and watch it fail in production. Um, but, uh, but if you're really wanting— if you're spending that real money, you're gonna— you're gonna want to ask them some properly engineering questions and, and probably look at more less at the code they write and more around architecture and engineering practice. 56:43 James MacDonald: Agreed. Do you think that return of the— you're actually coming in with your laptop is going to be a continued trend? Or they can be like— it's like your old whiteboard discussions, right? Right. You're coming in, you're talking about theoretical problem. How would you go about this? And you're looking for their mentality around their architecture, around their problem solving, around 57:04 Adam Witanowski: I think the whiteboard should come back. 57:06 James MacDonald: Yeah. 57:06 Adam Witanowski: Yeah, 100%. I think that's because you're really looking at the distinguishing marker. Like, you go and get Claude code and I get Claude code and we sit down and we, you know, and we're going to build a calculator app. We're probably going to end up with something which is 90% similar because of model weights, right? Model weights know how to build a calculator. And so we're going to end up with the same code. Now that's testing whether Claude is any good, not test— and our subscription is worth it, not whether you're a good engineer versus me. So I think getting through how do you think about problems, what questions do you ask, how would you architect this? Okay, so now let's build a calculator that 10 people can use simultaneously, or it can, you know, draw. It knows when you're writing dirty words in numbers, uh, when you— so when you hold it upside down, you know, and it can do something interesting with that, you know. So start to think through now, um, and now design this feature that's over and above. Because I think that's— that comes down to really the distinguishing mark of good engineering now is it comes down to taste. It's not— it's not just can this person write code? Does this have— does this person have taste? around what's good code, because I can manufacture— I can build a dark factory harness which will just produce hundreds of apps. Are any of them any good? And that's, that's a taste issue, not a can this person generate code issue. 58:38 James MacDonald: So you've come in for first round interview. We've done, let's call it the, the behavioral cultural conversation to start with, making sure you're the right cultural fit. We've got a technical conversation which It's similar to the old whiteboarding session. It's taking somebody through your actual code. You've mentioned before online around harnesses and coaching people through the harnesses. Talk to me about that. What does that look like? What does that question look like? What does a great answer look like? What are you looking for? 59:05 Adam Witanowski: Yeah, I mean, so it depends. So if you're talking about their coding harness, uh, because everything, everything, um, is, is somewhat a harness now, but In terms of their coding harness, uh, yeah, I think it's a— Claude code is a harness, Cursor is a harness. Um, so I think we mentioned this before, but do they just use it vanilla, or do they run a headless dark factory that's able to run, you know, loop engineering? Or are they thinking more exotically like graph-based harnesses? And so, you know, there's, there's various— and that's sort of the graduation, right, is you've got basic vanilla Claude, then you've got, you know, they've got a, they've got a harness which has some skills and plugins and MCPs attached. Then they're maybe they've moved on to headless loop. I'm 60:03 James MacDonald: somewhere in between 2 and 3 at the moment. 60:06 Adam Witanowski: You are? Yeah. Yeah. And so at NIB, I built a tool which was able to measure the complexity of someone's harness, right? So it was built for a particular context at a particular time, but it was able to— it had a 10-point grading system and then suggested, here's the next steps you would need to take to graduate to the next phase. 60:27 James MacDonald: Do you think we'll see a product like that be successful in market? Yeah. Or do you think the technology is just growing so quick that you'd need to continue to update that product? 60:35 Adam Witanowski: I think there's definitely— I mean, I've started more and more building in parallel. Anything that I build, I also build a testing harness for it now. And so my— even my own, because, because it's moving so fast, you need to be able to answer, is this better? Because now it's doing things so quickly. And this is going back to just a human in the loop as a concept. We take a context-poor human and put them in front of context-rich agents and, and say, tell me if it's good. Which is a stupid idea, um, because they're just never going to be able to answer that question in any meaningful time frame. Um, so we need to be building not just harnesses which, you know, or agents and agent harnesses which do things, but testing harnesses which can tell us if what it did was any good. And so, um, yeah, I mean, I, I've got 3, uh, testing harnesses that I'm building at the moment, one over Uh, my own coding harness, which I'm, you know, building a, uh, a recursive contract-based graph harness, um, which is an interesting thing to build. But it's, uh, anyway, so it's, uh, it runs, um, non, uh, non-deterministic, uh, depth graph loops, uh, so it's able to build its own sort of roadmap and plan and and, you know, build stuff, um, using some deterministic components. Anyway, measuring that, it needs to be able to measure my current harness, which isn't that, and measure this new harness, whether it's producing something that's better. So I think, yeah, absolutely, that fits into a— because like building harnesses is a little bit like, you know, engineering now is a little bit being like being a Jedi. And we, uh, You know, we, we get these engineers, AI engineers, and they use the, the standard Claude tool, but eventually they graduate into this Jedi mode where they want to start building their own harness like a Jedi builds a lightsaber. And, and every lightsaber is a little bit different, um, but you can see it still has all the hallmarks of being a lightsaber. And, and so, you know, that's, that's functionally a testable thing. So that's why I think it becomes a good test for an interview Um, and they're also closely held secrets, uh, for engineers. That becomes part of their, part of their distinguishing factor is how quickly my thing can do something. Is it 70% complete, 80% complete, or 90% complete at the end of a task? Um, how did you get that 90% completeness? That's all measurable, right? And so I, I think, um, uh, having something which, which is able to measure that is— it will be, will become a tool. We have some harness testing tools already. And, you know, there's various benches you can go and look at online. 63:19 James MacDonald: And that's probably the closest thing to an old school code review, right? 63:22 Adam Witanowski: Yeah, I think it's, it's most analogous to, probably analogous to a code review. I think the other thing, there's still code reviews. Does your harness produce something which is good or red hot garbage? Right. So, you know, there's still things like cyclometric complexity and other stuff you can measure. And so I built a, uh, you know, through going through the recruiting process, um, which was a bizarre thing to enter back into at this point because I was, you know, looking for, uh, looking through jobs and applying for things a couple of years ago. Doing it this time around was just so different, um, and, and I think it's a functionally broken industry at the moment, uh, in a lot of ways because, uh, you know, I get an agent to produce a CV particular to a JD. I send that through to you. You now have got 1,000 people doing the same thing. And so your system previously would have ingested that, found some things which were, uh, you know, which match— pattern matched me to the, to the job description. If 1,000 people are doing exactly the same thing, you don't have 20 bits of signal now, you've got 1,000 bits of signal. 64:31 James MacDonald: You've got AI interviewing AI. 64:33 Adam Witanowski: Yeah, 100%. Yeah. And Um, and so, you know, I, uh, I thought, what would this look— I mean, this is the engineer in me— how could this look different? And so I thought, what if you reverse the paradigm and you, uh, and engineers put themselves on a marketplace and then recruiters go and discover them there? You've now got the same problem that you've got with human in the loop, where you've got a context-rich engineer being sourced by a context-poor recruiter. So there needs to be some measures that you can put over that. So exposing their GitHub to a tool which can measure complexity and flaws and errors and run some passes over it, I think is helpful for the recruiter. 65:17 James MacDonald: One step further than that, are we going to a point where it's, I've got a recruiter agent, you've got your own agent, your job application agent, one of your 1,000 agents, 65:29 Adam Witanowski: right? 65:30 James MacDonald: Are we getting to a point where it's agent versus agent and maybe a human jumps in at the last point? 65:33 Adam Witanowski: Um, I mean, there's already some areas where this is becoming real, where, you know, my personal assistant agent will call, uh, something and they will have an AI agent. And then so you've got this sort of agent-to-agent interaction, um, you know, and it's— I think that there's— that probably needs to be formalized a little bit because there's already protocols to handle agent-to-agent communication. Yeah. Um, you know, my, my, um, uh, set of agents that I built— so it took— I had to— I fed it a huge corpus of information. Huge, uh, relative, but a large corpus of information about me, my work history. It interviewed me 3 times, built this whole sort of backlog of some canned answers. If they ask this, this is the direction I want you to answer, that kind of thing. Built a zero BS filter so it couldn't actually lie to the recruiter. And then all I need to do is give it a JD. It takes that JD, creates a CV, a website, an ATS-ready ingestible version of it, and then a chat agent that, that a recruiter could go and have a conversation with me about that particular role. And then of course it's capturing all of that, all of those questions and feeding it back into the system to create better responses for next time. Right. So that's, that's one side of that almost solved. Yeah. You know, from the my side, you guys need to catch up. And, and then they— that, that whole interview process, if we can get, you know, a code review across my, you know, across my personal GitHub repos, a benchmarking tool against my harness, And my, my agent being interviewed by your agent, we could probably cut down interview process to, you know, minutes. 67:27 James MacDonald: I think what you cut it down to is the most important part. And it's one of the things I probably— one of the first questions I always ask is that motivation part, right? So that's what you mentioned about like if I'm the company having to sell you on the vision, having to sell you on are you looking for the tooling, are you looking to solve a problem, or what are the problems that are important to you to solve? It's like, because when recruitment's done well, it's more than just a skill set. Oh, you can tick these boxes. It's like, are you gonna be motivated to be here long-term, solve problems here, add value here? What can I do for you as much as what you can do for me? And if you get that motivation alignment part right, that's where most of the success comes from. 68:02 Adam Witanowski: Oh, for sure. 68:03 James MacDonald: Even if you don't have the skills to start with, 'cause you can grow into those skills. 68:06 Adam Witanowski: And what I want out of a company is very different to what the next guy you interview is gonna. So I may look at your problems and go, Jeez, they're boring. Yeah. They may look at them and go, oh, I love the stability. I love sitting with a problem for 10 years. 68:19 James MacDonald: Yeah. So if you can take that technical part out of it, yeah, I think, and we, we talk to the human-to-human part, that's why I'm relatively confident in the future of the recruitment industry for a while. 68:28 Adam Witanowski: It's the same, it's the same problem as engineering, right? So it's, it comes down to a taste or a, you know, a set of things which are values, um, which, uh, which end up being applied And it becomes— that becomes intrinsically more human. 68:42 James MacDonald: Yeah, it's the communication, it's the empathy, it's the understanding of motivations. They're the things— problem solving, those core parts, taste. 68:50 Adam Witanowski: Yeah, 68:51 James MacDonald: these are the things that will survive, and the people that are good at those things will continue to grow. And I think, as you said, the technical side of things, we'll get to that point. It's just the good engineers are a couple of steps ahead of the people that are applying that to recruitment. 69:06 Adam Witanowski: Yeah, testing. Of course, there's bazillions being spent on it. 69:09 James MacDonald: Correct. 69:11 Adam Witanowski: So yeah, I think that again, that's a, that's a, I think it's a common theme within AI at the moment is this idea of an arms race. You know, you've got, you've got various arms races going on and one of them is in recruitment for sure. Yeah, 69:23 James MacDonald: there's more HR tools being invested in globally than I think most other markets. 69:27 Adam Witanowski: Right. 69:28 James MacDonald: Because it's a broken industry. It's been a broken industry for a very long time. 69:32 Adam Witanowski: Yeah. And it's a high expense too for organizations. 69:35 James MacDonald: Correct. All right, mate, we can both agree on the recruitment process is part, what, broken, especially this senior end of town. And I think recruitment becomes more important at senior end of town because we do play to that. You're in demand. A company needs to attract you to them and then assess your capability. If you had to give a summary piece, what can a company do? What does good look like in attracting senior level AI talent at the moment? 70:03 Adam Witanowski: Yeah, so I think alignment is really important. So, you know, not every senior is going to be a fit for every role. And so just making sure that they're culturally aligned and aligned with the vision that you have for the business and really selling something to them which is meaty enough to, you know, they can still learn in because people at that top end still want to learn. You know, I think salary is a, is a big thing. Being competitive in salary is, is increasingly important, particularly as the wages for this kind of role inflate more and more overseas and seem to stagnate here. I think having a strategy for how you assess what good looks like, and I think this is probably going to, you know, in the short term Because if you unlock someone senior who's good, they can unlock the rest of the team, you know, and that's part of the value they bring. But I think in the short to medium term, it's probably going to come back onto talent agencies more and more and leaning on them to help find good talent, because often the good talent is sitting in a chair somewhere in some other organization and they're not necessarily hunting. So I 71:19 James MacDonald: think applying on Seek. 71:20 Adam Witanowski: Yep, they're not applying on Seek. And so I think, um, and people like yourself, you have relationships and you build those relationships over time. Um, and so, you know, I think utilizing, um, organizations like yours to, uh, you know, who have a finger on the pulse a little bit, you're not throwing your hands in the air and saying, where are these people? Um, you know, you know where these people are and you can help put them into, uh, into problems that you believe they're going to be well equipped to solve. So I think I think trust, trust the you of the world. 71:52 James MacDonald: I appreciate that. 71:53 Adam Witanowski: And, and have a, have a plan for what good looks like. Be prepared to pay more than you're comfortable with and, and also have a clear vision or trust that this person can help you build a clear vision for what the future is going to look like. So it may even look like we bring you into this role, how will you shape it? 72:19 James MacDonald: Which aligns with something you mentioned earlier today about not being put in a box. Like, give you the freedom to implement your knowledge and solve problems across the business. Don't put you in a box, solve this one problem, one problem only. 72:32 Adam Witanowski: Yeah, absolutely. 72:33 James MacDonald: Mate, I think I might title the video, An Engineer Is an Apex Predator, a Jedi, a Stallion. And a unicorn. 72:44 Adam Witanowski: Yeah, 72:45 James MacDonald: I think they're the 4 we've gone with today. So, uh, not coming from somebody with no bias, obviously. So appreciate your time today. Uh, really interested to find out where you end up and the problems you're going to solve there. Cheers. 72:57 Adam Witanowski: Great, thanks, thanks for having me. 72:58 James MacDonald: Here's my take. Adam said one of the sharpest things I've actually heard on the show. Most companies have done a good job of measuring costs, what token costs, what their license costs, they know the salaries of their engineers, but most companies don't have a clear picture on the value added through AI. And that one problem explains the AI talent disconnect. Companies say they're seeing no ROI yet from AI, and Adam's answer is actually, you're not measuring it properly. And that's a core issue behind the disconnect between senior AI salaries in Australia being too low in comparison to the US. Companies not willing to pay the $300,000 plus that's needed to get this senior AI talent because they don't see the value in it yet. And it's a measurement problem, not a technology problem, but a measurement problem. So I encourage most companies, before you're going talking to a recruiter, going to market to try to find these senior AI engineers, understand the measurement side of it. How are you going to measure the value that this person's going to come in and provide? Once you can see that clarity around that, you can clearly measure the value that this person's going to provide to your organization Then you can more clearly map what it's worth paying to this as an individual. I think if we get that part right, companies will see a lot more alignment between what they're willing to pay and what the market's looking for, what the market's currently being offered. If they're prepared to go to the US, we get that part right, we get more senior AI talent staying in Australia, more companies seeing value from senior AI talent, and everyone's a winner. If you've got any value out of this episode today, follow the show. On any of the channels, subscribe to my newsletter on Substack. Look forward to providing more value next week. Cheers. 74:32 James MacDonald: This was the Building Tech Teams Podcast. We release new episodes every Tuesday. Really keen for feedback on guests you'd like to see, more topics you'd like to see in depth. If you'd love to provide any form of feedback to me, find me on LinkedIn, James McDonald AU. You won't miss me. This episode was produced by Day One. No transcript lines match that phrase. Try fewer words.
▶ 
▶ 
▶ 
▶ 
▶ 
▶ 
▶
▶
▶