--- title: Do We Need Frontier Intelligence for 80 Percent of Knowledge Work? slug: do-we-need-frontier-intelligence-for-80-percent-of-knowledge-work date: 2026-07-07 summary: A finance and strategy view on why model cost is becoming a first-class engineering problem, and why evals are the bridge between AI architecture and business judgment. categories: "[[Writing]]" tags: - "#ai" - "#evals" - "#finance" - "#strategy" featured: false published: false --- You have likely heard about evals when reading about models. As a non-technical person working predominantly in finance & strategy, I stayed away from them thinking them to be out of my domain. They still are in many ways. But I have started to think about evals less as a technical benchmark, but more as an effective tool to feed into our decisions to use specific models, and that largely dictates cost and margin profiles for a product. The question today is different from what it was a year and half back. It goes much beyond "which model is smartest?" We really need to start asking "Where do we actually need the smartest model, and who should help decide that?" Frontier models are amazing - I use them everyday to run my agentic workflows. If the task is vague, high-stakes, or hard to specify, the frontier models have given me the freedom to work within a margin for error while taking away the context management overhead. I have found them to recover from messy prompts, absorb context, and yet produce useful outputs. But the last couple of months have made me think that the defaults are expensive. And GLM 5.2 made me think about what the alternate future looks like. For a while, model selection could look like a mostly technical decision. Engineers picked the model that worked best, product teams judged the user experience, and finance eventually saw the bill. I do believe that is not good enough anymore. AI has started moving from demos and betas towards running repeated workflows. Cost has become a part of the architecture. Latency, quality, reliability, context length, observability, routing, and unit economics have all started sitting together. I build all of these considerations into my model assumptions - most of them are qualitiative, requiring an understanding of seemingly technical topics. The model is not just an API choice anymore - it defines the cost structure and in turn the product's commercial sustainability. That means strategy and finance is increasingly meant to be a stakeholder in the design of the product. If every workflow in a product routes to the expensive frontier models because we have not done the work to understand what the task really requires, then we may get great outputs, but we learn very little. We miss identifying which parts of the workflow require frontier reasoning. We do not learn which parts are routine. We do not learn which tasks should be handled by cheaper models. It does not learn where deterministic software would be better than a model altogether. Now that is a technical problem, but also a finance problem. And it is a business problem all of sudden - and not an engineering problem. As a thought exercise I considered knowledge work - unarguably the next growth lever for agentic applications. Cowork from Anthropic has pretty much driven a chunk of the 2026 hockey stick growth curve that propelled it to ~$50Bn run-rate and it also serves a huge profit pool that is hard to penetrate. I have often asked myself "do we need frontier intelligence for 80 percent of knowledge work?" as I saw Opus 4.8 tokens being used to search Slack threads and clean up finance datasets. At the frontier, GPT and Claude families of models are both extremely capable and either can be a plausible default for most use cases. However, they are not identical. Their capabilities can very across dimensions like long-horizon agentic work, structured extraction, tool use, coding, latency, instruction following, or cost. Evals at the frontier are often about choosing between strong defaults and finding edge-case differences. Below the frontier, evals become even more important because capability is less smooth. A model may look excellent in one area and brittle in another. It may be good at coding but bad at spreadsheet-like joins. It may reason well but fail to produce an artifact. It may be strong in benchmarks but unstable inside an agent scaffold. Now open-weight models are interesting to me here not because I think they automatically replace the frontier models. I don't think that. Nor do I think that companies should pick models based on ideology. Open-weight models create optionality and change cost curves and it is worth noting their effectiveness for that reason alone. That is where open-weight models become interesting to me. Not because I think open-weight models automatically replace frontier models. I do not. And not because I think companies should pick models based on ideology. They should not. Open-weight models are interesting because they create optionality. They can change cost curves. They can give companies more control over deployment. They can reduce dependence on a small number of closed providers. They can make it possible to route more work through cheaper infrastructure, especially when the work is bounded and checkable. That is why I think finance and strategy teams should care early. Not because they should dictate model choice from a spreadsheet, but because cost is now a first-class engineering constraint. If the system is designed without cost awareness, the business model will eventually push back. But that only matters if the models can actually do the work. So over the weekend, mostly out of curiosity, I ran a small eval on finance workflows. I wanted to test something close to the kind of work a strategy or finance team might actually care about. Not generic "does the model know finance?" questions. Those are too vague. I wanted messy, practical cases where the issue is not the formula, but the judgment around the formula. Things like: - A file name says Q3, but a worksheet title still says Q2. - A bank deposit matches three invoices, not one. - Customer names are slightly different, but probably refer to the same account. - A board-pack narrative implies churn caused the miss, but the revenue bridge does not actually prove that. - A close summary mixes actuals, bookings, cash, and billing cutoffs. These are small examples, but they are real in shape. A lot of finance work is not hard because the math is unknown. It is hard because the context is imperfect. The person doing the work has to understand which source of truth matters, which assumption is reasonable, what should be left open, and what should not be overclaimed. That is exactly where model quality starts to matter in practice. The model has to notice ambiguity. It has to make a reasonable business assumption. It has to explain that assumption. It has to avoid pretending the data says more than it does. And it still has to produce a useful output. That is what an eval is supposed to test. An eval is just a structured test. But a good eval is more than a quiz. It is a way to turn a vague model debate into evidence. Instead of saying "I feel like this model is better," you define a workflow, define what good looks like, run multiple models against the same task, and inspect what failed. The inspection matters more than the score. That was the biggest lesson from this exercise. In my first run, GLM looked like it failed everything. If I had stopped there, the conclusion would have been simple: GLM was not usable for this workflow. It also would have been wrong. The trace showed that the local relay was not running correctly, and the provider adapter was passing prompts in a way that caused invocation failures. That is not a model capability failure. That is an operational failure. Then, after fixing the relay and running the tests again, another issue showed up. Some of the grader checks were too literal. For example, the grader penalized a model for mentioning an unmatched invoice even though the model correctly left that invoice open. In finance terms, that was a reasonable answer. The grader was the brittle part. So the eval did two jobs. It tested the models. And it tested the testing method. That is important because this is exactly what will happen inside companies. A bad eval can make a good model look weak. A broken tool path can be mistaken for a reasoning failure. A brittle grader can punish a model for doing the right thing in a real-world business context. After fixing those issues, the result became more interesting. Both GLM 5.2 through the local relay and default Codex passed all five ambiguity-heavy finance tasks. GLM scored slightly higher in the clean run because it carried more numeric and source evidence directly into the structured output. Codex also made the right business calls. Here is the clean technical readout from the run: | Test | What it tested | GLM output | Codex output | Commentary | | ---------------------------- | ------------------------------------------------------------------------------------------------------- | ----------: | -----------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Conflicting source of truth | Whether the model follows the stronger business signal when file metadata and worksheet labels conflict | Pass, 1.000 | Pass, 0.905 | Both chose Q3 over the stale Q2 worksheet title. GLM carried more numeric evidence into the answer, including revenue of 1,284,000 and 72% gross margin. Codex made the right source-of-truth call but was lighter on the numeric support. | | Split customer payment | Whether the model can match one cash deposit to multiple invoices and leave the right item open | Pass, 1.000 | Pass, 1.000 | Both matched the 48,250 deposit to INV-1008, INV-1011, and INV-1014, and left INV-2002 open. This is the kind of task where a model has to behave like an operator, not just a calculator. | | Dirty customer names | Whether the model can consolidate messy customer names without over-merging distinct entities | Pass, 1.000 | Pass, 1.000 | Both grouped ACME INC, Acme Holdings LLC, and Acme US at 315,000 ARR, while keeping Acme Europe separate at 90,000. The key was using tax and domain evidence rather than just fuzzy name matching. | | Board-pack driver discipline | Whether the model resists an unsupported narrative when the bridge tells a different story | Pass, 1.000 | Pass, 1.000 | Both identified the revenue miss as primarily volume and renewal timing, with usage as a partial offset. Both avoided blaming churn or pricing without quantified support. This is a very finance-native judgment test. | | Partial cutoff | Whether the model can handle mixed close status across actuals, bookings, cash, and billing | Pass, 1.000 | Pass, 1.000 | Both correctly treated April as preliminary, not final. Actuals were through Apr 28, bookings and cash were complete through Apr 30, and billing was delayed for Apr 29-30. | Overall, GLM passed 5 out of 5 with an average score of 1.000. Codex passed 5 out of 5 with an average score of 0.981. That is not a sweeping conclusion. It does not prove GLM is generally better than Codex. It does not prove open-weight models are ready to replace frontier models. It definitely does not prove that five hand-authored finance tasks are enough to make a production routing decision. But it does suggest something useful. At least some finance and strategy work may not require the most frontier model available. It may require a model that is good enough, a workflow that is well understood, a grader that catches the right failure modes, and a human who knows where the risk is. That is the opinion I am forming. Frontier intelligence is still the right default for many things. If the task is open-ended, strategic, poorly specified, high-risk, or long-horizon, I still want the strongest model I can use. If I need a model to operate across a large codebase, reason through a messy business question, or recover from bad instructions without help, I am not trying to save pennies on the model call. But a lot of knowledge work is not like that. A lot of knowledge work is bounded and repeatable. The inputs may be messy, but the workflow is familiar. The output shape is known. The failure modes can be named. The work can be checked. That is especially true in finance. Finance teams live in repeated workflows: close, reporting, variance analysis, planning, reconciliations, board prep, customer analysis, budget reviews, forecast updates. These workflows are not easy, but they are structured. A lot of the value comes from judgment inside constraints. That is exactly the kind of environment where evals should matter. They give finance and strategy teams a way to participate in the AI architecture conversation without pretending to be model researchers. The contribution is not "pick the cheaper model." The contribution is to help define the work, the risk, the acceptable failure modes, the review process, and the unit economics. From a finance seat, I would not want to make model decisions based only on broad benchmarks. I would want to know: - Can this model handle our actual workflows? - Can it make the same assumptions a good analyst would make? - Does it know when to leave something open? - Does it cite the numbers that support its conclusion? - Does it overclaim? - Does it fail silently? - Can the output be checked? - What does it cost at scale? Those are not abstract AI questions. Those are operating model questions. They are also engineering requirements. And this is why I think open-weight models deserve a serious look from strategy and finance teams. Not because they are always better. Not because they are more fashionable. Not because "open" is automatically good. They deserve a serious look because cost and control are real strategic variables, and because those variables now shape the engineering design itself. If an open-weight model can do a bounded workflow well enough, and if the workflow has good evals around it, then it may be irresponsible not to consider it. The company could save money, reduce vendor concentration, and build a more flexible AI stack. But the opposite is also true. If the eval shows that the model is brittle, unstable, or unable to make reasonable assumptions, then the answer is simple: use the frontier model, or redesign the workflow. That is the point. The goal is not to prove that cheaper models are better. The goal is to know where they are good enough. Without evals, model routing becomes politics. Technical teams argue for capability. Finance teams argue for cost. Product teams argue for user experience. Everyone is directionally right, but nobody is measuring the workflow. With evals, the conversation gets more concrete. You can say: for this workflow, on these cases, with this grader, this model succeeds, this model fails, and here is why. You can distinguish a model weakness from a tool failure. You can distinguish a bad answer from a bad test. You can decide where frontier intelligence is worth paying for. That is the bridge. Engineering gets a better way to reason about model architecture. Finance gets a better way to reason about cost without flattening quality. Strategy gets a better way to reason about where AI is actually changing the operating model. That is the real takeaway for me. The future of enterprise AI is probably not one model choice. It is a routing problem. Use frontier models where ambiguity, risk, and long-horizon reasoning demand them. Use cheaper capable models where the work is bounded and measurable. Use deterministic software where correctness should not depend on a model at all. And build the evaluation layer that tells you which is which. That is where I think strategy and finance teams should weigh in. Not at the end of the AI conversation, when the invoice arrives. At the beginning, when the company is deciding what work actually needs frontier intelligence and what work simply needs a reliable, measurable system. In that sense, model cost is not just a budget line. It is a design constraint. And if cost is a design constraint, then finance is not outside the engineering conversation anymore. That is why this weekend experiment was useful to me. It did not answer the whole question. But it made the question sharper. We should not ask whether open-weight models are "as good as" frontier models in the abstract. We should ask where they are good enough, how we know, what it costs, what can go wrong, and whether the workflow has enough structure to make the answer verifiable. That feels like the more serious AI cost conversation.