Enterprise AI’s Hardest Problems Aren’t the Models — They’re Everything Around Them

Estimated reading time: 14–16 minutes

At 6DAI Melbourne 2026, the most revealing conversations were not about which model topped the latest benchmark.

The invitation-only audience brought together senior AI, data, security, technology and business leaders from banking, government, healthcare, manufacturing, mining, retail and utilities. The questions from the floor were practical rather than speculative: who owns the risk, what happens when the data is wrong, and how much authority should an agent be given?

Across those sectors, the discussion had moved beyond whether AI could perform useful work. The harder questions begin when that capability enters a real organisation.

Can organisations supply AI with information it can genuinely trust? Who is accountable when an agent changes a business system? What happens when an automated mistake propagates through several connected processes?

The technology is moving quickly. The organisations implementing it are discovering that data, governance, culture and operational discipline move at a different speed, and that tension ran through almost every session.

The foundations beneath enterprise AI

Joseph Sullivan from Informatica used the foundations of a skyscraper to describe enterprise AI. Models, agents and polished demonstrations occupy the visible upper floors, but underneath them sit fragmented databases, legacy applications, inconsistent definitions, access controls and years of undocumented business logic.

Most organisations can access broadly similar frontier models. The difference lies in the quality, currency and organisational context of the information surrounding them.

Having data is not enough. It must be current, traceable and understood within the business process where it will be used.

A pricing agent, for example, might have accurate sales records and still make the wrong decision if it cannot see service failures, customer sentiment or an elevated risk of that customer leaving. Multiple departments may also hold different but individually valid definitions of a new customer, completed transaction or active account.

The data can be correct while the answer is still wrong.

This was reinforced by Elizabeth Wilson from the Department of Education. An early attempt to apply AI across a policy library exposed contradictions between the policies themselves. Rather than simply blocking the project, AI helped reveal weaknesses in the information environment it had been asked to use.

HBF offered a more mature production example. Kathryn Gulifa described an AI-assisted workflow currently processing around half of the insurer’s ancillary claims and saving approximately $1 million a year in operating costs, with plans to extend the approach to medical claims.

Its value did not come from attaching AI to an arbitrary process. It began with a repetitive, expensive and well-understood operational problem. The strongest implementations discussed at 6DAI were led by real pain points, not by a newly purchased tool searching for somewhere to live.

When AI begins taking action

The risk profile changes significantly when AI stops recommending and starts doing.

A conventional generative AI tool might draft a document, summarise a meeting or suggest a response. A person can review the output before anything happens.

An agentic system may alter a database, initiate a payment, change an account, route a customer request or trigger another agent. A mistake is no longer confined to a disappointing paragraph. It can become an operational event.

Rachael Phelan from NAB framed this as a shift from a technology concern to an enterprise risk. A single AI-assisted decision may appear manageable, but thousands of decisions across multiple systems can create cumulative effects that are harder to detect and reverse, particularly when one agent’s output becomes another agent’s input.

The event repeatedly returned to the need for clear ownership. Central AI teams may select platforms and support deployment, but responsibility should remain with the business owner accountable for the affected customer, process or service. Deploying AI does not transfer that responsibility to the technology department.

Identity is also becoming part of the governance problem. James Schafer from AWS and Toby Allen from Okta argued that agents require identifiable accounts, limited permissions and traceable activity, much like human users.

Organisations need to know where an agent is operating, what it can access and what it is authorised to do. When it acts on behalf of an employee or customer, the chain of authority must follow the action across every connected system. An agent should never be anonymous, excessively privileged or impossible to audit.

Identity alone does not prove that a particular action was authorised. Policy enforcement needs to sit outside the probabilistic model rather than relying on the agent to police itself.

This becomes especially important as businesses move from internal agents toward systems that interact with agents deployed by customers, suppliers and other organisations. The next trust problem may be deciding not only what an organisation’s own agents can do, but what authority it is willing to recognise in somebody else’s.

Autonomy is earned in stages

Nicola Pastorello‘s description of staged trust at BlueScope felt immediately familiar from my work in controls and safety-critical systems. New systems are not given authority simply because they performed well in a demonstration.

They begin in advisory or shadow mode. Operators compare their recommendations against real outcomes over days, weeks or months, and only sustained evidence of reliability earns the system greater control. Manufacturing has developed this discipline because failures can affect equipment, production and human safety.

That model translates directly to agentic AI. A system can begin by offering information, progress to preparing actions for approval and eventually perform selected tasks independently. The speed of that progression should depend on the consequence, reversibility and observed behaviour of the individual use case.

In controls, oversight is meaningful only when the person watching has enough context to recognise a fault, the authority to stop it and a practical way to reverse the outcome. Placing a person somewhere in an AI workflow provides none of that automatically.

For low-impact and reversible actions, supervision may be enough. For consequential decisions involving finance, safety, vulnerability or customer rights, direct human approval remains considerably more important.

The aim is not to demand confirmation for every harmless action. Too many prompts create approval fatigue and turn governance into a reflexive click. Human intervention should be concentrated where it materially changes the risk.

A scalable approach is to let agents operate within policies approved in advance, while escalating exceptions and higher-risk actions for explicit human approval.

Where local models fit

The day also challenged the assumption that every AI workload needs the largest available cloud model.

Having begun working more deeply with local models and AI workflows myself, I found it encouraging to hear large organisations exploring the same architectural choices. Dr Imtiaz Bhayat described Regis Aged Care experimenting with local models on NVIDIA hardware after the cost of an existing AI-supported workflow became difficult to scale. Nicola Pastorello also pointed to smaller, quantised and open-weight models as increasingly practical options for suitable workloads.

That aligns with what I have been finding in my own testing. For tightly defined tasks such as extraction, classification, document retrieval and internal decision support, the largest model a system can run is not necessarily the most useful one. A smaller quantised model that performs the required task reliably and at an acceptable speed may be the better engineering choice.

A conversation after the sessions with Peter Chilcott from Natural Velocity made the remaining friction equally clear. He demonstrated a working Open WebUI-based RAG implementation using an NVIDIA embedding model, with reliable retrieval from technical documentation. The result showed that it can be done, but getting it to that point had required considerable time and iteration.

That closely mirrors my own experience. RAG sounds straightforward: embed the documents, retrieve the relevant passages and give them to the model. Technical manuals are rarely straightforward inputs. Multi-column layouts, split tables, repeated headers, image-based diagrams and command names separated from their descriptions can all damage the ingestion process before the language model receives a question. A capable model cannot retrieve information that never made it cleanly into the index.

This is why cloud AI remains easier to use: much of the difficult wiring has already been done and tested, then hidden behind a finished interface. Local tools offer greater control, but currently leave the user responsible for more of the stack. As consumer hardware and software improve, that trade-off should become less severe, making local execution practical for a much wider group of professionals and allowing more sensitive documents and routine inference to remain under organisational control.

Local deployment does not make frontier models redundant. Complex reasoning, broad general knowledge and large-scale training will continue to require substantial infrastructure. The more practical architecture is likely to be layered: frontier models for unusually complex work, with smaller models supporting well-defined tasks where cost, latency, privacy and operational dependence matter more than maximum capability.

The question is becoming less about which model is objectively best and more about which model is appropriate for the workload. That choice also has sustainability implications. Running every task through the most computationally demanding option available is difficult to justify when a smaller model can provide a sufficient result with fewer resources.

Building an organisational capability

The organisations moving beyond isolated pilots are treating AI as a capability rather than a software purchase.

Dilan Perera from Bendigo Kangan Institute identified four elements that need to develop together: workforce literacy, governance, an operating model, and data and technology foundations.

A model can be acquired quickly. The organisational capability needed to operate it responsibly takes considerably longer to build.

That breadth also explains why specialist vendors, technology partners and implementation teams have an important place in the ecosystem. Models, infrastructure, data preparation, retrieval, identity, governance and workflow design require different forms of expertise, and few organisations will sensibly build every layer alone.

External expertise should, however, build internal capability rather than permanent dependence. The measure of a good partner is not simply whether they can deploy the technology, but whether the organisation understands, governs and can ultimately take ownership of the result.

Manohar Esarapu from the City of Port Phillip added an important complication. Not all organisational knowledge is documented. Workarounds, judgement calls and practical experience often remain in the heads of long-serving employees.

An agent trained only on official procedures may understand how work is supposed to happen without understanding how it actually happens.

Before automating a process, organisations may need to uncover the informal knowledge currently holding it together. Removing experienced employees before that knowledge has been captured can leave a business with a sophisticated system and nobody who understands why the old process worked.

The City of Port Phillip’s work on a multilingual digital front door also illustrated where the human handover becomes important. An agent may be well suited to answering routine questions at any hour, but hardship, vulnerability and complex individual circumstances require a different response.

The harder design problem is knowing when the agent should stop and hand the conversation to a person.

This also reframes workforce adoption. AI literacy is not limited to prompt writing or familiarity with a particular chatbot. Employees need to understand what information a system is using, when its output deserves challenge, what authority they have delegated, which systems it can reach and when a task should return to human control.

The organisations that handle this best may be those able to absorb change without surrendering human judgement or institutional knowledge.

Governance, trust and the limits of good data

Governance was sometimes described as if it were the brake on innovation. The more convincing argument throughout the day was that it enables useful systems to move into production.

A deployment that cannot demonstrate where its information came from, what it changed, who approved it and how it can be stopped is unlikely to progress far in a regulated or safety-conscious organisation.

That requires maintaining an inventory of deployed AI, defining access and ownership, monitoring performance, preserving audit trails and planning for continuity if a model or provider disappears.

But trusted data has limits.

During the final audience panel, I asked what organisations were doing to address bias embedded in model training, particularly as AI becomes involved in areas such as recruitment, insurance claims and home-loan decisions.

The responses pointed towards grounding systems in approved information, comparing outputs across different models and retaining human approval for consequential decisions. Those are useful safeguards, but they do not fully resolve the underlying issue.

Trusted context can reduce factual errors. It cannot by itself correct demographic bias embedded in a model or discriminatory patterns contained within historical organisational data. Local deployment does not remove that problem either.

A system can be well documented, technically accurate and consistently unfair.

Responsible deployment therefore requires organisations to test outcomes, not merely validate inputs. In recruitment, lending, insurance, healthcare and other consequential settings, that means examining whether different groups are being treated differently and whether the human review process is capable of recognising it.

Governance must extend beyond provenance and technical performance to include the effect a system has on the people subjected to its decisions.

The hard part begins after the demonstration

6DAI Melbourne showed that the next phase of enterprise AI will not be determined simply by who has access to the largest model or the longest list of pilots.

Most organisations can purchase similar capabilities. The harder work lies in connecting them to trustworthy organisational knowledge, giving them carefully bounded authority, observing what they do and retaining the judgement to intervene.

That work is less theatrical than a model launch. It involves data integration, process mapping, access controls, workforce development, collaboration with experienced partners where needed, and difficult decisions about where responsibility ultimately sits.

Enterprise AI is rapidly becoming infrastructure rather than innovation. Organisations that succeed won’t necessarily be those with the most powerful models, but those able to combine trustworthy data, clear governance and experienced human judgement into systems people are willing to trust.

Further reading: Our exclusive interview with Ledger’s Ian Rogers explores why separating AI reasoning from human authorisation may become one of the defining principles of enterprise AI security.