Ungoverned AI in safeguarding, and the case for tools built by professionals, for professionalsOut of the Shadows

In December 2023, the Department of Families, Fairness and Housing in Victoria, Australia notified its privacy regulator of an incident. A child protection worker had used ChatGPT to help draft a Protection Application Report: the document submitted to the Children’s Court to inform decisions about whether a child needs protection, including whether they should be removed from their parents’ care. The case concerned a young child whose parents had been charged in relation to sexual offences.

The worker’s stated motive was not malicious. They wanted to save time and present their work more professionally. And the investigation did not stop at one worker. An internal review found indications of AI involvement in around one hundred other child protection documents from the same unit. Between July and December 2023, nearly nine hundred of the department’s employees had visited the ChatGPT website. One inaccurate court report was the visible tip. The iceberg was organisational: a workforce under pressure, reaching for an ungoverned tool, in the absence of a governed one.

Shadow AI describes the use of AI tools, most commonly consumer chatbots, by staff without their organisation’s approval, oversight or knowledge. It is no longer fringe behaviour. A Microsoft-commissioned survey of just over two thousand UK workers in October 2025 found that 71% admitted using consumer AI systems without IT approval, half of them weekly. Fewer than one in ten workers had received substantial AI training.

The caring professions are not exempt; if anything, the pressures that drive shadow use are sharper there. Research commissioned by Social Work England in 2025 found that of the social workers surveyed, one in four reported using generative AI without direction from their employer. Focus group participants gave examples of personal information being shared with publicly available tools. A survey of around one thousand UK GPs found 20% using generative AI in clinical practice despite an absence of guidance. In law, a tracking database now records more than fourteen hundred cases worldwide in which courts have commented on AI-generated fabrications in legal filings.

These are self-report studies with the limitations that implies. But the direction of the evidence is consistent and unambiguous: in every profession measured, a substantial minority of practitioners are already using ungoverned AI on real work. In safeguarding, that work involves the most sensitive information an organisation holds.

It would be convenient to frame shadow AI as a conduct problem. The evidence does not support that framing. The Victoria worker wanted to save time. BASW’s annual survey found almost two thirds of social workers unable to complete their work in the hours available. The Social Work England research found 83% of practitioners saw AI’s potential to reduce administrative burden, and that 86% of those who qualified in the past five years received no preparation whatsoever for using AI in practice. Shadow AI is unmet demand. It is what a workforce does when the workload is real, the technology is freely available, and the organisation has offered nothing better.

The instinctive organisational response is a ban. The Victoria case shows both why bans happen and why, on their own, they do not work. The department had policies; the regulator found them insufficient as controls. The ban arrived after the report had reached the Children’s Court, after a hundred documents showed signs of AI involvement, after nine hundred staff had already visited the website. Prohibition was the consequence of the incident, not its prevention.

Network blocks do not reach personal devices, where the professional liability is greatest. BASW’s practice guidance, the first of its kind in England, is explicit that accountability sits with the practitioner using the tool, and that liability increases where AI is used on a personal device without an employer’s permission. A prohibition that simply relocates use from a managed environment to an unmanaged one has made the position worse.

Prohibition also fails the test regulators now apply. The Information Commissioner’s Office frames its expectations around a single question: can you show how each decision was made, and are the safeguards proportionate to the risk? Shadow AI fails that test by definition. No logs, no impact assessment, no human oversight anyone can evidence, and no means by which a child or family could ever contest how their information was handled. An organisation cannot govern what it cannot see.

The choice facing safeguarding organisations is not between AI and no AI. That choice has already been made by the workforce. The real choice is between governed and ungoverned use.

Drawing the OVIC findings, the BASW guidance, the ICO’s test and the DfE’s generative AI product safety expectations together, a credible alternative to shadow AI in a safeguarding context must be able to demonstrate, not merely assert, the following:

1. Purpose-built for the context.  Designed around safeguarding practice and statutory guidance, not a general-purpose chatbot with a disclaimer attached.
2. Anonymisation by design.  No identifying information about children enters the tool; the design makes the safe path the easy path, screening at the point of input rather than relying on policy alone.
3. Enterprise data terms.  Inputs are not used to train models and are not retained on consumer terms, with a data protection impact assessment before deployment, not after an incident.
4. A human in the loop with named accountability.  The tool advises; the professional decides. Nothing generated enters a record or statutory process without review and approval.
5. A verification step built into the workflow.  The documented failure mode is generated text that softens risk. AI-drafted material is checked against what the professional actually observed and received before it is relied upon.
6. An audit trail.  Logged use and reviewable outputs: precisely what the ICO’s test demands and what shadow use can never provide.
7. Alignment with statutory guidance.  Grounded in Keeping Children Safe in Education, Working Together to Safeguard Children, and the organisation’s own policies.
8. Transparency.  Staff, leadership and the wider community know the tool exists, what it does, what it must never be used for, and who is accountable for it.

Any tool that cannot evidence these properties is not an alternative to shadow AI. It is shadow AI with better branding.

Ask RESPOND was built to this specification because it was built from inside the problem: developed within a school, by practising safeguarding professionals who carry the duty of care they are designing for, on the foundation of the RESPOND workflow (Recognise, Engage, Support, Pause, Offer, Notify, Document). It provides immediate, context-sensitive guidance aligned with the school’s own policies and escalation pathways at the moment a member of staff needs it. It does not make decisions. It behaves as an expert colleague, not a supervisor: accountability remains where the law and professional standards place it, with the human.

Its design answers the drivers of shadow AI directly. It meets the workload need that sends staff to consumer chatbots, so the governed route is also the convenient one. It requires no personal data about children to do its job. And the framework it runs on already contains the behaviour the Victoria case showed to be missing. Pause, the bridge step at the centre of RESPOND, is the discipline of stopping to check before acting. It applies to a first impression of a child’s presentation, and it applies equally to a fluent paragraph produced by a machine.

One more thing, because a paper that exposes ungoverned AI must not exempt its own tools from scrutiny: RESPOND’s position has always been that it demonstrates rather than certifies. The standards above are applied to the Ask RESPOND ecosystem as an ongoing programme of evidence, and where a capability cannot yet meet the standard, it is not deployed. We commend the same honesty to every organisation and every vendor in this space.

1. Do you actually know which AI tools your staff are using for work, or do you only know what your policy permits?
2. If a member of staff had used AI in a way they were unsure was allowed, would they feel safe telling you?
3. Has a data protection impact assessment been completed for every AI tool in approved use?
4. Does anything AI-generated enter a record or report without review by an accountable professional?
5. What approved alternative do you offer for the workload tasks that send staff to consumer chatbots?

OVIC, Investigation into the use of ChatGPT by a Child Protection worker (September 2024) · Social Work England, The emerging use of Artificial Intelligence (AI) in social work, with research by Research in Practice and The Open University (2025) · BASW, Generative AI and Social Work: initial guidance for practice and ethics (March 2025) · Microsoft/Censuswide survey of 2,003 UK workers (October 2025) · Blease et al., Generative artificial intelligence in primary care, BMJ Health and Care Informatics (2024) · DfE, Generative AI: product safety expectations · ICO, Guidance on AI and data protection. All statistics are self-report survey data and should be read as indicative of an emerging pattern. OVIC found the inaccuracies in the Victoria case did not ultimately alter the decisions of the department or the court; the findings concern the breach and the risk created.