How it works, why it gets things wrong, and how people stay in charge
When the ACSF AI Working Group surveyed members in September and October 2026, 22 of the 28 respondents named inaccurate or made-up answers as a top concern. Sixteen named over-reliance on AI or reduced human oversight. Those two concerns are connected, and both can be managed. It starts with understanding what these tools actually do.
1. How AI models work, in plain language
The AI chat tools most people use today are large language models. They are trained on huge amounts of text, and from that text they learn patterns: which words, phrases, and ideas tend to follow one another.
When you ask a question, the model does not look up the answer in a database or check a regulation. It predicts the most likely next word, then the next, then the next, based on the patterns it learned. The result often reads like an expert wrote it, because the model has seen a great deal of expert writing.
That is the key point for safety professionals. Sounding right and being right are two different things. Unless a tool is specifically connected to a trusted source and shows you where each statement came from, it is producing likely-sounding language, not verified facts.
2. Why made-up answers happen, and why it matters here
Because the model is predicting plausible text, it can produce statements that are fluent, confident, and false. This is often called a hallucination. A model might:
- Cite a regulation, advisory circular, or report that does not exist.
- Describe a trend in your safety data that is not really there.
- Merge details from two different events into one.
- Fill a gap in a report with a detail that was never reported.
A model does not know when it is guessing, so it rarely tells you. The wording sounds just as sure when it is wrong as when it is right.
In everyday office work, an error like that is an inconvenience. In aviation safety work, it can shape a risk assessment, a corrective action, or a conclusion about what a crew member did. An invented finding in an Event Review Committee (ERC) summary, or a false trend in an SMS report, can send resources in the wrong direction and leave the real hazard in place. It can also damage the trust that makes voluntary reporting work.
That is also why the over-reliance concern matters. When a tool is right most of the time, people naturally start checking it less. The errors that slip through are the ones nobody expected.
3. How humans stay in the loop
AI can be a useful assistant for sorting, summarizing, and drafting. It cannot be the decision-maker. Every organization using it on safety work should be able to answer three questions clearly.
Who verifies the output? A qualified person who knows the subject reviews every AI-produced summary, trend, or draft against the original reports and data before it is used. If a statement cannot be traced back to a source, it comes out.
Who signs off? The same accountable people who own safety decisions today. That means the safety manager, the ERC, the Director of Operations, or whoever your SMS names. AI assistance does not change who is responsible, and the record should show a named person approved the final product.
What data never goes into the tools? Set this in writing before anyone starts. For most operators, that includes identifiable ASAP reports, names of reporters and crew members, and tail numbers or dates tied to a specific event. It also includes protected or privileged safety information and anything covered by a confidentiality agreement. Consumer AI tools should never see this data. Even enterprise or firewalled products need their data handling, retention, and training terms confirmed in the contract, and legal counsel should weigh in before protected safety data is used at all.
4. Free and consumer tools can learn from what you type
Many free and consumer AI tools, and some paid individual plans, can keep what you type and use it to train future versions of the model, depending on the provider and the account settings. Once safety data is part of that training, you cannot take it back, and there is a chance that details could surface later in ways no one intended. Enterprise versions typically offer contract terms that keep your inputs out of training and limit how long they are retained, but confirm that in writing rather than assuming it.
The safe habit is simple. If you are not using an approved enterprise tool with confirmed data protections, de-identify everything before it goes in. Remove names, employee numbers, tail numbers, flight numbers, specific dates, locations, and any detail that could point to a person or an event. Even with an enterprise tool, de-identifying first is good practice, because it limits the damage if anything ever goes wrong.
5. Keep the model’s answers anchored to the regulation
Because a model predicts likely text instead of looking things up, it may answer a regulatory question from general patterns. It can mix in rules from the wrong part, an outdated version, or another country’s requirements. You can reduce that risk by constraining what the model works from.
- Give it the source. Paste in or attach the exact regulation, advisory circular, or company manual section you want it to use, such as the specific 14 CFR part or your own SMS manual.
- Tell it to stay inside that source. For example: “Answer only from the text provided. If the answer is not in the text, say so.”
- Ask for citations. Have it quote or point to the specific paragraph behind each statement, so a reviewer can check it quickly.
- Use connected tools carefully. Some tools can search a defined set of documents. That helps, but the output still needs to be verified against the source.
- Check currency. Make sure the version you provide is the current one, and do not rely on the model to know about recent changes.
Constraining the model does not remove the need for review. It makes review faster and errors easier to spot.
A practical checklist
- Know what the tool is. Confirm whether it only predicts text or is connected to a trusted source that it cites.
- Treat every output as a draft, never a finding.
- Verify against the source. Check every fact, figure, citation, and trend against the original reports or documents.
- Name a reviewer for each AI-assisted task, with the expertise to catch errors.
- Name the sign-off. A named, accountable person approves anything that goes into the safety record.
- Write a no-go data list that covers identifiable ASAP reports, names, tail numbers, and protected safety information.
- Keep consumer AI tools away from safety data, and confirm the terms of any enterprise tool in writing.
- De-identify before you paste, especially in any tool that is not an approved enterprise product, since your inputs may be used for training.
- Anchor answers to the regulation. Provide the exact source text, tell the model to answer only from it, and ask for paragraph citations.
- Talk to counsel before using AI with protected or privileged safety information.
- Test the tool on past reports where you already know the answers, and repeat the test when the tool changes.
- Watch for over-reliance. Build review steps that require real engagement, and rotate reviewers when you can.
- Tell your reporters how AI is and is not used, so trust in the program holds.
Join the ACSF AI Working Group
No operator should have to work this out alone. The ACSF AI Working Group is building practical guidance shaped by what members told us they need, including short guides like this one. Whether your organization is already using AI, just exploring it, or skeptical of it, your experience belongs in that work.
Join the ACSF AI Working Group and help shape how AI supports safer aviation operations.


