This reflection extends Behaviour & Responsibility — the trap of "human in the loop" when the human has quietly stopped watching.
Automation often works best when systems handle routine activity and humans intervene when something unusual occurs. In theory, this combines machine consistency with human judgement.
In practice, remaining alert while a system performs correctly is difficult. When automation succeeds repeatedly, people naturally pay less attention. Skills weaken because they are rarely used. Confidence in the system grows while readiness to take control declines.
Then, when an exceptional situation occurs, the person may be least prepared at the moment they are most needed. This is automation complacency. It can occur in transportation, cybersecurity, finance, manufacturing and any environment where people supervise intelligent processes.
The problem cannot be solved merely by placing a human "in the loop." The human must remain capable of understanding the situation and exercising meaningful control.
Organisations should ask whether employees still possess the knowledge to operate when the system fails. Training should include unusual and failure scenarios, not only normal operation.
Responsibilities must also be realistic. A person cannot attentively monitor hundreds of automated decisions while simultaneously completing other work. Human oversight becomes symbolic when the workload makes real review impossible.
Good system design should help people understand why intervention may be necessary and alert them before the situation becomes unmanageable. Automation should reduce avoidable burden — but it should not create the illusion that humans remain responsible while removing their practical ability to act.
The intelligent organisation must preserve both capability and attention — not simply assign accountability after failure.
Keeping a human in the process has little value when the system has trained that human to stop paying attention.