Tags
Agentic, Workflows, Roles
Author
Wout Helsmoortel
Agentic AI does not eliminate human capital, it repositions it. People remain the arbiters of quality, trust, and compliance, while agents absorb repetitive tasks and orchestration overhead. The aim is to redeploy expertise where it creates leverage: setting standards, judging edge cases, and teaching the system to improve.
Why human roles matter
Agentic systems change the shape of work. Instead of people pushing tasks through steps, the system coordinates actions, retrieves evidence, and performs first‑pass checks. Human leverage moves to three higher‑order activities: defining standards, arbitrating risk, and teaching the system to improve. This is a workforce design question before it is a model question: who does what, with which tools, and under what accountability?
What actually changes
Work allocation: agents take on retrieval, formatting, and routine evaluation; people own exceptions, interpretation, and policy trade‑offs.
Decision rights: explicit escalation paths and veto powers keep humans in command.
Competencies: method design, oversight, and feedback coaching become core skills.
Incentives: quality and learning metrics matter as much as volume throughput.
Tooling: tracing, evaluation dashboards, and prompt libraries become part of everyday work.
The three core roles
1. Method makers
Turn practical know‑how into reusable methods. They define metrics, scorecards, and test cases that capture what “good” looks like, and they write evaluation rubrics that Evaluator Agents can apply.
What they do: capture failure modes, design metrics, create labeled examples, decide pass and fail thresholds.
Deliverables: evaluation handbook, seed ground truth, prompt blocks for verification.
Avoid: vague criteria, one‑off reviews without codifying the method.
2. Judges and escalation owners
Handle uncertainty and risk. They review high‑criticality outputs, decide when to accept or reject, and author the escalation rules that keep control with humans.
What they do: spot‑check outputs, resolve agent vs human disagreements, author refusal rules.
Deliverables: sampling policy, escalation tree, audit notes.
Avoid: rubber‑stamping, hidden decision logic.
3. Continuous trainers
Feed the loop with targeted corrections so the system learns. They turn edits and comments into structured feedback that improves prompts, tools, or data.
What they do: annotate mismatches, add counter‑examples, request new tools or data.
Deliverables: feedback log, updated test cases, change requests.
Avoid: free‑form comments without labels or links to cases.
Example
Defense training: instructors co‑create a rubric with four metrics: accuracy, compliance, completeness, clarity. A draft evaluation is generated by the agent after each exercise. The Evaluator Agent scores the draft, lists missing evidence, and flags low‑confidence sections. Human judges review those flags, approve or reject, and leave labeled comments. Trainers convert recurring edits into new test cases, update the rubric, and adjust prompts. Over eight weeks, average review time falls by 60 percent while traceability improves because every decision is logged with sources and rationales.
Accountancy: juniors prepare fiscal advice based on client data. The agent checks rule applicability and adds citations. If a threshold is outdated or a deduction lacks justification, the Evaluator Agent lowers the completeness score and asks for clarification:
“Would you like me to insert the current small‑business VAT threshold and link the source?”
Seniors handle exceptions and update the sampling policy when the mismatch rate drops below an agreed target.
Conclusion
Agentic systems succeed when people and machines evolve together. By defining who designs methods, who judges exceptions, and who trains the system, organizations turn expertise into infrastructure. Over time, the same people who once reviewed outputs start improving the evaluators, governance loops tighten, and trust becomes measurable.
Human roles don’t disappear; they move closer to strategy, assurance, and learning.
References and Acknowledgments
Author: Wout Helsmoortel
Founder, Shaped — specializing in agentic AI systems, explainable architectures, and learning transformation for defense and enterprise environments.
Academic references
Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv.
Gu, J., et al. (2024). A Survey on LLM-as-a-Judge. arXiv.
Li, H., et al. (2024). LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv.
OpenAI. Evals Framework. OpenAI.
Anthropic. Third-Party Model Evaluations Initiative. Anthropic.
