Lesson 5.39: Assessing a high-risk AI system
AI systems used to make decisions about people need more than a security review. The EU AI Act classes AI used in employment, such as recruiting and selecting candidates, as high-risk (Annex III), with obligations for risk management, data governance, logging, human oversight and accuracy. The NIST AI Risk Management Framework 1.0 organises the work into Govern, Map, Measure and Manage. In the lab HR wants a tool that ranks job applicants. You read its documentation, measure outcome rates by group, find weak oversight and no logging, check which risk management functions were never done, and write the decision.
20 min
You will learn to
- Recognise high-risk AI uses under the EU AI Act
- Read a model's documentation critically
- Compare selection rates between groups
- Check oversight, logging and data governance
- Use NIST AI RMF functions to find gaps
Key terms
- High-risk AI system
- Under the EU AI Act, AI in listed areas such as employment (Annex III), with strict obligations.
- Selection rate
- The share of a group's applicants who get a positive outcome.
- Four-fifths rule
- A US screening heuristic: a group's rate below 80% of the highest group's rate signals possible adverse impact.
- Human oversight
- People able to understand, monitor and override the system's outputs.
- NIST AI RMF
- NIST AI Risk Management Framework 1.0 (January 2023): Govern, Map, Measure, Manage.
← Running a major incident · All lessons · Capstone: a three-year strategy with funding options →