.svg)
Jean-Marie John-Mathews, Ph.D.
.png)
How to implement LLM as a Judge to test AI Agents? (Part 2)
Testing AI agents effectively requires automated systems that can evaluate responses across several scenarios. In this second part of our tutorial, we'll explore how to automate test execution and implement continuous red teaming for LLM agents. Learn to systematically evaluate agentic AI systems, interpret results, and maintain security through ongoing testing as your AI application evolves.
.svg){kind=icon}
How to implement LLM as a Judge to test AI Agents? (Part 1)
Testing AI agents effectively requires automated systems that can evaluate responses across several scenarios. In this first part of our tutorial, we introduce a systematic approach using LLM as a judge to detect hallucinations and security vulnerabilities before deployment. Learn how to generate synthetic test data and implement business annotation processes for exhaustive AI agent testing.
Exclusive interview: our first television appearance on AI risks & security
This interview of Jean-Marie John-Mathews, co-founder of Giskard, discusses the ethical & security concerns of AI. While AI is not a new thing, recent developments like chatGPT bring a leap in performance that require rethinking how AI has been built. We discuss all the fear and fantasy about AI, how it can pose biases and create industrial incidents. Jean-Marie suggests that protection of AI resides in tests and safeguards to ensure responsible AI.
How to test ML models? #4 🎚 Metamorphic testing
Metamorphic testing are adapted to Machine Learning. This tutorial describes the theory, examples and code to implement it.
How to test ML models? #3 📈 Numerical data drift
Testing the drift of numerical feature distribution is essential in AI. Here are the key metrics you can use to detect it.
How to test ML models #2 🧱 Categorical data drift
Testing drift of categorical feature distribution is essential in AI / ML, requiring specific metrics
How to test ML models? #1 👉 Introduction
What you need to know before getting started with ML Testing in 3 points
Where do biases in ML come from? #7 📚 Presentation
We explain presentation bias, a negative effect present in almost all ML systems with User Interfaces (UI)
Where do biases in ML come from? #6 🐝 Emergent bias
Emergent biases result from the use of AI / ML across unanticipated contexts. It introduces risk when the context shifts.
Where do biases in ML come from? #5 🗼 Structural bias
Social, political, economic, and post-colonial asymmetries introduce risk to AI / ML development
Where do biases in ML come from? #4 📊 Selection
Selection bias happens when your data is not representative of the situation to analyze, introducing risk to AI / ML systems
Where do biases in ML come from? #3 📏 Measurement
Machine Learning systems are particularly sensitive to measurement bias. Calibrate your AI / ML models to avoid that risk.
Where do biases in ML come from? #2 ❌ Exclusion
What happens when your AI / ML model is missing important variables? The risks of endogenous and exogenous exclusion bias.
Where do biases in ML come from? #1 👉 Introduction
Research Literature review: A Survey on Bias and Fairness in Machine Learning
.jpeg)
What does research tell us about the future of AI Quality? 💡
We look into the latest research to understand what is the future of AI / ML Testing
.svg)