A newer version of the Gradio SDK is available: 6.22.0
Case Analysis: AI Mental Health System at a University
Consolidated analytical material covering the case from multiple angles. Source content: 9 chapters previously delivered to participants in the No-AI condition. Now used as input for the per-session perfect-answer generator.
How to read this — and the case it discusses
Here is the case:
"A university is considering deploying an AI system that analyzes students' digital activity (e.g., learning platform usage, assignment submission patterns, and optionally social media signals) to identify those at risk of severe mental health issues, including depression and self-harm. The system does not access private messages but uses behavioral patterns to generate risk scores. Students flagged as 'high risk' may be contacted by university counselors, even if they have not explicitly asked for help. The university argues that early detection could prevent harm and save lives. However, some students and staff are concerned about privacy, potential misclassification, and the possibility that students may feel constantly monitored.
You are asked to evaluate whether the use of this AI system is ethically acceptable."
Notice two things about how it is worded.
First, the case picks calm and helpful-sounding words. It says "analyzes", which sounds technical and neutral. The same activity could be called "monitors" or "watches". It says "identify those at risk", which sounds like a discovery. It could be called "labels" or "categorises". It says "contacted", which sounds gentle. It could be "approached" or "summoned". You can describe the same facts in either set of words. The case picked the calmer set.
Second, the case leaves some words out. It says "privacy" but not "consent". It says "misclassification" but not "stigma". It does not say "agency", "trust", or "vulnerability". The words it uses tend to support the school taking action. The words it leaves out would limit what the school can do. Anyone who wants to bring those missing words in has to first justify why they belong, and that "first justify" itself keeps the words out.
The chapters that follow walk through eight angles on this case. As you read each one, keep in mind that the case is not just describing a situation. It is describing it in a particular way, and most of what feels natural to think is partly the case's wording shaping you.
1. Purpose — what is the school actually trying to do?
This chapter is about the gap between what the school says the system is for and what the system actually does. Noticing this gap is one of the simplest ways to make your argument stronger.
The case tells you the purpose right at the top: "early detection could prevent harm and save lives." That is the natural place to start. You take "save lives" as the goal, then ask whether the privacy costs and misclassification risks are worth it. Your whole argument then runs with "saves lives" sitting in the background as a fact about the system.
That treats the slogan as the system, when it isn't. The system is not "saving lives". The system is producing a list of names, sending automated alerts, and writing a quarterly report.
Picture what comes out of this system over two years. It flags 200 students. 195 of them get a counsellor email. The annual report says: campus suicides, zero; students supported, 195. A school administrator can defend that on paper. It looks like success. But the campus suicide rate was already very low before the system existed, so the zero tells you nothing about whether the system caused the result. The system has reliably produced its operational outputs (the list, the alerts, the report). The bigger claim "we saved lives" is doing different work that the outputs do not actually support. A stronger argument engages with what the system reliably makes, week after week, not just with what the school says it is for.
2. Evidence — what we know, and what we don't
This chapter is about how little the case actually tells you, and how fast people decide what they think by guessing at the rest without noticing they are guessing.
The case states a small set of facts. The system looks at things like login activity and submission patterns. It does not read private messages. It produces risk scores. Flagged students may be contacted. The school's reason is early detection. The concerns are privacy, misclassification, and feeling monitored. That is most of what the case actually says. The natural move, when you read this, is to take that small set of facts and reach a confident answer on top of it.
That confident answer is built on a lot of missing information that the writer is quietly assuming.
Here are some of the things the case does not tell you that would change your judgement.
- How accurate is the system? What is the rate of students flagged who were not actually at risk?
- Who decides what "high risk" means?
- Are students told before the system analyses them? Can they opt out?
- What does "contacted" actually mean (an email, a required meeting, a call home, a note in a permanent file)?
- How long is the data kept?
- If the system is wrong, can a student appeal?
None of this is in the case, and different answers would change everything.
There is also a deeper problem. Suppose the school says the system is "90% accurate". Accurate against what? In real clinical practice, depression risk is decided through interviews and questionnaires, and trained clinicians often disagree. There is no neat independent list of who is "really" high risk that the AI can be checked against. So when the school says the system "works", that often just means "the AI does what the AI does." A stronger argument names what is missing from the case before reaching a answer.
3. Concept — the words that look settled but aren't
This chapter is about a few words in the case that sound clear and settled, but actually shift around when you look at them closely.
When you read the case, words like "high risk", "private", "behavioural patterns", and "contact" feel like they refer to definite things. The natural move is to accept those words and reason from there. You ask whether using the system is justified, whether the contact is appropriate, whether the risks are too high. The words themselves stay fixed in the background.
But each one is slipperier than it looks, and the slipperiness all goes one way: it expands what the school is allowed to do.
Take "high risk". It sounds like a finding the system makes. It is actually a decision someone makes. Risk of what (severe depression, self-harm, suicide attempts)? These are different things with different time-scales and signals. And where do you draw the line (top 5%, top 10%, top 20%)? Each cutoff picks out a different group of students. The case treats this as a technical decision the model takes care of. It is actually a political call about who gets watched.
Take "private". The case reassures you that the system "does not access private messages." That sounds like a privacy protection. But "private" here is being defined narrowly, by the format of the data. It says nothing about what can be inferred from the data. Login times, submission patterns, and posting frequency, taken together over months, can paint a picture more revealing than any single private message. Calling the system "non-private" because it does not read messages is like saying you are not reading someone's mail because you only watch what packages they receive, when they leave the house, who visits, and how late they stay up. A stronger argument does not just use the case's words back at the case.
4. Assumption — what the case quietly takes for granted
This chapter is about hidden assumptions in the case: things the case treats as obvious, without ever telling you they are choices being made.
When you read the case, your mind quickly fills in a complete picture. A school worried about its students. An AI that scores their digital activity. Counsellors reaching out to the ones flagged as high risk. Some students reasonably worried about privacy. The picture feels finished. The natural move is to argue about whether this picture is good or bad on balance, whether the safety gain is worth the privacy cost.
But that picture has things in it that the case never actually defended. They were just slipped in quietly, as if they were not even choices.
Here is the big one. The case treats it as obvious that being identified by the system means someone from the school has to act. Why? Why does flagging a student have to lead to a counsellor email or a required meeting? Imagine instead a system that shows students their own pattern privately ("your sleep, attendance, and submission rhythm look different from your usual baseline") and lets students decide what to do. Same data, same algorithm, no school contact. Suddenly the worry about being watched, and the worry about being labelled, both look very different, because the school is no longer the actor. The case never tells you it is making the school-must-act assumption. It just builds the assumption into the wording, and your mental picture inherits it. A stronger argument is the one that asks: what is the case treating as obvious that it should actually be defending?
5. Inference — same facts, different conclusions
This chapter is about why people read the same case and end up with opposite answers, and how to notice which starting point you are using.
When you think about this case, you can quickly come up with five or six different reasons for the same conclusion. They sound like a long list, but most of them come from one or two underlying ways of thinking about what matters.
There are roughly three of those starting points, and they pull in different directions.
The first is outcomes-based. Add up the consequences. If the system saves more lives than it harms, do it; if not, don't. Whenever you hear "save lives" given as the reason for the system, that is outcomes-based reasoning. The trouble is, the case does not actually give you the numbers, and as the Evidence chapter pointed out, the numbers may not even exist in principle.
The second is rights-based. Some things should not be done to a person without consent, regardless of outcome. From this angle, even if the system saved lives, it is not okay to scan students' digital activity and act on the results without telling them and getting their agreement. Privacy and consent are not just costs to weigh. They are conditions before the system can be done at all.
The third is care-based. Real care needs the person being cared for to want it, not just the person giving the care to decide on it. Acting on a student because the algorithm flagged her, when she did not share that information and did not ask for help, may not actually be care. It can be intrusion dressed up as care.
Here is the move that is easy to miss. These three approaches do not all converge into one wise answer. They reach different answers for principled reasons. If you list the three and then announce "weighing all of these, I conclude...", you have quietly given more weight to one of them without saying which. A stronger argument names which starting point it is using, and engages the strongest version of one that pushes back against it.
6. Implication — what comes after the first effect
This chapter is about what happens later with this system, after the first round of effects. It is the part that is easy to forget when you first read the case.
When you first think about this system, the easy thing to picture is the immediate effects. Some students get flagged, some get contacted, some find it helpful, some find it intrusive. The natural move is to judge the system from that first picture, in the first weeks after deployment.
But the most important effects of a system like this often start later, after students learn the system exists and start adjusting to it.
Once students know that login times, submission timing, and forum activity are being watched as signs of mental state, behaviour drifts. Students who want to look fine learn to look fine. They log in regularly, submit on time, post upbeat updates. Students who are genuinely struggling but do not want to be flagged learn to hide better, to keep their patterns inside the "normal" range even while their actual situation gets worse. Now look at what has happened to the system. The data it sees is no longer behaviour produced naturally. It is behaviour shaped by the knowledge that the system exists. After a few cycles, the system is not detecting students in trouble. It is detecting students who failed to learn the new presentation rules.
There is a second later effect that is even easier to miss. Deploying this system establishes that "using AI to look at students' digital activity to act on them for their own good" is the kind of thing universities do. In five or ten years, when someone proposes using AI to detect academic dishonesty or identify students at risk of dropping out, the new proposal will not start from scratch. It will start from "we already do this for mental health, how is this different?" Today's deployment quietly spends the political ground for opposing future systems. A stronger argument follows the system past its first month.
7. Point of view — who is in the room, who is not
This chapter is about who the case talks about and who it forgets to mention, and why the people it forgets are often the most important ones.
The case sets it up as a debate between two groups: the school and the students. "The university argues...students and staff are concerned." So when you write about it, the natural thing is to line up the school's reasons against the students' concerns and try to balance them.
But "the school" and "the students" are each shorthand for many different people with different positions, and several important voices are not in the case at all.
Take "the school" first. It is at least five different roles, each pushing for the system for different reasons:
- A vice principal worried about a scandal on her watch.
- A counselling director who has seen real crises and wants any tool that might help.
- An IT team finally with a project to build.
- A legal office wanting a paper trail in case something goes wrong.
- A communications office wanting a story to tell.
None of them is the villain. Each makes a sensible decision in her own role. The system that comes out is something no one alone would have chosen.
Now look at "the students". They are at least three different positions, and these positions actually conflict with each other:
- Some genuinely want the school to notice them, and would be relieved to be flagged.
- Some would rather decide for themselves when to ask for help, and would feel being flagged is an intrusion.
- Some are most likely to be wrongly flagged and most exposed to harm from being flagged at the same time.
You cannot design the system to satisfy all three groups at once.
Now think about who is not in the case at all:
- Students who got flagged but never told. They have no way of speaking about their situation because they do not know they have one.
- Future students who enrol after the system is deployed and never had a chance to opt in.
- The student's own future self, who might apply for insurance ten years later when the "high-risk" record from age 19 still exists somewhere in the school's data.
A stronger argument splits each group apart and brings in at least one perspective the case forgot to mention.
8. Question — what question are you actually answering?
This chapter is about the question the case asks at the end, and how that question quietly decides what kinds of answers feel possible.
The case ends with: "You are asked to evaluate whether the use of this AI system is ethically acceptable." That sounds like a clear question. So the natural thing is to accept it and try to answer it. You give a yes-with-conditions, or a no, or a balanced "it depends on the safeguards" answer.
But the question itself is doing more work than is easy to notice, and naming that is itself a strong move.
The case's question can be answered at two different depths, and each one leads to a different kind of argument. The surface depth: "How can we make this system acceptable?" That is the level of technical fixes. Add an opt-out. Set up an appeals process. Improve the false-positive rate. Train counsellors better. Your argument lives here when you take the system as given and ask how to soften its edges. The deeper depth: "Should this kind of system exist at this school at all?" At this depth you are questioning the project itself, not just its execution. Is "early detection of mental health risk through behavioural inference" the right shape for a school's response to student wellbeing in the first place? Your strongest move is usually to land at the second depth, but only after knowing you made that choice and being able to say why.
There is also something hidden in the word "acceptable." Acceptable to whom? Not to a flagged student, who is not in a position to accept or refuse. Not to a future student, who has not enrolled yet. The question "is this acceptable?" can only really be asked by the people who decide whether to deploy. Accepting the case's question quietly puts you in their position. A stronger argument notices which depth it is answering at, and from whose position.