{"id":39702,"date":"2025-06-15T11:15:00","date_gmt":"2025-06-15T11:15:00","guid":{"rendered":"https:\/\/www.lifeandnews.com\/articles\/?p=39702"},"modified":"2025-06-16T06:16:45","modified_gmt":"2025-06-16T06:16:45","slug":"protecting-the-vulnerable-or-automating-harm-ais-double-edged-role-in-spotting-abuse","status":"publish","type":"post","link":"https:\/\/www.lifeandnews.com\/articles\/protecting-the-vulnerable-or-automating-harm-ais-double-edged-role-in-spotting-abuse\/","title":{"rendered":"Protecting the vulnerable, or automating harm? AI\u2019s double-edged role in spotting abuse"},"content":{"rendered":"\n

Aislinn Conrad<\/a>, University of Iowa<\/a><\/em><\/p>\n\n\n\n

Artificial intelligence is rapidly being adopted to help prevent abuse and protect vulnerable people \u2013 including children in foster care<\/a>, adults in nursing homes and students in schools<\/a>. These tools promise to detect danger in real time and alert authorities before serious harm occurs.<\/p>\n\n\n\n

Developers are using natural language processing, for example \u2014 a form of AI that interprets written or spoken language \u2013 to try to detect patterns of threats, manipulation and control<\/a> in text messages. This information could help detect domestic abuse and potentially assist courts or law enforcement in early intervention. Some child welfare agencies use predictive modeling<\/a>, another common AI technique, to calculate which families or individuals are most \u201cat risk\u201d for abuse.<\/p>\n\n\n\n

When thoughtfully implemented, AI tools have the potential to enhance safety and efficiency. For instance, predictive models have assisted social workers<\/a> to prioritize high-risk cases and intervene earlier.<\/p>\n\n\n\n

But as a social worker with 15 years of experience researching family violence<\/a> \u2013 and five years on the front lines as a foster-care case manager, child abuse investigator and early childhood coordinator \u2013 I\u2019ve seen how well-intentioned systems often fail the very people they are meant to protect.<\/p>\n\n\n\n

Now, I am helping to develop iCare<\/a>, an AI-powered surveillance camera that analyzes limb movements \u2013 not faces or voices \u2013 to detect physical violence. I\u2019m grappling with a critical question: Can AI truly help safeguard vulnerable people, or is it just automating the same systems that have long caused them harm?<\/p>\n\n\n\n

New tech, old injustice<\/h2>\n\n\n\n

Many AI tools are trained to \u201clearn\u201d by analyzing historical data<\/a>. But history is full of inequality, bias and flawed assumptions. So are people, who design, test and fund AI.<\/p>\n\n\n\n

That means AI algorithms can wind up replicating systemic forms of discrimination<\/a>, like racism or classism. A 2022 study<\/a> in Allegheny County, Pennsylvania, found that a predictive risk model to score families\u2019 risk levels \u2013 scores given to hotline staff to help them screen calls \u2013 would have flagged Black children for investigation 20% more often than white children, if used without human oversight. When social workers were included in decision-making, that disparity dropped to 9%.<\/p>\n\n\n\n

Language-based AI can also reinforce bias<\/a>. For instance, one study<\/a> showed that natural language processing systems misclassified African American Vernacular English as \u201caggressive\u201d at a significantly higher rate than Standard American English \u2014 up to 62% more often, in certain contexts.<\/p>\n\n\n\n

Meanwhile, a 2023 study<\/a> found that AI models often struggle with context clues, meaning sarcastic or joking messages can be misclassified as serious threats or signs of distress.<\/p>\n\n\n\n

\"A
Language-processing AI isn\u2019t always great at judging what counts as a threat or concern. NickyLloyd\/E+ via Getty Images<\/a><\/figcaption><\/figure>\n\n\n\n

These flaws can replicate larger problems in protective systems. People of color have long been over-surveilled<\/a> in child welfare systems \u2014 sometimes due to cultural misunderstandings, sometimes due to prejudice. Studies have shown that Black and Indigenous families<\/a> face disproportionately higher rates<\/a> of reporting, investigation and family separation compared with white families, even after accounting for income and other socioeconomic factors.<\/p>\n\n\n\n

Many of these disparities stem from structural racism<\/a> embedded in decades of discriminatory policy decisions, as well as implicit biases and discretionary decision-making by overburdened caseworkers.<\/p>\n\n\n\n

Surveillance over support<\/h2>\n\n\n\n

Even when AI systems do reduce harm toward vulnerable groups, they often do so at a disturbing cost.<\/p>\n\n\n\n

In hospitals and elder-care facilities, for example, AI-enabled cameras have been used to detect physical aggression between staff, visitors and residents<\/a>. While commercial vendors promote these tools as safety innovations, their use raises serious ethical concerns<\/a> about the balance between protection and privacy.<\/p>\n\n\n\n

In a 2022 pilot program in Australia<\/a>, AI camera systems deployed in two care homes generated more than 12,000 false alerts over 12 months \u2013 overwhelming staff and missing at least one real incident. The program\u2019s accuracy did \u201cnot achieve a level that would be considered acceptable to staff and management,\u201d according to the independent report.<\/p>\n\n\n\n

\"A
Surveillance cameras in care homes can help detect abuse, but they raise serious questions about privacy. kazuma seki\/iStock via Getty Images Plus<\/a><\/figcaption><\/figure>\n\n\n\n

Children are affected, too. In U.S. schools, AI surveillance like Gaggle<\/a>, GoGuardian<\/a> and Securly<\/a> are marketed as tools to keep students safe. Such programs can be installed on students\u2019 devices to monitor online activity and flag anything concerning.<\/p>\n\n\n\n

But they\u2019ve also been shown to flag harmless behaviors \u2013 like writing short stories with mild violence, or researching topics related to mental health. As an Associated Press investigation<\/a> revealed, these systems have also outed LGBTQ+ students<\/a> to parents or school administrators by monitoring searches or conversations about gender and sexuality.<\/p>\n\n\n\n

Other systems use classroom cameras and microphones to detect \u201caggression.\u201d But they frequently misidentify normal behavior<\/a> like laughing, coughing or roughhousing \u2014 sometimes prompting intervention or discipline.<\/p>\n\n\n\n

These are not isolated technical glitches; they reflect deep flaws in how AI is trained and deployed. AI systems learn from past data that has been selected and labeled by humans \u2014 data that often reflects social inequalities and biases<\/a>. As sociologist Virginia Eubanks<\/a> wrote in \u201cAutomating Inequality<\/a>,\u201d AI systems risk scaling up these long-standing harms.<\/p>\n\n\n\n

Care, not punishment<\/h2>\n\n\n\n

I believe AI can still be a force for good, but only if its developers prioritize the dignity of the people these tools are meant to protect. I\u2019ve developed a framework of four key principles for what I call \u201ctrauma-responsive AI.\u201d<\/p>\n\n\n\n

  1. Survivor control: People should have a say in how, when and if they\u2019re monitored. Providing users with greater control over their data can enhance trust in AI systems<\/a> and increase their engagement with support services, such as creating personalized plans to stay safe or access help.<\/li>
  2. Human oversight: Studies show that combining social workers\u2019 expertise with AI support improves fairness and reduces child maltreatment<\/a> \u2013 as in Allegheny County, where caseworkers used algorithmic risk scores as one factor<\/a>, alongside their professional judgment, to decide which child abuse reports to investigate.<\/li>
  3. Bias auditing: Governments and developers are increasingly encouraged to test AI systems<\/a> for racial and economic bias. Open-source tools like IBM\u2019s AI Fairness 360<\/a>, Google\u2019s What-If Tool<\/a>, and Fairlearn<\/a> assist in detecting and reducing such biases in machine learning models.<\/li>
  4. Privacy by design: Technology should be built to protect people\u2019s dignity. Open-source tools<\/a> like Amnesia, Google\u2019s differential privacy library<\/a> and Microsoft\u2019s SmartNoise<\/a> help anonymize sensitive data by removing or obscuring identifiable information. Additionally, AI-powered techniques, such as facial blurring, can anonymize people\u2019s identities in video or photo data.<\/li><\/ol>\n\n\n\n

    Honoring these principles means building systems that respond with care, not punishment.<\/p>\n\n\n\n

    Some promising models are already emerging. The Coalition Against Stalkerware<\/a> and its partners advocate to include survivors<\/a> in all stages of tech development \u2013 from needs assessments to user testing and ethical oversight.<\/p>\n\n\n\n

    Legislation is important, too. On May 5, 2025, for example, Montana\u2019s governor signed a law restricting state and local government from using AI to make automated decisions<\/a> about individuals without meaningful human oversight. It requires transparency about how AI is used in government systems and prohibits discriminatory profiling.<\/p>\n\n\n\n

    As I tell my students, innovative interventions should disrupt cycles of harm, not perpetuate them. AI will never replace the human capacity for context and compassion. But with the right values at the center, it might help us deliver more of it.<\/p>\n\n\n\n

    Aislinn Conrad<\/a>, Associate Professor of Social Work, University of Iowa<\/a><\/em><\/p>\n\n\n\n

    This article is republished from The Conversation<\/a> under a Creative Commons license. Read the original article<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Aislinn Conrad, University of Iowa Artificial intelligence is rapidly being adopted to help prevent abuse and protect vulnerable people \u2013 including children in foster care, adults in nursing homes and students in schools. These tools promise to detect danger in real time and alert authorities before serious harm occurs. Developers are using natural language processing, […]<\/p>\n","protected":false},"author":56,"featured_media":39703,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[5,30,8025,291,42,10,25,36,28,8],"tags":[16271,10656,196,885,891,886,860,3808,3394,4943,15439],"_links":{"self":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/39702"}],"collection":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/users\/56"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/comments?post=39702"}],"version-history":[{"count":1,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/39702\/revisions"}],"predecessor-version":[{"id":39704,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/39702\/revisions\/39704"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/media\/39703"}],"wp:attachment":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/media?parent=39702"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/categories?post=39702"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/tags?post=39702"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}