The Department of Education’s Office for Civil Rights (OCR) has issued new guidance cautioning that the use of artificial intelligence (AI) in schools could lead to discriminatory practices against non-white and transgender-identifying students.

The guidance stems from an executive order signed by President Joe Biden last year, which directed the Department of Education to develop policies and guidance ensuring AI is used responsibly and without discrimination. The order emphasized AI’s potential impact on “vulnerable and underserved communities.”

“The growing use of AI in schools, including for instructional and school safety purposes, and AI’s ability to operate on a mass scale can create or contribute to discrimination,” the guidance stated. “This resource provides information regarding federal civil rights laws in OCR’s jurisdiction and includes examples of types of incidents that could, depending on the facts and circumstances, present OCR with sufficient reason to open an investigation.”

The OCR document outlines scenarios in which AI might result in biased outcomes. For instance, it highlights the use of plagiarism detection tools powered by generative AI, which may exhibit “a high error rate when evaluating essays written by non-native English speakers.” The guidance warned that such errors could be perceived as discriminatory and could prompt a federal civil rights investigation, according to a report by Fox News.

Another example cited involves AI being used to determine disciplinary measures for students. The guidance warns that racial bias could emerge from these systems, particularly if the AI relies on historical data reflecting past discriminatory practices.

“Significant disparities by race have persisted in the school’s application of student discipline, and Black students are disciplined more frequently and more harshly than other similarly situated students of another race,” the guidance said. “As a result, the historic school discipline data that the software relies on reflects the school’s discriminatory disciplinary practices.”

The Biden administration has made combating discrimination by AI a priority. Last year, federal agencies such as the Consumer Financial Protection Bureau and the Equal Employment Opportunity Commission signed a pledge saying that they would use existing federal civil rights and consumer protection laws in order to crack down on AI systems deemed discriminatory. 

This piece first appeared at TPUSA.