TriEval: A Resource-Efficient Pipeline for LLM Bias, Toxicity, and Truthfulness Assessment

Abstract

Large Language Models (LLMs) have transitioned from simple conversational agents to the foundational infrastructure of the artificial intelligence landscape, now integral to critical sectors such as healthcare, education, and public administration. This widespread integration underscores the urgent need for ongoing assessment to guarantee these systems operate safely and fairly. Post-deployment challenges frequently manifest as erratic responses and the generation of factual inaccuracies, or hallucinations. While various evaluation frameworks are currently available, they often suffer from significant limitations: many are restricted to analyzing individual parameters in isolation, while others demand substantial computational power that remains out of reach for the majority of the research community.

TriEval was developed to overcome these hurdles by offering a holistic evaluation method that simultaneously assesses bias, toxicity, and truthfulness within LLM outputs, all while optimizing for computing efficiency. Designed for accessibility, the pipeline functions seamlessly on standard laptops without the need for GPU clusters, supporting both open- and closed-source model architectures. The framework’s efficacy was demonstrated through testing on four distinct models: Llama 3 8B, Mistral 7B, Gemma 2 9B, and Claude Haiku. The analysis revealed significant disparities between open-source and closed-source systems, particularly regarding their handling of toxicity and factual accuracy. To foster inclusivity in AI research, TriEval is being released as an open-source tool, providing scholars with limited computational resources a robust means to evaluate model performance.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC