Welo Data, a leader in delivering exceptionally high-quality AI training data, announces the launch of its Model Assessment Suite, a research tool designed to enhance the performance of large language models (LLMs). The centerpiece of this launch is a novel, multilingual approach to Causal Reasoning Research, a high priority area for improving LLM performance. This innovative Assessment suite provides a framework for improving frontier models, particularly in addressing complex cause-and-effect relationships.

Model Assessment Suite Approach

The Model Assessment Suite offers a two-stage approach to model testing:

• Public Model Assessment – Public models are tested against a suite of proprietary benchmarks, with comprehensive recommendations for improving performance. These assessments are designed to provide actionable insights for enhancing model fine-tuning and reinforced learning from human feedback (RLHF).
• Private Model Assessment – Upon request, the suite can be extended to test non-public models. This private assessment allows for deeper insights, customized to each model's architecture and design. In addition, we provide you with a clear comparison to see how your model ranks against others—ensuring valuable metrics before release.

Tackling Causal Reasoning

Despite significant advances in LLMs, causal reasoning remains a difficult challenge. Most LLMs rely on their pre-training data to respond to prompts, often overgeneralizing or misidentifying causal relationships. To address this, Welo Data has developed a completely novel dataset and prompt design methodology, utilizing domain experts to generate fact-based, industry-specific scenarios.

Key features of causal reasoning assessment include:

• Novel human-generated datasets created by subject-matter experts to ensure uniqueness and specificity.
• Complex prompts are designed to test different aspects of causality, such as the discovery of causal and non-causal relationships and the ability to recognize normality violations in cause-and-effect relationships as well as the impact of language variation.
• Multilingual and cross-linguistic capabilities, evaluating LLMs in English, Arabic, Japanese, Korean, Spanish, and Turkish, with the flexibility to test causal reasoning across these languages as well as various evaluations across different language pair permutations.

“We worked closely with Dr. Larry Carin, a leader in Machine Learning from Duke University, to identify the most critical gaps in current LLM assessments,” said Dr. Fernando Migone, VP of Transformation and Head of Welo Data’s Research Lab. “Our research shows that causal reasoning tasks are particularly challenging for models, as they often lack the necessary training data or struggle with linguistic diversity. This assessment suite addresses these gaps, offering deeper insights into model performance that can guide future training and development,” said Dr. Migone.

A Focus on Future Challenges

Evaluating causal reasoning is just the first step in Welo Data’s roadmap for LLM assessment, led by its Research Lab. Future iterations will focus on multi-hop reasoning and additional components of causality, ensuring comprehensive coverage of complex reasoning tasks. These iterations will include new languages – with additional languages prioritized on request. The suite is also designed to scale, supporting models in multiple locales and industries.

Market Impact and Industry Feedback

In addition to Dr. Carin, Welo Data has been collaborating with senior data scientists and machine learning engineers from key foundation model builders. The consensus is clear: the Model Assessment Suite’s focus on causal reasoning is impactful for developers and researchers, providing valuable insights into one of the most challenging areas of AI development.

For more information about Welo Data’s Model Assessment Suite, visit welodata.ai.