Post
2215
Benchmarking xLLM and Specialized Language Models: New Approach & Results https://mltblog.com/4nzaKUb
Standard benchmarking techniques using LLM as a judge have strong limitations. First it creates a circular loop and reflects the flaws present in the AI judges. Then, the perceived quality depends on the end user: an enterprise LLM appeals to professionals and business people, while a generic one appeals to laymen. The two have almost opposite criteria to assess the value. Finally, benchmarking metrics currently in use fail to capture many of the unique features of specialized LLMs, such as exhaustivity, or the quality of the relevancy and trustworthiness scores attached to each element in the response. In fact, besides xLLM, very few if any LLMs display such scores to the user.
I now discuss these points, as well as the choice of test prompts, and preliminary results about xLLM, compared to others.
-- Structured output vs standard response --
A peculiarity of xLLM is that if offers two types of responses. The top layer is the classic response, though much less reworded than in other systems to keep it close to the original corpus, and well organized. The layer below — we call it the structured output — is accessible to authorized end users via the UI; it displays clickable summary boxes with raw extracts and contextual elements (title, category, tags, timestamp, contact person and so on). It also shows relevancy and trustworthiness scores:
➡️ Trustworthiness score: it tells you how trustworthy the input source is, for each summary box. In particular, if the same information is found in two different input sources but with a mismatch, the trustworthiness score tells you which one is most reliable.
➡️ Relevancy score: it tells you how relevant a summary box is to your prompt.
The structured output provides very precise links to where the information is coming from. Also, models based mostly on [...]
Read full article at https://mltblog.com/4nzaKUb
Standard benchmarking techniques using LLM as a judge have strong limitations. First it creates a circular loop and reflects the flaws present in the AI judges. Then, the perceived quality depends on the end user: an enterprise LLM appeals to professionals and business people, while a generic one appeals to laymen. The two have almost opposite criteria to assess the value. Finally, benchmarking metrics currently in use fail to capture many of the unique features of specialized LLMs, such as exhaustivity, or the quality of the relevancy and trustworthiness scores attached to each element in the response. In fact, besides xLLM, very few if any LLMs display such scores to the user.
I now discuss these points, as well as the choice of test prompts, and preliminary results about xLLM, compared to others.
-- Structured output vs standard response --
A peculiarity of xLLM is that if offers two types of responses. The top layer is the classic response, though much less reworded than in other systems to keep it close to the original corpus, and well organized. The layer below — we call it the structured output — is accessible to authorized end users via the UI; it displays clickable summary boxes with raw extracts and contextual elements (title, category, tags, timestamp, contact person and so on). It also shows relevancy and trustworthiness scores:
➡️ Trustworthiness score: it tells you how trustworthy the input source is, for each summary box. In particular, if the same information is found in two different input sources but with a mismatch, the trustworthiness score tells you which one is most reliable.
➡️ Relevancy score: it tells you how relevant a summary box is to your prompt.
The structured output provides very precise links to where the information is coming from. Also, models based mostly on [...]
Read full article at https://mltblog.com/4nzaKUb