Laya is a library for zero-shot decision-making and calibration, currently available in version 0.3.27. The tool allows developers to perform classification tasks without generating text, as it scores provided options instead of producing output tokens.
The library uses a specific checkpoint, convaiinnovations/laya, and operates by encoding each question as a separate row. A key design principle for developers is to prioritize single choice questions with multiple options over multiple yes/no questions, as the former is more computationally efficient.
In testing against the CLINC150 banking domain benchmark, Laya achieved an accuracy of 0.804 in a zero-shot setting using intent names and one-line descriptions. This performance was compared to a TF-IDF and logistic regression classifier, which required labeled examples to reach similar accuracy levels.
The system provides two confidence metrics: answer_confidence, representing the probability of the selected answer, and confidence, derived from normalized entropy. The library also includes mechanisms for temperature scaling to adjust probability distributions.
