The developer of the Charoit assistant tested the mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 model to identify contradictions and duplicates in meeting protocols. The model was converted to ONNX format, taking up 1.1 GB, and requires 1.5 seconds to load. Despite its ability to identify paraphrases, the model struggled with contradictions, often misinterpreting different tasks as conflicting statements.
The author attempted to refine the model using a custom dataset of 16 pairs, but found that the model failed to handle long, complex paragraphs effectively. Consequently, the author concluded that simple string matching and direct transcript searches are more efficient for identifying contradictions. The NLI model was ultimately repurposed for deduplication within the assistant's live feed.
In the deduplication process, the model compares new points with the last 16 entries. A pre-filter using difflib with a 0.3 threshold discards unrelated entries before the NLI model evaluates the remaining pairs in both directions. If both probabilities exceed 0.8, the entry is marked as a duplicate. This remains the only daily use case for the model.
The author also implemented a revision system for meeting topics (cores) using bge-m3 for candidate selection and NLI for final judgment. However, the system was disabled on October 5 after it produced numerous false positives, often flagging different topics with identical template statuses as duplicates. The revision process consumed significant resources without providing actionable insights.
An attempt to use NLI as a gatekeeper for a quick-response feature failed to yield significant efficiency gains. The model correctly identified non-question inputs in only 1.7% of cases, with a precision of 55.6%. The author noted that silent failures, such as incorrect file path handling and resource prioritization issues, were the most significant technical challenges encountered.
