The developer of the Veynrel plugin for Obsidian conducted a study on LLM performance when processing structured notes. The test involved 360 requests across three models, using six variations of training and nutrition notes, including prompt injection attempts. The experiments aimed to evaluate data retention and the impact of instructions on output accuracy.
Results showed that models frequently lost context, such as the word 'today' in workout logs, even when explicitly instructed to preserve relative time. In 56 out of 60 clean workout inputs, the Qwen model failed to include the term 'today'. Furthermore, models occasionally hallucinated data, such as assigning specific volumes to food items that were not specified in the source text.
The study also highlighted technical constraints, specifically latency. 11.7% of Qwen requests exceeded the plugin's initial 30-second timeout, with the longest response taking 65.92 seconds. Consequently, the developer increased the timeout to 90 seconds in version 2.3.1 of the plugin to ensure complete data retrieval.
Total costs for the 360 requests amounted to approximately $0.04. The developer noted that while tables generated by the models often appeared complete, they frequently contained errors such as missing information, substituted measurements, or unauthorized additions, necessitating manual verification of all AI-generated outputs.
