Evaluation of an LLM-based Chatbot

One relevant academic paper that evaluates an LLM-based chatbot is a study on AI chatbots for mental health support published in the Journal of Artificial Intelligence and Autonomous Intelligence.(DOI: 10.54364/JAIAI.2024.1105)

The paper investigates the effectiveness of a chatbot designed as a mental health coach. The evaluation was conducted using a User Experience Questionnaire (UEQ), which measures dimensions such as efficiency, dependability, stimulation, and novelty. The results show that users found the chatbot engaging and helpful, particularly in providing motivational and supportive responses. However, slightly lower scores in efficiency and dependability indicate limitations in maintaining consistent conversational flow .

I selected this paper for three main reasons. First, it clearly involves a large language model-based chatbot in a specific context (mental health support), which aligns with the assignment requirements. Second, the paper includes a substantive evaluation using a structured questionnaire (UEQ) rather than merely describing the system. This makes the findings more reliable and measurable. Third, the evaluation considers multiple UX dimensions, allowing for a more comprehensive assessment of chatbot effectiveness.

This paper is particularly valuable because it demonstrates how LLM-based chatbots can positively influence user experience while also highlighting practical limitations. It provides a balanced perspective on both the strengths and weaknesses of chatbot systems in real-world applications.

Reference

Pan, X. (2024). Evaluation of AI-driven chatbots for user experience and task effectiveness. Journal of Artificial Intelligence and Autonomous Intelligence. https://doi.org/10.54364/JAIAI.2024.1105

评论

发表评论

此博客中的热门博文

Comparing Foodie San vs. Genki Sushi Singapore Apps

图片
Ever tried to order yakiniku in Singapore and felt like you needed a treasure map? That’s Foodie San for you. In contrast, Genki Sushi Singapore feels like your sushi is already waiting on a silver platter. Let’s break down why. I will show 3 top diferences below: 1. Interface Clarity - The name and icon of foodie san have nothing to do with yakiniku like, so every time I think about which app I want to go to for a long time. The name and icon of genki sushi correspond very well. 2. Voucher/Rewards Access - Foodie San displays only a stamp card when opening the rewards section, forcing users to navigate further to see actual vouchers, which can easily be overlooked. Genki Sushi, however, immediately shows all available vouchers, ensuring users don’t miss any benefits. 3. Take-Out Ordering – Foodie San does not include take-out ordering in the app, even though the service exists, forcing users to rely on external delivery apps like Grab. This adds unnecessary steps and reflects poor u...
阅读全文

Dark Patterns in Food Mobile Applications

图片
Dark patterns are interface design strategies that manipulate users into making decisions that benefit the company rather than the user. These patterns reduce transparency and limit users’ ability to make informed choices. Two common examples are hidden costs and difficult subscription cancellation. Example 1: Hidden Delivery Fees Many food delivery applications such as Foodpanda display food prices when users browse restaurant menus, but additional costs such as delivery fees, service fees, or small-order fees only appear at the final checkout stage. This design is considered a dark pattern known as Hidden Costs . Users may spend time browsing menus and selecting items while believing the total price is lower than it actually is. When the additional charges appear at checkout, users may still proceed with the order because they have already invested time in the process. A more ethical design would clearly display all expected costs earlier in the ordering process . For example, the ap...
阅读全文

Welcome to Taste & Touch

图片
Hello! I’m Hua, a Master’s student in Information Systems in NTU and a passionate food lover. I enjoy exploring different food applications, from ordering meals to discovering new restaurants. At the same time, I am deeply interested in user experience design and how digital interfaces shape our daily interactions. Food apps have become an essential part of modern life. Whether ordering delivery, browsing menus, or reading reviews, users rely heavily on these applications. A well-designed food app makes the process fast, intuitive, and enjoyable. In contrast, poor design can lead to confusion, frustration, and even abandoned orders. Through this blog, I will analyze food-related applications from a UX perspective. I will evaluate how these apps perform in terms of usability goals such as learnability, efficiency, feedback, and user satisfaction. I will also identify design strengths and weaknesses and suggest improvements. By combining my interest in food and user experience design, th...
阅读全文