∷ TIAGoGPT: Exploring the Potential of LLMs in Tiago Robot-Assisted Interventions for Supporting Children with Social Skill Difficulties
Date: August 2023
This robotic interactive system, TIAGoGPT is an MSc research project on "Exploring the Potentials of LLMs in TIAGo Robot-Assisted Interventions to Support Children with Social Skill Difficulties". The project implements cutting-edge technologies to revolutionise the field of child support. We employed state-of-the-art large language models in robotic assistive interventions for children with social skill difficulties. This means that the project system is deployed on a robot, and as part of our research study, children are expected to interact with the robot to enable us reimagine the potentials of this robot.
The aim of exploring the potential of Tiago robot-assisted interventions for supporting children in need project is to validate the potential of the Tiago robot in developing and implementing a robot-assisted intervention system that utilises voice transcription, generative text response, and text synthesis capabilities to assist children with social skill difficulties, such as autism spectrum disorder (ASD). By integrating voice transcription technology, a large language model (LLM) such as the generative pretrained transformer (GPT), and voice synthesis capabilities, the project aims to create a reliable and personalised response generator within the Tiago robot.
The project specific objectives are:
- To develop a robust and reliable voice transcription module that accurately converts spoken prompt into text format using the Whisper large-v2 model API from OpenAI.
- To implement a response generation module utilising the generative pretrained transformer model (GPT-3.5-turbo) API to generate appropriate and contextually relevant responses based on the transcribed input.
- To integrate the Google Text-to-Speech API to synthesise the generated responses into natural-sounding speech.
- To design a user-friendly graphic interface to facilitate seamless interaction and the display of synthesised responses.
- To incorporate a responsible AI implementation culture by utilising a digital consent form to create awareness of the data policy applicable to the usage of the developed system.
- To deploy the developed AI systems onto the TIAGo elite robot.
- To evaluate and test the AI systems deployed on the TIAGo elite robot.
- Voice Recording and Processing Module
- Voice Transcription Module - Whisper Model
- Response Generation Module - GPT3.5-Turbo
- Text Synthesis Module - GoogleCloud TTS
- Dual Conversation Mode Module - Text-based and Voice-based
- Storage Management Module - Database and File
- Response Displaying and Sounding Module
- Consent handling and managing Module
- Data Security Framework
- File Access Control
- Audit Log
- Thread Management
- Unit Testing Framework
- Automatic Coversation Loop
- Personalised User Interaction
Milestone for the 3rd Month August 2023
- Incorporating face detection a. to identify previous user b. to welcome previous user c. for gesture detection
- Ending each session when the user say "I'm done for now" or "Goodbye".
- Log
Milestone for the 3rd Month August 2023
- Implemented data handling security: hashing and encryption.
- Improved on the structure of codebase.
- Implemented app audit logs.
- Implemented file permission and access controls.
- Log data integrity.
- Security file deletion
From the Initial system design and the flowchart, we have the following section implemented in this version release:
- TiagoGPT starts conversation with greetings
- TiagoGPT greetings upgraded. Automatic conversation improved
- MVC implementation structured adopted and migration started
- Bug with checkboxes fixed in dark orange skin
- Saved Chat history implemented
- Saved consent implemented
- Unit testing framework implemented
- Bug with 'listening' and 'waiting' buttons fixed
- TiagoGPT starts conversation with greetings
- Automatic conversion implemented
- Thread handlers implemented
- Special Token implementation for data integrity protection
- Changed how conversation are displayed
- Added third-party image and logo Disclaimer
- Updated menu list
- Bug fixed in the dual conversion mode switching
- Package requirement listed
- Bug with voice mode recording fixed. Each voice input now requires clicking the start button to commence recording.
- User personalisation using their names instead of generic name
- Speech Recognition and Processing Module
- Dual Conversation mode Module - Text-based and Voice-based
- Voice transcription Module - Whisper Model
- Response generation Module - GPT3.5-Turbo
- Text Synthesis Module - GoogleCloud TTS
- Response Displaying and Sounding Module
- Consent handling and managing Module
- Storage management Module
- In your project folder, create a virtual environment with Python 3.9+
python -m venv virt-env
- Activate the virtual environment by
source virt-env/Scripts/activate
- Install the required dependencies by
pip install -r Requirements.txt
- Edit the openAI API file file with your openai api key
- Edit the google cloud TTS file file with your .json api obtained from GoogleCloud
- Run the TIAGoGPTAppLauncher.py file
- In your project folder, create a virtual environment with Python 3.9+
sudo apt install python3-pip
python3 -m venv virt-env
- Activate the virtual environment by
source virt-env/bin/activate
- Install with the packages in requirement.txt as shown below:
pip3 install -r Requirements.txt
- Edit the openAI API file file with your openai api key
- Edit the google cloud TTS file file with your .json api obtained from GoogleCloud
- Run the TIAGoGPTAppLauncher.py file
We appreciate the open source of the following projects in which this code is influenced by: