NOTE: This repository does NOT have all the files we ran on the Zeus2Q robot, as the Zeus2Q codebase included several large files (Llama model files, dependencies, etc.) totaling 25 GB and every file could not be exported to Github. All of the files we changed, modified, or created in any capacity have been included in this repository, and all the other files can be found in the default software system for Zeus2Q.
To execute our code on a Z2Q robot, run the Zeus2Q.py file.
Manned space missions often have small crews, severely limiting astronauts' physical social interactions and meaningful relationships over long periods of time. As deep-space missions to Mars and beyond become increasingly frequent, communication delays can . With these hazards - a closed environment, several stressful responsibilities, and the isolation and unfamiliarity in a place away from Earth - astronauts are at a much higher risk for depression and anxiety. Previously, NASA has reported that crew members have had hallucinations on missions, made mistakes due to frustration and fatigue, and been at risk for harming themselves and others. Therefore, our team reasoned that while astronauts can do nearly every task on their own, they need a kind companion to talk with them and ensure their mental well-being: AstroMax, a robot inspired by Disney's Baymax!
AstroMax is a humanoid robot that uses machine learning and computer vision to analyze facial expressions and determine the emotions of people it's talking to. Based on visual analysis, AstroMax sends data to a micro-controller which displays the "astronaut's" emotional state on a Liquid Crystal Display (LCD). After classifying a person's emotional state, AstroMax gives grounding words of advice and support, and continues the conversation with the user. From there, it analyzes an individual's tonal cues to better understand stress and fatigue levels of "astronauts", and provides more in-depth responses to the astronaut's emotional state.
AstroMax can also take measures to reduce loneliness and provide extra support, such as playing back a recording from "family members or friends back on Earth" or giving astronauts an encouraging fist bump. Specifically for movement, we implemented a custom algorithm to smooth out the physical actions of our Zeus2Q robot.
To effectively meet the challenge of helping astronauts maintain a stable psychological state across long-term, isolating space missions, we started by outlining the engineering requirements of our solution: emotion-centric context awareness, human-mimicking interaction, and effective communication.
On the emotional-awareness side of the problem, we created a facial emotion classifier pipeline using DeepFace, OpenCV, and the Zeus2Q humanoid robot's camera. We also implemented a tone analysis feature that records speech, splices it into 88 features through an OpenSmile pipeline, and categorized the emotion through a threshold approach. AstroMax is capable of categorizing 7 different emotions.
Our approach to making the robot more human-like involved a combination of biomimetic gestures and endearing behaviors. Through a custom, joint-space motion planning model, we used a gradient descent trajectory plan over a virtual potential function to manage nullspace redundancies caused by the high DoF humanoid design, avoid hitting joint singularities, and make more natural motions. We coupled this approach with an arctan() rate saturator to dampen whole-body accelerations and create smoother, more fluid movements. By creating a fading expression for the posture term in the virtual potential function, we ensured that goal achievement increases in priority over the course of a movement, balancing the needs of movement functionality and human-like aesthetic. These movements were applied to make human-like gestures like fist bumps.
Complementing these motions, we prompt engineered an LLM to make personalized jokes and behave compassionately toward the “astronaut”
Motion Algorithm: High degree of freedom system with lots of motion parameters to tune, combined with mechanical restrictions to motions (i.e. certain joints could not be moved unless others were moved out of the way). We had no sensor data about the joint positions, and thus could not leverage a closed loop controller for movements. Additionally, lower level servo code on the humanoid robot created variable latency between command signals and their implementation. These issues were systematically worked through with a combination of real-time position tuning using Maestro, command signal printouts for response tuning, posturing and rate limiters to mitigate latency. The tuning process in particular for the motions proved particularly time-consuming.
Computer Vision Pipeline & Llama LLM: Generating responses based on the Llama LLM took several seconds and even minutes; the entire system was extremely laggy and had a massive delay before the auditory output from the robot.
Tonal Analysis: The library we used for analysis of user voices and tone, OpenSMILE, extracted 88 features from audio files, from pitch and volume to jitter and shimmer. A challenge from our team was understanding which of these features was most important for understanding a person's emotional state when speaking, and how to find out how much feature values would vary based on different emotional states; often, the emotional state of the tone was a very sensitive (1-2 point) numerical feature change. Another challenge was understanding changes to the user's voice as opposed to all the background noise detected by the robot, especially because we were constrained to working in an area where multiple other teams were working as well.
Functionality/Use Case: As we received an (amazing!!) pre-programmed Zeus2Q STW humanoid robot with several functions designed for educational purposes, there were several functions in the Zeus2Q code-base that executed automatically and were not applicable to our use case. This meant we had to understand the existing configuration of the Zeus2Q code and remove/modify parts to fit our use case (so the robot would not randomly speak on its history, for example) without breaking the code-base, another challenge for us.
Arduino Integration: When we initially integrated the Arduino the humanoid robot wasn’t sending signals correctly. We realized each time we replug the Arduino we have to check and update the port name it is plugged into to correctly send information. Additionally, when we first turned on the LCD display you couldn’t see the words being printed. To solve this we included a potentiometer to decrease the back light of the LCD.
Despite the complexity of integrating hardware, machine learning, and real-time interaction under tight time constraints, we successfully built an end-to-end intelligent humanoid companion. We are proud of designing AstroMax with functionality to be able to perceive users and their input, analyze them, and act physically or vocally upon the knowledge it gained.
As a team with no prior experience working with AI/ML models, successfully integrating machine learning into a real-time humanoid robotic system was a major accomplishment. We moved from limited familiarity with computer vision and audio-based analysis to deploying models that could meaningfully interpret facial expressions and vocal cues and influence the robot’s behavior. We are also especially proud of stabilizing the robot’s physical motion by tuning a high degree of freedom motion profile. Given our initial challenges with the motion planner (rigid gestures), successfully executing a smooth motion that could establish a more authentic connection between the user and the robot was a big accomplishment for our team. Finally, working within a large existing codebase presented constant challenges, with background functions interfering with new logic. Achieving a reliable, end-to-end interaction, where a human could initiate a conversation, be analyzed, and receive a coherent verbal and physical response from the robot, was a significant milestone for our team.
We learned about the capabilities and challenges posed by high degrees of freedom, humanoid robots; we had minimal experience with humanoid robots entering the competition. We also learned about how to structure “adaptive” LLM models within a logic framework to combine predefined procedural code with the advantages posed by AI. The unique motion planning restrictions taught us how to create biomimetic movement without use of geometric, Jacobian-centered inverse kinematics.
Taking AstroMax from a prototype to a production-grade idea involves increasing the sophistication of the system. One such approach is the use of Retrieval Augmented Generation (RAG) to allow ground stations to upload manuals and procedures like the DSM 5 for interpretation and search by the AI, in-the-moment. To further increase how engaging AstroMax is, its motion capacity could be used to allow it to play board games with astronauts or even walk for extended periods of time. The ability to leverage the LLM for longer chats with users could yet still improve the adaptability.