Multi-purpose instead of dedicated devices are changing how humans interact with them. Human machine interfaces (HMIs) are moving from keyboards, graphical interfaces, to advanced gesture recognition, voice control, and ultimately brainwave control. In the context of the brainwave project we have a challenging “team project” that deals with EEG signal processing, neural networks, and HW/SW programming. The goal is to develop a brainwave-controlled application. The app consists of thinking of a colored geometrical object and displaying it on a smartphone. The person playing the game wears a helmet with sensors capturing EEG signals that are then processed in real-time in a microcontroller (or in a FPGA) using a convolutional neural network (CNN). This CNN matches a specific brainwave pattern that identifies the object to be displayed. The results are then beamed via Bluetooth to the smartphone. The app running in the smartphone enables a graphical user interface for the thought object.
Project description
The desktop application that was made with the purpose of gathering EEG data sets was tested firstly just using the 3D accelerometer from the Mobita and plotting the data on the graph. This method was also used to check if the connectivity was good with the Mobita device. And during the project the desktop application was modified accordingly so that it will suit the experiment designed at that period of time. the experiment was changed a couple of times as the results that the project group was getting were not good and the CNN was not learning. The desktop application works perfectly and it is in coordination with the current experiment design which gave better results with the process of learning the CNN.
The CNN has also been designed and created successfully and the group made sure of that by testing it with a data base containing hand written numbers (from 0 to 9). The results of the CNN after learning with that set of data was of 97%. After gathering EEG data sets with the corresponding experiment design and putting those into the CNN, it was obvious that the CNN needed a lot more data sets than what the project group currently collected. The filter is running live as the Mobita is measuring the brainwaves. It works accordingly and removes with success the frequency below 0.5Hz and 50Hz. It gives understandable and visibly good data that can be further used in the learning process of the CNN.
The mobile game application was finished and it is able to receive the necessary inputs so that the game can be played it also offers an option for the user to visualize the corresponding incoming commands (up, down, base) in a graph. Since it was not possible to run the CNN and the filtering live on the phone and at the same time receive data from the Mobita, the solution to this was to do the processing of the data on the laptop and have the application connect to it wirelessly. The computer sends the commands of up, down or base to the mobile application after the EEG data was filtered and analyzed through the CNN.