This repository contains the machine learning code of the multimodal XR Hand Tracking system for Extended Reality Musical Instruments (XRMIs), as described in our research paper "Combining Vision and EMG-Based Hand Tracking for Extended Reality Musical Instruments". Included are files for processing and analyzing sEMG (surface electromyography) data using the Myo armband with Python. The code is part of a research project at Queen Mary University of London, which aims to develop a multimodal hand tracking system for XRMIs.
Note: The Unity implementation for the multimodal hand tracking system can be found here.
data/: Training data from several recording sessions.main_rnn.py: The main training loop for the RNN model.exporter_pytorch_to_onnx.py: Script to export the trained PyTorch model to ONNX format.requirements.txt: List of dependencies.DataModule.py: Data handling module for the project.hyperparameters.py: Hyperparameters configuration.README.md: Documentation and setup instructions (this file).
- Git
- Python (version >= 3.8)
-
Clone the repository:
git clone https://github.com/maxgraf96/sEMG-myo-python -
Navigate to the cloned directory:
cd sEMG-myo-python -
Ensure that you have the correct version of Python installed (>= 3.8).
-
Install the required dependencies:
pip install -r requirements.txt
To run the main training loop, execute the following command:
python main_rnn.py
After training the model, if you wish to export it to ONNX format, run:
python exporter_pytorch_to_onnx.py
Feel free to fork the repository, make changes, and submit pull requests. For major changes, please open an issue first to discuss what you would like to change.
- Max Graf - max.graf@qmul.ac.uk
- Project Link: https://github.com/maxgraf96/sEMG-myo-python
If you use this work, please cite
@misc{graf2023combining,
title={Combining Vision and EMG-Based Hand Tracking for Extended Reality Musical Instruments},
author={Max Graf and Mathieu Barthet},
year={2023},
eprint={2307.10203},
archivePrefix={arXiv},
primaryClass={cs.CV}
}