Researchers used AI models based on the current neural networks (RNN) to read and accurately decode the amputee’s intent of moving individual fingers from peripheral nerve activities.
The AI models are deployed on an NVIDIA Jetson Nano as a portable, self-contained unit. With this AI-powered nerve interface, the amputee can control a neuroprosthetic hand with life-like dexterity and intuitiveness.
Watch the video
►Subscribe to my newsletter: http://eepurl.com/huGLT5
References
[1] Nguyen & Drealan et al. (2021) A Portable, Self-Contained Neuroprosthetic Hand with Deep Learning-Based Finger Control: https://arxiv.org/abs/2103.13452
[2]. Luu & Nguyen et al. (2021) Deep Learning-Based Approaches for Decoding Motor Intent from Peripheral Nerve Signals: https://www.researchgate.net/publication/349448928_Deep_Learning-Based_Approaches_for_Decoding_Motor_Intent_from_Peripheral_Nerve_Signals
[3]. Nguyen et al. (2021) Redundant Crossfire: A Technique to Achieve Super-Resolution in Neurostimulator Design by Exploiting Transistor Mismatch: https://experts.umn.edu/en/publications/redundant-crossfire-a-technique-to-achieve-super-resolution-in-ne
[4]. Nguyen & Xu et al. (2020) A Bioelectric Neural Interface Towards Intuitive Prosthetic Control for Amputees: https://www.biorxiv.org/content/10.1101/2020.09.17.301663v1.full
Video Transcript
00:00
in this video i will talk about a
00:02
randomly picked application of
00:03
transformers from the 600 new papers
00:06
published this week
00:07
adding nothing much to the field but
00:09
improving the accuracy by 0.1 percent on
00:11
one benchmark by tweaking some
00:13
parameters
00:14
i hope you are not too excited about
00:16
this introduction because that was just
00:18
to mess with the transformers recent
00:19
popularity
00:20
of course they are awesome and super
00:22
useful in many cases
00:23
and most researchers are focusing on
00:25
them but other things exist in ai
00:28
that are as exciting if not more you can
00:31
be sure
00:31
will cover exciting advancements of the
00:33
transformers architecture applied to nlp
00:36
computer vision or other fields
00:38
as i think it is very promising but
00:40
covering these new papers making slight
00:42
modifications to them is not as
00:44
interesting to me
00:45
just as an example here are a couple of
00:47
papers shared in march
00:49
applying transformers to image
00:51
classification and since they are all
00:53
quite similar and i already covered one
00:55
of them
00:56
i think it is enough to have an overview
00:58
of the current state of transformers
00:59
in computer vision now let's enter the
01:02
real subject
01:03
of this video which is nothing related
01:05
to transformers or even
01:06
gans in that case no hot words at all
01:09
except maybe
01:10
cyberpunk and yet it's one of the
01:12
coolest applications of ai
01:14
i've seen in a while it attacks a real
01:16
world problem and can change the lives
01:18
of many people
01:19
of course it's less glamour than
01:21
changing your face into an anime
01:23
character
01:23
or a cartoon but it's much more useful i
01:26
present you the portable
01:28
self-contained neuroprosthetic hand with
01:30
deep learning based
01:31
finger control by anguian drilan ital
01:34
before diving into it i just wanted to
01:36
remind you of the free nvidia gtc event
01:39
happening
01:40
next week with many exciting news
01:42
related to ai
01:43
and the deep learning institute giveaway
01:45
i am running if you subscribe to my
01:46
newsletter if you are interested i
01:48
talked about this giveaway with much
01:50
more details in my previous video
01:52
also i just wanted to announce that from
01:54
now on all new youtube members will have
01:57
a specific role on my discord channel as
01:59
a thank you for your support
02:00
now let's jump right into this unique
02:03
and amazing paper
02:04
this new paper applies deep learning to
02:06
a neuroprosthetic hand to allow
02:08
real-time control of individual finger
02:11
movements
02:11
all done directly within the arm itself
02:14
with
02:15
as little as 50 to 120 milliseconds of
02:18
latency
02:19
and up to 99 accuracy an arm amputee who
02:22
has lost his hand for 14 years can move
02:24
its cyborg fingers
02:26
just like a normal hand this work shows
02:28
that the deployment of deep neural
02:30
network applications embedded directly
02:32
on wearable biomedical devices is first
02:35
possible
02:36
but also extremely powerful here deep
02:38
learning is used to process and decode
02:40
nerve data acquired from the amputee to
02:43
obtain dexterous finger movements
02:45
the problem here is that in order to be
02:47
low latency this deep learning model
02:50
has to be on a portable device with much
02:52
lower computational power
02:53
than our gpus fortunately there has been
02:56
recent development of compact hardware
02:58
for deep learning users to fix this
03:00
issue
03:01
in this case they use the nvidia jetson
03:03
nano module
03:04
specifically designed to deploy ai in
03:07
autonomous applications
03:08
it allowed the use of gpus and powerful
03:10
libraries like tensorflow and pytorch
03:13
inside the arm itself
03:14
as they state this offers the most
03:16
appropriate trade-off
03:18
among size power and performance for our
03:20
neural decoder implementation
03:22
which was the goal of this paper address
03:24
the challenge of
03:26
efficiently deploying deep learning
03:28
neural decoders
03:29
on a portable device using real-life
03:31
applications
03:32
towards long-term clinical uses of
03:35
course there are a lot of technical
03:36
details that i will not enter into
03:38
as i am not an expert like how the nerve
03:40
fibers
03:41
and bioelectronics connect together the
03:44
microchip's designs that allows the
03:46
simultaneous neural recording
03:48
and stimulation or the implementation of
03:50
software and hardware
03:51
to support this real-time motor decoding
03:54
system you can read a great explanation
03:56
of these in their papers
03:58
if you'd like to learn more about it
03:59
they are all linked in the description
04:01
of the video
04:02
but let's dive a little more into the
04:04
deep learning side of this insane
04:06
creation
04:07
here their innovation leaned to
04:09
optimizing the deep learning motor
04:10
decoding to reduce as much as possible
04:13
the computational complexity
04:15
into this jetson nano platform this
04:17
image shows
04:18
an overview of the data processing flow
04:20
on the gesture nano
04:21
at first the data in the form of
04:23
peripheral nerve signals
04:25
from the amputee's arm is sent into the
04:28
platform
04:29
then it is pre-processed this step is
04:31
crucial to cut
04:32
raw input neural data into trials and
04:35
extract their main features
04:36
in the temporal domain before feeding to
04:39
the models
04:40
this preprocessed data correspond to the
04:42
main features
04:43
of one second of past neural data from
04:46
the amputee
04:47
cleaned from all noise sources then
04:50
this process data is sent into the deep
04:52
learning model
04:53
to have a final output controlling each
04:55
finger's movement
04:56
note that there are five outputs one for
04:58
each finger
04:59
to quickly go over the model they used
05:01
as you can see it starts with a
05:03
convolutional layer
05:05
this is used to identify different
05:06
representations of data input
05:09
in this case you can see the 64 meaning
05:11
that there are
05:12
64 convolutions made using different
05:15
filters
05:15
so 64 different representations these
05:18
filters are the network parameters
05:20
learned during training to correctly
05:22
control the hand when finally deployed
05:25
then we know that time is very important
05:27
in this case since we want fluid
05:28
movements of the fingers
05:30
so they opt for gated recurrent units or
05:33
gru to represent this time dependency
05:35
aspect when decoding the data
05:37
grews will allow the model to understand
05:40
what the hand was doing in the past
05:41
second
05:42
what is first encoded and what it needs
05:44
to do next
05:45
what is decoded to stay simple gru's are
05:49
just an improved
05:50
version of recurrent neural networks or
05:52
rnns solving computational problems
05:54
rnns had with long inputs by adding
05:57
gates to keep only the relevant
05:59
information
06:00
of past inputs in the recurrent process
06:02
instead of washing out
06:04
the new input every single time it's
06:06
basically allowing the network to decide
06:08
what information should be passed to the
06:10
output
06:11
as in recurrent neural networks the one
06:13
second data here
06:14
in the form of a 512 features is
06:17
processed iteratively
06:19
using the repeated gru blocks each dru
06:22
block
06:22
receives the input at the current step
06:25
and the previous output to produce the
06:27
following output
06:28
we can see gru's as an optimization of
06:30
the basic recurrent neural network
06:32
architecture finally this decoded
06:35
information is sent to linear layers
06:37
basically just propagating the
06:38
information and condensing it
06:40
into probabilities for each individual
06:42
finger
06:43
they studied many different
06:44
architectures as you can read in their
06:46
paper
06:47
but this is the most computationally
06:49
effective model they could make
06:50
yielding incredible accuracy of over 95
06:53
percent for the movement of the fingers
06:56
now that we have a good idea of how the
06:57
model works and know that it's accurate
07:00
some questions are still left such as
07:02
what does the person using it
07:04
feels about it does it feel real does it
07:06
work
07:07
etc in short is this similar to a real
07:10
arm
07:10
as the patient himself said i feel like
07:13
once this thing is fine tuned as
07:15
finished products that are out there
07:17
it will be more lifelike functions to be
07:19
able to do everyday tasks
07:21
without thinking of what positions the
07:23
hand is in
07:24
or what mode i have the hand programmed
07:26
in it's just like if i want to reach and
07:28
pick up something
07:29
i just reach and pick up something
07:31
knowing that it's just like my able hand
07:34
for every functions i think we will get
07:36
there i really do
07:38
please just take one more minute of your
07:40
time to watch this short
07:41
touching video where the amputee uses
07:44
the hand
07:44
and shares his honest feedback
07:50
is it pleasurable playing with it oh
07:52
yeah
07:53
it's just really cool like this is this
07:58
is crazy cool
08:00
to me these are the most incredible
08:02
applications that we can work on
08:04
with ai it directly helps real people
08:07
improve their lives quality
08:09
and there's nothing better than that i
08:11
hope you enjoyed watching this video
08:13
and don't forget to subscribe to the
08:14
channel to stay up to date with
08:16
artificial intelligence news
08:18
thank you for watching and as he just
08:20
said in the video
08:21
i will say the same about ai in general
08:24
this is
08:24
crazy cool