Wise Skin for Tactile Prosthetics (WiseSkin)" Hadrien Michaud, EPFL Partners, PI and co-‐PI’s CSEM: Dr. J. Farserotu (PI), Prof. Dr. C. Enz and Prof. Dr. J-‐D Deco.gnie EPFL: Prof. Dr. S. Lacour, Prof. Dr. O. Blanke M.D. (3rd party) The Bern University of Applied Sciences: Prof. Dr. V. Koch Advisory Board Member: S. Micera (EPFL)
Nano-‐Tera Annual mee.ng, Bern, May 4th 2015
04.05.2015
Hadrien Michaud -‐ Wiseskin
1
Downloaded from stm.sciencemag.org on De
Problem"
Hook not included
SENSORY MOTOR tem, this cannot exactly mimic the INTEGRATION natural, multimodal sensory input In addition to an accurate control, to the brain during movement. The motor tasks require sensory infor- Fig. 2. No sensory feedback. Although sensory feedback is fun- stimuli provided roughly resemble mation (Fig. 1) (12). Able-bodied damental for natural movements, a good internal model of the the real activity of some sensory individuals tend to underestimate prosthesis, obtained through training, and accurate feed-forward pathways so that the brain can asthe importance of sensory feedback control together allow for the execution of certain tasks with a myo- sociate these stimuli to specific during movement, because feedback electric prosthesis, without the delivery of artificial sensory input. consistent interactions between This is in addition to the natural feedback available to amputees. A is naturally integrated in each task. the artificial limb and the external prosthetic user is here using the rich sensory information already The lack of sensory information available to him (vision and muscle afferent activity from the mus- environment. The three studies leads to difficulties in performing cles used for control) in an activity of daily living requiring fine con- in Science Translational Medicine not only precise motor commands trol, without additional sensory feedback. The patient is using an (4–6) used this concept for closing but also the simplest tasks (12). The OttoBock Michelangelo hand activated by proportional, direct con- the loop, and their contribution in three studies (4–6) included sensory trol from signal sites that have been maximally separated with TMR. showing reestablished sensoryintegration with motor commands, motor integration is substantial. providing sensory feedback through Nonetheless, with the exception of 5. Functional tasks with sensory feedback. (A) they Without electrical stimulation of nerve fibers. Al- tiple sites in the nerve and wasFig. selective in preliminary results by Tan et al. (5), do the sens systemnot enabled, theprove subject was often unable to adequately c though each study has its own findings and each site. Although the impactfeedback of the type directly a functional advantage of trolbe thejudged grip forcethe in inclusion a delicate of task as holding a cherry while remov goals, the collective contribution is the dem- of electrode for encoding cannot thesuch sensory stimulation with thethat stem.used (B) With the sensory feedback enabled, thewith subject onstration that it is possible to deliver sensa- by comparing the three studies respect to the “open-loop� situation vi- felt con with theduracherry and force applied. accurate He successfully gripped the ch tions to the human brain by stimulating pe- different protocols and implantation sion the only. Although task-sensory removed the stem without the fruit. (C) Total force fr ripheral nerves for extended periods of time tions, the multichannel, selective,and intraneural information maydamaging provide advantages—such thumb subject the 1 had audiovisual [for Tan et al., up to 2 years (5)], concurrent system used by Raspopovic et al. is in and prin-index as tip for sensors the task when of separating stem from a feedb (sighted) but not sensory feedback (feedback off). Peak force during the with upper-arm prosthesis control (sensory- ciple able to better tune the sensation by a cherry (Tan et al.)—with appropriate train-
CREDIT: O. ASZMANN/MEDICAL UNIVERSITY OF VIENNA
Fishing Rod Attachment
motor integration). Ortiz-Catalan (6) and Tan (5) used cuff electrodes for nerve stimulation, which are wrapped around the peripheral nerve (Fig. 1). Stimulation was therefore not very selective, but the nerve integrity was preserved. Conversely, Raspopovic et al. (4) applied a more advanced electrode that was made of a very thin filament of polyimide inserted transversally into the nerve (transversal, intrafascicular electrode). In this electrode, stimulation was obtained by use of mul-
larger number of selective stimulation sites
ing it becomes not strictly necessary (Fig. 2),
combination of sites) than cuff elec- especially whenaconsidering the Rod limitations This convenient(ordevice allows thecanamputee to hold Fishing trodes. However, there is a risk of intraneural in the myocontrol algorithm accuracy and securely while keeping the terminal device for grasping. scarring with irreversible nerve damage that in free the precision in force exertion This by the rowas recently reported for this type of nerve botic device. Fishing Rod attachment utilizes the Fillauer “Zippy Med., Quick Release� electrode (13). For this reason, stability over Last, considering that 2014]! there are limits to [Tan et al., Sci.Transl. longer periods of time needs to be proven for the number of physical variables attachment so that it can be easily removed when not in use.that are codthis extremely flexible and selective electrode ed, the choice of those that are functionally Constructed of lightweight it isre-corrosion structure. RaspopovicDelrinŽ et al. (4) presented relevant is notresistant. an obvious task. For example,
61577
sults from measures made in 1 week, whereas the cuff electrode interfaces were implanted in the other two studies for 1 to 2 years (5, 6).
www
it can be argued that force is an important variable for the user because it cannot be estimated by vision. However, in experiments in
Hosmer Fishing Rod Attachment ..............................................$122.13 www.ScienceTranslationalMedicine.org 8 October 2014 Vol 6 Issue 257 257ps12
3
[Farina et al., Sci.Transl. Med., 2014]!
Fillauer LLC s 4EL s &AX Hosmer s 4EL s &AX OTS Corp s 4EL s &AX [Fillauer LLC]!
Center for Orthotics Design s 4EL s &AX CentriÂŽ s 3WEDEN s 4EL s &AX
Restore of a natural sense of touch for persons using myoelectric prosthetic limbs, through an integrated sensing and stimulation system."
[Biddiss et al., Prosthet. Â Orthot. Â Int, 2007]!
04.05.2015 Â
Hadrien  Michaud  -Ââ€?  Wiseskin Â
2 Â
The WiseSkin solution"
- A network of wireless pressure sensing nodes embedded in an elastomer skin." - Low power consumption through ad hoc radio, communication protocol, and waveguide." - Wearable haptic, non invasive stimulation system. "
04.05.2015
Hadrien Michaud -‐ Wiseskin
3
each site. Although the impact of the type of electrode for encoding cannot be judged by comparing the three studies that used different protocols and implantation durations, the multichannel, selective, intraneural system used by Raspopovic et al. is in principle able to better tune the sensation by a larger number of selective stimulation sites (or combination of sites) than can cuff electrodes. However, there is a risk of intraneural scarring with irreversible nerve damage that was recently reported for this type of nerve electrode (13). For this reason, stability over longer periods of time needs to be proven for this extremely flexible and selective electrode structure. Raspopovic et al. (4) presented results from measures made in 1 week, whereas the cuff electrode interfaces were implanted in the other two studies for 1 to 2 years (5, 6).
not directly prove a functional advantage of the inclusion of the sensory stimulation with respect to the “open-loop” situation with vision only. Although accurate task-sensory information may provide advantages—such as for the task of separating the stem from a cherry (Tan et al.)—with appropriate training it becomes not strictly necessary (Fig. 2), especially when considering the limitations in the myocontrol algorithm accuracy and in the precision in force exertion by the robotic device. Last, considering that there are limits to the number of physical variables that are coded, the choice of those that are functionally relevant is not an obvious task. For example, it can be argued that force is an important variable for the user because it cannot be estimated by vision. However, in experiments in
D
though each study has its own findings and goals, the collective contribution is the demonstration that it is possible to deliver sensations to the human brain by stimulating peripheral nerves for extended periods of time [for Tan et al., up to 2 years (5)], concurrent with upper-arm prosthesis control (sensorymotor integration). Ortiz-Catalan (6) and Tan (5) used cuff electrodes for nerve stimulation, which are wrapped around the peripheral nerve (Fig. 1). Stimulation was therefore not very selective, but the nerve integrity was preserved. Conversely, Raspopovic et al. (4) applied a more advanced electrode that was made of a very thin filament of polyimide inserted transversally into the nerve (transversal, intrafascicular electrode). In this electrode, stimulation was obtained by use of mul-
Elastomer ! membrane!
ΔP!
www.ScienceTranslationalMedicine.org Silicon! membrane!
Contact! PCB!
Side (sensor height = 170 µm)"
8 October 2014 Vol 6 Issue 257 257ps12
0.1 0
Voltage (V)
CREDIT: O. ASZMANN/MEDICAL UNIVERSITY OF VIENNA
Sensing tactile stimuli: sensor technology"
0
3
500
Time (a.u.) 1000
1500
2000
2500
-‐0.1 -‐0.2 -‐0.3 -‐0.4 -‐0.5
Implanted! metal-PDMS! composite ! 200 µm! electrode! Top! 04.05.2015
Hadrien Michaud -‐ Wiseskin
4
Sensing tactile stimuli: sensor node integration"
5 mm!
E = 1.5 MPa!
PDMS!
Front side: sensor!
Back side: electronics! 04.05.2015
TangoBlack!
20 mm!
E = 0.3 MPa!
20 mm!
Hadrien Michaud -‐ Wiseskin
5
each site. Although the impact of the type of electrode for encoding cannot be judged by comparing the three studies that used different protocols and implantation durations, the multichannel, selective, intraneural system used by Raspopovic et al. is in principle able to better tune the sensation by a larger number of selective stimulation sites (or combination of sites) than can cuff electrodes. However, there is a risk of intraneural scarring with irreversible nerve damage that was recently reported for this type of nerve electrode (13). For this reason, stability over longer periods of time needs to be proven for this extremely flexible and selective electrode structure. Raspopovic et al. (4) presented results from measures made in 1 week, whereas the cuff electrode interfaces were implanted in the other two studies for 1 to 2 years (5, 6).
not directly prove a functional advantage of the inclusion of the sensory stimulation with respect to the “open-loop” situation with vision only. Although accurate task-sensory information may provide advantages—such as for the task of separating the stem from a cherry (Tan et al.)—with appropriate training it becomes not strictly necessary (Fig. 2), especially when considering the limitations in the myocontrol algorithm accuracy and in the precision in force exertion by the robotic device. Last, considering that there are limits to the number of physical variables that are coded, the choice of those that are functionally relevant is not an obvious task. For example, it can be argued that force is an important variable for the user because it cannot be estimated by vision. However, in experiments in
D
though each study has its own findings and goals, the collective contribution is the demonstration that it is possible to deliver sensations to the human brain by stimulating peripheral nerves for extended periods of time [for Tan et al., up to 2 years (5)], concurrent with upper-arm prosthesis control (sensorymotor integration). Ortiz-Catalan (6) and Tan (5) used cuff electrodes for nerve stimulation, which are wrapped around the peripheral nerve (Fig. 1). Stimulation was therefore not very selective, but the nerve integrity was preserved. Conversely, Raspopovic et al. (4) applied a more advanced electrode that was made of a very thin filament of polyimide inserted transversally into the nerve (transversal, intrafascicular electrode). In this electrode, stimulation was obtained by use of mul-
CREDIT: O. ASZMANN/MEDICAL UNIVERSITY OF VIENNA
Transmitting tactile information: waveguide"
www.ScienceTranslationalMedicine.org 8 October 2014 Vol 6 Issue 257 257ps12
3
0
UHF antennas!
S11 S12 S21 S22
VDD!
Waveguide plane! GND!
Cross-section!
04.05.2015
S−parameter (dB)
−20 −40 −60 −80 −100 0
Hadrien Michaud -‐ Wiseskin
1
2 3 4 Frequency (GHz)
5
6
6
Transmitting tactile information: radio"
FM-‐Ultra Wide Band receiver: -‐ Simulated power consump.on 420 µW . -‐ Es.mated sensi.vity is -‐80 dBm at 200 kb/s.
Core of the receiver occupies area of ~400×500 µm2
-‐ Consumes less power while providing similar performance and offering greater versaHlity (data rate, number of users, etc.) than the state of the art.
SPI interface for calibra.on and configura.on 04.05.2015
Hadrien Michaud -‐ Wiseskin
Test structures 65 nm technology ! 7
Transmitting tactile information: protocol" • Challenges: – Design a MAC and rou.ng protocols to cope with the variability of the traffic.
• Strategy: – Adapt to changes in the traffic. – Under high traffic, communicate only the important tac.le events with high reliability.
• Currently: – Developing a proac.ve (Node Ini.ated) rou.ng protocol with high reliability.
Omnet ++ simulation!
04.05.2015
Hadrien Michaud -‐ Wiseskin
8
goals, the collective contribution is the demonstration that it is possible to deliver sensations to the human brain by stimulating peripheral nerves for extended periods of time [for Tan et al., up to 2 years (5)], concurrent with upper-arm prosthesis control (sensorymotor integration). Ortiz-Catalan (6) and Tan (5) used cuff electrodes for nerve stimulation, which are wrapped around the peripheral nerve (Fig. 1). Stimulation was therefore not very selective, but the nerve integrity was preserved. Conversely, Raspopovic et al. (4) applied a more advanced electrode that was made of a very thin filament of polyimide inserted transversally into the nerve (transversal, intrafascicular electrode). In this electrode, stimulation was obtained by use of mul-
of electrode for encoding cannot be judged by comparing the three studies that used different protocols and implantation durations, the multichannel, selective, intraneural system used by Raspopovic et al. is in principle able to better tune the sensation by a larger number of selective stimulation sites (or combination of sites) than can cuff electrodes. However, there is a risk of intraneural scarring with irreversible nerve damage that was recently reported for this type of nerve electrode (13). For this reason, stability over longer periods of time needs to be proven for this extremely flexible and selective electrode structure. Raspopovic et al. (4) presented results from measures made in 1 week, whereas the cuff electrode interfaces were implanted in the other two studies for 1 to 2 years (5, 6).
the inclusion of the sensory stimulation with respect to the “open-loop” situation with vision only. Although accurate task-sensory information may provide advantages—such as for the task of separating the stem from a cherry (Tan et al.)—with appropriate training it becomes not strictly necessary (Fig. 2), especially when considering the limitations in the myocontrol algorithm accuracy and in the precision in force exertion by the robotic device. Last, considering that there are limits to the number of physical variables that are coded, the choice of those that are functionally relevant is not an obvious task. For example, it can be argued that force is an important variable for the user because it cannot be estimated by vision. However, in experiments in
CREDIT: O. ASZMANN/MEDICAL UNIVERSITY OF VIENNA
Feedback to the prosthesis user"
LRA
ERM ERM = Eccentric Rota.ng Mass
www.ScienceTranslationalMedicine.org 8 October 2014 Vol 6 Issue 257 257ps12
3
LRA = Linear Resonant Actuator
Vibro-‐tac.le feedback devices
Sensory glove
Tes.ng with subjects
Mechano-‐tac.le feedback devices 04.05.2015
Hadrien Michaud -‐ Wiseskin
9
Feedback to the prosthesis user" Functional prototype under construction"
Sensor Data Actuator command Control unit
04.05.2015
EMG Data E Hap.c device and EMG electrodes
Robo.c hand
Hadrien Michaud -‐ Wiseskin
Commercial force sensi.ve resistors
10
Conclusion"
Pressure sensor!
Elastomeric skin! 04.05.2015
ULP receiver!
Board!
Waveguide! Hadrien Michaud -‐ Wiseskin
Haptic display!
Protocol! 11
Next steps"
Sensor characterization, transmitter, µ-controller interface!
Skin electromechanical testing! 04.05.2015
Hadrien Michaud -‐ Wiseskin
Functional prototype!
Hardware deployment! 12
Sensor!
Protocol!
Radio!
Skin technology!
Haptic feedback!
Thank you!! 04.05.2015
Hadrien Michaud -‐ Wiseskin
13
References" § [Bidiss et al., 2007]E. a Biddiss and T. T. Chau, “Upper limb prosthesis use and abandonment: a survey of the last 25 years.,” Prosthet. Orthot. Int., vol. 31, no. 3, pp. 236–57, Sep. 2007. § [Farina et al., Sci. Transl. Med., 2014] D. Farina and O. Aszmann, “Bionic Limbs : Clinical Reality and Academic Promises,” Sci. Transl. Med., vol. 6, no. 257, 2014. § [Tan et al., Sci. Transl. Med., 2014] D. W. Tan, M. a. Schiefer, M. W. Keith, J. R. Anderson, J. Tyler, and D. J. Tyler, “A neural interface provides long-‐term stable natural touch percep.on,” Sci. Transl. Med., vol. 6, no. 257, pp. 257ra138– 257ra138, Oct. 2014.
04.05.2015
Hadrien Michaud -‐ Wiseskin
14