Share This Post

Featured News

A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback

A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback

Abstract

Current myoelectric prostheses allow transradial amputees to regain voluntary motor control of their artificial limb by exploiting residual muscle function in the forearm. However, the overreliance on visual cues resulting from a lack of sensory feedback is a common complaint. Recently, several groups have provided tactile feedback in upper limb amputees using implanted electrodes, surface nerve stimulation, or sensory substitution. These approaches have led to improved function and prosthesis embodiment. Nevertheless, the provided information remains limited to a subset of the rich sensory cues available to healthy individuals. More specifically, proprioception, the sense of limb position and movement, is predominantly absent from current systems. Here, we show that sensory substitution based on intraneural stimulation can deliver position feedback in real time and in conjunction with somatotopic tactile feedback. This approach allowed two transradial amputees to regain high and close-to-natural remapped proprioceptive acuity, with a median joint angle reproduction precision of 9.1° and a median threshold to detection of passive movements of 9.5°, which was comparable with results obtained in healthy participants. The simultaneous delivery of position information and somatotopic tactile feedback allowed both amputees to discriminate the size and compliance of four objects with high levels of performance (75.5%). These results demonstrate that tactile information delivered via somatotopic neural stimulation and position information delivered via sensory substitution can be exploited simultaneously and efficiently by transradial amputees. This study paves a way to more sophisticated bidirectional bionic limbs conveying richer, multimodal sensations.

INTRODUCTION

Commercially available myoelectric prostheses typically only provide efferent control, which is achieved by measuring muscle activity from the forearm and extracting the user’s movement intentions (1), while afferent information is absent. Consequently, amputees using these systems often complain about the need to rely on visual cues during everyday prosthesis use (2, 3). Although studies have successfully used invasive and noninvasive approaches to restoring tactile feedback in upper limb amputees (416), direct elicitation of selective proprioceptive percepts remains elusive and is only rarely reported (48). Efforts to restore proprioceptive feedback invasively have been limited to preliminary studies showing only modest functional benefits or lacking extensive characterization (1720). Using a different approach, Marasco et al. (21) recently exploited the well-documented muscle vibration illusion to provide homologous proprioceptive feedback in amputees having undergone targeted muscle reinnervation, with promising functional results. However, reinnervated muscle vibration often induced accompanying referred cutaneous sensations on the phantom hand, limiting the possibility to provide tactile feedback simultaneously without interference, a key aspect for clinical translation.

Proprioception is known to be mediated in part by type Ia and type II sensory afferents from the muscle spindles (22). The proximity of proprioceptive afferents and motor neurons within the nerve may explain the difficulty in activating proprioceptive pathways without inducing undesirable motor twitches. Neurophysiological evidence indicates that microstimulation of proprioceptive afferents does not lead to perceptual responses, unless accompanied by muscle activity (23). This suggests that selective homologous proprioceptive feedback (i.e., where the restored sensation closely matches the natural sensation and where there is no coactivation of muscles) could be difficult to achieve with current neural stimulation approaches (in transradial amputees). Instead, sensory substitution (remapping) may be a viable alternative, potentially enabling substantial functional gains. Sensory substitution (remapping) has been used extensively for restoring other sensory modalities (11) and pioneered by Bach-y-Rita and colleagues (24) with tactile substitution of visual stimuli, including recently using brain implants in nonhuman primates (25, 26) and augmented reality in healthy participants (27, 28), with promising results.

For this reason, we implemented a hybrid approach for restoring multimodal sensory information to transradial amputees, where finger position information (referred to as remapped proprioception) was provided using sensory substitution based on peripheral intraneural stimulation, whereas tactile information (referred to as somatotopic tactile feedback) was restored using a somatotopic approach, where the elicited sensation was correctly perceived on the fingers and palm, as previously shown (4, 5). Specifically, joint angle information was delivered through spared neural afferent pathways using intraneural stimulation of the peripheral nerves in the amputee’s stump. Furthermore, because sensory substitution can be implemented using a large number of approaches (i.e., using any other sensory modality than the one being substituted), we performed a side-by-side comparison of our results with the same sensory substitution approach implemented using noninvasive electrotactile feedback with one participant.

Two transradial amputees were implanted with transverse intrafascicular multichannel electrodes (TIMEs) in the ulnar and median nerves (Fig. 1) (29). Participant 1 performed a pilot study, whereas participant 2 performed a more comprehensive set of experiments. Both participants reported stable sensations of vibration, pressure, and electricity over the phantom hand and stump during intraneural stimulation (fig. S1). Position information was provided using active sites that elicited sensations referred to the lower palm area or the stump. This choice avoided any conflict with tactile feedback, which used active sites providing sensations referred to the phantom fingers (4). The feedback variable was the hand aperture (either one or two degrees of freedom depending on the experiment; see Materials and Methods), encoded using linear amplitude modulation.

Fig. 1 Overview of the multimodal sensory feedback experimental setup.

(Left) Bottom: The robotic hand is driven using sEMG activity acquired from the participant’s forearm muscles and classified into distinct motor commands. As the robotic hand closes its fingers around an object, both pressure and position are measured in real time. Top: Information about pressure and position is then encoded into stimulation pulses, where stimulation amplitude is directly proportional to finger position or pressure. Pressure perception is restored using a somatotopic approach, where the induced sensation corresponds to the fingers being touched. Position information (remapped proprioception) is restored using sensory substitution, whereas the sensation does not correspond to the natural area. (Right) Top: Both sensory streams are delivered using intraneural stimulation through TIME electrodes implanted in the median and ulnar nerves. Bottom: The TIME implant is inserted transversally through the exposed nerve fascicles.

RESULTS

We first characterized the acuity of the remapped proprioceptive sense alone. We administered two clinical tests, namely, threshold to detection of passive motion (TDPM) and joint angle reproduction (JAR) (30). During the TDPM test, we measured the smallest prosthesis displacement necessary for the participants to detect passive motion of the artificial hand, starting from randomly chosen positions across the hand’s range of motion (constant speed, 27.5° s−1). This test measured the sensibility to stimulation amplitude and is reported in terms of remapped hand aperture. The overall TDPM was 9.5° [interquartile range (IQR), 9.1], with 12.5° (IQR, 10.4) for participant 1 and 6.5° (IQR, 6.6) for participant 2 (fig. S2). No statistically significant correlation was found between TDPM and initial hand position (P = 0.52) or with movement direction (P = 0.11), indicating that remapped proprioceptive sensibility was equal across the range of motion and independent of the direction of movement of the hand (Fig. 2A).

Fig. 2 TDPM and JAR tasks.

(A) The TDPM is reported for each prosthesis position tested. The median is reported as a dashed line. A histogram of the data, with bin sizes of 3°, is shown on the right. A total of 244 measures were collected with two participants (115 for participant 1 and 129 for participant 2). (B) The robotic fingers’ range of motion and the way the angle are reported. (C) JAR precision during the fixed position reproduction task for four target positions. The reproduced positions are reported as median (full, colored line) and IQR (shaded area). The MAD for the pooled performance on all positions is reported for each participant. (D) Box plots reporting the detailed absolute deviation for each requested position. The median is reported as a blue line, and the box represents the IQR. The whiskers encompass all data samples (no outliers removed). A total of 80 (40 for participant 1 and 40 for participant 2) repetitions were collected for the task. Asterisks indicate conditions found to be statistically different after a Kruskall-Wallis test with multigroup correction. (E) A box plot showing the absolute deviation around the median for randomly switched prosthesis actuation speeds (three speeds). For this task, only participant 2 participated, and a total of 48 repetitions were performed. (F) Scatterplot of the measured error in JAR for each position tested during the JAR task with random and continuous positions. A histogram of the data, with bin sizes of 3°, is shown on the right hand size. A total of 171 measures were collected with two participants (81 for participant 1 and 90 for participant 2).

During a first variant of the JAR test (fixed positions), both participants were asked to actively move the hand to one of four self-selected angular positions. The angle of closure was measured from the fully open state (Fig. 2B). For each reproduced position, the median absolute deviation from the median (MAD; a robust measure of variability) was computed. MAD was measured at 10.2° for participant 1 and 4° for participant 2 (Fig. 2C). Overall, MAD was significantly lower when the target position was at the extremes of the range of motion (fully open or fully close) compared with intermediate positions due to the impossibility to “overshoot” the target at both extremes of movement (P < 0.05; Fig. 2D).

Participant 2 also performed a control condition to dismiss the possibility of using movement duration to infer finger position. During the same task, hand prosthesis actuation speed was randomly switched between three values (22°, 43°, and 68° s−1), without the participant’s knowledge. Despite receiving unreliable information about timing, no significant increase in spread was observed for any of the tested actuation speeds (P = 0.76), nor for the overall performance (P = 0.75), indicating that timing did not play a critical role in achieving high task performance (Fig. 2E).

We also performed a more challenging JAR experiment using random and continuous positions. In this case, the robotic hand was first passively closed with a random joint angle. Then, the hand was passively opened again, and the participants were asked to control the robotic hand and bring it back to the same position. The JAR precision was constant across the entire range of motion (P = 0.68), with a median error of 9.1° (IQR, 14.6; Fig. 2F). Median error was 8.6° for participant 1 (IQR, 12.7) and 9.9° for participant 2 (IQR, 15.9; fig. S2).

To study how the remapped position sense could be exploited during functional tasks, we performed an object size identification experiment, where participants had to determine the size of an object chosen randomly from a pool of four cylinders with varying diameter (Fig. 3A). The objects resulted in different final degrees of closure of the hand (Fig. 3D). Overall, the two participants identified the objects correctly in 78% of cases (77.5% for participant 1 and 80% for participant 2; fig. S3 and Fig. 3B), whereas five healthy controls had a higher score of 98.5% (fig. S4A). Movies S1 and S2 show a few example trials of the object recognition task.

Fig. 3 Identification of object size.

(A) Schematic representation of the four different objects used during the object size identification task and their labeling (not to scale). (B) Overall performance during the task with remapped proprioception only for both participants in the form of a confusion matrix (left) and performance in identifying each object (right). Median correct identifications and a 95% CI for each object are reported alongside the matrix. Asterisks identify levels that were statistically different from chance level. A total of 160 repetitions (40 for participant 1 and 120 for participant 2) were performed with two amputee participants. (C) Overall performance during the object size recognition task with simultaneous touch and remapped proprioceptive feedback in the form of a confusion matrix. A total of 100 repetitions were performed with participant 2. (D) Representative position traces obtained during the experiments. One example was chosen for each cylinder size to illustrate the difference in measured position obtained in each case. In addition, the stimulation amplitude computed from the position is reported on the second y axis. (E) Overall performance for each tested hand actuation speed during a control trial with changing speeds. A total of 96 repetitions were performed with participant 2. (F) The performance obtained during a control condition where only touch feedback was delivered. In this case, 100 repetitions were performed with participant 2.

Fig. 4 Identification of object size and compliance.

(A) Schematic representation of the four different objects used during the object size and compliance task and how they were labeled (not to scale). (B) Overall task performance with both remapped proprioception and touch, for both participants, reported as a confusion matrix. The combined performance is shown under the image. Median correct identifications and a 95% CI for each object feature (size and compliance) are reported alongside the matrix. Asterisks identify levels that were statistically different from chance level. A total of 220 repetitions were performed with two participants (40 for participant 1 and 180 for participant 2). (C) Performance during the same object size and compliance task when only remapped proprioceptive feedback is provided. A total of 80 repetitions were performed with participant 2. (D) Representative force and position traces, as measured by the robotic hand, for each object type. The full lines represent hand position (0° to 110°), and the dashed lines represent measured force (normalized). The four patterns are not contiguous (illustrated by dashed lines), but the relative duration of each pattern is conserved to allow meaningful comparison of the slopes. (E) Performance during the same task when only touch feedback is provided. A total of 80 repetitions were performed with participant 2. (F) Compliance decoding performance broken down by object, with touch only, remapped proprioception only, or both sensory modalities. Compliance decoding performances above chance level are shown with an asterisk. A total of 380 repetitions were used for this panel [combination of data from (B), (C), and (E)].

Several control conditions were tested with participant 2. First, the same task was repeated with tactile feedback alone (Fig. 3F). In this scenario, performance was poor but remained above the 25% chance level [47% correct identification; 95% confidence interval (CI), 36.9 to 57.2]. However, further analysis showed that only the largest object was correctly identified above chance level (Fig. 3F). This indicated that tactile feedback alone was not sufficient to perform this task (i.e., recognizing all objects). Second, the task was performed with both tactile and remapped position feedback. The measured performance (70% correct identification) was not statistically lower than the performance obtained with remapped proprioception only (P = 0.449, Fisher’s exact test), indicating that the addition of touch did not interfere with the interpretation of position feedback (Fig. 3C). Third, when remapped proprioception was provided alone and the prosthesis movement speed was randomly switched between three values, the performance was 67%, which was not statistically different from the condition with constant speed (P = 0.226, Fisher’s exact test; Fig. 3E), suggesting that the participant did not rely on timing as a proxy to infer hand aperture.

Both participants were also asked to identify the size and compliance of four different cylinders (Fig. 4A). In this case, tactile and remapped proprioceptive feedback was provided simultaneously (Fig. 4B). Overall, performance for this task was high, with 75.5% correct answers (87.5% for participant 1 and 73% for participant 2; Fig. 4D and fig. S3). By comparison, two healthy controls had a perfect score of 100% (fig. S4B). Participant 2 performed the same task while receiving only remapped position (Fig. 4E) or tactile (Fig. 4F) feedback. In both cases, performance significantly worsened (no overlap of 95% CIs). When only position feedback was provided, object size was identified above chance level, but object compliance was not (Fig. 4E). Conversely, when tactile feedback was provided, only object compliance was correctly identified (Fig. 4F). Furthermore, providing both modalities simultaneously can improve performance, as seen from the superior compliance decoding achieved using both touch and remapped proprioception compared with either modality individually, for the large object (Fig. 4C).

Data obtained with participant 2 for the object size task show a steady increase in performance over time, indicating that, although remapped proprioception can successfully be exploited almost immediately, training may confer an advantage and could lead to further improvements in performance over time (fig. S5A). Additional measurements, obtained over longer periods of time, could confirm the effect of training on performance.

Fig. 5 Comparison of invasive and superficial sensory substitution on TDPM.

(A) TDPM values are reported for all tested starting positions. Median TDPM is reported as a dashed line. A histogram of the data, with bin sizes of 1.5°, is shown on the right. A total of 65 trials are shown. (B) A comparison of TDPM task metrics between the two approaches used for sensory substitution (invasive and superficial electrical stimulation). Both median TDPM (shown with IQR) and number of direction detection errors are shown (along with the 95% CI). A total of 65 TDPM trials were performed using noninvasive stimulation, and 129 trials were performed using intraneural stimulation.

In another experiment, we provided two channels of remapped proprioceptive feedback simultaneously (one channel for the first three digits and one for the last two). In this case, two channels giving rise to different sensations on the stump were used. Using this “multijoint” feedback, participant 2 could simultaneously detect the diameter of two cylinders with a very high performance of 93.7% (fig. S6), demonstrating that the sensory remapping approach presented here can also be applied to more than one finger simultaneously.

Fig. 6 Comparison of invasive and superficial sensory substitution during functional tasks.

(A) Schematic representation of the experimental setup. Top left shows the normal object size task, where all fingers wrap around four objects of different sizes. Bottom left shows the multijoint object size task, where the first three fingers, and the last two, come into contact with different parts of the objects, which can be of different sizes. The right side shows the object size and compliance task. In this case, two noninvasive approaches were used (TSPS and TIPS). In both cases, size (proprioception) was encoded using noninvasive sensory substitution. In the case of TSPS, tactile feedback was also encoded using noninvasive sensory substitution. In the case of TIPS, a hybrid approach was used, where tactile feedback was encoded using intraneural electrical stimulation leading to somatotopic tactile sensations. Also shown are the abbreviations used for the various objects. (B) Overall performance during the object size task performed with noninvasive sensory substitution. Identification performance broken down by object is reported with 95% CIs, and asterisks identify levels that were statistically different from chance level. Overall (median) correct identifications are reported underneath the plots. A total of 80 trials were recorded for this task. (C) Performance of the multijoint object size task displayed using the same data presentation structure as in (B). A total of 80 trials were recorded for this task. (D) The overall performance during the size and compliance task using the TSPS encoding, shown as the percentage of correct identification broken down by object (shown with 95% CIs). Underneath, the overall task performance is reported. A total of 160 trials were recorded for this task. (E) The overall performance during the size and compliance task using the TIPS encoding displayed using the same data presentation structure as in (D). A total of 160 trials were recorded for this task. (F) Comparisons of overall task performances for the object size (left), the multijoint object size (middle), and the object size and compliance (right) tasks using either superficial or intraneural stimulation as a source of sensory substitution.

Next, participant 2 performed all three functional tasks reported above during a demanding verbal fluency task to test the effect of cognitive loading on task performance. For both purely remapped proprioceptive tasks (object size identification and multijoint remapped proprioception task), performance remained statistically unchanged (fig. S7A). For the combined tactile and remapped proprioceptive task (combined size and compliance identification), performance was diminished, in accordance with the anecdotal reports that judging object compliance was more challenging, requiring higher focus. In all cases, tasks were performed above chance level, even in the presence of cognitive loading (fig. S7, B to D).

Last, we tested whether using noninvasive electrotactile feedback as a source of sensory substitution would result in significantly different results compared with the same approach based on intraneural feedback. We replicated the TDPM, object size, and multijoint and object size and compliance tasks with electrotactile feedback delivered at the shoulder level with participant 2, as depicted in fig. S8.

The median TDPM obtained using superficial stimulation was 12.5° (IQR, 11.8; Fig. 5A). We found a statistically significant correlation between starting robotic hand position and proprioceptive acuity (P = 0.0003, Spearman’s correlation), indicating that the participant was able to perceive finer movements when the hand was closer to fully open (low-angle values) than when the hand was closer to being closed (high-angle values). We then compared these results with the TDPM values obtained when invasive electrical stimulation was used for sensory substitution and found a statistically significant difference (P = 0.0000006, Kruskall-Wallis) in median TDPM values between the two conditions (Fig. 5B). Specifically, proprioceptive acuity was lower when sensory substitution was provided using noninvasive stimulation compared with intraneural stimulation. Furthermore, a significant difference was also observed in the number of direction detection errors (i.e., when the participant correctly detects the presence of movement but not the direction of the movement), with errors being more common during noninvasive sensory substitution (P = 0.000000002, Fisher’s exact test).

We also evaluated the impact of the two types of sensory substitution approaches on subjective prosthesis embodiment by administering an embodiment questionnaire (fig. S9). Of the 10 questions, only 5 are reported in fig. S9. The other five questions, which were control questions testing for participant suggestibility, were consistently answered with a score of 0 (not at all) for all conditions, indicating that the participant was not suggestible. Statistically significant differences between invasive and noninvasive sensory substitution were observed for certain questions (questions 1, 3, and 9) but not for others (questions 2 and 10), indicating that prosthesis embodiment was lower when providing remapped proprioceptive feedback using noninvasive electrical stimulation and higher when providing the same feedback using intraneural electrical stimulation. In both cases, embodiment was higher than in the absence of any stimulation.

During the object size task performed with noninvasive sensory substitution, the participant recognized the cylinders with a median performance of 81% (Fig. 6B), whereas during the multijoint remapped proprioception task, overall performance was measured at 89% correct answers (Fig. 6C). When compared with the result obtained using intraneural stimulation as a source of sensory substitution, we found no statistically significant difference in task performance (P = 0.83 and P = 0.82, Fisher’s exact test; Fig. 6F).

During the object size and compliance tasks, three encoding strategies were compared. Although the aim was to compare the use of superficial and intraneural electrical stimulation to convey remapped proprioceptive information, we also tested conveying tactile feedback using either invasive somatotopic stimulation or superficial feedback (Fig. 6A). Overall performance for the task using the TSPS (tactile feedback superficial, remapped proprioceptive feedback superficial) encoding was 68% correct identifications (Fig. 6D), whereas performance using the TIPS (tactile feedback intraneural, remapped proprioceptive feedback superficial) encoding was 73% correct identifications (Fig. 6E). When comparing both superficial encoding strategies with the invasive sensory substitution (Fig. 6F), no statistically significant differences in task performance was observed.

DISCUSSION

The TDPM experiment indicates that intraneural sensory substitution is capable of restoring high remapped proprioceptive acuity, defined as the participants’ ability to perceive small changes in hand aperture (conveyed as small changes in injected current). Previous results have shown that healthy individuals obtain TDPM values for single finger joints between 6.5° and 1.5° (31). Although these results are not directly comparable, our approach enabled participant 2 to obtain acuity within this range, whereas participant 1 obtained a slightly lower acuity. This indicates that the bandwidth of the remapped position sense, which, in this case, is driven by the participant’s ability to discriminate changes in current amplitude, may be sufficient for a wide range of functional tasks.

Furthermore, high JAR values demonstrate that the participants are capable of exploiting the precise information provided via sensory substitution during an active task to efficiently close the loop. The median JAR errors measured during the experiment (8.6° and 9.9°) were consistent with the participants fully exploiting the precise position information they received. The slightly higher errors obtained, when compared with the passive ability to perceive the degree of aperture of the robotic limb as measured by TDPM, are likely explained by the participant’s inability to perfectly control their prosthetic limb. Despite the imprecision introduced by the controller delay (about 100 ms), the JAR errors compare favorably with results obtained with healthy individuals, where the matching error for the metacarpophalangeal joint was measured between 5.94° and 10.9° (32, 33).

Functionally, this ability to accurately sense hand aperture, even during active movements, translated into high performance during the object size–sensing task. Both participants were able to fairly easily recognize the four different cylinders presented to them, despite this task being impossible to achieve without sensory feedback. The four objects used during the object size task were separated by about 25° in terms of hand aperture (with significant variability depending on the placement of the object in the hand). The fact that both participants had TDPM values significantly below this value (12.5° and 6.5°) suggests that this approach may allow the discrimination of a larger number of objects. This conclusion would also hold for noninvasive sensory substitution, although to a lesser extent considering the higher measured TDPM.

This laboratory-based object exploration task maintained some of the characteristics an amputee is likely to encounter during everyday life despite representing a simplified situation (e.g., number of available objects is limited to four), such as the necessity to identify an object despite not having access to visual cues (e.g., moving in the dark or grabbing an object from inside a bag). The fact that sensory substitution can provide functionally meaningful information during a simplified object exploration task offers promise that such an approach could lead to clinical benefits for prosthesis users. Demonstrating such clinical advantages will require larger, home-use studies that test for a much bigger number of prosthesis use cases.

The ability to interpret the provided position information was preserved even when we introduced an additional feedback variable, namely, somatotopic tactile feedback. We demonstrated that each feedback variable mainly informed the user about one of the object’s physical properties (compliance or size). Specifically, tactile feedback informed about object compliance, as previously shown by Raspopovic et al. (4), whereas position feedback provided information about object size. The performance obtained by both participants was higher than previous results reported in the literature [Horch et al. (19) reported about 54% correct identification in one participant] using direct elicitation, indicating that sensory substitution may be more effective at restoring position information. The high bandwidth of the remapped proprioceptive sensory substitution channel reported here may help explain these differences.

We carefully designed control conditions to eliminate the most obvious potential bias, which was the possibility that the participants were relying on timing as a proxy for hand aperture. The results reported here indicate that this was not the primary strategy used by the participants, because performance remained unchanged despite large and randomized modifications of the robotic hand’s actuation speed.

Unexpectedly, even under cognitive load, the participant’s ability to interpret the remapped position information remained high. The sense of limb position and movement arises from a complex interplay between various afferent signals coming from the skin, muscle spindles, Golgi tendon organs, and joint capsules (23, 3436). To this day, the exact contribution of each of these factors remains debated (34). In the case of the hand, cutaneous information is believed to play a major role (23). In this study, the channels used to deliver remapped proprioceptive information resulted in cutaneous sensations referred to the palm and stump. Because skin stretch mediates proprioception in the healthy hand, using cutaneous channels for remapped proprioceptive feedback arguably may not be entirely nonsomatotopic, perhaps helping to explain the limited cognitive burden reported here.

To extend the main results of this study, we showed that the sensory substitution approach presented here can be generalized to more than one degree of freedom. The limit of how many simultaneous degrees of freedom could be restored using this approach remains an open question. However, the limited cognitive load shown for the use of one or two channels indicates that it might be possible to further increase the number of restored channels while maintaining good performance. Although this experiment demonstrates the ability of the participant to distinguish and attend to two distinct channels of sensory substitution, the total amount of delivered information is not increased (because four objects are being recognized, as in the single channel case).

Considering that sensory substitution can, by definition, be delivered using numerous sensory channels, we compared the results obtained using intraneural sensory substitution with a noninvasive electrotactile sensory substitution channel. With noninvasive sensory substitution, the same participant obtained significantly higher TDPM values (12.5° against 6.5°). Furthermore, the higher TDPM values were accompanied by a significantly higher number of direction detection errors. As discussed above, these differences are driven by the participant’s ability to perceive changes in the amount of injected charge (because injected charge is directly proportional to hand aperture) and the charge modulation range. These two factors will determine the amount of information that can be reliably transmitted through a given sensory substitution channel. In our case, almost twice as much information about hand position was relayed to participant 2 using intraneural stimulation compared with noninvasive stimulation.

Despite these significant differences in remapped proprioceptive acuity, no significant changes were observed in task performance (Fig. 6). This indicates that the lower proprioceptive sensitivity achieved using superficial stimulation–based sensory substitution was not sufficient to affect the types of functional tasks performed in this study. Although one might expect a drop in sensitivity to be accompanied by a corresponding drop in task performance, the tasks performed by the participant may not have been challenging enough to detect such differences. Recognizing four different object sizes does not necessarily require very high proprioceptive acuity (because, as noted, the objects differ by sizable amounts of hand aperture when grasped). We therefore hypothesize that the observed differences in remapped proprioceptive acuity would lead to differences in performance for tasks requiring finer position sensibility. This hypothesis will need to be investigated with further experiments. Nevertheless, both approaches appear to be viable tools for improving task performance during simple sensory exploration.

Last, we looked at prosthesis embodiment using a subjective questionnaire, which is a qualitative assessment and therefore intrinsically limited. However, because prosthesis embodiment is likely to play an important role in limb rejection rates (13), these types of questionnaires are increasingly used to evaluate sensory feedback approaches in prosthetics (15). In our case, it provides useful information when choosing between both strategies for the delivery of remapped proprioception. Although both types of sensory substitution resulted in an improved sense of embodiment, superficial stimulation led to statistically lower answers on specific embodiment questions compared with invasive feedback.

In this study, the noninvasive electrotactile stimulation used for sensory substitution was delivered in the shoulder area of the corresponding hand. Theoretically, such a sensory substitution approach could be applied to any part of the body, with potentially differing results in terms of prosthesis embodiment (e.g., if the stimulation is applied contralaterally). We cannot exclude the possibility that the difference in prosthesis embodiment observed between the two feedback strategies was driven by the respective positions of the applied stimuli (fig. S8). Applying the noninvasive stimuli in the same location as the invasive one would not have been possible, because the invasive stimuli were perceived on the phantom limb (i.e., wrist). On the other hand, applying the noninvasive stimuli closer to the phantom hand would have required additional processing steps to remove the stimulation artifacts from the surface electromyographic (sEMG) signal used for prosthesis control (10). Although several such approaches have been described in the literature (37, 38), they are dependent on hardware specifics (e.g., an amplifier stage that does not saturate) and were not implemented in our setup. We therefore chose a location sufficiently removed from the site of sEMG acquisition, bypassing any difficulties with the stimulation artifacts, which are particularly strong when using two channels of superficial feedback in the stump area (i.e., during TSPS).

A limitation of the current study is that remapped proprioceptive information was provided for two groups of fingers at most (two channels). Whether the sensory substitution approach presented here would enable the delivery of feedback about all five fingers remains an open question (this has also yet to be demonstrated for tactile feedback). Furthermore, we focused our efforts on restoring position information about the fingers. This choice was predicated on the importance of hand proprioception in healthy individuals, for instance, during haptic exploration (39). However, future efforts could focus on restoring position or angle information around other joints, such as the wrist and the elbow (the latter for transhumeral amputees), which are likely to significantly affect functional outcomes during reaching and grasping tasks. Providing position information about other joints could be achieved using the same sensory substitution approach described here.

Last, current implantable nerve stimulation systems have high redundancy, usually offering dozens of active sites over all electrodes. In practice, only a fraction of these active sites are used for tactile feedback. Our approach takes advantage of an existing implant by exploiting unused channels. This makes it easier to implement a multimodal feedback scheme on top of existing sensory neuroprostheses with minimal added cost and engineering burden. Furthermore, our results indicate that it is beneficial to use intraneural stimulation as a source of sensory substitution when providing proprioceptive feedback to upper limb amputees, compared with noninvasive sensory substitution. However, considering the absence of functional differences between intraneural and superficial stimulation during relatively simple tasks, we argue that the reported results are not sufficient to justify a neural implant for the sole purpose of providing remapped proprioceptive feedback. Instead, intraneural sensory substitution may be the optimal choice when a neural interface is already in place, such as for providing tactile feedback (4, 5). This will be especially true as higher bandwidth becomes necessary for restoring an increasingly large number of feedback channels (i.e., multimodal feedback). Nevertheless, future studies will need to quantify the impact of the higher bandwidth offered by intraneural sensory substitution during more challenging tasks.

In conclusion, this study shows that transradial amputees can effectively exploit a hybrid multimodal stimulation approach, which combines somatotopic tactile feedback with sensory substitution (remapped proprioception). The functional results demonstrate that the two streams of information can be used simultaneously to achieve high task performance. Our results pave a way to more sophisticated bidirectional bionic limbs conveying richer, multimodal sensations.

MATERIALS AND METHODS

Study design, participant recruitment, and experiment logistics

Two amputees participated in the study (a 54-year-old female with a left wrist disarticulation incurred 23 years before the study and a 54-year-old female with a proximal left transradial amputation incurred 2 years before the study). Ethical approval was obtained by the Institutional Ethics Committees of Policlinic A. Gemelli at the Catholic University, where the surgery was performed. The protocol was also approved by the Italian Ministry of Health. Informed consent was signed. During the entire length of our study, all experiments were conducted in accordance with relevant guidelines and regulations. This study was performed within a larger set of experimental protocols aiming at the treatment of phantom limb pain and robotic hand control. The clinical trial’s registration number on the online platform (www.clinicaltrials.gov) is NCT02848846.

The data reported in this manuscript were obtained over a period of several days in two amputees. The first participant (participant 1) was recruited as a pilot case toward the end of an ongoing long-term study of intraneural electrodes (5 months after implantation) and performed a more limited number of experiments (particularly with regard to control conditions). Participant 1 performed all experiments reported here over a period of 4 days (divided in two sessions of two back-to-back days over 2 weeks), although each type of experiment was not performed more than once (there are no data for the same experiment over multiple days). The second participant (participant 2) was recruited at an earlier stage (2 weeks after implantation) and performed a larger number of trials and a more complete set of control experiments. All data for participant 2 were obtained over a period of 6 days (sessions spread over a period of 6 weeks), with several experiments grouping data over multiple days (and allowing a comparison of performance over days, as shown in fig. S5A). The noninvasive experiments were performed with participant 2.

Bidirectional setup and prosthesis control

For the functional tasks, participants were fitted with a custom bidirectional research prosthesis, allowing control of hand opening and closing by processing sEMG signals and providing sensory feedback by means of electrical stimulation of the peripheral nerves. A robotic hand with tension force sensors integrated within each digit (IH2 Azzurra, Prensilia, Italy) was controlled using a custom, multithreaded C++ software running on a Raspberry Pi 3 single-board computer (Raspberry Pi Foundation, UK). A recording and stimulating device (Neural Interface Processor, Ripple LLC., USA) was also connected to the central single-board computer, acquiring sEMG data from two or four bipolar channels and providing stimulation outputs to the four neural electrodes. Custom-molded sockets were built with integrated screws to easily fix the robotic hand on the end. Holes were drilled to allow for the placement of sEMG electrodes on the stump.

For prosthesis control, a simple three-state (open, close, and rest) threshold controller was used for participant 1, and participant 2 used a k-nearest neighbors (KNN; k = 3) classifier with three classes (40). Two or four bipolar channels of sEMG signals were acquired from forearm residual muscles (for participants 1 and 2, respectively), where palpation was used to place the electrodes in the optimal positions. The sEMG data were acquired with a sampling frequency of 1 kHz and filtered using an infinite impulse response filter with fourth-order Butterworth characteristics, between 15 and 375 Hz, as well as a notch filter to remove any 50-Hz power hum. For the threshold controller, the mean absolute value (MAV) was computed for each channel, and a threshold was set manually to indicate when the hand should be opened or closed. The amplitude of the sEMG signal (MAV) controlled hand actuation speed (proportional control). For the KNN classifier, the waveform length was computed over a window of 100 ms for each channel and fed to the classifier every 100 ms. The decoded class was used to send open or close commands to the prosthesis.

Tactile feedback based on intraneural electrical stimulation

Both participants were implanted with four TIMEs in the median and ulnar nerves (two per nerve), above the elbow, each with 14 active sites and two counterelectrodes on the substrate (29). A total of 56 actives sites per participant were thus available. After an extensive mapping phase, during which the stimulation parameter space (defined by the following variables: electrode, active site, stimulation amplitude, stimulation pulse width, and frequency) was explored, relationships between stimulation parameters and sensation quality, location, and intensity were established, as described in Raspopovic et al. (4), where an analogous preparation was used. Briefly, for every active site, injected charge was increased progressively at a fixed frequency and pulse width by changing the stimulation amplitude. If the range afforded by the selected pulse width and the maximum deliverable current amplitude (imposed by the stimulator) was too small, then the pulse width was incremented, and the same thing was performed again. The threshold for minimum sensation was noted as soon as the participant detected any sensation related to the stimulation. The maximum parameters were saved when one of the following conditions was met: the sensation became painful, when the stimulation started inducing a muscle twitch, or simply when the participant did not feel comfortable increasing it further. This was repeated three times per active site, giving an average value. These two values, threshold and maximum, were saved for every active site and could be used later when choosing a modulation range. The effects of changing the frequency were not investigated in this work, and it was always fixed at 50 Hz. Injected current levels were always below the chemical safety limit of 120 nC for each stimulation site.

During the experiments reported in this work, a single tactile channel was used for sensory feedback in both participants at any given moment (the optimal electrode and active site for the experiments were chosen every week based on the sensations reported by the participants and were not always the same). The active sites used, as well as the sensations they elicited and the exact parameter ranges, are reported in fig. S1. The measured force applied by the prosthetic digits was encoded using a linear amplitude modulation scheme designed to associate perceived stimulation intensity with measured force. Parameters were chosen in such a manner as to optimally cover the whole dynamic range of sensations reported by each participant. For participant 1, tactile feedback was provided using charge-balanced, square pulses with an amplitude between 230 and 500 μA and a pulse-width duration between 80 and 120 μs, which resulted in a sensation of vibration referred to the base of the middle finger. For participant 2, tactile feedback was provided using an amplitude between 90 and 980 μA and a pulse-width duration between 50 and 200 μs depending on the day, which always resulted in a sensation of pressure or contraction referred to most of the ulnar innervation area (although less intense over the fourth finger). A more detailed set of parameters is provided in fig. S1B. Only two sets of parameters were used simultaneously at any given time (one for each feedback channel, respectively, tactile and remapped proprioceptive). The high number of parameters reported in the table (fig. S1B) is a result of changes in parameters between days and sessions, especially for participant 2, who performed these experiments soon after implantation, when stimulation parameters may still vary strongly from day to day. The mapping procedure was repeated every week, often leading to the discovery of better active sites and sensations that were then used during the experiments.

For the typical time scales involved in our experiments (trials lasting on the order of minutes), neither of our participants reported relevant changes in sensation intensity, which would indicate the presence of adaptation. Such effects were anecdotally observed only for much higher stimulation durations (tens of minutes). For all practical purposes, adaptation was insignificant during our experiments.

Sensory substitution for remapped proprioceptive feedback

To convey position information to the participants, we used sensory substitution. To avoid any cross-talk with tactile feedback, we used an active site resulting in a sensation that was not referred to the fingers. In participant 1, the selected stimulation parameters resulted in paresthesia located in the lower palm area, whereas in participant 2, the area involved was the medial part of the forearm, occasionally extending into the lower palm and wrist. Encoding of the position information retrieved from the robotic hand (a value between 0 and 255, corresponding to hand aperture angles of 0° to 110°, as measured on the robotic hand) was achieved using a simple linear encoding scheme. After establishing a suitable modulation range for each selected active site, the hand position value was used to modulate stimulation amplitude, whereas pulse width and frequency were kept constant (f = 50 Hz). Amplitude modulation resulted in changes to the perceived sensation intensity. The range of parameters used for stimulation was as follows: an amplitude between 230 and 260 μA and a pulse-width duration of 80 μs for participant 1 and an amplitude between 100 and 600 μA and a pulse-width duration of 100 or 200 μs depending on the day for participant 2.

Both participants underwent a brief learning session (<20 min) to help map the stimulation intensity to the prosthesis opening angle. We first instructed each participant to explore the new information by looking at the robotic limb while it was passively opened and closed. Then, we turned the control on and instructed the participant to actively explore their environment, grasping various objects and performing opening and closing movement with the prosthesis. Both participants quickly expressed confidence in interpreting the sensation, as well as a readiness to initiate the trials. Over the entire duration of the trials, the subjective experience associated with the remapped proprioceptive stimulation remained constant (perceived as paresthesia or contraction, respectively). Both participants reported a complete inability to perform the functional tasks reported in this study when not provided with sensory feedback.

Neural encoding of somatotopic tactile and remapped proprioceptive information

Two types of information were provided using the same neural interface in this study: somatotopic tactile and remapped proprioceptive. Remapped proprioceptive information was computed as followsEmbedded Image

Embedded Image

where RPSA, RPSPW, and RPSF are the remapped proprioception stimulation amplitude, pulse width, and frequency, respectively; Embedded Image

Embedded Image

is the amplitude corresponding to sensation threshold; Embedded Image

Embedded Image

is the maximum allowed amplitude (discomfort); θ is the recorded degree of closure of the fingers (one degree of freedom); and θmax is the maximum degree of closure the prosthesis can achieve. A constant value c was used for the pulse width but varied between sessions, as described in fig. S1B.

Remapped proprioceptive feedback was used during the TDPM, JAR, object size, object size and compliance, and the multijoint object size tasks. When two channels of proprioceptive feedback were needed during the multijoint object size task, the same encoding strategy was used with two distinct active sites.

On the other hand, as reported above, somatotopic tactile feedback was delivered using an active site resulting in a referred sensation on the phantom fingers. This approach was analogous to the approach presented in Raspopovic et al. (4) and is described hereEmbedded Image

Embedded Image

where TSA, TSPW, and TSF are the tactile feedback stimulation amplitude, pulse width, and frequency, respectively; Embedded Image

Embedded Image

is the amplitude corresponding to sensation threshold; Embedded Image

Embedded Image

is the maximum allowed amplitude (discomfort); p is the recorded tension in the finger tendon sensors; and pmax is the maximum tension that can be measured by the prosthesis under our experimental conditions (determined empirically). This feedback method was only used during the object size and object size and compliance task, which were the only conditions where tactile information was delivered alongside remapped proprioceptive information.

Noninvasive sensory substitution

For noninvasive sensory substitution, electrotactile stimulation was used (41), as shown in fig. S8. A surface stimulator (RehaStim, Hasomed, Germany) was used to deliver square charge balanced biphasic pulse trains with a fixed frequency (50 Hz). A pair of stimulation electrodes were placed on the shoulder of the participant (PALS neurostimulation electrodes, Axelgaard, USA; round, 2.5 cm in diameter), in such a way as to elicit only in-loco sensations under the electrodes [without distally referred sensation as described in (10)]. Perceptual thresholds and pain thresholds were identified using the same approach used for intraneural stimulation described above. In the case of noninvasive sensory substitution, the prosthetic hand’s finger position was encoded using pulse-width modulation of the delivered pulses. Pulse-width modulation was used rather than current amplitude modulation, simply because the RehaStim stimulator offers finer control over the injected charge by changing pulse width rather than amplitude (amplitude in steps of 2 mA and pulse width in steps of 20 μs).

In addition, during the multijoint remapped proprioception task, two channels of proprioceptive feedback were used. In this case, the two remapped sensations were delivered using two stimulation sites leading to distinct spatial sensations (i.e., two different locations on the shoulder).

Threshold to detection of passive motion

During the TDPM task, the robotic hand was moved passively using a software interface controlled directly by the experimenter. Remapped proprioceptive information was provided during the entire trial. The participants were instructed to announce when a movement was felt and in what direction. Whenever a movement was detected, the initial position and the detection position were saved. Then, after a small pause, the experiment continued starting from the last position. During these experiments, the participants were acoustically and visually isolated, using a sleeping mask and a set of headphones playing music. Falsification trials with no stimulation were also carried out. Prosthesis actuation speed was 27.5° s−1. To eliminate the possibility that time of actuation was being used as a proxy for degree of closure, we used a random amount of time between each repetition. Thus, the time between the beginning of each trial and the first movement of the hand was not fixed.

An important limitation in the setup used to perform the TDPM test with participant 1 was found a posteriori and subsequently resolved for the experiments with participant 2. Specifically, the software used to generate passive movements (used for participant 1) was found to be inaccurate, resulting in an average minimum finger movement of 9.5° ± 5.5°. That is, in the case of participant 1, when passively moving the hand, the experimenter could not generate movements smaller than 9.5° on average. Consequently, if the “true” TDPM precision was lower than this (as it was found to be for participant 2), then our experimental setup would not have been accurate enough to measure it. This is an important limitation to keep in mind when looking at the TDPM results for participant 1. For participant 2, the control algorithm for passively moving the hand was modified to ensure that the minimum finger displacement would be of lower magnitude (1.25°, fixed).

Joint angle reproduction

We performed two variants of the JAR test. In the first variant, the participants were instructed to bring the hand to one of four self-selected positions. Before starting the experiments, we asked each participant to show us the chosen positions using their intact hand. This was performed to ensure that they had understood the task. During the rest of the trial, the positions were recalled from memory. For every requested position, the final position of the hand was recorded, and after a brief pause, the next position was requested (the same position was never asked twice in succession). Participant 2 performed an additional set of control trials, where the prosthesis actuation speed was randomly drawn from a set of three possible speeds (22°, 43°, and 68° s−1).

In the second JAR variant, there were no predefined positions. Instead, the robotic hand was closed to a random and continuous position passively by the experimenter, and the participants could “feel” the sensation for a few seconds. The hand was then opened again, and the participants were instructed to bring it back to the same position actively. Both the initial position and the reproduced position were recorded, before the next repetition would start. During all variants, the participants were acoustically and visually isolated, as described above. In addition, falsification trials with no stimulation were carried out. As with the TDPM task, the time between the beginning of each trial and the first movement of the hand was not fixed during the last JAR variant (this was impossible during the first variant, because the trial was initiated by the participant).

Object size identification

During the size identification task, four three-dimensional (3D)–printed cylinders of equally spaced diameters were used (2, 4.33, 6.66, and 9 cm, referred to as sizes very small, small, large, and very large, respectively). The choice of four cylinders was based on pilot results that indicated that using a smaller number would result in the task not being challenging enough (fig. S5B). After being acoustically and visually isolated, both participants were asked to close the robotic hand while one of the four objects was placed in its grip. The participants announced which object was thought to be held in the hand, and both the actual object and reported object were recorded. A simple control trial with no stimulation was also carried out. In addition, participant 2 performed a series of control trials. First, the same task was carried out using only tactile feedback. Second, the task was performed with both tactile and remapped proprioceptive feedbacks together. Last, the task was performed while prosthesis actuation speed was randomly drawn from a set of three possible speeds (43°, 68°, and 86° s−1).

To establish a baseline precision of natural hand proprioception, we recruited five right-handed healthy participants to perform the same size recognition task. Their right arms were placed in a fixed position on a table, allowing for palmar grasps. To more closely match the experiment performed with the robotic limb, we presented the objects in such a manner that they would not touch the thumb, being wedged instead between the fingers and the palm.

Combined size and compliance identification

The combined size and compliance identification task was performed the same manner as the object size identification experiment described above. Here, the objects had two different sizes and two different compliances (hard 3D-printed plastic and soft foam), allowing a total of four different combinations. In addition to the remapped proprioceptive stimulation provided in all the other experiments, tactile feedback was delivered by means of electrical nerve stimulation during this trial. Different object compliances led to different temporal dynamics in the hand sensors readout (as visible in Fig. 4B), which in turn caused different stimulation profiles to be delivered to the participant (using linear amplitude modulation). Both participants performed this task. In addition to the base task, participant 2 performed two control conditions. In the first condition, the same task was performed while only remapped proprioceptive feedback was turned on. In the second condition, the same was performed with only tactile feedback turned on. To establish a baseline precision of natural hand proprioception, we recruited two right-handed healthy participants to perform the same size recognition task.

Multijoint remapped proprioception task

Instead of providing one channel of tactile feedback and one channel of remapped proprioceptive feedback, as in previous tasks, participant 2 also performed a task where two channels of remapped proprioceptive feedback were provided simultaneously. In this case, one channel encoded the degree of closure of the median area (first three fingers), whereas the second channel encoded the degree of closure of the ulnar area (last two fingers). In this case, two channels giving distinct sensations on the forearm were used, with the same overall approach described above. With this multijoint feedback, participant 2 was asked to recognize four conditions: two small objects placed in the median region and ulnar region, a small object placed in the median region and a large object placed in the ulnar region, the opposite condition with the small object in the ulnar region, and a final condition with two large objects. The rest of the task’s details were kept identical to the object size experiment described above.

Cognitive load task

All functional tasks were repeated under increased cognitive load with participant 2. A verbal fluency task was performed alongside the experiments, where the participant was given a letter and was asked to say words starting with the chosen letter as quickly as possible. The average spoken word rate was computed for each task (words per minute) to confirm that the participant was constantly performing the verbal task with the same intensity. Task performance for each type of experiment (object size, object size and compliance, and multijoint remapped proprioception) was measured the same manner as during the noncognitive loaded tasks to compare the effect of performing a mentally demanding task in parallel with each of the functional experiments.

Embodiment evaluation

The embodiment questionnaire was administered after each type of active task (object size, multijoint object size, and object size and compliance tasks). A series of 10 questions were used, adapted from previous studies on prosthesis embodiment to fit our experiments (proprioceptive feedback instead of tactile feedback) (13, 4244). Half of the questions were control questions, designed to dismiss the possibility that the participant was suggestible. The participant was asked to report how much she agreed with each statement on a scale ranging from 0 (not at all) to 6 (completely). The five questions, which tested embodiment, are reported in Fig. 5. The five control statements were “I felt like the stump started to move toward the robotic hand,” “I felt like I had three hands,” “I felt the sensation somewhere between the stump and the robotic hand,” “I felt like the robotic hand started to move toward the stump,” and “The robotic hand started to look like my real hand, in terms of shape, skin tone, or other characteristics.”

Statistics and data analysis

All data were analyzed using MATLAB (R2016a, The MathWorks, Natick, USA). All statistics were performed using the available built-in functions. A one-sample Kolmogorov-Smirnov test was used to determine whether the datasets associated with the various experiments were normally distributed. None of our datasets passed the test because they are highly asymmetrical due to the nature of the tasks. We therefore used nonparametric alternatives [Kruskall-Wallis instead of analysis of variance (ANOVA)] and reported the median and IQR instead of the average and SD. All reported P values resulting from Kruskall-Wallis tests measure the significance of the chi-square test statistic. When appropriate, multigroup correction was applied using Tukey’s honestly significant difference procedure [multcompare(), MATLAB]. Spearman’s rank correlation coefficient was used to test whether the scatterplots shown in Figs. 2 and 6 had correlation values significantly different from 0. Spearman’s rank correlation was used instead of Pearson’s linear correlation coefficient, which assumes normality. To measure the spread of data in the JAR experiments (Fig. 2), we used the robust and nonparametric MAD. In Fig. 2F, a Kruskall-Wallis statistic was computed to test the hypothesis that the measured deviation was dependent on the position tested. A multiple comparison correction was applied. Levels that were found to be statistically different are marked with an asterisk (P < 0.05). All plots representing median and 95% CI in Figs. 3 (B, C, E, and F), 4 (C to F), and 6 were generated using a binomial parameter estimate, with chance level being estimated at 25% for correctly recognizing one among four objects and 50% for correctly identifying one feature (hard versus soft and big versus small). Nonoverlap of 95% CIs was used as a sufficient criterion to identify statistically significant differences, whereas a Fisher’s exact test was used in cases where it was not possible to draw conclusions directly from the intervals (i.e., 95% CI overlap). Additional details about the number of repetitions for each experiment are reported in the corresponding figure legends. When random numbers were needed (e.g., generating object presentation sequences), random permutations of an equipopulated sequence [randperm(), MATLAB] were used.

SUPPLEMENTARY MATERIALS

robotics.sciencemag.org/cgi/content/full/4/27/eaau8892/DC1

Fig. S1. Reported referred sensations and stimulation parameters.

Fig. S2. TDPM and JAR performance broken down by participant.

Fig. S3. Object size and compliance recognition broken down by participant.

Fig. S4. Performance of healthy controls during object identification tasks.

Fig. S5. Control condition and time progression of object recognition tasks.

Fig. S6. Multijoint proprioceptive task performance.

Fig. S7. Functional tasks performed under increased cognitive load.

Fig. S8. General overview of the two sensory substitution approaches compared in this study.

Fig. S9. Comparison of invasive and superficial sensory substitution on embodiment.

Movie S1. Sensing object size (participant 1).

Movie S2. Sensing object size (participant 2).

Acknowledgments: We are grateful to the participants who willingly committed 6 months of their lives for the advancement of knowledge and for a better future of persons that undergo hand amputations. Funding: This work was supported by the EU grant CP-FP-INFSO 224012 (TIME project), the project NEMESIS (NEurocontrolled MEchatronic hand prostheSIS) funded by the Italian Ministry of Health, the EU grant FP7-611687 NEBIAS (NEurocontrolled BIdirectional Artificial upper limb and hand prosthesiS), and the EU grant FP7-HEALTH-1602547 EPIONE (natural sensory feedback for phantom limb pain modulation and therapy). The work was also supported by the Swiss National Science Foundation through the National Centre of Competence in Research (NCCR) Robotics and by the Bertarelli Foundation. Author contributions: E.D., G.V., A.M., and S.M. conceived the experiments. E.D., G.V., I.S., A.M., and J.P. conducted the experiments. E.D. analyzed the results and created the figures. E.D. and S.M. wrote the manuscript. E.D., G.V., I.S., S.R., and F.M.P. characterized sensations elicited by stimulation during experiments. G.G., R.D.I., and P.M.R. recruited the participants and were responsible for all the clinical activities. M.C. and C.C. developed the sensorized hand prosthesis. T.S. developed the TIMEs. All authors reviewed the manuscript. Competing interests: S.R., F.M.P., and S.M. hold shares of “Sensars Neuroprosthetics,” a startup company dealing with potential commercialization of neurocontrolled artificial limbs. M.C. and C.C. hold shares of “Prensilia,” a startup company commercializing robotic hands. The other authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper or the Supplementary Materials.

Science Robotics

Share This Post

Leave a Reply