Virtual reality validation of naturalistic modulation strategies to counteract fading in retinal stimulation

Experiments were approved by the human research ethics committee of École polytechnique fédérale de Lausanne (decision number 042-2018/16.10.2018). Ten sighted volunteers were involved in the study (table 1).

SPV was generated by adapting a previously described approach [22]. The experiment was performed using a Dell Precision 3630 computer with an Intel Xeon E-2146G CPU (3.50 GHz) and an Nvidia GeForce GTX 1080 GPU. The VIVE Pro Eye head mounted display was used. Tracking of head orientation and position was provided using two VIVE Base Station 2.0 tracking cameras placed in opposite corners of the room (roughly 2.5 m apart) and aimed at the centre of the room where the participant was sat. SPV was developed using Unity and computed using Cg shaders, allowing real-time operation. The code is available online (https://github.com/lne-lab/polyretina_vr). Both the spatial and temporal properties of phosphenes were considered. Images were converted into phosphenes distributed over a visual angle of 45° using the layout of the POLYRETINA prosthesis (80/120, electrode diameter in µm/electrode pitch in µm) [18]. The total number of phosphenes was 9914. For each frame, three sources of random variability were included: the phosphene size was randomly varied between ±30% of the electrode size, the phosphene brightness was randomly varied between 50% (grey) and 100% (white) of the phosphene's default brightness, and 10% of the electrodes were considered not functional. Then, a distortion due to unintended activation of axon fibres was introduced (λ = 2) [22]. The frame rate of SPV was set to 5 Hz with a 11 ms frame duration (1 frame on/17 frames off). SPV was presented to the right eye only.

A novel aspect of the SPV was the inclusion of perceptual fading, which is the reduction in brightness of phosphenes over a short amount of time due to retinal desensitisation.

We previously showed using blind retinal explants upon repeated illumination of the same POLYRETINA electrode at 5 Hz that the RGC response persisted for only 0.4 s [7]. In agreement with previous results describing a two-phases desensitisation process in RGCs [6], we found a first rapid decay occurring immediately after the first stimulation pulse followed by a slow decay during prolonged stimulations. On average, the rapid decay occurred over the first 25% of the total response duration (i.e. 0.1 s) and the slow decay for the remaining 75% of the total response duration (i.e. 0.3 s).

In SPV, phosphene fading was simulated via a decrease in brightness (figure 1(a)) whose dynamics matched that of RGCs desensitisation. The brightness decay was approximated as a double linear decay with two time constants (t_f, fast decay constant; t_s, slow decay constant). Brightness (B) decreased linearly in t_f time until at 50% brightness and then decreased linearly in t_s time until 0% brightness as in equation (1), where t_a is the elapsed time since the phosphenes onset. Brightness values range between 0 and 1. This range was chosen specifically because it corresponds to the values expected by application programming interfaces when rendering pixels on the screen

$$\begin{equation}B = \left\{ \begin{aligned} &1 - \frac{{{t_{\text{a}}}}}{{2{t_{\text{f}}}}},\;\;\;{\text{if}}\,{t_{\text{a}}} < {t_{\text{f}}} \\ &0.5 - \frac{{{t_{\text{a}}} - {t_{\text{f}}}}}{{2{t_{\text{s}}}}},\;\;\,{\text{if}}\,{t_{\text{f}}} < \,{t_{\text{a}}} < {t_{\text{f}}} + {t_{\text{s}}} \end{aligned} \right..\end{equation} \tag{ 1 }$$

Zoom In Zoom Out Reset image size

Phosphene recovery was simulated via an exponential increase in brightness (figure 1(b)) whose dynamics matched experimental evidence of recovery of retinal excitability after desensitisation [8, 34]. This recovery is a time-dependent process, during which the neuronal membrane is absolutely refractory immediately after a spike train and then gradually increases to its resting potential [35, 36]. Similarly in SPV, individual pixel brightness switched from decay to recovery after 2 s without stimulation (absolute refractory period). The brightness recovery followed an exponential growth (equation (2)), according to classical models of neuronal excitability [37–39], where t_n is the time the phosphene has been inactive, a is the time for the phosphene to completely recover (set to 2 s) and b defines the arc of recovery (set to 3)

$$\begin{equation}B = 1 - {\left( {1 - \frac{{{t_{\text{n}}} - 2}}{a}} \right)^b}.\end{equation} \tag{ 2 }$$

In SPV, the 'normal fading' (F) condition corresponds to the natural desensitisation of the retina upon 5 Hz stimulation from the same electrode (figure 1), while the 'no fading' (NF) condition corresponds to stable brightness over time (B = 1). Concurrently, compensation approaches for fading were simulated based on the naturalistic spatiotemporal modulation strategies previously characterised with retinal explants from blind mice (figure 2) [7]. The 'saccade' (S) strategy is a spatial modulation strategy (figure 2(a)). It simulates natural microsaccades by having the entire image oscillate horizontally by the pitch of one phosphene once every second. This way, new phosphenes will become active and refresh at least some part of the prosthetic image, thus counteracting fading. The total response duration in RGCs was measured by alternating the illumination (10 ms light pulses at 5 Hz repetition rate) of two neighbouring electrodes at 1 Hz switching rate. The RGC response persisted for 1.8 s. The 'random' (R) and the 'interrupt' (I) conditions are temporal modulation strategies. In the R condition, the duration of the light pulse was varied, leading to variable inter-pulse intervals (figure 2(b)). This strategy emulates the highly irregular temporal profile of light reaching a steady location on the retina due to both eye and object movements. This dynamic change contributes by counteracting the desensitisation of bipolar cells to natural stimuli [40]. Similarly, it reduces desensitisation during static electrical stimulation [7, 41]. In the R condition, the RGC response persisted for 0.8 s. In the I condition, for each five pulses, only the first three were delivered (figure 2(c)), since the last would not have generated a RGC response anyway. The RGC response persisted for 0.4 s, as in the static stimulation. All three strategies were temporally synchronous across each phosphene. The remaining conditions, were a combination of the three strategies already described: saccade/random (SR, figure 2(d)), saccade/interrupt (SI, figure 2(e)), random/interrupt (RI, figure 2(f)), and saccade/random/interrupt (SRI, figure 2(g)).

Zoom In Zoom Out Reset image size

The rate of brightness decay in SPV for the compensation approaches was generated from the total response duration measured in RGCs [7]. The durations were split with a ratio of 1:3 to produce the fast (t_f) and slow (t_s) decay constants used in equation (1). For each approach, the computed time constants are reported in table 2.

A representative example of image perception at various frames is shown in figure 3 for various conditions.

Zoom In Zoom Out Reset image size

The fading logic was applied simultaneously to each phosphene every time there was a stimulus presentation (figure 4). Parameters were updated in real-time on the GPU where the simulation is also being processed. To read/write such an amount of data on the GPU we used multiple render textures and accessed them in a double-buffered fashion to create a read/write data matrix where the values could be stored and updated in real-time. In other words, all processing and necessary data for the simulation remained solely on the GPU, where computations could be completed fast enough to enable real-time SPV.

Zoom In Zoom Out Reset image size

The experiment consisted of a single repeated task in which participants had to read aloud three six-letter words. Words were presented in the mother tongue of the participant and on separate lines. Each word was 7° tall and the entire stimulus (i.e. the three words) took up 27° vertically and around 40° horizontally. Participants were informed that phosphenes would appear and fade over a few seconds and that head movements could be used to reduce fading. Participants could rotate their heads in order to adjust their view of the words. However, translational movements did not affect the view of the stimulus. Only rotational movements were used as they are more natural and comfortable for participants to perform. A similar approach was taken to analyse only the angular head movement of patients implanted with the Argus® II [20]. Once a participant had given their answers, the experimenter would mark the trial as finished and the participant would have a 2 s pause until the next trial. As participants were able to move their heads, their virtual head position was reset at the start of each trial.

The task was designed to not be too difficult but to also take time to complete. As the study was focused on the impact of phosphene fading, it was important that participants could not complete the task in such a short amount of time that the fading was not perceived. By having three words instead of one, it guaranteed that a participant could not complete the task before the image had completely faded prior to any accommodating head rotation. On the other hand, the words were displayed with a generous size, so that participants could actually complete it. SPV includes several laborious aspects of artificial vision such as a low spatial resolution, restricted field of view and refresh rate, as well as unintended visual distortions from axon fibre activation and perceptual desensitisation. In pilot tests, making words smaller, in combination with the already unaccommodating SPV, increased the difficulty to unacceptable levels. Another important aspect of the task was that the three words fit entirely within the field of view of the prosthetic vision (45°). By doing this, any head rotation from participants could be largely attributed to their purposeful effort to reduce fading and not due to participants having to move their heads simply to see the stimulus.

The task was completed under the nine different conditions, presented randomly. Each condition was presented five times for a total of 45 trials per session. Participants were asked to come in for a total of six sessions to control for learning effects. Therefore, each participant completed 270 trials (30 per condition). Participants were given instructions on the task before starting. Most importantly, they were given the information that the SPV would fade over time and that head rotations could be used to counteract the fading.

Data analysis was performed in MATLAB (MathWorks). The head rotation data consisted of a non-zero amount of quaternion values depending on the duration of the trial (figure 5(a)). Quaternions are four-dimensional vectors which describe the rotation of an object around three perpendicular axes. To analyse the data, each quaternion was first converted into rotations around the vertical and lateral axes (figure 5(b)), since rotations were almost absent in the longitudinal axis. Then, head rotation data was split into two separate measurements: one focused on the magnitude of head rotations and the other on the spread of head rotations. Magnitude measures the amount that participants had to move their heads during a trial while spread measures the extent of dispersion of head movements away from the stimulus. Here, a head movement away from the stimulus refers to a rotation of the head that produces an angle between the centre of the stimulus (i.e. the three words) and the gaze position. This could be achieved through either lateral or vertical head rotations or a combination of both. Measurements were split this way due to the possibility that participants might have moved their heads a lot, but only in a small area, or alternatively, only had to move their heads a few times, but did so in large arcs.

Zoom In Zoom Out Reset image size

To measure the magnitude, head rotations from each trial were segmented into individual rotation vectors (figure 5(b)). Segments were identified using a peak analysis on the head's inverse angular velocity. The peak analysis was performed using the MATLAB function 'findpeaks'. This function returns all local maximums from a one-dimensional signal. In this case, the signal was the head's inverse angular velocity. The signal was inverted so that the peaks corresponded to the instances when participants' were slowing their head movement. Segments were then filtered using the k-means algorithm using the peak's height, width and the turning angle at the point of the peak. The first and last data points of each trial were always marked as peaks. If multiple data points were identified for a single segment then the data point closest to the correct k-means cluster centre was chosen. The segment magnitude was calculated by summing the rotation between each data point in the segment. Finally, the magnitude of these vectors was averaged. This approach was chosen over simpler options such as the total head rotation as this was heavily affected by the time taken to complete the task. Similarly, head velocity proved to be a misleading metric as participants' head rotations were often discontinuous, pausing while participants focused on the stimulus and then subsequently moving as fading began to degrade vision. As a consequence of the stop-start nature of head rotations, the data could be segmented into individual rotation vectors, with an associated magnitude.

To measure the spread of head rotations, we calculated the average rotation of the head away from the stimulus for each trial, using the MATLAB function 'dist'. This function returns the angle between two quaternions in unit radians, which is analogous to the Euclidean distance between two points, substituting x and y positions for lateral and vertical rotations. Using this function in combination with an identity quaternion angled directly towards the stimulus, an angle was calculated for each head rotation away from the stimulus at each frame of a trial (figure 5(c)). Then, the data was converted into degrees from radians and averaged per trial.

Statistical analysis and graphical representation were performed with Prism 8 (Graph Pad). The D'Agostino and Pearson omnibus normality test was performed to justify the use of a non-parametric test. The box plots extend from the 25th to 75th percentiles. The line is the median. The + is the mean. The whiskers extend to the smallest and the largest value. In plots, p-values were reported as: * p < 0.05, ** p < 0.01, *** p < 0.001, and **** p < 0.0001.

First, we evaluated the time taken by each subject to complete the task (figure 6). This parameter provides the first information on the performance for each condition.

Zoom In Zoom Out Reset image size

The Friedman one-way repeated measure analysis of variance by ranks reported a significant difference among the conditions (p < 0.0001). Dunn's multiple comparison test showed that three conditions resulted in a time to complete the task significantly lower than the fading condition: RI (p < 0.0001), SRI (p = 0.0393) and NF (p < 0.001). The remaining conditions (S, R, I, SR, SI) do not result in a statistically significant difference in the time to complete the task compared to the fading condition (table 3), indicating that those strategies might not be as powerful as the others. Interestingly, the RI and SRI strategies are not only significantly better than F condition, but also reported results statistically similar to the NF condition, which indicates that they are extremely efficient in counteracting fading. The other strategies (S, R, I, SR, SI) are significantly worse than NF condition, which confirms their poor performance in counteracting fading. In summary, the most promising strategies to counteract fading were RI and SRI.

In order to counteract the fading of phosphenes during prosthetic vision, implanted patients are instructed to continuously move their head. The side-effect is a high cognitive demand and a decline in comfort of the use of the device from constant scanning. We hypothesised that head movements could be reduced by employing strategies to slow down the fading.

3.2.1. Magnitude of head rotations

During trials under normal fading (F), we observed large head rotations away from the stimulus (figures 7(a) and (b)), while head rotations were virtually absent during the NF condition (figures 7(e) and (f)). Spatiotemporal modulation strategies allowed the reduction of head rotations by a variable factor depending on the strategy (figures 7(c) and (d) for RI strategy).

Zoom In Zoom Out Reset image size

The average magnitude of the head rotations was calculated then for all the conditions to determine which strategy would provide a statistically significant reduction (figure 8).

Zoom In Zoom Out Reset image size

The Friedman one-way repeated measure analysis of variance by ranks reported a significant difference among the conditions (p < 0.0001). Similar to the time to complete the task, Dunn's multiple comparison test showed that four conditions resulted in a magnitude of head rotations significantly lower than the fading condition: SI (p = 0.0293), RI (p < 0.0001), SRI (p = 0.0023) and NF (p < 0.001). The remaining conditions (S, R, I, SR) do not result in a significant difference in the magnitude of head rotations compared to the fading condition (table 4), indicating that those strategies might not be as powerful as the others. Interestingly, SI, RI and SRI strategies are not only significantly better than the F condition, but also reported results statistically similar to the NF condition, which indicates that they are extremely efficient in counteracting fading and reducing the magnitude of head rotations. The other strategies (S, R, I, SR) are significantly worse than the NF condition, which confirms their poor performance. In summary, the most promising strategies to reduce the magnitude of head rotations were SI, RI and SRI.

Table 4.  P-values from the Dunn's multiple comparison test. Significant comparisons are in bold (p < 0.05).

	F	S	R	I	SR	SI	RI	SRI
S	>0.9999	—	—	—	—	—	—	—
R	>0.9999	>0.9999	—	—	—	—	—	—
I	=0.1184	=0.9895	>0.9999	—	—	—	—	—
SR	>0.9999	>0.9999	>0.9999	=0.6441	—	—	—	—
SI	=0.0293	=0.3233	=0.6441	>0.9999	=0.1981	—	—	—
RI	<0.0001	=0.0005	=0.0016	>0.9999	=0.0003	>0.9999	—	—
SRI	=0.0023	=0.0393	=0.0907	>0.9999	=0.0218	>0.9999	>0.9999	—
NF	<0.0001	<0.0001	<0.0001	=0.1184	<0.0001	=0.4093	>0.9999	>0.9999

3.2.2. Spread of head rotations

Next, we quantified the spread of the head rotations. Similar to the magnitude, during trials under normal fading (F), we observed a spread of rotations away from the stimulus in both vertical and lateral direction (figure 9, black). Modulation strategies reduce the spread by a variable factor depending on the strategy (figure 9, red for RI strategy), which are further reduced in NF condition (figure 9, blue).

Zoom In Zoom Out Reset image size

The average spread of the head rotations was then calculated for all the conditions to determine which strategy would provide a statistically significant reduction (figure 10).

Zoom In Zoom Out Reset image size

The Friedman one-way repeated measure analysis of variance by ranks reported a significant difference among the conditions (p < 0.0001). Similar to the previous analysis, Dunn's multiple comparison test showed that five conditions resulted in a spread of head rotations significantly lower than the fading condition: I (p = 0.0393), SI (p = 0.0016), RI (p < 0.0001), SRI (p < 0.0001) and NF (p < 0.001). Also, the remaining conditions (S, R, SR) do not result in a significant difference in the spread of head rotations compared to the fading condition (table 5), indicating that those strategies might not be as powerful as the others. Interestingly, I, SI, RI and SRI strategies are not only significantly better than the F condition, but also reported results statistically similar to the NF condition, which indicate that they are extremely efficient in counteracting fading and reducing the spread of head rotations. The other strategies (S, R, SR) are significantly worse than the NF condition, which confirms their poor performance. In summary, the most promising strategies to reduce the spread of head rotations were I, SI, RI and SRI.

Table 5.  P-values from the Dunn's multiple comparison test. Significant comparisons are in bold (p < 0.05).

	F	S	R	I	SR	SI	RI	SRI
S	>0.9999	—	—	—	—	—	—	—
R	>0.9999	>0.9999	—	—	—	—	—	—
I	=0.0393	>0.9999	>0.9999	—	—	—	—	—
SR	>0.9999	>0.9999	>0.9999	>0.9999	—	—	—	—
SI	=0.0016	=0.6441	=0.1536	>0.9999	=0.5150	—	—	—
RI	<0.0001	=0.0045	=0.0005	=0.8008	=0.0032	>0.9999	—	—
SRI	<0.0001	=0.0907	=0.0161	>0.9999	=0.0690	>0.9999	>0.9999	—
NF	<0.0001	<0.0001	<0.0001	=0.0522	<0.0001	=0.6441	>0.9999	>0.9999

Last, we asked if there was any learning effect during the task. Participants completed six sessions in total, spread out over at most two weeks, in order to see what effect practice would have on performance. The dataset was then separated for each session (figure 11). For the time to complete the task, we observed a marked learning effect, shown by a performance increase after each session (figure 11(a)). When only the last sessions for each condition are compared, all conditions approach a qualitatively similar performance not significantly different than the F condition (p > 0.9999 for conditions S, R, I, SR, SI, RI and SRI; Kruskal–Wallis with Dunn's multiple comparison test), except for the NF condition which remains significantly better than the F condition (p = 0.0006; Kruskal–Wallis with Dunn's multiple comparison test).

Zoom In Zoom Out Reset image size

This result indicates that learning alone helps subjects to reduce the time to complete the task, independently of the use of fading compensation strategies. However, the situation is remarkably different for head rotations. Learning is not observed for both the magnitude (figure 11(b)) and the spread (figure 11(c)) of head rotations. For the magnitude, considering only the last session for each condition, RI strategy is significantly better than the F condition (p = 0.0049; Kruskal–Wallis with Dunn's multiple comparison test) together with NF (p < 0.0001; Kruskal–Wallis with Dunn's multiple comparison test); also, RI and NF showed a statistically similar magnitude of head rotations (p > 0.9999; Kruskal–Wallis with Dunn's multiple comparison test). A similar result was shown for the spread of head rotations. When considering only the last session for each condition, RI strategy is significantly better than the F condition (p = 0.0084; Kruskal–Wallis with Dunn's multiple comparison test) together with NF (p < 0.0001; Kruskal–Wallis with Dunn's multiple comparison test); also, RI and NF showed a statistically similar magnitude of head rotations (p > 0.9999; Kruskal–Wallis with Dunn's multiple comparison test).

In summary, this data indicates that practice improves the time to complete the task, but only modulation strategies (in particular the RI strategy) can significantly reduce the burden associated with active head scanning.