Studies have documented that utilizing peer feedback can enhance students’ English academic writing skills. Little is known, however, about the effects of incorporating peer feedback to enhance English as a second language (L2) medical students’ academic writing performance.n = 124).This longitudinal interventional study examines Chinese medical students’ English academic writing skills development via peer feedback in four parallel classes over an 18-week semester between the experimental and control groups , and the effects were retained in the delayed post-test 6 weeks later . Little improvement was found in the control group in the post-test  or the delayed post-test . Between-group comparisons indicate that the experimental group outperformed the control group in the post- and the delayed post-tests, as shown in their overall score and scores on the four components.Significant increases in the experimental group’s performance in the post-test were found after 18-week instructions (pre- vs. post-test: overall score, Incorporating peer feedback into process-oriented medical English writing classroom teaching can effectively enhance Chinese medical students’ English academic writing skills over time, while the traditional product-oriented writing instructions had little help in improving Chinese medical students’ academic writing skills. This longitudinal intervention study develops our understanding of the effectiveness of peer feedback in L2 academic writing pedagogy. It offers instructional implications for L2 writing teachers to teach English academic writing among medical students in China and beyond. Limitations and suggestions for future studies are discussed. Researchers have proposed several approaches for teaching writing, e.g., the product- and process-oriented approaches , 39. ThePeer feedback, a commonly used peer-supported positive learning strategy in the writing process, refers to a process where learners provide feedback on their peers’ work in oral or written forms and learners also receive feedback from other peers see , 45, 47), 4745, 4n = 198) in a Chinese university [Given the advantages of peer feedback above and the potential of utilizing peer feedback for English writing instructions, previous studies have incorporated peer feedback activities into the process-oriented approach for improving students’ writing performance e.g., , 42, 45), 4542, 4iversity . They deiversity  investigiversity . Additioiversity , 46), whiversity .RQ1: Does the use of peer feedback improve Chinese medical students’ English academic writing development over time?RQ2: Is there any difference in the effects of peer feedback and traditional writing instructions on Chinese medical students’ English academic writing development?To address some of the existing research gaps, the current study applies a longitudinal intervention design to investigate the effects of peer feedback on L2 medical students’ academic writing skills development over time. Specifically, this study is guided by the following two research questions (RQ):n = 62) and the control group (CG = 62). Students in the experimental group were 10 males and 52 females between 18 and 19 . Similarly, students in the control group were 9 males and 53 females between 18 and 19 . Regarding their past English learning experience, they started to learn English in the first year of primary school. In addition, the homogeneity tests were run to ensure students’ writing abilities were homogenous in the pre-test. The results illustrated that students’ writing skills in overall scores and the five subscores were similar at the pre-test  degrees in English language education and have taught English writing for 15 years. Four key genres  were chosen as the teaching target for both groups. Although the English writing tasks involved various topics such as education, career, and culture, most of the writing topics were health-related due to their medical major.Peer feedback activities were implemented in the experimental group and the writing course design. The writing teachers for the experimental group  incorporated peer feedback activities, while the teachers for the control group  applied the conventional product-oriented approach.In the experimental group, students received feedback on different writing dimensions from peers, and eventually completed a revised draft each week. To help students provide effective peer feedback, the Department of English of the university arranged weekly peer feedback workshops to teach students how to use a rubric to offer feedback on each dimension of their peers’ work. Specifically, the details of each dimension on the rubrics were explained in the weekly workshops. Writing samples for different score ranges were also presented, and teachers explained how to use the rubric to provide feedback to these writing samples, which could ensure the inter-rater reliability and the uniformity of the marking process to a large extent. Furthermore, when they experienced difficulties in giving feedback to their peers, they could immediately raise their hands and seek help from the teacher in writing classes. The rubric used in this study was adapted from the rubric for marking writing sections of College English Test-6 (CET-6) , which iTo alleviate external factors affecting learners’ writing performance, weekly pre-class teaching training sessions were organized with the two teachers for the experimental group to guide them to implement the same teaching procedures, and the two teachers for the control group attended other sessions for the same purpose. The first author observed the actual writing classes to ensure that teachers teach English academic writing in a planned manner.At the pre-test stage, a mock writing test for CET-6 was arranged by the Department of English at this university, and students should finish the writing section in the classroom. The writing samples were collected to obtain baseline data from the experimental and control groups. Then, 18-week writing instructions were carried out in the experimental group. At the post-test stage, another mock writing test was conducted again immediately after the interventions. In addition, to investigate the follow-up effects, we conducted a delayed post-test 6 weeks after the intervention was completed. The topics of the mock writing test for pre-, post-, and delayed post-tests were adapted from the writing section of the CET-6 test, which requires students to write an essay with around 200 words within 30 minutes. Three experienced English writing teachers were consulted to ensure the comparability and feasibility of the two writing topics. And then, students from another parallel class were invited to finish the two tasks to guarantee that the difficulty of the two topics was moderate for freshmen medical students.A double-blinded expert (two teachers) assessment mechanism was applied to score students’ writings. The 9-point scoring rubric was used to rate all essays from four dimensions with equal weight: task response, coherence and cohesion, lexical resource, and grammatical range and accuracy. Two teachers were invited to an academic seminar to guarantee acceptable inter-rater reliability and the uniformity of marking criteria. At first, twenty compositions were given to the two writing teachers, and they scored the compositions individually. Then, they went to check whether they could give the same mark on the same compositions. If not, they were asked to have a communication to explain why they scored this mark. During this process, they can agree on the scoring standard, and then they were allowed to evaluate the rest of the compositions independently. Finally, Cronbach’s Alpha was calculated and reported at 0.802 in pre-tests, 0.855 in post-tests, and 0.875 in delayed post-tests, which indicated their scoring has good reliability.ηρ2) was used for measuring the effect sizes for mixed ANOVAs. Cohen’d was calculated for the t-tests, and r was calculated for Wilcoxon and Mann–Whitney tests. The interpretation of the results was based on Cohen’s (1992) classification that ηρ2 values of 0.01, 0.06, 0.14; d values of 0.20, 0.50, and 0.80; and r values of 0.10, 0.30, and 0.50 were considered small, medium, and large, respectively.The assumptions of the statistical tests were checked via standard diagnostic tests and procedures. The visual inspection of histograms, normal Q-Q, normal P-P, box plots, skewness and kurtosis, and the Shapiro-Wilk tests were used to check the normality of distribution, Levene’s test was used for homogeneity of variance, and Mauchily’s test was used for sphericity of the data. Then, 2 × 3 mixed factorials analysis of variance (ANOVAs) and Friedman test were conducted to investigate the effects of peer feedback on L2 students’ writing skills. Besides, simple effect analysis and Wilcoxon signed-ranks tests were used as the posthoc test within groups. Where there were multiple comparisons, Bonferroni corrections were used to avoid Type I errors. Partial eta squared .Given that the assumptions of normal distribution, homogeneity, and sphericity were met, mixed ANOVAs were conducted for the overall writing scores and subscores including task response, coherence and cohesion, and lexical resource. Friedman test was conducted as an alternative analysis for grammatical range and accuracy scores since the scores were not normally distributed. The results of the baseline test illustrate that students’ writing skills in overall scores and the five subscores were similar at the pre-test  = 31.85, p < 0.001, ηρ2 = 0.15, a significant main effect of group, F  = 34.60, p < 0.001, ηρ2 = 0.09, and a significant interaction effect for time × group, F  = 18.61, p < 0.001, ηρ2 = 0.09, on students’ overall writing scores.As shown in Table p < 0.001; pre- vs. delayed post-test, p < 0.001), and the effect was retained in the delayed post-test . Nevertheless, the overall writing score of students in the comparison group did not improve significantly across the three tests .Then, a simple effect analysis was used to explain the time × group interaction effect further. Within-subject comparisons (see Table p = 0.115), while the experimental group significantly outperformed the comparison group in the post-test, p < 0.001, with large effect size, d = 1.02, and in the delayed post-test, p < 0.001, with large effect size, d = 1.04.Further, between-group comparisons  = 12.69, p < 0.001, ηρ2 = 0.07, a significant main effect of group, F  = 12.95, p < 0.001, ηρ2 = 0.03, and a significant interactive effect of time × group, F  = 6.08, p = 0.003, ηρ2 = 0.03, with respect to the task response score. Furthermore, a significant main effect of time, F  = 22.74, p < 0.001, ηρ2 = 0.11, a significant main effect of group, F  = 25.43, p < 0.001, ηρ2 = 0.07, and a significant interactive effect of time × group, F = 14.64, p < 0.001, ηρ2 = 0.07 were found in terms of the coherence and cohesion score. The mixed-design ANOVA results also showed that there was a significant main effect of time on lexical resource, F  = 17.75, p < 0.001, ηρ2 = 0.09, a significant main effect of group, F = 21.36, p < 0.001, ηρ2 = 0.06, and a significant interactive effect of time × group, F = 9.61, p < 0.001, ηρ2 = 0.05.The results of 2 × 3 mixed factorials analysis of variance (ANOVAs) suggested that there was a significant main effect of time, p < 0.001; pre- vs. delayed post-test, p < 0.001), and the effect was maintained in the delayed post-test , whereas the score for the control group did not improve over time . Moreover, a simple effect analysis revealed that peer feedback enhanced the coherence and cohesion scores for students in the intervention group, and the effect was retained in the delayed post-test . However, no significant improvement was found for the comparison group over the three-time points . Simple effect analysis also shows that the lexical resource scores improved significantly for the experimental group , and the effect was retained in the delayed post-test . However, for the control group, no significant improvement was found over time .As shown in Table p = 0.444), while students in the experimental group performed much better than those in the control group in the post-test (p < 0.001) and in the delayed post-test (p < 0.001), with medium effect sizes, d = 0.62 and d = 0.68, respectively. Between-group comparisons (see Table p = 0.135). However, the experimental group showed better performance regarding coherence and cohesion than the control group in the post-test, p < 0.001, with a large effect size (d = 0.93), and in the delayed post-test, p < 0.001, with a large effect size (d = 0.83). Between-group comparisons revealed that no significant difference was found between the two groups in lexical resource scores in the pre-test (p = 0.367). However, the experimental group showed better performance regarding lexical resource scores than the control group in the post-test, p < 0.001, with a medium effect size d = 0.74), and in the delayed post-test, p < 0.001, with a large effect size, d = 0.84.Additionally, the task response scores of the two groups in the pre-test showed no significant difference p = 0.44, whileχ2 = 43.33, p < 0.001), and had no effects on that of the control group across time .Given that the scores were not normally distributed, the Friedman test examined the differences in grammatical range and accuracy scores over the three tests. Table p < 0.017 (0.05/3≈0.017). There was a statistically significant improvement in the grammatical range and accuracy scores in the experimental group in the pre-test vs. the post-test , the pre-test vs. the delayed post-test . And no significant difference was found for the experimental group in the post-test vs. the delayed post-test , meaning that the effect was retained in the delayed post-test. However, no significant difference was found in the grammatical range and accuracy scores in the control group in the pre-test vs. the post-test , the pre-test vs. the delayed post-test , and the post-test vs. the delayed post-test , indicating that the traditional product-oriented approach had little effects on students’ grammatical range and accuracy scores. Additionally, Mann-Whitney U tests were conducted to compare the differences between the two groups at the three tests. The results showed that students in the experimental group and the control group in the pre-test were quite similar concerning the grammatical range and accuracy scores , whereas statistically significant differences were found in the grammatical range and accuracy scores between the experimental group and the control group in the post-test  and the delayed post-test .Post-hoc analysis with Wilcoxon signed-rank test was conducted with a Bonferroni correction applied, resulting in a significant level set as task response, coherence and cohesion, lexical resource, and grammatical range and accuracy) for the experimental group and the control group were compared at both pre- and post-test stages.The current study aimed to investigate the effects of peer feedback on enhancing medical students’ English writing skills development over time at a university in China. Peer feedback (see Table task response and coherence and cohesion). The current study extended previous studies and suggested that peer feedback in process writing instructional framework, although not conclusively, can improve medical students’ English writing performance in terms of task response, and coherence and cohesion.The research question concerns testing the effects of peer feedback on developing students’ writing performance over time. This study has shown that students in the experimental group performed better at the post-test stage in the overall writing score and its four dimensions after processing writing instructions for 18 weeks. This was in line with the findings of other studies e.g., , 28, 44), 4428, 4“As the medical science continues to develop, people have no choice but to learn new knowledge…” and “With the development of the Internet, a wide range of amusements have filled our life”. Although these sentences had no problems concerning grammatical accuracy, it can be easily recognized that these sentences do not have direct relations with the writing topic the importance of reading ability, or the way to improve it. In contrast, in the post-tests, this phenomenon did not occur in most students in the experimental group.The improvement in task response for students in the experimental group has suggested that students had realized the importance of presenting relevant and well-supported opinions, and their ability to answer the writing requirements can be improved through the teacher’s instructions on model texts and brainstorming activities in the planning stage. Therefore, they could address the issues with relevant and persuasive statements. This can be supported by the analysis of students’ essays. For example, in the pre-tests, many students tend to write some sentences that did not have a direct relationship with the writing topic such as lexical resource, which echoes the findings of several recent studies , 28lexicMuncie’s  study. Tcoherence and cohesion after the peer feedback intervention. Compared with teacher feedback, a significant benefit of peer feedback is that it put great emphasis on providing feedback at the meaning level [coherence and cohesion. Based on the analysis of students’ written texts in the experimental group, one of the most apparent changes happened in the use of conjunctions. In the pre-tests, we found that some students might not have realized the necessity to use conjunctions to organize their sentences logically. Consequently, students wrote sentences like “I am keen on the ancient history regarding China, it offers me the opportunities to enjoy ancient cultures.” and “Practice makes perfect, you should read authentic books regularly.”. However, our analysis of students’ writing samples in the post-tests has shown that a wide range of conjunctions like “because”, “so” “since” “while” and “but” were accurately used to organize their sentences. This may be because they have understood the logical relationship  between simple sentences and were able to use specific conjunctions to present their ideas.Students in the experimental group also made great achievements in ng level , 38. Thagrammatical range and accuracy, which confirms the empirical findings that peer feedback could significantly develop the grammatical accuracy of learners’ English writing . Fu. Fugrammg , 4527, 4To conclude, this longitudinal interventional study examined the effects of peer feedback on Chinese medical students’ English writing performance over time. Compared with prior studies using the conventional product-oriented writing approach, the finding of the current study suggests that incorporating peer feedback into L2 writing teaching can effectively enhance medical students’ overall writing performance and performance in various aspects, including task response, coherence and cohesion, lexical resource, and grammatical range and accuracy.


X-ray microtomography is a nondestructive, three-dimensional inspection technique applied across a vast range of fields and disciplines, ranging from research to industrial, encompassing engineering, biology, and medical research. Phase-contrast imaging extends the domain of application of x-ray microtomography to classes of samples that exhibit weak attenuation, thus appearing with poor contrast in standard x-ray imaging. Notable examples are low-atomic-number materials, like carbon-fiber composites, soft matter, and biological soft tissues. We report on a compact and cost-effective system for x-ray phase-contrast microtomography. The system features high sensitivity to phase gradients and high resolution, requires a low-power sealed x-ray tube, a single optical element, and fits in a small footprint. It is compatible with standard x-ray detector technologies: in our experiments, we have observed that single-photon counting offered higher angular sensitivity, whereas flat panels provided a larger field of view. The system is benchmarked against known-material phantoms, and its potential for soft-tissue three-dimensional imaging is demonstrated on small-animal organs: a piglet esophagus and a rat heart. We believe that the simplicity of the setup we are proposing, combined with its robustness and sensitivity, will facilitate accessing quantitative x-ray phase-contrast microtomography as a research tool across disciplines, including tissue engineering, materials science, and nondestructive testing in general. DespiThe technology for x-ray phase-contrast imaging is more demanding in comparison to what is needed for conventional x-ray imaging, specifically in terms of the required spatial and temporal coherence of the radiation . While sIn this context, we have identified a solution for lab-XPCµT that has minimum requirements on the experimental setup. Our approach is based on a low-power, fixed target and sealed x-ray tube; it requires a single optical element (modulator), and it is compatible with flat-panel detectors as well as with single-photon counters. All of these components are readily available and require a minimal amount of maintenance. The modulator is fabricated through laser ablation of a tungsten foil and requires positioning accuracy of approximately 5–10 µm. The total length of the system is less than 1 m, making it one of the most compact for lab-XPCµT. Our approach shares similarities with a Shack–Hartmann wavefront sensor and in our work, it evolved from one-directional to two-directional sensitivity of beam tracking ,38. It c2.A.The core element of the imaging system  is the sWhen a sample is in place, the intensity distribution of each probe is attenuated and shifted by the local properties of the sample, e object . The tralgorithm . These dBy linking the two refraction images to the orthogonal gradients of the phase shift   can be o follows : (7)F[A better signal-to-noise ratio can be seen by direct visual comparison with the attenuation-image contrast of the same phantom . The retB.The x-ray system features a fixed W-target microfocus Hamamatsu x-ray source (L12161-07), which we run at 40 kVp and 10 W. The modulators were manufactured with a laser ablation procedure from readily available 100-µm-thick tungsten foils (Goodfellow). Two pitches, 50 and 100 µm, were tested. The apertures have a conical shape with diameters of Three detectors were tested: i) Pixirad-2/PIXIE-III photon counter with a 650 µm CdTe sensor (98% detection efficiency up to 50 keV), 62 µm pixel pitch and a The geometry was defined as follows: To characterize the system and fine-tune the quantitative retrieval algorithm, a test phantom consisting of a poly(methyl methacrylate) (PMMA) calibration sphere  embedded in wax and polyethylene foam. Images were acquired at n angles , nine diThe ability of the system to measure The performance of the system for soft-tissue imaging was demonstrated with two animal-derived small organs; a heart and an esophagus, extracted from a 300 g Sprague–Dawley rat and a 3 kg piglet, respectively. The heart sample was obtained from the UCL Biological Services Unit (BSU), while the esophagus from piglets was procured from JSR Genetics Ltd, a Home Office-approved supplier. Animals were euthanized via Schedule 1 methods, and sample sizing has been implemented following the NC3R principles. The organs were prepared for imaging through paraformaldehyde (PFA) fixation, followed by critical point drying (as described in previous work ). For thAll reconstructions were performed using the Feldkamp–Davis–Kress algorithm implementation in CUDA of the Astra Toolbox . The vol3.The angular sensitivity depends on the exposure time and closely follows the trend expected from the Poisson statistics, proportional to the power of −1/2 . This inTomographic reconstructions of ted ones  in Tableexpected ; howeverLab-XPCµT of the animal-derived small organs are presented in For the piglet esophagus , we idenat heart , lab-XPC4.We presented a compact and cost-effective system for performing XPCµT in a laboratory setting. The system is based on a single modulator and a low-power sealed x-ray tube and is compatible with a range of readily available detectors. We report submicrometer angular sensitivity starting at 2 s of exposure time at 50 µm spatial resolution and 32 s at 16 µm spatial resolution, and a maximum angular sensitivity of 220 nrad in 64 s at 50 µm spatial resolution. These results are compatible with the state of the art and were achieved with a simple and compact setup. The proposed approach is quantitative, as validated through a phantom composed of known materials, and flexible in terms of spatial resolution and sampling. By introducing the amplitude modulator, sampling and resolution are driven by the structure of the illumination and can be tuned to the requirement of the sample by adapting the data acquisition strategy. The potential of the system for soft-tissue imaging was demonstrated on two biological specimens prepared without staining: five different tissue layers were identified in a piglet esophagus, and the orientation of microfibers within the myocardium of a rat heart was visualized. We believe that the concurrent simplicity, sensitivity, robustness, compactness, and efficiency of our approach will be instrumental in making XPCµT more accessible and available to a wider community.


Nature Communications 10.1038/s41467-022-29506-y, published online 20 April 2022Correction to: In the original version of the manuscript, the image shown in Figure 4c, third row down for the condition “WT SCD” was incorrect. This image incorrectly showed the same sample as shown in Figure 4a, top row for the condition “WT DMSO”. This has been corrected in the PDF and HTML versions of the article. The correct version of Figure 4 is as follows:


In our prior studies, human participants were required to generate long sequences of targeted hand movement when task difficulty varied between conditions, and where full vision of the hand and target was always available. The movement amplitude—that is, the actual distance travelled—for each movement was measured; consecutive movement amplitude values were formed into time series; then, the time series were submitted to spectral analysis. As task difficulty increased, there was a pink-to-white-noise shift in movement amplitude time-series structure. Those changes could be attributed to a difficulty-induced increase in the need to engage visual feedback processes, which maintain accurate guidance of the hand to the target. The current study was designed to provide a more direct test of the hypothesis that difficulty-induced increases in visual feedback processing modulate movement amplitude time-series structure. To that end, we examined cyclical aiming performance under four unique conditions created from the crossing of two index of difficulty (2 and 5 bits) and two visual feedback  conditions. That allowed us to examine how variations in visual feedback quality might influence difficulty-induced changes in time-series structure. In the visual feedback condition, we predicted that the increase in difficulty should result in a pink-to-white-noise shift in time-series structure. If that expected shift resulted from increased engagement of visual feedback processing, then in the no-visual feedback condition—where visual feedback processing was disabled—we should observe a strengthened pink-noise time-series structure that does not change with the increase in difficulty. The current results confirmed those predictions. That provides further support for the hypothesis that engagement of closed-loop visual feedback processing modulates movement amplitude time-series structure. Traditionally, the control of manual aiming has been understood in terms of two types of information processes: Feedforward and feedback control e.g., –5. FeedfOur research has shown that when participants are required to generate long sequences of cyclical aiming movements under conditions of full vision and movement amplitude—the target-to-target travel distance—is recorded, movement amplitude time-series structure increases in complexity—shifting from pink to white noise—as task difficulty increases ,7. Fig 12(2A/W), where ID is the index of difficulty expressed in binary units (bits), A is the amplitude or distance between target centers, and W is the target width [In addition, in that research ,7 we defet width . We hypoet width –5,10–13.et width ,7. In adet width .Although it is reasonable to infer that pink-to-white-noise shifts in time-series structure reflect a transition from feedforward to feedback control, our aforementioned studies did not directly manipulate visual feedback availability. A manipulation of visual feedback would provide a more direct test of the hypothesis that shifts in time-series structure under visual feedback conditions reflect shifts in closed-loop motor control processes. For example, if increased closed-loop visual feedback processing is responsible for the ID-induced pink-to-white-noise shift in movement amplitude time series, then blocking visual feedback processing through the removal of visual feedback should result in increased dominance of feedforward control and a strengthened pink-noise time-series structure that remains constant across increases in the ID.We are aware of one study that examined manual aiming movement amplitude time-series structure under both visual feedback and no-visual feedback conditions: In Miyazaki et al. , long seIn the current experiment, participants performed in a cyclical aiming task under two ID levels—2 and 5 bits—and two visual feedback conditions—visual feedback and no-visual feedback, during which closed-loop visual feedback processing was enabled or disabled, respectively. Following our prior findings, under the visual feedback condition we predict a pink-to-white-noise shift in movement amplitude time-series structure with the increase in ID ,7,17. IfThirty healthy individuals, 19 of whom were female, and whose mean age was 23.933 years (SD = 3.523), served as participants. All participants reported that they were right-hand dominant, had no prior history of neurological disease or damage, and had normal or corrected-to-normal vision. They responded to advertisements for healthy right-hand dominant volunteers between the ages of 18 and 2(2A/W)—we created two target displays, one that had an ID value of 2 bits  and one that had an ID value of 5 bits . With the increase in ID, there was an eight-fold increase in A while W was held constant. During the experiment itself, participants performed under both levels of ID twice, once under the visual feedback condition and once under the no-visual feedback condition. Thus, each participant performed in four unique experimental conditions. Each condition required the production of 500 consecutive cyclical aiming movements. The same apparatus and customized software reported in detail by Slifkin and Eder ,21 was ucursor on the video monitor would correspond to the position of the mouse on the graphics tablet. They were told to move back and forth between the two targets, being as fast and accurate as possible—where the instruction to be fast was of equal importance to the instruction to be accurate—and that a target hit would register if the cursor “landed” anywhere within the target region at the time of a mouse-button press. For each movement, a crosshairs marker appeared at the center of the currently active target; when the mouse button was pressed to indicate the end of the action, the marker moved to the opposite target to mark it as the newly active target. Any button press occurring when the cursor was outside of the target was classified as a target miss and resulted in a “beep” sounded by the computer. The mouse cursor, targets, target marker, and error beep were active during the first 100 movements of all four experimental conditions , which translates to instantaneous acquisition rates of either 66.67 or 62.50 Hz (M ≈ 64.52 Hz), respectively. The spatial resolution of each sample was 0.1 mm. In addition, during data acquisition we recorded the x-dimension movement-trajectory position at the moment of each mouse-button press.All data presented in this report came from the x-axis distance between consecutive mouse-button press positions. We did not analyze the initial 200 movements of the time series in any experimental condition. That excluded the initial 100-movement calibration period (movements 1–100) and the first 100 movements following the transition (movements 101–200). The latter was meant to exclude any transient adjustments that may have been present following the transition. The remaining 300 movements in each condition (movements 201–500) for each participant were submitted to data analyses. The main analysis involved spectral analysis of time series of consecutive movement amplitude measurements, where each movement amplitude measurement was the While analysis of movement amplitude time-series structure via spectral analysis was the primary focus of the current study, we also present an analysis of mean movement amplitude and mean movement time; the latter was the average time from one mouse-button press to the next. Analysis of those variables allowed an assessment of whether the ID manipulation was effective, which should be evident in between-ID differences for movement amplitude and movement time. For example, the analysis of mean movement amplitude allowed a determination of the scaling of movement amplitude to the different amplitude requirements imposed under ID 2 and ID 5. Following Fitts’ law, its numerous replications, and our prior research, it was expected, at least under visual feedback conditions, that movement time would increase with the increase in ID e.g., ,13,14,21The average movement amplitude and movement time were calculated for each participant’s time series under each of the four unique conditions. Separate two-way ID (2) by visual feedback (2) ANOVAs, with repeated measures on both factors, were used to test the reliability of the change in mean movement amplitude and mean movement time with the change in ID and visual feedback conditions.pwelch function. We chose pwelch parameter values that allowed each series to be analyzed as a single segment (window) of 300 data points.  Then, the 300-point segment was multiplied by a Hamming window followed by the calculation of the power spectral density. The power spectrum from each time series was divided into 150 bins of equal width where the frequencies associated with the upper limits of the lowest and highest frequency bins were 0.0033 and 0.5000 cycles per movement (Hz), which translates to 300 and 2 movements per cycle, respectively. Each frequency bin spanned a 0.0033 Hz range. The amount of power in each frequency bin was related to the magnitude of the movement amplitude oscillations within that bin’s frequency range.Assessments of movement amplitude time-series structure were made through spectral analysis. Prior to spectral analysis, each 300-point movement amplitude time series was linearly detrended e.g., . The detβ from the power spectrum. That involved a log10 transformation of both the spectral power and frequency of each participant’s power spectrum in each condition. Then, linear regression—y = bx + a—was used to describe changes in log10 power as a function of log10 frequency (see 10 transformation) was removed from the analysis. That step was taken because the aforementioned linear time-series detrending procedure has been shown to attenuate power in the lowest frequency bin, which may bias estimates of β [see 10 transformation—to the frequency bin with an upper limit of 0.1000 Hz—or −1.0000 Hz after the log10 transformation. The reason for imposing a −1.0000 Hz upper limit is that a common feature of human performance time-series data exhibiting long-range correlations (including the current data) is a linear reduction in power across the lower frequencies until about −1.0000 Hz, after which there is a flattening, a sharp downturn, or a sharp upturn in power across the higher frequencies [e.g., see Our main measure of movement amplitude time-series structure was an estimate of b describing the relation between log10 frequency and log10 power is equivalent to the exponent β in the equation for the power function S(f) = 1/fβ. In that equation, S(f) is the spectral power at each frequency, 1/f is the inverse of frequency, and β specifies the degree of proportionality between S(f) and 1/f [e.g., see b has a value of −1, then the power function β has a value of 1. If β = 1, then the fall-off of power is inversely proportional to the increase in frequency, which is the signature of a pink-noise time-series structure  For each dependent variable—mean movement amplitude, mean movement time, and β—the Tukey HSD post-hoc test was used to compare pairs of group means, which allowed us to investigate the locus of any significant main effects or interactions. All results reported as “significant” had p values less than .05. Exact p values were reported, unless the p value was less than .0001; in that case, p < .0001 was reported. All statistical analyses were performed using Statistica v.13.3.A two-way ID (2) by visual feedback (2) ANOVA, with repeated measures on both factors, was used to examine changes in the group-mean β values with an average across the remaining participants within the relevant condition. When separate two-way ANOVAs based on the two solutions were compared, the between-ANOVA statistical outcomes  were the same for each dependent variable’s main effects and interaction. Furthermore, the two analyses applied to each dependent variable yielded essentially identical patterns of group means. Here, we report the results based on the missing data replacement procedure. Before reporting the statistical analyses of our results, we present representative time series from each condition.For three participants, data from 1 of the 4 experimental conditions was incomplete or missing. In two cases, the data files were corrupted. In the third case, one of the four conditions was run twice, which resulted in the absence of data for one condition. Two solutions were considered: The first solution entailed removing all data of the three participants from all analyses; the second solution entailed replacing the missing mean movement amplitude, mean movement time, and As with all data submitted to spectral analysis in the study, each panel in As shown in the time series and as highlighted by the polynomial regression functions, the contribution of low-frequency oscillations appears to be similar across the ID-2 no-visual feedback (top left panel), ID-5 no-visual feedback (top right panel), and ID-2 visual feedback (bottom left panel) conditions, but at the ID-5 visual feedback condition (bottom right panel) the polynomial regression function has flattened. In particular, at ID 2 a similar low-frequency-dominant time-series structure can be seen either when visual feedback was absent or present (cf. top left panel and bottom left panel). However, at ID 5 the low-frequency-dominant structure was attenuated when visual feedback was added (cf. top right panel and bottom right panel). It is at ID 5 where the control of manual aiming should require greater reliance on visual feedback processing and there should be increased time-series whitening. While the main results of this study will focus on further analyses of movement amplitude time-series structure , prior to that presentation we provide an analysis of the group-mean movement amplitude and the group-mean movement time.F = 634.297, p < .0001, ηp2 = .956. When averaged across ID levels within each visual feedback condition, the group-mean movement amplitude was 69.892 mm and 70.466 mm in the visual feedback and no-visual feedback conditions, respectively. Those values were nearly identical, resulting in the absence of a significant main effect of visual feedback, F = 0.014, p = .907, ηp2 = .0005. As seen in F = 0.552, p = .464, ηp2 = .019.As shown in p = .0002) and from ID-2 no-visual feedback to ID-5 no-visual feedback (p = .0002). Differences between group means were not significant for either the ID-2-visual-feedback vs. ID-2-no-visual-feedback comparison (p = .924) or the ID-5-visual-feedback vs. ID-5-no-visual-feedback comparison (p = .973).  The pattern of Tukey HSD results reflects both the ANOVA results and the pattern of group means depicted in According to the Tukey HSD post-hoc test, there was a significant increase in the group-mean movement amplitude both from ID-2 visual feedback to ID-5 visual feedback  = 273.108, p < .0001, ηp2 = .904. When averaged across ID levels within each visual feedback condition, movement time under visual feedback and no-visual feedback was 780.100 ms and 653.696 ms, respectively. The reduction in movement time with the removal of visual feedback resulted in a significant main effect of visual feedback, F = 11.326, p = .002, ηp2 = .281. However, the magnitude of that effect differed at the different ID levels  = 17.163, p = .0003, ηp2 = .372.As shown in vels see : At ID 2p = .0002) and from ID-2 no-visual feedback to ID-5 no-visual feedback (p = .0002). While a statistically significant difference between ID-2 visual feedback and ID-2 no-visual feedback was absent (p = 0.999), there was a significant increase of the ID-5-visual-feedback movement time over the ID-5-no-visual-feedback movement time (p = .0002). That provides further confirmation of the larger ID-induced increase in movement time under visual feedback as compared with no-visual feedback.  The pattern of Tukey HSD results reflects both the ANOVA results and the pattern of group means depicted in According to the Tukey HSD post-hoc test, there was a significant increase in the group-mean movement time both from ID-2 visual feedback to ID-5 visual feedback  of the regression line is negative at ID 2 and clearly flattens to a near-zero value at ID 5. In contrast, under no-visual feedback, the regression bs are negative and parallel for the ID-2 and ID-5 power spectra. That pattern of regression b results based on the group-mean power spectra matched the pattern of group-mean β coefficients plotted in β was reduced from 0.521 to −0.004 with the change from ID 2 to ID 5; thus, the distribution of power was in the pink-noise range at ID 2 and clearly whitened at ID 5. However, under no-visual feedback, β was essentially unchanged with the increase in ID, that is, β at ID 2 and ID 5 was 0.846 and 0.842, respectively; thus, in the absence of visual feedback, the distribution of power exhibited a strengthened pink-noise structure that persisted with the change in ID level.β declined from 0.684 at ID 2 to 0.419 at ID 5. The overall ID-induced decline in β was reflected by a significant main effect of ID, F = 9.807, p = .004, ηp2 = .253. When averaged across ID levels within each visual feedback condition, β increased from 0.259 under visual feedback to 0.844 under no-visual feedback, which yielded a significant main effect of visual feedback, F = 38.164, p < .0001, ηp2 = .568. The differential influence of the visual feedback manipulation on the ID-induced change in β—that is, the ID-2-to-ID-5 reduction in β under visual feedback and the absence of an ID-induced change in β under no-visual feedback—resulted in a significant ID by visual feedback interaction, F = 6.323, p = .018, ηp2 = .179. Under visual feedback, the ID-induced change from pink to white noise closely replicates our prior findings: In the current study , but the very small difference between ID-2 no-visual feedback and ID-5 no-visual feedback was not significant (p = 0.999). In addition, the reduction in the group-mean β from ID-2 no-visual feedback to ID-2 visual feedback was not significant (p = .143), whereas the reduction from ID-5 no-visual feedback to ID-5 visual feedback was significant (p = .0002).  The pattern of Tukey HSD results reflects both the ANOVA results and the pattern of group means depicted in The Tukey HSD post-hoc test confirmed, namely, that the reduction in the group-mean Our earlier research showed that movement amplitude time-series structure in a cyclical aiming task shifted from pink to white noise with increases in the ID ,7. In thThe current study provided a more direct test of the hypothesis that increased whitening of movement amplitude time series reflects increased engagement of closed-loop visual feedback processes ,7,17. Ifβ values for the no-visual feedback and visual feedback conditions were in the pink-noise range and did not differ statistically. That finding is consistent with the idea that at low-ID levels there is little if any need to engage closed-loop visual feedback processing, even when visual feedback is available [e.g., When viewing the results as changes across visual feedback conditions within each ID level, we also found confirmation of our predictions: Within ID 2, the group-mean le e.g., . In contle e.g., ). That sle e.g., .As might be predicted by Fitts’ law and its numerous replications e.g., ,13, we fFurthermore, the current pattern of movement time results is consonant with the findings of Wu et al. . They exAs shown in β values to the absence of visual feedback  under visual feedback. Thus, while movement time was always sensitive to variations in ID, β was only sensitive to variations in ID when visual feedback was available and there was an ID-induced need to engage closed-loop visual feedback processing. How can those findings be explained in terms of our model of movement amplitude time-series structure [β’s insensitivity to the increase in ID in the absence of visual feedback—when visual feedback processing was disabled—and 2) β’s sensitivity to the increase in ID in the presence of visual feedback—when visual feedback processing was enabled.In contrast, there was no ID-induced change in tructure ,7? Accorβ. In turn, there was a strong relation between variations of β and variations of movement time, r2 = .981 [β must reflect β’s dependence on closed-loop visual feedback processing too [β from the four conditions of the current experiment was examined, the r2 value was more than halved, r2 = .413, compared to the same relation examined in Slifkin and Eder [r2 = .981. The reduction in the current r2 value occurred because, although under visual feedback the increase in movement time was related to a decrease in β, under no-visual feedback the increase in movement time was related to no change in β  and under a set of large-scale  target displays [scale doubled. We found that with the increase in the ID, movement time increased and β changed from pink-to-white-noise values (those outcomes were observed in the visual feedback condition in the current study too [see Figs β there were no statistically significant differences between scale levels within either the 2-bit or 5-bit ID level [β [Would the current results have differed if the 2-to-5-bit increase in the ID was achieved through a reduction in target width while the movement amplitude requirement was held constant? In our prior research, under conditions of visual feedback we examined changes in movement time and displays . Within β would have been expected. That expectation follows from our previous results where we found a systematic reduction of β over the ID levels of 2, 3, 4, and 5 bits [β values. We would not expect different β values at intermediate ID levels. Thus, adding more ID levels should result in a similar pattern of results as that of the current study: Namely, we would still expect a reliable ID by visual feedback interaction like that shown in One might also wonder whether including additional levels of the ID would have changed the main findings of the current study. Our interest has been in determining whether ID-induced increases in visual feedback processing are related to changes in time-series structure ,7,17. ThWhile including conditions where an increase in the ID was achieved through a reduction in target width, including variations in target display scale, and including additional ID levels would provide more detail about the influence of the ID and visual feedback conditions on movement amplitude time-series structure, doing so would come at a cost: Reliable time-series analyses require lengthy time series, which increase both the duration of experimental sessions and the likelihood of participant fatigue . Consequβ is uniquely sensitive to closed-loop visual feedback processing. Future research with the goal of quantifying the degree of closed-loop visual feedback engagement may benefit from assessments of motor-event time-series structure via β: Variations in β might predict the degree of closed-loop visual feedback processing in a similar fashion as variations in time-series structure predict neurological disease severity [The original motivation for this study was to test the possibility that the proximal, direct cause of increased movement amplitude time-series whitening with increases in ID was the ID-induced need to engage the available visual feedback through closed-loop processing. The current findings provided support for that hypothesis. Furthermore, the results suggest that Last, the current study adds to other attempts to understand the processes and mechanisms underlying the structure of long sequences of discrete motor events e.g., ,34–36. I


Recent success stories in reinforcement learning have demonstrated that leveraging structural properties of the underlying environment is key in devising viable methods capable of solving complex tasks. We study off-policy learning in discounted reinforcement learning, where some equivalence relation in the environment exists. We introduce a new model-free algorithm, called QL-ES , which is a variant of (asynchronous) Q-learning tailored to exploit the equivalence structure in the MDP. We report a non-asymptotic PAC-type sample complexity bound for QL-ES, thereby establishing its sample efficiency. This bound also allows us to quantify the superiority of QL-ES over Q-learning analytically, which shows that the theoretical gain in some domains can be massive. We report extensive numerical experiments demonstrating that QL-ES converges significantly faster than (structure-oblivious) Q-learning empirically. They imply that the empirical performance gain obtained by exploiting the equivalence structure could be massive, even in simple domains. To the best of our knowledge, QL-ES is the first provably efficient model-free algorithm to exploit the equivalence structure in finite MDPs. Reinforcement learning (RL) aims to develop computer systems with the ability to learn how to behave optimally, or nearly so, in an unknown dynamic environment. An RL task typically involves an agent interacting with the environment, which is often modeled as a Markov decision process (MDP), and the agent’s goal is to find a policy maximizing some notion of reward. In most settings in RL, the MDP is initially unknown beyond its state and action spaces. Hence, the agent aims to learn a near-optimal policy using the experiences collected from the environment.behavior or logging policy. Perhaps the most famous off-policy learning algorithm is the celebrated Q-learning algorithm )Assumption A2.The agent knows the reward function R.In this section, we present a variant of Q-learning that exploits the equivalence structure in the environment to speed up the learning of the optimal Q-function. We call this algorithm QL-ES, which is short for ‘Q-learning with equivalence structure’.t to update potentially multiple entries of t, the agent can construct samples for all other pairs in counterfactual next-state for QL-ES follows the same machinery of QL but is also built on the idea that the knowledge on Algorithm 2 QL-ESInput: dataset T, learning rates Initialization: for do    Sample action     Find for do                 Compute +1 using end for   end forThe agent thus uses st,at,st+ to updatWhen the underlying MDP admits some equivalence structure, QL-ES performs multiple updates of Q-function at any slot, in contrast to structure-oblivious Q-learning that updates only the Q-function of the current state–action pair. Thus, we expect learning the optimal Q-function under QL-ES to be faster than Q-learning; this will be corroborated by the numerical experiments in Remark 1.The multiple updates used in QL-ES can be straightforwardly combined with many other variants of Q-learning, such as Speedy Q-learning  and UCB-f is some suitable function f satisfying the Robbins–Monro conditions, e.g., We finally remark that some works in the literature on Q-learning use learning rates of the form T time steps. A relevant notion of deviation often studied in the literature  is tfirst time step such that all state–action pairs are visited at least once with probability at least M and the behavior policy M. Further, we have Let us first recall the classic definition of cover time ariants) ,38,39. LNext, we introduce a notion of cover time for equivalence classes, which is relevant to the performance analysis of QL-ES. We believe it can be of independent interest.Definition 3.Let M be a finite MDP and The following theorem provides a non-asymptotic sample complexity for QL-ES. It concerns constant learning rates, i.e., Theorem 1.There exist some universal constants A proof of this theorem is provided in Comparison with sample complexity of Q-learning. Theorem 1 tells us that the number of steps to have rem 2 in ) revealsM and M and M, the ratio M appears to be complicated and tedious, if possible at all. Nonetheless, it is possible to construct simple problem instances, where one can derive analytical bounds on This ratio S but on C, the number of classes in We refer the reader to the results in This section is devoted to reporting numerical experiments conducted to examine the performance of QL-ES against the (structure-oblivious) Q-learning algorithm. First, we present the considered evaluation metrics and environments. Then we present numerical assessment of (i)Max-norm Q-value Error defined as (ii)Total Policy Error defined as s.We consider two evaluation metrics in the experiments:t. Equivalently, we may compute the metric (ii) viaThe metric (i), which is in line with the definition of sample complexity studied in Working with  is prefeRiverSwim and GridWorld. These are classical MDPs widely used in the RL literature. Both render suitability to demonstrate the numerical performance of QL-ES since each allows us to define a family of MDPs with progressive difficulty levels.We consider two environments: RiverSwim and variants. A generic RiverSwim MDP with L states is shown in L of states, the more exploration is required. The L-state RiverSwim , the state–action space is of size 84 , while the number of classes remains 8 in both. In S of states grows. Specifically, we consider RiverSwim, Perturbed RiverSwim , These results reveal that We now turn to reporting experimental results for QL-ES and Q-learning in RiverSwim and GridWorld. In the following figures, QL indicates the standard Q-learning algorithm (Algorithm 1). We used a constant learning rate To demonstrate the scalability of QL-ES, in We now turn to the results for GridWorld MDPs. As in RiverSwim MDPs, QL-ES significantly outperforms Q-learning in the grid-world environments. For Q-learning, the Q-value error remains considerable, even for Modified RiverSwim and Modified GridWorld defined as follows. The Modified RiverSwim is identical to 6-state RiverSwim  and QLES(6) manage to find optimal actions significantly faster than Q-learning. In addition, QLES(10) is far superior to QLES(6) because of the quite smaller Q-value error and total policy error. The results show that algorithms with a suitable number of equivalent pairs are sufficient to learn an optimal policy in most states reasonably fast.S, the size of the state space. Through extensive numerical experiments in standard domains, we demonstrated that QL-ES significantly improves over (structure-oblivious) Q-learning. These results revealed that exploiting state–action equivalence favors faster convergence of the Q-function and policy learning in large MDPs. A limitation of our approach is the need for the prior knowledge on the structure. To the best of our knowledge, existing methods for learning the equivalence structure are all model-based. Hence, an interesting question is whether it is possible to exploit the equivalence structure without prior knowledge on the structure using only model-free algorithms. Devising such model-free algorithms (or otherwise establishing an impossibility result) is an interesting yet challenging topic for future work. Another interesting direction for future work is to investigate ways to combine the knowledge of the equivalence structure with function approximation methods. Finally, another avenue for future work is to study model-free algorithms for the regret minimization setting in average-reward MDPs ..S, the s