Systematic review of learning curves in robot-assisted surgery

1Learning curves describe the rate of progress in gaining experience or new skills and are widely reported in surgery. Surgeons typically exhibit improvements in performance over time, often followed by a plateau where minimal/limited additional improvement is observed1. Generally, surgical learning curves are measured as a change in an operative variable (which can be considered a surrogate for surgeon performance) over a series of procedures. Studies investigating learning curves for surgical procedures are becoming increasingly important, as learning curves can have substantial impact on surgical metrics, clinical outcomes and cost–benefit decisions.

2There has been particular interest in learning curves in robot‐assisted surgery, especially in gynaecology and urology2, 3. Despite the reported operative benefits and improved hospital experience provided by robot‐assisted surgery compared with traditional minimally invasive approaches4, 5, uptake of robotic technology has been slow, largely due to high capital and maintenance costs, and uncertainty regarding the potential benefits of robot‐assisted approaches over conventional laparoscopic approaches. For example, robot‐assisted approaches have been associated with longer operating times for many procedure types6. A large proportion of these comparative studies, however, may have been generated from surgeons who were still learning the robotic technology in question4, potentially underestimating the full benefits of robotic assistance. Robot‐assisted approaches have the potential to expedite surgeon learning, but methods used to measure and define learning curves seem inconsistent1. Studies evaluating the learning curve for surgical procedures often aim to determine the number of sequential procedures that comprise the learning curve, or that are required to ‘overcome’ the learning curve (sometimes referred to as the learning curve length). To achieve this aim, studies often define a particular threshold in surgeon performance. A common threshold includes reaching a plateau in performance, yet the performance thresholds used are highly inconsistent1. The way learning curves are described can lead to misinterpretation. Terms such as ‘overcome’, for instance, could be considered misnomers, implying surgeons have mastered a procedure, for which certain performance thresholds may not provide sufficient evidence. For example, a plateau in performance does not necessarily equate to high‐quality performance; it only implies that a surgeon is no longer improving1.

3There remains a need to understand better the learning curve of robot‐assisted surgery and broadly characterize how learning curves are defined and reported. This systematic review was performed to characterize the current evidence base and appraise the methods used to define and measure learning curves for surgeons performing robot‐assisted surgery, taking a holistic, panspecialty view.

4This systematic review was conducted in accordance with a prespecified protocol and the PRISMA guidelines7.

5MEDLINE, MEDLINE In‐Process, Embase, the Cochrane Database of Systematic Reviews, the Cochrane Central Register of Controlled Trials, and the NHS Economic Evaluation Database were searched. Surgical training has evolved alongside the rapid development of robotic technologies. As such, database searches were limited to the period from 1 January 2012 to 5 February 2018, in order to capture studies investigating learning curves in the context of training relevant to current practice. The search terms used are provided in Tables S1 and S2 (supporting information). The two most recent abstract books of relevant surgical congresses were also searched from 1 January 2016 to 14 February 2018. This review considered only primary research, and excluded review articles. Supplementary hand searches of the bibliographies of relevant systematic reviews were conducted to identify any primary studies not identified elsewhere.

6The review process was performed by two independent reviewers, who assessed the titles and abstracts of all search results (stage 1), as well as the full texts of all potentially eligible studies identified in the first stage (stage 2). In the event of discrepancies, the two reviewers came to a consensus for each decision. In the absence of a consensus, a third independent reviewer resolved any disagreements.

7Eligible publications included any randomized or non‐randomized, comparative or observational studies involving wet‐ or dry‐lab testing, simulations, patients or registry/economic analyses that performed a learning curve analysis (consisting of a graph and/or reported data for at least 4 time points) of surgeons performing robot‐assisted surgery in any specialty. Studies were required to report learning curve results from more than one surgeon alone or as part of a surgical team, of any specialization (robot‐assisted, laparoscopic or open). In the absence of reporting the number of surgeons involved, included studies were required to have multiple authors. Only studies that included at least 20 surgical procedures in the analysis of the learning curve were considered. At least one of the following metrics had to be reported: time to plateau/number of ‘phases’ in the learning curve; statistical differences in metrics assessed over time; or learning percentages. Detailed eligibility criteria are shown in Table S3 (supporting information). The studies reported in this review are restricted to learning curve analyses of procedures on patients.

8For each eligible study, data were extracted into a prespecified grid by one reviewer, with verification by a second, independent reviewer. Where there was a discrepancy, the two reviewers attempted to come to a consensus; a third reviewer resolved any disagreements in the absence of a consensus.

9Captured data included study design, methodology, surgeon experience, robotic technology used and the metric measured to evaluate the learning curve. Information relating to the learning curve itself was captured, including the number of phases of the curve, the number of operations per phase and the number of procedures to overcome the learning curve (denoted in this review as the point where the chosen performance threshold was considered to have been overcome). Where reported, the specific performance threshold used was captured. If the learning curve had not been overcome within the study period, the number of procedures to overcome the learning curve was reported to be greater than the total number included in the study period.

10The quality of each eligible study was assessed using either the UK National Institute for Health and Care Excellence (NICE) RCT checklist8 or a modified version of the Downs and Black checklist for non‐randomized studies9.

11A total of 2316 records from electronic database searches, conference abstract searches and hand searches were identified. Of these, 281 full‐text articles were assessed for eligibility, of which 213 were excluded (Table S4, supporting information). The remaining 68 records (reporting on 68 unique studies) were found to meet the eligibility criteria, 49 of which reported on patient data and are presented here (Fig. 1).

12Characteristics of the 49 eligible studies presenting learning curves derived from patient procedures are presented in Table 1 10–58. All were observational in design. Data were analysed retrospectively in 40 of 49 studies (82 per cent), and the remaining nine studies (18 per cent) were prospective in design. The majority (35 of 49, 71 per cent) were single‐arm studies, and the remainder (14 of 49, 29 per cent) were comparative. Among the 41 studies that explicitly defined the number of robotic surgeons per study, the number of participating robotic surgeons was generally small. Most studies (33 of 41, 80 per cent) included fewer than five robotic surgeons, over half of which (18 of 33, 55 per cent) included fewer than three (Fig. 2). The captured studies spanned ten surgical specialties (Fig. 3). Learning curves were reported most frequently for urology, general surgery and gynaecology.

13Time‐based metrics were the most commonly reported variables used to assess the learning curve, reported by 42 of the 49 studies (86 per cent). Other measures, including length of hospital stay, morbidity and mortality rates, and procedure‐specific metrics, were reported less commonly (Fig. 4). Of the categories of metrics captured, duration of surgery, length of stay (LOS) and complication rate were reported most frequently within each category. The number of procedures required to overcome the learning curve for these metrics is shown in Table 2.

14Across 33 studies that investigated the learning curve based on duration of surgery, 27 reported whether the learning curve had been overcome. In 21 of these 27 studies (78 per cent), at least one of the included surgeons was reported to have overcome the learning curve, with the remaining six studies (22 per cent) stating that the learning curve had not been overcome by any surgeon within the number of procedures in the study period (Table 2).

15Among studies where the learning curve was not overcome by any surgeon, the number of procedures assessed over the study period varied significantly, with a range of 6–404 patients12, 26, 34, 37, 51, 58. Of the studies in which the learning curve had been overcome, 12–140 patients were needed for urological procedures16, 18, 30, 50, 56, 0–80 for general surgical procedures (where 0 suggests no apparent learning process required, if the surgeon was already experienced)15, 20, 39, 40, 42, 47, 57, 0–60 for gynaecological procedures31, 43, 0–74 for colorectal procedures27, 29, 38, 15–20 for thoracic procedures28, 35 and 7–25 for paediatric procedures14, 21.

16Among the 27 studies that reported whether the learning curve had been overcome, learning curve analyses were conducted for 22 unique procedures, of which only five (23 per cent) were supported by multiple studies. In such instances, the number of patients required to overcome the learning curve for these varied substantially between studies. For example, three studies evaluated the learning curve for duration of surgery for robot‐assisted distal pancreatectomy. In one study47 the learning curve was overcome after 40 patients, whereas in the other two12, 58 it had not been overcome within the study period (of 11 and 83 patients).

17Of five studies reporting LOS, two reported on whether the learning curve had been overcome by at least one robotic surgeon (Table 2). One study38 estimated that the number of patients required to overcome the learning curve was between 0 and 15. In the other study35, the learning curve was not overcome within a study period of 185 patients.

18Of nine studies assessing complications, eight reported whether the learning curve was overcome for complication rate. Of these, five26, 31, 34, 52, 58 found that the learning curve for complications had not been overcome for at least one robotic surgeon within the study period. With the exception of one study31, which included a surgeon with a study period of only 24 patients, these studies generally involved large patient numbers (132–404), and spanned urology, general surgery, gynaecology and cardiovascular specialties (Table 2). Only four studies reported that the learning curve for complications had been overcome. The numbers of procedures were estimated as: 0–84 for robot‐assisted sacrocolpopexy31, 36, 12–14 for robot‐assisted hysterectomy55 and 0–15 for robot‐assisted total mesorectal excision38.

19Of the 49 studies, eight (16 per cent) evaluated whether the learning curve for clinical metrics had been overcome (Table 3). Metrics included oncology‐specific metrics such as surgical margin status and recurrence rate, and urology‐specific metrics such as urinary continence. The number of procedures to overcome the learning curve varied substantially. Of the two studies assessing urinary continence after robot‐assisted radical prostatectomy, one25 reported that 100 procedures were required to overcome the learning curve, whereas in the other23 the learning curve was not overcome by any of the four robotic surgeons, with study periods ranging from 112 to 541 patients. For all other clinical metrics, the learning curve was overcome by at least one surgeon during the study period, with a wide range of 0–300 patients to achieve this target.

20Some studies in the review evaluated the learning curve using more than one metric. The number of patients to overcome the learning curve was sometimes inconsistent between metrics. For example, of the two studies35, 38 that reported whether or not the learning curve was overcome for both duration of surgery and LOS, one35 indicated that substantially greater procedural experience was required for LOS, with more than 170 additional patients required to overcome the learning curve based on this metric.

21The overall standard of reporting and level of detail provided in the included studies was low, often lacking sufficient information to interpret the learning curve for robot‐assisted procedures. For example, although 34 of the 49 studies (69 per cent) made some acknowledgement relating to the previous experience of included surgeons (Table 4), the detail in the reporting was mixed, with only 17 of 49 studies (35 per cent) quantifying previous experience of robot‐assisted (8 of 49, 16 per cent), laparoscopic (13 of 49, 27 per cent) or open (6 of 49, 12 per cent) operations completed. Only four of 49 studies (8 per cent) indicated whether simulation, dry lab or cadaver training had been completed before patient enrolment.

22Variability was observed in the performance thresholds used to measure the learning curve. For example, among the 27 studies that defined whether the learning curve for duration of surgery had been overcome (Table 2), the most common performance threshold used was the number of procedures needed to reach a plateau in performance, but other thresholds included the number of procedures to reach a change in phase, or the number to achieve a predetermined skill threshold set by an expert surgeon, and some studies did not specify the performance thresholds used (Fig. 5 a). In nine of the 27 studies (33 per cent), no statistical or quantitative assessment of the learning curve was reported beyond visual fit or a qualitative description (data not reported). Similar variation in learning curve definitions was observed for analyses of LOS and complication rates (Fig. 5 b,c).

23Several studies used these methods to define whether surgeons had achieved a high level of performance, characterized by terms such as proficiency or competency. However, these terms were used inconsistently. In 13 of 14 studies (93 per cent) that employed performance terms, at least one term was used to describe the point where the learning curve had been overcome; ‘proficiency’ was used for this purpose in ten studies22, 27, 28, 31, 36, 38, 39, 42, 50, 55, ‘competency’ in five studies10, 27, 33, 36, 49 and ‘expertise’ in one study50. In the four studies that reported more than one performance term, the terms were either used interchangeably36, 50 or assigned divergent definitions33, 49.

24Although quality assessment of included studies (Table S5, supporting information) revealed a relatively low risk of bias for several quality assessment items, risk of bias was unclear for a large proportion of the questions, suggesting poor reporting of methodology. In particular, the risk of bias with respect to the blinding of subjects, external validity of included populations and study centres, and statistical power was either high or could not be determined (at least 45 of the 49 studies, 92 per cent).

25This review identified substantial variation in the lengths of learning curves, included metrics and methods employed to assess the learning curve, as well as the reporting of the analyses and terminology used across ten surgical specialties. Reported learning curve estimates are therefore subject to substantial uncertainty, and the generalizability of these findings is limited.

26The results of the 49 eligible studies suggested that surgeon learning curves were complex. They varied significantly between studies, procedures and specific metrics assessed. A variety of factors could account for much of the variation in reported learning curve length.

27The surgeon's previous experience may have been a significant factor; in three of five studies comparing the operating time learning curves of robotic surgeons, those with greater experience required fewer procedures to overcome their learning curve16, 29, 38. Although the captured studies often compared surgeons with different experience levels, such as trainees versus those who had completed training or robotic versus laparoscopic surgeons, studies generally did not report the participants' specific grade or training experience.

28Robotic training programmes are becoming increasingly common to enable surgeons to overcome the learning curve faster59, 60. In addition to previous experience, participation in specific training programmes may influence the learning process. Although based on a small sample size, Guend and colleagues27 reported that a lower procedure volume was required to overcome the learning curve for robot‐assisted colorectal resections for three surgeons who had participated in an institutional training programme (25–30 procedures each), compared with the volume required for an earlier surgeon who joined the institution before the programme was established (74 procedures). Few studies, however, provided details of their training programmes, where these existed. Recent training programmes have considered innovations such as feedback loops that aim to provide specific recommendations for improvement, shortening the time required to achieve adequate performance61.

29Differences in procedural complexity may also have contributed to variation in the learning curves observed. In surgical practice, following initial improvement and subsequent stabilization of performance, a decline in performance is often observed62. This decline is thought to reflect the point at which, following mastery of simpler procedures, surgeons take on more challenging, technically complex procedures, that impact on a learning curve63.

30Some procedures are inherently more complex and challenging than others. Studies64, 65, 66 of simulated robotic training tasks have observed learning curves of different duration for tasks of varying complexity. Learning curves are likely to be influenced by numerous observed and unobserved confounders. To account better for such differences, and to permit comparisons between studies, enhanced reporting of surgeon baseline characteristics, experience and procedure complexity is required.

31This review has highlighted a number of limitations associated with reporting learning curves for robot‐assisted surgery. Many studies failed to describe the characteristics of the surgeons, patients or methods of assessment in sufficient detail to make valid comparisons between studies or to enable a study to be reproduced.

32Although the majority of captured studies were determined to be of reasonable methodological quality, the studies were observational and usually included few surgeons. These study designs are associated with significant drawbacks, particularly with respect to confounding and selection bias67, 68, although this is expected given that they are more suited to measuring learning curves. Regarding sources of bias in the included studies, the risk of bias was unclear for a large proportion of the quality assessment items, particularly in relation to blinding, external validity and statistical power, suggesting poor reporting of methodology.

33There was little consistency in the performance thresholds used to measure the learning curve, making between‐study comparisons challenging. A large proportion of studies measured the number of procedures required to reach a plateau in surgeon performance. Although a variety of methods can define quantitatively the point at which a plateau is reached69, 70, 71, 72, there is currently no widely accepted and validated method, and some studies used visual fit alone1. The number of procedures required to overcome learning curves reported are subject to considerable uncertainty. Several studies measured the number of procedures to achieve a threshold set by experts. These were sometimes based on the performance of expert robotic surgeons50, whereas others38, 56 included expert laparoscopic surgeons. Many studies did not report the specific performance thresholds used to measure the learning curve, precluding any ability to make comparisons.

34These methods were frequently used to define the points at which surgeons reached ‘proficiency’, ‘competency’ or other related terms. These terms, however, were used inconsistently or interchangeably36, 50, or used with distinct definitions33, 49. In one study49 competency was used to describe performance that reached a steady state or plateau, whereas proficiency described further improvement after plateau and mastery as the achievement of outcomes better than the set target value. These definitions are not well aligned with guidelines for assessing surgical competence73, 74, recommended by the US Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties, nor the criteria developed for procedure‐based assessment in the Intercollegiate Surgical Curriculum Programme in the UK.

35The mismatch between the performance thresholds used to measure the learning curve and the terminology used to describe the results of the analyses can lead to misinterpretation. The term ‘overcome’ was commonly used to describe a point when surgeons reached a given performance threshold, implying a high level of performance. These thresholds, particularly time to plateau, are often simplistic and may not capture sufficient evidence about the learning process to support this implication. A plateau in performance does not always equate with high‐quality performance, as surgeons will not necessarily plateau at the same level1. Likewise, using thresholds of performance to define terms such as competence and proficiency could be considered inappropriate, as these terms also imply a specific level of performance. A recent study61 investigated proficiency‐based progression training programmes in which residents who failed to show progressive improvement (reached a plateau in performance) were not considered proficient unless they had achieved predetermined proficiency benchmarks set by experienced surgeons.

36The lack of consistency in methods used to describe surgical performance and the use of simplistic and inappropriate methods adds to the complexity of interpreting learning curves. Using thresholds that provide meaningful measures of surgeon performance alongside standardized terminology seems vital to realize the full potential of learning curve analyses for optimization of surgical training programmes.

37Time‐based variables were the most common metrics used to assess learning curves, as is the case for other systematic reviews assessing surgical learning curves in other contexts63, 75. Although common across learning curve analyses, the present review suggests that variation can exist between the learning curve profiles of different metrics for a given procedure, with recovery and safety metrics (LOS, complications) exhibiting substantially longer learning curves than those for operating time, often with continued improvement for extended periods of time after the learning curve for operating time has been overcome. Given that improvements in clinical outcomes may be important drivers for the uptake of robot‐assisted approaches, the value of comparisons based on learning curves for operating time alone is unclear. In addition, the metrics captured may not directly measure surgical performance, with surrogate markers, such as operating time and LOS, reported frequently. Real‐time automated performance metrics, coupled with machine learning algorithms to process automatically collected data, may enable more direct measurement of surgeon performance76.

38Several data gaps were identified in the reported data. Many of the identified studies enrolled fully trained surgeons, investigating the transferability of skills for conversion from laparoscopic or open surgery to robot‐assisted procedures. These studies may be of limited value for informing the design of surgical training programmes. The limited data investigating the learning curves of trainee surgeons may result in missed opportunities for the optimization of programmes to accelerate the training of surgeons who are novices with robot‐assisted devices. No study reported data related to the economic impact of the learning curve, such as training costs or financial impacts of suboptimal outcomes.

39This systematic review was a broad, exploratory search of the literature reporting on the surgeon learning curve for robot‐assisted surgery, with broad search terms and eligibility criteria. Incomplete and variable reporting created challenges for data synthesis, especially given some of the exploratory and subjective outcomes this review intended to identify. The exploratory nature of the review may have introduced a number of limitations, which may have resulted in relevant data being overlooked. For example, to include evidence of suitable quality, studies that involved fewer than 20 surgical procedures in total (across surgical approaches or surgeons) were excluded, regardless of the number of robot‐assisted procedures completed by any one surgeon or included within the learning curve analysis. Studies were captured only if they reported actual learning curve data (as a graph or table presenting at least 4 time points), so that studies reporting potentially relevant data (for example the economic impact of the learning curve) could have been excluded if they did not meet these criteria. Studies that included just a single surgeon were also excluded, as the innate differences in technical ability that may exist between surgeons were anticipated to limit the reliability of the data reported in these studies. Only the learning curves of robotic procedures were considered in this review, and although the review did not set out to compare robotic‐assisted procedures with other surgical approaches, this prevented any conclusions to be drawn regarding lengths of the learning curve for robot‐assisted versus non‐robotic procedures. Only studies originally published in English were included, and database searches were limited to 2012–2018 in order to capture evidence most relevant to present‐day training practices. Although comparisons between robot‐assisted and other surgical approaches are warranted, studies with appropriate evaluation methods, standardized terminology and necessary context are essential for robust comparisons to be made. These kinds of study should provide better estimates of learning curves for robot‐assisted procedures, enhance surgical training programmes and improve patient outcomes.