Reliability of HIV rapid diagnostic tests for self-testing compared with testing by health-care workers: a systematic review and meta-analysis

1Interest in HIV self-testing—an approach to increase access to HIV testing—has increased since 2014. As a discreet and convenient approach, HIV self-testing might be most useful in reaching people who are reluctant or unable to access existing HIV testing services because of concerns about privacy, stigma, discrimination, and, in some contexts, criminalisation. According to various studies,,,,,, HIV self-testing is highly acceptable among many different population groups, including those with low testing coverage and who report barriers to and low uptake of existing HIV testing services. Despite this, some policy makers and users have raised concerns that self-testers might not be able to do the test or interpret the test results correctly.

2We did a systematic review to assess the reliability and performance of HIV rapid diagnostic tests when used by self-testers, compared with health-care workers. Although previous reviews assessed the accuracy of rapid diagnostic tests for self-testing,, they primarily focused on sensitivity and specificity and did not consider the validity of the reference standard. Thus, we systematically measure and report test concordance between self-testers and health-care workers to account for imperfect reference standards to establish the reliability and performance of rapid diagnostic tests used for self-testing.

3Evidence before this studyTo diagnose HIV, at least two or three tests, depending on the HIV prevalence among the population being tested, are needed. The validity of using a single test as a reference standard is imperfect. Previous systematic reviews focused on sensitivity and specificity of HIV rapid diagnostic tests for self-testing. An initial search of PubMed, for studies published from Jan 1, 1995, to Jan 26, 2016, with the search terms “HIV self-testing” and “review”, indicated that the validity of the reference standard has not been considered previously.Added value of this studyTo inform a WHO recommendation, we assessed the reliability and performance of HIV rapid diagnostic tests used by self-testers compared with health-care workers, by calculating statistics on test concordance to account for the imperfect reference standard. We included studies that used products designed for self-testing in diverse country settings. Previous reviews were done when HIV self-testing was emerging; these reviews primarily drew from US and European studies that used professional-use products or prototypes that have since been adapted for HIV self-testing.Implications of all the available evidenceSelf-testers could reliably and accurately do an HIV rapid diagnostic test, whether assistance was provided or not, when compared with a trained health-care worker. Errors in the test procedure might be reduced by refinement of the design of rapid diagnostic tests for self-testing, improvement of manufacturer labelling and instructions for use, and provision of additional support with instructional videos. Modifications should always be the responsibility of the manufacturer.

4Evidence before this study

5To diagnose HIV, at least two or three tests, depending on the HIV prevalence among the population being tested, are needed. The validity of using a single test as a reference standard is imperfect. Previous systematic reviews focused on sensitivity and specificity of HIV rapid diagnostic tests for self-testing. An initial search of PubMed, for studies published from Jan 1, 1995, to Jan 26, 2016, with the search terms “HIV self-testing” and “review”, indicated that the validity of the reference standard has not been considered previously.

6Added value of this study

7To inform a WHO recommendation, we assessed the reliability and performance of HIV rapid diagnostic tests used by self-testers compared with health-care workers, by calculating statistics on test concordance to account for the imperfect reference standard. We included studies that used products designed for self-testing in diverse country settings. Previous reviews were done when HIV self-testing was emerging; these reviews primarily drew from US and European studies that used professional-use products or prototypes that have since been adapted for HIV self-testing.

8Implications of all the available evidence

9Self-testers could reliably and accurately do an HIV rapid diagnostic test, whether assistance was provided or not, when compared with a trained health-care worker. Errors in the test procedure might be reduced by refinement of the design of rapid diagnostic tests for self-testing, improvement of manufacturer labelling and instructions for use, and provision of additional support with instructional videos. Modifications should always be the responsibility of the manufacturer.

10This systematic review and meta-analysis followed the PRISMA standards (appendix pp 1–2). We searched PubMed, PopLine, and Embase for studies published from Jan 1, 1995, to April 30, 2016. We also reviewed six electronic HIV/AIDS conference databases (ie, Conference on Retroviruses and Opportunistic Infections, International AIDS Conference, International AIDS Society, American Public Health Association, National HIV Prevention Conference, and the HIV Diagnostics Conference) for all available years (appendix p 3).

11We searched for grey literature through Google Scholar (first 100 titles of 201 results). We screened bibliographies of included articles and purposely selected and contacted experts (ie, academic researchers with ongoing studies on HIV self-testing) to identify additional sources. We contacted authors of relevant studies (up to two attempts) to retrieve relevant study information. We placed no language, age, study type, or geographical limitations on the search.

12We included studies reporting the performance of rapid diagnostic tests by self-testers and those reporting the concordance or the sensitivity and specificity of rapid diagnostic tests compared with the results of testing done by a health-care worker. Two reviewers (CF, CJ) screened records independently and resolved disagreements through discussion and consensus.

13We defined HIV self-testing as a process where an individual collects his or her specimen, does a test, and interprets their own test result. In the directly assisted approach, self-testers received an in-person demonstration of how to do the test or to interpret the test result; in the unassisted approach, self-testers were provided only with manufacturers' instructions for use included in the kit. All self-testers, irrespective of type of approach used, could access or receive assistance over the phone, through the internet, or with additional instructions (eg, videos, animations, or diagrams). We did not consider HIV counselling, linkage to care, and referral information as part of HIV self-testing assistance. We considered observed studies when participants were directly observed or video recorded to evaluate their HIV self-testing performance.

14We excluded studies reporting on home specimen collection, concordance or sensitivity and specificity of self-testing, or self-monitoring devices for conditions other than HIV.

15We defined the testing strategy used to establish the reference result as any testing sequence used to identify HIV infection (appendix pp 4–7). We classified testing strategies as aligned or not aligned with WHO guidance on the basis of the 2015 Consolidated guidelines on HIV testing services.

16We defined measures of concordance (inter-reader reliability) as the percentage agreement and Cohen's κ between the health-care worker and the self-tester.

17We defined measurements of accuracy as specificity and sensitivity. HIV positivity among participants was based on the number of HIV-positive participants with known status or who received an HIV-positive diagnosis during the study. HIV positivity was then categorised as high (≥5%) or low (<5%).

18Given the imperfect or absent reference standards among studies, to evaluate performance of HIV rapid diagnostic tests used by self-testers we first assessed whether the result of the index and the reference test agreed or disagreed. We then calculated the raw estimates of sensitivity and specificity.

19We extracted data for true reactive, true non-reactive, false reactive, and false non-reactive results to calculate κ and raw estimates of sensitivity and specificity, and explored the effect for oral fluid and blood-based rapid diagnostic tests separately, by type of assistance (direct assistance or no assistance) and type of observation. We calculated raw estimates of sensitivity and specificity with Meta-DiSc software; we did not consider invalid or indeterminate values to avoid skewness of results.

20We determined quality of studies using the Standards for Reporting Studies of Diagnostic Accuracy (STARD) checklist, and the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). We considered high risk of partial verification bias if more than 10% of study participants did not have their HIV test results and status confirmed, and if the selection of patients to receive the reference standard was not randomised. We considered a study to have a high risk of differential verification bias if more than 10% of patients received testing with a different reference standard. CF scored studies for quality in terms of risk of bias and concerns regarding applicability.

21Given the high study variability and the inclusion of multiple reference standards, we pooled κ estimates using a random-effects model with the R package metaphor, version 3.4.4. We assessed heterogeneity by visual inspection of forest plots and calculation of the I2 statistic (>25–50% moderate).

22The funder of the study had no role in study design, data collection, data analysis, data interpretation, writing of the report, or the decision to submit for publication. The corresponding author had full access to all study data and final responsibility for the decision to submit for publication.

23After screening and removing duplicates, we included 25 studies,,,,,,,,,,,,,,,,,,,,,,,, in the review (figure 1). All studies,,,,,,,,,,,,,,,,,,,,,,,, evaluated concordance, 15 studies,,,,,,,,,,,,,, evaluated sensitivity and specificity, and one only evaluated sensitivity. 15 studies,,,,,,,,,,,,,, used oral fluid-based rapid diagnostic tests, six,,,,, used blood-based rapid diagnostic tests, and four,,, used both.

2413 studies,,,,,,,,,,,, reported on unassisted HIV self-testing, 11,,,,,,,,,, reported on directly assisted HIV self-testing, and one reported on both approaches (table 1). 23 of 25 studies were observational in design (three cohort,,, 18 cross-sectional,,,,,,,,,,,,,,,,,, and two cross-sectional and qualitative, ), and two were randomised controlled trials., Sample size varied from 22 to 5662 participants. HIV positivity among participants was available in 28 reports from 22 studies; 19 (68%) of 28 reports had a high HIV positivity,,,,,,,,,,,,,,, and eight (29%) had low HIV positivity.,,,,,,, Reference test strategy was not available in five of 25 studies, or was not aligned with WHO testing guidance in another five studies (appendix pp 4–7). 16 (64%) of 25 studies were considered to be at low risk of bias and applicability across all key domains for QUADAS-2 (appendix p 8). 17 (68%) of 25 studies also failed to fulfil at least 60% of the STARD criteria, with a mean of 16·2 available items out of 34 (appendix pp 9–34).

25Of the 25 studies evaluating concordance between the result of an HIV rapid diagnostic test used by a self-tester compared with a result obtained by a health-care worker, 18,,,,,,,,,,,,,,,,, reported raw percentage of agreement, three,, reported a κ statistic, and four,,, reported both (table 2).

26Reported κ ranged from fair (κ 0·277, p<0·001) to almost perfect (κ 0·99).,,,,,, The raw proportion of agreement was high, ranging from 85·4% to 100%. Overall, our estimates of pooled agreement across studies were almost perfect for both types of approaches (directly assisted κ 0·98, 95% CI 0·96 to 0·99; unassisted κ 0·97, 0·96 to 0·98; I2=34·5%, 0 to 97·8; figure 2). Pooled estimates according to whether HIV self-testing was observed or not also had almost perfect agreement (observed 0·98, 0·96 to 0·99; unobserved 0·96, 0·94 to 0·99; I2=43·0%, 38·8 to 98·4; figure 2). The lowest estimated agreement (κ 0·47, −0·04 to 0·97) was in rural Zimbabwe; the study investigators attributed poor performance to low literacy in the population tested, and verbose instructions that needed further optimisation.

27The proportion of disagreements, assuming the self-tester perspective compared with health-care worker, ranged from 0% to 14·6%. Across 29 reports from 25 studies, four reports,,, found no difference in interpretation between self-testers and health-care workers. Most reported differences resulted from the interpretation of a reactive result as non-reactive (0·01–4·8%, 13 of 29 reports), a reactive result as invalid (2·7–6·7%, five of 29), a non-reactive result as reactive (0·1–4·5%, 14 of 29), an invalid result as reactive (0·5–3·1%, three of 29), an invalid result as non-reactive (0·3–12·5%, 13 of 29), or a non-reactive result as invalid (0·5–50%, seven of 29).

28Reasons for disagreements were higher in directly assisted studies (2·7–6·7%) than unassisted studies (4·6%) when interpreting a reactive result as invalid, and were higher in unassisted studies (0·01–4·8%) than directly assisted studies (0·06–2·7%) when interpreting a reactive result as non-reactive.

29Across 20 reports from 16 studies,,,,,,,,,,,,,,,, 16 (80%) of 20 reports had specificity of more than 98%. Sensitivity varied substantially; 18 (90%) of 20 reports had sensitivity of at least 80%. Two studies reported sensitivity of less than 80%: one had insufficient information on how to interpret faint positive lines, and the other suggested lengthy instructions were a barrier to participants in the rural arm, in which literacy levels were lower than the urban arm. Excluding these two studies,, sensitivity estimates were higher for blood-based rapid diagnostic tests (96·2–100%),,, than oral fluid-based rapid diagnostic tests (80–100%),,,,,,,,,,, as were specificity estimates (blood-based 99·5–100% vs oral fluid 95·1–100%). Studies,,,,,,,,,,,, in which testing was observed reported a modest difference in sensitivity (80–100%) compared with unobserved studies,,,, (88·9–100%; table 3).

30A study from Zimbabwe with oral fluid-based rapid diagnostic tests, with data disaggregated by setting, found that urban populations had higher sensitivity (80%, 95% CI 28·4–99·5) than rural populations with lower literacy (66·7%, 9·4–99·2), and that this was also the case for specificity (urban 97·8%, 88·5–99·9 vs rural 94·7%, 85·4–98·9).

31All studies,,,,,, addressing directly assisted HIV self-testing used oral fluid-based rapid diagnostic tests. The estimated sensitivity was similar to that in studies,,,,, with oral fluid rapid diagnostic tests within the unassisted approach (table 3).

32Three studies included some participants taking antiretroviral drugs. In two studies,, participants had non-reactive test results, but later received confirmatory testing and were diagnosed or disclosed their HIV statuses afterward. In the third study, self-testers and health-care workers both obtained non-reactive results because they used the same oral test.

33We identified 25 reports from 20 studies with information on invalid results: seven reports,,,,,, used the directly assisted approach, six,,,,, used the unassisted approach and two, used both approaches. Invalid results ranged from one (0·2%) of 478 tests to 197 (56·3%) of 350 tests (table 2).,,,,,,,,,,,,,,,,,,

34Although most participants were able to obtain a correct result, user errors among self-testers were noted in 15 reports. Of these reports, two found a high proportion of user error: one reported most users were unable to take blood samples or transfer blood specimens correctly (197 invalid results from 350 tests; 56·3%), and the other reported users were aware of making mistakes (36 invalid results from 239 tests; 15·1%). Excluding these studies, the proportion of invalid results was similar in studies,,,,,,,, of the directly assisted approach (0·3–9·5%) and studies,,,,, of the unassisted approach (0·2–7·9%). The proportion of invalid results was higher in studies,,,,,,,,,,,,, in which testing was observed (0·2–9·5%) when compared with unobserved studies,,,, (0·4–7·9%).

35The proportion of studies reporting invalid results among self-testers was greater in studies,,,,,,,, using blood-based rapid diagnostic tests (0·4–9·5%) than studies,,,,,,,,,, using oral fluid-based rapid diagnostic tests (0·2–4·5%). Excluding studies,,,,, with feasibility of less than 60%, the proportion of invalid results was less than 5% (0·2–4·6%), regardless of the approach or specimen.

36User errors described in studies of the directly assisted approach were incorrect or incomplete specimen collection (finger prick or oral swab),,,,, incorrect use or spillage of buffer,,,,,, incorrect transfer of blood specimen, and problems with the interpretation of results.,,,,,, Reported errors in studies of the unassisted approach included specimen collection (finger prick or oral swab),,, misinterpretation of test results,, incorrect time to read the results,, test kit opened incorrectly,, incorrect use or spillage of buffer, instructions not followed or read, or incorrect transfer of the blood specimen.

37In general, reported errors in performance were similar by type of specimen; however, studies using oral fluid rapid diagnostic tests reported errors in the interpretation of test results and studies using blood-based rapid diagnostic tests reported errors in transfer of the blood specimen.

38Two studies, found that people with known HIV status had a higher proportion of errors (ie, when collecting the specimen) when self-testing compared with people with unknown HIV status (0·8% vs 0·2%), whereas a third study found that known HIV-positive participants were more likely to do the test correctly.

39Self-testers can achieve the same results as health-care workers when using HIV rapid diagnostic tests and diagnostic accuracy of rapid diagnostic tests for self-testing is high. Application of the estimated ranges of sensitivity (80–100%) and specificity (95·1–100%) to a hypothetical group of 100 000 people with 1% of HIV prevalence would result in 0–200 HIV-positive cases being missed, and 0–4851 HIV-negative individuals being misidentified with a reactive result, excluding two outliers.,

40This systematic review and meta-analysis suggests that in the hands of self-testers, the sensitivity and specificity of blood-based rapid diagnostic tests were higher than oral fluid rapid diagnostic tests, although fewer studies used blood-based rapid diagnostic tests. The reduced sensitivity is probably explained by the lower quantity of HIV antibodies in oral fluid compared with whole blood, as observed in professional-use assessments. Although blood-based rapid diagnostic tests might have the potential to deliver more accurate results, more invalid results might occur because the greatest number of user errors was related to standard procedures when capillary tubes and pipettes were used.

41Most studies had a high HIV positivity among participants where tests are expected to have a higher positive-predictive value than lower prevalence populations. Furthermore, imperfect reference standards might also decrease the degree of accuracy. We found wide variability in sensitivity estimates, which could be explained by the use of adapted rapid diagnostic tests not specifically designed for self-test use,,,,,,,, or used test kits before approval by national regulatory authorities, in 23 of 25 included studies.,,,,,,,,,,,,,,

42When we excluded six studies,,,,, with low feasibility, the proportion of invalid results met the minimum acceptable criteria for rapid diagnostic test performance (<5%); however, we still found no significant differences in the proportion of invalid results by type of approach, suggesting that use of a rapid diagnostic tests for self-testing without assistance will not increase the possibility of obtaining an invalid result.

43Most invalid results and errors in performance reported by studies included in this review related to user errors and manufacturing defects. These invalid results and errors can be mitigated with instructions for use because the complexity of the test procedure or the complexity of the instructions can increase the possibility of failure in performance and incorrect interpretation of a result.

44Recommendations include use of simple and clear language and well designed pictorial instructions, especially for the steps related to specimen collection, buffer use, and interpretation of results; easily identifiable kit components; reduction of the volume of specimen needed to do the test; and intuitive single-step test kits with controlled and automatic specimen collection, transfer, and buffer use.

45In some settings, instructions could be adapted and validated for the cultural context and for less-skilled users, including individuals with low literacy or visual impairments. This could include translation in local languages, clear and large print instructions for use, detailed images and descriptions, or electronic documents or audio instructions. To improve performance of less-skilled users, instructions for use could be coupled with in-person or video demonstrations on how to do the test and interpret the result.

46Product labelling should clearly state that people with reactive or invalid test results should seek further testing at a health facility. The labelling should also include information on test limitations in detecting HIV infection during the window period, for people taking pre-exposure prophylaxis, or in people with a suppressed immune response, such as people on antiretroviral drugs. This is a crucial issue because reports show that people with HIV on antiretrovial therapy might be using HIV self-testing kits to check and reconfirm their HIV status, and could obtain a false-negative result.,

47Strengths of this study include completeness of the search strategy, explicit inclusion criteria, a systematic approach to data collection, and independent assessment of each included study. Among the limitations were that most included studies used oral fluid-based rapid diagnostic tests, and studies used different and imperfect reference standard tests to identify HIV-positive individuals. Most studies did not compare approaches or specimens head-to-head. Results were considered biased in studies where the reference test strategy was not aligned with WHO testing guidance. Our last search was done on April 30, 2016, and since then 11 studies reporting on HIV self-testing performance have been published,,,,,,,,,,, of which eight are abstracts. These studies reached the same conclusion as we did, reporting that most participants were able to do the self-test correctly, and, where reported, the raw proportion of agreement was also high, ranging from 84% to 99%.,,,, Six of 11 studies would not have met our inclusion criteria, and the four that did meet the inclusion criteria might not have influenced our findings. Furthermore, most studies used adapted test kits that were not specifically designed or packaged for self-testing, and in some studies participants did not interpret their own results, but interpreted contrived devices or pictures or photographs.

48No study provided information on people recently or acutely infected with HIV, and no study disaggregated data by individuals taking antiretroviral drugs. Because little data were available, we could not explore the influence of HIV prevalence, type of reference used, or study design. Selection bias is likely because most studies carefully selected participants; some studies included only known HIV-positive individuals. We did not assess publication bias because analytical methods are not well suited for testing observational data. Finally, although most studies were judged to be at low risk of bias, concerns remained about studies with small samples and the extent to which the findings can be generalised.

49In summary, self-testers can achieve a high level of agreement with the results obtained by a health-care worker when using an HIV rapid diagnostic test for self-testing, whether or not assistance was provided. Errors in performance of the test procedure might be reduced through improvement of the design of rapid diagnostic tests for self-testing, clearer product labels, inclusion of simple instructions for use, and provision of additional support, such as instructional videos.