| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
ORIGINAL RESEARCH |
From the Program for Appropriate Technology in Health (PATH), Seattle, Washington; Pan American Health Organization (PAHO), Lima, Peru; International Agency for Research on Cancer (IARC), Lyon, France; Department of Pathology, Columbia University, New York, New York; Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada; and JHPIEGO Corp., Baltimore, Maryland.
Address reprint requests to: J. W. Sellors, MD, Program for Appropriate Technology in Health, 1455 NW Leary Way, Seattle, WA 98107-5136; E-mail: jsellors{at}path.org.
 |
ABSTRACT |
|---|
TOP
 ABSTRACT MATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
METHODS: Three raters individually assessed 144 photographs as negative, positive, or suspicious for cancer. The inter-rater agreement was analyzed using the unweighted and weighted 
coefficient. To explore the reasons for concordancy and discordancy, photographs were compared on histologic evidence of cervical intraepithelial neoplasia and on testing for oncogenic types of human papillomavirus.
RESULTS: Overall raw agreement among the three raters was 66.7% (96 of 144) with a of 0.57 (95% confidence interval 0.48, 0.66). Pair-wise agreement using unweighted and weighted was moderate to substantial: 0.54–0.60 and 0.56–0.63, respectively. There was concordance on negative in 25.7% (37 of 144) and on positive or suspicious for cancer in 41.0% (59 of 144). Cervical intraepithelial neoplasia II or III was not present on biopsy if photographs were concordant-negative, and the human papillomavirus test was less likely to be positive (relative risk 0.3; 95% confidence interval 0.2, 0.6) in concordant-negatives compared with concordant-positives, including suspicious for cancer. Cervical intraepithelial neoplasia II or III was more common in photographs that were concordant-positive, including suspicious for cancer, compared with discordants (relative risk 3.4, 95% confidence interval 1.5, 7.6).
CONCLUSION: Based on photographs of the cervix taken after acetic acid wash, the level of agreement among raters using visual inspection with acetic acid categories was moderate to substantial, consistent with other commonly used tests.
Visual inspection with acetic acid, also referred to as direct visual inspection or cervicoscopy, has attracted increasing interest as a promising screening method to prevent cervical cancer—a leading cause of death in resource-poor settings.1–6 (Sankaranarayanan R, Shyamalakumary B, Wesley R, Sreedevi Amma N, Parkin DM, Nair MK. Visual inspection with acetic acid in the early detection of cervical cancer and precursors [letter]. Int J Cancer 1999;80:161–3.) In contrast to training laboratory technicians to read cervical cytology, para-medical personnel can be trained over several days to perform visual inspection with acetic acid at a high level of competence.3–5 A vaginal speculum is inserted, table vinegar (3–5% acetic acid) is applied to the cervix, and a white light (eg, flashlight or torch) is used to inspect the cervix for acetowhite lesions. The clinician assesses the cervix as visual inspection with acetic acid-negative, visual inspection with acetic acid-positive, or suspicious for cancer. Women with cervices that are visual inspection with acetic acid-positive might then be either referred for further assessment using colposcopy or another screening test (eg, two-stage screening), or immediately treated using a simple outpatient ablative technique such as cryosurgery. Women with cervices scored as suspicious for cancer would be referred for appropriate workup and therapy. Visual inspection with acetic acid offers a potentially feasible strategy to screen women in resource-poor settings where traditional cytology is not available.
Because visual inspection with acetic acid is a relatively new clinical test, it should be carefully evaluated for both accuracy and reliability.7 The accuracy of visual inspection with acetic acid has been well characterized in one study,4 which avoided spectrum bias, verification bias, and review bias—biases that can inflate estimates of a test’s accuracy.7 With respect to reliability, there has not been a published evaluation of observer agreement on visual inspection with acetic acid. Such a study would assist in the appraisal of this new test and could suggest ways to improve the definitions of its categories.
This study was done 3 months after an invitational workshop on visual inspection with acetic acid training,8 attended by international experts who were evaluating this test as a screening tool for resource-poor settings. Because it was clear that cervical photographs likely would be an important component of visual inspection with acetic acid training and that teaching of this test would benefit from a more standardized approach, it was decided to use cervical photographs to measure the inter-rater agreement among three of the physicians (from India, Peru, and the United States) who had attended the conference.
|
MATERIALS AND METHODS |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
The raters used one of three mutually exclusive categories: "positive," "negative," or "suspicious for cancer" for each photograph. "Positive" was described as a well-defined, dense acetowhite area touching or very near to the squamocolumnar junction. "Suspicious for cancer" referred to a lesion with an irregular, elevated surface that might be hemorrhagic. "Negative" was defined as the absence of any acetowhite area or, at most, the presence of a very faint or pale acetowhite area(s). The years of field experience with visual inspection with acetic acid and colposcopy were estimated for observer A, B, and C as 1.0 and 0.1, 2.5 and 5.0, and 0.1 and 10.0, respectively.
At the time the photographs were taken, colposcopists took directed biopsies of any areas that appeared abnormal. They also performed an endocervical curettage if a lesion extended into the endocervical canal or if the referral Papanicolaou smear showed high-grade squamous intraepithelial lesion or cancer and there was no corresponding colposcopic lesion. When reading histology, the pathologists were blinded to the colposcopic findings. For the Canadian histology, two expert gynecologic pathologists independently read the histology specimens; they were blinded to the colposcopy and cytology findings. The pathologists used CIN terminology for histology reporting: changes consistent with human papillomavirus (HPV) were classified as CIN I if no higher degree of dysplasia was present. If a woman had multiple biopsies, including endocervical curettage, the most severe diagnosis was used as the reference standard. For any disagreement in diagnosis, the pathologists reviewed the histology together and reached a consensus. In Peru, the histology specimens were read as usual by pathologists at the National Cancer Institute.
The cervix was designated as normal if colposcopic examination showed no lesion and the colposcopists submitted no biopsies or curettings for histologic examination, or if tissue was taken for histologic examination but it showed no CIN or cancer.
The transformation zone was sampled for the Hybrid Capture II assay (Digene Corp., Gaithersburg, MD) with a soft, cone-shaped cervical brush (Digene). The cervical brush was placed in specimen transport medium and stored at 4C, then shipped within 2 weeks to the McMaster University Regional Virology and Chlamydiology Laboratory for HPV testing according to the manufacturer’s protocol.9 The Hybrid Capture II assay can detect 13 of the most common oncogenic types of HPV as a group, and gives a semiquantitative result in relative light units. A specimen was considered positive if the light emitted was equal to or greater than the light emitted by the positive control at the standard 1 pg cutoff level.
Data were entered and analyzed using SPSS 10.0.5 (SPSS Inc., Chicago, IL). The coefficient can range from -1.0 to 1.0, with 0 indicating chance agreement. A of less than 0 indicates poor agreement, 0–0.2 indicates slight agreement, 0.2–0.4 indicates fair agreement, 0.4–0.6 indicates moderate agreement, 0.6–0.8 indicates substantial agreement, and 0.8–1.0 indicates almost perfect agreement.10 Agreement among the three raters was determined using the multirater statistic with three categories of assessment.11 Agreement was calculated (SAS, 8.1, SAS Institute Inc., Cary, NC) for each pair of raters using Cohen’s , with and without weighting.12 In the weighted calculation, a disagreement between two adjacent categories (ie, negative versus positive and positive versus suspicious for cancer) is weighted less heavily than a disagreement between non-adjacent categories (ie, negative versus suspicious for cancer). Sensitivity and specificity of the photo assessments were calculated for each rater and for the majority (at least two of the three raters in agreement), based on colposcopy and histology as the reference standard. The 
2 test for trend was based on the Mantel-Haenszel statistic.13
|
RESULTS |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
The overall raw agreement among the three raters was 66.7% (96 of 144). The assessments were concordant for a negative rating in 25.7% (37 of 144) and concordant in 41.0% (59 of 144) for either a positive or a suspicious for cancer rating. Comparing negative with positive or suspicious for cancer—for the decision on whether a woman needs referral for further assessment—the observers disagreed on 33.3% (48 of 144). The pair-wise agreement () for the three raters, without weighting and with weighting to take into account the magnitude of categoric differences, was moderate to substantial, ranging from 0.54 to 0.60 and 0.56 to 0.63, respectively (Table 1
). The for multiple raters was 0.57 (95% confidence interval [CI] 0.48–0.66).
|
). When the assessments were concordant on negative, HPV was less likely to be detected (relative risk 0.3; 95% confidence interval 0.2, 0.6) than when there was concordance on positive, including suspicious for cancer. The prevalence of HPV was similar among women whose assessments were discordant (62.2%) and those who had positively concordant assessments, including suspicious for cancer (57.6%) (P = .64). The prevalence of biopsy-confirmed CIN II or III or squamous carcinoma was zero in the concordant-negatives but 3.4 (95% confidence interval 1.5, 7.6) times higher in the concordant-positive category, including suspicious for cancer, compared with the discordant photographs. These findings suggest that raters might be influenced by the presence of low-grade lesions, causing them not to adhere to the criteria for positivity. Consistent with this hypothesis, the coefficients for multiple raters were 0.62 (95% CI 0.45–0.77) and 0.35 (95% CI 0.18–0.52) in those who tested negative and positive for HPV, respectively. The numbers were too small to compare the level of agreement for each stratum of histologic diagnosis.
|
). Rater "C" had the highest sensitivity for CIN II or III or squamous carcinoma (96.8%) and the lowest specificity (38.9%) among the three raters. Improved specificity for CIN II or III was associated with years of field experience with visual inspection with acetic acid ( 2 for trend = 8.55, P = .003) and with years of colposcopy experience (2 for trend = 6.37, P = .012). There was no evidence for an association between years of experience with colposcopy (2 for trend = 2.06, P = .151) or visual inspection with acetic acid (2 for trend = 0.81, P = .37) and the sensitivity of photographic assessment for CIN II or III.
|
|
DISCUSSION |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
The histology and HPV test data, in addition to the visual images, allowed further examination of images for which agreement and disagreement occurred. The histology profiles of the concordant-positive and concordant-negative women showed a predominance of high-grade lesions or carcinoma (42.4%) and normal (100%) findings, respectively. The prevalence of HPV was also distributed as one would expect: 18.9% in concordant-negative women and 57.6% in those who were concordant-positive, including suspicious for cancer. Although one of the aims of visual inspection with acetic acid is to correctly identify women with more severe lesions, it would be ideal if it could also help to discriminate them from those with lower-grade lesions, which are more likely to regress.21 The relatively high prevalence of HPV in the discordants suggests that low-grade lesions may be common and easily confused with mild ace-towhitening of areas of immature metaplasia. This situation is in contrast to cervical cytology, for which agreement on squamous intraepithelial lesions increased when smears that were positive for HPV were separated from those which were not.17
It has been argued that ensuring the reliability of a test is an important step in achieving accuracy.7 Because all of the women had been examined by colposcopy, with histology as appropriate, it was also possible to evaluate the sensitivity and specificity of photographic assessment using the visual inspection with acetic acid categories. Colposcopy with directed biopsy is acknowledged to be an imperfect reference standard, but it is the best available.22–27 Using this reference standard, the accuracy of the assessments based on these photographs was comparable with visual inspection with acetic acid performed by nurse-midwives screening women.4 This is quite plausible because our study used experienced physicians, selected images, and ample time in a nonclinical setting to make an assessment. More years of field experience with either visual inspection with acetic acid or colposcopy were associated with higher specificity of the photographic assessments for high-grade lesions, but there was no discernible association between experience and sensitivity.
This study is timely for the continuing growth and development of visual inspection with acetic acid, which is a relatively new test. The current performance characteristics are comparable with many other tests in common use, and the findings of this study will hopefully lead to further improvement. Improved quality and consistency will increase effectiveness, decrease inequity in delivery of health services in resource-poor settings, and facilitate teaching of visual inspection with acetic acid. The use of photographs may be helpful for teaching visual inspection with acetic acid and for quality control within screening programs. It is preferable to evaluate and perfect a new test before, not after, its dissemination and implementation.7 The next steps will be to convene these raters and other teachers of visual inspection with acetic acid to discuss why the criteria work when there is agreement and to better understand if criteria can be improved to avoid disagreement.
|
Footnotes |
|---|
We wish to acknowledge Lynne Gaffikin, epidemiologist consultant with JHPIEGO Corp., who reviewed the manuscript and made many helpful suggestions.
Received August 17, 2001. Received in revised form November 1, 2001. Accepted November 19, 2001.
|
REFERENCES |
|---|
|
TOP
ABSTRACTMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
2. Cullins VE, Wright TC, Beattie KJ, Pollack AE. Cervical cancer prevention using visual screening methods. Reprod Health Matters 1999;7:134–43.
3. Sankaranarayanan R, Wesley R, Somanathan T, Dhakad N, Shyamalakumary B, Amma NS, et al. Visual inspection of the uterine cervix after the application of acetic acid in the detection of cervical carcinoma and its precursors. Cancer 1998;83:2150–6.[Medline]
4. University of Zimbabwe/JHPIEGO Cervical Cancer Project. Visual inspection with acetic acid for cervical-cancer screening: Test qualities in a primary-care setting. Lancet 1999;353:869–73.[Medline]
5. Denny L, Kuhn L, Pollack A, Wainwright H, Wright TC Jr. Evaluation of alternative methods of cervical cancer screening for resource-poor settings. Cancer 2000;89: 826–33.[Medline]
6. Blumenthal PD, Gaffikin L, Chirenje ZM, McGrath J, Womack S, Shah K. Adjunctive testing for cervical cancer in low resource settings with visual inspection, HPV, and the Pap smear. Int J Gynaecol Obstet 2001;72:47–53.[Medline]
7. Reid MC, Lachs MS, Feinstein AR. Use of methodologic standards in diagnostic test research: Getting better but still not good. JAMA 1995;274:645–51.[Abstract]
8. JHPIEGO. Proceedings of the Symposium on Training Issues in Cervical Cancer Prevention, September 2000. Baltimore, MD: JHPIEGO Corp.; 2000.
9. Lytwyn A, Sellors JW, Mahony JB, Daya D, Chapman W, Ellis N, et al. Comparison of human papillomavirus DNA testing and repeat Papanicolaou test in women with low-grade cervical cytologic abnormalities: A randomized trial. HPV Effectiveness in Lowgrade Paps (HELP) Study No. 1 Group. CMAJ 2000;163:701–7.
10. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74.[Medline]
11. Siegel S, Castellan NJ Jr. Nonparametric statistics for the behavioral sciences. 2nd ed. New York: McGraw-Hill; 1988.
12. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46.
13. Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst 1959;22:719–48.
14. Sellors JW, Niemenen P, Vesterinen E, Paavonen J. Observer variability in the scoring of colpophotographs. Obstet Gynecol 1990;76:1006–8.
15. Etherington IJ, Luesley DM, Shafi MI, Dunn J, Hiller L, Jordan JA. Observer variability among colposcopists from the West Midlands region. Br J Obstet Gynaecol 1997; 104:1380–4.[Medline]
16. Ferris DG, Cox JT, Burke L, Litaker MS, Harper DM, Campion MJ, et al. Colposcopy quality control: Establishing colposcopy criterion standards for the National Cancer Institute ALTS trial using cervigrams. J Lower Genital Tract Dis 1998;2:195–203.
17. Sherman ME, Schiffman MH, Lorincz AT, Manos MM, Scott DR, Kurman RJ, et al. Toward objective quality assurance in cervical cytopathology. Correlation of cytopathologic diagnoses with detection of high-risk human papillomavirus types. Am J Clin Pathol 1994;102:182–7.[Medline]
18. Ismail SM, Colcough AB, Dinnen JS, Eakins D, Evans DM, Gradwell E, et al. Observer variation in histopathological diagnosis and grading of cervical intraepithelial neoplasia. Br Med J 1998;298:707–10.
19. Robertson AJ, Anderson JM, Beck JS, Burnett RA, Howatson SR, Lee FD, et al. Observer variability in histological reporting of cervical biopsy specimens. J Clin Pathol 1989; 42:231–8.
20. Preti M, Mezzetti M, Robertson C, Sideri M. Inter-observer variation in histopathological diagnosis and grading of vulvar intraepithelial neoplasia: Results of a European collaborative study. BJOG 2000;107:594–9.[Medline]
21. Holowaty P, Miller AB, Rohan T, To T. Natural history of dysplasia of the uterine cervix. J Natl Cancer Inst 1999;91:252–8.
22. Buxton EJ, Luesley DM, Shafi MI, Rollason M. Colposcopically directed punch biopsy: A potentially misleading investigation. Br J Obstet Gynaecol 1991;98:1273–6.[Medline]
23. Chappatte OA, Byrne DL, Raju KS, Nayagam M, Kenney A. Histological differences between colposcopic-directed biopsy and loop excision of the transformation zone (LETZ): A cause for concern. Gynecol Oncol 1991;43: 46–50.[Medline]
24. Howe DT, Vincenti AC. Is large loop excision of the transformation zone (LLETZ) more accurate than colposcopically directed punch biopsy in the diagnosis of cervical intraepithelial neoplasia? Br J Obstet Gynaecol 1991;98:588–91.[Medline]
25. Massad LS, Halperin CJ, Bitterman P. Correlation between colposcopically directed biopsy and cervical loop excision. Gynecol Oncol 1996;60:400–3.[Medline]
26. Mitchell MF, Schottenfeld D, Tortolero-Luna G, Cantor SB, Richards-Kortum R. Colposcopy for the diagnosis of squamous intraepithelial lesions: A meta-analysis. Obstet Gynecol 1998;91:626–31.[Abstract]
27. Belinson JL, Pretorius RG, Zhang WH, Wu LY, Qiao YL, Elson P. Cervical cancer screening by simple visual inspection after acetic acid. Obstet Gynecol 2001;98:441–4.
This article has been cited by other articles:
|
|
 |
|
  L. Denny, L. Kuhn, M. De Souza, A. E. Pollack, W. Dupree, and T. C. Wright Jr Screen-and-Treat Approaches for Cervical Cancer Prevention in Low-Resource Settings: A Randomized Controlled Trial JAMA, November 2, 2005; 294(17): 2173 - 2181. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. C. Wright Jr. Chapter 10: Cervical Cancer Screening Using Visualization Techniques J Natl Cancer Inst Monographs, June 1, 2003; 2003(31): 66 - 71. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Molander, P. Finne, J. Sjoberg, J. Sellors, and J. Paavonen Observer Agreement With Laparoscopic Diagnosis of Pelvic Inflammatory Disease Using Photographs Obstet. Gynecol., May 1, 2003; 101(5): 875 - 880. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
