Skip to main content
SpringerLink
Log in

Quantitative assessment of target delineation variability for thymic cancers: agreement evaluation of a prospective segmentation challenge

  • Original Research
  • Published:
Journal of Radiation Oncology

Abstract

Objectives

We sought to quantitatively determine the interobserver variability of expert radiotherapy target-volume delineation for thymic cancers, as part of a larger effort to develop an expert-consensus contouring atlas.

Methods

A pilot dataset was created consisting of a standardized case presentation with pre- and post-operative DICOM CT image sets from a single patient with Masaoka-Koga Stage III thymoma. Expert thoracic radiation oncologists delineated tumor targets on the pre- and post-operative scans as they would for a definitive and adjuvant case, respectively. Respondents completed a survey including recommended dose prescription and target volume margins for definitive and post-operative scenarios. Interobserver variability was analyzed quantitatively with Warfield’s simultaneous truth, performance level estimation (STAPLE) algorithm and Dice similarity coefficient (DSC).

Results

Seven users completed contouring for definitive and adjuvant cases; of these, five completed online surveys. Segmentation performance was assessed, with high mean ± SD STAPLE-estimated segmentation sensitivity for definitive case GTV and CTV at 0.77 and 0.80, respectively, and post-operative CTV sensitivity of 0.55; all volumes had specificity of ≥0.99. Interobserver agreement was markedly higher for the definitive target volumes, with mean ± SD DSC of 0.88 ± 0.03 and 0.89 ± 0.04 for GTV and CTV, respectively, compared to post-op CTV DSC of 0.69 ± 0.06 (Kruskal-Wallis p < 0.01.

Conclusion

Expert agreement for definitive case volumes was exceptionally high, though significantly lower agreement was noted post-operatively. Technique and dose prescription between experts was substantively consistent, and these preliminary results will be utilized to create an expert-consensus contouring atlas to aid the nonexpert radiation oncologist in the planning of these challenging, rare tumors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Gomez DR, Fuller CD, Chennupati S et al. Mediastinal and Tracheal Cancer. In: Perez and Brady’s Principles and Practice of Radiation Oncology. Halperin EC, Wazer DE, Perez CA, Brady LW eds. 6th edition. © 2013 by Lippincott, Williams & Wilkins. Philadelphia, PA

  2. National Comprehensive Cancer Network. Thymomas and Thymic Carcinomas. Version 1.2014. http://www.nccn.org/professionals/physician_gls/pdf/thymic.pdf. Accessed 09/29/2014

  3. Falkson CB, Bezjak A, Darling G, Gregg R, Malthaner R, Maziak DE et al (2009) Lung cancer disease site group of cancer care Ontario’s Program in evidence-based care. The managmenet of thymoma: a systematic review and practice guideline. J Thorac Onc 4:911–919

    Article  Google Scholar 

  4. Hsu HC, Huang EY, Wang CJ et al (2002) Postoperative radiotherapy in thymic carcinoma: treatment results and prognostic factors. Int J Radiat Oncol Biol Phys 52:801–805

    Article  PubMed  Google Scholar 

  5. Curran WJ Jr, Kornstein MJ, Brooks JJ, Turisi A 3rd (1988) Invasive thymoma: the role of mediastinal irradiation following complete or incomplete surgical resection. J Clin Oncol 6:1722–1727

    PubMed  Google Scholar 

  6. Ogawa K, Uno T, Toita T, Onishi H, Yoshida H, Kakinohana Y et al (2002) Postoperative radiotherapy for patients with completely resected thymoma: a multi-institutional retrospective review of 103 patients. Cancer 94:1405–1413

    Article  PubMed  Google Scholar 

  7. Mornex F, Resbeut M, Richaud P et al (1995) Radiotherapy and chemotherapy for invasive thymomas: a multicentric retrospective review of 90 cases. The FNCLCC trialists. Federation Nationale des Centres de Lutte Contre le Cancer. Int J Radiat Oncol Biol Phys 32:651–659

    Article  CAS  PubMed  Google Scholar 

  8. Nako T, Kanaya H, Namura M et al (1990) Complete atrioventricular block following radiation therapy for malignant thymoma. Jpn J Med 29:104–110

    Article  Google Scholar 

  9. Hirota S, Tsujino K, Hishikawa Y, Watanabe H, Kono K, Soejima T et al (2001) Endoscopic findings of radiation esophagitis in concurrent chemoradiotherapy for intrathoracic malignancies. Radiother Oncol 58:273–278

    Article  CAS  PubMed  Google Scholar 

  10. Lucchi M, Mussi A, Basolo F et al (2001) The multimodality treatment of thymic carcinoma. Eur J Cardiothorac Surg 19:566–569

    Article  CAS  PubMed  Google Scholar 

  11. Gomez DR, Komaki R (2010) Technical advances of radiation therapy for thymic malignancies. J Thorac Oncol 5:S336–343

    Article  PubMed  Google Scholar 

  12. Jeanneret-Sozzi W, Moeckli R, Valley JF et al (2006) The reasons for discrepancies in target volume delineation: a SASRO study on head-and-neck and prostate cancers. Strahlenther Onkol 182:450–457

    Article  PubMed  Google Scholar 

  13. Njeh CF (2008) Tumor delineation: the weakest link in the search for accuracy in radiotherapy. J Med Phys 33:136–140

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Leunens G, Menten J, Weltens C et al (1993) Quality assessment of medical decision making in radiation oncology: variability in target volume delineation for brain tumours. Radiother Oncol 29:169–175

    Article  CAS  PubMed  Google Scholar 

  15. Muijs CT, Schreurs LM, Busz DM et al (2009) Consequences of additional use of PET information for target volume delineation and radiotherapy dose distribution for esophageal cancer. Radiother Oncol 93:447–453

    Article  PubMed  Google Scholar 

  16. Rasch C, Barillot I, Remeijer P et al (1999) Definition of the prostate in CT and MRI: a multi-observer study. Int J Radiat Oncol Biol Phys 43:57–66

    Article  CAS  PubMed  Google Scholar 

  17. Fitton I, Steenbakkers RJ, Gihuijs K et al (2008) Impact of anatomic location on value CT-PET co-registration for delineation of lung tumors. Int J Radiat Oncol Biol Phys 70:1403–1407

    Article  PubMed  Google Scholar 

  18. Kalpathy-Cramer J (2010) Fuller CD (2010) Target contour testing/instructional computer software (TaCTICS): a novel training and evaluation platform for radiotherapy target delineation. AMIA Annu Symp Proc 13:361–365

    Google Scholar 

  19. Kalpathy-Cramer J, Bedrick SD, Boccia K et al (2011) A pilot prospective feasibility study of organ-at-risk definition using Target Contour Testing/Instrutional Computuer Software (TaCTICS), a training and evaluation platform for radiotherapy target delineation. AMIA Annu Symp Proc 2011:654–663

    PubMed  PubMed Central  Google Scholar 

  20. Koch GG, Landis JR, Freeman JL et al (1977) A general methodology for the analysis of experiments with repeated measurement of categorical data. Biometrics 33:133–158

    Article  CAS  PubMed  Google Scholar 

  21. Landis JR, Koch GG (1977) An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 33:363–374

    Article  CAS  PubMed  Google Scholar 

  22. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  CAS  PubMed  Google Scholar 

  23. Kong FM, Ritter T, Quint DJ et al (2011) Consideration of dose limits for organs at risk of thoracic radiotherapy: atlas for lung, proximal bronchial tree, esophagus, spinal cord, ribs and brachial plexus. Int J Radiat Oncol Biol Phys 81:1442–1457

    Article  PubMed  PubMed Central  Google Scholar 

  24. Chapet O, Kong FM, Quint LE et al (2005) CT-based definition of thoraric lymph node stations: an atlas from the University of Michigan. Int J Radiat Oncol Biol Phys 63:170–178

    Article  PubMed  Google Scholar 

  25. Taylor A, Rockall AG, Powell ME (2007) An atlas of the pelvic lymph node regions to aid radiotherapy target volume definition. Clin Oncol (R Coll Radiol) 19:542–550

    Article  CAS  Google Scholar 

  26. Michalski M, Lawton C, El Naqa I et al (2010) Development of RTOG consensus guidelines for the definition of clinical target volume for postoperative conformal radiation therapy for prostate cancer. Int J Radiat Oncol Biol Phys 76:361–368

    Article  PubMed  PubMed Central  Google Scholar 

  27. Nielsen MH, Berg M, Pedersen AN et al (2013) Delineation of target volumes and organs at risk in adjuvant radiotherapy of early breast cancer: national guidelines and contouring atlas by the Danish Breast Cancer Cooperative Group. Acta Oncol 52:703–710

    Article  PubMed  Google Scholar 

  28. Myerson RJ, Garofalo MC, El Naqa I et al (2008) Elective clinical target volumes for conformal therapy in anorectal cancer: a radiation therapy oncology group consensus panel contouring atlas. Int J Radiat Oncol Biol Phys 74:824–830

    Article  PubMed  PubMed Central  Google Scholar 

  29. Goodman KA, Regine WF, Dawson LA et al (2012) Radiation Therapy Oncology Group consensus panel guidelines for the delineation of the clinical target volume in the postoperative treatment of pancreatic head cancer. Int J Radiat Oncol Biol Phys 83:901–908

    Article  PubMed  Google Scholar 

  30. Ng M, Leong T, Chander S et al (2012) Australasian Gastrointestinal Trials Group (AGITG) contouring atlas and planning guidelines for intensity-modulated radiotherapy in anal cancer. Int J Radiat Oncol Biol Phys 83:1455–1462

    Article  PubMed  Google Scholar 

  31. Small W Jr, Mell LK, Anderson P et al (2008) Consensus guidelines for delineation of clinical target volume for intensity modulated pelvic radiotherapy in postoperative treatment of endometrial and cervical cancer. Int J Radiat Oncol Biol Phys 71:428–434

    Article  PubMed  PubMed Central  Google Scholar 

  32. Toita T, Ohno T, Kaneyasu Y et al (2010) A consensus-based guideline defining the clinical target volume for pelvic lymph nodes in external beam radiotherapy for uterine cervical cancer. Jpn J Clin Oncol 40:456–463

    Article  PubMed  Google Scholar 

  33. Gregoire V, Ang K, Budach W et al (2014) Delineation of the neck node levels for head and neck tumors: a 2013 update. DAHANCA, EORTC, HKNPCSG, NCIC, CTG, NCRI, RTOG, TROG consensus guideline. Radiother Oncol 110:172–81

    Article  PubMed  Google Scholar 

  34. Li XA, Tai A, Arthur DW et al (2009) Variability of target and normal structure delineation for breast cancer radiotherapy: an RTOG multi-institutional and multiobserver study. Int J Radiat Oncol Biol Phys 73:944–951

    Article  PubMed  PubMed Central  Google Scholar 

  35. Berkelman S, Wolden S, Lee N (2009) Head-and-neck target delineation among radiation oncology residents after a teaching intervention: a prospective, blinded pilot study. Int J Radiat Oncol Biol Phys 73:416–423

    Article  Google Scholar 

  36. Fuller CD, Nijkamp J, Duppen J et al (2010) Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. Int J Radiat Oncol Biol Phys 72:481–489

    Google Scholar 

  37. Awan M, Kalpathy-Cramer J, Gunn GB et al (2013) Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs at risk in the head and neck: Quantitative assessment of conformance to expert delineation. Pract Radiat Oncol 3:186–193

    Article  PubMed  Google Scholar 

  38. Vorwerk H, Beckmann G, Bremer M et al (2009) The delineation of target volumes for radiotherapy of lung cancer patients. Radiother Oncol 91:455–460

    Article  PubMed  Google Scholar 

  39. Gomez D, Komaki R, Yu J et al (2014) Radiation therapy definitions and reporting guidelines for thymic malignancies. Zhongguo Fei Ai Za Zhi 17:110–115

    PubMed  Google Scholar 

  40. Kim ES, Putnam JB, Komaki R et al (2004) Phase II study of a multidisciplinary approach with induction chemotherapy, followed by surgical resection, radiation therapy, and consolidation chemotherapy for unresectable malignant thymomas: final report. Lung Cancer 44:369–379

    Article  PubMed  Google Scholar 

  41. Cardillo G, Carleo F, Giunti R, et al. Predictors of survival in patients with locally advanced thymoma and thymic carcinoma (Masaoka stages III and IVa). Eur J Cardiothorac Surg 37:819–823

  42. Hanna GG, Carson KJ, Lynch T et al (2010) 18F-fluorodeoxyglucose positron emission tomography/computed tomography-based radiotherapy target volume definition in non-small-cell lung cancer: delineation by radiation oncologists vs. joint outlining with a PET radiologist? Int J Radiat Oncol Biol Phys 78:1040–1051

    Article  PubMed  Google Scholar 

  43. Endo M, Nakagawa K, Ohde Y et al (2008) Utility of 18FDG-PET for differentiating the grade of malignancy in thymic epithelial tumors. Lung Cancer 61:350–355

    Article  PubMed  Google Scholar 

  44. Kumar A, Regmi SK, Dutta R et al (2009) Characterization of thymic masses using (18)F-FDG PET-CT. Ann Nucl Med 23:569–577

    Article  PubMed  Google Scholar 

  45. Inoue A, Tomiyama N, Fujimoto K et al (2006) MR imaging of thymic epithelial tumors: correlation with World Health Organization classification. Radiat Med 24:171–181

    Article  PubMed  Google Scholar 

  46. Tomiyama N, Honda O, Tsubamoto M et al (2009) Anterior mediastinal tumors: diagnostic accuracy on CT and MRI. Eur J Radiol 69:280–288

    Article  PubMed  Google Scholar 

  47. Fuller CD, Duppen J, Rasch CR et al (2009) A Prospective Randomized Pilot Study of Site-specific Atlas Incorporation into Target Volume Delineation Instructions in the Cooperative Group Setting: Preliminary Results from a Southwest Oncology Group Pilot using Big Brother. Int J Radiat Oncol Biol Phys 75:S136–S137

    Article  Google Scholar 

  48. Castillo R, Castillo E, Guerra R et al (2009) A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets. Phys Med Biol 54:1849–1870

    Article  PubMed  Google Scholar 

  49. Castillo R, Castillo E, Fuentes D et al (2013) A reference dataset for deformable image registration spatial accuracy evaluation using the COPDgene study archive. Phys Med Biol 58:2861–2877

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Unit 97, Houston, TX, 77030, USA

    Emma Holliday, Clifton D. Fuller, Daniel Gomez & Ritsuko Komaki

  2. Department of Radiology, Massachusetts General Hospital, Boston, MA, USA

    Jayashree Kalpathy-Cramer

  3. Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, USA

    Andreas Rimner

  4. Department of Radiation Oncology, The University of Texas Health Science Center San Antonio, San Antonio, TX, USA

    Ying Li

  5. Department of Radiation Oncology, University Medical Center, Amsterdam, The Netherlands

    Suresh Senan

  6. Department of Therapeutic Radiology, Yale Cancer Center, New Haven, CT, USA

    Lynn D. Wilson

  7. Department of Radiation Oncology, Kaiser Permanente Southern California, Los Angeles, CA, USA

    Jehee Choi

  8. Department of Radiation Medicine, Oregon Health and Science University Knight Cancer Center, Portland, OR, USA

    Charles R. Thomas Jr.

Authors
  1. Emma Holliday
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Clifton D. Fuller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jayashree Kalpathy-Cramer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Gomez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andreas Rimner
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Suresh Senan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lynn D. Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jehee Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ritsuko Komaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Charles R. Thomas Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Clifton D. Fuller.

Ethics declarations

Conflict of interest

Emma Holliday, Jayashree Kalpathy-Cramer, Daniel Gomez, Andreas Rimner, Ying Li, Suresh Senan, Lynn D. Wilson, Jehee Choi, Ritsuko Komaki, and Charles R. Thomas declare that they have nothing to disclose.

Statements of ethical standards

This article does not contain any studies with human or animals performed by any of the authors.

Funding source(s)

Dr. Fuller received/receives grant support from: the SWOG Hope Foundation Dr. Charles A. Coltman, Jr. Fellowship in Clinical Trials; the National Institutes of Health Paul Calabresi Clinical Oncology Award Program (K12 CA088084) and Clinician Scientist Loan Repayment Program (L30 CA136381-02); Elekta AB/MD Anderson Consortium; GE Medical Systems/MD Anderson Center for Advanced Biomedical Imaging In-Kind Award; the MD Anderson Center for Radiation Oncology Research, and an MD Anderson Institutional Research Grant Program Award. These listed funders/supporters played no role in the study design, collection, analysis, interpretation of data, manuscript writing, or decision to submit the report for publication.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Holliday, E., Fuller, C.D., Kalpathy-Cramer, J. et al. Quantitative assessment of target delineation variability for thymic cancers: agreement evaluation of a prospective segmentation challenge. J Radiat Oncol 5, 55–61 (2016). https://doi.org/10.1007/s13566-015-0230-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13566-015-0230-7

Keywords

Search

Navigation