Skip to main content
SpringerLink
Log in

A Java-based fMRI Processing Pipeline Evaluation System for Assessment of Univariate General Linear Model and Multivariate Canonical Variate Analysis-based Pipelines

  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

As functional magnetic resonance imaging (fMRI) becomes widely used, the demands for evaluation of fMRI processing pipelines and validation of fMRI analysis results is increasing rapidly. The current NPAIRS package, an IDL-based fMRI processing pipeline evaluation framework, lacks system interoperability and the ability to evaluate general linear model (GLM)-based pipelines using prediction metrics. Thus, it can not fully evaluate fMRI analytical software modules such as FSL.FEAT and NPAIRS.GLM. In order to overcome these limitations, a Java-based fMRI processing pipeline evaluation system was developed. It integrated YALE (a machine learning environment) into Fiswidgets (a fMRI software environment) to obtain system interoperability and applied an algorithm to measure GLM prediction accuracy. The results demonstrated that the system can evaluate fMRI processing pipelines with univariate GLM and multivariate canonical variates analysis (CVA)-based models on real fMRI data based on prediction accuracy (classification accuracy) and statistical parametric image (SPI) reproducibility. In addition, a preliminary study was performed where four fMRI processing pipelines with GLM and CVA modules such as FSL.FEAT and NPAIRS.CVA were evaluated with the system. The results indicated that (1) the system can compare different fMRI processing pipelines with heterogeneous models (NPAIRS.GLM, NPAIRS.CVA and FSL.FEAT) and rank their performance by automatic performance scoring, and (2) the rank of pipeline performance is highly dependent on the preprocessing operations. These results suggest that the system will be of value for the comparison, validation, standardization and optimization of functional neuroimaging software packages and fMRI processing pipelines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Beckmann, C. F., & Smith, S. M. (2004). Probabilistic independent component analysis for functional magnetic resonance imaging. IEEE Transactions on Medical Imaging, 23(2), 137–152.

    Article  PubMed  Google Scholar 

  • Blankertz, B., Curio, G., & Müller, K. (2002). Classifying single trial EEG: Towards brain computer interfacing. In T. G. Diettrich, S. Becker, & Z. Ghahramani (Eds.), Advances in Neural Inf. Proc. Systems (NIPS 01). 14, 157–164.

  • Bluemke, D. A., Gatsonis, C. A., Chen, M. H. , DeAngelis, G. A., DeBruhl, N., Harms, S., et al. (2004). Magnetic resonance imaging of the breast prior to biopsy. JAMA, 292(22), 2735–2742.

    Article  PubMed  CAS  Google Scholar 

  • Bullmore, E. T., Brammer, M., Rouleau, G., Everitt, B., Simmons, A., Sharma, T., et al. (1995). Computerized brain tissue classification of magnetic resonance images: A new approach to the problem of partial volume artifact. NeuroImage, 2(2), 133–147.

    Article  PubMed  CAS  Google Scholar 

  • Chance, M. R., Bresnick, A. R., Burley, S. K., Jiang, J. S., Lima, C. D., Sali, A., et al. (2002). Structural genomics: A pipeline for providing structures for the biologist. Protein Science, 11, 723–738.

    Article  PubMed  CAS  Google Scholar 

  • Cohn, D., Atlas, L., & Ladner, R. (1994). Improving generalization with active learning. Machine Learning, 15(2), 201–221.

    Google Scholar 

  • Cox, R. W. (1996). AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. Computers and Biomedical Research, 29, 162–173 [http://afni.nimh.nih.gov/afni/].

    Article  PubMed  CAS  Google Scholar 

  • Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37(1), 36–48.

    Article  Google Scholar 

  • Fissell, K., Tseytlin, E., Cunningham, D., Iyer, K., Carter, C. S., Schneider, W., et al. (2003). Fiswidgets: A graphical computing environment for neuroimaging analysis. Neuroinformatics, 1(1), 111–126 [http://grommit.lrdc.pitt.edu/fiswidgets].

    Article  PubMed  Google Scholar 

  • Ford, J., Makedon, F., Megalooikonomou, V., Shen, L., Steinberg, T., & Saykin, A. J. (2001). Spatial comparison of fMRI activation maps for data mining. NeuroImage, 13(6), 1302.

    Article  Google Scholar 

  • Friston, K. J., & Penny, W. (2003). Posterior probability maps and SPMs. Neuroimage, 19(3), 1240–1249.

    Article  PubMed  CAS  Google Scholar 

  • Genovese, C. R., Noll, D. C., & Eddy, W. F. (1997). Estimating test-retest reliability in fMRI I: Statistical methodology. Magnetic Resonance in Medicine, 38, 497–507.

    Article  PubMed  CAS  Google Scholar 

  • Gevins, A. S., Morgan, N. H., Bressler, S. L., Cutillo, B. A., White, R. M., Illes, J., et al. (1987). Human neuroelectric patterns predict performance accuracy. Science, 235, 580–585.

    Article  PubMed  CAS  Google Scholar 

  • Gold, S., Christian, B., Arndt, S., Zeien, G., Ted Cizadlo, T., Johnson, D. L., et al. (1998). Functional MRI statistical software packages: A comparative analysis. Human Brain Mapping, 6(2), 73–84.

    Article  PubMed  CAS  Google Scholar 

  • Grant, J. D., Somers, L. A., Zhang, Y., Manion, F. J., Bidaut, G., & Ochs, M. F. (2004). FGDP: Functional genomics data pipeline for automated, multiple microarray data analyses. Bioinformatics, 20(2), 282–283.

    Article  PubMed  CAS  Google Scholar 

  • Hansen, L. K., Larsen, J., Nielsen, F. A., Strother, S. C., Rostrup, E., Savoy, R., et al. (1999). Generalizable patterns in neuroimaging: How many principal components? Nueroimage, 9(5), 534–544.

    Article  CAS  Google Scholar 

  • Haynes, J. D., & Rees, G. (2006). Decoding mental states from brain activity in humans. Nature Reviews. Neuroscience, 7, 523–534.

    Article  PubMed  CAS  Google Scholar 

  • Herskovits, E. H., & Gerring, J. P. (2003). Application of a data-mining method based on Bayesian networks to lesion-deficit analysis. NeuroImage, 19(4), 1664–1673.

    Article  PubMed  Google Scholar 

  • Holmes, A. P. (1994). Statistical issues in functional brain mapping. PhD thesis, University of Glasgow.

  • Iyengar, V. S., Apte, C., & Zhang, T. (2000). Active learning using adaptive resampling Proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining. pp. 91–98.

  • Kay, K. N., David, S. V., Prenger, R. J., Hansen, K. A., & Gallant, J. L. (2008). Modeling low-frequency fluctuation and hemodynamic response timecourse in event-related fMRI. Human Brain Mapping, 29(2), 142–156 (Feb).

    Article  PubMed  Google Scholar 

  • Kippenham, J. S., Barker, W. W., Pascal, S., Nagel, J., & Duara, R. (1992). Evaluation of a neural-network classifier for PET scans of normal and Alzheimers disease subjects. Journal of Nuclear Medicine, 33, 1459–1467.

    Google Scholar 

  • Kjems, U., Hansen, L. K., Anderson, J., Frutiger, S., Muley, S., Sidtis, J., et al. (2002). The quantitative evaluation of functional neuroimaging experiments: Mutual information learning curves. NeuroImage, 15(4), 772–786.

    Article  PubMed  CAS  Google Scholar 

  • Kustra, R., & Strother, S. C. (2001). Penalized discriminant analysis of [15O] water PET brain images with prediction error selection of smoothing and regularization hyperparameters. IEEE Transactions on Medical Imaging, 20, 376–387.

    Article  PubMed  CAS  Google Scholar 

  • LaConte, S., Anderson, J., Muley, S., Ashe, J., Frutiger, S., Rehm, K., et al. (2003). The evaluation of preprocessing choices in single-subject BOLD fMRI using NPAIRS performance metrics. NeuroImage, 18(1), 10–27.

    Article  PubMed  Google Scholar 

  • LaConte, S., Strother, S., Cherkassky, V., Anderson, J., & Hu, X. (2005). Support vector machines for temporal classification of block design fMRI data. NeuroImage, 26(2), 317–329.

    Article  PubMed  Google Scholar 

  • Lange, N., Strother, S. C., Anderson, J. R., Nielsen, F. Å., Holmes, A. P., Kolenda, T., et al. (1999). Plurality and resemblance in fMRI data analysis. NeuroImage, 10(3), 282–303.

    Article  PubMed  CAS  Google Scholar 

  • Lautrup, B., Hansen, L. K., Law, I., Morch, N., Svarer, C., & Strother, S. C. (1994). Massive weight-sharing: A cure for extremely ill-posed problems. In H. J. Hermann, D. E. Wolf, & E. Poeppel (Eds.) Proceedings of the workshop on supercomputing in brain research: From tomography to neural networks. Ulich, Germany: World Scientific.

    Google Scholar 

  • Le, T. H., & Hu, X. (1997). Methods for assessing accuracy and reliability in functional MRI. NMR Biomedicine. NMR Biomedicine, 10, 160–164.

    CAS  Google Scholar 

  • Lehman, C. D., Peacock, S., DeMartini, W. B., & Chen, X. (2006). A new automated software system to evaluate breast MR examinations: Improved specificity without decreased sensitivity. AJR American Journal of Roentgenology, 187(1), 51–56.

    Article  PubMed  Google Scholar 

  • Lewis, J. P., & Neumann, U. (2004). Performance of Java versus C++. http://www.idiom.com/~zilla/Computer/javaCbenchmark.html.

  • Liberman, L., Morris, E. A., Kim, C. M., Kaplan, J. B., Abramson, A. F., Menell, J. H., et al. (2003). MR imaging findings in the contralateral breast of women with recently diagnosed breast cancer. AJR American Journal of Roentgenology, 180(2), 333–341.

    PubMed  Google Scholar 

  • Liu, L., Meier, D., Polgar-Turcsanyi, M., Karkocha, P., Bakshi, R., & Guttmann, C. R. G. (2004). Event-driven workflow management for medical image processing and analysis in a large image database Proc. of Distributed Databases and Processing in MedImg. Comp. France. 74–83.

  • Liu, L., Meier, D., Polgar-Turcsanyi, M., Karkocha, P., Bakshi, R., & Guttmann, C. R. G. (2005). Multiple sclerosis medical image analysis and information management. Journal of Neuroimaging, 15(s4), 103–117.

    Article  CAS  Google Scholar 

  • Lukic, A. S., Wernick, M. N., & Strother, S. C. (2002). An evaluation of methods for detecting brain activations from PET or fMRI images. Artificial Intelligence in Medicine, 25, 69–88.

    Article  PubMed  Google Scholar 

  • Madsen, R. E. (2003). Multi-subject fMRI generalization with independent component representation. IMM Technical Report. [http://www2.imm.dtu.dk/~rem/index.php?page=publications].

  • Maitra, R., Roys, S. R., Gullapalli, R. P. (2002). Test-retest reliability estimation of functional MRI Data. Magnetic Resonance in Medicine, 48, 62–70.

    Article  PubMed  Google Scholar 

  • Morch, N., Hansen, L. K., Strother, S. C., Svarer, C., Rottenberg, D. A., Lautrup, B., et al. (1997). Nonlinear versus linear models in functional neuroimaging: Learning curves and generalization crossover. In J. Duncan & G. Gindi (Eds.), Lecture notes in computer science 1230: Information processing in medical imaging (pp. 259–270). Springer-Verlag.

  • Mungall, C. J., Misra, S., Berman, B. P., Carlson, J., Frise, E., Harris, N., et al. (2002). An integrated computational pipeline and database to support whole-genome sequence annotation. Genome Biology, 3(12), research0081.1–0081.11.

    Article  Google Scholar 

  • Nandy, R. R., & Cordes, D. (2003). Novel ROC-type method for testing the efficiency of multivariate statistical methods in fMRI. Magnetic Resonance in Medicine, 49(6), 1152–1162.

    Article  PubMed  Google Scholar 

  • Nicolaou, N., & Nasuto, S. J. (2004). Temporal independent component analysis for automatic artefact removal from EEG, in Proc. MEDSIP’04, 2nd International Conference on Medical Signal and Information Signal Processing, Sliema, Malta.

  • Prechelt, L. (2000). An empirical comparison of seven programming languages. IEEE Computer, 33(10), 23–29.

    Google Scholar 

  • Rex, D. E., Ma, J. Q., & Toga, A. W. (2003). The LONI pipeline processing environment. NeuroImage, 19(3), 1033–1048.

    Article  PubMed  Google Scholar 

  • Shaw, M. E., Strother, S. C., Gavrilescu, M., Podzebenko, K., Waites, A., Watson, J., et al. (2003). Evaluating subject specific preprocessing choices in multisubject fMRI data sets using data-driven performance metrics. NeuroImage, 19(3), 988–1001.

    Article  PubMed  Google Scholar 

  • Sherlock, R., Mooney, P., Winstanley, A., & Husdal, J. (2002). Shortest path computation: A comparative analysis. Conference paper accepted for presentation at the GISRUK 2002 conference, University of Sheffield, Sheffield, UK.

  • Skudlarski, P., Constable, R. T., & Gore, J. C. (1999). ROC analysis of statistical methods used in functional MRI: Individual subjects. NeuroImage, 9(3), 311–329.

    Article  PubMed  CAS  Google Scholar 

  • Smith, S. M., Beckmann, C. F., Ramnani, N., Woolrich, M. W., Bannister, P. R., Jenkinson, M., et al. (2005). Variability in fMRI: A re-examination of inter-session differences. Human Brain Mapping, 24, 248–257.

    Article  PubMed  Google Scholar 

  • Smith, S. M., Jenkinson, M., Woolrich, M. W., Beckmann, C. F., Behrens, T. E. J., Johansen-Berg, H., et al. (2004). Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage, 23(S1), 208–219 [http://www.fmrib.ox.ac.uk/fsl/].

    Article  Google Scholar 

  • Stone, M (1973). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society B, 36(2), 111–147.

    Google Scholar 

  • Strother, S. C. (2006). Evaluating fMRI preprocessing pipelines. IEEE Engineering in Medicine and Biology Magazine, 25(2), 27–41.

    Article  PubMed  Google Scholar 

  • Strother, S.C., Lange, N., Anderson, J.R., Schaper, K.A., Rehm, K., Hansen, L.K., et al. (1997). Activation pattern reproducibility: measuring the effects of group size and data analysis models. Human Brain Mapping, 5, 312–316.

    Article  Google Scholar 

  • Strother, S. C., Anderson, J., Hansen, L. K., Kjems, U., Kustra, R., Sidtis, J., et al. (2002). The quantitative evaluation of functional neuroimaging experiments: The NPAIRS data analysis framework. NeuroImage, 15(4), 747–771.

    Article  PubMed  Google Scholar 

  • Strother, S. C., LaConte, S., Hansen, L. K., Anderson, J., Zhang, J., Pulapura, S., et al. (2004). Optimizing the fMRI data-processing pipeline using prediction and reproducibility performance metrics: I. A preliminary group analysis. NeuroImage, 23(1), 196–207.

    Article  Google Scholar 

  • Strother, S. C., Liow, J.-S., Moeller, J. R., Sidtis, J. J., Dhawan, V., & Rottenberg, D. A. (1991). Absolute quantitation in neurological PET: Do we need it? Journal of Cerebral Blood Flow and Metabolism, 11, 3–16.

    Google Scholar 

  • Tanabe, J., Miller, D., Tregellas, J., Freedman, R., & Meyer, F. G. (2002). Comparison of detrending methods for optimal fMRI preprocessing. NeuroImage, 15, 902–907.

    Article  PubMed  Google Scholar 

  • Tegeler, C., Strother, S., Anderson, J., & Kim, S. (1999). Reproducibility of BOLD-based functional MRI obtained at 4 T. Human Brain Mapping, 7(4), 267–283.

    Article  PubMed  CAS  Google Scholar 

  • Wang, X., Hutchinson, R., & Mitchell, T. (2003). Training fMRI classifiers to discriminate cognitive states across multiple subjects, The 17th Annual Conference on Neural Information Processing Systems.

  • Winterer, G., Ziller, M., Klöppel, B., Heinz, A., Schmidt, L. G., & Herrmann, W. M. (1998). Analysis of quantitative EEG with artificial neural networks and discriminant analysis—A methodological comparison. Neuropsychobiology, 37, 41–48.

    Article  PubMed  CAS  Google Scholar 

  • Witten, I. H., & Frank, E. (2000). Data mining: Practical machine learning tools and techniques with JAVA implementations (pp. 10–35). Morgan Kaufmann Publishers.

  • Woods, R. P., Grafton, S. T., Holmes, C. J., Cherry, S. R., & Mazziotta, J. C. (1998a). Automated image registration: I. General methods and intrasubject, intramodality validation. Journal of Computer Assisted Tomography, 22(1), 139–152.

    Article  PubMed  CAS  Google Scholar 

  • Woods, R. P., Grafton, S. T., Watson, J. D., Sicotte, N. L., & Mazziotta, J. C. (1998b). Automated image registration: II. Intersubject validation of linear and nonlinear models. Journal of Computer Assisted Tomography, 22(1), 153–165.

    Article  PubMed  CAS  Google Scholar 

  • Zijdenbos, A. P., Forghani, R., & Evans, A. C. (2002). Automatic “pipeline” analysis of 3-D MRI data for clinical trials: Application to multiple sclerosis. IEEE Transactions on Medical Imaging, 21(10), 1280–1291.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We thank the Fiswidgets group led by Kate Fissell for their technical support, cooperation and various help. We are also grateful to James Ashe, M.D. and Suraj Muley, M.D. for providing the static-force data, and to Kelly Rehm, Kirt Schaper for technical assistance. This work was partly supported by the NIH Human Brain Project P20 Grant MN EB002013.

Author information

Authors and Affiliations

  1. Health Informatics Graduate Program, University of Minnesota, Minneapolis, MN, 55455, USA

    Jing Zhang, Lael Gatewood & Stephen C. Strother

  2. Department of Electrical Engineering, University of Minnesota, Minneapolis, MN, 55455, USA

    Lichen Liang

  3. Department of Neurology, University of Minnesota, Minneapolis, MN, 55455, USA

    Jon R. Anderson, David A. Rottenberg & Stephen C. Strother

  4. Rotman Institute, Baycrest and University of Toronto, Toronto, ON, M6A2E1, Canada

    Stephen C. Strother

  5. Mt. Sinai Medical Center, New York, NY, 10029, USA

    Jing Zhang

Authors
  1. Jing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lichen Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jon R. Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lael Gatewood
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David A. Rottenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Stephen C. Strother
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Zhang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, J., Liang, L., Anderson, J.R. et al. A Java-based fMRI Processing Pipeline Evaluation System for Assessment of Univariate General Linear Model and Multivariate Canonical Variate Analysis-based Pipelines. Neuroinform 6, 123–134 (2008). https://doi.org/10.1007/s12021-008-9014-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-008-9014-1

Keywords

Search

Navigation