Evaluation of Two CD-ROMs from a Series on Cell Biology

INTRODUCTION

An increasing number of multimedia programs provide valuable resources for science education in molecular and cell biology. Many textbooks are accompanied by software, and numerous stand-alone products are available. However, little research has been done on the impact of educational CD-ROMs and how these resources affect teaching and learning in molecular and cell biology. One reason for this is that far-reaching evaluations of such multimedia programs is difficult to achieve, since learning with multimedia often takes place as individual study, e.g., at home, independently of instructors or fellow learners. This is considerably different from traditional learning, which can be observed and evaluated directly. Traditional evaluation methods such as observation and group discussion may be adequate for the evaluation of group-based instruction, but they often may be inappropriate or inadequate with regard to expenditure of effort and time for the evaluation of interactive learning programs. Traditional group-based instruction works best when preconditions in the learner group are distributed homogeneously, whereas computer-based learning does not have this restriction. What is learned and when, how long, and how often the learning medium is used are controlled by the learner if not explicitly regulated otherwise. These heterogeneous starting situations for multimedia learners automatically lead to different user expectations. Since media-supported learning provides a variety of access corridors for learners, it likewise allows a variety of learner intentions and motivations. This variance in learners' motivation affects learning achievements and, even more fundamentally, the mere definition of what learning achievement is. In traditional learning, goals (exam, certificate) are usually predefined. In media-supported learning, this does not have to be the case. Motivation may range from sporadically seeking individual interesting pieces of information to absolving a complete course including examinations (Baumgartner, 1999).

In this report, two CD-ROMs from a series dealing with various major aspects of cell biology are evaluated. What lessons can be learned by comparing them? The objectives of the CD-ROM series project have been published elsewhere (Siebert et al., 2000) and are briefly summarized here. The CD-ROM series was targeted at students of secondary schools, colleges, and introductory courses at universities, as well as at private users with a scientific background. In the CD-ROM series, navigable three-dimensional objects, microscopic shots of the interior of cells, and experiments to visualize cellular activities are to be combined and used interactively. The CD-ROM series is intended to allow explorative and discovery learning. Additionally, it should support autonomous learning, helping students to acquire knowledge and review subject matter. Access to the contents should be as easy as possible; therefore, two kinds of navigation were implemented—first, a site map that allows a graphical overview of the entire contents of the program and, second, context-sensitive links to related content screens. As the individual content screens were not structured hierarchically, a”next” button was omitted. This should underline the explorative and active character of the CD-ROM. A”back” function to return to the last screen, however, was included. The navigation was altered and improved in CD-ROM 2 (see CD-ROM 2, below). A clear and professional graphic design was considered to be of major importance for an easy and intuitive usability of very differently organized screens; a combination of symbols and realistic representation was to be used as a design principle. Additional on-screen text was to be dispensed with. A high level of interactivity was given major emphasis; reactive interactivity, instead of simulations, was to be used to allow nonambiguous feedback to reactions of the users. Table 1 gives an overview of the main objectives.

DESCRIPTION OF THE CD-ROM PROJECT

The CD-ROM series on cell biology (see www.cells.de) has been developed by the German IWF Wissen und Medien gGmbH (see www.iwf.de) as a nonprofit activity. The IWF is an educational institution that has an assignment in higher education, i.e., it provides audiovisual media for higher education. From 1998 to 2002, in cooperation with the Bio-Center at the University of Frankfurt and two private companies, the IWF compiled a comprehensive media archive on cell biology. This project's objective was the progressive preparation and application of an interactive knowledge representation in cell biology—a modular media service (for details see Sander et al., 2002). The CD-ROMs are one major spinoff of this project. The four-part Cell CD-ROM Series combines learning objects on individual subjects:

1. Life from Light and Air—Chloroplast and Photosynthesis (Bereiter-Hahn and Peters, 1999)

2. The Power Plant—Mitochondrion and Energy Metabolism (Bereiter-Hahn and Peters, 2001)

3. Internal Boundaries—Membranes and Transport (Bereiter-Hahn and Peters, 2002a)

4. The Cell Nucleus—From Gene to Protein (in preparation)

Each CD-ROM provides a virtual virtual laboratory (see Figure 1) in which experiments can be planned and carried out. All the films screened in the learning program are listed and can be accessed individually. The success of knowledge acquisition can be checked by taking a multiple-choice quiz. A site map index allows a graphical overview of the entire contents of the program. It displays a complete list of chapters and subjects. There is also an abundance of supplementary information, in the form of texts, images, graphics, films, and virtual animations, to provide a comprehensive theoretical background to each topic. Help texts round out the offering. The content of each CD-ROM is structured in three chapters, namely,“ Function,”“Structure,” and“ Development.”

CD-ROM 1. “The Cell I—Life from Light and Air—Chloroplast and Photosynthesis”

The “Function” chapter presents a comprehensive overview of the biochemical processes involved in photosynthesis. A further interactive option is to simulate biochemical reactions in order to reinforce what has been learned. The “Structure” chapter deals with the fine structure of the chloroplast. It makes use of numerous 3-D animations and a virtual flight through a chloroplast (see Figure 2). The chapter is rounded off with electron micrographs of a chloroplast in which it is possible to enlarge and highlight certain details. The origin development of chloroplasts is dealt with in the chapter“ Development.” As in the other chapters, learners become actively involved in the program. The individual cell organelles (e.g., nucleus, cell skeleton, microbodies, mitochondria, vacuole, endoplasmatic reticulum, and Golgi apparatus) are illustrated using text and visuals. A description of this CD-ROM is given in a laudatory speech:

CD-ROM 2. “The Cell II—The Power Plant—Mitochondrion and Energy Metabolism”

This second CD-ROM in the series on cell biology focuses on mitochondria. These are the organelles where the most important energy-providing catabolic processes take place. Mitochondria, the power plants of the cell, are found in both animal and plant cells. In addition to 3-D models and computer animations, photographs and videos (see Figure 3) provide information on structural and functional characteristics while employing special electron, light, and laser-scan microscopic techniques to aid in comprehension and add depth to the presentation. Detailed interactive animations explain the metabolic pathways in cellular respiration, e.g., glycolysis, fermentation, the respiratory chain, and the tricarboxylic acid cycle; interactive laboratory experiments ascertain how the respiratory activity of mitochondria is influenced by different parameters. The latest scientific findings have been considered, for example, with regard to the function of the ATP synthase. A chapter on reproduction includes the topics division and fusion of mitochondria, mitochondrial DNA, and import of proteins into the organelle. The CD-ROM also provides a quiz with approximately 100 questions and four interactive videos, an index and a help function, a printing function and a glossary.

Differences in Navigational Design between the two CD-ROMs

Differences in the navigational design of the three CD-ROMs are considered briefly here, as they have some relevance on the evaluation results. A graphical chart index—similar to a Web site's sitemap—is implemented on both CD-ROMs. It is termed “Index” on CD-ROM 1 and“ Compass” on CD-ROM 2. On CD-ROM 1, two clicks are generally needed to open this index, compared to the single click required on CD-ROM 2 (see Figure 4). However, an alphabetical index was implemented only on CD-ROM 2. Moreover, CD-ROM 2 provides a didactically appropriate linear tour through coherent parts of the contents. Additionally, CD-ROM 2 provides a simple personalization tool, a notepad, allowing users to write down their remarks.

MATERIALS AND METHODS

Qualitative and quantitative analyses were used for the evaluation of the two CD-ROMs. CD-ROM 1 was evaluated through the qualitative analysis of available material from users and evaluators. Thus, we expected to become oriented with regard to users of the CD-ROM series and their opinions, as well as to obtain information on the perceived quality of design of the work and its possible applications in the learning and teaching context. For CD-ROM 2, which was available for testing 2 years after CD-ROM 1, the evaluation framework was narrowed to obtain more specific results. In addition to qualitative analysis, quantitative analysis with questionnaires within defined target groups as well as a log file analysis was performed.

When setting up the design of the study, we contemplated using a comparison of different learning groups, e.g., a comparison of students learning with the CD-ROM versus students using “traditional” material like a textbook. However, as several publications have stressed the difficulty of such an approach, we decided not to include this approach into our current investigation. Mayer (1997) points out that “there are serious methodological confounds in comparing two media.” He stresses that “it is not possible to determine whether media effects are attributable to differences between computer-based and book-based delivery systems or to differences in the content and study conditions of the lessons.” Additionally, the quality of the instructional design of textbooks varies and therefore leads to the question of which textbook should be chosen to be compared with the CD-ROM. Mayer (1997) highlights this problem: “It is possible to produce effective and ineffective instruction in both computer-based and book-based media. ...”

Evaluation of CD-ROM 1

Qualitative material was available in the form of seven reports on the CD-ROMs' use in classroom instruction, three reviews in trade journals, and two recognitions at award presentations. The seven reports on the CD-ROMs used in classroom instruction are based on observations made by teachers and students in the classroom while actually using or testing the multimedia software. The recognitions at award presentations were based on defined criteria from specialized educational institutions that had been applied when evaluating the programs, usually using rating and ranking methods. The reviews in trade journals are based on the subjective impressions of the reviewers. An assessment of these different kinds of evaluation approaches was made by Baumgartner (1997). We realize that the quality of this material is very heterogeneous, and conclusions should be drawn from it only with greatest care. We did not aspire to obtain quantitative results, such as those achieved by empirical studies, with this approach, but we hoped to be able to detect trends.

Such material was taken into consideration only when a clear opinion was expressed in it concerning one of the CD-ROMs in the series; mere descriptions without any expression of an author's own viewpoint were omitted. Reviews, recognitions at award presentations, letters, e-mails, and four of the reports contained no more than two pages of text each, whereas Schürholz's (2000) report had 30 pages and Durst's (1999), text 5 pages. Grube's (2000) text was counted as four individual reports, because statements from four different evaluation groups (teachers, students, interns, and didactical experts) could clearly be distinguished in it.

The following five categories were derived from the material using the method described by Bortz and Döring (2002). The category“ quality of media” embraces the conceivable quality of films, graphics, and images. “Content” states whether the authors find the presented content to be scientifically correct. “Use of multimedia” considers whether multimedia features, such as 3-D objects, virtual laboratories, and interactive animations, have been satisfactorily integrated into the CD-ROM. “Didactical approach” assesses whether a satisfactorily didactical approach has been achieved.“ Navigation” deals with the ease of finding relevant content and maintaining orientation when browsing in the program.

The material was input into a database and encoded by assigning passages of the text to these categories. The frequency of occurrence of certain categories was taken as a basis for the verification of the given hypothesis. The general rating of the CD-ROMs was assessed using a scale ranging from +2 for very good to –2 for very bad. All explicit and implicit statements about the quality and usefulness of the product were taken into account. Three researchers (Gertraude Kerlen, Ulrich Roters, and Uwe Sander) examined the encoded passages and results carefully and critically to verify the interpretations.

Evaluation of CD-ROM 2

For practical reasons, we decided to restrict the the students' time for working with the CD-ROMs to only 40 min. Expansion of the working sessions would have lead to breaks, as each school period lasts only 45 min. These breaks could have lead to discussions between students concerning the software. These exchanges of opinions between students might have had an impact on student feedback and thus interfered with the design of the study.

Quantitative Evaluation. A total of 464 students participated in the evaluation program. They were from German secondary-school biology courses (classes 11 to 13, ages 16 to 20). All students participated in pretests to determine prior computer literacy and prior biological knowledge as described by Huk et al. (2003). After two consecutive 20-min working sessions with the CD-ROMs, the students could then state their opinion by filling in a questionnaire (see Figure 5) and by adding comments. Differences in opinions were analyzed with binomial tests. Impact of pretest results on impression was analyzed using nonparametric Mann–Whitney U tests because the students scores in pretests were not normally distributed and hence did not allow application of parametric tests (e.g., ANOVA, t-test).

Qualitative Evaluation. In addition to the quantitative analysis of the questionnaire, written comments from the 464 participating students were categorized to obtain new insights. The evaluation of CD-ROM 1 focused on the observations and impressions of teachers and specialists in the field of either media or biology, whereas we were particularly interested in the students and learners' perspective in the survey of CD-ROM 2. The comments were categorized in a manner similar to the five categories used in the evaluation of CD-ROM 1 as described above (see Evaluation of CD-ROM 1). However, the category “content” was omitted, as school students were not expected to be able to consistently assess the scientific correctness of CD-ROM 2.

Log File Analysis. Furthermore, log files were recorded when the CD-ROM was used. They allowed the navigational behavior of users in the survey to be tracked. Log files were extracted that showed which navigational element of the start page (see Figure 6) was chosen: the guided tour, the explorative tour, or the compass (sitemap). We included only log files of students who worked with the software on their own (N = 284), as opposed to working in groups.

EVALUATION RESULTS

CD-ROM 1

Eight of the 14 statements containing opinions about the five derived categories stem from authors who have a background in secondary education; five statements can be assigned to those having a college and university background. Table 2 shows the number of statements that contain opinions about the five derived categories. Eight statements are found in the category “quality of media.” Typical examples are cited to illustrate the quality of remarks: “The quality of pictures and videos is of the highest standard,”“ Graphics and short footage are abundant and very impressive,”“ high standard of technical realisation,” “excellent pictures and films,” and “Graphically it is very impressive and each image gives the impression of being very thoroughly prepared.” All statements are unambiguously positive and, thus, give testimony about the perceived high quality of media integrated into the CD-ROM series.

For the most part, the scientific content of CD-ROM 1 is also valued highly: “expertise from scientific research,” “excellently prepared specialist information,” “comprehensive content,”“ exhaustive information,” and “Exemplary design with regard to content” are some of the remarks obtained. However, there are two critical statements: “Less impressive are the presentations of the scheme sequences of the photosynthetic electron transport, or the Calvin–Benson cycle” and “This depiction is principally wrong!” Both statements are from cell biologists and concentrate on details of the presented content that are perceived to be incorrect. There are, however, three more statements that are not included in Table 2, as they are not about scientific correctness, but that nevertheless have some relevance in the context. These statements criticize the inhomogeneous levels of information presented in the CD-ROMs, e.g., “Some aspects of this CD-ROM are very sophisticated (e.g., mentioning the different steps of the Calvin Benson cycle); whereas others are too simplistic (e.g., the interactive quiz with sound approving the correct answer).” and “The level of difficulty in the different parts of the CD-ROM is too heterogeneous. Some components have a level of difficulty suitable for grade 7 or 8, whereas others are appropriate for secondary school.”

Four remarks offer a positive opinion about the use of multimedia in the CD-ROM 1, e.g., “It offers what multimedia can achieve today,”“ excellent integration of new means of multimedia,” and “The multimedia design is diversified and inventive.” Only one author states that there are “too many multimedia `gimmicks.'”

The didactical approach is discussed controversially in the available texts.

The navigation on CD-ROM 1 is perceived critically: “User guidance is extremely unclear. After a short time one has `lost one's way.'”“ Finding one's way through the program is difficult, because one cannot retain an overview.” “User guidance is hardly appropriate, because it is unclear, too playful and does not correspond with the Internet standard.” “The CD-ROM has an archive of available films, but the individual films cannot be accessed directly.” “When surfing a CD-ROM, I have often wondered whether I have covered it all, or whether I have missed important sections; however, in this case a very well arranged index gives the user an excellent overview of which layer he is presently in.”

Four of the 12 statements about the CD-ROM 1 are interpreted to contain a negative overall judgment, while 7 are conceived to express a positive overall opinion. Statements about other aspects of the CD-ROM series are not taken into account in this survey, as their frequency is too low to allow valid results.

An additional source of information about the CD-ROM 1's use is sales numbers, as the program is distributed commercially. In Germany, three publishers, including IWF, market the CD-ROM. Foreign-language versions are distributed by publishers in Italy, Denmark, The Netherlands, Slovenia, and Brasil. About 34,00 German and 16,000 foreign-language versions of the CD-ROM have been sold. German publishers report that biology teachers in secondary school are among the main customers. However, these publishers were not able to provide us with differentiated data about their customer base.

CD-ROM 2

Quantitative Evaluation. The tested students' assessment of the six statements about CD-ROM 2 (see Figure 5) varied considerably, ranging from “I agree” (2 points, maximum) to “I disagree” (–2 points; minimum) or—as in statement 6—to“ I disagree more or less” (–1 point; see Figure 5). However, only a minority had a negative opinion of the CD-ROM, whereas the majority of the students declared themselves to be “neutral” or to “agree more or less” with regard to the statements presented (Figure 7). The number of students with positive assessments (“I agree” and “I more or less agree”) significantly exceeded the number of students with negative assessments (“I disagree” and “I more or less disagree”) with reference to the statements that the information on the CD-ROM is easily found and that the navigation is clearly structured (Statements 1 and 2; p < .001). A significant majority of the students liked the design of the CD-ROM (Statement 3; p < .001; 353 positive versus 29 negative assessments), considered the software to be a suitable tool for preparation for examinations (Statement 5; p < .001; 223 positive versus 35 negative assessments), and felt that the software is a suitable complement to or extension of school lessons (Statement 6; p< .001; 336 positive versus 35 negative assessments). A total of 173 students would like to work with the CD-ROM again, 148 were undecided, and 131 would not consider further usage. As in the statements mentioned above, the number of positive assessments significantly exceeded the number of negative ones (Statement 4; p < .019). Taken together, this analysis indicates that students appreciate the CD-ROM's design and navigation features. They find the CD-ROM to be a suitable complement to or extension of their lessons and the majority is convinced that the CD-ROM is suitable for preparation for examinations and tests of their knowledge. However, correlation of pretest results revealed that computer literacy as well as prior biological knowledge had a significant impact on whether students agreed or did not agree to use the CD-ROM in future (see Figure 8). Students who agreed more or less (1 point and 2 points on the rating scale) with Statement 4 had higher computer literacy (Mann–Whitney U test, p = .043) and enhanced prior biological knowledge (p = .023) compared to students who disagreed with this statement (–1 point and –2 points on the rating scale).

Qualitative Evaluation. The qualitative analysis of the students' comments written on the questionnaire provided further information. Some of the categorized statements are listed in Table 3. The high quality of media modules is emphasized by 29 students. A few comments are cited here:“ ... The animations make the CD-ROM understandable ...”; “The animations, films and graphics are all very good”; and “I find the experiments and the 3-D images very appealing”. A total of 27 students point out that the use of multimedia is appropriate. They state, for example, “The use of different forms of imparting information is positive” and “As a result of the use of visuals and animations, the CD is easily understood and motivates people who previously had no desire to learn to now do so.” In contrast, 14 students ventured negative comments about media design or multimedia use. One major point here was the students' demand for more interactivity within the media modules, for example: “... have to listen too much. Being able to interact more frequently is fun and thus facilitates learning.”

Among the statements about navigation, there are a few comments about the compass as a sitemap and navigation tool. Three people stated that the compass is helpful when seeking navigation, while another three do not consider the compass to be clearly structured. The necessity to become accustomed to the CD before obtaining orientation is underlined by the comments of 14 participants, who claimed that it takes its time to find relevant information, for example,“ Until one has achieved an overview and has found the compass, it is somewhat difficult to find what one is searching for.” Five students wrote about becoming easily distracted by irrelevant content not related to the tasks given to them in the survey, especially at the beginning of a work session. Another 14 students mention that they felt disoriented when working with the CD.

Log File Analysis.Table 4 lists differences in the use of the compass (sitemap) between the first and the second work session with the CD. During the second session, the compass is used significantly more often. In the first session, 23.6% of the users click on the compass; in the second session, 43.5%.

In addition to these evaluation results, we have added a statement from the Digita 2000 award laudation speech about CD-ROM 2:

DISCUSSION

Evaluation of CD-ROM 1

All statements about the quality of media are unambiguously positive, thus giving testimony to the perceived high quality of media integrated in the CD-ROM series. Additionally, a positive opinion about the use of multimedia in the CD-ROMs predominates. This is an indication that the technical aspects of the CD-ROMs are valued highly by users and evaluators. The scientific content is also mostly valued highly; however, there are statements that the level of difficulty in the different parts of the CD-ROM is too heterogeneous. The reason for this is that the authors of the CD-ROM wanted to present the topics completely and not just provide fragments of them as is often the case in a school curriculum. Areas such as growth, cell structure and functioning, cell division, gamete formation, sexual and asexual reproduction, inheritance, and genetics are often taught at different points in the school year—or even in different years. It is therefore hard for students to see the coherent thread that unites these areas. This leads to problems in understanding concepts in cell and molecular biology; this problem extends even to biology specialists at the undergraduate level (Lewis and Leach, 2000; Wood-Robinson et al., 2000). The didactical approach is discussed controversially in the available sources. The critical statements are from evaluators who tested the CD-ROM 1 in secondary school (Grube, 2000; Schürholz, 2000). The lack of appropriate worksheets is criticized, the character of a course of studies should be introduced, and the CD-ROMs should not be designed as “edutainment.” These statements suggest that in these cases the CD-ROM could not be integrated into the learning process as easily as had been hoped. The navigation in CD-ROM 1 is perceived critically, whereas only a few negative statements were made about CD-ROM 2. As stated above, on many screens on CD-ROM 1 two clicks are needed to open the graphical chart index. Additionally, no alphabetical index or didactically appropriate linear tour through the content exists. These shortcomings may be the reasons for the negative evaluation of CD-ROM 1's navigational design. Navigation was substantially improved in the subsequent CD-ROM.

Evaluation of CD-ROM 2

The students' rating of CD-ROM 2's design and navigation is mainly positive both in the quantitative and in the qualitative evaluation. In particular, the graphical quality of the whole CD-ROM is rated very well by the students on the questionnaire, and indeed this is emphasized by many students' comments about the high quality of the media modules.

Navigation. In the questionnaire and the comments, the navigation also receives a positive student rating. The compass implemented as a sitemap and central navigational tool seems to fulfill this task. The results when analyzing log files and the clicks on the starting page reveal a significant increase in the use of the compass in the second work session with the CD-ROM 2. However, their comments reveal that there were some students who felt disoriented and frustrated in their search for relevant information. These effects could be a result of the disorientation (Conklin, 1987) that occasionally occurs when learning with hypermedia and/or the serendipity effect (Kuhlen, 1991). The latter can lead to distraction of users in hypermedia environments as the result of the abundance of information. This may result in their not pursuing their original goal further. On the other hand, however, this effect may support explorative behavior and learning (Mayes et al., 1990; Kuhlen, 1991).

As described above (Differences in Navigational Design between the Two CD-ROMs), the access to the compass has been facilitated on the second CD-ROM, and an alphabetical index and a guided tour have been additionally implemented. Nevertheless, as revealed by the students' comments, a few of them still miss guidance through the program. The reason might be a usability problem, i.e., when users choose not to click on the “guided tour” button on the starting screen, there is subsequently only a small button—labeled “tours”—in the upper-right corner of the compass available to access the guided tour. It might have been preferable to have positioned this button clearly and always visible in the main CD-ROM navigation area, the test tube. Such an increased variation of navigation tools and access possibilities might then lead to even better acceptance by the users. Various studies reviewed by Chen and Macredie (2002) recommend taking different learning styles into consideration when designing navigation support for hypermedia systems. An alphabetical index may best support learners who tend to be analytical and task oriented. A sitemap may better support learners who prefer to process information in a global fashion. Didactically appropriate linear tours might facilitate learning processes for users who are externally directed and who, at least in the initial phase of a learning session, prefer guided learning. A study on the relationship of learner characteristics, software design, and navigation behavior is further described by Steinke et al., (2003).

Didactical Approach and Content. Despite their positive assessment of the CD-ROMs design and navigation, 14.4% of the students were not sure if they would use the CD-ROM in the future, and 14.6% even rejected possible future use. Only 50% agreed that the CD-ROM is suitable for the preparation for examination and comprehension tests. One reason for this might be that the CD-ROM was developed for a broad target group, including students in secondary schools, colleges, and universities in different countries. Additionally, the CD-ROMs were not intended to be a course replacement, but as a supplementary learning tool. Therefore, the content does not exactly fit into the curriculum of the German secondary schools that participated in the survey. Nevertheless, many students—about 73%—found the CD-ROM to be suitable as a supplement for school lessons.

The comments noted on the questionnaire may indicate reasons why many students did not find the CD-ROM suitable for test preparation but would like to use it as a supplement. A large number (about 27%) of students criticized the fact that too much jargon was used and that there were not enough explanations in the narration. Some of the students mentioned that even with the help of the built-in glossary, it was difficult to understand some of the content modules, for example, “The glossary is very good, but new subject matter is explained with new terms in the films; the net result is that one tends not to understand the information as a whole.” Five students wished that the glossary had contained more words. Despite the high-quality module design it still appears to be difficult for students to fully understand the content when working with the CD-ROM at school within a limited time frame. But even when working at home it might be tiring to grasp narration with a lot of specialist terms. For easier use and a better integration of the CD-ROM into school lessons, the integration of appropriate worksheets as claimed by Schüurholz (2000) might be a valuable asset. The quiz proved to be a learning tool frequently used by the students. This was observed in the work sessions. Eight students mention in their comments that the quiz is a good thing. One way of improving the quiz would be to display the correct answer directly after a wrong choice has been made or to provide a link to the appropriate passage. Six students mentioned this in their comments; others said so during the working session.

Another important thing we noticed when analyzing the comments is the fact that about 10% (46) of the students missed written on-screen text accompanying the oral and visual presentation. Twelve students mentioned that they prefer to learn with written text or books. Another six mentioned their wish to print out the screen contents or write on-screen when learning with the CD. However, CD-ROM 2 does support the option of printing out text and graphics and making on-screen notes using a “notepad.” But only five students commented that the notepad—a newly introduced tool on CD-ROM 2—was useful for them. This discrepancy between the availability and the actual use of multimedia tools indicates that students do not take advantage of the whole range of options when working with hypermedia. Since most students are still primarily accustomed to working with books and written documents, it is not surprising that they experience difficulties working in a goal-oriented manner with learning material that is not text-oriented. Exercises on using new media might be necessary to improve hypermedia literacy.

Conclusions

Two CD-ROMs from a series on cell biology were evaluated using different approaches, including qualitative, quantitative, and log file analyses, making a direct quantitative comparison difficult. Nevertheless, there are some findings that depict similarities and differences between the two CD-ROMs and shed light on how the programs could be used in the learning process and how not to do so. For both programs, the overall impression is predominantly good. This is also true for graphical and technical issues. CD-ROM 1 is criticized for navigational problems. New navigational functions have been successfully implemented on CD-ROM 2. Thus, the usability of at least CD-ROM 2 has been demonstrated. However, when considering how the programs were used in the learning process the results are more ambiguous. Most students found CD-ROM 2 to be suitable as a supplement for school lessons, but a minority of students were not sure that, or even did not think that, the CD-ROMs were suitable for preparation for examinations and tests of knowledge. Students with good computer literacy and above average prior biological knowledge found the CD-ROMs more useful than others. Overall, we conclude that the main goals of the project have been achieved. Defined target groups were reached (e.g., students in secondary schools), different learning approaches could be supported (e.g., discovery and autonomous learning), the content was easy to access (on CD-ROM 2), the usability was appreciated as easy and intuitive, and most evaluators were satisfied with the level of interactivity.

Having now analyzed some of the demands of the educational community on interactive learning software that came up in our survey, and having compared them with the features offered in our programs, we find heterogeneous and even contradictory expectations. There is a need for self-paced and discovery learning in a learner-centered approach, as opposed to having course structure and worksheets and focusing on preparation for examinations and knowledge tests in a more teacher-centered process (Baumgartner, 2001). Clearly, not all the requirements for the various stages of the learning process can be satisfied with the CD-ROM series. Instead of revising the existing CD-ROMs to accommodate the understandable wishes of the evaluators, we decided to design and evaluate a new product, based on the existing one, the Cell Media Archive (Bereiter-Hahn and Peters, 2002b). Further research is planned to evaluate this new product and compare it with the existing CD-ROMS.

Our study began by collecting and analyzing available anecdotal reviews about our CD-ROMs. This method of qualitative evaluation, used quite frequently in social science (Bortz and Döring, 2002), proved also to be very helpful in the evaluation of our educational material. It allowed us to quickly obtain a rough idea of the strong and weak points of these multimedia products. It immediately led to improvements in subsequent CD-ROMs, at the same time influencing the design of a more sophisticated quantitative evaluation. Anecdotal information also played an important role when discussing the results of the quantitative analysis of questionnaires and log files. The comments of students that filled out the questionnaires were helpful in obtaining further information. They informed us about unfulfilled requirements and about problems using the multimedia product we had not considered previously. Consequently, we believe the combination of qualitative and quantitative evaluation to be an effective way to evaluate multimedia learning material.

FOOTNOTES

ACKNOWLEDGMENTS