Complete information on Selection Criteria for PI & CAI

The joint Committee of Programmed Instruction and Teaching Machines in 1963 recommended distinguishing between internal characteristics of the PI program (which can be judged by an inspection of the program) and external characteristics (features which are only revealed by the performance of users after the completion of the program or by objective features). Many of the same selection criteria can be used when evaluating CAI programs or designing web-based courses.

Internal:

Is the subject matter appropriate for, or needed by, the intended audience? Are the objectives of the program, if stated, appropriate for or needed by the intended audience? Are they relevant to some real-world need or prerequisite to further study? Is the sequence of frames consistent with the structure of the subject-matter content? Is the sequence of frames appropriate for the type of users in the intended audience? Is the step size appropriate for the ability of the users?

Are the responses opportunities frequent enough for the type of users or the structure of the content? When a response is required, does feedback follow at some appropriate point, not too delayed from the response? Are there any provisions for individualizing the sequence or content of the program (e.g., branching, diagnostic tests, surveys) or is every user required to proceed through each step? Is an instructor’s guide available that contains recommendations for administration, placement, or integration with other study materials? Is a posttest available which measures attainment of the objectives?

External:

Is there evidence of any testing or user tryout of the program prior to distribution? Was the group used to field-test the program similar to the group for which you are selecting a program, in terms of age, ability, computer experience, and prior knowledge? Did the program, as tested in the tryout, produce a sufficient level of learning? The National Education Association cautioned that an internal inspection alone can be deceptive. External data from tests involving users at your institution or a similar group should be included in the evaluation process. The users should represent, as closely as possible, the intended population of users.

Other Criteria:

Are the directions on how to use the program clear, adequate, and appropriate for the target users? Are the technical terms and symbols adequately defined? Does the content cover the stated objectives without unnecessary redundancy and superfluous information? (Some redundancy for the purpose of practice is a normal Part of PI.) Does the program periodically review what has been taught most recently throughout the program? Does the achievement test measure the user’s comprehension of the content of the program as opposed to mere recall of facts? Does the program allow for individual differences in learning other than pacing, such as prior knowledge or background, ability level, and learning styles? Are active responses required of the user? Do they require comprehension of the material, rather than mere recall or copying of information?

Are the responses an appropriate form for the users and relevant to the objectives? Are there enough responses for each skill or unit of information? Do the questions require the user to utilize the critical information in the frame, or to recall only trivial facts? Are the prompts necessary for making the correct response, or do they replace the thought processes in which you want users to engage? Are there too many prompts? Are enough questions asked to allow the user to apply the information to a variety of situations? Will the skills taught in the program transfer to a real-world setting? Are there at least one question requiring users to respond to each important piece of information or skill? Is the feedback accurate and clear to the user? It is surprising how often this simple criterion is violated.

Does it immediately follow the response? Can the feedback be seen by the user while responding? Are alternative answers to questions acceptable and are the acceptable responses indicated? Are helpful, remedial comments provided when wrong or alternative answers are anticipated? Do the instructions allow the user to skip unnecessary repetition or to return to material when necessary? If a non-print medium is used, is it an appropriate format for the content of the program? Could the program be more effectively delivered in another format?

Is the delivery mode (machine/text, CAI, web-based) or control of pacing (self-paced, programmed, or group- paced) appropriate for the maturity level and ability of the intended users? Are there a sufficient number of review questions and feedback to keep all parts of the program current and active in memory?

Research is underway on standards for web-based instruction in higher education. There are two models for designing interactive educational programs. The first, instructional systems design (ISD), the traditional model, determines a goal, sets objectives, delivers instruction, formulates test questions, and evaluates learning. The second, hypermedia design (HDM) focuses on the student’s goal and how the student chooses to access information. While ISD is concerned with design goals, HDM focuses on the user’s goals. Selection of CAI or web- based design should be based on whether the program is well designed and meets the needs of the intended users.

There are many design models for CAI available today. One model, developed in 1994, consisted of seven phases:

1. Determination Of the need and situation analysis

2. Formulation of aims and performance objectives and development of items for evaluation

3. Design of study material, including development of a teaching strategy and media selection and integration (e.g. the inclusion of sound and video)

4. Development and preparation (this includes story boarding and programming)

5. Implementation and use

6. Assessment of student progress

7. Formative and summative evaluation on a continuous basis throughout all phases.

After each phase is completed, it must be evaluated before moving on to the next phase. CAI design projects should consist of several members including a project manager, subject experts, advisors, evaluators, programmers, and graphic artists. The CAI must meet the needs of its users to be effective. Also, computer literacy can be a major problem. Students without technology skills will have to master basic computer knowledge before using the CAI successfully.

Good web instruction asks students to interact in some way and not just to memorize information. It must be flexible and allow for differences in learning abilities. It should encourage deep learning and not merely surface learning. Students must understand concepts and how they fit into the whole, be able to integrate parts, apply the information practically, and receive feedback. It is essential that web instruction provides opportunities for interactivity to make it meaningful for the student.