MultiMedia developMent
and evaluation
Mechelle D. Tumanda( MAIT)
-Presenter -
Multimedia
comes from the word
“multi” and “media”.“
Multi” means various
“Media” refers to any hardware
or software used for
communicating.
Multimedia represent the consolidation of all
elements of technology as they combine sound,
image, video, drawing & text with a high quality in
addition to the interactive environment.
CURRENT DEFINITION OF MULTIMEDIA IN ICT
In the field of Information and Communication Technology, multimedia means more
than the use of the various media. A computer user interacts with the computer to
perform tasks such as finding information or play games to develop a skill.
Thus, the meaning of multimedia has changed as technology advanced in our lives.
MULTIMEDIA
TODAY
Multimedia and its related
applications have almost
become synonymous with
modern technology; given
the kind of explosion the
technological realm has
seem.
Multimedia makes our life
easier several times fold. It
is through multimedia that
mobile phones can be
used for a number of
purposes.
Multimedia and its related
applications have almost
become synonymous with
modern technology; given
the kind of explosion the
technological realm has
seem.
Multimedia makes our life
easier several times fold. It
is through multimedia that
mobile phones can be
used for a number of
purposes.
USES of Multimedia
COMMERCIAL
 Much of the electronic old and new media used
by commercial artists is multimedia.
 Exciting presentations are used to grab and
keep attention in advertising
COMMERCIAL
 Much of the electronic old and new media used
by commercial artists is multimedia.
 Exciting presentations are used to grab and
keep attention in advertising
ENTERTAINMENT AND FINE ARTS
 Multimedia is heavily used in the
entertainment industry, especially to
develop special effects in movies and
animations.
 Multimedia games are also very popular.
ENTERTAINMENT AND FINE ARTS
 Multimedia is heavily used in the
entertainment industry, especially to
develop special effects in movies and
animations.
 Multimedia games are also very popular.
USES of Multimedia
EDUCATION
 Multimedia is used to produce computer-based
training courses.
 Edutainment is an informal term used to describe
combining education with entertainment, especially
multimedia entertainment.
EDUCATION
 Multimedia is used to produce computer-based
training courses.
 Edutainment is an informal term used to describe
combining education with entertainment, especially
multimedia entertainment.
ENGINEERING
 Software engineers may use multimedia in Computer
Simulations.
 Multimedia for software interfaces are often done as
a collaboration between creative professionals and
software engineers.
ENGINEERING
 Software engineers may use multimedia in Computer
Simulations.
 Multimedia for software interfaces are often done as
a collaboration between creative professionals and
software engineers.
advantaGeS oF MultiMedia
Multimedia enhances the effect of text presentations.
Improves the quality of presentation and retains the
attention of audience.
It can be used for educational as well as
entertainment purpose.
 It is quick and easier to operate for the instructor.
Multimedia presentations can be modified very easily.
Multimedia is Entertaining as Well as Educational.
diS- advantaGeS oF MultiMedia
Non-interactive – if one-way, no
feedback.
Complex to create.
Time consuming.
Use of multimedia is expensive.
TEXT is the basic element of multimedia. It involves the
use of text types, sizes, colors and background color.
In a multimedia application, other media or screen can be
linked through the use of text. This is what you call
Hypertext.
To produce an effective multimedia program there are
three things that need to be considered. They are:
1. The position of the text on the screen.
2. Length of the message
3. Legibility of the text.
GRAPHICS- make the multimedia application attractive.
They help to illustrate ideas through still pictures.
There are two types of graphics used:
Paint
graphics
Draw
graphics
Bitmaps images are real images that can be captured from devices such as cameras or scanners.
Vector graphics are drawn on the computer and only require a small amount of memory.
AUDIOA multimedia application may require the use of speech, music and sound effects. These are
called audio or the sound element.
There are two basic types of audio or sound: analog and digital audio.
original sound signal
digital sampling of the actual sound.
The sound used in multimedia is
digital audio.
ANIMATION
-is a process of making a static image look like it is moving. In
multimedia, digital animation is used. Animation adds visual impact to
the multimedia project.
- Digital animation can be categorised into two broad area:
STANDARD FILE FORMATS
Each of the five elements mentioned has various standard file formats. file format is a particular way to
store information in a computer.
The formats will allow you to make decisions of the type of text, graphic, audio, video and animation to
use when developing your own multimedia program.
INTERACTIVITY OF MULTIMEDIA
If you use the first multimedia application, the content progresses without you having to
control the flow of the movie. Your only control is to press the “play” and the “stop” buttons.
This interaction is called linear interactivity.
On the other hand, the second multimedia application allows you to decide which part you
would like to view, or how much you want to know about the content.
This interaction is named non-linear interactivity.
MEDIUM OF DELIVERY
CD-BASED
MULTIMEDIA
HARDWARE AND EDITING SOFTWARE FOR MULTIMEDIA PRODUCTION
In producing a multimedia program, we need to: Gather data for the 5 basic
elements of multimedia: text, animation, graphics, video and audio by using
hardware.
To edit the elements, we need special editing software.
EDITING SOFTWARE FOR MULTIMEDIA
MULTIMEDIA AUTHORING TOOLS
CRITERIA IN EVALUATING AN AUTHORING TOOL :
 It should be easy to use; thus non-programmers do not have to
learn a programming language in order to develop multimedia
courseware.
 It should have ready-made templates, including popular buttons or
signs and navigation tools.
 It offers compatibility with the World Wide Web.
 It has on-line help and learning aids.
USER INTERFACE PRINCIPLES
User interface is a program that provides an interface or medium for
humans to interact with the system.
8 main principles of the user interface.
1. Consistency
2. Clarity
3. Context
4. Navigation
5. Search
6. Personalisation
7. Learnability
8. Flexibility
Consistency means the interface design is in harmony and the same applies
to all screen in a software program.
The same words or commands perform the same functions throughout the
user interface.
Clarity means clearness of labels on all icons to make the system easy
to understand. Users should use relevant icons or graphics to indicate
the information.
Icons, words or commands should be clearly labelled so that users can
understand them easily.
Context means every part of a lesson should be relevant to a particular
title. Ideas presented need to relate to the title.
The user interface should be structured. For example, if the lesson is
about Flow Chart, all the sections including the introduction, content,
activity, evaluation, summary and extension should be related to the
Flow Chart.
Navigation means users can move around the menus, help files or
other screens in a system
Learnability means the system provides support information and
help files to make the system easy to understand.
Support information is
important, especially in helping
the user to use the system.
The help function should be
available to speed up the
interaction for both
experienced and inexperienced
users.
Flexibility means a user has the authority to navigate through
all the sections without any limitations.
For example, the user is allowed to access any of the lessons,
sections and pages of the program.
Personalisation means the users can make their own
personal or individual learning.
Users can choose their display options.
Evaluation is testing whether a multimedia programme fulfills the
objectives set, and suggesting improvements it requires to make the
programme useful for its target audience. Evaluation is not a uniform
process and evaluation cannot be identical for all programmes. Evaluation
invariably will have to be made of the objectives that the multimedia
software wishes to fulfill. For our purpose, we may say, evaluation will
involve testing of the content it transacts vis-à-vis the target learners,
keeping in view the prime objective of the expected learning which may
take place in the learners after they go through the programme.
Evaluation is testing whether a multimedia programme fulfills the
objectives set, and suggesting improvements it requires to make the
programme useful for its target audience. Evaluation is not a uniform
process and evaluation cannot be identical for all programmes. Evaluation
invariably will have to be made of the objectives that the multimedia
software wishes to fulfill. For our purpose, we may say, evaluation will
involve testing of the content it transacts vis-à-vis the target learners,
keeping in view the prime objective of the expected learning which may
take place in the learners after they go through the programme.
At the broadest level, evaluation of a programme
should be done at two levels: at the level of content
and at the level of technology employed. Technology
plays the role of only the means to attain the
identified objectives. Optimal use of technology is
desirable and in evaluating multimedia software it
needs to be kept in mind that the technology itself
should not become too cumbersome for the users,
because the competence level of individuals using
technology varies a lot.
At the broadest level, evaluation of a programme
should be done at two levels: at the level of content
and at the level of technology employed. Technology
plays the role of only the means to attain the
identified objectives. Optimal use of technology is
desirable and in evaluating multimedia software it
needs to be kept in mind that the technology itself
should not become too cumbersome for the users,
because the competence level of individuals using
technology varies a lot.
The inherent strength of multimedia lies in the power of integration.
Moreover, a television/video programme can also have all the
components of multimedia like the text, pictures, graphics, audio,
movies etc. What is special to multimedia is its, which is limited or
mostly absent in video.
A major tenet of multimedia education is the acceptance of the fact that
no single method or medium is appropriate and perfect for all
individuals. Every individual has his/her own preferred methods of
information reception and processing. The ideal condition would be to
provide numerous options to learners. However, if numerous options
cannot be made available, a number of alternative instructional options
based on various learning approaches must be given. This is what a
multimedia programme offers.
The inherent strength of multimedia lies in the power of integration.
Moreover, a television/video programme can also have all the
components of multimedia like the text, pictures, graphics, audio,
movies etc. What is special to multimedia is its, which is limited or
mostly absent in video.
A major tenet of multimedia education is the acceptance of the fact that
no single method or medium is appropriate and perfect for all
individuals. Every individual has his/her own preferred methods of
information reception and processing. The ideal condition would be to
provide numerous options to learners. However, if numerous options
cannot be made available, a number of alternative instructional options
based on various learning approaches must be given. This is what a
multimedia programme offers.
Formative evaluation is done as a continuous process in the
development of multimedia and even before the development process
actually starts. Decisions taken at the beginning of the process of
software development affect various aspects of the software. Answers
to questions like who, why, where, and how become the guidelines for
the development of the software. Depending upon the time and
resources, both quantitative and qualitative methods of feedback are
utilized in formative evaluation. No programme can fulfill all
requirements of all learners. In fact, if a single programme can provide
all the information, and answer all queries on a single topic, it should be
considered successful. So, every software developer must decide and
delimit the scope of the software beforehand. In other words, we have
to spell out the objectives of the programme.
Formative evaluation is done as a continuous process in the
development of multimedia and even before the development process
actually starts. Decisions taken at the beginning of the process of
software development affect various aspects of the software. Answers
to questions like who, why, where, and how become the guidelines for
the development of the software. Depending upon the time and
resources, both quantitative and qualitative methods of feedback are
utilized in formative evaluation. No programme can fulfill all
requirements of all learners. In fact, if a single programme can provide
all the information, and answer all queries on a single topic, it should be
considered successful. So, every software developer must decide and
delimit the scope of the software beforehand. In other words, we have
to spell out the objectives of the programme.
Some questions like the following ones need to be answered
because these will affect the content and the selection of
technology.
(i) Who are the target users of this software and what is the level
of the target users?
(ii) What is the level of computer familiarity expected of the
learners?
(iii) What would be the objective (in terms of content) to be
covered by the programme?
(iv) How will the programme be used?
a. as supplementary to classroom teaching?
b. as independent programme providing complete courseware?
Some questions like the following ones need to be answered
because these will affect the content and the selection of
technology.
(i) Who are the target users of this software and what is the level
of the target users?
(ii) What is the level of computer familiarity expected of the
learners?
(iii) What would be the objective (in terms of content) to be
covered by the programme?
(iv) How will the programme be used?
a. as supplementary to classroom teaching?
b. as independent programme providing complete courseware?
Summative evaluation is targeted at the end-users, in
educational software a major focus of study is the software's .
The teaching learning objectives identified during the needs
analysis become the base of summative evaluation of academic
software. Evaluation should keep the educational,
entertainment, ease of use or design features in mind while
conducting summative evaluation of these and then overall
impact on learning.
Suggestions on the basis of summative evaluation may be for (i) short-term and
(ii) long-term changes in the programme. Short-term changes may be based on
your own observations and the feedback from the users and the long-term
changes may be made on the basis of the decisions of the curricular design and
on the basis of suggestions given by the development agencies and the
organisations using your software, if it is being used outside your institution.
Summative evaluation is targeted at the end-users, in
educational software a major focus of study is the software's .
The teaching learning objectives identified during the needs
analysis become the base of summative evaluation of academic
software. Evaluation should keep the educational,
entertainment, ease of use or design features in mind while
conducting summative evaluation of these and then overall
impact on learning.
Suggestions on the basis of summative evaluation may be for (i) short-term and
(ii) long-term changes in the programme. Short-term changes may be based on
your own observations and the feedback from the users and the long-term
changes may be made on the basis of the decisions of the curricular design and
on the basis of suggestions given by the development agencies and the
organisations using your software, if it is being used outside your institution.
RefeRences
www.iso.org/iso/iso_catalogue www.macrotech.co.nz
searchsoa.techtarget.com cemca.org.in
www.merriam-webster.com https://sites.google.com
techterms.com Files.eric.ed.gov
Multimedia Development and Evaluation

Multimedia Development and Evaluation

  • 1.
    MultiMedia developMent and evaluation MechelleD. Tumanda( MAIT) -Presenter -
  • 3.
    Multimedia comes from theword “multi” and “media”.“ Multi” means various “Media” refers to any hardware or software used for communicating.
  • 4.
    Multimedia represent theconsolidation of all elements of technology as they combine sound, image, video, drawing & text with a high quality in addition to the interactive environment.
  • 5.
    CURRENT DEFINITION OFMULTIMEDIA IN ICT In the field of Information and Communication Technology, multimedia means more than the use of the various media. A computer user interacts with the computer to perform tasks such as finding information or play games to develop a skill. Thus, the meaning of multimedia has changed as technology advanced in our lives.
  • 6.
    MULTIMEDIA TODAY Multimedia and itsrelated applications have almost become synonymous with modern technology; given the kind of explosion the technological realm has seem. Multimedia makes our life easier several times fold. It is through multimedia that mobile phones can be used for a number of purposes. Multimedia and its related applications have almost become synonymous with modern technology; given the kind of explosion the technological realm has seem. Multimedia makes our life easier several times fold. It is through multimedia that mobile phones can be used for a number of purposes.
  • 7.
    USES of Multimedia COMMERCIAL Much of the electronic old and new media used by commercial artists is multimedia.  Exciting presentations are used to grab and keep attention in advertising COMMERCIAL  Much of the electronic old and new media used by commercial artists is multimedia.  Exciting presentations are used to grab and keep attention in advertising ENTERTAINMENT AND FINE ARTS  Multimedia is heavily used in the entertainment industry, especially to develop special effects in movies and animations.  Multimedia games are also very popular. ENTERTAINMENT AND FINE ARTS  Multimedia is heavily used in the entertainment industry, especially to develop special effects in movies and animations.  Multimedia games are also very popular.
  • 8.
    USES of Multimedia EDUCATION Multimedia is used to produce computer-based training courses.  Edutainment is an informal term used to describe combining education with entertainment, especially multimedia entertainment. EDUCATION  Multimedia is used to produce computer-based training courses.  Edutainment is an informal term used to describe combining education with entertainment, especially multimedia entertainment. ENGINEERING  Software engineers may use multimedia in Computer Simulations.  Multimedia for software interfaces are often done as a collaboration between creative professionals and software engineers. ENGINEERING  Software engineers may use multimedia in Computer Simulations.  Multimedia for software interfaces are often done as a collaboration between creative professionals and software engineers.
  • 9.
    advantaGeS oF MultiMedia Multimediaenhances the effect of text presentations. Improves the quality of presentation and retains the attention of audience. It can be used for educational as well as entertainment purpose.  It is quick and easier to operate for the instructor. Multimedia presentations can be modified very easily. Multimedia is Entertaining as Well as Educational.
  • 10.
    diS- advantaGeS oFMultiMedia Non-interactive – if one-way, no feedback. Complex to create. Time consuming. Use of multimedia is expensive.
  • 12.
    TEXT is thebasic element of multimedia. It involves the use of text types, sizes, colors and background color. In a multimedia application, other media or screen can be linked through the use of text. This is what you call Hypertext. To produce an effective multimedia program there are three things that need to be considered. They are: 1. The position of the text on the screen. 2. Length of the message 3. Legibility of the text.
  • 13.
    GRAPHICS- make themultimedia application attractive. They help to illustrate ideas through still pictures. There are two types of graphics used: Paint graphics Draw graphics Bitmaps images are real images that can be captured from devices such as cameras or scanners. Vector graphics are drawn on the computer and only require a small amount of memory.
  • 15.
    AUDIOA multimedia applicationmay require the use of speech, music and sound effects. These are called audio or the sound element. There are two basic types of audio or sound: analog and digital audio. original sound signal digital sampling of the actual sound. The sound used in multimedia is digital audio.
  • 16.
    ANIMATION -is a processof making a static image look like it is moving. In multimedia, digital animation is used. Animation adds visual impact to the multimedia project. - Digital animation can be categorised into two broad area:
  • 17.
    STANDARD FILE FORMATS Eachof the five elements mentioned has various standard file formats. file format is a particular way to store information in a computer. The formats will allow you to make decisions of the type of text, graphic, audio, video and animation to use when developing your own multimedia program.
  • 18.
    INTERACTIVITY OF MULTIMEDIA Ifyou use the first multimedia application, the content progresses without you having to control the flow of the movie. Your only control is to press the “play” and the “stop” buttons. This interaction is called linear interactivity. On the other hand, the second multimedia application allows you to decide which part you would like to view, or how much you want to know about the content. This interaction is named non-linear interactivity.
  • 20.
  • 22.
    HARDWARE AND EDITINGSOFTWARE FOR MULTIMEDIA PRODUCTION In producing a multimedia program, we need to: Gather data for the 5 basic elements of multimedia: text, animation, graphics, video and audio by using hardware.
  • 23.
    To edit theelements, we need special editing software.
  • 24.
  • 25.
  • 26.
    CRITERIA IN EVALUATINGAN AUTHORING TOOL :  It should be easy to use; thus non-programmers do not have to learn a programming language in order to develop multimedia courseware.  It should have ready-made templates, including popular buttons or signs and navigation tools.  It offers compatibility with the World Wide Web.  It has on-line help and learning aids.
  • 27.
    USER INTERFACE PRINCIPLES Userinterface is a program that provides an interface or medium for humans to interact with the system. 8 main principles of the user interface. 1. Consistency 2. Clarity 3. Context 4. Navigation 5. Search 6. Personalisation 7. Learnability 8. Flexibility
  • 28.
    Consistency means theinterface design is in harmony and the same applies to all screen in a software program. The same words or commands perform the same functions throughout the user interface.
  • 29.
    Clarity means clearnessof labels on all icons to make the system easy to understand. Users should use relevant icons or graphics to indicate the information. Icons, words or commands should be clearly labelled so that users can understand them easily.
  • 30.
    Context means everypart of a lesson should be relevant to a particular title. Ideas presented need to relate to the title. The user interface should be structured. For example, if the lesson is about Flow Chart, all the sections including the introduction, content, activity, evaluation, summary and extension should be related to the Flow Chart.
  • 31.
    Navigation means userscan move around the menus, help files or other screens in a system
  • 32.
    Learnability means thesystem provides support information and help files to make the system easy to understand. Support information is important, especially in helping the user to use the system. The help function should be available to speed up the interaction for both experienced and inexperienced users.
  • 33.
    Flexibility means auser has the authority to navigate through all the sections without any limitations. For example, the user is allowed to access any of the lessons, sections and pages of the program.
  • 34.
    Personalisation means theusers can make their own personal or individual learning. Users can choose their display options.
  • 44.
    Evaluation is testingwhether a multimedia programme fulfills the objectives set, and suggesting improvements it requires to make the programme useful for its target audience. Evaluation is not a uniform process and evaluation cannot be identical for all programmes. Evaluation invariably will have to be made of the objectives that the multimedia software wishes to fulfill. For our purpose, we may say, evaluation will involve testing of the content it transacts vis-à-vis the target learners, keeping in view the prime objective of the expected learning which may take place in the learners after they go through the programme. Evaluation is testing whether a multimedia programme fulfills the objectives set, and suggesting improvements it requires to make the programme useful for its target audience. Evaluation is not a uniform process and evaluation cannot be identical for all programmes. Evaluation invariably will have to be made of the objectives that the multimedia software wishes to fulfill. For our purpose, we may say, evaluation will involve testing of the content it transacts vis-à-vis the target learners, keeping in view the prime objective of the expected learning which may take place in the learners after they go through the programme.
  • 45.
    At the broadestlevel, evaluation of a programme should be done at two levels: at the level of content and at the level of technology employed. Technology plays the role of only the means to attain the identified objectives. Optimal use of technology is desirable and in evaluating multimedia software it needs to be kept in mind that the technology itself should not become too cumbersome for the users, because the competence level of individuals using technology varies a lot. At the broadest level, evaluation of a programme should be done at two levels: at the level of content and at the level of technology employed. Technology plays the role of only the means to attain the identified objectives. Optimal use of technology is desirable and in evaluating multimedia software it needs to be kept in mind that the technology itself should not become too cumbersome for the users, because the competence level of individuals using technology varies a lot.
  • 46.
    The inherent strengthof multimedia lies in the power of integration. Moreover, a television/video programme can also have all the components of multimedia like the text, pictures, graphics, audio, movies etc. What is special to multimedia is its, which is limited or mostly absent in video. A major tenet of multimedia education is the acceptance of the fact that no single method or medium is appropriate and perfect for all individuals. Every individual has his/her own preferred methods of information reception and processing. The ideal condition would be to provide numerous options to learners. However, if numerous options cannot be made available, a number of alternative instructional options based on various learning approaches must be given. This is what a multimedia programme offers. The inherent strength of multimedia lies in the power of integration. Moreover, a television/video programme can also have all the components of multimedia like the text, pictures, graphics, audio, movies etc. What is special to multimedia is its, which is limited or mostly absent in video. A major tenet of multimedia education is the acceptance of the fact that no single method or medium is appropriate and perfect for all individuals. Every individual has his/her own preferred methods of information reception and processing. The ideal condition would be to provide numerous options to learners. However, if numerous options cannot be made available, a number of alternative instructional options based on various learning approaches must be given. This is what a multimedia programme offers.
  • 47.
    Formative evaluation isdone as a continuous process in the development of multimedia and even before the development process actually starts. Decisions taken at the beginning of the process of software development affect various aspects of the software. Answers to questions like who, why, where, and how become the guidelines for the development of the software. Depending upon the time and resources, both quantitative and qualitative methods of feedback are utilized in formative evaluation. No programme can fulfill all requirements of all learners. In fact, if a single programme can provide all the information, and answer all queries on a single topic, it should be considered successful. So, every software developer must decide and delimit the scope of the software beforehand. In other words, we have to spell out the objectives of the programme. Formative evaluation is done as a continuous process in the development of multimedia and even before the development process actually starts. Decisions taken at the beginning of the process of software development affect various aspects of the software. Answers to questions like who, why, where, and how become the guidelines for the development of the software. Depending upon the time and resources, both quantitative and qualitative methods of feedback are utilized in formative evaluation. No programme can fulfill all requirements of all learners. In fact, if a single programme can provide all the information, and answer all queries on a single topic, it should be considered successful. So, every software developer must decide and delimit the scope of the software beforehand. In other words, we have to spell out the objectives of the programme.
  • 48.
    Some questions likethe following ones need to be answered because these will affect the content and the selection of technology. (i) Who are the target users of this software and what is the level of the target users? (ii) What is the level of computer familiarity expected of the learners? (iii) What would be the objective (in terms of content) to be covered by the programme? (iv) How will the programme be used? a. as supplementary to classroom teaching? b. as independent programme providing complete courseware? Some questions like the following ones need to be answered because these will affect the content and the selection of technology. (i) Who are the target users of this software and what is the level of the target users? (ii) What is the level of computer familiarity expected of the learners? (iii) What would be the objective (in terms of content) to be covered by the programme? (iv) How will the programme be used? a. as supplementary to classroom teaching? b. as independent programme providing complete courseware?
  • 49.
    Summative evaluation istargeted at the end-users, in educational software a major focus of study is the software's . The teaching learning objectives identified during the needs analysis become the base of summative evaluation of academic software. Evaluation should keep the educational, entertainment, ease of use or design features in mind while conducting summative evaluation of these and then overall impact on learning. Suggestions on the basis of summative evaluation may be for (i) short-term and (ii) long-term changes in the programme. Short-term changes may be based on your own observations and the feedback from the users and the long-term changes may be made on the basis of the decisions of the curricular design and on the basis of suggestions given by the development agencies and the organisations using your software, if it is being used outside your institution. Summative evaluation is targeted at the end-users, in educational software a major focus of study is the software's . The teaching learning objectives identified during the needs analysis become the base of summative evaluation of academic software. Evaluation should keep the educational, entertainment, ease of use or design features in mind while conducting summative evaluation of these and then overall impact on learning. Suggestions on the basis of summative evaluation may be for (i) short-term and (ii) long-term changes in the programme. Short-term changes may be based on your own observations and the feedback from the users and the long-term changes may be made on the basis of the decisions of the curricular design and on the basis of suggestions given by the development agencies and the organisations using your software, if it is being used outside your institution.
  • 54.