Archive for December, 2007

CALL – MASTER ASSIGNMENT

December 10, 2007

NETLANGUAGES – LOWER INTERMEDIATE

Sample – ‘House & Home.’

 <The sample pages for the Lower Intermediate General English Course are taken from a unit called ‘House and Home’. The unit looks at accommodation, places to live, etc. On this page you can see the contents of this unit and try a small sample of the material taken from the following sections- Reading, Vocabulary, Grammar & Listening>

Introduction

As a relatively new way to deliver English Language teaching, little research into what constitutes a ‘good’ or ‘bad’ on-line course exists although as Dudeney (May 2004) points out: ‘from the pedagogical point of view, the value of a good online teaching program has been represented by the acronym DISC, which stands for: dialogue (which includes the notion of interaction – obviously crucial in online language learning, given the central role that interaction plays in language acquisition) – involvement, support, and control. These key issues can be phrased in the form of questions that the potential leaner needs to ask when choosing between options.’

In this assignment I will evaluate Netlanguages following the criteria laid out in the questionnaire but I will use the DISC evaluation in summation as I feel it is a more exact evaluation tool for on-line courses.

 

1. Communicative Skills 

Reading – 2

The reading is in four parts. The subject of Feng Shui is introduced with a short three paragraph text. There are hyperlinks on selected words to the dictionary. The general point about the subject matter of the reading is that it is up-to-date and has a modern ‘feel.’

 

More authentic material is readily available for on-line courses and unlike a book it can be updated on a weekly basis if needs be. On-line course writers and teachers do not need to wait for two weeks or more for the latest foreign language newspaper or magazine – today’s issue is online now. The learners using Netlanguages are looking at the same texts online as native speakers are reading online.

After the first short paragraph there are two global meaning comprehension questions followed by four short paragraphs and a ‘choose the title’ exercise. The last exercise has a visual component in that it combines text and picture matching.

The learner has immediate feedback when doing the exercise with a pop-up box which simply says ‘correct’ or ‘incorrect.’ If the learner is incorrect they are advised to ‘have another go!’

There is not a lot of text to be read by the learner. This may be because it is designed for low-intermediate level but also it is felt that people do not like to spend long periods reading text from a screen as it can cause eye strain. Although the hyperlinks to the dictionary gives another opportunity to read perhaps there should and could be links to further reading materials on Feng Shui for fast finishers.  Instead of a simple ‘correct’ or ‘incorrect’ the answers could include links to further reading on the subject. I say this for these reasons:

Reading is best enhanced by reading, and the more, the better. Students who use the web to find information often use scanning skills, as well as extended reading. If you do not find what you are looking for you click on a link and see where it takes you. This is the way most people use the web in their own language so it seems a missed opportunity to not include links in a reading text. Although it must be said that the main menu includes a web search (button 11) ‘Find more information about Feng Shui and famous homes…’ which I am unfortunately not able to check out as it is not included in the sample lesson.

 

Listening – 3

There are three listening exercises. The first exercise involves three speakers who describe a room and the learner has to match the speaker to the name of the room. The listening is for general meaning. The feedback is limited yet again to ‘Congratulations!’ or ‘Try again.’

The second exercise is a listening exercise for more specific meaning. The feedback is again quite limited.

In the last exercise learners have to listen for adjectives and make a note of them. Finally the learner can click on a hyperlink to bring up a transcript to check their answers.

I would categorize the listening section as only average mainly because it repeats the format and approach of the other sections.

Writing (Not possible to give evaluation)

We are not able to comment directly on the writing exercises as we are not allowed access to them in the sample lessons. However writing is included in four sections:

8 – Tell your tutor about where you live…

12 – Write a letter giving information about how to organise an office…

13 – Send your work to your tutor and get feedback…

15 – Send us your feedback on this unit…

Obviously, writing is directly covered in section 12 when the learners are asked to write a letter about how to organize an office.  It is a pity that we cannot directly comment on the section so I will limit myself to some general comments about writing practice in online courses.

 

Students tend to write more and revise more when they word-process their texts, and they take more pride in the finished product. Writing, using a PC is more authentic and ‘real world’ nowadays, much more so than writing with pen and paper in a classroom. Writing online and using a PC is taken more seriously as it mirrors the way most people write in their everyday lives.

One of the most-researched areas in this field is the use of computers to enhance writing skills. Word-processors have been seen as tools that promote writing skills in various ways. Process writing in particular has been helped by the use of information technology (Boone, 1991). For example, brainstorming is more visible with different types of concept mapping (Mikulecky, et al., 1989); revising on the word-processor is more likely to occur than rewriting on paper and peer critiquing occurs more readily in many cases when the discussion is online rather than face-to-face (Herrmann, 1989; Marx, 1990). Just using a word-processor or online discussion is not enough to create permanent beneficial effects on writing, however.

An exploratory study by Krause (1995) with native speakers using online discussion suggests that online discussion may be valuable as a teaching tool, but just using online discussion does not necessarily spill over to affect offline writing. Clearly, the tasks set by the teacher and instructions given on use of computer tools are essential to long-term benefits in writing skills. I would assume that the online tutor guiding learners using Netlanguages would be aware of these points.

Speaking 4 (No speaking practice on Netlanguages )

I cannot find any speaking practice in Low Intermediate apart from pronunciation practice which I will comment on later. This is an obvious gap in Netlanguages. We live in a face to face (f2f) world. Technology exists and is used everyday to enable synchronous or live f2f interaction via the net. Online courses should use this technology to allow learners to talk to each other and their tutors.

It seems to be an omission as well that Netlanguages does not use some of the excellent already existing software which enables learners to practice speaking online.

In general a link to the Internet can provide a source of authentic, up-to-the-minute information in the target language with opportunities for real communication with native speakers. The speaking component of Netlanguages falls short of utilizing what is possible and giving learners speaking practice.

2. Linguistic Component

Grammar – 2

The first exercise is on prepositions of place using pictures. The learners have to match the prepositions to the pictures. The feedback is limited to ‘There are 9 correct matches. Try one more time.’

Exercise 2 is a click and drag activity where the  learner has to use prepositions in context in a text on Feng Shui.

In the final activity learners can listen to how the prepositions are corrected.

I think the exercise is fine as far as it goes  but is perhaps a shade limited in that there is little pre-teaching of the grammar. The drawback with these types of exercises is that the learner could in theory get the correct answers through trial and error in that if you do the exercises enough times you will get 100% on the basis of elimination.

The way the grammar is taught in the sample lacks a constructivist and cognitive element which I feel is more appropriate when teaching grammar.

 

Pronunciation (Not possible to give evaluation)

In section 10 there is a specific pronunciation activity ‘Minimal pair questions – identifying differences between pairs of questions…’ Unfortunately we are not able to access this section in the sample so it is not possible to evaluate how pronunciation is tackled on Netlanguages.

However pronunciation is covered in the grammar and vocabulary sections. It is limited to listening to how the words are pronounced. There is software available which can record a learner speaking and play back the speech. This can be compared to a model pronunciation. If Netlanguages does not use this technology it would be another gap in the design of the course which should be addressed.

Vocabulary – 3

There are three parts to the vocabulary section. Exercise 1 is a text entitled ‘A Room of my own’ where selected vocabulary items are hyperlinked to the dictionary. In exercise 2 the learner has to use the text as a guide to fill in a vocabulary list using a drawing on which objects are numbered. An interesting drawback in this kind of activity is that the technology does not always reflect the flexible nature of language. In exercise 2 question 1 the answer is ‘coffee table.’ Yet it would be perfectly acceptable to call the object simply ‘table.’ However the system rejects the answer ‘table’ as incorrect and tells the learner to try again.

There is a tendency to present vocabulary in black and white terms on online courses when the reality of living language is grey. For that reason I would only give the vocabulary section on Netlanguages an average score.

It is informative to compare the Netlanguages approach to teaching vocabulary to some very interesting research done on vocabulary development . As most reading teachers know, having an adequate vocabulary is critical to reading success. While students can do some guessing from context, they still need to be able to understand the bulk of what they are reading in order to make sense of the rest. Various studies in cognitive science have pointed to the usefulness of multiple modes of input, such as aural and graphical as well as textual data, for enhancing recognition and recall. This is cited as one of the strengths of multimedia (Liaw, 2001).

The main factor influencing vocabulary learning, however, as well as reading skill development in general, is exposure. Vocabulary is learned best through repeated exposure to words in context, as part of reading. Concordancing is one approach to giving students more exposure to a word in a variety of contexts. A concordance scans a very large text base (usually made up of hundreds of documents or more) and pulls out all the occurrences of a target word, along with some of the context. Most concordance displays have the search term in the middle, then five to ten words of the context on either side. With a large text base, almost any word will have several occurrences, and the learner can see them all together. Cobb (1997) explains the benefits of vocabulary learning with a computer tutor based on a concordancer:

“It is indeed a tough paradox that you need words to learn words, but it can be softened by three factors. First, … with several contexts accessible, a learner is likely to find one where he knows enough ambient words to make a useful inference. Second, a finer-grained picture of exactly how many words are needed to make inferences is becoming available through corpus analysis…. Third, a corpus tutor can be designed to contain elements of both direct instruction and dictionary work in the initial bootstrapping phase.”

I feel the Netlanguages online course approach to teaching vocabulary could benefit from Cobb’s advice.

3. Methodology

Method – 3

 

I would like to make some general points about methodology used in on-line courses before analyzing Netlanguages. An online course does not constitute a method in of itself, rather it is a means of delivering English teaching using whatever method(s) the course designers favour.

However the way E-Learning in general and online courses in particular are structured opens up the possibility of multiple methods and approaches being packaged together. An eclectic approach in practice rather than purely in words.

 

For example the relationship between teacher/tutor and learner is completely different on an online course to the normal relationship we have in the traditional classroom. For  those teachers who are accustomed to being the sole source of information and authority, this can be a very large change. It is hard to avoid as more students have access to computers and the Internet at home, they can get information and ask questions independently. It is no effort for a student to pose a grammar or vocabulary question at Dave’s ESL Cafe or other similar sites to get an answer from another teacher. If you’re wrong, the students can find out. I feel that this exciting aspect of online learning is missing from the Netlanguages approach. The learners do not seem to be encouraged to go out into the digital world to search for answers and meanings themselves. I did not see many hyperlinks in the sample I studied to the super information highway.

 

Experiential learning can be easier with electronic media because large amounts of data are available and students can easily build and revise electronically. I do not think Netlanguages really takes advantage of this possibility.

 

Research in cognitive psychology has supported the concept that people use past knowledge and experience to help provide a structure for new ideas, and that people interpret new information based on their previous knowledge and experience. In other words, we create meaning for ourselves; we do not incorporate it wholesale from others. Unfortunately is that not the mistaken method of Netlanguages with their ‘correct’ ‘incorrect’ pop-ups? It is indeed unfortunate because the technology available for online courses gives us the opportunity to include more than one method.

 

Language learners should be active participants in learning, not just empty vessels to be filled with the teacher’s knowledge. Piaget’s contemporary, Lev Vygotsky, also believed that students were not just passive recipients of knowledge. They learned best in their “zone of proximal development” – where they can, with guidance, learn more than they would on their own. I cannot really comment on the tutor’s role in Netlanguages because we do not see their role in the sample but in the sections I have commented on the learner does seem to be left on their own until they come up with ‘correct’ answers.

Netlanguages could do a lot more to create opportunities for experiential learning. Students should be gathering and analyzing data and demonstrating their knowledge in a tangible way.

 

I believe that constructivist learning is easier to design and set up electronically than using other media. For instance tools for constructing meaning can be included on online courses: In the constructivist approach to language learning, students should be able to create their own meaning by producing newsletters, web pages, and multimedia presentations. In the samples I have seen I feel Netlanguages has missed an opportunity to achieve this. It might be said that this could be a big drawback in Netlanguages if the same ‘correct – incorrect’ approach is used in the higher levels.

 

On the positive side Netlanguages offers immediate feedback. Students do not have to hunt for an answer key at the back of a book or wait for the teacher to return an exercise to know if they understood the material. This brings an immediacy to the learning which suits some learners’ individual styles.

There also exists the possibility on Netlanguages for limitless repetition. This should not be underestimated as a strategy especially at lower levels. Computers and online courses are superb at handling repetitive tasks. They can present the same material again and again at times when their human instructors are not available. Repetition plays an important role in reinforcing learning. In this respect Netlanguages method works well in that the learner can tailor the course to their own needs. If they need a slower pace and constant repetition they can have it. There is no teacher to hurry them along. They choose when they feel they have learnt something and it is time to move on.
On a general point computers and online courses can also be said to provide a “Neutral” medium. Netlanguages’  performance via the computer is consistent. It never loses patience with students, never gets angry, and never plays favourites. Language learners continually create mental hypotheses about how the target language works. The computer can provide a safe environment in which to test linguistic hypotheses.

Without complete access to Netlanguages it is difficult to make definite judgments about the method used (particularly as regards the role of the tutor) but it seems to me that the course is based on the International House eclectic approach to methodology. However, I must say that the samples I have evaluated smack of a Presentation, Practice and Production approach to teaching which is prevalent on the CELTA courses which are closely linked to International House. I did not see much evidence of a task based approach, cognitive activities or communicative exercises in the samples. This may be an unfair criticism because I have only seen a snapshot of the course and a lower level aswell.

Motivation – 3

As I have already mentioned multiple learning styles can be supported more easily in online courses. Often, similar material is available in text, graphical, audio, and video format. All of these can be accessed in one place, via the computer, rather than having to bring in books, tapes, and VCRs

Students can be more motivated doing online courses. It feels different; it appeals to the MTV generation; all of the positive aspects of technology create more motivation for students. These points are all valid but the question is: does Netlanguages take advantage of these advantages? I think in this respect Netlanguages is only an average online course. It has the feel of only going so far in tapping the potential of the internet.

 

The internet is a technology that is constantly developing and changing. Netlanguages could be quickly left behind. It is true that the shift to the computer lab has provided a change of pace, and some students may work more eagerly on a computer than in other settings. However, I believe that the “motivation” factor may have been more significant in the past when computers were a novelty. For example Hollywood will always be ahead of education in terms of its ability to produce sophisticated multimedia, and today’s students may not be impressed with the relatively low level of sophistication of most online courses.

It is not enough to impress today’s learners just to put The Headway series on the Internet. I am not saying that Netlanguages is just ‘Headway online’ but it could be much much better than it is.

To justify the expense and effort involved in using computers and online courses, I think we need to look for more compelling reasons than any motivation that is provided by the medium itself.
Having said that I think there are two areas which Netlanguages has worked on very well and which greatly aids motivation – individualization and control of input.

There is a tremendous opportunity for individualization of learning inherent within online courses which is taken advantage of by Netlanguages. Students can work at their own pace. Those who learn more slowly can use CALL materials for remediation while others can use them to accelerate learning.
Control over language input: In Netlanguages the target language is presented in manageable segments. The language remains natural and at normal speed since digital audio and video clips can easily be played as often as necessary. Text is associated with each segment and displayed or hidden according to the needs of the learner. This aspect works very well in Netlanguages.

Interaction -4

 

It is interesting that the slogan of Netlanguages is: ‘e-languages with a human touch.’ This is obviously a reaction to the negative feelings many learners and indeed teachers have about online learning due to the approach of companies such as Opening, Wall Street and Blah Blah & Co (amongst others) which gave CALL an undeserved bad name. The slogan gives the impression that online courses are not cold and unfriendly and that there is interaction and not just with your computer! The question of questions in this assignment is: what is the reality?

In this section I will assume that we understand ‘interaction’ to mean interaction with a tutor and other learners as I have already described the interaction the learner has with the computer. I will however make one point about the interaction I noticed on the samples I evaluated.

Immediate feedback: The almost instantaneous feedback on Netlanguages  can save learners hours of misguided work. Even the most unintelligent “no, you’re wrong”  (seen in Netlanguages) type of feedback can alert students that they are on the wrong track. This can be considered positive interaction.

There are five possibilities to interact on the sample:

7 -Post messages about Feng Shui and your house, and find out about other   students’ houses…

9 – Tell your tutor about where you live…

11 – Find more information about Feng Shui and famous homes…

13 – Send your work to your tutor and get feedback…

15 – Send us your feedback on this unit…

They range from sending feedback to the tutor, through a web search and chat session. We could say that a third of the section involves interaction of some sort which is laudable. However it seems all the interaction is written.

If ‘7′ (above) is a synchronous chat then that is an excellent opportunity for interaction although it seems most of the interaction is asynchronous and written. This probably reflects the kind of interaction most people indulge in on the internet although synchronous voice chat is widely available with broadband connection and I would have liked to see this included in Netlanguages as well.

Contents – 5

Netlanguages is advertised as a General English course. The sample I have evaluated has the title ‘House & Home’ and the contents cover this theme. The ‘Feng Shui’ reading is topical, trendy and actually quite interesting. The vocabulary connected with rooms and objects found within is vital vocabulary needed by all learners. For these reasons I consider the contents to be excellent.

4. Technical Features

Interface appearance – 5

It should be said that in general opinions about the design of websites are very much a personal matter. Some people like the screen to be filled up with images and hyperlinks while others are looking for lots of type. One point which should be borne in mind is that it is widely accepted it is difficult to look at pages of text for long periods on a PC, as  headaches or eyestrain usually are the result. For this reason English language online course designers usually try to keep text to the bare minimum and give the option of printing out longer texts. Incidentally I did not see this option on Netlanguages.

The Netlanguages interface is clean and uniform. The design is minimalist. The red numbered buttons are clearly identifiable as the entry points for the different sections of the course.

It is also important that each activity is contained on one screen with an acceptable amount of scrolling down to get to the last exercise on each screen.

The design is very logical and avoids confusing the learners with too many ‘bells and flashy buttons,’ for example the buttons for activating the listening activities look like the buttons on a tape recorder. What could be simpler?

Navigation – 4

Some of the points mentioned above apply to navigation as well. Netlanguages is an easy site to get around. There is not a lot of scrolling and the various sections are clearly marked.

My only criticism is that the designers have not taken advantage of the possibilities available with software today.

For example there are tremendous advantages of hypertext: On a computer, text can be linked in a non-linear fashion to explanatory text, to sound, to images, and to video. This capability can provide learners with links to extend and enhance their understanding of the new language. By choosing which paths to explore, readers can create their own meaning and organization. Netlanguages uses hypertext to link to the dictionary. I would have liked to see a more imaginative use of hypertext on Netlanguages.

It should be added that an internet novice would need some initial training to do the Netlanguages course. It is designed for people who already know their way around the keyboard and internet a little.

Summary

As I mentioned in my introduction the value of a good online teaching programme has been represented by the acronym DISC.Online learning: it is all about dialogue, involvement, support and control – according to the research(Coomey, M, and Stephenson, J)

According to Dudeney (May 2004) the following questions should be asked when evaluating online courses: I will add my opinion as to how I think Netlanguages measures up after each question.

 

Dialogue. How much opportunity is provided to interact with the tutors, other learners, and proficient users of the language in general? – Netlanguages has ample opportunities to interact with tutors and other learners but I did not see opportunities to interact with proficient users of the language.

What kinds of interaction are available (e.g. synchronous/ asynchronous, written, spoken, etc) and how integral are these to the program – as opposed to being an optional “frill”? – Netlanguages uses mostly written asynchronous interaction. It could increase the options for synchronous spoken interaction.

Using e-mail, bulletin boards, ‘real-time’ chat, asynchronous chat, group discussions and debate, the tutor or moderator structures interactive opportunities into the content of the course? – Netlanguages uses all these options on the course.

Involvement.

How motivating are the materials and tasks? – Netlanguages materials appear  to be motivating.

How much constructive feedback will I get on the tasks that I do, and how explicit ? – As I explained above the immediate feedback is very sparse, limited to ‘correct’ and/or ‘try again,’ but this type of feedback does have it’s place.

How much online collaboration with other learners is involved? – In the sample I evaluated a third of the activities involved some sort of collaboration, even if that was only emailing the tutor.

Does the course enshrine a traditional “delivery” view of learning (e.g. is it simply an online reference book) or does it adopt a more constructivist approach, whereby learning is jointly constructed through involvement in meaningful tasks? – Netlanguages clearly enshrines a traditional “delivery” view of learning.

Includes responses in structured tasks, active engagement with material, collaboration and small group activities? -  I did not see much evidence of this in the sample.

Support.

Will I be assigned a personal tutor for the length of the course? – I assume this is the case with Netlanguages as sessions with a tutor are mentioned in the menu.

How much tutor contact will I get, how regularly, and in what form (e.g. email, phone, net meeting etc)? – I get the impression that the tutors are readily available.

Is the tutor’s role purely managerial, or will it involve pedagogical and pastoral support as well? – They seem to offer pedagogical and pastoral support.

Includes periodic face-to-face contact, online tutorial supervision, peer support, advice from experts, feedback on performance, support services and software tools? – I am not aware that Netlanguages offers these services as it does not appear in the sample or indeed the advertising blurb.  This is perhaps the most important feature of successful online courses according to surveys.

Control.

Will I be able to view sample materials before committing myself to the course? – Netlanguages is a lot more liberal than other online course providers in that it allows you to get a fair idea of the course from the sample. I have understood that the site is broken into four areas; a study room, a library, a cafe, and a teacher’s room. The sample has about 16 different activity types. However it is difficult to gauge the amount of support which is available.

How much control will I have over the pace and timing of the course? – This is unclear as well.

How much say will I have in the choice of topics and texts, and the sequencing of tasks? – It does not seem that there is a lot of choice available to the learner. They seem to presented with the course ‘as it is’ but it does do ‘what is says on the bottle!’

How much support will be provided in terms of training me to be an autonomous and resourceful online learner? – This does not come across clearly from the sample.

Finally, although I have been critical of some aspects of Netlanguages it is in my opinion the best example of an all-around English language course based on Web technology. Areas which need improvement such as ‘Involvement ‘and ‘Control’ will undoubtedly be addressed in the coming years.

BIBLIOGRAPHY

_________________________________________________________________________

Bangert-Drowns, R.L. (1993). The word processor as an instructional tool: A meta-analysis of word processing in writing instruction. Review of Educational Research, 63 (1), 69-93.

Boone, R. (1991) (Ed.). Teaching process writing with computers. Revised Edition. Eugene, OR: ISTE.

Coomey, M, and Stephenson, J (2001)  in Stephenson, J. (ed.) Teaching and Learning Online, London: Kogan Paul.

Cobb, T. (1997). From concord to lexicon: Development and test of a corpus-based lexical tutor. [Online] Available: http://www.er.uqam.ca/nobel/r21270/webthesis/Thesis0.html (10-28-01).

Cobb, T., Greaves, C., Horst, M. (2001). Can the rate of lexical acquisition from reading be increased? An experiment in reading French with a suite of on-line resources. [Online] Available: http://www.er.uqam.ca/nobel/r21270/cv/BouleE.htm (10-28-01).

Cole, M.J. and Wertsch, J.V. Beyond the individual-social antimony in discussions of Piaget and Vygotsky. [Online] Available: http://www.massey.ac.nz/~alock//virtual/colevyg.htm (10-27-01).

Daiute, C. (1985). Writing and computers. Reading, MA: Addison-Wesley.

Dewey, J. (1938). Experience and education. New York: Collier Books.

Healey, D. (1993). Learner choices in self-directed second-language learning. Dissertation: Thesis (Ph. D.) University of Oregon, 1993.

Dudeney, Gavin. (May 2004) Technology in Teaching: New Directions – Speech, Aristotle University of Thessaloniki, Greece.

Herrmann, Andrea. (1989).Teaching writing with peer response groups. Encouraging revision. ERIC Digest. ERIC Clearinghouse on Reading and Communication Skills, Bloomington, IN. ED307616.

Kitao, K. and Kitao, S.K. (2001). Keypal opportunities for students. [Online] Available: http://ilc2.doshisha.ac.jp/users/kkitao/online/www/keypal.htm (10-28-01).

Krause, S. (1995). “How will this improve student writing?” Reflections on an exploratory study of online and off-line texts. Computer-Mediated Communication Magazine, 2 (5), 10. [Online] Available: http://www.ibiblio.org/cmc/mag/1995/may/krause.html (10-27-01).

Lai, S. (2000). Influence of audio-visual presentations on learning abstract concepts. International Journal of Instructional Media, 27 (2), 199-206. Accessed 10-27-01 from Academic Search Elite database.

Liaw, S. (2001). Designing the hypermedia-based learning environment. International Journal of Instructional Media, 28 (1), 43-56. Accessed 10-27-01 from Academic Search Elite database.

Marx, M. S. (1990). Distant writers, distant critics, and close readings: Linking composition classes through a peer-critiquing network. Computers and Composition, 8 (1), 23-39.

Mikulecky, L., Clark, E., and Adams, S. (1989). Teaching concept mapping and university level study strategies using computers. Journal of Reading, 32, 694-702.

Piaget, J. (1976). To understand is to invent: The future of education. G-A Roberts (Trans.) Où va l’éducation?, 1948. New York: Penguin Books.

Vygotsky, L. (1986). Thought and language. A. Kozulin (Trans. and Ed.). Cambridge, MA: MIT Press. Original work published 1934.

ASSESSMENT AND TESTING – MASTER ASSIGNMENT

December 10, 2007

Introduction:

This is an actual end of unit progress test from a project I am working on. The project is an on-line General and Technical English course. The course consists of fifty units which takes learners from elementary to advanced level. I have chosen progress tests from Unit 37 which is intermediate level (Common European Framework level B2). I have included two tests because the learners I work with do the general and technical English courses in parallel.

Each unit of the course has seven sections:

1. Introduction

2. Vocabulary

3. Reading

4. Structures

5. Functions

6. Listening

7. Communication Activities

8. Self Evaluation

1. The INTRODUCTION section serves two purposes in that it reviews the previous unit and introduces the new one.

2. The VOCABULARY section introduces the lexical element of the unit.

3. READING skills are practised in this section

4. Grammatical STRUCTURES are covered in this section.

5. Language FUNCTIONS are practised.

6. LISTENING skills are covered in this section.

7. Learners take part in communicative activities in the COMMUNICATION ACTIVITIES section.

8. In the SELF EVALUATION section learners decide for themselves if they are able to move on to the next unit or if they need to go over the unit again or require help from their tutor.

The learners do the course at home or at work on-line. They have access to a virtual classroom and are in touch with fellow learners and a tutor by e-mail and chat.

Common European Framework of Reference for Languages

The European Language Portfolio.

The institution where I teach has given a commitment to work to implement the ELP. Therefore the test I am presenting for this assignment has to be seen in this context. It is an attempt to introduce self-evaluation and learner responsibility and autonomy which are at the heart of the ELP approach. However we have to work in the language teaching world as it is at the moment and so my test contains aspects of older forms of testing as well as the new approach of the ELP.

I feel it would be useful to summarize the main objectives of the ELP.

  • Promotion of multilingualism and cross-cultural dialogue.
  • Facilitation of mobility in Europe.
  • Strengthening and preservation of cultural diversity.
  • Promotion of autonomy in learning.
  • Encouragement of life-long language learning.

The ELP also has three main aspects:

  • The Language Passport gives an overview of the current level of language proficiency and summarizes the learning and intercultural experiences of its holder.
  • The Language Biography documents the personal history of language learning and intercultural experiences. It also contains instruments for self-assessment of language proficiency and helps reflection and planning of learning.
  • The Dossier is a collection of work that illustrates what the learner has done and is able to do in different languages. On the one hand the dossier can be used as a “working dossier” which accompanies daily language learning and documents the learning process, on the other hand as a “showcase dossier” which illustrates the present level of language proficiency.

THE TYPE OF TEST ACCORDING TO PURPOSE?

The first point to be made about the type of test I have designed is that it needs to be seen in the overall context of the course. The on-line course aims to be communicative and interactive. However it must be borne in mind that the learner is physically isolated from and never meets their fellow learners or tutor. Therefore the end of unit test or ‘Self Evaluation Test’ has to serve multiple purposes.

The three main purposes are:

1. As a Progress Test: Most normal classroom tests take this form. They assess progress students make in mastering material taught in the classroom. They are often given to motivate learners. They also enable learners to assess the degree of success of teaching and learning and to identify areas of weakness and difficulty. Progress tests can also be diagnostic to some degree.

However, as this is an on-line course it takes a slightly different form. The tutor receives the data from the learner’s Self Evaluation test but there is nothing to stop the learner ploughing on to the next unit / level even if they have scored quite low in the test. If the tutor sees that the learner is consistently scoring low in the tests they should point it out to the learner and offer counselling but the idea is that the learner themselves should take responsibility for monitoring their own progress and asking for advice if they feel it is needed.

2. I would also say that the Self Evaluation test should be considered as a Diagnostic Test as well. Diagnostic tests enable teachers to identify specific weaknesses and difficulties so that an appropriate remedial programme can be planned. With my Self Evaluation Test the emphasis is on the learner to discover their own weaknesses and seek assistance and solutions. They are also encouraged to recognise their strengths and build upon them.

3. The final purpose of the Self Evaluation Test is to raise awareness of learning strategies and promote learner autonomy. The advice and tips given at the end of the test are devices to encourage learners to take an interest in the processes of learning a second language, in effect, to learn how to learn.

TEST GENERATION?

I would not consider any aspect of my test to be first generation. Questions 17 and 18 in the General English section could be interpreted as first generation if looked at superficially but in reality they are real-world tasks and involve a degree of peer correction. The questions are testing the communicative ability of the learner and as such are not open to the subjective judgement of an examiner which is the hallmark of first generation testing.

I would consider questions 6 – 16 in the General English section and all the questions (apart from Q9) in the Technical English section to be Second Generation type questions. They test discrete items including pronunciation, vocabulary and grammatical structures. The questions relate to specific items which have been taught in the module.

Overall I would consider that my test is an example of a Third Generation Test. I have tried to contextualize the questions and make them as real-world as possible. There are examples of integrative questions and questions which aim to test global comprehension. (Q17 & Q18 General English and Q9 Technical English) These types of questions also test the productive skills in that the learner has to write and speak and exchange information with the tutor and other learners. I have tried to create a test which both tests the learner’s language knowledge and the capacity to use that knowledge and language in communicative situations.

I would also add that my test is an attempt to integrate the self evaluation and learner autonomy / responsibility components of the European Language Portfolio. There is a clear attempt in the test to edge the learner towards self assessment and to think about and understand the language learning process. One of the aims of my test is for the learner to realize that the teacher is merely a guide and for a successful outcome of the language learning process the learners themselves have to assume a greater responsibility. They are encouraged to diagnose problem areas, set objectives and assess their learning styles and progress.

PRINCIPLES OF LANGUAGE TESTING?

As I have explained above my test is a combination of Second and Third Generation Tests and Self and Peer Evaluation. I will comment on each of the pairs of principles.

COMPETENCE VS. PERFORMANCE

I feel there is a balance in the test between finding out what the learner knows about the language and how they use it in concrete communicative situations.

USAGE VS. USE

Again there are examples of both in my test. The learners are asked to give examples of vocabulary and structures to indicate their knowledge of the rules (usage) but they are also asked to produce language in performance (use).

DIRECT/INDIRECT ASSESSMENT

Although there are examples of indirect testing, I have endeavoured to make my test a direct test overall.

DISCRETE POINT VS. INTEGRATIVE ASSESSMENT

As I have explained above my test includes examples of both discrete item and integrative assessment.

OBJECTIVE VS. SUBJECTIVE ASSESSMENT

Again the test contains examples of both objective and subjective assessment. The subjective assessment centres on the productive skills. In an attempt to overcome the usual problems with subjective assessment, peer assessment and self-evaluation have also been introduced.

RECEPTIVE VS. PRODUCTIVE SKILLS

The test attempts to assess receptive and productive skills.

BACKWARD AND FORWARD LOOKING ASSESSMENT

In general I would say this test is more forward looking than backward. Although it tests what was taught in the module it also attempts to assess the use of real-world language and so in that sense it is definitely forward looking.

CONTEXTUALIZED VS. DISEMBODIED LANGUAGE

The whole course I have written attempts to use contextualized language. Great care has been taken to give a context to all activities and tasks. (See above – ‘Communication Activity’) Therefore the Self-Evaluation Test aims to reflect this.

CRITERION REFERENCED AND NORM-REFERENCED ASSESSMENT

The main idea behind the test is to introduce the learners to Self-Evaluation as defined by the ELP. The test is therefore certainly not Norm-Referenced. It leans more towards Criterion Referenced assessment but with the emphasis being on the learner themselves to make the judgement whether or not they have successfully completed the various tasks.

RELIABILTY VS. VALIDITY

Overall the test is subjective in that the majority of the questions or reflections are not objective. This should in theory reduce reliability but I strongly feel that the self-evaluation and peer evaluation element actually increases the validity. Initial studies on the implementation the ELP show that self-evaluation is extremely accurate.

Content Reliability: In my opinion the test reflects perfectly the language of the syllabus.

Construct validity: My test not only is mainly a third generation test but is an attempt to promote autonomy in learning and self-evaluation, which are both current theories in language learning and testing.

Predictive validity: I think test does assess future language performance quite accurately, especially the LEARNING DIARY reflection at the end of the test.

Concurrent validity: I am confident that my test would produce similar results to established tests.

The underlying principle in my test is self-evaluation. I believe self-evaluation enables learners to develop their critical faculties. Self-assessment also enables learners to look at language in more concrete terms. Self-assessment hopefully will motivate learners to look at their strengths and weaknesses and become more autonomous learners which is a fundamental part of the learning process.

According to Oscarsson (1989), there are six different reasons why self-assessment can be beneficial to language learning.

1. Self-assessment promotes learning, plain and simple. It gives learners training in evaluation which results in benefits to the learning process.

2. It gives both learners and teachers a raised level of awareness of perceived levels of abilities. Training in self-assessment, even in its simplest form, like asking “What have I been learning?” encourages learners to look at course content in a more discerning way.

3. It is highly motivating in terms of goal-orientation.

4. Through the use of self-assessment methodologies, the range of assessment techniques is expanded in the classroom. As a result of using self-assessment, the learner broadens’ his/her range of experience within the realm of assessment.

5. By practicing self-assessment, the learners participate in their own evaluation.

They, in effect, share the assessment burden with the teacher.

6. By successfully involving learners in their own assessment, beneficial post-course effects will ensue.

RELEVANT SCALES AND INSTRUCTIONS FOR MARKING TEST?

Overall the test is based on the ELP self-assessment common reference levels. The global scales are as follows:

Basic User:

A1

A2

Independent User:

B1

B2

Proficient User:

C1

C2

There are also specific questions which relate to the material covered in the module which are backward looking but overall the test is an attempt to introduce self-evaluation in the context of an on-line course.

The self-evaluation is achieved by means of three devices:

Rating Scales

Check lists

Questionnaires

These three techniques have been used in self-evaluation. They have been used as a means where learners can rate their perceived general language proficiency or ability level. A lot of developmental work has been done in this area through the use of “ability statements” such as “I can read and understand newspaper articles intended for native speakers of the language.” I have adapted the statements in my test from ELP materials.

The learners are instructed to indicate their estimated ability to cope with situations by reflecting on and ticking the described level of performance. (In the web version of my test the learner can tick a check box to indicate if they agree with the self-evaluation statements. This is not possible to show in this printed version.)

TEST FORMATS?

The test uses three main formats.

1. The questions relating to the material taught in the module (mainly grammar and vocabulary) are in a multiple choice format.

2. The self evaluation is achieved by means of statement ‘Check lists.’

3. Communicative competence is tested authentically where possible and is interaction based, unpredictable, and varied in context and has a communicative purpose.

These formats have been used because I feel they are most appropriate to the type of test I wanted to create.

UNIT 37 GENERAL ENGLISH JAM SESSION

After finishing the unit can you…

  1. express yourself using similes and comparisons?
    1. Yes
    2. No
  1. distinguish if a word has the sound /s/ or /z/?
    1. Yes
    2. No
  1. name at least four similes using ‘like’ or ‘as’ and name several words related to music?
    1. Yes
    2. No
  1. use different words and expressions that have either positive or negative meaning?
    1. Yes
    2. No
  2. review a film, a song or a book?
    1. Yes
    2. No

Listen to the following words and click on the sound they have.

  1. a. /z/ sound

b. /s/ sound

  1. a. /z/ sound

b. /s/ sound

  1. a. /z/ sound

b. /s/ sound

  1. When my sister wants to go out she gets dressed as ________ as a flash.

a. cold

b. quick

c. stubborn

  1. I’ve been eating like a _______ for the last week because I’ve been really nervous.

a. bear

b. fish

c. bird

  1. Yesterday a man came to the surgery and he was as deaf as a ________.

a. post

b. clock

c. log

12. Which is the odd one out?

    1. Tacky
    2. Cool
    3. Rubbish
  1. What does ‘haven’t got a clue’ mean?
    1. that you are an expert at something
    2. that you have no idea about something
    3. that you haven’t got an opinion about something
  1. Click on the right option.
    1. a leather pair of comfortable shoes
    2. a pair of leather shoes comfortable
    3. a comfortable leather pair of shoes
  1. I think James Joyce books are extremely boring and difficult to read.
    1. gradable adjective
    2. un-gradable adjective
  1. Yesterday I met an English man and he spoke to me…
    1. as slowly like he could.
    2. like slowly like he could
    3. as slowly as he could

Writing an E-mail

17. Try to remember the best concert you have ever been to. Write an e-mail to your study-mate telling them about the event. Try to use some of the words you have learned in this unit. Use comparisons and adjectives. Send me a copy of the message as well.

Reply to any messages you receive from study mates.

Leaving a voice mail message.

18. Your favourite group are playing in your town. Send a message to your study-mate telling when and where the concert will be and how much the tickets will cost. Try to persuade your study-mate to go with you. Send me a copy of the message as well.

Reply to any messages you receive from study mates.

Pronunciation & Vocabulary

19. You should record all the new vocabulary in your Audio Diary and check your pronunciation with the model pronunciation in the Audio Dictionary.

GENERAL ENGLISH FEEDBACK

POINTS TO THINK ABOUT

0-6: Perhaps you need to work more on some points covered in this unit. This could be because you’ve skimmed over some sections, or because you’ve had trouble concentrating. Have a break and go over the things you aren’t sure about again. Remember – this part of the unit is to help you consolidate what you’ve learned.

7-11: Well done – you’ve passed the test. However, you might need to look through some sections again. You’ve obviously understood a large part of this unit, but you should make sure you are clear about all aspects of this part of the course. Don’t go on to the next unit until you are clear about this one.

12-16: Congratulations! You’re doing very well so far. If you want more things to study, have a look at the other sections in this course, by clicking on comunidad. If you are interested in studying abroad, click on servicio lingüístico. Keep on working hard and good luck!

17 & 18: Make sure you send the e-mail and voice message. I’ll get back to you about them as soon as I receive them. Don’t forget to reply to any messages you receive.

19. Did you check your pronunciation? How does it compare to the model pronunciation in the Audio Diary. Get back to me if you have any doubts. You can record any words you are not sure about and send them to me in a sound file and I’ll let you know my opinion. You can also send the same files to your study-mates to see what they think as well.

UNIT 37 TECHNICAL ENGLISH – SOUND TECHNOLOGY

It’s that time again – The Self Evaluation Test. You can only do your best!

1. Which word is the odd one out?

    1. amplified
    2. amplifier
    3. noisy

2. What does ‘faint’ mean in relation to sound?

    1. That a sound is very loud
    2. That a sound is of low intensity
    3. that a sound is silent

Listen carefully:

3. What is the sound? Click on the right option.

a. Loud

b. Low

c. Fuzzy

Listen carefully:

4. What’s wrong with this cassette?

a. It’s fuzzy

b. It’s faint

c. There’s no bass

What’s the word?

5. A device that uses lasers to produce sound

a. stand

b. CD player

c. speaker

6. A device which converts electrical energy into sound waves

a. stand

b. a cassette deck

c. speaker

7. Surround sound is…..

a. spatial

b. spacey

c. special

8. Two optical tracks on the film to create four distinct channels of sound

a. Dolby

b. Matrixing

c. THX

Writing an E-mail

9. Electrical equipment that didn’t break would be like a dog that didn’t bark. Stereo systems, of course, are no different and every major retail store has a customer service department where broken or faulty equipment can be returned for repair.

ACTIVITY ONE

You’ve just accepted a job in a shop which sells audio equipment. It’s your first day at work and you’ve been left in charge of customer services. There are five pieces of faulty equipment. In order to make sure everything is repaired correctly you have to package the broken items and enclose a note explaining what you think may be wrong with each system. But all you’ve got to guide you are the sounds of the broken equipment, and some comments from their owners. Click on each picture to hear what’s wrong – and DO read the comment accompanying it. Then select the best word to describe the problem.

(Learners click on buttons to hear 5 different sounds indicating what is wrong with machine.)

Instructions for the tape The recordings have the following specifications of sound.

Recording 1- Is fuzzy and not clear

Recording 2- Lacks some bass

Recording 3- Has a lot of background noise

Recording 4- Has almost no sound at all and is very faint

Recording -5 Very loud

1. Model Number: Sanyo 635

Customer complaint: When I put on a CD it sounds a bit funny.

2. Model Number: Marantz pm-57

Customer complaint: There don’t seem to be any low frequency sounds.

3. Model Number: Harman HD 710

Customer complaint: When I use the head phones there is a lot of hissing in the background.

4. Model Number: Sony SY-767

Customer complaint: The speakers seem to be of very low intensity. In fact I am not sure they work at all. Do you think the sound has been switched off ?

5. Model Number: Panasonic PN323

Customer complaint: When I listen to anything on the tape deck, CD player or radio, it hurts my ears and I think it may cause damage to my ear drum.

Activity Two

Now that you’ve worked out what the customer is trying to say, you have to package and write a short e-mail to the repair man, Jim*. Quote the model number and summarise what the problem is with each system.

* Jim can be one of your study-mates.

  • You will probably receive an email commenting on a similar problem from another study-mate. If there is anything you don’t understand in the e-mail you receive write a reply and send a copy to me.

Pronunciation & Vocabulary

10. You should record all the new vocabulary in your Audio Diary and check your pronunciation with the model pronunciation in the Audio Dictionary.

TECHNICAL ENGLISH FEEDBACK

POINTS TO THINK ABOUT

1. b Amplifier is a machine which makes sounds louder. The other two are adjectives which describe sound. If you didn’t know that revise the Vocabulary and Language Functions of this unit.

2. b If you didn’t know the answer was “of low intensity” have a look at Language Functions again.

  1. a If you didn’t know the answer and can still hear go to the Listening Section again and the Language Functions.
  1. c If you didn’t know there was no bass on that tape go back to Communication Activities and listen to the faulty cassettes.
  1. b Go and look at the Vocabulary section again if you weren’t sure about that.
  1. c As above!
  1. a You’ll have to look at the Reading section again if you didn’t know that surround sound comes from spatially distinguishable sources.
  1. Dolby did indeed invent this system but it’s called “Matrixing.” If you guessed the right answer okay…. But why not go back and look at the Reading section again?
  1. Activity 1

a) There isn’t any bass

b) The sound is very faint.

c) There is a lot of background noise.

d) The sound is very fuzzy.

e) The volume control button is broken and everything is so loud it’s deafening.

Activity 2

Reflect on any replies you receive from study-mates. If you have any questions for me do not hesitate to get in touch.

10. Did you check your pronunciation? How does it compare to the model pronunciation in the Audio Diary. Get back to me if you have any doubts. You can record any words you are not sure about and send them to me in a sound file and I’ll let you know my opinion. You can also send the same files to your study-mates to see what they think as well.

FINAL REFLECTION & FEEDBACK

Learning Diary:

We are now coming to the end of this level (B2) It is time to reflect on your overall progress. Please read these summaries of what a typical learner should be able to do at your level. Think about both General English and Technical English.

Record your responses in your learning diary. If you think you need to talk to me about anything we can arrange a chat. If you feel you are ready to move onto the next level after doing this reflection please let me know.

UNDERSTANDING

Listening: I can understand extended speech and lectures connected with my profession and follow even complex lines of argument provided the topic is reasonably familiar. I can understand most TV news and current affairs programmes. I can understand the majority of films in standard dialect and technical videos about my work.

Reading: I can read articles and reports concerned with contemporary problems in which the writers adopt particular attitudes or viewpoints. I can also read most technical reports and articles pertaining to my profession. I can understand contemporary literary prose.

SPEAKING:

Spoken Interaction: I can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible. I can take an active part in discussion in familiar contexts, accounting for and sustaining my views. I can take part in business and technical meetings and am able to negotiate.

Spoken Production: I can present clear, detailed descriptions on a wide range of subjects related to my field of interest and profession. I can explain a viewpoint on a topical issue giving the advantages and disadvantages of various options.

WRITING:

I write clear, detailed text on a wide range of subjects related to my interests. I can write an essay or report, passing on information or giving reasons in support of or against a particular point of view. I can write e-mails and faxes. I can write letters highlighting the personal significance of events and experiences.

C1 LEVEL – YOUR NEXT LEVEL

This is the profile of a proficient user of the language at the next level. Read the profile and think about whether you are ready to attempt to move to this level. Think about what you need to do to achieve a similar profile.

<C1 – Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express him/herself fluently and spontaneously without much obvious searching for expressions. Can use language flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organizational patterns, connectors and cohesive devices.>

Further Self-Evaluation

If you want to self-analyse your language learning abilities in more depth go to this page:

DIALANG

http://www.dialang.org/english/

You will need to download the software but it is well worth it. Let me and your classmates know how it goes.

BIBLIOGRAPHY

Self-assessment checklists from the Swiss

version of the European Language Portfolio (2004)

http://culture2.coe.int/portfolio/documents/appendix2.pdf

The EAQUALS-ALTE European Language Portfolio (2004)

http://www.eaquals.org/about/portfolio.asp

DIALANG

http://www.dialang.org/english/

Cohen, Andrew D (1994) Assessing Language Ability in the Classroom. Boston: Heinle & Heinle

http://www-writing.berkeley.edu/TESL-EJ/ej03/r12.html

Oscarsson, M. (1984). Self-Assessment of Foreign Language Skills: A Survey of Research and Development Work. Strasbourg: Council of Europe.
Oscarsson, M. (1997) “Self-Assessment of Foreign and Second Language Proficiency”. In The Encyclopedia of Language and Education, Vol. 7. Kluwer Academic Publishers, pp 175-187.
Oscarsson, M. (1998). “Learner Self-Assessment of Language Skills”. IATEFL TEA SIG Newsletter, Nov. 1998