Case study: The CERCLU Project: Certification of Language Competence in Italian University Language Centres

© Maurizio Gotti and Carol Taylor Torsello

Table of contents

• Background
• Description of the Project
• Perspectives for CERCLU
• Bibliography
• Related links

Background

In 1999, the Italian University system underwent a significant process of reform, one aspect of which made knowledge of a second EU language compulsory for all students in order to obtain their university degree. Universities are obliged to ascertain that each student has achieved a required level of language competence. The type, level and evaluation of the competence required, are, however, left to be established locally.

In view of this reform, the Italian Association of University Language Centres (AICLU) immediately decided to set up a small working group to prepare a common system for testing and certifying students' language competence in relation to the Common European Framework of Reference for Languages (CEF), and gave it the name CERCLU (Taylor Torsello 2001; Gotti 2002). The group, comprising Language Centre Directors from the Italian universities of Padua, Bergamo, Bologna, Siena, Cagliari, Verona, Trieste, Rome 3 and the Rome Institute for Motor Sciences, began work in September 2000.

Description of the Project

The levels to be certified are those established by the CEF as Level B1 (Threshold Level) and Level B2 (Vantage) (Council of Europe 1998 , 2001a , 2001b). Since they are based on targets fixed at a European level, the tests resulting from this project can be administered in different environments and with different purposes: as entrance tests, placement tests, pass tests, evaluation at the end of a course for which the objectives set correspond to the same European levels, external evaluation of the quality of language learning.

A database of validated testing items has been created to be used, following established protocols, to create the tests to be administered in the individual centres. These protocols have been based on the models provided in Bachman & Palmer (1996), adapted to the situation of Italian University language teaching. Each item has a score so that the candidates examined can be given clear information about their position relative to the level of the test they have taken. For both of the levels available, modules have been created corresponding to the four communicative skills (reading, writing, listening and speaking) and these can be evaluated separately. The choice of the possibility of using the modules separately as well as in combination derives from the specific reality of Italian universities, where skills are not always required at the same level. Clearly, however, the CERCLU certification will be awarded only on the basis of the results on all four skills, whereas scores can be supplied, for institutional purposes, on the individual modules. CERCLU, in fact, caters for the specific needs of students in Italian universities, which means that the texts, topics, and activities are chosen on the basis of relevance to such students and to their situation. Furthermore, some differentiation is introduced between tests according to the disciplinary macro-area of the candidate: humanities, social sciences and physical sciences.

Although at present the certification system is limited to English and Italian, it is hoped that in the future it will be possible to extend it to other languages. The fact of basing the levels of testing and certification on the European parameters makes it possible to insert the CERCLU tests into a system of recognition along with tests created in other countries for which the same parameters (based on the Council of Europe levels) are adopted. The CERCLU certification issued by Italian language centres is meant to be recognised by language centres members of the European Confederation of Language Centres in Higher Education (CERCLES) in other European countries. The CERCLU project has, in fact, been developed in Italy in close connection with analogous projects developed by other members of CERCLES, and agreements are being negotiated whereby the national associations will promote recognition of CERCLU certification, and also of certification developed by other CERCLES members which are similarly based closely on the descriptors of the CEF.

This project marks the beginning of a process of internationalisation in university language testing which, it is hoped, will be further strengthened by adding new agreements with partners in other countries. It is also hoped to acquire further funding, making use of opportunities such as those offered by the European Union.

From constructs to prototype

After an initial stage of research on testing and certification in 2000-2001, and of general planning of the action to be undertaken, the AICLU pilot group devised a set of constructs and created a complete, working prototype test for B1 and B2 levels both for English and Italian.

The first concern of the CERCLU project researchers was to construct tests tailored to the needs of their specific situation (Alderson, Clapham & Wall 1995:11-12, Bachman & Palmer 1996:88). For both levels, B1 and B2, the operative distinction into subtests linked to the four traditional abilities reading, listening, writing, speaking - has been kept. For each level the characteristics of the authentic texts to be used have been identified. Levels B1 and B2 have also been differentiated on the basis of the cognitive complexity of the elaboration process the candidate would be required to perform (Brown & Hudson 2002:216), as illustrated in the descriptors found in the CEF (Council of Europe 2001a: 65-72). Another differentiating criterion was the distinction between degrees of formality: in the productive tasks, the informal register required in the students' communication with their peers as opposed to the more formal register required in tasks involving interaction with non-peers is seen as distinguishing between levels B1 and B2.

Our method of operation was that of transforming the CEF statements into a construct of language input and expected response for both levels and both languages, as summarized in Figures 1 and 2, where the type of input given and the type of performance required from the candidate are described for each module of the tests (for further discussion see Taylor Torsello & Ambroso forthcoming).

Figure 1 - B1 level: input and output

Figure 2 - B2 level: input and output

Once the types of input and output had been determined, the AICLU pilot group produced the prototype tests, taking into consideration the number and type of questions to present to candidates in order to have a significant sample of their level for each ability, the time to allow the candidates to perform each task, and the relative weight to give the various tasks and the modules for the four abilities . The resulting schemes are summarised in Figures 3 and 4 .

Fig. 3 - Structure of the CERCLU tests in English and Italian for B1 level

Fig. 4 - Structure of the CERCLU tests in English and Italian for B2 level

The tests described above were created using QuestionMark Perception and administered to the first sample groups of candidates. The software made it possible in many cases to mark responses automatically. Detailed procedural decisions had to be taken regarding the modules testing the productive skills, so as to be able to score them as objectively as possible. The following carefully defined and weighted parameters were set for both B1 and B2 levels:

Oral production:

• pronunciation
• grammatical resources
• lexical resources
• discourse organisation
• register and interaction
• fluency

Written production:

• lexical resources
• grammar
• cohesion
• content and organisation
• discourse appropriateness
• spelling and punctuation

It was decided that a proportion of marks would be assigned on the basis of the overall evaluation of whether or not the performance had achieved its communicative purpose.

Since the students who sat the CERCLU exams during the experimentation phases required information about their level, the pilot group decided that the percentage on each module indicating success in terms of actually being a B1 or a B2 in the language tested would be 60%, which became the pass mark. It was also decided that, in case a student reached the average score of 60% for the whole test, but with one or more individual modules below this, the CERCLU certification would not be granted. To be certified as B1 or B2, the candidate would have to pass on all four modules.

The analysis of the results of the first phase of experimentation allowed us to refine some initial choices made in relation to the above-mentioned points and to arrive at the solutions presented in Figures 3 and 4. In the points' column, the total number of points for each of the test parts in the reading and listening modules simply corresponds to the number of possible right answers, since these are machine-corrected. The writing and speaking modules, on the other hand, are scored directly in percentages by the assessors, who consider the performance as a whole, giving points on the basis of different language parameters rather than of different test items. Since each ability is considered equally important, the points given to reading and listening had to be turned into percentages in order to be weighted equally with writing and speaking.

Duration and phases of the Project

The project is an ongoing one that will require development, adaptation, and upkeep for all the time of its existence. However, the time anticipated for bringing it into operation was four years.

In the first year (September 2000 August 2001), information about the various systems of certification in use in Europe was acquired. A detailed bibliography on testing and systems of certifying language competence was elaborated. A web site was created for the materials collected and the documents produced by the group. The software (QuestionMark Perception) was selected for the writing, storage and retrieval of test items, for the production of tests according to the agreed protocols and for the analysis of the statistics relating to the results. The objectives related to the European B1/B2 levels and to the target language use domains and text types were closely examined, and the test constructs were defined.

In the second year (September 2001 August 2002), the design statement was elaborated and the elements necessary to consider an item adequate for a particular level and a particular objective were defined. The protocols for the creation of the tests were detailed. The hardware and software for the management of the database of items and of the tests themselves were prepared. Meetings were organised with our first non-Italian partner, Plymouth, to discuss the project and recognition of the certification. The prototypes were prepared and trials were run using groups of students carefully selected for level on the basis of other tests. Tables of criteria with relative points for scores were drawn up for tests of the productive skills, and instructions for the oral examinations were detailed. These tables and instructions were also tested. Reports on the results were written both at local and at national levels (discussion in Italian are available on the AICLU web site) .

In the third year (September 2002 August 2003), and on the basis of the results to date, the prototypes, the oral examination, and the tables for evaluation were modified where necessary. A further calibration was performed, using the tests of the Association of Language Testers in Europe (ALTE) as control tests for the corresponding levels of the CERCLU candidates. Once this second phase of testing was completed, the databank of items for all the test parts envisaged for the Italian and English certification was created. In September 2002 the project was presented at the CERCLES Conference in Paris (Taylor Torsello & Ambroso forthcoming) and in June 2003 it was presented at the AICLU Conference in Trieste (Gotti & Taylor Torsello forthcoming).

During the fourth year (September 2003 August 2004) the test-prototypes and the methods and results of their validation were examined in detail by Prof. Charles Alderson from Lancaster University, who discussed them with the CERCLU team in October 2003. Some changes have been made in the tests on the basis of Professor Alderson's comments. In particular, pull-down lists have been replaced by radio-button multiple choice items, the answering machine item in the B2 oral production test has been replaced by a peer-interaction item, and both oral tests have been shortened (B1 to 11 minutes and B2 to 12 minutes). A further phase of experimentation has begun. As new tests are created and added to the database, they are calibrated against the previous ones. At the same time, t he process of negotiating the recognition of the certification at an international level has continued.

At the end of the fourth year of the project it is intended to evaluate CERCLU's pentration and impact. CERCLU's contribution in applying that part of the university reform concerned with students' proficiency in language will also be evaluated. Finally, decisions will be taken on how to go about turning the CERCLU project into a system with an ever-increasing data bank of items, and one which can be made use of on a wide scale and can function on an independent and permanent basis. A book is being published on the project (Evangelisti, Gotti & Taylor Torsello forthcoming).

Perspectives for CERCLU

CERCLU has made significant progress since the idea was first conceived in 1999. But there is still a lot of work to be done. The CERCLU group must now undertake an intensive training campaign. Indeed, the training process will have to be an on-going part of the project from now on (Council of Europe Language Policy Division 2003:65-88). A permanent editing committee must be set up, with fixed procedures for checking and revising the tests produced by the item writers (Alderson, Clapham & Wall 1995:223-28). Finally, a certification system is a formidable enterprise to administer and manage, and cannot remain within the confines of the research interests of a group of university professors who are also language centre directors. Nonetheless, the group has agreed that before the end of 2004 some of the language centres involved in the pilot project would be ready to administer the CERCLU tests to small groups of students and the first CERCLU certificates for B1 and B2 levels would be issued. As the databank of test items grows and the certification system is consolidated, more and more Erasmus/Socrates exchange students visiting Italy will return to their countries with a CERCLU certification of B1 or B2 level in the Italian Language, and more and more students from Italy will arrive in European universities with a CERCLU certificate as proof that they are level B1 or B2 in English.

With CERCLU, an important step toward the internationalisation of the Italian university system has been taken, which should contribute to greater mobility within Europe. It is hoped that European funding might be acquired for the further development of the project. The initial recognition agreement signed with a small number of UK universities has been seen as an anticipation of wider recognition within Europe, to be obtained especially thanks to the network of language centres created by CERCLES, and to the positive interaction between the national associations of language centres. Among the types of inter-association collaboration envisaged, two promising possibilities are: 1) agreements for reciprocal recognition of CEF-based certification systems produced and endorsed by national associations; 2) agreements for development of the CERCLU system in a European project that would permit the rapid expansion of the data bank for English and Italian and the extension of the system to cover other European languages.

Bibliography

Alderson, J.C., C. Clapham & D. Wall (1995). Language Test Construction and Evaluation. Cambridge: Cambridge University Press.

Bachman, L.F. & A. Palmer (1996). Language Testing in Practice . Oxford: Oxford University Press.

Brown, J.D. & T. Hudson (2002). Criterion-referenced Language Testing . Cambridge, Cambridge University Press.

Council of Europe (1998). Threshold . Cambridge: Cambridge University Press.

Council of Europe (2001a). Vantage . Cambridge: Cambridge University Press.

Council of Europe (2001b). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press.

Council of Europe Language Policy Division (2003). Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF) - Manual: Preliminary Pilot Version (Council of Europe Document DGIV/EDU/LANG (2003) 5). Strasbourg: Council of Europe.

Evangelisti, P., M. Gotti & C. Taylor Torsello (forthcoming). Il Progetto CERCLU per la certificazione delle competenze linguistiche nei centri linguistici universitari . Roma: Carocci.

Gotti, M. (2002). La valutazione delle competenze linguistiche in un'ottica europea: il Progetto CERCLU. In C. Taylor Torsello, M. Catricalà & J. Morley (eds.), Anno europeo delle lingue: proposte della nuova università italiana , 27-40. Siena: Terre de Sienne editrice.

Gotti, M. & C. Taylor Torsello (forthcoming). Il Progetto CERCLU: Work in progress e primi risultati. In Ch. Taylor, C. Taylor Torsello & M. Gotti (eds.), Atti del 3° Convegno Nazionale dell'AICLU. Trieste: Edizioni Universitarie Triestine.

Taylor Torsello, C. (2001). La Certificazione delle competenze linguistiche. Universitas , 79:26-29.

Taylor T orsello, C. & S. Ambroso (forthcoming). The CERCLU Project for Certifying Language Competence in Italian University Language Centres. In Satchell R.& N. Chenik (eds.) University Language Centres: Forging the Learning Environments of the Future . Paris: CercleS.

Related links

AICLU - Associazione Italiana Centri Linguistici Universitari.
www.aiclu.it/

AICLU CERCLU validation data
www.aiclu.it/cerclu/nuovo/validazionedoc.pdf

CERCLES - European Confederation of Language Centres in Higher Education
www.cercles.org/

CERCLU - Linguistic Competence Certification in Italian Universities
www.aiclu.it/cerclu

Council of Europe - Modern Languages: Learning, Teaching, Assessment. A Common European Framework of Reference
www.coe.int/T/E/Cultural_Co-operation/education/Languages/Language_Policy/default.asp#TopOfPage

QuestionMark Perception
www.questionmark.com/uk/tryitout_features.htm

Referencing this article

Below are the possible formats for citing Good Practice Guide articles. If you are writing for a journal, please check the author instructions for full details before submitting your article.

• MLA style:
  Canning, John. "Disability and Residence Abroad". Southampton, 2004. Subject Centre for Languages, Linguistics and Area Studies Guide to Good Practice. 7 October 2008. http://www.llas.ac.uk/resources/gpg/2241.
• Author (Date) style:
  Canning, J. (2004). "Disability and residence abroad." Subject Centre for Languages, Linguistics and Area Studies Good Practice Guide. Retrieved 7 October 2008, from http://www.llas.ac.uk/resources/gpg/2241.