CADETS’ PERCEPTIONS TOWARDS THE AUTHENTICITY OF SUMMATIVE TEST

https://media.istockphoto.com/vectors/seaman-in-white-uniform-vector-id472729602

Abstract

The demands for getting a good job for cadets in voyage schools becomes the main reason to learn English. Therefore, the institutions facilitate them by holding an additional program. Since they are emphasized on getting a job after graduating, the test must be authentic. This study purposes to investigate the cadets’ perceptions towards the authenticity of the new evaluation method on the summative test. A case study of qualitative design is chosen to analyze the transcribed data from the semi-structured interview. It involves four cadets of eleventh grade. The finding indicates that involving some experts of voyage and foreign people in the test already support the authenticity. Also, the new evaluation method effects positively on their motivation and future education and career planing. The researcher suggests that a continuation of the implementation of this study will be more detail not only on the authenticity but also on validity, reliability, and practicality.

Keywords: authentic test, summative test, vocational high school, voyage school, students’ perceptions, ESP.

INTRODUCTION

To get a better job, the cadets in voyage schools are obligated to be able to communicate in English fluently. To facilitate them, the institutions hold a non-based curriculum teaching and learning program because the classroom teaching and learning activity does not cover the needs. It is also aimed to habitualize them with English and the real situation of their future career. The coaches use free teaching methods and emphasize on the result. However, it becomes a bad program when the evaluation method is inappropriate and does not reveal the cadets ability and progress. Hughes (2003, p.124) stated that if we want to test oral skill, we must ask the students to speak.

However, a program in one voyage school evaluated the cadets’ speaking ability by translating 10 Bahasa Indonesia expressions into English and 10 English expressions into Bahasa Indonesia in writing whereas they were drilled in speaking. After being revised, the evaluation method obligated the cadets to communicate and emphasized on its authenticity to help them get used to the real situation of sailing. According to Carr (2011, p. 20), authenticity refers to how the test corresponds TLU (Target Language Use) or reflects the real situation. In short, a test is authentic when it mixes the theories and the skills (Alkubaidi, 2010, p. 3). Also, Spolski (1985) in Alkubaidi (2010, p. 3), stated that the material of an authentic test should be adapted from main sources. Relating to it, they adapted the material from IMO (International Maritime Organization). In short, the new method of summative test is considered more authentic than the old one (translating method).

Bachman (1991, p. 671) proposed authenticity through an international model which links both the test method's characteristics with the actual language proficiency being tested. Authenticity is a term that cannot be achieved for certain. Furthermore, Bachman & Palmer (1996, p. 23) stated that it is the 'degree of communication', to which the test task represents the actual taught language abilities that simulate common responses from testers. This can be attained in language classes when the cadets communicate with one another, requesting directions for instance as a given task previously taught in class. On the other hand, McNamara (1996, p. 11) argues that a performance assessment is authentic if it resembles real-life and promotes real-life responses.

Therefore, the new evaluation method of summative test in the voyage school has already been in line with those arguments. It resembles the real situation such as interview for the board internship, interview for the higher education, explaining to tourists, describing the things and their functions, and explaining the steps to use a certain thing.

To support the data, the researcher conducted an observation on the day of the summative test. It was done to frame the differences between the old and the new material. Since the researcher was a coach on the old evaluation method, she did not need another observation. However, an interview was addressed to the chief of the coach regarding the changing of evaluation method. It was obtained that improvement was done to habitualize and prepare the cadets for their career. Thus, authenticity is firstly emphasized.

Suryaningsih (2014) investigated the students’ perceptions towards TOEFL and IELTS. She divided the aspects to be analized intro three; on the tests, in the tests, and effects of the tests. First, participants’ perceptions of the on test aspect highlighted that technology and environment in the TOEFL test were not supportive for the test takers. In contrast, IELTS test provided test takers with a more comfortable and convenient environment and technology. Second, participants’ perceptions of the in test aspect emphasized that the TOEFL test did not measure test takers’ language ability due to its impracticality. Participants also expressed their opinion that the TOEFL test did not intend to measure language ability, but rather knowledge of a subject. However, their perceptions towards the IELTS test were the opposite. Finally, participants’ perceptions of the effects of tests aspects pointed out more negatives than positives, referencing gatekeeping, fairness, ethicality, and morality.  From her study, it is proved that even the standardized test like TOEFL and IELTS could provide different views from the test takers’ perceptions’

Regarding the students’ perception on the test, Struyven, Dochy, & Janssens (2005) conducted a study to investigate the students’ perceptions about evaluation and assessment in higher education. It was found that learners’ perceptions towards the assessment and the approaches to learning are strongly correlated. The type of the assessment influences on their effort to prepare themselves. However, this impact was not always positive. The inappropriate type of assessment will encourage the surface  approach to learning. Thus, this study suggested that the surface approach to learning is simply created since the deep approach is more complex to be promoted.

This big changing influences the cadets preparation not only on the material but also mental. It means that they realize the difficulty level and the authenticity between the old evaluation and the new one. The institutions tried to adapt the real situation of their future career such as continuing their education, being interviewed for applying a job in a big ship, and speaking to foreigners. Authenticity is different on each person (Gulikers, Bastiaens, Kirschner, & Kester, 2006, p. 4). Since it is decided authentic by the coaches, this study tries to figure out the authenticity based on the cadets’ perceptions. Also, they have fewer experiences in testing and sailing and might also differ to how they perceive authenticity from the test takers’ point of view. Thus, the research problem of this study is how do the cadets view the authentic test?

RESEARCH METHODOLOGY

The cadets’ perceptions towards the new authentic summative test can only be described through words rather than numbers (Ary, Jacobs, Sorensen, & Walker, 2010). Thus, this study is more appropriate to be framed under qualitative design rather than quantitative design. Moreover, it is categorized as a case study research which aim is to gain information related to the current phenomena (Postlethwaite, 2015). Also, it is constructed to expose the parts which are considered very authentic and motivated and attractive from the cadets’ perception (Cohen, Manion, & Morrison, 2007). The other purpose is to explain the phenomena in particular (Ary et al., 2010, p. 29).

Before taking the data for this study, the researcher did a preliminary observation on December, 5th 2017 to have an overview towards the technical stages of the new evaluation method in summative test. It was the day of summative test. It was conducted in one room in the cadets’ boarding house. There were 72 cadets do the test. Also, the researcher gained some information about their preparation and feeling before facing the test from uncontsructed interview. The researcher did not observe the old evaluation method because she used to be a coach a year ago. Thus, to compare the similarities and to contrast the differences, she only needed the information of the new evaluation method.

This present study involved one vocational high school of voyage in Surabaya as the field of this study. This school was chosen because it is one of voyage schools which hold a regular and routine activity for the cadets in order to make them use English communicativey. It is also the only school which hold a non-curriculum teaching and learning program in the boarding house. Thus, the cadets will keep practicing to speak English although the class ends. In short, this program is held to facilitate the cadets who stay in the dormitory.

There were 3 male cadets and 1 female cadet of eleventh grade participated in this study. Two cadets (A1 and A2) come from class A and the rests (B1 and B2) are from class B. The A1, A2, and 2 are male cadets and the B1 is a female cadet.They were chosen to represent their population. Cadets A1 and B1 are the good learners in their classes while cadet A2 and B2 are the lower learners in their classes. These classifications were obtained from the coaches based on their progress since they were in the tenth grade. Also, the researcher was not able to present the equal number of cadets’ gender because there is only one female cadet of eleventh grade who stay in the dormitory.

As a consideration, the eleventh grade cadets were chosen because they already experience the old evaluation method in the previous semester. Therefore, it can be used to help them compare the similarities and contrast the differences between the old and the new evaluation method. However, the tenth graders have not experience any summative test of this program yet. Moreover, the twelve grade cadets did not join this program anymore because they have to prepare themselves for the final examination and for the internship on board.

To collect the data, interview was chosen as the instrument of this study. It was conducted to answer the two research questions related to the cadets’ view on the new evaluation method of summative test involving the parts which they think authentic and the parts which they think attractive and motivated. Ary, Jacobs, & Sorensen (2010, p.29) proposed several methods in obtaining information. According to Berg (2001, p.71), the interviewers are permitted to probe far beyond the answers to their prepared and predetermined questions. This study specifically utilized face-to-face semi-structured interview. It was chosen because it is considered flexible Also, this is presented with a certain intention (Burgess, 1984, p.5 in Heigham and Croker, 2009, p.183). There were four formulated questions yet the researcher can give additional question towards the unexpected responses given  (Given, 2008, p. 62). The interview was recorded and transcribed to be presented in the result and discussion part.Besides, eight self-construct questions were addressed to the cadets to answer the two reasearch questions of this study and developed based on the relevant theoretical framework. The type of interviews were chosen in order to obtain more information by presenting additional question related to the unexpected response (Given, 2008, p. 62).

There were some steps taken in accomplishing this present study. First, the consent both from the school in which the program was observed, and the interviewees were obtain prior to data collection. The program observes was not the teaching and learning activity but the first session of summative test on December 5th 2017. It was done to obtain some information regarding the technical stages of the new evaluation method in the summative test. She obtained much information regarding the technical stages. It seemed like the program had improved a lot. Then, the interviews were conducted on December 7th 2017 after the summative test was done because the researcher still formulated the questions regarding to the observation result. In order to strengthen the information and avoid the cadets’ nervousness and rigidity, a semi-structured interview was conducted.

The analysis is based on the data from the interview. Since qualitative design was chosen, the analysis was done through description. First, the researcher conducted the interviews to the four cadets and those were recorded. Secondly, the recording was transcribed and organized into some classifications. Then, it was analyzied descriptively based on its content so that the required information can be drawn. Finally, some related researches and theories were presented to strengthen the findings.

FINDING AND DISCUSSION

Based on the cadets’ perceptions, in which part do the cadets think that the test is authentic?

The following data are the responses regarding the part that the cadets think the test authentic. Most of the respondents think that invoving some experts of voyage and foreign people resemble the real situation. The questions given by both the sailors and the foreigner replicated the real situation. The responses are presented below.

A1 :	The test is more serious than the last one. There was a sailor test me and the questions are like applying a job. Maybe he is a captain on a board. ... The questions about activity on board like maintenance and direction. He also asked me where do I want to work after graduate from here and about continue my school in Poltekpel.
A2 :	There are many strangers. I think that some of them are captains or people work in ship. There is also a foreigner and she asked me about my job. Then, I explain to her. She also asked about some things on board, the name and the function.
B1 :	The situation is like the real situation on board. If in the classroom I learn about the theory, in the test I must show what I can do. I must know the name of equipment, the function, and how to use it. ... The real situation make me nervous and I think that someone who test me know well about sailing.
B2 :	There is a foreigner and this is my first time speak with foreigner. I am very nervous and forget what I aready memorized. Maybe it’s the part that the test look like the real situation. I must speak with people from other country if I sailing to abroad one day.

The result of the interviews correlate with McNamara (1996, p. 11). The cadets think that resembling real-life and promoting real-life responses make the test authentic. That was proved by involving the experts of voyage and foreigners who pretended to be tourists on board. The response from B2 also indicates that the pressure and the anxiety place himself in a real situation of sailing. Also, the anxiety of B2 supports the findings of Nadeem, Ali, Maqbool, & Zaidi (2012). Since he is considered a poor learner, the anxiety level is higher than the good learners. The general finding shows that the higher the IQ level, the lower the anxiety level, vice versa.

Based on the cadets’ perceptions, how does the authentic test positively affect the cadets’ progress and motivation?

The following responses are addressed to answer the second research question about how the authentic test positively affect their progress and motivation.

A1 :	The captain motivates me to gradate soon and study English well because it can help me to get a good job. I want to be a great sailor, travel the world, and work in oirl tanker, and get big salary.
A2 :	I am motivated to evaluate my speaking and pronounciation because the woman don’t understand what I speak. So, I want to practice harder every day.
B1 :	The test becomes a media to measure my ability in speaking and explaining something. I also get some advices to be self-confident and to practice more frequently.
B2 :	It influences my mental preparation to face the test. I was very shocked what the foreigner tested me. It gave me more pressure. I usually can speak English with my friend in the dorm but I’m very nervous and shy to speak with her. But after that, I become confident.

Although the atmosphere should be serious and gave more pressure to the cadets, the interviewee already build rapport by being open and friendly as proposed by Gerhard (2009). Since B2 was anxious in facing the foreigner, Brosan et al. (2010, p. 7) consider that it is a good chance to build a rapport and turn it into motivation. The motivation and advices encourage them to learn and practice harder than before. Also, Aritonang (2014) stated that motivation affects positively to confidence. Therefore, motivation gives them a belief that they are able to do something. The response of B1 supports the finding of Baleghizadeh and Masoun (2013) that self-assessment on a formative basis in an EFL setting leads to increased self-efficacy.

CONCLUSION AND RECOMMENDATION

This study is designed to figure out the cadets’ perspective of the new evaluation method of summative test. The basis that motivated the investigation was the emphasizing on the authenticity on the test.  The findings indicated that involving some experts of voyage and foreign people in the test already support the authenticity. Also, the new evaluation method effects positively on their motivation and future education and career planning. Therefore, based on those findings, it can be concluded that the cadets recogize strongly about the authenticity of summative test and well motivated. The researcher suggests that a continuation of the implementation of this study will be more detail not only on the authenticity but also on validity, reliability, and practicality.

In addition, this presents study offers recommendation to consider. First, it is imperative for teachers or coaches to have comprehensive knowledge about making a good test especially for vocational high school students (ESP). Second, this recommendation goes to those who will conduct further research in the same area. As the present study only focused on cadets’ perspectives with small number of participants, more comprehensive study on different perspectives and focuses with more representative sample will be beneficial to be conducted. Also, this study will be better to conduct more detail not only on the authenticity but also on validity, reliability, and practicality.

REFERENCES

Alkubaidi, M. A. (2010). Authenticity in Language Test Design. Research Gate, (March), 1–12.

Aritonang, M. 2014. Motivation and Confidence of Indonesian Teachers to Use English as a Medium of Instruction. TEFLIN Journal, Volume 25, Number 2. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=DB61DFBF66EF403AAA9BF3E10C3BB3FE?doi=10.1.1.658.5134&rep=rep1&type=pdf

Ary, D., Jacobs, L. C., Sorensen, C. K., & Walker, D. A. (2010). Introduction to Research in Education (8th ed.). Belmont: Nelson Education Ltd.

Bachman, L. F. (1991). What does language testing have to offer? TESOL Quarterly, 671-704.

Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Oxford University Press.

Baleghizdeh, S., and Masoun, A. 2013. The Effect of Self-Assessment on EFL Learners’ Self-Efficacy. TESL CANADA JOURNAL/REVUE TESL DU CANADA Volume 31, issue 1, 2013. http://teslcanadajournal.ca/index.php/tesl/article/viewFile/1166/986

Berg, B. L. (2001). Qualitative Research Methods for the Social Sciences. USA: Allyn & Bacon.

Brosan, L., Hogan, B., Flight, E., Freyland, J., Kult, F., Lightning, E., & Heeswyk, R. Van. (2010). Coping with Anxiety. Cambridge: NHS.

Carr, N. T. (2011). Designing and Analyzing Language Tests (First Edit). Oxford: Oxford University Press. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.322.8417&rep=rep1&type=pdf

Cohen, L., Manion, L., & Morrison, K. (2007). Research Methods in Education. British Journal of Educational Studies (Vol. 55). https://doi.org/10.1111/j.1467-8527.2007.00388_4.x

Given, L. M. (2008). The SAGE Encyclopedia of Qualitative Research Methods. (L. M. Given, Ed.), The Sage Encyclopedia of Qualitative Research Methods (1st&2nd ed.). United States of America: SAGE Publications, Inc. https://doi.org/10.4135/9781412963909

Gulikers, J. T. M., Bastiaens, T. J., Kirschner, P. A., & Kester, L. (2006). Authenticity is in the eye of the Beholder. Journal of Vocational Education and Training, X(X), 1–160. https://doi.org/10.1080/13636820802591830

Hughes, A., 2003. Testing for Language Teachers. Second Edition. Cambridge. Cambridge University Press.

McNamara, T. F. (1996). Measuring second language performance. New York: Addison Wesley Longman.

Nadeem, M., Ali, A., Maqbool, S., & Zaidi, S. U. (2012). Impact of anxiety on the academic achievement of students having different mental abilities at university level in Bahawalpur (southern Punjab) Pakistan. International Online Journal of Educational Sciences, 4(3), 519–528. Retrieved from http://www.acarindex.com/dosyalar/makale/acarindex-1423904276.pdf

Postlethwaite, T. N. (2015). Educational research: some basic concepts and terminology.

Ramirez, V. A. C., 2010. Students’ Perceptions about the Development of Their Oral skills in an English as a Foreign Language Teacher Training Program. A thesis of Universidad Technologica De Pereira. Colombia.

Struyven, K., Dochy, F., & Janssens, S. (2005). Students’ perceptions about evaluation and assessment in higher education: a review. Assessment & Evaluation in Higher Education, 30(4), 331–347. https://doi.org/10.1080/0260293042000318091

Suryaningsih, H. 2014. Students’ Perceptions of International English Language Testing Systems (IELTS) and Test of English as a Foreign Language (TOEFL) Test. Theses and Disertations (All). 1235. http://knowledge.library.iup.edu/etd/1235