Ms. Tran Thi Thanh Mai, an EFL lecturer at Van Lang University, has been teaching English for more than fifteen years. As an EFL lecturer and a material writer, she has published extensively on various EFL issues both nationally and internationally. Her current professional interests include techniques and approaches in teaching English skills and language units, literature learning and play activities. Affiliation: Faculty of Foreign Languages, Van Lang University, Vietnam; email address: mai.ttt@vlu.edu.vn

Ms. Võ Hồng Ngọc, an undergraduate student at Van Lang University, has great interest in AI application and its effect. She aims to hone her skills with a master degree in the upcoming years. Email: ngoc.207na68953@vanlanguni.vn
 

Abstract

Technological advancement has brought about dramatic changes in the face of education over the years. Noticeably, ChatGPT, a novelty and cutting-edge tool, contributes to alleviating workload problems faced by educators and providing support in various domains. The objectives of the study are to examine the effectiveness of ChatGPT on testing and assessment in the English subject at Tan Hiep Secondary school, and to learn about English teachers’ thoughts on using ChatGPT for testing and assessing purposes. To reach the above-mentioned aims, a quasi-experimental design and a quantitative method were employed. The data were analyzed with SPSS 23 and Microsoft Excel. 88 seventh-grade students (particularly 45 students of class 7/3 and 43 students of class 7/2) during the second semester of the 2023-2024 academic year were assigned to control and experimental groups. Both groups took the same pre-test and two different post-tests (teacher-made tests and ChatGPT-generated tests, respectively) to figure out whether there was any disparity. Also, the survey was carried out through a paper-based questionnaire that was delivered to English teachers at the end of the experiment. The findings of the study indicated that there were no statistically significant differences in the pre- and post-tests between the control group and the experimental group. In other words, the result of the study revealed that tests generated by ChatGPT could classify students' academic levels and assess their competence as effectively as teacher-made tests. What is more, analysis of survey responses showed all English teachers at Tan Hiep Secondary school had an unenthusiastic attitude toward the implication of ChatGPT in testing and assessment aspects and did not expect to apply this AI tool in these areas in the future owing to some personal and technological reasons. 
 

Introduction 

Scientific knowledge is increasingly being advanced and is widely used in all aspects of social life in recent years, and education is no exception. To catch up with the trend, educators also have to actively enhance skills and build up knowledge to acquire new technologies. However, there are still a lot of challenges that teachers as well as educators meet in the teaching process. 

The appearance of Artificial Intelligence (AI) has been changing our future in remarkable ways shaping human development and existence. One field that has garnered huge concern and significant investment is education. Due to the current conventional curriculum, students find it boring and uninteresting to keep learning and acquiring knowledge. Similarly, Gürbüz, et al. (2017) claimed that the motivation of learners could be negatively affected by traditional methods or resources such as textbooks and lectures. As a result, language teachers have constantly been attempting to keep up with innovation in education trends, including applying technology in teaching, conducting modern teaching techniques for the purpose of increasing attractiveness and effectiveness in methods, and contributing various activities in addition to bringing learners new experiences coming from AI. Besides, it is possible for AI systems to create and provide learning materials that are customized to the needs and the ability of each individual (Raj & Renumol, 2022). 

With unexpected success, AI technology signified a major breakthrough in the digital age with the production of ChatGPT – one of the most valuable and potential applications was developed by the OpenAI company with models as a conversation that responds to information from humans. In the educational setting, it is useful for ChatGPT to support active learning and teaching. Thanks to its large amount of data and conversational capabilities, ChatGPT is assessed as an advantageous instrument for assisting students in providing knowledge (Qadir, 2022). Also, it helps students give quick feedback and exchange or discuss learning materials as a peer (Wang et al. 2023). Not surprisingly, ChatGPT is trusted and embraced by numerous instructors because of its diverse advantages.  A number of teachers’ tasks seem to be much more practical. They have to manage different tasks, consisting of lesson plans, activities, testing, assessments, etc. Teachers have just constrained time to complete all responsibilities (Ramanathan et al., 2022) which causes quick exhaustion and burnout that adversely affects the teaching quality. In fact, it's clear that all schools are not always able to meet good educational conditions such as technology, classroom materials, and so on (Akram et al., 2021). That shortage results in students not having the best learning experience. Hence, to create a better educational environment, teachers must struggle with limitations and difficulties. Furthermore, ChatGPT allows lecturers to save their time in creating quizzes, lectures, activities, and even comments from students. It is suitable to use AI in designing assignments, tests, and questions and answers (Sinha et al., 2023). Also, it may evaluate tasks and offer feedback, giving teachers time to design engaging lessons and gain their students more attention. The incorporation of intelligent technology into schools has broadly been used and is considered an essential method in the current context of education. On the other hand, there are still some negative impacts or debates on how to effectively use ChatGPT for educational purposes. Although ChatGPT is an advanced and revolutionary tool, potential risks can arise from using it incorrectly (Qureshi, 2023). Thus, educators must take into consideration its benefits as well as its risks when incorporating ChatGPT into education.

In line with the trend of integrating AI into teaching, there have been numerous studies focusing on many aspects of education ranging from its application in lesson planning and teaching activities to its impact on writing skill development, its notable benefits in higher education, its impact on learners’ motivation, its influences on students’ performance and behavior as well as its enhancement of computer-human interaction. Nonetheless, the amount of research on the efficiency of ChatGPT in testing and assessment remains limited, particularly in Vietnamese middle schools. Hence, this study, entitled “The effect of ChatGPT in testing and assessment at Tan Hiep Secondary school”, was conducted to examine the effectiveness of ChatGPT in testing and assessment, alongside delving into the viewpoints of English teachers at Tan Hiep Secondary school regarding the integration of ChatGPT in testing and assessment.

The following two research topics are intended to be discussed to accomplish the goal of the study: 

Research question 1: Does using ChatGPT have any significant effects on testing and assessment in English at Tan Hiep Secondary school?

Research question 2:  How do English teachers at Tan Hiep Secondary school respond to the use of ChatGPT in testing and assessment?

The null hypothesis for this study was tested at 0.05-level of significance, i.e. There will be no statistically significant difference in testing analysis score as measured by the periodic achievement tests between the experimental group and the control group.

 

Literature review

An overview of ChatGPT

Chat Generative Pre-Trained Transformer (ChatGPT) is a new artificial intelligence (AI) tool that uses the processing for natural language and innovative technology to create a remarkable ability of human-like conversational dialogue (e.g. Cotton, Cotton & Shipway, 2023; De Winter, 2023; Topsakal & Topsakal, 2022; Wenzlaff & Spaeth, 2022; Zhai, 2022, etc.). This special AI technology was released in November 2022, and developed by Open AI which has been a cutting-edge technology (Gilson et al., 2022). Moreover, ChatGPT is used as a virtual assistant that can respond or prompt to user input, thanks to its advanced capabilities as well as a large quantity of text data (Gozalo-Brizuela & Garrido-Merchan, 2023). In spite of its impressive performances, there are some limits that ChatGPT cannot support as a human, especially emotion. That’s because humans have experiences and emotions through thoughts and actions while ChatGPT runs on and analyzes data to offer the inquiries of the users. Therefore, it is impossible for ChatGPT to understand or experience emotions in the same way that humans do (Borji, 2023).

According to Mathew (2023), the advent of ChatGPT is generally defined as a versatile tool that changes the world and initiates a discussion of Generative Pre-trained Transformer (GPT) models over a six-month period. With the purpose of developing AI technology, OpenAI has been successfully promoting their ideas and has released an optimized language model that is trained on an enormous corpus of text data, assisting it to understand the patterns of natural language and produce appropriate replies (Greyling, 2022). Indeed, ChatGPT, an AI model is recognized as one of the most potent language models (Cotton, Cotton & Shipway, 2023) beyond the capabilities of ChatGPT, which achieve state-of-the-art performance on a multiplicity of various real-world scenarios, including humanlike conversation, reject inappropriate requests, recognize and learn from its own mistakes, and to name just a few (Jiao et al., 2023). Unlike search engines like Google, ChatGPT has unique features that contribute to success in a natural language processing model and popularity with a broad range of applications in various domains as well (Macdonald et al. 2023). Ray (2023) also points out that seven key features of ChatGPT are making waves of change in human-machine interaction. According to Macdonald et al. (2023), thanks to the separated features of comprehension within a given context, capabilities in language generation, task adaptability, fluency in multiple languages, zero-shot and few-shot learning, and fine-tuning, this advanced AI’s model makes a substantial contribution to its widespread popularity as well as represents a significant breakthrough in the AI (artificial intelligence) field. 

With the importance of AI ever increasing, it is expected to open a new and better future and shape the way society operates. One of the fields in which AI technology creates remarkable success and serves as a significant asset is the academic environment. The implementation of ChatGPT in education has garnered significant attention due to its potential to enrich learning and pedagogical experiences. The high technology contributes to instrumental support and is a considerable AI tool for teacher grading in essay writing (Babitha et al., 2022) although it remains unclear about the accuracy and efficiency of this AI-powered grading tool. 

 

The roles of ChatGPT in education

Numerous researchers have highly assessed the various functions of ChatGPT in an academic environment and a unique ability that few current tools have that is a remarkable and potential language ability. Some of the studies mentioned that there are many advantages of learning that ChatGPT supports, while others express concerns about the negative effects on the integrity of learning and the fairness in education (Yan, 2023; Shoufan, 2023). Therefore, ChatGPT may be a promising and beneficial tool if applied correctly for its intended usage and purposes. Particularly, writing skill is one of the skills that ChatGPT can support a lot for students. Students can enhance their writing skills and make their writing more effective by interacting with ChatGPT and receiving feedback along with suggestions for correction of grammatical mistakes and ideas for improvement (Osorio, 2023). In addition, the response from AI is always faster and saves more time than humans since AI can analyse, search, and give answers in seconds. Instead of the time-consuming process of checking and giving feedback by teacher, this technology directly provides comments and indicates writing issues immediately, which proves ChatGPT is a potential personal tutor for students (Wu et al., 2023). By generating immediate and personalized answers, ChatGPT has supported students in various ways and helped them boost their academic performance as well as improve their learning. With special ability, it facilitates expanding students’ knowledge by providing complex concepts, solving individual needs, and responding quickly. Following this line, it adapts their learning pace and supports necessary information and the process of acquiring knowledge (García Sánchez, 2023).

Furthermore, ChatGPT is considered a professional observer who finds out the weaknesses of the learning process and provides a particular plan for separated students. It also facilitates enhancing group discussion and engaging collaboratively with students in tasks or projects. This is a useful advantage for students who rarely interact with their partner or expect to work independently since AI has the same resources as their peers. Communication and idea-sharing are fundamental aspects of the learning procedure as a result of the strength of collaborating in groups among peers (Qadir, 2022). A wide range of research has been undertaken to discover the impact of ChatGPT on educational backgrounds, in particular some aspects such as predictive analytics, adaptive assessment, and adjusted learning environments. Consequently, it shows enormous potential for supporting learning and providing both personalized teacher and student assistance.

In an ever-changing society, the burden of the workload of educators is reduced significantly thanks to the advancement of AI in the future (Baskara & Mukarto, 2023). Because of this, they have more time to improve teaching quality with better lesson design, have a chance to observe individual students, and attend professional development events. All of these actions also contribute to the success of education in the next period. Additionally, student grading is no longer a concern when optimized by ChatGPT. By determining the frequent mistakes or errors students make in multiple assignments, ChatGPT can be utilized as a semi-automating process to grade students and detect both their strengths and weaknesses through their tasks (Kasneci et al., 2023). It has also been reported that ChatGPT's expectation is to automate and update scoring systems, help educators quickly identify problems or difficulties that students are facing and actively propose better solutions in order to help students improve academically. Moreover, it is true that human knowledge and creativity cannot be replaced by AI; however, that does not mean that it has no position in the context of informatics education (A. El-Seoud et al., 2023). Teachers can completely take advantage of the advanced features of ChatGPT to enhance pedagogical practices. According to Rudolph et al. (2023), educators are able to receive ChatGPT’s support in teaching methods and activities. Moreover, Shidiq (2023) expressed that AI was a pedagogical tool that teachers can apply in their classroom. Similar to the assistance of students through instructing and demonstrating, teachers completely have the ability to create an independent classroom where all students can freely access the resources outside of the classroom. Rather than concerning the utility of ChatGPT by students, educators should pay attention to how ChatGPT is used best in both teaching and learning, which is the best way to increase teaching quality and limit the academic misuse of ChatGPT for dishonest or unethical purposes. As a result, it is vital for instructors to effectively master the use of the tool and its limitations and potential obstacles to prevent as well as address the ethical and pedagogical challenges (García-Peñalvo, 2023).

Besides, ChatGPT serves as an easy-to-use and accessible tool for creating interactive activities, detailed lesson plans, and other teaching materials. According to Atlas (2023), the teaching aspects, such as quizzes, tests, and syllabi, are simplified thanks to ChatGPT's outstanding features. With this support, teaching methods as well as material resources become more creative and varied to meet the numerous learning needs of students. Since the teachers do not spend much time preparing, completing lesson plans, and other tasks at the same time, they have many opportunities to make their PowerPoint or lesson creative and attractive, and come up with teaching ideas. Therefore, it is also a multi-task assistant to support the creation of instructional materials, such as learning objectives and requirements needed to complete the course (Whalen et al., 2023).

 

Ethical considerations and limitation of ChatGPT

It is not denied that AI brings us a range of substantial benefits as well as solutions. ChatGPT, an AI-powered language model developed by OpenAI, has been making waves in both the tech world and the AI market (Rudolph et al., 2023; Ruby, 2023). However, apart from its undoubted benefits, the sophisticated AI model also contains potential downsides. Numerous pieces of research indicate that the incorporation of AI tools has opened new horizons in the educational context. On the other hand, others highlight an alarming trend concerning the use of ChatGPT in education.

First of all, plagiarism continues to be a top concern for educators due to the common usage of AI in writing. The misuse of AI machines inevitably leads to interventional cheating and plagiarism, and it is highly possible to provoke plagiaristic behavior. Bašić et al. (2023) indicated that assignments with the unauthorized use of AI-assisted writing, particularly ChatGPT, were greatly likely to be plagiarism. The written work being supported by virtual assistance is assessed similarly, as the student-generated work is not fair, and this can destroy academic integrity (Eke, 2023). Plagiarism is one of the disallowable behaviors in educational settings that violates six academic integrity standards: honesty, trust, fairness, respect, responsibility, and courage (International Centre for Academic Integrity, 2021). The teaching-learning process always emphasizes fairness, honesty, and integrity, so the application of Al writing technology to intentionally cheat has a huge impact on education. Therefore, educators may face difficulties in assessing students' learning, which is unavoidable. Cassidy (2023) is also concerned that there was a wrong evaluation that negatively influenced the learning process while using ChatGPT for cheating. Hence, an abundance of plagiarism applications and detection tools have been developed to assist educators in checking plagiarized material. To identify plagiarized text, this software detects content similarity through the analysis of sophisticated linguistic patterns using various methods (Khalil & Er, 2023). Nevertheless, it is not always possible to detect AI-generated content; there are numerous cases in which the detection cannot identify AI-generated text (Katz et al., 2023; Bašić et al., 2023) that impact the success of plagiarism-detecting tools and diminish faith in academic achievement.

In a world where AI has become an omnipresent force in nearly every aspect of our lives, instructors raise concerns about social biases and prejudices toward ChatGPT. According to Kasneci et al. (2023), the language model ChatGPT may have influenced social biases, prejudicial views, and preconceptions based on the data used for training. The quality of the educational process is increasingly improving, motivating, and collecting a lot of new knowledge that comes from human civilization. ChatGPT’s data is also collected from big data; thus, it can respond to most user requirements. On the other hand, the AI-powered tool includes certain limitations. The output that ChatGPT provides is still limited to 2021 information and has not yet been fully updated beyond this timeframe (Gilson et al., 2023). As a result, it unavoidably provides inaccurate or unreliable answers, especially on recent timelines or specific topics. In addition, ChatGPT may generate responses that are mistaken or even incorrect (Megahed et al., 2023). Due to limited data, AI-generated responses are untrustworthy since they are straightforward to duplicate with available sources, resulting in likely plagiarism in the academic environment. 

 

Testing and assessment

Testing and assessment are crucial and essential techniques in education that satisfy purposes and objectives aimed at promoting the teaching process, providing information to change teaching approaches, and making plans that are appropriate for students. 

In simplistic terms, collecting information related to knowledge, skills, and students’ attitudes toward learning as the basis for assessing students can be referred to as testing. According to Linn & Gronlund (1995), the test is described as a form of assessment that involves a series of questions conducted in a fair and appropriate manner for all students at a particular time. Furthermore, it is always unacceptable that there is no testing in teaching procedure (Das, 2018). Beyond being seen as a tool, tests can also be considered a type of assessment that measures individuals’ capacity, knowledge, and performance within a specific area (Brown & Abeywickrama, 2019). In the teaching-learning environment, assessment is one of the most frequent and major activities in the process of teaching. Actually, a number of educators, assessors, and other researchers as well as related materials such as articles, specialized books, and dissertations have contributed much to finding the concept of assessment appropriate (Ghaicha, 2016). According to Bachman (2004), there is no agreement on the precise concept of the term because it is commonly used with a multitude of various meanings in the field of language testing and educational measurement. Besides, Brown (2009) claims that the term assessment refers to actions of interpretation of measurement results obtained from a wide range of techniques and procedures. However, other researchers observe the term in other ways. Mousavi (2009) notes that assessment means appraising or measuring a person’s capacity while assessment is an analysis of the process of students' performance through observation of current progress and the ability to acquire something such as language or knowledge. Along with the ideas of the previous authors, Nagaraj (2010) adds that the performance of individuals or things needs to be assessed or categorized in all contexts, either academic courses or other situations.

The purpose of assessment is to assess student performance through tests or scores they achieve in order to identify how well they comprehend a particular area. In other words, it serves as a tool to determine the learning process, strengths, and weaknesses of each student. Moreover, the educators can use the test as an assessment device by assigning tasks or assignments since they can observe the administering procedure and provide assessments for individuals. Based on the observations, it is possible for the instructors to adjust teaching strategies to meet the learning demands of students effectively. Brown & Abeywickrama (2019) stated that the responsibility of a teacher was to give assessments to students subconsciously based on their answers, comments, or processes of administering tasks that facilitate following student progress, giving valuable feedback, and enhancing teaching effectiveness. Similarly, by measuring student performance through the results of the tests, the educators can understand and monitor the learning development of students and, as a result, assess and determine their ability. Furthermore, the teacher can measure specific skills based on particular tests, such as vocabulary tests, writing tests, etc., that help teachers discover the lack of knowledge and skill of individuals. Additionally, on the basis of the general knowledge domain the students achieve, the educators rank each student and identify distinctions between individual performers.

Overall, testing and assessment need to have a general purpose and specific objectives. For students, it assesses their results and indicates a lack of skill and knowledge, helping them to learn more actively and improving their learning quality. Besides, it also assists the educators to recognize the student strengths and weaknesses, take proper measurements as well as change and improve teaching methods to appropriate the level of students.

 

Research method

This study is quasi-experimental research with pre-test and post-test design. Based on the purpose of the paper, a quantitative approach was an appropriate choice for the researcher to measure the effects of ChatGPT in assisting testing and assessment in education as well as to find out English teachers’ perceptions of the use of ChatGPT in designing a test. To begin with, the research samples are 88 seventh-grade students, including 43 students of class 7/2 (served as experimental group) and 45 students of class 7/3 (served as control group) at Tan Hiep Secondary school, whose English competence was determined through the placement test (functioned as the pre-test). Therefore, the effect of ChatGPT on testing and assessing students’ English competence was based on quantitative analyses of the results of the pretest and posttest. The researcher would then make a comparison between the control group and experimental group’s results to find out whether ChatGPT can be as effective as teachers regarding testing and assessing students’ performance. What is more, after collecting the data from the tests, the researcher delivers the open-ended questionnaire to teachers with the aim of exploring their perceptions of utilizing ChatGPT in testing and assessment.

The study was conducted with the participation of 88 seventh-grade students at Tan Hiep Secondary school, particularly 45 students of class 7/3 and 43 students of class 7/2.  At present, they are in the second semester of the 2023-2024 school year and have finished one mid-term test. Both classes are instructed by Mr. Nguyen Xuan Dat - Head of the English Department at Tan Hiep Secondary school. There were four English teachers who approved of taking part in the questionnaire at Tan Hiep Secondary school. Specifically, there are 3 teachers with more than 30 years of teaching experience and 1 teacher with less than 15 years of that. Actually, there is one teacher who has known and had experience with ChatGPT. However, three have not experienced using ChatGPT yet. 

Pre-test and post-tests are designed to examine whether ChatGPT can classify students or not as well as to measure the ability of ChatGPT to test and assess through tests. In order to ensure the validity and accuracy of the pre-test and post-tests, the teacher who participated in the research checked and approved the test and could examine the ability of the students. Moreover, with the purpose of ensuring the reliability of the tests, the researcher used Cronbach's α on the data collected from pre-test and post-tests. The scores from the 30-minute pre-test, the 15-minute post-test 1 and the 45-minute post-test 2 served as instruments to measure learners’ English competence and scores before and after intervention. The reliability of the test by Cronbach’s α was 0.873 (>0.5), showing good reliability in internal consistency. 

Table 3.1. Cronbach’s Alpha 

Cronbach’s Alpha	N of items
0.873	3

In addition, the questionnaire consists of 3 parts with the goal of finding out English teachers' attitudes toward the application of ChatGPT in testing and assessment at Tan Hiep Secondary school. The first part focuses on personal information including gender, and teaching experience. The second part includes 3 questions concerning the teachers' experience with the use of ChatGPT. The last part aims to investigate their perception of the utilization of ChatGPT in testing and assessing in school.

The goals of the study are to assess the effectiveness of the incorporation of ChatGPT into testing and assessing students as well as to uncover the views of English teachers at Tan Hiep Secondary school on the application of ChatGPT in testing and assessment. To carry out the study, the data are collected through tests and questionnaires based on quantitative and experimental methods. There are 5 phrases in the procedure of the study that can be observed as follows:

Phrase 0: The essential instruments in the study (pre-test, post-test, and questionnaires) were prepared, and Mr. Nguyen Xuan Dat completed the tests first to determine the accuracy of both post-tests and pre-test, which ensured there were no two answers to a question as well as no mistakes in the tests. Furthermore, 7/2 and 7/3 classes were chosen to conduct the 30-minute placement test to consider whether the two classes’ students have the same level or not.

Phrase 1 (Week 1): At this phase, both groups did a pre-test to categorize students. The grading system is divided into Excellent (over 8 marks); Good (6.5–7.9 marks); Average (5.0–6.4 marks); Poor (3.5–4.9 marks); and Very Poor (lower 3.5 marks) (recorded by the Minister of Education and Training in Vietnam). Based on their similar English competence, Class 7/3 was assigned randomly to be control group while the other class served as experimental group.

Phrase 2 (Week 2): The teacher-in-charge named Nguyen Xuan Dat conducted the intervention with the responsibility of teaching the lessons to two groups. He provided both groups with similar lessons via the same teaching methodology and activities (Unit 10_sections: Getting Started; A Closer Look 1; A Closer Look 2 of English 7 Global Success) and after finishing the lessons, the control group did a 15-minute teacher-made post-test and the experimental group did a 15-minute ChatGPT-generated post-test. The researcher collected the data from the two groups.

Phrase 3 (Week 3, 4): After finishing post-test 1, two groups continuously learned the content of the next lessons for 3 weeks, aiming to accumulate enough knowledge for post-test 2.

Phrase 4 (Week 5): Until week 5, the students learned the necessary lessons before performing post-test 2 in 45 minutes. The content of post-test 2 is knowledge that was acquired in the last 3 weeks (the whole knowledge from Unit 10 and 3 sessions from Unit 11: Getting started; A Closer look 1; A closer look 2) with the same method as the first post-test. 

Phrase 5 (Week 5): The questionnaires would be delivered to English teachers at Tan Hiep Secondary school to uncover their attitudes toward the incorporation of ChatGPT into education, especially in testing and assessment. 

 

Findings and discussion 

Research question 1: Does using ChatGPT have any significant effects on testing and assessment in English at Tan Hiep Secondary school?

T-test analysis was used to see if there was any significant difference between the experimental and control group regarding English competence on the pre-test. In addition, the data obtained from the post-test administration to the participants after five weeks of intervention were also analyzed by the T-test to examine the differences in scores among them. Also, the responses from the questionnaires would be analyzed to find out the English teachers’ attitudes towards using ChatGPT for testing and assessment. 

Results indicated that before the intervention, the average score of the control group on the pre-test was 7.0444, and that of the experimental group was 6.3814, with standard deviations  (SD) of 2.10892 and 2.37820, respectively. The control group’s average pre-test score (M = 7.0444; SD = 2.10892) was higher than the experimental group’s (M = 6.3814; SD = 2.37820). The mean difference between the two groups’ pre-test scores is MD = −66305. However, the obtained p-value gained from Levene’s test for equality of variances was .168, much greater than the critical value of P = 0.05, which meant the findings in Equal variances assumed were used for the analysis of the T-test. According to the table, the calculation of the observed p-value between the experimental and control groups (2-tailed) was 0.170, which is much higher than the Alpha value of 0.05. Consequently, there was no statistically significant difference between the mean of pre-test scores of the control group and that of the experimental group regarding their learners’ competence on the pre-test prior to treatment.

Table 4.1. Sample Statistics for the Pre-test
	Group	N	Mean	Std. Deviation	Std. Error Mean
Pre-test	Experimental group	43	6.3814	2.37820	.36267
	Control group	45	7.0444	2.10892	.31438

 

Table 4.2. Independent Samples T- test for the Pre-test
		Levene's Test for Equality of Variances		t-test for Equality of Means
		F	Sig.	t	df	Sig. (2-tailed)	Mean Difference	Std. Error Difference	95% Confidence Interval of the Difference
									Lower	Upper
	Equal variances assumed	1.929	.168	-1.385	86	.170	-66305	.47865	-1.61456	.28847
	Equal variances not assumed			-1.381	83.714	.171	-66305	.47996	-1.61756	.29146

After the intervention, the mean score for post-test 1 in Table 4.3 (5.7558) with standard deviation (SD) of 2.21835 for the experimental group compared to the control group (6.3778) with SD of 1.70612 was considerably higher. The significance of Levene’s test for equality of variances in post-tests 1 and 2 were .060 and .509, respectively, which are all higher than the Alpha value of 0.05. In consequence, the value in Equal variances assumed line must be applied. 

Table 4.3. Sample Statistics for the Post-tests
	Group	N	Mean	Std. Deviation	Std. Error Mean
Post-test 1	Experimental group	43	5.7558	2.21835	.33830
	Control group	45	6.3778	1.70612	.25433
Post-test 2	Experimental group	43	6.5349	1.84712	.28168
	Control group	45	5.7467	2.14472	.31972

As can be seen in Table 4.4, the significant level of the reported p-value for post-test 1 was 0.143 while the other test was 0.69, which was higher than the critical p-value, which concluded that there were no significant distinctions in the mean scores among the post-tests. In other words, the null hypothesis was supported based on the data obtained from T-test.

Research question 2:  How do English teachers at Tan Hiep Secondary school respond to the use of ChatGPT in testing and assessment?

To address this question, quantitative data from English teachers at Tan Hiep Secondary school were collected. The themes of the questionnaire consisted of (1) personal information, (2) their experience with the use of ChatGPT, and (3) their standpoints on using ChatGPT in testing and assessing in education. 

Part 1: Demographic information 

Figure 4.1 indicated the data regarding the teachers’ genders and their teaching experience. According to the statistics, the majority of respondents were female, accounting for 75% of the sample (N=3) and the other 25% was male. Notably, they have had extensive experience teaching English. In particular, three teachers had over 30 teaching years while only one had less than 15 years, with no respondents with less teaching experience than 10 years. It showed that more than half of the teachers had high seniority years and experience teaching English to various grades at the middle level. 

Part 2: Teachers’ practices on using ChatGPT in testing and assessment

No.	Items	Answers	Percentages
3	How often do you use ChatGPT in your teaching?	Never	75%
		Rarely	25%
4	Have you ever applied ChatGPT to your teaching? If yes, how did you apply it?	Yes, I used to find activities for students and contents for a test	25%
		No	75%

Regarding their experience with the use of ChatGPT, all respondents had gained common knowledge of ChatGPT. However, the bulk of instructors who have used ChatGPT were extremely few, with 3 participants responding with “never” usage and 1 individual with “rare” usage. This proved that ChatGPT was not widely used among the teachers at Tan Hiep Secondary school. Not surprisingly, when it came to the employment of ChatGPT in teaching, 3 of the instructors in the total number of teachers never applied, and only 1 individual utilized it in their work. 

Part 3: Teachers’ perspectives toward the use of ChatGPT in testing and assessing

Question 1: In your opinion, what are the advantages of applying ChatGPT to testing and assessing? Please specify the advantages.

Due to the limited adoption of ChatGPT, the majority of the instructors offered ideas based on individual perspectives and comprehension of ChatGPT. All of the participants noted that ChatGPT did encompass diverse positive impacts on testing and assessment in educational contexts. According to the findings, there were indicated as (1) prompt responses, and (2) diverse databases. These potential benefits demonstrated the versatility of the language AI model as a teaching assistant, which offers tailored aid to teachers. Specifically, there was a teacher who used to experience the utility of ChatGPT for creating items for a test and discovering games. She pointed out that ChatGPT assisted her in saving time by producing items of tests in a short period, a wide range of sample tests, and activities. Meanwhile, other participants assumed that it was possible to provide various prompts such as questions, forms, or testing options. Though they have not employed ChatGPT, they also agreed with its usefulness in testing and assessment. 

 

Prompt responses 

T1: “I used ChatGPT to create questions for a test and find fun games for my students. It's certainly efficient, quickly providing a range of various questions and activities on specific topics. This proves advantageous as I need to prepare diverse tests for various classes and levels.”

Providing varied and quick items was one of the unique features of AI assistants. The participant agreed that ChatGPT helped generate different items that alleviated the great amount of time she took to make a test. It allowed users to effortlessly seek information through a multitude of topics and respond with adequate and quick answers.

 

Diverse databases

T2: “I have never used it, but I do find it helpful for providing different ideas to assess students.

T3: “Although I've never tried it myself, I believe teachers could use ChatGPT to search various activities or teaching methods. Additionally, incorporating such technology could provide educators with more diverse and efficient assessment options.”

T4: “To my knowledge, I believe it's capable of generating different types of tests for testing students although I've never used it myself.” 

Other teachers confirmed that it had the ability to provide diverse ideas that made their courses, activities, or teaching approaches more creative and varied. With a vast knowledge base of ChatGPT, users could leverage large different sources of information to explore new and attractive approaches to the same scenario instead of searching on a lot of other websites. 

Nonetheless, there were plenty of questions and concerns about its efficacy as well as capacity. One out of four teachers added that she experienced issues with ChatGPT while using it as a generated-test tool. The test being constructed by ChatGPT caused some errors from items, which took much time to adjust. Additionally, other respondents expressed doubts over ChatGPT’s ability. Particularly, one educator posited that the role of monitor was very important in assessing learners’ capability; hence, testing and assessing through AI was unacceptable, and another instructor felt uncertain about its efficiency.

T1: “However, I spent much time rechecking and changing items because the questions produced by ChatGPT tend to be straightforward, and there are often errors in the answers. So, I stopped using this tool, I believe designing a test without using ChatGPT will be much faster.”

T2: “However, I'm unsure about its effectiveness in education.”

T4: “However, I disagree with its ability to accurately assess students' proficiency levels. ChatGPT relies solely on existing data, and it cannot monitor students' progress effectively.”

Question 2: In your opinion, what are the disadvantages of applying ChatGPT to testing and evaluation? Please specify the disadvantages.

When asked about the drawbacks of ChatGPT regarding testing and assessing purposes, four participants shared the anticipated limitations and challenges that they could suffer from during the use of ChatGPT in these fields. The options provided cover important concerns, including (1) the accuracy of responses provided by ChatGPT, (2) misalignment with user expectations, and (3) difficulty for technophobes. These concerns emphasized reliability, accuracy, and the extent of ChatGPT accessibility. 

 

Lack of accuracy

T1: “The accuracy of ChatGPT isn't always reliable; the questions and answers it generates can sometimes contain mistakes and be inaccurate.”

As far as ChatGPT’s shortcomings were concerned, it was likely to produce unreliable and inaccurate answers because its responses completely depended on the data being trained. Therefore, it could make mistakes in its replies if there were ambiguous, incorrect, or limited inputs. One instructor stated that the contents created by ChatGPT contained errors and were incorrect which resulted in its low degree of accuracy. Importantly, a test must be entirely accurate and appropriate to the lesson’s aim and students’ level. Consequently, lack of accuracy was regarded as a significant disadvantage of ChatGPT.

 

Misalignment with user expectations

T3: “In my opinion, ChatGPT is able to truly misunderstand our demands and offer unexpected responses.”

Regarding fully meeting teachers’ anticipated outcomes, ChatGPT sometimes could not completely comprehend and meet users' requests, leading to its answers being inappropriate and incorrect with their wishes. Besides, one participant illustrated that the AI chatbot's answers could be irrelevant, misleading, and inaccurate with the user's expectations. It could be explained that ChatGPT's responses were based on the provided data. In addition, it completely did not have the ability to discover itself and update other knowledge; as a result, generating misleading answers at times was clear. 

 

Difficulty for technophobe

T2: “I feel challenged to adapt to new technology, so it often takes me a while to figure out how to use it. I'm accustomed to creating tests in the traditional way, drawing from textbooks and my years of teaching experience to formulate questions.”

T4: “This technology is unfamiliar to me, and I haven't had the chance to explore it thoroughly. I struggle to keep up with new developments because I'm usually busy with other tasks and not very skilled with technology. For me, mastering new technological tools is quite challenging.”

With regards to which difficulties users faced in the application of ChatGPT, innovative instruments access like ChatGPT was a dramatic challenge for individuals who lacked proficiency in computer-related skills. According to two participants' perspectives, they found it difficult to deploy advanced devices due to the complexity and lack of comprehension of this digital platform. Moreover, ChatGPT consistently released new versions in the future, which could cause technophobes to struggle to start to adopt new technology, especially when it came to accessing an advanced tool like ChatGPT.     

Question 3: Do you plan to apply ChatGPT to testing and assessing your students in the future? According to you, to what extent integrating ChatGPT into testing and evaluation is feasible? Please support your reasons:

Surprisingly, when asked about the intention of employing ChatGPT as a useful tool in testing and assessing, all of the participants did not plan to exploit ChatGPT for making a test. Furthermore, they gave several strong ideas linked to unnecessary support for the application of ChatGPT in the mentioned fields. This suggested that AI tools were not always appropriate for testing and assessing purposes at Tan Hiep Secondary school. Specifically, one teacher stated that the incorporation of ChatGPT for tasks, such as lesson planning would be more applicable than testing and assessing. Also, the participant mentioned that schools should provide training or professional development opportunities for academic staff in order to effectively utilize ChatGPT if necessary. Similarly, another participant also highlighted that the utilization of ChatGPT was quite tough for people who rarely engaged with digital innovation; consequently, the teacher was not willing to adopt this instrument. Out of all the teachers, others intensely confirmed that keeping the traditional assessment approaches was sufficient and effective; thus, it was not essential to make use of assisting teachers. It could be said that the English teachers at Tan Hiep Secondary school did not express the desire for the implication of ChatGPT for testing and assessing purposes in the educational context. In other words, they remained in preference for conventional methods over the utility of an innovative approach, particularly ChatGPT. 

T1: “In my opinion, it's more effective to assess students with the guidance of a teacher rather than relying solely on ChatGPT. ChatGPT cannot replace a teacher in this aspect. If ChatGPT were to be used in education, I believe it would be more suitable for tasks like lesson planning rather than student assessment. Additionally, there should be training programs or workshops available to help educators utilize ChatGPT effectively and strategies to mitigate any errors it may produce.”

T2: “I believe there's no need to introduce it into education because sticking to traditional methods works well. Embracing traditional approaches provides a strong base in teaching methods that have been successful over time.”

T3: “In my opinion, it is unnecessary to integrate it into education because the traditional ways remain good and efficient.”

T4: “I find it difficult to use most technological devices because they take me a lot of time to find out how to use them, so I don’t expect to apply it for my job”

Research question 1 investigated whether using ChatGPT has any significant effects on testing and assessment in English at Tan Hiep Secondary school. The results of the current study indicated that ChatGPT could assess students’ English competence as effectively as experienced teachers do. In other words, the efficiency of the ChatGPT-made test has approximately the same effectiveness in assessing students’ performance as that of the teacher-made test. Specifically, based on the T-test’s results, it was evident that the test being produced with the support of ChatGPT possesses the capability of categorizing students with a conventional approach, which is capable of categorizing, testing students, and making a judgment about their proficiency, knowledge, and progress. 

Research question 2 investigated the teachers’ standpoints on using ChatGPT in testing and assessment. The data obtained from the questionnaire indicated that most of the participants experienced teaching English for over 30 years, which shows their extensive teaching years. 

Regarding the experience of using ChatGPT, four participants confirmed that a comprehensive understanding of this AI tool was still limited since ChatGPT was a novelty and they had not gotten many chances to become familiar with its capabilities. To be more specific, all of the teachers failed to use it in their tasks, except for one who used ChatGPT with rare frequency, and the teacher employed it to explore interesting activities and design items for a test with the assistance of ChatGPT. This suggested that ChatGPT was a little approved and challenging to apply at Tan Hiep Secondary school.

With regards to teachers’ perspectives on the use of ChatGPT in testing and assessment in teaching, they provided valuable insights into the potential benefits and drawbacks of ChatGPT’s testing and assessment purposes. Saving time and possessing various data were two strengths mentioned in the survey. Another crucial aspect revealed was the anticipated challenges and limitations of implementing ChatGPT, which included a lack of accuracy, misleading queries, and challenges for technophobes. The respondents agreed that its responses were diverse and fast; however, these answers were not completely reliable and accurate, which could provide misleading, obviously false information. In addition, it could not understand contexts, leading to generating responses that did not meet previous demands. On the other side, ChatGPT was an advanced and innovative tool; nevertheless, individuals who are technophobes could find it tough to navigate. 

Concerning English teachers’ perspectives on the implication of ChatGPT in testing and assessment in the future, it was surprising that all participants did not intend to employ educational technology, such as ChatGPT for the above-mentioned purposes. This was because its accuracy and reliability were not ensured, and its ability could not accurately assess students' proficiency levels, based on teachers' viewpoints. Noticeably, more than half of teachers indicated that the application of ChatGPT to testing and assessment was not essential in this context. Instead, they advocated traditional methods, which remained effective and accurate. This suggested that the integration of language AI models could be limited to education.

 

Conclusion

Artificial intelligence’s advancement opens many opportunities for the development of educational contexts, thanks to its potential effects on the future of academics. However, despite abundant studies regarding ChatGPT’s impact, there is a limitation of empirical research concerning its application in pedagogical settings, particularly in testing and assessment. To address this research gap, the paper was conducted to investigate ChatGPT’s effect on testing and assessment as well as how English teachers at Tan Hiep Secondary school perceive the employment of ChatGPT in such a field. By using both a quasi-experimental method and a quantitative data analysis obtained throughout tests and paper-based questionnaires, the result obtained from the detailed analysis of the data demonstrated that ChatGPT-made tests can gauge students' competence corresponding to teacher-made tests. It was evident that before and after the treatment, the number of students who got over-average levels of both tests (pre- and post-tests) was approximately equivalent. Furthermore, the findings also suggested that the ChatGPT’s capacity could be productively applied to the classification of students since there was no significant disparity between control and experimental groups in the categorization of students’ levels. From the analyzed data, consistency in scores and the equivalency of proficiency levels were recognized in both groups, which concludes the potential assistance of ChatGPT in testing and assessment.

The findings collected from the questionnaires revealed that English teachers at Tan Hiep Secondary school held varied perspectives on the application of ChatGPT in testing and assessment. Only one teacher used this AI tool, while others never used ChatGPT in their tasks since the majority of teachers are quite old (over 30-year teaching experience), so it was a huge challenge for them to use modern technology. Another significant finding was that the teachers were still not familiar with this advanced technology due to its novelty. In other words, they did not have many opportunities to access this powerful AI tool. 

 

Acknowledgements

The author of this article acknowledged the support of Van Lang University at 69/68 Dang Thuy Tram St. Ward 13, Binh Thanh Dist., Ho Chi Minh City, Vietnam.

 

References 

El-Seoud, S., Ayman, S. E., Nagaty, K., & H. Karam, O. (2023). The impact of CHATGPT on student learning/performing. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4532913 

Akram, Z., Sethi, A., Khan, A. M., & Zaidi, F. Z. (2021). Assessment of burnout and associated factors among medical educators. Pakistan Journal of Medical Sciences, 37(3). https://doi.org/10.12669/pjms.37.3.3078 

Babitha, M. M., Sushma, C., & Gudivada, V. K. (2022). Trends of Artificial Intelligence for online exams in education. International journal of Early Childhood special Education, 14(1), 2457-2463.

Baskara, R., & Mukarto, M. (2023). Exploring the implications of chatgpt for language learning in higher education. Indonesian Journal of English Language Teaching and Applied Linguistics, 7(2), 343-358.

Borji, A. (2023). A categorical archive of CHATGPT failures. Retrieved from the WWW: https://arxiv.org/abs/2302.03494 

Brown, H. D. (2003). Language Assessment Principles and Classroom Practice (2nd ed.). New York: Pearson Education, Inc. 

Brown, H., & Abeywickrama, P. (2019). Language assessment: Principles and classroom practices (3rd ed.). Pearson Education. 

Brown, T.L.G. (2004). Teachers’ conceptions of assessment: implications for policy and professional development. Assessment in Education, 11(3), 305-322. https://doi.org/10.1080/0969594042000304609

Cassidy, C. (2023). Australian universities to return to ‘pen and paper’ exams after students caught using AI to write essays. The Guardian news. Retrieved from the WWW: https://www.theguardian.com/australia-news/2023/jan/10/universities-to-return-to-pen-and-paper-exams-after-students-caught-using-ai-to-write-essays

Cotton, D. R., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating. Ensuring academic integrity in the era of ChatGPT. Preprint. https://doi.org/10.35542/osf.io/mrz8h

Das, E. (2018). Testing and evaluation: An effective process of learning language. Latest TOC RSS, 1(1). https://doi.org/10.15864/ijelts.1105

De Winter, J. C. F. (2023). Can ChatGPT pass high school exams on English language
comprehension?. Researchgate. Preprint. https://doi.org/10.1007/s40593-023-00372-z

Eke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology,13. https://doi.org/10.1016/j.jrt.2023.100060

Erna, N., Jabu, B., & Salija, K. (2018). An analysis of English teacher-made tests nana. eprints  http://eprints.unm.ac.id/id/eprint/10907

García Sánchez, O. V. (2023). Usage and perception of ChatGPT in higher education. Revista De Investigación En Tecnologías De La Información, 11(23), 98–107. https://doi.org/10.36825/RITI.11.23.009

García-Peñalvo, F.J. (2023). La percepción de la Inteligencia Artificial en contextos educativos tras el lanzamiento de ChatGPT: Disrupción o pánico. Educ. Knowl. Soc, 24, e31279. https://doi.org/10.14201/eks.31279

Ghaicha, A. (2016). Theoretical Framework for Educational Assessment: A Synoptic Review. Journal of Education and Practice, 7, 212-231. https://files.eric.ed.gov/fulltext/EJ1112912.pdf

Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022). How does ChatGPT perform on the medical licensing exams? The implications of large language models for medical education and knowledge assessment. medRxiv. Preprint.
https://doi.org/10.1101/2022.12.23.22283901 

Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R., & Chartash, D. (2023). How does CHATGPT perform on the United States Medical Licensing Examination? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9, e45312. https://doi.org/10.2196/45312 

Gozalo-Brizuela, R., & Garrido-Merchan, E. C. (2023). CHATGPT is not all you need. A state of the art review of large generative AI models. Retrieved from the WWW: https://arxiv.org/abs/2301.04655 

Gürbüz, R., Gülburnu, M., & Şahin, S. (2017). Oyun Destekli Kesir Öğretimi Hakkinda Öğretmen Görüşleri: Video Destekli Bir Çalişma. Adıyaman Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, (25), 98-132. 

Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., … Kasneci, G. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. https://doi.org/10.35542/osf.io/5er8f

Khalil, M., & Er, E. (2023). Will CHATGPT get you caught? rethinking of plagiarism detection. Lecture Notes in Computer Science, 14040, 475–487. https://doi.org/10.1007/978-3-031-34411-4

Mathew, A. (2023). Is Artificial Intelligence a World Changer? A Case Study of OpenAI’s Chat GPT. Recent Progress in Science and Technology, 5, 35–42. https://doi.org/10.9734/bpi/rpst/v5/18240D

Mondal, H., Marndi, G., Behera, J. K., & Mondal, S. (2023). Chatgpt for teachers: Practical examples for utilizing Artificial Intelligence for educational purposes. Indian Journal of Vascular and Endovascular Surgery, 10(3), 200–205. https://doi.org/10.4103/ijves.ijves_37_23 

Osorio, J.A.C. (2023) Explorando el potencial de ChatGPT en la escritura científica: Ventajas, desafíos y precauciones. Scientia et Technica, 28, 3–5. DOI: https://doi.org/10.22517/23447214.25303

Qadir, J. (2022) Engineering Education in the Era of ChatGPT: Promises and Pitfalls of Generative AI for Education. TechRxiv,1, 1–10. DOI: 10.36227/techrxiv.21789434.v1

Qureshi, B. (2023). Exploring the use of CHATGPT as a tool for learning and assessment in Undergraduate Computer Science Curriculum: Opportunities and challenges. Retrieved from the WWW: https://arxiv.org/abs/2304.11214 

Ruby, D. (2023). ChatGPT Statistics for 2023: Comprehensive Facts and Data. Demand Sage. Retrieved from the WWW: https://www.demandsage.com/chatgpt-statistics/ 

Rudolph A, J. (2022). Higher education in an age of war. Journal of Applied Learning & Teaching, 5(1), 6-9.https://doi.org/10.37074/jalt.2022.5.1.1

Rudolph, J., Tan, S., & Tan, S. (2023). Chatgpt: Bullshit spewer or the end of traditional assessments in higher education? JALT: Journal of Applied Learning and Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.9 

Shoufan, A. (2023). Exploring Students' Perceptions of ChatGPT: Thematic Analysis and Follow-up Survey. IEEE Access, 11(38805-38818). https://doi.org/10.1109/ACCESS.2023.3268224

Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology. Cureus, 15(2): e35237. https://doi.org/10.7759/cureus.35237

Topsakal, O., & Topsakal, E. (2022). Framework for a foreign language teaching software for children utilizing AR, voicebots and ChatGPT (large language models). The Journal of Cognitive Systems, 7(2), 33-38. https://doi.org/10.52876/jcs.1227392 

Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O. C., & Dimitrova, V. (2023). Artifcial Intelligence in Education. 24th International Conference, AIED 2023, Tokyo, Japan., 13916. https://doi.org/10.1007/978-3-031-36272-9 

Whalen, J., Mouza, C. (2023) ChatGPT: Challenges, Opportunities, and Implications for Teacher Education. Society for Information Technology & Teacher Education, 23(1), 1–23. Retrieved from the WWW: https://www.learntechlib.org/primary/p/222408/.

Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28(11), 1-25. http://dx.doi.org/10.1007/s10639-023-11742-4

 

Please check the Pilgrims in Segovia Teacher Training courses 2025 at Pilgrims website.

Please check the Pilgrims f2f courses at Pilgrims website.