Examining mode effects for an adapted Chinese critical thinking assessment
We examine the effects of computer-based versus paper-based assessment of critical thinking skills, adapted from English (in the U.S.) to Chinese. Using data collected based on a random assignment between the two modes in multiple Chinese colleges, we investigate mode effects from multiple perspectives: mean scores, measurement precision, item functioning (i.e. item difficulty and discrimination), response behavior (i.e. test completion and item omission), and user perceptions. Our findings shed light on assessment and item properties that could be the sources of mode effects. At the test level, we find that the computer-based test is more difficult and more speeded than the paper-based test. We speculate that these differences are attributable to the test’s structure, its high demands on reading, and test-taking flexibility afforded under the paper testing mode. Item-level evaluation allows us to identify item characteristics that are prone to mode effects, including targeted cognitive skill, response type, and the amount of adaptation between modes. Implications for test design are discussed, and actionable design suggestions are offered with the goal of minimizing mode effect.
Introduction: Assessing Student Learning Outcomes in Higher Education Until the last decade, objective information on student learning and student learning outcomes in higher education at the national and international levels was scarce. This area was largely underrepresented in comparison to other areas of formal education such as school. In the context of current developments in higher education such as internationalization of study programs and ever-increasing student mobility and the ensuing increase in heterogeneity of students’ learning conditions, the need for objective, valid, and reliable assessment tools that adhere to the Standards for Educational and Psychological Testing set out by the American Educational Research Association (AERA) has become urgent. This has led to intense research efforts being made within and across many countries, which are of great practical and political importance. This book presents the most significant of these initiatives and developments in order to highlight the tremendous work national and international research communities have done in this area over the past decade. A broad range of national and international assessment research projects and curricular innovation initiatives in higher education focusing on both domain-specific and generic student learning outcomes are presented in this volume. Results and lessons learned from various research programs such as the German Modeling and Measuring Competencies in Higher Education (KoKoHs) and feasibility studies such as the Assessment of Higher Education Learning Outcomes (AHELO; an international comparative study by the Organisation for Economic Co-operation and Development (OECD) of students’ generic skills and economic and engineering competencies) form the basis of several ongoing initiatives by testing institutes to make assessments suitable for use in higher education abroad. Examples include the Educational Testing Service’s (ETS) Heighten Outcomes assessment and the Council for Aid to Education’s (CAE) Collegiate Learning Assessment CLA+. At the European level, the CALOHEE initiative on Measuring and Comparing Achievements of Learning Outcomes in Higher Education in Europe aims to develop a joint basis for learning outcomes in higher education as well as curricula in five disciplines, including education. One of the most current international initiatives, the International Collaborative for Performance Assessment of Learning in Higher Education (iPAL), focuses on developing performance assessments of learning that meet high standards in psychometric quality criteria and are suitable for use at higher education institutions across nations. The compilation of this work in the present book shows where we stand today and the progress that has been made in this field of research with newly developed theoretically conceptualized approaches to modeling and measurement instruments for empirical studies. It also illustrates which issues have not yet been thoroughly addressed by the – indeed very active – research community measuring student learning in higher education. Therefore, this book offers a sound basis for further research, highlighting the current challenges and future perspectives in measuring learning and learning outcomes in higher education we need to deal with in the next decades.
The need to foster critical thinking has long been one of the key issues in education. It is essentially vital nowadays against the background of an increased volume of cross-cultural communications due to the present-day demand for collaboration to tackle pressing global issues through joint efforts of different nations. While the format of debates has been recognized by researchers as one of the most efficient tools of setting off critical thinking, it is up to the new technologies in education to make it possible to bring this platform to a cross-cultural level. Since a cross-cultural dialogue in most cases supposes the mastery of a foreign language, e-learning in the form of cross-border video-conference debates present an invaluable opportunity for educators to enhance the pedagogy of foreign language acquisition around the globe. The present paper focuses on a case-study of an on-going project of implementing the tool of synchronous cross-cultural video-conference debates.
In the era of abundance of ICT in education, the focus of academics is gradually shifting from initial urge for testing all possible emerging devices in order to improve learning process to a concern over potential overload with digital media and thus the need for its sifting. Foreign language learning is no exception. The fundamental skills of reading, listening, speaking, and writing have all been ICT enhanced in the past decade in both successful and failed attempts to boost language proficiency of learners. At the same time, the only indisputable need for technology in foreign language acquisition in academic environment has always been traced in the sphere of developing listening skills. For many researchers, listening as a basic skill is closely connected to the ability of comprehension [3, 5, 9]. However, true for lower level language learning this argument is fading at a more advanced level of language mastery dominated by reinforcement of critical thinking. A qualitative leap from basic comprehension to critical analysis has to be addressed in the process of developing language competencies. Listening, in particular, requires the design of special learning material, which, on the one hand, meets the criterion of authenticity [2, 7] and, on the other hand, leaves room for scaffolding since formal learning implies the use of specifically-built system of learning tools and not just exposure to random language experiences. [3, 4] In this regard, digital audiobooks present a unique opportunity to tackle the development of critical listening at advanced stages. Audiobooks have recently found a wide application in education from its elementary stage to adult learning [1,16]. Primarily, audiobooks are used as a supplement to reading. The present study, however, is focused on the use of audiobooks for critical listening skills for ESL students without considering the reading component of the traditional approach to audiobooks. In such context, the present article deals with issues of developing a methodology of implementing the use of digital audiobooks in advanced ESL classroom, presents specific examples of scaffolding exercises and analysis of the data collected from field-testing this approach on the stage of higher education through the case study of the Russian students learning English at National Research University Higher School of Economics in Moscow.
The author teaches to awaken creativity in oneself, using emotions as a factor of motivation, explains the concept of critical thinking, gives the reader tools to add / edit publications to increase the clarity and rationality of their own judgments, and also shows where a particular theory is applicable
The article examines Husserl project of the renovation of European cultural mankind in context of the genesis of Husserl’s practical philosophy, existential motivations for the emergence of the idea of renovation (Erneuerung) and in perspective of its adaptations to the contemporary. The key point of the article – Husserl’s thesis about internationalization through the phenomenological "spirit of autonomy". Аnalyzing of the problem of harmonization between universal theoretical and concrete practical dimension of the idea of renovation, author explicate the socio-critical and cultural creative potential of Husserl's phenomenology.
The paper covers a case of teaching critical reading and speaking to students of Economics, ICT and Mechanics at senior levels at National Research University Higher School of Economics. Relevant tested teaching materials aimed at the development of required competencies in reading and speaking are presented as well as the methodical principles used in elaboration of these materials.
Writing-to-learn benefits have been explored in various educational settings. However, little research has been done on how a WTL approach in combination with two different languages of instruction can influence historical reasoning learning. The main objective of the present study is to examine the effects of a particular WTL instruction in two languages (L1 is Russian and L2 is English) on historical reasoning learning outcomes. The paper presents the results of a case study of first year students of the History Faculty. Learners received small-group L1/L2 instruction by a team of two teachers in a Logic module which included evidence based direct instruction and a set of WTL activities. The instruction explicitly targeted argumentation skills such as argument structure, validity of an argument, fact vs opinion and using documented historical sources. The Critical Thinking Analytic Rubric was used for both pre and post-course metacognitive competencies assessments while argumentation skills were assessed with the help of a new developed rubric. The results showed various patterns of positive change in all the categories and seem to support the hypothesis that this approach to writing-to-learn in L1 and L2 leads to successful acquisition of disciplinary knowledge and skills.
Contemporary argumentation theorists claim that argumentation has interactive, social, dynamic and dialogical nature and reflect social constructivist perspective. Yet, there are multiple approaches to promoting a most effective learning approach and instruction. How can students learn to construct strong arguments and distinguish between facts and opinions?
There are three educational approaches for developing argument skills: oral, written and web-based discussions. Research results reveal that expended engagement in argumentative discourse improves the quality of arguments even if there is no instruction provided.
In our book we take a mixed approach to teaching which is based on experiential learning and direct instruction. Direct instruction provides students with requirements for their writing while the experiential approach emphasizes their engagement and practice.
For Russian students, engaging in a “two-sided” argument (versus "one-sided") seems to be challenging. A two-sided argument addresses the opposing argument, rather than simply arguing for one's own position. It is crucial that Russian students learn to engage in evidence-based argumentation where they provide a claim and support it by evidence or reasons in a certain way.
Another challenge is that English instructors may also have difficulty in explaining how to make arguments and how evidence can be applied in reading and writing. Many teachers seem to be unprepared to provide instructional support for learning argumentation skills.
Development of cognitive competence requires acknowledging the academic and disciplinary discourses. Russian students often struggle to attend to opponent's claims and stay focused on their own claims. They also fail to identify any weaknesses in the opponent’s arguments. However, when proper instruction is available students are able to apply arguments.
There are various approaches to defining argument structure. Some researchers highlight a claim supported by grounds, warrants and backing. Others suggest argument-counterargument integration for defining an argument schema and suggest strategies to construct an integrative argument: refutation, constructing a design claim and weighing.
Whatever approach is chosen, one learning goal is for teachers and for students to become aware of the existing strategies and decide why they follow it. This is part of the goal of metacognitive development. It is also important to incorporate reflective activities into learning as they help to ensure that reasoning skills become internalized.
In addition to gaining awareness of the strategies and reflection, a learner should gain a deeper understanding of the content and persuading others with their arguments. Students should learn generate arguments that incorporate multiple perspectives of an issue. Our book employs reflective activities as a primary pedagogical tool for the improving argument and reasoning skills.
Our book is in line with the experiential learning which we see as the process of learning when students’ knowledge is based on their experience. “Knowledge results from the combination of grasping and transforming experience" (Kolb, 1984, p. 41 in Kolb, et al., 2000). The learning process is structured in four stages including first experience (preliminary questions in each section), reflective observation (theoretical excerpts and sample writings), abstract conceptualization (end-of-the-section questions) and active experimentation (thinking, speaking, writing activities).
In our book we also scaffold argumentative written discourse and break up learning activities into different aspects. There are such techniques as working in collaborative pairs, reflective activities, dialogues and discussions.
The development of metacognition is based on extended reading activities and speaking. It will help ensure students become reflective about their reasoning and evidence. Apart from having students develop stronger arguments, we invite them to engage more with the opposing position which isn't necessarily false.
We do believe that mastering critical writing skills help students become better critical thinkers.
Institutions affect investment decisions, including investments in human capital. Hence institutions are relevant for the allocation of talent. Good market-supporting institutions attract talent to productive value-creating activities, whereas poor ones raise the appeal of rent-seeking. We propose a theoretical model that predicts that more talented individuals are particularly sensitive in their career choices to the quality of institutions, and test these predictions on a sample of around 95 countries of the world. We find a strong positive association between the quality of institutions and graduation of college and university students in science, and an even stronger negative correlation with graduation in law. Our findings are robust to various specifications of empirical models, including smaller samples of former colonies and transition countries. The quality of human capital makes the distinction between educational choices under strong and weak institutions particularly sharp. We show that the allocation of talent is an important link between institutions and growth.