Sunday, October 6, 2013
Friday, September 20, 2013
Indian Institutes of Management
CAT Exams: Development and Scoring
Development :
The CAT 2013 test development process was conducted in alignment with the Standards for Educational and Psychological Testing. The exam was designed with two sections: (1) Quantitative Ability & Data Interpretation, (2) Verbal Ability and Logical Reasoning. These two sections are consistent with the knowledge domains historically assessed by the IIMs and are also aligned with the content areas covered in equivalent global admission examinations that measure performance along similar scales.
Your raw scores are calculated for each section based on the number of questions you answered correctly, incorrectly, or that you omitted.
The IIM scaling model is as follows:
The CAT exam has been developed to accurately identify top performing candidates and that design makes use of a scaled score range of 0 to 450. In order to appropriately identify the top performing candidates, the CAT exam is, by design, a very difficult exam. As would be expected with the more difficult CAT exam, no candidate would likely answer 100% of the items correctly or achieve the top theoretical score. The exam design will accomplish the goal of identifying the top performing candidates who are, indeed, ranked at the top of the list. If the exam were designed to be substantially easier, it would be theoretically possible for a candidate to achieve a score of 450. However, an exam constructed to be that easy would not serve the distinct purposes of the IIMs.
Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling and linking: Methods and practices. 2nd Ed. Springer.
Fairness and Equivalency in IIM Exams
Your raw scores are calculated for each section based on the number of questions you answered correctly, incorrectly, or that you omitted.
| ||||||||||||
The CAT exam has been developed to accurately identify top performing candidates and that design makes use of a scaled score range of 0 to 450. In order to appropriately identify the top performing candidates, the CAT exam is, by design, a very difficult exam. As would be expected with the more difficult CAT exam, no candidate would likely answer 100% of the items correctly or achieve the top theoretical score. The exam design will accomplish the goal of identifying the top performing candidates who are, indeed, ranked at the top of the list. If the exam were designed to be substantially easier, it would be theoretically possible for a candidate to achieve a score of 450. However, an exam constructed to be that easy would not serve the distinct purposes of the IIMs.
Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling and linking: Methods and practices. 2nd Ed. Springer.
Fairness and Equivalency in IIM Exams
Content of the examination was developed and confirmed by individuals with high levels of expertise in each of these content domains. Post-administration analysis will be conducted by credentialed psychometricians to confirm the validity of the examination scores and to ensure that every candidate was provided a fair and equal opportunity to display their knowledge.
Scoring :
Prometric employs an industry-standard, psychometrically-sound approach to the scoring process for all IIM candidates. The three-step process is outlined here and is supported by the Standards for Educational and Psychological Testing and the ETS Standards for Quality and Fairness.
Step 1: Raw Score is Calculated
This scoring methodology ensures that candidates are only awarded points for what they know, while having points deducted for inappropriate random guessing. This is a standard process in the testing industry and is a methodology employed in scoring similar admissions tests such as the Graduate Record Examination (GRE).
Step 2: Raw Score is “Equated”
Equating is a statistical process used to adjust scores on two or more alternate forms of an assessment so that the scores may be used interchangeably. Industry standard processes were used for equating, such as those outlined within the ETS Standards for Quality and Fairness.
Step3: Equated Raw Score is “Scaled”
In order to ensure appropriate interpretation of an equated raw score, the scores must be placed on a common scale or metric. A linear transformation is used for this scaling process, which is an industry standard practice (Kolen & Brennan, 2004).
Section Scores = 0 to 225
Total Exam Score = 0 to 450
Three scaled scores are presented for each candidate: an overall scaled score and two separate scaled scores for each section. As the two sections evaluate two distinct sets of knowledge and skills, scores may not correlate across sections. A high score in one section does not guarantee a high score in another section. Percentile rankings are provided for each individual section as well as for the overall exam score.
About Test Difficulty
Reference
American Educational Research Association (AERA), American Psychological Association (APA), and the National Council on Measurement in Education (NCME). (1999). Standards for Educational and Psychological Testing. Washington, D.C.: Author.
Educational Testing Service (ETS). (2002) ETS Standards for Quality and Fairness. Princeton, N.J.
A significant number of examination forms are used on behalf of the IIMs to evaluate the large candidate population. With the use of multiple forms comes the need to ensure fairness and equivalency of the examinations used for assessment. A post-equating process is necessary to ensure validity and fairness.
Equating is a psychometric process to adjust differences in difficulty so that scores from different test forms are comparable on a common metric and therefore fair to candidates testing across multiple days. The equating process was designed with three phases: exam creation, post-equating, and scaling.
Each form contains a pre-defined number of statistically profiled questions selected from a large bank. These questions form an equating block within a form, which can be used as an anchor to rescale candidates’ scores to the metric of the item bank. This rescaling process adjusts for the difference in the form difficulties, taking into account of candidates’ differential performance on the equating block. As a result, the candidates’ rescaled scores can be placed and compared on the common metric regardless which form they take.
This approach provides support for equating without significantly impacting the security of the items. The second phase of the process is post equating. In this process, items are concurrently analyzed and the estimated item parameters (item difficulty and item discrimination) are put onto a common metric. Item Response Theory (IRT), a psychometrically supported statistical model, is utilized in this process. The result is a statistically equated raw score that takes into account the performance of the candidate along with the difficulty of the form administered.
Once post-equating has resulted in an equated raw score, scaling of the scores is done to reduce confusion to candidates. Scaling can be done using a linear or non-linear transformation of the original, equated number correct. Though the number as presented to candidates is placed on a common scale for ease of interpretation, the position of candidates in the score distribution does not change.
Lastly, once scaled scores are established, the final step in the scoring process is to rank candidates in their performance. A percentile rank is the percentage of scores that fall below a given score. With the total scale scores arranged in rank order from the lowest to highest, in 100 equally sized groups, a table with the total scale scores to percentile ranks will be created. This ranked list of candidates will allow for the identification of candidates from the highest performers at the very top of the list to the lower performers in the middle and low end of the scale.
The test development and equating models outlined have substantial advantages to candidates. First, they confirm with a high level of psychometric rigor that all examination scores are valid, equitable and fair. Post equating takes into account any statistical differences in examination difficulty and ensures all candidates are evaluated on a common scale. Reporting scores on this statistically equivalent scale creates an environment where the very high performing candidates will be ranked appropriately at the top end of the scale.
Test Duration and Pattern
The CAT 2013 will have only two sections. The first section is Quantitative Ability & Data Interpretation; the second section is Verbal Ability & Logical Reasoning. These two sections will be implemented sequentially with separate time limits. The examination will be 140 minutes. Candidates will have 70 minutes to answer 30 questions within each section which will have an on-screen countdown timer. Within the 70 min provided for the section, candidates can review the answers. Once the time ends for the first section, they will move to the second section and will no longer be able to go back.
A 15-minute tutorial will be provided before the start of the test, and candidates are advised to go through that carefully before starting the main examination. The total duration will be two hours and thirty five minutes including the tutorial. A practice test to provide candidates with a feel of the timed sections, as well as navigation and functionality of the actual test will be made available in the coming weeks. Once again candidates are advised to go though them carefully to be familiar with themselves with the screens, layout, and navigation.
At the test venue, each candidate will be seated at a desk with a computer terminal and he/she will be provided with a scratch paper for calculations. After the test, candidate must leave the scratch paper at the desk. Rough work cannot be done on any other paper/sheet, as nothing will be allowed inside the testing room. No breaks will be given during the test.
Disclaimers Regarding Conduct of CAT
Disclosing, publishing, reproducing, transmitting, storing, or facilitating transmission and storage of the contents of the CAT or any information therein in whole or part thereof in any form or by any means, verbal or written, electronically or mechanically for any purpose, shall be in violation of the Indian Contract Act, 1872 and/or the Copyright Act, 1957 and/or the Information Technology Act, 2000. Such actions and/or abetment thereof as aforementioned may constitute a cognizable offence punishable with imprisonment for a term up to three years and fine up to Rs. two lakhs. Candidates who want to appear for CAT have to agree to a Non-Disclosure Agreement at the time of the test.
Computer based testing is a proven and reliable process that is administered to tens of millions of people each year. As with paper and pencil testing, or virtually every other human endeavor, a very small number of problems could occur that might prevent a test from being delivered and/or a result from being generated. In the unlikely event this does occur, every effort will be made to correct the problem, up to and including the administration of another test.
Common Admission Test
The Common Admission Test (CAT) is a computer based test held in India. This test scores a person on the bases of quantitative ability, data interpretation, verbal ability and logical reasoning. The Indian Institutes of Management(IIMs) started this exam and use the test for selecting students for their business administration programs.The test is conducted every year by one of the IIM's based on a policy of rotation. In August 2011, it was announced thatIndian Institutes of Technology (IITs) and Indian Institute of Science (IISc) would also use the CAT scores, instead of the Joint Management Entrance Test (JMET), to select students for their management programmes from 2012-14 year.
History
Before 2009, CAT was a paper based test conducted on a single day for all candidates. The pattern, number of questions and duration have seen considerable variations over the years.
On 1 May 2009, it was announced that CAT would be a Computer Based Test starting from CAT 2009. The American firm Prometric has been entrusted with the responsibility of conducting the computer based test.[3] The first computer based CAT was marred by technical snags.[4] The issue was so serious that it prompted the Government of India to seek a report from the convenor.[5] The trouble was diagnosed as 'Conflicer' and 'W32 Nimda', the two viruses that attacked the system display of the test, causing severe slow down.[6]
Approval
CAT is conducted by the Indian Institutes of Management as a pre-requisite for admission to various management programmes of IIMs, Indian Institutes of Technology (IITs), Indian Institute of Science (IISc), National Institutes of Technology (NITs), Faculty of Management Studies (FMS) and few other institutions.[7]
CAT is normally conducted every year during a 21 day testing window (estimate but may vary) in the months of October and November. A candidate can appear for CAT only once during the 20-day testing window. The test score is valid for admission to the forthcoming academic year only.[8]
Exam format
The CAT, like virtually all large-scale exams, utilises multiple forms, or versions, of the test. Hence there are two types of scores involved viz. raw score and scaled score.
The raw score is calculated for each section based on the number of questions one answered correctly, incorrectly, or omitted. Candidates are given +3 points for each correct answer and -1 point for each incorrect answer. There are no points for questions that are not answered. The raw scores are then adjusted, as necessary, through a process called equating. Equated raw scores are then placed on a common scale or metric to ensure appropriate interpretation of the scores. This process is called scaling.
Three scaled scores will be presented for each candidate: an overall scaled score and two separate scaled scores for each section. As the two sections evaluate distinct sets of knowledge and skills, scores do not correlate across... sections. A high score in one section does not guarantee a high score in another section. Percentile rankings are provided for each individual section as well as for the overall exam score.[9]
Subscribe to:
Posts (Atom)