Measurement and Analysis of Student Learning and Performance

Many institutions use the ETS Major Field Test for students receiving a Bachelor’s Degree in Business as an end-of-program student learning outcomes assessment instrument. The test itself is a multiple-choice test “designed to measure a student’s subject knowledge and the ability to apply facts, concepts, theories and analytical methods.” [ETS website] The Dauch College of Business and Economics has administered this test as a key summative assessment for undergraduate COBE programs. The test is given to students taking the BUS499 Senior Assessment class. In addition to an overall score, the mean percent correct for all students taking the exam is calculated for each of nine specific assessment indicators. Comparative data tables provided by the ETS enable comparisons with other institutions using the test. The following chart shows the relationship between the Dauch College of Business and Economics (COBE) overall mean score for all students taking the exam and the mean score for senior students at all domestic institutions for one or more years prior to the test semester.

ETS Major Field Test Mean Scores, COBE vs. National Average

ETS Major Field Test Mean COBE Scores and Corresponding Percentile

In general, the overall mean for COBE students taking the exam was close to the mean for all institutions during the 2006-2007 and 2007-2008 academic years. The COBE faculty and administrators reviewed these results and devised a plan to modify the curriculum with the goal of improving student performance on this assessment. Following the changes, student performance improved significantly in the 2008-2009 academic year, to a level that we rated as exceeding expectations. Performance of COBE students improved further in the 2009-2010 academic year. The mean program score achieved by COBE students in the spring 2010 semester was at the 85th percentile nationally for all institutions using this assessment instrument, a level that we rate as exceeding expectations. Student performance during the fall 2010 semester dropped to a level just below the national mean, but improved again in the spring 2011 semester to the 65th percentile nationally, exceeding expectations. In general we conclude that our students have performed at a level well above the national average for the past few years.

As shown in the following figure, overall COBE performance across all eight of the assessment indicators measured by the ETS Major Field Test has improved over the past five academic years. Variability among the nine indicators has been reduced, with the range between the best and worst indicators ranging from 50 to 65 percentile points during the first four semesters of data shown while dropping to only 20 percentile points in the spring 2010 semester. Further, all nine indicator means were at the 70th percentile or higher in the spring 2010 semester, the first time in eight semesters in which this occurred. We rated this level of performance as exceeding expectations. Faculty members in each functional area review the results and discuss whether curriculum changes are warranted, “closing the loop” and completing our assessment process. Overall, we are satisfied with the performance of our students on this external assessment of their business knowledge and technical skills, and we would rate our students as meeting or exceeding expectations.

ETS Major Field Test Mean Scores for Nine Content Areas

Student learning outcomes assessment related to program-specific outcomes for undergraduate students also are assessed when students complete the mandatory internship or formal work experience requirement. When a student registers for an internship, he or she must select at least three or four outcomes relevant to the type of internship, and the internship supervisor confirms that the student will be able to accomplish the outcomes. Upon completion of the internship or work experience, the COBE internship coordinator evaluates the extent to which the student accomplished each of the selected outcomes. In addition, beginning in the fall 2010 semester (including internships completed during summer 2010) the internship supervisor (the student’s employer) completes and submits an evaluation of the student’s performance on several of the College outcome areas. Having an external assessment of student performance in real work situations is tremendously appealing, as it represents an authentic assessment of student learning outcomes.

Beginning with the summer 2010 semester, the employer survey was administered via either an online survey or a hard-copy questionnaire. The employer survey, completed by the student’s internship supervisor (employer), enables the supervisor to evaluate the intern’s performance in five of the seven student learning outcome content areas, including communication skills, leadership and teamwork, business knowledge and technical skills, ethics, and analytical and quantitative skills. Results from the first survey (summer and fall 2010) are provided in the following graphs. In general, we are very pleased with the strong performance displayed by our students during their internships. In all areas, the great majority of students are evaluated as either accomplished or proficient, the top two categories. This reinforces the anecdotal evidence received informally from our internship employers, who have been very complimentary about our students’ preparation, ability and motivation.

Results from Internship Supervisor Survey – Ethics

Results from Internship Supervisor Survey – Business Knowledge

Results from Internship Supervisor Survey – Communication

Results from Internship Supervisor Survey – Leadership/Teamwork

Results from Internship Supervisor Survey – Analytical/Quantitative

We believe that this approach for assessing student learning outcomes in internships represents an innovative, best in class assessment that enables us to gain valuable feedback from one of our key stakeholders, employers. Participation in internships enables students to demonstrate their competency in a real world business setting. Because the internship process is formal and structured, and because we require this feedback of all internship employers, we feel that this assessment feedback is more accurate and complete than would be a similar assessment of employers of our graduates. As we continue to employ this approach during the coming years, we will closely monitor our students’ performance and use this data to inform future curriculum design.

Eagle Investment Group Investment Performance (Undergraduate)

The Eagle Investment Group (EIG) manages three separate portfolios totaling approximately one million dollars of the Ashland University endowment. The objective (and primary student learning outcome) of the group is to earn a reasonable rate of return relative to the economic conditions while following ethical principles and maintaining and appropriate level of diversification in the portfolio. An excellent authentic assessment of student learning outcomes is provided by the performance of the EIG group’s portfolios relative to the major market indices. The EIG group provides information on their achievement to the public by publishing an annual report. The EIG portfolios have outperformed the Dow Jones and S&P indices for nine of the past ten years. A summary of the results is shown the following table and graph. Although the high level of performance by Eagle Investment Group students did not suggest any specific changes, the performance of the Eagle group in 2009 led to a specific change in the way the Eagle group handles the summer transition between spring and fall semesters, as well as providing a “teachable moment”. The 2009 Annual Report of The Eagle Investment Group describes the situation as follows:

In 2009 … market volatility and uncertainty of the effects of government bailouts sent many investors running for very conservative investments including members of the Eagle Investment Group. After some well-evaluated investments posted negative returns for the group at the beginning of the year, the class of 2009 developed a very conservative bond ladder strategy. With this still in effect, and the inactivity of managing the portfolio during the summer, the group missed out on the summer run-up of about 30%. We have addressed this omission to make sure that the incoming class is vigilant about the markets and the portfolios.” (p.6)

When the next group of students took over the portfolio in August 2009, the investment strategy changed significantly, as students disbanded the bond ladder, generated cash, and reinvested the cash into stocks using a new strategy. As a result of the changes, the EIG outperformed the market indices in 2010.