Overview:
Heather Nelson, Logistics Management (LGM) Program Chair at the Ozark campus, is improving her student’ achievement in obtaining industry credentials through assessment. The Certified Logistics Technician (CLT)® certification process was incorporated into the LGM capstone course in 2018. After four years of offering the exam, data analysis revealed lower than ideal results. Ms. Nelson’s two-pronged approach to improve outcomes has resulted in a course rewrite and a shift in program timing of the credentialing exam. The results are highly anticipated - read on for the whole story…
Assessment Process:
Program Learning Outcome (PLO) assessed:
Demonstrate the ability to perform demand management to improve efficiency, perform inventory control measures, and optimize warehouse layouts.
Description of Measure:
LGM program learning outcomes are supported by the Certified Logistics Technician (CLT) ® certification program which “recognize(s) through certification, individuals who demonstrate mastery of the core competencies of material handling at the front-line (entry-level to front-line supervisor) through successful completion of the certification assessments.” The goal of incorporating this certification in the capstone course, LGM 2083 Operations Management, was to assess if the curriculum is in line with industry expectations. This certification was chosen due to the direct correlation with the LGM program learning outcomes and more importantly that the industry recognizes this credential, including Walmart using it for training front line workers. The CLT® certification program is also utilized by many other renowned employers including Amazon, Anderson Corporation, and Caterpillar.
Description of Expectations/Target:
The LGM program is offered in 100% online delivery. Students must pass the Foundational-level Certified Logistics Associate (CLA) Certificate exam to qualify to sit for the CLT certification exam. Students are not fully certified until they pass the CLT exam. Both exams have been given to students in the capstone course since 2018 and must be taken at an MSSC-approved proctor site. Prior to 2020, students took the exam in person at the Ozark campus. Since 2020, all students test online via ProctorU, which better aligns with the fully online program delivery.
Expectation of Target:
75% of students in the class will obtain CLT certification.
Findings/Results:
Review of the exams’ scores for four assessment cycles shows a lower than ideal passing rate for the CLT certification (2018: 0%, 2019: 17%, 2020: 80%, 2021: 42%). The chart below shows passing rates for the CLA and CLT exams.
Reflection on Findings:
Year over year, efforts have been made to improve passing rates including recommending students review course materials of select courses in the program and reorganizing and updating content of the CLT study guide. These changes were prompted by a survey item included as part of student’s final assignment for the spring 2019 and 2020 capstone course. The assignment stated, “Include in your Word document what you felt was the most difficult part of the exams, what material you felt you were NOT prepared for, and any recommendations for improvement in preparation/testing site/etc.” Students’ responses pointed to lack of preparation immediately prior to the exam and a large portion of the exam topics being covered in one course, LGM 2033 Warehousing and Inventory, taken in their first year of study. While these initial efforts appeared to have worked, since passing rates jumped from 17% in 2019 to 80% in 2020 despite the onset of the Covid-19 pandemic that spring semester, spring 2021 passing rates fell to 42%.
Plans for Improvement/Next Steps
The student feedback received during the spring 2019 and 2020 semesters had already led to a review of the LGM 2033 Warehousing and Inventory course. In that review, faculty determined that the course needed to be rewritten to more closely align with the CLT exam competencies. Over the last year and with faculty input, the LGM 2033 has been rewritten to not only align with the CLT competencies but also meet Quality Matters™ standards for online courses. Additionally, the credentialing exam has been relocated to the course with students taking the CLA for the midterm exam and the CLT for the final exam.
The redeveloped course is offered this fall 2021 semester and the course will be reviewed for QM approval during the spring 2022 semester. The fall 2021 and spring 2022 certification exam pass rates will be reviewed at the end of the academic year with 2022 annual assessment plan final analysis occuring in January 2023.
When Arkansas Tech commenced the HLC reaccreditation process in the Fall of 2018, departments across campus began building their evidence documents. The Math Department at Arkansas Tech was no exception. Kristi Spittler-Brown assisted in building a 70-page department assessment evidence document, which covered the overall mathematics success of the 2018-2019 academic year for the Russellville campus, Ozark campus, and Concurrent enrollers, as well as pre/post analysis, and remedial and gen-ed comparative analysis.
The depth of assessment the Math Department has gone through is what is noticeably impressive. Across Russellville, Ozark, and concurrent enrollment, the Math Department has broken down assessment across several factors, including:
 |
 |
The Math Department also implemented a Pre vs Post test for MATH 1113, MATH 1003, and MATH 0903. Across these courses, summary statistics covering mean, standard deviation, confidence interval, and many other statistics provided insight into their assessment work. Additionally, each test was broken down into a percentage of correct and incorrect responses, listing what each question covered. The report concluded with success rates for Remedial and General Education Mathematics courses from Fall 2016-2019 and recommendation for improvements.
Although 2020 had many challenges due to the pandemic, the Math department was still able to make a few changes based on assessment data. First, the post test data was gathered and assessed, looking specifically at the performance of each exam question. The Math Department Curriculum Committee examined these questions, rewriting them or making changes in education strategy to ensure a more consistent exam performance.
Next, there were topics that students still struggled with at the end of the semester. Different strategies were implemented into the program to improve the performance and retention of these topics. Unfortunately, recent semesters were impacted by the pandemic. Implementation was not ideal and data gathered was not considered normal. However, the department will be meeting prior to the fall 2021 semester to measure successful implementation fidelity of their strategies.
Finally, the department used student enrollment data to reach the decision to drop Intermediate Algebra. In addition to that, the Math Department will also only offer corequisite college algebra for STEM track students who need the course for remediation and college algebra.
Arkansas Tech’s Study Abroad program has kept up to date with their yearly assessment, as well as recently completing their CAS review. Gabriele Haulmark, Study Abroad Coordinator, has dedicated Study Abroad’s Mission to “increasing ATU’s presence and effective participation in the nation and the world.” The program partners with National Student Exchange and Global Endeavors to offer students a chance to broaden perspectives and experience history and culture firsthand.
Study Abroad uses two forms of assessment data. In addition to pre and post surveys (mostly rating scales), short answers from the surveys and student blog posts are used to measure their learning outcomes. These outcomes, created in 2017 by the Study Abroad Committee with the Study Abroad Coordinator, consist of:
Questions and blog-posting directions have been crafted to align with the learning outcomes. For example:
Haulmark finds the most satisfying and rewarding part of the assessment process to be reading the short answer and blog posts. The following excerpts of student responses and posts show alignment to the learning outcomes:
Outcome 1.
So far, I’ve learned a great deal about life in Graz and how people go about their routines. For example, a huge adjustment I’ve had to make is that stores and markets close extremely early (no Walmart people!). People here tend to run all of their errands before 6:00pm. This has been a big change, because you have to be prepared to get everything you need during the day.
Outcome 2.
For when I didn’t have class, I had arranged a small teaching assistant job for the duration of my stay in Austria at a primary school not far from Graz. I met with the professor at the beginning of February, and we had arranged for me to come to the school every Friday beginning on March 6th. Anywho, the teachers were excited and nervous to practice English with me, and the students looked at me as though I was magical, which really boosted my self-esteem. We played English games, sang songs, read books together, and learned dances all in the span of four hours. And I loved every second of it.
Outcome 3.
However, there is no doubt in my mind that I will return to Austria in the future. The life I had made for myself there completely on my own is something I’m very proud of! I’ve gained new independence and confidence in myself and with others. I can’t wait to see where that helps me with wherever life takes me. I am thankful again for the opportunity I had to go and the connections I’ve made all over the world.
Outcome 4.
On the 6th of February, I started German lessons. As I am sure to no one’s surprise by my intro, I started at the very bottom of the German knowledge pyramid. A1/1. This class is extremely fast paced as it is a full semester of German in only three weeks and very overwhelming at times as we are in class for three hours a day every day. However, I feel mildly more confident in my German capabilities and I have made it an effort to read all signs aloud to practice more and to repeat the announcement voice on public transport.
The first change Study Abroad made based on assessment findings was after the first year of surveys. The original surveys were taken with pencil and paper. The lack of efficiency was fixed with the surveys being taken online. This allowed the data to be analyzed easier and more efficiently.
The next issue that Study Abroad is planning to address is the lack of conclusive and useful data. This issue stems from students failing to complete both surveys (pre and post). Thus, there is no way to compare results. Haulmark will explore options to ensure the completion of both surveys.
Next, Study Abroad noticed that the survey questions did not go into enough depth to ensure that outcomes were reached. It was especially apparent for long-term study abroad students. Special questions were implemented for these long-term students to answer in the form of blogs or reports. Study Abroad will continue to evolve their assessment practices to improve the program and ensure student learning outcomes are reached.
Arkansas Tech’s College of Business recently implemented a new assessment process. This process helps the college meet the standards for assurance of learning as required by the Association to Advance Collegiate Schools of Business (AACSB), the accreditor for the college. A key feature of this new process was the development of rubrics by the COB faculty to assess the primary learning objectives of the college.
Process:
Dr. Hunter and Dr. Huang attended two AACSB seminars focused solely on the assurance of learning process. As a result of this training and because the college revised its mission and vision statement in Spring 2018, the learning goals and related objectives were revised by the curriculum/AOL committee to ensure the goals were consistent with the mission and were then submitted to the COB faculty for its approval. At a faculty meeting in September 2018, the undergraduate business faculty were divided into groups to develop appropriate rubrics for each of the relevant learning goals. Each group began the process by asking each group member to list as many items as possible on individual “sticky” notes that they felt should be considered for the rubric. After that step was completed, the group collected all notes relating to a single topic (for example, correct punctuation for the writing rubric); then, the group developed subheadings and assigned the various ideas to a subheading. Dr. Hunter then created a rough draft of each rubric that was analyzed, discussed, and revised by the COB’s curriculum/AOL committee. Once the committee approved the rubrics, they were distributed to the entire COB faculty. After some additional discussion and minor revisions, the rubrics were approved by the COB faculty. The graduate rubrics were developed by the graduate faculty using the same process.
Implementation:
The College of Business undergraduate faculty developed goals/objectives and then the related rubrics to assess oral communications, written communications, ethics, technology, and professionalism. The graduate faculty created goals/objectives and then rubrics to assess ethics, oral and written communications, and technology. The decision was made to implement the assessment plan gradually. The COB began by using the ethics and oral communications (UG) and Technology (G) rubrics at the end of the Fall 2018 semester; during Spring 2019, data was collected using the written and oral communications rubrics (UG) and the ethics and oral communications rubrics (G). In Fall 2019, the COB used the professionalism, written, and oral communications rubrics (UG) and the written communication rubric (G) to collect data.
Findings and Interventions:
With the introduction of the rubrics, it was found that the students consistently struggled with two items on the oral communications rubric at both the formative and summative level for undergraduates and the summative level for graduate students. The first issue was the students failed to use a significant portion of the allotted time or ran over their allotted time. The second finding was that students did not comply with the mandated dress code (regardless of whether it was professional dress or business casual). When students were told to dress business casual, business professional, etc. they had trouble grasping exactly what that meant. The first intervention made was to purchase large clocks/timers for the students. This allowed for them to see exactly how long their presentation had been and adjust their speed to either reach the minimum or not exceed the maximum amount of time. The second intervention was the coordination of the “Dress for Success” luncheon held by the College of Business. Faculty, staff, and 86 students attended the luncheon in business professional clothing to learn about office dress wear. The luncheon featured several speakers as well as a representative from Modern Woodman Financial, Matt Ward. It was used to demonstrate appropriate interview, business professional, and business casual dress. Using both the clock and Dress for Success interventions, the COB reassessed these objectives during the Fall 2019 semester and found that the students scored significantly better on the rubrics in both the timing and appropriate dress objectives.
Future Plans:
The College of Business will use the following assessment schedule over the next five years. The five undergraduate and four graduate objectives will continue to be assessed using rubrics and the resulting data will be analyzed to ensure continuous improvement within the college. The following assessment schedules allow the COB to “close the loop” two and one-half times within each five year accreditation cycle.
[menu]
Summer Bruch, Associate Professor, took over the leadership of the Art Department as Department Head in the fall of 2018. As a significant function in keeping the department on track, Ms. Bruch uses a combination of tools to complete the assessment process and to stay up to date from year to year. Summer uses Weave to break down student learning in each degree program throughout the Art department. Each project within Weave lists current learning outcomes, measures, expectations, and the findings/results. Summer believes staying current with assessment makes the accreditation process manageable. She also uses QuestionPro for informal evaluation in the form of senior exit surveys along with an alumni survey. Since Arkansas Tech has a university-wide QuestionPro account, the price was right.
“When we have programs with rapid growth, like Game and Interactive Media, current assessment materials make it easier to allocate resources.”
Since taking over the leadership of the Department of Art, Summer has implemented an effective assessment practice to stay current and continuously take action to improve every aspect of the student learning in the Department of Art.
