"I spend my time in the library because it is a quiet place where I can focus and work to my best ability."

Stephanie Zulla, freshman, Social Work

Assessment Planning- Strengthening the Teaching and Learning Environment at Mercy College

General Education Student Artifact Rubric Evaluation Pilot Summary - Summer 2012

            The Mercy College Libraries initiated a pilot program to evaluate a representative sample of student artifacts from the undergraduate General Education core curriculum. This sample consisted of outlines, bibliographies, research drafts, completed library instruction assignments and final drafts of research papers. The sample artifact items were produced during the Spring 2012 semester and gathered from General Education courses English 112, Communication 110, Sociology 101, Critical Inquiry and Junior Seminar. A total of 52 student artifacts were evaluated in the pilot program. 

Overall, the evaluation process proved efficient and time-effective. Artifact upload and rubric scoring were accomplished with minimal delay and software/system malfunction.  Based on the pilot program results, it is recommended that the student artifact analysis be expanded and integrated into library information literacy instruction and student outcomes assessment across all four Mercy College campuses beginning Fall 2012.  It is further recommended that the sample size for the artifact pool from each campus reflect a representative percentage of each campus library’s instruction presence in the General Education Curriculum.

Moving forward, intensified focus of library instruction on information literacy skills for upper-division undergraduates is strongly recommended to enable an increase in the level of competency to “2” or “Meets” for all criteria.  Results of the current study will be presented to the Seminar Faculty Cohort and their collaboration will be sought for the introduction of an expanded information literacy program for Junior Seminar. A two-step library instruction program will be recommended this academic year for Junior Seminars sessions.  It will include:

1. Expansion of  basic research skills
2. Integration and acknowledgment of information resources in scholarly communication format.

Results will be compiled after Fall and Spring semesters and analyzed for signs of improvement in each level of competency.  The full report is available below in Attachments as General Education Student Artifact Rubric Evaluation Pilot 2012.

Library Consultation Project - Spring 2012

Much of our self-assessment efforts have focused on first modifying our current reference and research assistance practices to make them more engaging and effective for students.  We removed the traditional reference desk model in fall 2011 and replaced it with a three tiered system of assistance.

Level One provides basic, navigational and technical assistance.  This is provided by all library staff.  If students require more intensive assistance, they are channeled towards an on –duty librarian for a just-in-time Level Two assistance, in which the librarian provides direct assistance to the student for as long as they need it.  Level Three consists of a scheduled, one on one, Consultation Session with the student in which the librarian provides highly intensive, in depth assistance that can last from 30 to 60 minutes, and in several cases, even longer.  Often, they will continue into a second and third scheduled session at future dates.   In many cases, students are asked to bring material and assignments to the meeting and the librarian will often prepare for the session beforehand to maximize the session. 

These sessions have proven to be highly effective and satisfying for students.  Several key learning outcomes have been integrated into the design of the Consultation session as follows:

• Students will be able to develop a research topic and effective research questions
• Students will be able to navigate library resources and tools
• Students will be able to develop keyword search strategies for searching databases
• Students will be able to search the web for relevant and reliable information
• Students will understand when and how to cite resources to avoid plagiarism
• Students will know how to access materials held in other libraries

The Consultation sessions are most successful when they take place in a quiet area.  Currently we utilize the Library Reference Department and offices for this purpose.  Librarians are available 10-9 Monday through Thursday and 10-5 Friday through Saturday for these scheduled sessions.

Consultation Learning Outcomes Study and Data

To understand better how students were being impacted by the Consultation Sessions, in the Spring of 2012 the Dobbs Ferry librarians launched  a survey, both through use of a Lime Survey and through direct questionnaire, which deployed key questions to measure student learning outcomes and were based on the library’s information literacy rubrics.  189 consultation sessions were conducted in total and 67 students responded to the survey – a 33% return rate.   The full questionnaire with charts is included as an attachment below. 

Summary of Results

The results indicate that formalized, one on one consultation sessions with students yield highly effective results for students.  Roughly 90% or more of all students surveyed, for all questions, strongly agreed that the session had a positive impact on success in their research assignments and in their learning about specific information literacy components and issues. 

Intended as a pilot study, we have determined that these surveys point toward the need for increasing the level and depth of one on one consultation sessions with students and the exploration of a Personal Librarian initiative for students in the majors and capstones.  By increasing our level of one-on-one engagement with students, more students will benefit from research assistance. 

The results also indicate that we will need to deploy the same survey technique in the 2012/13 academic year, with more students, in order to expand the breadth of our measurement and assessments.    We are energized and pleased by the results of the Spring pilot surveys and with a new Assessment Librarian in place for the fall, we are positioned well to begin robust student learning outcomes measurement going forward.

Comprehensive Learning Commons Service Forms:

As part of its ongoing commitment to self-reflection and evidence-based improvement, the libraries and learning centers continually place high importance on the assessment process.  The Title V Grant also requires that we develop a comprehensive Learning Commons Service Form to be completed by librarians and other staff working with students and patrons.  These two goals address the same concerns for gathering feedback on library usage and resources.  In 2009 we specifically included assessment goals and objectives into our Strategic Plan (listed below) and as part of this plan, we continue to find new tools and methods to assess our operations, services and the utilization of our resources. In 2005 and 2009 we applied the LibQual survey to assess perception of the libraries across three parameters: library as place, information control and affect of service.  The LibQual represents the gold standard in library service quality surveys.  We will be utilizing this survey again near the 2015 academic year. 

We also regularly disseminate library-built satisfaction surveys at least once a year for more direct assessment of resources and services.  For our information literacy program, we use general questionnaires distributed to students and/or faculty. The ongoing process consists of analyzing survey results and planning improvements based on this feedback, ultimately to measure and increase our impact on student learning outcomes and retention.  Many of these home-grown instruments themselves will be assessed and improved in future semesters.  Together, these instruments comprise the comprehensive learning commons service forms.  Highlights are described below: 

For 2010-11 we developed and launched several new improvements on our surveys to help in this critical process:

• Student Discovery Survey:  Like our "Blue Card" surveys from the 2009-10 Grant period, the Student DIscovery Surveys are general user satisfaction instruments designed to elicit feedback on services and resources as well as capturing student commentary.  Launched in November, 2011, the survey results were reviewed and evaluated in January 2012.  Sample report and summary are attached.

• Faculty Library Instruction Survey: To evaluate the effectiveness of our library instruction and information literacy program, we have also deployed retrospectively a faculty library instruction survey to all faculty receiving a library instruction session in Fall, 2011 at Dobbs Ferry.  This survey will lay the groundwork for a more detailed and comprehensive module that will be deployed in spring, 2012 to all classes that receive instruction, but tracked especially closely for the Freshman Seminar classes, where Information Literacy is a general education requirement.  For Psych 101 and Freshman Seminar, the survey will take the form of a pre/post questionnaire in order to gauge student impact more accurately.  Sample report attached.

• The Bronx Info Lit Survey:  Our Bronx Library regularly administers an assessment survey to students receiving an information literacy session in that location. 

• Manhattan Info Lit Survey:  Our Manhattan library has developed a survey to address effectiveness of IL sessions occurring at that location.

Evaluation of Information Literacy Sessions in Bronx and Manhattan Spring 2011:

• An information literacy questionnaire was distributed to Freshman Seminar and other course specific classes at the Bronx and Manhattan campuses.  The questionnaire was made up of five multiple choice questions focused on the usefulness of the particular library instruction session.  A total of 207 students from course specific classes filled out the survey.  A total of 121 freshman seminar students filled out the survey.  Results showed that most of the students were in agreement about the importance of understanding how to find and use information.

• When asked the critical question of how they begin their research, most students disagreed with "I start my research by going to the Mercy College Libraries homepage"   The response range for this question was from 1% of Freshman Seminar students in the Bronx to 8% of Freshman Seminar students in Manhattan.   This outcome indicates the library needs to examine and adapt best practices for teaching  information  literacy.  The library also needs to conduct one on one  interviews and/or focus panels with Mercy College students to better understand their research habits.

Yorktown Survey: Our Yorktown librarians administer a survey to all classes receiving instruction (attached).  Based on assessment of the results, a number of areas needing to be improved have been identified, as follows:

• Need to focus on freshman year and sophomore transfer courses.  Low scores indicate need to work more closely with these students.
• General Education Classes seem to be actively encouraging library instruction. We need to concentrate on the Freshman Seminar and Junior Seminar
• More emphasis on RefWorks citation software needed, so students understand how to cite properly.
• Using the computer labs or instruction rooms with computers would enhance presentation.

• Digital Storytelling:  This highly engaging exercise and authentic assessment tool has expanded in use from an initial group of 4 faculty in Fall 2010 to 11 in Fall 2011 reflecting a diffusion of innovation curve pointing toward increasing faculty adoption.  Freshman Seminar Faculty have been the majority adopters.  A retrospective faculty survey to understand how digital storytelling has impacted student learning in AY 10-11 will be deployed by the end of fall term.  This Fall FRS faculty has incorporated a post-reflection component that will measure success toward achieving student learning outcomes). This data will be forthcoming at this semester's conclusion as well. More information, the Survey, its results, etc, are available in the sub-section focused on Digital Storytelling.

In addition to regular surveys administered to patrons, the library has also updated other secondary materials that are also used as tools to assess patron experience.  These include a new Reference Form, used by the Dobbs Ferry Library has created a new reference tracking form to better track how we assist students.

Another powerful assessment objective is our goal of creating a Student Advisory Team – to be launched in Spring 2012.  This focus group will be comprised of a cross section of Mercy students and will utilize regular meetings as well as social media outlets to enable close and guided assessment of all library resources and services as well as provide a dedicated testing group for evaluating new initiatives.

Finally, there are a number of automated, information technology based data tracking systems and tools regularly employed by the libraries to gather data on various aspects of library services and resource utilization.  Attached is a table providing details on these systems.  These include:

• Koha: Library Management System reporting functions (for Circulation, Reserves, System Loans, etc)+
• Database tracking systems
• Google Analytics and Calendar

The attached chart details the other automated systems used.

Assessment Plan for Library Learning Commons

2010-2014

Strategic Goal #4.  Enable an atmosphere of collaboration and intellectual activity by involving patrons in the critical assessment of LLC functions and growth.

• Establish a student advisory board to partner with MCLLC to promote the tandem discovery of new ideas for creative growth of MCLLC.

• Develop and maintain a workable MCLLC usage policy which includes regular student feedback on environmental quality, service and resources.

• Conduct usability studies to improve the MCLLC website to make it more accessible, intuitive, and user-friendly.

• Adopt “green issues”, ADA concerns and cultural diversity as learning models and explore their application to the MCLLC physical space and functions.

• Create a responsive, creative learning environment by celebrating the achievements of the Mercy academic community and its programs by leveraging the MCLLC space for events and activities.

• Coordinate with Schools and programs to provide timely information on events, awards, etc.

• Utilize current Web 2.0 technologies to post changes, developments, new programs, and services to user.

Strategic Goal #6:  Assess the effectiveness of service to Mercy College community.

• Adjust the strategic plan on an annual basis in response to the changing needs of patrons.
• Assess MCLLC staff functions, librarian engagement, library service responsibilities, and Mercy
  College Community satisfaction through standardized assessment tools.
   
• Form a student advocacy group made up of a cross-section of students actively assisting the
  library with critical service assessment.
   
• Increase awareness of library resources through outreach, collaboration, information literacy,
  and student activities.
   
• Participate in a LibQual+ User Preference Survey to follow up the 2005 and 2010 surveys.
   
• Conduct annual workshops for MCLLC faculty and staff for performance improvement.