Research Practices Survey
The Gould library has taken a lead role in a MITC / NITLE grant to create a web-based assessment tool to measure the information literacy of incoming students, before they've had any college library instruction. This grant-funded project was originally called the First Year Information Literacy in the Liberal Arts Assessment (FYILLAA), and has since become a nationally administered survey called the Research Practices Survey.
The grant included librarians, faculty, institutional research staff, and academic technologists from St. Olaf, Macalester, Carleton, Grinnell, Lake Forest, the University of Chicago, Ohio Wesleyan, and DePauw University. The survey provides participating schools with data about their students, along with comparison data across from the other participating institutions.
The assessment tool focused on five dimensions of student information literacy: Experience, or what have students done; Attitude, or what do students value; Epistemology, or what do students believe; Knowledge, or what do students know; and Critical Capacities, or how do students evaluate.
The first full implementation of the survey happened in late summer and early fall 2005. Five of the schools, Carleton, University of Chicago, Grinnell, Macalester, and St. Olaf, had high enough response rates to allow for inter-institutional comparisons. A presentation of the project (linked to the right) was given at the AAC&U conference in March 2006. Another presentation (also linked) was given at the 2007 ACRL conference.
2007 ACRL presentation now available (linked on the right)
The full text of the 2006 survey is available for download from NITLE (referred to as the "Research Practices Survey").
The FYILLAA 2006-07 survey of incoming first year students was originally conducted Aug. 21 – Sept. 11, 2006 – before any of the incoming students would have had any library instruction. The survey asked students about their library experience, and their perceptions of their research skills, attitudes about research, and their proficiency with research. 250 students were randomly selected for the survey, and 187 of those responded (a 75% response rate).
We chose to re-administer the FYILLAA survey in the spring 2007 to the group of students who responded to the fall survey. We hoped to measure changes in students’ information literacy perceptions and skills that would have occurred during their first two terms at Carleton. 88 of the 187 students surveyed responded (a 47% response rate).
As benchmark data, the picture that emerges about students’ experiences, attitudes, and abilities is, in many ways, not surprising. However, there are several key findings we noted in the two surveys. Below are charts from these findings and our thoughts on what they may mean about the information literacy of our incoming students and their instruction and experiences during the first year at Carleton.
- Library use not only increases during the first year of college, but our students are more likely to come to the library for academic-related reasons.
- The kinds of resources and materials students are likely to use in their first year shift slightly away from more general materials.
- There is little change in the amount of support students seek from professors, friends, and librarians. However, the percentage of students who take advantage of the Writing Center increases significantly, and the percentage of students relying on parents or family members for academic support decreases significantly.
Chart 1: A full three quarters of the students responded in Spring 2007 that they used a college or university library once a month or more. That's up from 48% of the students who used a high school library once a month or more (only 13% used a university or college library once a month or more before coming to Carleton). None of the students surveyed in Spring 2007 said that they never used a college or university library.
Chart 2: The reasons for visiting a library changed. In spring 89% of students visited the library for academic-related reasons. That's up from 70% in the fall. A quarter of the students used the library for non-academic reasons in the fall. That fell to only 9% in the spring.
Chart 3: The types of sources used by our students in their papers. In high school, more students used reference sources and newspapers or magazines. Fewer used such sources at Carleton, indicating a preference for more scholarly material.
Chart 4: How do students search for sources? In high school, more students were more likely to used general search engines. At Carleton, the use of general search engines dropped, and the number of students using the library catalog and online indexes increased, showing a better understanding of some of the research tools available.
Chart 5: Who do our students turn to for advice? Writing centers showed the greatest increase in number of students who use their services when working on assignments. We saw only a modest increase in the number of students turning to librarians for support. This may be a result of the types of assignments that first year students typically have. It would be very interesting to see this question posed to seniors after they have finished comps. It is also noteworthy that the percentage of students turning to a parent or adult family member dropped significantly. This is surprising given the amount of both data and anecdotal evidence in higher education right now suggesting that the current generation of students’ connectedness to parents extends to help with academic work.
Our library instruction program is reaching first year students. Carleton does not have specific courses that all first year students, so it is difficult to have any program that specifically reaches all students. Despite this limitation, 88% of the students who took the survey in the spring indicated that they had received library instruction in the past academic year.
Perceptions of Research:
- Perceived difficulty of research increases once students are at Carleton College.
- The percentage of students reporting that they “enjoy” research drops during their first year at Carleton.
Chart 6: The majority of our students do not consider research to be difficult. However, more of the students surveyed after their first year at Carleton were likely to find various components somewhat or very difficult. Since students are more likely to admit that research can be difficult once they’re at Carleton, this may be an opportunity to reach more of the students and offer assistance.
Chart 7: Not only are students finding research to be more difficult at Carleton, they are also enjoying it less. The number of students who enjoy research "very much" or "quite a bit" dropped from 44% to 28% once students were at Carleton.
- Students are more likely to understand Boolean logic, truncation, and sources of citation after a year at Carleton.
- Many students struggle with some of the “basics” of information literacy—boolean logic, truncation, and understanding the pieces of a citation.
- Students increase their understanding of available research tools and make decisions about research based on the “scholarly” nature of the resources.
- Students continue to struggle with characterizing “scholarly” information.
Chart 8: While more students were able to correctly identify the best way to retrieve the most results in a search after their first year at Carleton, 40% still do not know the best search strategy for combining terms in this scenario.
Chart 9: Similar to the previous chart, our students are more likely to know the correct ways of truncating words when doing a search after they’ve been at Carleton. However, the number of students who knew the correct method is well below half. Given their proficiency at using technology, we often think that our students would be bored by discussions of Boolean search and truncation methods. That may well be the case, but given this information, we need to make sure we do not assume that students know these basics when we work with them in classes, appointments, and at the Research/IT desk.
Chart 10: Prior to coming to Carleton, when given various citations of different types of sources, most students could only correctly identify the book citation. After their first year at Carleton, more students were able to identify the journal article and book section citations. However, only slightly more than half (51%) were correct about the book section citation.
Chart 11: Prior to Carleton, only slightly more than half of our students would search an academic database or index to find comprehensive results on a topic. After their first year at Carleton, that number increased to 72%, indicating that are students are learning about some of the research tools available to them.
Chart 12: Our students are still confused about the scholarly nature of popular magazines. Before Carleton, 34% of students though articles in magazines like Time, Newsweek, or US News and World Reports were scholarly. That number dropped after their first year at Carleton. However, more than half still believe that articles in these magazines may be scholarly or that you cannot determine the scholarly nature of those articles. This is another information literacy basic that is often left out of instruction in favor of more complex concepts due to time constraints.
Chart 13: After their first year at Carleton, more students are likely to choose a source based on its scholarship. However, a significant number of students consider all of the reasons listed as equally valid, which includes reasons such as the convenience of retrieving the source.
Attitudes about research:
The following statements were posed to the students and they were asked whether or not they agreed with the statements. The differences between the responses of students prior to coming to Carleton, and then again after their first two terms is an important look at how our students attitudes change and how some remain the same.
|Statement||Percent of students agreeing in fall 2006 (pre-Carleton)||Percent of students agreeing in spring 2007 (after two terms at Carleton)|
|Skillful researchers know the best way||83.8%||69%|
|A course in research skills would be useful||67.7%||55.8%|
|Successful researchers understand things quickly||28.6%||14.9%|
|Good research yields clear results, poor research yields ambiguous results||47.6%||33.3%|
|Some people are just naturally better at research||70.3%||74.7|