Faculty and student perceptions and behaviours related to information literacy : a pilot study using triangulation

This pilot study was developed to determine if the University’s students were proficient in IL based on the requisite skills defined by ALA (2000), to define faculty and student perceptions and behaviours related to information literacy (IL) and to test an evaluation rubric using empirical inquiry and triangulated methods. Findings suggested that not all students (n=164) had satisfactory IL skills even at the senior student level. While 4th year college students (seniors n=91) fared better on an IL survey when compared to 1st year college students (freshmen n=53), analysis of the senior students’ theses led researchers to believe that students were most likely not skilled in this area, and had an inflated opinion of their own IL abilities. Overall, students felt they were less IL challenged compared to the faculty’s (n=55) observation of the IL challenges experienced by the students. Students’ self-assessment of their literacy skills may have been coloured by the propensity of the faculty to over-edit students’ papers rather than simply providing constructive feedback, thus altering the natural end result. These authors used a triangulated approach including thesis review, comparisons between student and faculty survey responses and comparison of findings from the theses and the student and faculty surveys. Findings and discussion of methodology will hopefully provide valuable lessons for those interested in assessing students’ IL.


Background
Information Literacy (IL) is defined as the cognitive ability to know when information is required, and the associated skills to locate, evaluate and effectively use the information (Association of College and Research Libraries [ACRL] 2012; The American Library Association [ALA] 2000).According to Green (2010), IL is '...an effective solution to the difficulties of accumulating, appraising, and managing large bodies of information, knowledge, and ... literature ' (p. 313).Green proposes the most significant gauge of IL is probably one's ability to determine when enough information has been retrieved and recognition that continuing to gather data will yield nothing new.
According to ACRL and ALA (2000, p. 8-14), an information literate person should be able to: Gilbert and Rosario. 2013. Journal of Information Literacy, 7(2).81 http://dx.doi.org/10.11645/7.2.1793 • define and articulate the need for information • access the needed information effectively and efficiently • evaluate information and its sources efficiently • incorporate selected information into one's knowledge base and value system • individually or in a group use information effectively to accomplish a specific purpose • understand the economic, legal and social issues surrounding the use of information, and access and use the information ethically and legally.

The problem
At a small university in California, faculty and librarians were challenged in assessing the IL of students across the curriculum.Much of the feedback the librarians received about IL was anecdotal and unlikely valid or reliable.There was a paucity of empirical data to support students' IL skills and knowledge, yet one of the educational goals of the university is that the students are able to 'use the information technology proficiently with the ability to evaluate critically the quality of sources' .

Review of the literature
In the spring of 2009, Head and Eisenberg (2009) collected self-report data from 27,666 students from 6 colleges and universities in the United States.Their analysis focused on a sample of 2,318 respondents.Their online survey was designed to provide a better understanding of students' experiences in the research process and to gauge their IL.The questions they posed were 1) 'How do early adults define and conceptualise the process of research?' and 2) 'What steps do early adults take to locate, evaluate, select and use resources required for course-related and everyday research?' (p.40).
They found that students embraced brevity, consensus and currency of research.Students were savvy about sources, systems and services, and developed problem-solving strategies as necessary.Of the respondents, 65% wanted to know the big picture when beginning course-related research; almost all used course readings, Google or Wikipedia for everyday research.Of these, 80% reported 'rarely, if ever' asking librarians for help with research assignments.However, 90% of the students used library databases for online course-related research and believed these resources provided credible content and in-depth information sufficient to meet the expectations of the instructor.
Between March 2010 and May 2010, Head and Eisenberg (2010) surveyed students on 25 US university campuses (n=112,844) to determine how students evaluated and used information, specifically information-seeking processes, and any difficulties they had in conducting research.Analysis of this self-report data was based on 8,353 student responses.This work included students' criteria for evaluating web content, in asking for help with evaluation of content, evaluating library sources, student research styles and techniques, the use of productivity tools, difficulties with steps and stages of research, and determining what was important to them when conducting research.
In this sample, 84% found getting started to be the most challenging part of course-related research; 66% found it difficult to identify a topic and 62% found it difficult to narrow the topic; 61% were challenged in eliminating irrelevant information.Almost 50% of this sample asked faculty for support when assessing sources for the course; only 11% asked librarians.These students were concerned with currency when searching web content (77%) and when using library sources (67%).Students' primary concerns were passing the course (99%), completing the assignment (97%) and getting good grades (97%).Less than 80% said learning something new was important.Diep and Nahl (2011) were interested in obtaining views of Vietnamese library administrators, staff and faculty about their students' IL.Researchers surveyed four universities; the sample included 149 online responses and 133 paper/pencil responses.When asked if students were knowledgeable about citing references, 'strongly agreed' and 'agreed' were collapsed into one variable.In doing so, authors found that 35% of the librarians strongly agreed/agreed yes, 42% were unsure, and 22.5% strongly disagreed or disagreed that their students were knowledgeable about citing references.Of the faculty, 47% believed that students were familiar with citing sources, whereas 44.7% thought otherwise; 37% of the librarians assumed students were informed about copyright when utilising sources, 24.7% agreed, while 54% disagreed or strongly disagreed.
Diep and Nahl identified faculties' perceptions about IL, including the perception that IL required little attention, that the need to communicate the importance of IL to the students was unnecessary and a lack of understanding about the role of IL in 'helping students to become effective learners'.Further, they identified lack of collaboration between faculty and librarians, insufficient support and lack of resources and limited knowledge about certain subjects among librarians (75% of faculty thought the lack of librarian's knowledge on certain subjects was a salient issue).
In 2001, Maughan studied students at a northern California university and found vast differences in undergraduate students' perceived abilities and the actual results of a skills test.
In this study 70-77% self-rated as excellent or good and about 14% rated themselves as poor.However, on the skills test, 35-81% either scored poorly or could not pass the test.Gross and Latham (2009) discovered that students chose to ask for assistance with information resources from acquaintances or unfamiliar persons who seemed friendly rather than seeking help from an instructor or librarian.Diep and Nahl (2011) found that 89% of the faculty assumed that students sought advice from instructors when searching for information, whereas 55% of librarians were uncertain about this.Gross and Latham (2011) found that students who scored below proficient on objective IL tests often had a 'miscalibrated view of their own abilities'.Their sample included students attending two community colleges (n=577).Of these, 52% said they were self taught in terms of information research, 34% said they learned from a friend and 14% learned from a parent, self or a combination of the three.At the first school, the students' mean self-estimated performance was 76%, but the mean student IL test score was 44%.At the second school, the students' mean self-perceived literacy pre-test was 78% but the mean student IL test score was 54%.The difference between the mean test score and the estimated ability in both schools was significant at p<.0000.
Among a sample of faculty (n=419), Singh (2005) found that only 4% thought all of their undergraduate students met the ACRL criteria for IL; 42% thought some met the standards; 23% thought that only a few of their students met the criteria; about 1% thought that none of their students met the criteria.Among the graduate faculty (n=362), 1% said that none of their students met the criteria; 9% said all students met the criteria; 44% said that most of their students actually met the criteria.Although a high percentage of the faculty felt that not all of their students were information literate, the most frequent response to a question about assignments that required research was 'every' and the most frequent response to the question about requiring library instruction, was 'none'.

Purpose
For our colleagues who may not be familiar with the terms, in the United States freshmen refers to first year college students.rubric and to conduct empirical inquiry using triangulated methods to assess whether students at the university were able to: • define the scope of a thesis or research question • select relevant information and sources • incorporate a variety of information and sources • evaluate information for bias, fairness and accuracy • access and use the information ethically and legally • critically evaluate the quality of resources.
These researchers looked at students' knowledge, skills, best practices and challenges, and compared differences between freshmen and seniors.Also students' perceptions of their challenges were compared with how challenged the faculty perceived the students to be.Student capstone projects (theses) were analysed to determine student levels of IL based on a rubric with a scale of 4 to 1, with 4 being the most desirable evaluation.This information was used as part of the Library's Self Study in the spring of 2013 (the Self Study is the University's way of evaluating each major programme on a five-year cycle).!

Ethical considerations
This project was approved by the University Institutional Review Board (IRB) for the Protection of Human Subjects, IRB #10022.No formal consent was required as consent was implied if one completed the anonymous survey.The students' theses were anonymous to the reviewers.!

Methods
This project used triangulation by comparing faculty and students, students and students and comparing quantitative results from student responses with narrative work of the students.

Instruments, validity and reliability
The surveys were developed from the reports of Head andEisenberg (2009, 2010)  The instruments were vetted for content validity by librarians, library staff and faculty; changes were made based on the feedback from these individuals and students.External validity must be considered limited due to the narrow demographics of the sample.Reliability was supported by having six students pilot the survey and using Cronbach's alpha with the total sample.

Data collection and analysis
Student surveys were distributed in a classroom setting.Faculty surveys were sent via email with a survey link.It is unclear if the faculty who allowed us into their classroom to survey students completed the survey themselves as it was anonymous; however, participation was encouraged among all faculty.Senior theses were collected from a sample of spring 2012 senior students' work.The authors asked the departments to send a random sample of theses for review; we received 12, and it is unclear whether they were randomly selected or not.
Data were analyzed using SPSS v.17 for Windows.Analytical procedures included frequencies, independent t-tests and ANOVA.Some data were collapsed to measure certain constructs; other data were reported in detail.For the purpose of student / student comparisons, only the freshmen and seniors were included in the analysis to measure change from incoming to exiting students; graduate students were not part of this sample, and we did not include the sophomores and juniors in the student / student comparison because of the low sample size.For faculty / student comparisons, all students were included in the analysis since the mix of courses taught by the faculty included freshmen, sophomores, juniors and seniors.
This study only looked at 12 senior theses because of the pilot nature of this study.It was the desire of the researchers to refine and vet the rubric so that it could be used more successfully next year.It became clear when reviewing one thesis that modification was necessary.Theses were initially assessed using the AAC&U VALUE rubric (Association of American Colleges and Universities 2012).This rubric proved difficult as it was intended to assess a portfolio of work over time rather than an individual thesis.The rubric was revised by the librarians using a similar scale and criteria, focusing on quality, relevance, variety and use of sources as indicators.The first thesis was read and vetted by six librarians and recommendations made for changes.Afterwards, each thesis was read and assessed by two different librarians and an average score was obtained.The inter-rater reliability was established and a means calculated.

Results
The total sample (n=219) included 164 students and 55 faculty members.Among the students 53 were freshmen, 7 sophomores, 12 juniors, 91 seniors and 1 did not respond to class level.Among the students, 60 were from nursing, 27 from communications, 22 from business, 10 from biology and the remaining from chemistry, comparative literature, English, humanities, international studies, music, liberal studies, occupational therapy, political science, psychology or undeclared.This distribution is somewhat consistent with the overall makeup of the university, where nursing makes up the major part of the population.However, this distribution primarily reflects the faculty's willingness to allow data collection in their courses.The majority of the students (87%) reported their grade point average as 3.00 to 3.99 (46 freshmen, 4 sophomores, 11 juniors and 81 seniors).Of the faculty who responded to the survey, 4 were instructors, 21 adjunct professors, 18 assistant professors, 3 associate professors and 9 full professors.Grade level teaching distribution showed that 5 faculty taught mostly freshmen, 16 taught sophomores, 15 taught juniors, 9 taught seniors and 8 taught graduate students.!

Student responses: best practices, knowledge and challenges
Freshmen and senior student responses were compared to see if there were significant differences in self-reported best practices.Seniors were significantly more likely to say they determined search terms early, used different types of resources and used interlibrary loan.Freshmen were more likely to say they were easily frustrated when researching topics.Although not significant, when seniors were compared to freshmen, seniors were somewhat more likely to say they created a thesis or problem statement and developed an overall search plan before beginning research.25.9% Fr. = Freshmen (9 th grade students) Sr. = Seniors (12 th grade students) SA/A = collapsed strongly agree and agree 1 = strongly agree 2 = agree 3 = unsure 4 = disagree 5 = strongly disagree *independent t-tests significant differences in the means = .05or less For the six collapsed variables for knowledge, the multiple choice questions were coded either 1 for correct or 0 for incorrect and summed.Using this collapsed data, the Cronbach's alpha was 0.53.Even though the reliability was low, there was a significant difference between seniors and freshmen.The seniors' mean knowledge was 3.18 (sd=1.36,n=89) and the freshmens' mean knowledge was 2.06 (sd=1.36,n=52), t=4.717, p=.000.Table 3 provides greater detail and chi-square analysis for correct answers between the freshmen and the seniors.To determine students' overall perception of their challenges, 19 items were collapsed; Cronbach's alpha for this group of questions was 0.90.Comparisons were made between freshmen and seniors and there was no significant difference in the challenges perceived by the students.However, when comparing faculty and students' perceptions, there were significant differences.The student overall mean score was 2.14 (sd=.50); the faculty mean score was 2.56 (sd=.36,t=6.132, p=.000).Most significant scores from the disaggregated data are detailed in Table 4.In every case except 'getting started', the faculty perceived the challenge to be greater for the student than the student perceived the challenge.Students were asked to identify which criteria were most important to them when evaluating a website as a reference for an academic paper (they were told to choose five).These findings are found in Table 5.

Thesis Analysis
Twelve senior theses were surveyed to determine the general level of IL skills for undergraduate seniors.Theses were submitted to the university to fulfill the Bachelor of Arts or Science degree.The sample included three theses from nursing, four from humanities, one from political science, one from history and three from communications.The theses are the students' final work and may reflect suggested revisions and input from faculty; this being the case, these authors recognise the limitation of this work in determining students' IL.Also, we do not know what criteria the faculty gave the students for developing their research papers and so it would seems somewhat unfair to analyse the students' work based on a scale with which they were not familiar.
However, overall the theses seemed well written and engaging, and thesis development tended to be good to average.While some thesis statements were clear and direct, they were expository rather than analytical, evaluative or argumentative.One student's topic was explained in detail but never clearly defined in a thesis statement.Most theses were similar to reviews or summaries of existing literature or ideas.Also, it was expected that senior students would have more experience in eliminating irrelevant sources, which was not the case.Correct use of citation styles was inconsistent.Three of the theses were free of major errors; however the rest had difficulty with consistency, used mixed styles or used no formal style in referencing.Few provided copyright permission for images and tables pulled from other sources, and at least one thesis provided no credits whatsoever.In some cases authors provided in-text citations for the source material, but relied too heavily on lengthy direct quotes rather than paraphrasing; they did not discuss or provide insight on the information quoted.
Overall students did not incorporate a variety of scholarly sources.Problems included too few sources, overreliance on one type of source, overreliance on non-scholarly sources (newspapers, popular websites), little use of peer-reviewed scholarly journals and very little use of books even though librarians felt books might have been useful for some of the topics.There was an overreliance on newspapers and popular websites, and students' inability to evaluate bias and accuracy tended to be problematic.Table 6 identifies the mean scores of the students.This score was calculated by adding the rating scores allocated by the librarian reviewers and dividing by the number of reviewers.
In reviewing the seniors' theses, the librarians reflected on how the university once required students to take a one-unit IL class prior to graduation.This class was cancelled in 2010 in favor of embedding these skills in freshman year courses.Anecdotally, since the change, faculty and librarians have noted an improvement in student success.However, many faculties now invite librarians to give guest lectures or require one-on-one meetings between students and librarians in sophomore, junior and senior level classes.When students' theses scores were compared to the number of librarian-led guest lectures and librarian-student research appointments since August 2011, the programs with greater librarian contact showed higher scores on the rubric (Table 7).It is difficult to make any meaningful correlations given the sample size and an inability to determine if these 12 students took the preparatory research course, attended a guest lecture or had an appointment with a librarian.It may also be the case that the students' work was edited by the faculty member and returned for re-editing multiple times.Additionally, we do not know what guidelines were provided to the students in the development of their thesis and how that compares with the rubric used to evaluate their work by the librarians.

Faculty specific items
When faculty were asked how many times they provided feedback on student's papers before the final was due, 5 (9%) answered 0, 15 (27%) said 1 time, 13 (24%) said 2 times, 13 (24%) said 3 times and 9 (16%) said they provided feedback more than 3 times before the final student paper was due.One faculty member wrote, 'For some courses, it depends on the number of students enrolled and how many units I get as the instructor.'Also, faculty were asked several questions about grading and rubrics.When asked, 'How do you assess your students' ability to determine the nature and extent of the information required for a class project, proposal or thesis?', 53% said these criteria are formally incorporated into the course and are graded using a detailed rubric; 31% said these criteria are stated in assignment but not graded using a rubric; 11% said these criteria were not stated in the assignments or in rubric form but are implied in expectations of the students' work; and 6% said they did not measure these criteria.I break the thesis into small chunks to make it more doable.
Moving forward, my students will be required to collect data and to tell a story with that information.How do you assess the student in his/her ability to use information effectively to accomplish a specific purpose?
46.3% 25.9% 24.1% 3.7% These questions are too abstract.I ask lateral thinking questions on an exam -I teach economics.
How do you assess students' ability to understand the economic, legal, and social issues surrounding the use of the information and whether they access and use the information ethically and legally?
18.5% 29.6% 25.9% 25.9% The syllabus has the standard language about plagiarism but in class there is much discussion of reliable sources.

Conclusions and discussion
There were mixed results in students' reports.For example, 77% of all students said they created a problem statement or thesis statement before beginning the research process, 64% developed an overall search plan before beginning and 65% figured out search terms early; yet almost 60% still reported having a hard time getting started.Also consistent with Head and Eisenberg, many students agreed had difficulty narrowing down topics, determining credibility of source, etc.Of these students, 43% reported getting frustrated researching topics, which may account for the 23% who reported using the same topic for multiple assignments.Consistent with the findings of Gross and Latham (2011), these students also seemed to have an inflated view of their abilities, though they mostly reported struggling with the process.
A contributing factor to the students' lofty notions of their abilities could be the enabling activities of the faculty.It was discovered that many of faculty aggressively edit the students' papers rather than simply providing constructive feedback.Of the faculty respondents 40% said they allowed the students to turn in drafts three or more times before the final grade was assigned; this also included correcting format, suggesting citations and references, correcting sentence and sequencing structure, etc.In the end, incorporating the corrections and suggestions from the faculty could give students a false sense of the quality of their work, rather than considering how much of the work may simply be that of the faculty member.The review of the theses was consistent with the empirical data from the faculty surveys; students had difficulty with many aspects of the research process.

Limitations
Using the VALUE rubric proved difficult, and the newly developed rubric was not thoroughly vetted.The narrow demographics and small sample size limited external validity, there was low internal reliability for best practices and knowledge and the survey requires further testing.Also, there were questions about the validity of the theses in providing a true picture of students' IL.Reflective discussions with faculty and librarians posed the questions: 1) How much of the final thesis work could be attributed to the student vs. the faculty's edits and support, 2) Would a student paper on how he or she describes the process of developing a new research paper be more telling in terms of true student IL and 3) Would use of a formal IL assessment tool (not initially considered) such as SAILS (Standardized Assessment of Information Literacy Skills 2013) be more informative and helpful in assessing our students' IL.Not addressing these questions before the research began was a limitation.These authors recognise the limitation of using student theses that have been edited and corrected by faculty before final submission; using the students' drafts before they were graded and retuned for revisions would be a better gauge of the student's actual skills.This pilot study was an excellent means of identifying these limitations and will guide our future endeavors related to assessing IL.

Lessons learned and recommendations
Faculty and librarians have the opportunity to collaborate in order to better support our students regarding IL.While these authors were pleased that the seniors demonstrated greater skills, knowledge and practice compared to the freshmen, seniors were still not at a level consistent with a senior bachelor's degree student.To know when enough information has been retrieved and recognition that additional data or information will yield nothing new means that the students must be able to identify, categorise and synthesise the data they collect which typically was not the case.These authors note the following lessons learned and have recommendations or ideas to consider for improving students' IL: • Lesson 1: The student's final papers may not be a true reflection of their abilities.
o Recommendation: to determine true IL among students it may be better to evaluate students' first drafts and/or require the students to write a paper describing the process they followed in creating a research paper, i.e. search techniques and outline of their process, search terms and database and sources they searched.This descriptive paper should be attached to the student's first draft of the paper and the paper reviewed for IL.This would provide greater insight into the student's abilities.o Recommendation: provide workshops or guidelines for faculty to help them enhance their ability to provide appropriate feedback on students' work that does not include heavy editing and rewriting of the student's paper.• Lesson 2: Students do not seem to be taking advantage of the mentoring and support that is available through their librarians.
o Recommendation: Ask faculty to assign the students one-on-one meetings with the librarians during each academic semester.This should be facilitated through classes that require research and a paper.This will lessen the burden on faculty, may increase student retention from one year to the next, will increase and support students' appreciation for the value of library consultation and will broaden the students' skills and knowledge related to IL and the topics of their research.This will also result in a more dynamic learning environment that is interactive and engaged learning.o Recommendation: Provide a library orientation each semester and a library tour during new student orientation.This would allow the students to be introduced to their own personal librarian (depending upon their major), to have direct contact information for their librarian and to gain knowledge of the support available at the library.o Recommendation: Provide online access to librarians during hours when the library is closed.This allows the students to get support in the evenings when they may be working on their papers.• Lesson 3: Development of a survey may be less valid and reliable than using a preexisting tool.
o Recommendation: Investigate an existing survey for use rather than creating one's own.o Recommendation: Involve the students in the IL assessment process, i.e. seek their help in assessing existing rubrics, evaluating and validating surveys, collecting data, etc.
and Rosario.2013.Journal of Information Literacy, 7(2).87 http://dx.doi.org/10.11645/7.2.1793 and Rosario.2013.Journal of Information Literacy, 7(2).88 http://dx.doi.org/10.11645/7.2.1793 Sophomores are second year, juniors are third year and seniors fourth year college students.The purpose of this pilot study was to test an evaluation . An IL rubric was modified from an AAC&U VALUE rubric (Association of American Colleges and Universities 2012) and used to evaluate a random sample of senior theses for IL (see Rubric below).3. A faculty survey which asked about faculty assessment practices and how faculty perceived students' abilities and challenges related to IL (view survey at: https://www.surveymonkey.com/s/D2RHNRC).