Reports

Output Measures Task Force Charge
-The task force has clarified and redefined the charge; please see section II, below.

1. Summary of recommendations:
   1. Create a higher-level Output Measures Task Force in 2003-04 with one member being at the director level to better emphasize administrators' need to incorporate output measures in institutional strategic planning. Action items for that task force might include:
      • Identify specific BLC initiatives to be assessed, such as the processing turnaround time for ILL among libraries, as suggested at the May 27th workshop; or the qualitative impact on staff and workflow in those libraries using the Virtual Catalog.
      • Continue monitoring of assessment tools - LibQUAL+™ (http://www.libqual.org/), E-metrics (http://www.arl.org/stats/newmeas/emetrics/index.html), local assessment and keep the BLC community apprised of these;
      • Investigate the feasibility of the BLC's (as a consortium) registering for LibQUAL+™, and contact previous consortial participants to determine questions asked, comparisons made, etc.; incorporate output measures and outcomes in the organization's long range planning;
      • Consider assessing the impact of the BLC task force programs offered over the past two years - how have these benefited services and workflow within BLC libraries?
        
        
   2. Clarify the task force's charge when that group is first convened;
      
      
   3. Provide that or any task force with BLC's funding guidelines for programs, speakers, etc.;
      
      
   4. Continue work on activating the output measures COI, possibly convening a face-to-face meeting of the COI. There is obviously a lot of interest in this area and the BLC could help keep up enthusiasm for the topic;
      
      
   5. Identify experts in assessment among BLC libraries and form an "assessment bureau" of librarians willing to share their expertise.
      
      
2. Background information (optional):
   See Interim Reports of November 8th and January 28th.
   
   
3. Programs planned or implemented:
   As the former Staff Development Task Force strongly recommended in its June, 2002 report, we organized a well-attended, well-received half-day workshop on output measures on May 27th, 2003, at the BPL. The program included: Jo-Ann Michalak and Ed Oberholtzer (Tufts) on their long-term locally created surveys and recent experience with LibQUAL+™; Dan O'Mahony (Brown) on his institution's experience with LibQUAL+™, complete with local and aggregate data; Brinley Franklin (UConn) on his experience with the ARL E-Metrics Project for electronic resources; and Barbara Preece on the BLC's use of inputs and outputs. The Q & A portion's going into overtime was an indication of the success of the program and the enthusiasm for the topic.
   
   To view these presentations, please see the BLC website -http://www.blc.org/task_forces/FY03/output_measures/om_program.html
   
   
4. Resources used (how have BLC resources been used?)
   We drew on BLC staff for help and advice (specifically Melissa Petze and Barbara Preece) and made extensive use of the conference call technology.
   
   
5. Activities (summary, include milestones and challenges faced)
   We met 14 times via conference calls between February 6th and June 4th. Our primary goal was the planning and organization of the May 27th workshop. Our biggest challenges were the uncertainty of our charge and working without funding guidelines.

Output Measures Task Force Charge
-The task force has clarified and redefined the charge; please see section II, below.

1. Meetings held (virtually and/ or face-to-face)

   Since our November 8 report, the BLC Output Measures Task Force has met via conference call on November 19, January 7, January 9, January 16, January 22, and January 28.

2. Activities/Accomplishments to date (list or summarize)
   
   
   1. The task force clarified its charge, confirming that it should look "at ways to evaluate and assess BLC-sponsored projects," as per Barbara Preece.
   2. The task force formulated a short list of questions which Barbara Preece sent to the Consortium listserv on January 17, attached as Appendix A along with the two responses received to date.
   3. The task force received responses from 12 BLC institutions, representing 15 libraries, to its survey of BLC member libraries regarding assessment. A summary of responses appears in
      Appendix B.
   4. Regarding the workshop on output measures, the task force is seeking to clarify the focus of the workshop, as envisioned by the BLC; see "Requests," below, section IV/C.
      
      
3. List of COI's with which you communicate

   The Output Measures COI. The task force sent several questions to the COI in November to initiate discussion, without results. The task force will try engaging the COI again once the workshop is planned.

4. Requests
   
   
   1. Funding: Once workshop plans are underway, the task force will submit a budget to the BLC to cover speaker fees and expenses for a workshop on output measures.
   2. Resources: none requested at present.
   3. Board/ MC queries: The task force would like to clarify the nature of the workshop, as envisioned by the BLC. Rather than focus on specific off-the-shelf products such as LibQual+ or projects such as E-Metrics, the task force is looking to create a workshop on general subjects of outcomes and assessment, as supported by results of the task force survey and board feedback. These would include evaluation procedures and the process involved in developing an evaluation plan, such as establishing evaluation objectives, determining methods (quantitative vs. qualitative), design selection, and assessment of evaluation itself. The task force is investigating potential speakers. Charles Friedman (Univ. of Pittsburgh), who gives workshops on evaluation design, is one possibility; members of the ACRL staff who can speak on the use of statistics are another. The task force hopes to create a workshop of interest to BLC member libraries as well as the staff of the BLC itself, but requests clarification that this approach is what the BLC has in mind.
      
      
5. Next Steps
   
   
   1. Once the above inquires are answered, the task force will focus on workshop planning.
   2. The task force is in the process of contacting the task forces on the Virtual Catalog and the 24x7 reference program, to coordinate our work.

Output Measures Task Force Charge
-The task force has clarified and redefined the charge; please see section II, below.

1. Meetings held (virtually and/or face-to-face)

   The BLC Output Measures Task Force has met five times: September 26 at Northeastern University, October 7 at UMass Dartmouth, and via conference call on Oct. 23, Oct. 30, and Nov. 5. The task force plans to meet chiefly by conference call, given our far-flung locations. There is no leader, as each member plays an equal role.

2. Activities/Accomplishments to date (list or summarize)

   At our Oct. 7 meeting with director liaison Ann Montgomery Smith, the task force clarified its charge. Our principal focus will be to investigate assessment tools and evaluation measures of use to BLC member libraries, with the hope that tools and measures to evaluate the consortium itself will emerge as part of that investigation. To that end:

   1. Task force members have familiarized themselves with ARL's New Measures Initiative, LibQUAL+, E-Metrics, student learning outcome, and other output measure issues; the list of resources below includes some of the articles and Web sites we've consulted.

   As per Action Item 3, Contact NEASC to learn about their plans for assessment and evaluation measures and/or tools as part of their current and future accrediting standards, Nancy Stafford attended a NELA presentation on NEASC. Her report follows as Appendix A.

   2. An output measures COI was established on October 21 after 13 staff of BLC libraries expressed interest, in addition to the four task force members. The task force plans to seed the COI with questions and issues to encourage listserv discussion and to engage COI participants.

   3. The task force has developed a survey on present and projected use of assessment tools among BLC libraries. The survey was distributed to BLC directors on November 5, with responses due November 12. The survey includes questions on interpretation and sharing of data, use of the results, and on what type of workshop BLC libraries would find most useful. A copy of the survey is attached as Appendix B.

   The task force will draw on survey results and on the Output Measures COI to determine the nature and focus of a workshop for the BLC on assessment tools and evaluation measures.

3. List of COI's with which you communicate
   
   Output Measures COI
   
   
4. Requests
   
   
   1. Funding

      Once plans are underway, the task force will submit a budget to cover speaker fees and expenses for a workshop on assessment tools to the BLC.

   2. Resources
      
      Web sites:
      
      ARL New Measures Initiative http://www.arl.org/stats/newmeas/newmeas.html
      
      LibQUAL+ http://www.arl.org/libqual/
      
      ARL Higher Education Outcomes (HEO) Research Review http://www.arl.org/stats/newmeas/outcomes/heo.html
      
      NEASC Commission on Institutions of Higher Education: Standards for Accreditation http://www.neasc.org/cihe/stancihe.htm
      
      University Libraries Group http://www.Lehigh.EDU/~inulg/
      
      International Coalition of Library Consortia http://www.library.yale.edu/consortia
      
      Articles:

      "ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC: Measurements of Research Libraries" [links to numerous articles, chiefly on LibQUAL+ and student outcome; two articles on E-Metrics] http://www.arl.org/newsltr/meas.html

      Blixrud, Julie. "Where's the Evidence? Discovering the Measures That Make Up a Library's Contribution to Student Learning Outcomes," delivered to the Iowa Library Association Annual Conference, October 10, 2002. http://www.arl.org/stats/newmeas/outcomes/
      ILAjblix_files/v3_document.htm

      Heath, Fred, Colleen Cook et al. Various articles on LibQual+ in portal: Libraries and the Academy, 2:1, January, 2002. http://muse.jhu.edu/journals/portal_libraries_and_the_academy/
      toc/pla2.1.html

      Kyrillidou, Martha, "From Input and Output Measures to Quality and Outcome Measures, or, from the User in the Life of the Library to the Library in the Life of the User." The Journal of Academic Librarianship, 28:1, pp.42-46.

      Kyrillidou, Martha and Fred M. Heath, eds. "Measuring Service Quality." Library Trends, Spring, 2001.

      Smith, Kenneth R. "New Roles and Responsibilities for the University Library: Advancing Student Learning Through Outcomes Assessment," paper prepared for the Association of Research Libraries, May 2000 http://www.arl.org/stats/newmeas/outcomes/HEOSmith.html

   3. Board/ MC queries

      The task forces requests the Board or Managing Committee to confirm that our understanding of the charge, as stated above, corresponds with theirs.

5. Next Steps

   The survey will be sent to BLC directors on Nov. 5 and will be due Nov. 12, with results to appear in the Jan. 31 interim report.

   The task force will initiate discussion on the output measures COI.

   The task force will contact consortia regarding consortial use of LibQUAL+ and other assessment tools.

   The task force will begin planning its workshop on assessment tools.

   Please note: Charts and graphs included in a document will be converted into Adobe format in order to be published to the BLC website.

Action Item 3: Contact NEASC to learn about their plans for assessment and evaluation measures and/or tools as part of their current and future accrediting standards.

The following information came from the NEASC/CIHE web site, email correspondence with Brown University Library Director and NEASC/CIHE Commissioner Merrily Taylor, and from a presentation given by Merrily Taylor and Dr. Robert Froh, Associate Director of CIHE, at the New England Library Association (NELA) Conference on Oct. 22, 2002.

Assessment should not be viewed as something that is imposed from the outside. The commitment to assessment should come from within and be integrated into institutional culture. According to Merrily Taylor, NEASC's position on assessment and the main thing to remember about the Commission's standards is that they are designed to help discern an institution's success at meeting the standards in light of its own mission and purpose. The standards are neither quantitative nor prescriptive.

The Commission wants to see outcomes and how data collection, surveys, and other tools are used to improve teaching, learning, and institutional decision-making. The Commission is not only interested in knowing that the institution is fulfilling its mission, it is also interested in knowing how it is doing it or plans to do it. The Commission would like for each institution to be asking itself, "Are we achieving what we've set out to do?" For libraries, it is not a question of "Are we a good library?" but being able to state "We're a good library for our institution and here's why." Two areas that are difficult to measure but of particular interest are how institutions can best assess students' critical thinking skills and how librarians can best assess information literacy skills.

To assist institutions in assessment processes from the perspective of accreditation, the Commission offers workshops, and has piloted the Institutional Assessment Portfolio project which was designed as an instrument to work across various kinds of institutions.

Additionally, NEASC has developed a web site that makes available the following resources: http://www.neasc.org/cihe/cihe.htm

NEASC standards (to be revised, possibly by Dec. 2002) are found at http://www.neasc.org/cihe/stancihe.htm
Standard 7 pertains to Library and Information Resources. The standards, especially number 7, are constantly revised to take into account new technologies and practices, distance education in particular. There has been discussion on whether information literacy should be considered as a course requirement. Additionally, it has been suggested that Standard 7 should emphasize use of resources as well as the existence of resources.

CIHE "Policy on Institutional Effectiveness"
http://www.neasc.org/cihe/instuteffect.htm

CIHE Initiatives: Institutional Assessment Portfolios Project Description http://www.neasc.org/cihe/assessment_description_2002.htm

For use during the self-study phase, CIHE has drafted "Cues" to assist Institutions in strengthening their assessment of student learning. See Standard 7: http://www.neasc.org/cihe/self-study_cues.PDF

American Association for Higher Education Assessment Forum http://www.aahe.org/assessment/

The Urban Universities Portfolio Project http://www.imir.iupui.edu/portfolio/

Sponsored by AAHEA, the Urban Universities Portfolio Project is a collaboration among six urban public universities to develop prototypes of electronic institutional portfolios:
BLC Member Institution participation:
University of Massachusetts Boston

The task force plans to monitor further NEASC developments and include any findings in future reports.

Increasingly, libraries are seeking to evaluate their services and resources in terms of output measures. Output measures focus on the effectiveness of library services, meeting user expectations, and student learning outcomes, among other issues. Used in addition to traditional input measures, such as collection size, expenditures, and staffing, output measures help a library answer the question, "do we make a difference?"

Libraries are using new tools to make these evaluations. To determine present and projected use of assessment tools and evaluation measures among BLC member libraries, the BLC Output Measures Task Force has developed a short survey on your use of such tools, as well as the analysis and use of their results. Your response to the survey will help the Task Force in planning a BLC workshop on output measures, to be presented to BLC member libraries next spring.

We welcome your participation. Please send your completed online survey to task force member Donna Leveillee by Tuesday, Nov. 12, Donna_Leveillee@brown.edu, and thank you for taking part in the survey.

1. What tools does your Library currently use to assess and evaluate output measures? (i.e., LibQual+, E-metrics, focus groups, surveys.)
2. If your Library does not currently use tools to assess and evaluate output measures, which ones would you consider using in the future?
3. Is there a group or team within your Library charged with the responsibility of interpreting the data? If so, please explain.
4. What is the background or experience of the individual(s) interpreting the data? What analysis and research methods do they use?
5. Do you share the results with the library staff, campus community, peer institutions? If so, how is the information made available to them?
6. In what ways have these data been used to inform management decisions? Please provide one or two brief examples.
7. What outreach strategies do you employ to insure a good response rate?
8. What kind of presentation regarding output measure tools would be helpful to you?
9. If you use LibQual+, E-metrics, etc., do you do further analysis of the data in-house? If so, do you use SPSS, SAS, STATA, or some other package? Please explain.