Reports

1. Meetings held:
   
   
2. November 17, 2003, February 26, 2004, March 5, 2004, March 15, 2004, March 25, 2004, March 31, 2004
   
   
3. Activities/Accomplishments to date:
   Searched literature to determine what other consortia are doing in terms of assessment, interviewed other consortium members about assessment, reviewed individual libraries' assessment web pages, surveyed Boston Library Consortium libraries regarding assessment activities and created a graphic spreadsheet to present results. Interviewed members of two large consortia about assessment activities.

   Executive Summary
   The Task Force members explored the meaning of assessment for a consortium in very practical terms. The main issues to address or questions to ask when performing assessment are:

   What is it that we do or offer and to whom?
   
   What is the cost of operations?
   
   What is the value?
   
   How do you measure the value? What tools are available to do so?
   
   Once the questions are asked and tools chosen and utilized, the next steps are imperative:
   
   Analysis of assessment findings
   
   Exploring changes to effect better service, streamlined operations, more cost-effective ways of working to provide services
   
   Implementation of changes that will improve the current situation

   In order to complete action items one and two Task Force members developed questions to ask the institutions and questions to ask library users to get the information sought in regard to value of services and costs of operations. These are listed below.

4. Action Items One and Two
   
   On Site Access and Card Borrowing Program Questions for the Institutions
   
   What is the cost in terms of staff time and materials for: Home institution borrowers:
   
   
   1. Interviewing people
   2. Time it takes to write up card
   3. Do circulation check
   4. Cost of applications, cards
   5. Publicity costs/time
   6. Cost of books, journals
   7. How many items did the institution users borrow from another BLC library and what did this borrowing save the home institution in purchases
   
   
   Visiting borrowers:
   
   
   1. Applications - paper, printing costs
   2. Staff time to create online user records
   3. Time to do quality control of user records
   4. Time/paper/emails for overdue or recall notices, bills
   5. Cost of books, journals
   6. Publicity costs/time
   
   
   Use statistics - no. of cards created, given out, number of circulations, bills - to get staff time spent.
   
   Questions for the Users
   
   
   1. Does the service facilitate research, meet research needs?
   2. How good is the quality of service?
   3. Is it worth a trip to another library?
   4. Ease of access to sites (physical and virtual)?
   5. What is the quality of collections?
   6. Are you looking for space for browsing/studying?
   
   
   Virtual Catalog/Interlibrary Loan
   
   Questions for institutions:
   
   
   1. How is BLC/VC different from traditional ILL?
      
      faster ___
      staffing ___
      other ___
      
      
   2. Which is more effective/efficient? Why?
      
      staff time ____
      cost ____
      delivery time ____
      equipment ____
      
      
   3. Have we / will we alter allocations of our collection development funds?
      
      
   4. Have we / will we alter our collection development profiles?
      
      Yes ___ No ___
      
      
   5. Are the items requested beyond our collection development scope or are they items we "missed" when we might have purchased them?
      
      
   6. How have library users reacted/responded to 'user initiated' services?
      
      
   7. How have the services been publicized?
      
      Library web page ___
      Campus web page ___
      Campus publication ___
      Library publication ___
      Instruction sessions ___
      Other (please list)
      
      What publicity has been effective?
      
      
   8. Do you allow your users to request items owned but not available?
      
      Yes ___ No ____
      
      
   9. Do you think the BLC libraries should coordinate collection development around each others strengths?
      
      Yes___ No ___
      
      
   10. Has the VC changed your collection development practices? If so, how?
       
       
   Questions for users:
   
   
   1. Do you know what these services are?
      
      VC ___
      ILL ___
      
      
   2. How often do you use each service?
      
      weekly ___
      monthly ___
      once a semester ___
      once a year ___
      
      
   3. How did you find out about the VC?
      
      library staff ____
      friend/colleague ____
      instruction ____
      signage ____
      
      
   4. What is your impression of the VC vs. traditional ILL?
      
      faster/slower ____
      easier to use ____
      more choices ____
      other? ____
      
      
   5. How could the VC be improved?
      
      faster delivery ___
      interface ___
      other ___ (please list)
      
      
   Electronic Resources:
   
   Questions for the institution:
   
   
   1. Do you negotiate your e-resource license agreements to include those users who are not directly affiliated with your user community? Please check all those groups included in your license agreements other than current faculty, staff and students.
      
      alumni ___
      visiting scholars ___
      summer workshop attendees ___
      BLC users ___
      general public ___
      other (please define) __________________________
      
      
   2. Do you allow users with a BLC card or with an ID from a BLC library to access your subscription-based electronic resources within your library? Check one.
      
      yes ___
      no ___
      
      If yes, how many workstations are generally available for these users?
      
      If no, please explain the qualifications used in making this decision.
      
      
   Users

   Task Force members do not currently see an effective way of polling users on their satisfaction with electronic resources provided through the BLC. Most users do not distinguish (or are unable to) whether or not the individual institution is providing the resource to them directly or through the BLC.

5. Action Items Three and Four

   Members of the Task Force reviewed the "Evaluation of the Virtual Catalog Project" carried out in 2002, the "Report of the Consortium Card Policies Task Force" and the Interim Report of the Resources Sharing Task Force as assessment projects recently carried out or in progress by the BLC. As mentioned above, members of the Task Force surveyed individual BLC libraries to see what each are doing in terms of assessment. The results of the consortium-wide survey of assessment activities in the form of a graph are attached: blc_assessment_2004.htm. Members of the Task Force do not believe that the assessment activities listed in the graph effectively meet the needs as identified in action items one and two. The majority of the activities are primarily individual institutional measurements and not a measure of consortium activity in terms of costs to the consortium as a whole or measures of services offered by the BLC as a whole. In light of this finding Task Force members make the following recommendations.

   We recommend that the BLC institutions participate in the LibQUAL+ survey individually on a regular agreed upon basis and then use results to compare and improve services and to develop ideas for more ways of sharing collection development. Some suggested readings about the OhioLINK participation (as a consortium) in LIBQUAL+ are listed below.

   Task Force members propose paper and/or online surveys as tools for measuring services such as on-site access, the Virtual Catalog/Direct Distance Borrowing and Interlibrary Loan. The survey may be as simple as 4-5 questions at the bottom of the application form for renewing Boston Library Consortium cards or a short online survey. The questions would be aimed at determining the value of the service to users at each of the BLC institutions and at getting at information that leads to improvement of services.

   To be most effective in carrying out regular ongoing assessment activities the Task Force recommends that a new Task Force or standing committee be created to oversee all assessment activities for the BLC or, alternatively, that each COI be assigned the task of developing assessment tools related to their areas. The COI's or task force or committee will:

   1. Look at the following and consider using or modifying and implementing as standards and/or assessment tools across the BLC:
      
      ACRL Standards
      
      ARL New Measures Initiatives, particularly:
      
      
      • E-Metrics Project
      • Assessing ILL/DD
      • Technical Services Cost Study
        
        
   2. Perform an extensive review of Resource Sharing within the Boston Library Consortium with an eye toward cooperative collection development
      
      
   3. Design and carry out surveys such as those suggested in Action Steps 3 and 4.
      
      
   4. Explore the idea of publishing consortium-wide assessment findings so as to establish benchmarks for other consortia.
      
      The link below leads to a model of steps to be a carried out in an assessment process that Task Force members recommend for use.
      
      http://www.lib.clemson.edu/aboutlib/plans/2003_2004Plan.pdf
      
      
   Sample Survey OF BLC Card Holders:
   
   We would like to know how you as a BLC card holder are using the facilities and resources of the BLC cooperating libraries. Please answer the following questions.
   
   
   1. How often do you visit a member BLC Library?
      
      weekly ____
      monthly ____
      semester ____
      
      
   2. Why do you choose a particular member BLC Library?
      
      location ____
      hours ____
      resources ____
      facilities ____
      
      
   3. Which resources do you use there?
      
      monographs (books) ____
      journals/magazines/newspapers ____
      online resources ____
      reference ____
      study space ___
      e-mail ___
      
      
   List of COI's with which you communicate
   
   All COI's
   
   Requests
   
   a. Funding
   b. Resources
   c. Board/ MC queries
   
   Reading Sources:
   
   Libraries with Assessment Pages:
   
   George A. Smathers Libraries, U. of Florida
   Library Planning and Performance Measurements
   http://web.uflib.ufl.edu/admin/toolbox/StratPlan.html
   
   Yale University Library Assessment Toolkit
   http://www.library.yale.edu/assessment/toolkit/
   
   U. of Washington Libraries Assessment
   http://www.lib.washington.edu/assessment/
   
   Assessment at the Raymond Walters College Library
   http://www.rwc.uc.edu/library/assessment/
   
   Southern Utah University
   How Does the Library at Southern Utah University Compare?
   http://www.suu.edu/general/ir/library.html
   
   Brown University
   Library User Needs Team (LUNT)
   http://www.brown.edu/Facilities/University_Library/MODEL/lunt/index.html
   
   University of Northern Colorado
   University Libraries Assessment Profile
   http://www.unco.edu/library/assessment/profile.htm
   
   Annotated Bibliography on Library Assessment
   http://www.carleton.edu/campus/library/staff/karen/Bibliography.htm
   
   See in particular the state-wide Interlibrary Loan Assessment article by Wiley, L. and Chrzastowski, T.E.
   
   Additional Reading When Available:
   
   Gatten, Jeffrey N. "Measuring Consortium Impact on User Perceptions: OhioLINK and LibQUAL+." [IN PRESS: Journal of Academic Librarianship (May 2004)]
   Gatten, Jeffrey N. "The OhioLINK LibQUAL+ 2002 Experience: A Consortium Looks at Service Quality." [IN PRESS: Journal of Library Administration]
   
   
   Attached Graph: blc_assessment_2004.htm.