Reports

1. Executive summary

   Since January 2003, the 24x7 Reference Task Force has completed the following major tasks: 1) refine, update and document BLC ASK 24/7 policies and procedures; 2) evaluate the Pilot Project from its inception in November 2002 through April 2003; and 3) develop a proposal for the ongoing governance of the program.

   After a review of the results of our quantitative and qualitative evaluations of the service to date, we recommend the BLC continue BLC ASK 24/7 while working with the current vendor, 24/7 Reference, to improve and enhance it. We recommend BLC conduct another evaluation of the progress and performance of the service at the end of the second year.

   We further recommend a governance structure comprising a Project Coordinator and a Management Committee.

2. Introduction

   Charge
   The 24 x 7 Reference Task Force II was charged with refining the polices and management structure of the consortium's virtual reference two-year pilot program. The previous 24/7 task force had selected a virtual reference software package and outlined a model of service for staffing and managing the collaborative project. A pilot project was launched in November of 2002. The task of the second Ask 24/7 Task Force has been to evaluate the pilot to date and to make recommendations for future policies and procedures based on our initial experience with the pilot. In response to this charge, this second 247 Task Force has done the following:

   • Revised the policies and procedures prepared by the implementation group in the fall to reflect current practice
   • Sponsored a program that allowed participating librarians to voice their questions and concerns about the project to date
   • Evaluated the service to date using the guidelines established by the first task force
   • Prepared documentation and guidelines for the ongoing evaluation of the project
   • Communicated with the vendor to improve the service
   • Devised a plan for the ongoing governance and management of the project
     
     
3. 24/7 Task Force Activities

   Meetings
   Since February, the Task Force has met frequently in conference calls or in person. Evaluation of the 24x7 service has been a major topic of those meetings, the minutes of which are available at www.blc.org/task_forces/FY03/24X7/24X7_minutes.html. Our discussions focused on the quality of the after-hours service, i.e., hours not monitored by BLC librarians (Monday-Friday, 9:00am-5:00pm), amount of use of the service by BLC patrons, technical problems encountered and their causes and solutions, and general satisfaction of patrons and librarians with the 24x7 service. In order to obtain an evaluative response from as many BLC participants as possible, the Task Force conducted a presentation on the pilot program to date, inviting all BLC librarians to attend.

   BLC Ask 24/7: Present and Future Program
   On May 22, 2003 24 librarians from BLC libraries attended the program BLC Ask 24/7: Present and Future at the Lamar Soutter Library at the University of Massachusetts Medical School in Worcester. Julie Jersyk presented a short summary of the project to date. Anne Moore gave a Powerpoint presentation of an analysis of UMass Amherst's transcripts. The lively discussion that followed raised many issues that later appeared in the evaluation surveys and were communicated to the vendor. The presentation was of benefit not only to the Task Force in the completion of its charge, but also to the participating BLC librarians not represented on the Task Force. The opportunity to discuss common problems and viewpoints proved valuable and cathartic.

4. Evaluation
   a. Process
   The charge to the Task Force included the directive to "implement the evaluation process recommended by the initial Task Force." The group's recommendations for evaluation of the service included analysis of session transcripts to assess both quality and quantity and analysis of surveys from patrons using the service as well as participating BLC librarians.

   The Task Force sought feedback from the ten institutions participating in the pilot in order to complete an effective evaluation of the project to date. Each of the ten institutions was asked to analyze transcripts from the month of April and record results in a spreadsheet created by the Task Force (see appendix B). They also were given three subjective questions to answer. The questions were designed to be as general as possible to give librarians the latitude to make as many suggestions and comments as possible. Librarians were asked to respond to the following questions:

   1. Please tell us whether your institution and its participating librarians are satisfied with the Ask 24/7 service. Explain.
   2. Do you consider the service valuable to your students? Why or why not?
   3. Should the Task Force recommend to the BLC Board of Directors that the Ask 24/7 service be continued beyond the initial two year program? Why or why not?

   Summary of Quantitative Evaluation Responses
   Participating BLC libraries were asked to review the transcripts for all inquires submitted through their website for the BLC Ask 24/7 service for the month of April 2003. They recorded specified information about each transcript in a spreadsheet (see Appendix, Spreadsheet for evaluating 24/7 transcripts), which was then submitted to the Task Force for compilation and analysis. The following details were aggregated and analyzed about the questions submitted: day of the week, hour, category of user, type of librarian responding, type of question, features used in the response, technical problems, and whether the transaction was completed. See Appendices C, D, and E for further information. Here are the highlights:

   • Users from participating BLC schools/libraries asked an average of 17 questions per day (500 questions over 30 days)
   • Undergraduates submitted 65% of questions
   • 84% of the inquiries occurred between Sunday and Thursday; 62% occurred Monday-Wednesday
   • 87% of the questions were asked between 10 am and midnight
   • 44% of the questions were Research/Reference, often clearly stated
   • MCLS librarians (after hours librarians working for the vendor) answered 58% of the questions. A librarian at the user's home library answered less than 1% submitted at the Website
   • MCLS librarians were too busy to answer 43% of the BLC questions they picked up
   • A technical problem occurred during 45% of the transactions
   • 57% of the transactions were incomplete

   This quantitative assessment indicated that undergraduates use the service heavily and early in the week between the hours of 10 am and midnight. Overall traffic is higher than anticipated. Users who submit questions to the virtual reference service have already attempted to find resources on their own and have well-formed research questions. The quality of responses from BLC and other academic librarians is consistently outstanding. Academic librarians engage in a full reference interview and teach users skills for conducting research on their own next time even in the virtual environment. Unfortunately, most questions are answered by MCLS rather than BLC librarians even during the hours BLC librarians are staffing the service. MCLS librarians keep our users waiting for extended periods. When they do pick up the call, they frequently say they are too busy to answer and ask if the user could wait for an email response. The quality of the responses by MCLS librarians was poor.

   Technical problems and quality control issues with responses from MCLC librarians were discussed with the vendor in June 2003.
   
   Please see: charts (Adobe Acrobat format)
   
   Summary of Qualitative Evaluation Responses

   The Task Force sought information from the ten institutions participating in the pilot in order to complete an effective qualitative evaluation of the project to date. Each institution was given an evaluation form with three questions designed to be as general as possible in order to allow latitude for suggestions and comments. Following are the three questions and summarized responses.

   1. Please tell us whether your institution and its participating librarians are satisfied with the Ask 24/7 service.
      
      Positive responses:
      A majority are pleased that their patrons can get research help around the clock.
      Many approved of the collaborative aspect of the service, which they otherwise would not be able to offer.
      Respondents feel the BLC librarians are providing excellent service to their patrons.
      
      Negative responses:
      There are unresolved technical issues, chiefly sessions crashing and patrons disappearing or being disconnected. The co-browsing feature often doesn't function.
      MCLS librarians often pick up BLC patrons faster than BLC librarians can.
      Lack of customers is frustrating to the librarians; without sufficient opportunity to practice using the software it is difficult to maintain a competent skill level.
      The 24/7 software is compatible with EZProxy, but since only four of the libraries use EZProxy, librarians often can't co-browse databases with patrons, thus limiting the teaching function of the software.
      The greatest concern, expressed by all the respondents, is the quality of the after-hours service provided by the MCLS librarians. These librarians rely heavily on web search engines to answer research questions that require different sources, tending to provide the patron with quick answers instead of instructing them on the use of online resources. At peak times the amount of time patrons spend waiting to receive a response to a question sometimes as much as twenty-five minutes is unacceptable. Instead of referring patrons back to their home institutions, MCLS librarians sometimes suggest that patrons use their own e-mail service.
      
      
   2. Do you consider the service valuable to your students? Why or why not?

      The majority feel that some form of collaborative virtual reference service is beneficial to their students. However, the majority opinion is that the service's value is limited by the concerns mentioned above.

   3. Should the Task Force recommend to the BLC Board of Directors that the Ask 24/7 service be continued beyond the initial two year pilot program?

   While nearly all the participating institutions feel that some form of collaborative virtual reference service should continue beyond the two year pilot, the participants want to see improvement in the areas outlined above in order for the project to remain viable. Suggestions for improving the service include:

   • Hire a Project Coordinator who will serve as a consistent contact person for the 24/7 vendor. This will stabilize relations between the BLC and the vendor. Many suggested that a part-time Project Coordinator could improve services by developing a marketing plan, providing staff with continuous training, and evaluating the service on a regular basis.
   • Request that MCLS librarians who pick up our patrons during our coverage period transfer that patron back to us.
   • Request that ref247 hire additional academic librarians to staff the MCLS service.
   • Extend the hours of coverage provided by BLC librarians into week nights.
   • Suggest to the 24/7 administrators that the MCLS librarians be better trained to handle academic research questions.
   • Find a solution to the proxy issue.
   • Switch to software that is simpler to use and more stable.
   • Develop a coverage schedule that takes into account the size of the staff in each library.
   • Develop a more comprehensive policy manual.

   Summary of Communication with the Vendor (June 2003)
   Members of the Task Force conveyed the concerns of the BLC Ask 24/7 reference librarians to the vendor by means of email, phone and in-person meetings. The Task Force was impressed with the enthusiasm and commitment of the vendor to resolve the technical and quality issues we raised. The vendor will visit and meet with the librarians to offer further training, introduce new features, and report back on the improvements made in response to our concerns.

   A new software release implemented in June 2003 resolved most of the technical issues, including spontaneous user and librarian disconnections. Additional testing continues. Plans for addressing quality control issues include:
   • MCLS librarians will receive additional training regarding academic library reference interviews, philosophy, and databases.
   • MCLS librarians will be reminded that they have access to WorldCat and Gale research databases.
   • MCLS librarians will assume users with academic research questions have already visited Internet search engines and will move quickly to academic databases.
   • MCLS librarians will wait 20 dings before picking up users when academic librarians are staffing the queue.
   • MCLS will hire additional academic librarians.
   • An academic librarian will be on duty for MCLS until 1 am California time.
   • MCLS public librarians will transfer BLC questions to a BLC librarian when we are on duty, will transfer to the MCLS academic librarian on duty after BLC hours, and will refer to the user's home library when no academic librarian is available.
   • MCLS academic librarians will be clearly identifiable while online.
   • 247ref staff will provide training and meet with BLC librarians in August 2003.
   • BLC librarians will forward problematic transcripts to 247ref staff immediately upon detection of a quality issue.
5. Recommendations
   
   a. Future directions

   The recommendation of the Task Force is to continue the service with this vendor and work to develop and enhance it. We recommend the service be reviewed on an annual basis.

   Virtual reference is here to stay. The importance of Virtual Reference Service is supported by the effort ALA is making to develop virtual reference guidelines and the increasing use of this software application in the commercial sector. Libraries are under intense pressure to provide our services digitally at the convenience of the user. Our patrons are increasingly turning to electronic means to send and receive information.

   b. Governance

   The Task Force recognizes the success of this service is contingent on a commitment to adequate support as outlined in the following proposed governance structure.

   Project Coordinator: A half-time, paid position, reporting to the BLC Executive Director, to coordinate and oversee all activities of the Ask 24/7 service. See Position Description below.

   Management Committee: A standing committee made up of one librarian representing each participating institution.

   Management Committee members are responsible for working with the Project Coordinator as outlined below and also for overseeing the operation of the service at their home libraries. Responsibilities include in-house training and scheduling of individual librarians to monitor the service and mediating communications between the Vendor, the Project Coordinator and the librarians.

   Together, the Project Coordinator and Management Committee are responsible for the policies and procedures for the Ask 24/7 service and revising them as necessary. This team is also responsible for ongoing assessment and planning for the future of the service.

   We recommend that the Project Coordinator and the Management Committee hold regular, formal meetings.

Summary: The Project Coordinator serves as the primary liaison with the Vendor (24/7 Reference) and with the Boston Library Consortium Board of Directors. In collaboration with the Management Committee, develops policies and procedures, designs and carries out ongoing training, and arranges associated programming as appropriate. Carries out day to day oversight and administration of the service. Works scheduled hours staffing the service along with other reference librarians.

Liaison with the vendor: Mediates all communication between the Vendor and the BLC participating libraries. Relays announcements and queries from the Vendor and coordinates a group response when necessary. Relays information and inquiries from the BLC libraries to the Vendor. Arranges for initiating and terminating librarian accounts with the Vendor. Routes and tracks problem reports.

Liaison with BLC Board of Directors: Keeps the Board advised of the activities of the Ask 24/7 service. Prepares and submits regular reports and provides other information as requested.

Liaison with the Management Committee: The Management Committee is composed of one member from each participating BLC library. The Project Coordinator schedules and leads regular meetings of the Management Committee. Together, the Management Committee and the Project Coordinator develop, and revise as needed, policies and procedures for the Ask 24/7 service; develop and schedule ongoing training for participating librarians; provide other programming as appropriate. Develops marketing and publicity.

Administration: Prepares and coordinates service schedules for academic terms, summer sessions and holidays; prepares and distributes monthly statistical reports; relays, tracks and responds to queries and problem reports between participating libraries and vendor. Prepares and submits content for the Web page (maintained at Brandeis). Periodically reviews and revises library policy pages on the Vendor Web site. Works scheduled hours monitoring the Ask 24/7 Service.

Since submission of the last report, the Task Force has held three meetings, all by means of conference calls. A face-to-face meeting is planned for April 25.

The 24/7 Implementation Group, whose work is complete, has been disbanded, and the online discussion list dedicated to that group has now been converted to a users' discussion list, with all BLC Ask 24/7 participating librarians invited to join.

The Task Force decided to rotate the duties of Project Coordinator in light of a lack of resources to hire a permanent coordinator. Julie Jersyk has been appointed coordinator of the Pilot Project for the period of March 1 to August 31

Because there may be additional libraries interested in participating, the Task Force has decided to try to submit its final report by June 2. This would enable new libraries to join without undue delay. Tasks to be completed before integrating new institutions include evaluation of the pilot project and development of a set of policies.

• Developed an outline for a BLC Ask 24/7 Policies Manual, which will be made available in print and online. That work is ongoing.
  
  
• Held extensive discussions regarding ways and means of evaluating the Pilot Project quantitatively and qualitatively. As a result, the following steps have been taken:
  • Drafted an outline of information/data to be collected from 24/7 transcripts for the purpose of evaluating the Pilot Project.
  • Developed a transaction summary sheet to be filled out by participating librarians. Information compiled from the sheets will be used in the evaluation process.
  • Discussions of means of qualitative evaluation are ongoing.
    
    
• Addressed technical problems that have been experienced by participants.
  
  
• Begun planning a program to be offered for the participating institutions and for the BLC community at large. Purposes of the program will be to acquaint the BLC community with the progress of the Pilot Project, provide information to institutions that may be contemplating joining, and provide a discussion forum for participating librarians. It is anticipated that the Task Force will benefit from information gained.
  
  
• Begun reviewing all existing polices and procedures for updating and revision.

The Task Force met as a group for the first time on January 14, 2003 to outline tasks to be completed and issues to be addressed by July 14, 2003. These tasks are all specific goals that will enable The Task Force to carry out the broader charges listed above. Below is a summary of the Task Force's tasks for the next six months:

• The Task Force will decide on how the project will continue to be managed. The possibility was suggested that the responsibility for managing the project can be rotated among interested BLC librarians. The group also discussed the option of having the management tasks be divided up among more than one person. The Task Force understands that this is an issue that needs to be decided on as soon as possible as a project manager is currently needed.
  
  
• The Task Force will propose a policy for dealing with proprietary resources among the participating libraries.
  
  
• The Task Force will propose a plan for expanding the project (bringing in more BLC members).
  
  
• The Task Force will evaluate the first phase of the project to date by implementing the evaluation process recommended by the initial task force. The group will then propose a continuing evaluation process for the ongoing management of the project.
  
  
• The Task Force will determine how transcripts will be managed, and this includes addressing the following questions:
  
  -Who has access to the transcripts?
  
  - How will the transcripts be evaluated and analyzed, and who will evaluate them?
  
  - Is there a standard for "good" session transcript?
  
  
• The Task Force will outline an ongoing publicity plan for the project.
  
  
• The Task Force determined that how it will operate as a group. The group will rotate the responsibility of convening each meeting among its five members. There will be an assigned minute keeper for each meeting, and the minute keeper will be the convener for the next meeting. The convener will also be responsible for drafting an agenda for the appropriate meeting. Meetings will take place primarily through conference calls, the "meeting function" of the 24/7 software, or in person.