Minutes

The first item on the agenda was a continuation of the discussion of the evaluation of the ASK 24/7 pilot program. All present expressed concern about the quality of the responses to our patrons from MCLS librarians. The late evening transcripts often indicate that the librarians are too busy to respond or, when they do respond, they fail to take an instructional approach to queries. Anne has analyzed the transcripts for UMass Amherst and found that in 20% of cases, MCLS librarians said they were too busy to answer the question immediately. It seems to us that 24/7 Reference needs to hire more after-hours librarians and needs to hire more librarians who take the academic approach. We were, on the other hand, very pleased with the responses from academic librarians in BLC and elsewhere.

Debbie suggested that it would be useful if the vendor could make available a passworded report that would allow us to access a list of all transcripts generated by librarians at a given institution. As it is, we have to access each librarian's transcripts separately.

Anne distributed a sheet she uses to analyze transcripts. After some discussion it was decided to adopt the sheet as part of the evaluation packet we will send to participating libraries. We decided to ask participating institutions to analyze all transcripts for the month of April using Anne's sheet, and also to answer some general questions about the pilot program, which Julie will draft. We will plan to have the packet ready for distribution on May 22 and will ask to have it returned by June 9

There was discussion of the interpretation of the resolution codes. There seems to be no consistency in what different librarians consider a completed transaction. It was decided to ask the implementation group about dropping "completed" from the monthly statistics distributed by the project manager at least for the time being, until a firm definition of "completed" is agreed upon. Anne said that at UMass Amherst a session is considered complete if the patron says thank you or goodbye.

Next on the agenda was the topic of content for the program to be presented on May 22. As decided at the previous meeting, the program will have four components: a demo of the service, a presentation on the history of the pilot program to date, a recap of task force activities to date, and a discussion period. It was decided to start with the demo, with Kate as librarian and Jim as patron, followed by the 24/7 history, by Anne, using the UMass Amherst experience as the example, followed by a recap of task force activity, by Julie, and than leave an ample amount of time for discussion, as it is expected that attendees will have many questions and comments.

Anne offered to bring her laptop, which already has its settings calibrated for the 24/7 service. We all agreed to arrive an hour early on the 22nd to make whatever on-the-spot preparations need to be made.

We approved the minutes from the February meeting and Julie will send them to the user's group and Melissa Petze.

We jumped around the topics a bit, so these notes are organized by topic rather than actual order of occurrence.

Privacy:
Since 24/7 has not yet deleted any of the transcripts for BLC, we discussed privacy implications. The previous task force agreed to have transcripts deleted after two weeks. We feel that is not sufficient time to allow us to review the transcripts. We need a minimum of two weeks after the end of the month, but preferably a month to 6 weeks. Some of us have touted the service as setting the standard for not retaining personal information. But we still need to keep sufficient information long enough for us to analyze the transcripts and ensure quality service. We believe that the name, phone, zip, and email address would be stripped. We are not sure about the IP address. Julie will clarify with 24/7.

Evaluation/Transcript Analysis:
We were all extremely satisfied with the quality of the responses from BLC librarians. Kate had an experience in which an MCLS librarian asked for a username and password/pin from a Boston College faculty member and that information appeared in the transcript. She contacted Susan Barb at Quality Control. In reviewing the transcripts from MCLS librarians who responded to our patrons during the evening and weekend hours, we made the following observations about non-BLC librarian performance:

• Frequent use of the we're too busy, hang on, leave a message, or try back later message
• Exclusive use of the Internet to respond to research questions rather than taking the user to the appropriate list of databases and locating an appropriate source
• Confusion of basic services: for example an MCLS librarian did not know what WorldCat was and directed the patron to the Virtual Catalog instead
• Frustration by users at the lack of basic knowledge of academic library services by the MCLS librarians
• Heavy use of referrals to the home campus rather than attempting to answer the inquiry

Boston College has been following Bernie Sloan's method of analyzing transcripts:

• Number of sessions by hour
• Number of patrons by type
• Librarian institution
• Referring URL
• Technical problems such as slow computer, co-browsing not working, user disconnected
• Type of inquiry

We recommended we add the following to this list:

• Number of sessions completed successfully (parties actually say thank you or goodbye)
• Track browser type (so we can analyze technical problems)

Kate said we should decide what we expect from MCLS librarians in responding to our patron's inquiries:

• We want them to do more than just say call your local library on the next workday.
• We don't want them to say they are too busy.
• We don't want our users turned off from virtual reference by being kept waiting or receiving mis-information.

We discussed having all 10 participating BLC libraries analyze the MCLS librarian transcripts for one month and rate them as satisfactory or unsatisfactory and provide a justification.

Kate was assigned to make forms for individual librarians to use to evaluate each question after it occurs. These are to be used at all 10 libraries. (When are we to begin doing this?)

We will then report recurring problems to 24/7.

Listservs:
There is still confusion about which listserv is which and getting all those who do virtual reference subscribed to the user's group listserv. We asked Julie to follow up with Melissa to try to change the name of the user's group listserv. The name is: 247users-l@blc.umb.edu.

Scheduling:
The schedule is complete through Memorial Day. We would like to rotate shifts in the summer as the traffic during some shifts is lighter than during others. Kate asked that the shifts not be assigned on a first-come, first-serve basis next time and that schools not be penalized for responding on the deadline. Boston College should have first choice this time since they were assigned the only remaining shift last time.

Ongoing Training:
We need refresher sessions and an opportunity to share tips. Since presenting a program once a year is in our charge, we will plan a program/mini-conference in the summer. Julie will clarify with Barbara the requirements for our program. We are considering a forum on general concerns, evaluation of the program, and continuing education/sharing session for participants.

Policy Manual:
We strongly believe we need a written policy manual librarians can refer to when they staff the service. Jim will come up with an outline, which we will add to. Julie will email or print out the online policies for our libraries and send them to Jim. We described frustration with the 24/7 interface to the online policies since the interface is inflexible and we must email Carol Bonnefil to have a change made.

Next Report:
We want to get our next report done by June 2, so it will be received by the Board of Directors in time for their June 27 meeting. We will discuss the outline for the report during the next conference call.

Anne will write up the minutes, arrange the conference call, and be the convener for the March 24, 9 am meeting.

Kate called the meeting to order and asked for a scribe and next meeting convener. Julie volunteered.

Barbara announced that Julie has agreed to take the first rotation as Project Manager, with a term running from March 1 to August 31, 2003. She reiterated that there is not sufficient staff at the BLC to take on projects such as this and that it is something the directors will need to consider as the consortium takes on more projects. Debbie suggested that the institutions contribute money, to which Barbara replied that there has been mixed response to that idea from the five directors on the Management Council.

Barbara remarked that since the implementation phase of the project is now complete, that group can be disbanded. It was decided to convert the discussion list for that group to a wider Community of Interest or Users Group for the service and to invite all participating librarians to join. The lists are:

247-l@blc.umb.edu for the users group, and 24x7-l@blc.umb.edu for the task force.

Barbara said she would send an announcement to the list.

Project Expansion

Barbara said that informal interest has been expressed by three institutions. UMass Worcester is interested and possibly Brown and Wellesley. BPL is participating in the Mass Answers program. There is still the outstanding issue that the 24/7 and other (LSSI) systems do not "talk" to each other, which would be a detriment to partnering with institutions, such as MIT, who are using other products.

It was thought that we should develop policies and evaluate the project in its current makeup before integrating other institutions. While our final report is due on July 14, we could move it up to an earlier date, so that we could invite additional institutions to join by July 1. There was discussion of whether we could complete the work by May so that the report could be submitted in advance of the Management Council meeting on May 20. Issues to be resolved include handling and analysis of data, budget, the question of whether to add another seat, scheduling of shifts, analysis of transcripts. Related to transcripts are privacy issues such as the stripping of personal data. It was decided that we would divide the transcripts once the personal information is stripped.

An evaluation process was outlined by the first 24/7 task force. The current group could revisit it with the idea of amending it. The report on the BLC web site includes the evaluation method.

Barbara cited a relevant article:

Sawhill, John C. and David Williamson. "Mission Impossible? Measuring Success in Nonprofit Organizations." Nonprofit Management and Leadership. 11(3) 2001.

Julie agreed to make initial contact with the Output Measures task force for input. Jim offered to help.

Microsoft Virtual Machine

Anne said that the 24/7 service is not compatible with anything that is not Microsoft. This means that many patrons at UMass Amherst cannot use the service. She believes this ought to be taken up with the vendor. The group wondered if other groups have experienced this and whether we should contact them. The users group which is being set up may be a contact point. Anne will send more information by email.

Policy Manual

The group thought that we need more than what is included at the Web site. It would be helpful to have a hard copy manual. Suggestions for inclusions included:

• Key Web sites
• Telephone list
• List of subject specialists
• Circulation policies
• Database list

The next step is to look at the transcripts. Jim suggested that we look at them in a uniform manner, having a list of things we are looking for:

• Types of questions
• How the after-hours librarians are doing
• Do we have to go into databases with patrons? And, if so, are we successful?
• Co-browsing success rate.
• General statistics

Some results may depend on the equipment patrons have. It will be important to look at patterns. Anne suggested we each look at 20 transcripts and then decide what we need to look at, then set standards.

The next meeting will be on Monday, March 3 at 9 a.m., once again by conference call.