 |
|
Reports
May 28, 2004 Final Report I
24x7 Evaluation Team Charge
Executive Summary:
(All appendixes are linked at the end of the report)
Since February, 2004, the Task Force has:
- Conducted an environmental scan to determine what virtual reference products are currently available.
- Surveyed participating BLC librarians to elicit qualitative information.
- Analyzed transcripts for quantitative and qualitative data.
- Participated in local sponsorship of the two-part teleconference Realities of Digital Reference, which was open to all BLC librarians.
- Met with many of the participating and non-participating BLC librarians at refresher training and the Reference COI at the BLC Annual Celebration.
- Communicated with 24/7 Reference (hereafter, the vendor), the local ASK 24/7 coordinators and the directors of BLC institutions.
As a result of our efforts, we recommend that BLC continue with 24/7 Reference for one additional year while assessing alternative software and service models. It will be important to monitor the results of the current negotiations between 24/7 Reference and OCLC Question Point. A merger between the two would have a significant impact on the virtual reference landscape.
Charge:
The 24/7 Reference Taskforce III is charged with evaluating the two-year virtual reference pilot project and making a recommendation regarding the continuation of this service, and, if continued, how to manage this service beyond the pilot. The previous task forces had selected software, outlined a staffing model, established and redefined policies, outlined a management structure for the project, and collected and evaluated data from the first half of the pilot.
Using the assessment methodology from the previous task force, we are further charged to:
- Analyze session transcripts, surveys from patrons using the service, and surveys of participating BLC librarians to assess both quality and quantity of service.
- Answer questions on satisfaction with service, value to students, and continuation of the program.
- Compare current assessment results to those of the previous evaluation.
- Make a recommendation to: continue service with current vendor, continue service and explore other vendors, or discontinue service. Provide information supporting this recommendation.
History:
Task Force I was formed in January 2002, and charged with evaluating the currently available options and recommending to the Board of Directors a virtual reference software package. They were also responsible for outlining a model of service for staffing and managing the collaborative project. The BLC Board of Directors accepted the Task Force recommendation to contract with 24/7 Reference for software and support.
The 24/7 Implementation Group was created on August 21, 2002 to start an online reference service for participating members of the Boston Library Consortium. Ten BLC members agreed to participate in the initial implementation of the service (Boston College, Boston University, Brandeis University, Northeastern University, Tufts University, UMass/Amherst, UMass/Boston, University of Connecticut, University of New Hampshire, and Williams College). Each participating library assigned one representative to the Implementation Group.
The Pilot Project began when service went live on November 12, 2002.
Taskforce II began meeting in February 2003. Their charge was to evaluate the first year of the pilot project and to make recommendations for future policies and procedures based on that evaluation. To that end, they:
- Devised a plan for the ongoing governance and management of the project.
- Evaluated the first year of service using the guidelines established by the first taskforce.
- Prepared documentation and guidelines for the ongoing evaluation of the project.
- Revised the policies and procedures prepared by the implementation group.
- Sponsored a program that allowed participating librarians to voice their questions and concerns about the project to date.
- Communicated with the vendor to improve the service.
Task Force III:
Activities:
- Meetings:
We held in-person meetings on February 20th (our initial organizational meeting at which we received our charge), April 22nd, and May 7th.
- Programs:
Feb 6th: Conducted in-service refresher training in conjunction with part one of the teleconference Realities of Digital Reference.
April 1st: Met with the BLC Reference COI and other interested individuals to solicit input after the BLC Annual Celebration.
April 16th: Part two of the teleconference Realities of Digital Reference.
- Summary of communication with the vendor:
In response to concerns expressed by the BLC's ASK 24/7 participants, we requested immediate implementation of the beta "overflow" service. As of May 10, 2004, BLC questions go directly to a BLC librarian when one is available. With the "overflow" program, BLC librarians are the only ones monitoring our queues, along with Metropolitan Cooperative Library System (hereafter MCLS) backup librarians. This system prevents staff from other libraries from picking up BLC patrons. If one BLC librarian is online, and not busy, she will get the call and no one else (other than the MCLS librarians) will be notified. If, while she is busy with the first patron, a second person comes in, then that person will wait in the BLC queue and no one else (other than the MCLS librarians) will be notified. If a 3rd person comes into the queue while the 2nd person is waiting, and the 2nd person has been waiting more than 30 seconds, then this 3rd person will go into the overflow queue, so that anyone staffing the academic queue can pick that patron up.
We have also been in ongoing communication with the vendor about their future plans for software and service enhancements and about other service models. The vendor has been highly responsive in fine-tuning the software to meet our needs. The California State Library is not investing additional funds to further enhance the software. The 24/7 Reference software, based on eGain, is the best software available at this time.
Evaluation:
- Environmental scan of software:
As part of the review process, we undertook an environmental scan of other virtual reference products, specifically targeting Docutek, LivePerson, and QuestionPoint. Our activities included a literature review, studying available product reviews drafted by clients, and gathering information from the vendors directly. We found that the software used by Docutek and QuestionPoint shares the same features as the 24/7 software (real-time chat, co-browsing, transcript saving/sending, statistics reports, etc) but also some of the same technical difficulties reported by the BLC users (issues with Macintosh computers, disconnections, sluggish co-browsing). The QuestionPoint software also offers streaming video and voice-over IP, enhancements which do not seem to be desired by the BLC users. A strength of the QuestionPoint service is its Knowledge Base, a database of completed Q&As, which have been and continue to be contributed by participating libraries. Some of the goals of the QuestionPoint software include support for multiple languages and translations, automated routing at the local level, interoperability among reference networks, and NISO support. LivePerson has been offering a simpler chat program with the features of page-pushing, transcript saving, and statistics-keeping as well as a self-service Knowledge Base (including an FAQ). The company has recently introduced a number of enhancements, including co-browsing, transcript-sending, and patron evaluation forms. We would like to monitor this particular software over the next year.
- Surveys:
A. Quantitative Activity Analysis
B. Librarian Survey Questionnaire and Comments
C. Responses to Hours Library Could Staff the Service
D. Publicity Survey
E. User Survey Data
F. Questions to Library Directors Regarding Participation
G. Survey of UMass LivePerson Users
Summaries of quantitative and qualitative survey data are below.
More detailed responses to the surveys, and charts of quantitative data, are available in the corresponding appendices.
A. Quantitative Activity Analysis: We performed quantitative analysis on the questions submitted to the BLC ASK 24/7 websites for the month of April 2004 (with comparative data from April and October 2003).
Results:
Quantitative Summary Questions Submitted to BLC ASK 24/7 Websites April 2004 (April and October 2003)
Note: Unless otherwise labeled, percentages are for April 2004. When multiple percentages are listed, the format is: April 2004 (October 2003; April 2003).
Use
- Users from participating BLC schools/libraries asked an average of:
17 questions a day in both Apr 04 and 03 23 questions a day in Oct 03
- Undergraduates submitted 66% (62%; 65%) of the questions.
- Students (undergraduate and graduate) asked 77% (79%; 77%) of the questions.
- Users from Brandeis University submitted the largest percentage of questions (19%) in Apr 04, but only 7% in Oct 03 and 13% in Apr 03.
- Users from Northeastern University submitted the largest percentage of questions in 2003 (27%; 29%), but only 14% in Apr 04.
- Users from UMass Boston asked 17% of the questions in Apr 04 and 20% in Oct 03, but only 8% in Apr 03.
Conclusion: Students, and undergraduate students in particular, gravitate to virtual reference. School use patterns change from semester to semester although we could not find a correspondence between promotion and traffic. Traffic is stable for the service and somewhat higher in fall than spring.
Day of the Week
- 64% (68%; 73%) of the questions were submitted Monday-Thursday. Thursday was the most active day (19%) in Apr 04 with Tuesday a close second (18%) and Monday next (16%). While Monday, Tuesday, and Wednesday received the heaviest traffic and accounted for 62% of the questions in Apr 03, Thursday (17%) surpassed Monday and Tuesday in Oct 03. Wednesday was the heaviest day (19%) in Oct 03 while Monday and Tuesday were tied for heaviest days (21% each) in Apr 03.
- 22% (21%; 16%) of the questions were asked on Friday and Saturday.
- 78% (79%; 84%) of the questions were asked between Sunday and Thursday.
Conclusion: Virtual Reference use is spreading throughout the week.
Time of Day
- 89% (91%; 87%) of the questions were asked between 10 am and midnight. 85% (87%; 82%) were asked between 10 am and 11 pm.
- 32% (48% in Oct 03) of the questions were asked during the hours BLC staffs the service (Monday-Friday 9 am-5 pm).
Conclusion: The present hours BLC libraries staff the service do not meet the needs of our users. However, if we could extend our hours of coverage to 11 pm, we would satisfy most (about 85% of) users. We should continue the 24/7 coverage to meet the needs of the other 15% of users. The addition of other BLC and regional academic libraries might enable us to staff longer hours that are in demand.
Source of Librarian who Answered Question
- A librarian from the user's home school/library received 1% (6) of the questions submitted to that school/library website in Apr 04 and less than 1% (3; 2) in 03.
- MCLS librarians (primarily public, and later academic, librarians employed by the vendor) answered 61% (55%; 58%) of the questions.
- The percentage of questions answered by BLC librarians fell 6% in the last year to 12% (13%; 19%). Questions answered by other academic librarians dipped slightly 25% (28%; 12%) although it was still up 13% from the initial measurement.
Conclusion: BLC librarians have not been answering the questions posed by BLC users. On May 6, 2004 the BLC 247 Evaluation Task Force contacted the vendor to ask that we be placed in the "overflow" program as soon as possible. We thought we were already in this program in which questions from BLC users would go directly to a BLC librarian if one was online. Apparently, the program was just out of beta testing when we asked. We were up and running on "overflow" on May 10, 2004.
Types of Questions
- 45% (41%; 44%) of the questions were Research/Reference, often extremely specific and well-formed.
- All of the questions were appropriate for an academic library reference desk. 84% (82%; 84%) of the questions were those typically asked at a Reference Desk or what we call "reference" questions (catalog, database, ready reference, and research/reference).
Conclusion: Users attempted to locate the information on their own before submitting their question. They have already checked search engines and many library resources as well. Users seek serious research assistance through BLC ASK 24/7 Reference. They expect this core service.
Types of Responses
- Co-browsing was used in only 8% (9%; 9%) of the transactions. 66% (65%; 60%) of the transactions were conducted exclusively with chat. Websites were pushed to the user in 26% (26%; 31%) of the transactions.
Conclusion: Co-browsing is not heavily used, but this may be a result of incompatibility of the software with some proxy servers and lack of confidence in the stability of the software.
Technical Problems
- A technical problem occurred in 41% (36%; 45%) of the transactions. Technical problems increased by 5% between Oct 03 and Apr 04, but have still fallen 4% in the last year.
- 47% (51%; 57%) of the transactions were incomplete (session is considered complete when user says thank you or goodbye). This represents an improvement of 10% in transactions completed in the past year.
- The user was dropped or disconnected in 37% (22%; 26%) of the transactions. The fact that disconnections have increased indicates there are still significant technical issues with the software or improper default settings in the browsers of our users. It is also possible that the librarians who are analyzing the transcripts are more proficient at recognizing disconnections than during the previous evaluation.
- Other technical problems occurred in 4% (14% in Oct 03) of transactions. These difficulties included: librarian never appeared or disappeared; push failed; transfer problem; co-browse not working; user could not see text.
Conclusion: Technical problems declined in some categories, but disconnections continue. The vendor is proactive in tweaking and improving the software.
Quality
- For 11% (13%; 25%) of all the questions submitted to the BLC Ask 24/7 service, MCLS librarians said they were too busy helping other users to respond to the question. Increases in staffing at the vendor led to a 14% increase in immediate responses to BLC users after hours.
- MCLS librarians responded to 61% (55%; 58%) of BLC user questions and they were too busy to assist our patron 11% (13%; 25%) of the time. They were too busy to answer 17% (23%; 43%) of the BLC questions they picked up. In one year MCLS librarians improved their response rate 26% even though they were answering a higher percentage of our questions than in previous evaluations.
- And the vendor added a team of academic librarians to respond to our questions and improve quality of service to our users. 16% (14% in Oct 03; not tracked in Apr 03) of questions received some type of post-chat follow-up. 12% (7% in Oct 03) received a follow-up email and 3% (5%; 4%) were transferred to another librarian or library.
Conclusion: Significant progress occurred between the evaluations to improve MCLS staffing levels. A new quality control program implemented early in 2004 sent incomplete transactions directly to the Library/School coordinator for prompt resolution. We are highly satisfied with the quality control measures in place with the vendor.
(See Appendix A below for detailed charts)
B. Librarian Survey Questionnaire and Comments: We surveyed the BLC librarians who are staffing the current pilot service with the following questions:
- Please tell us whether you are satisfied with the ASK 24/7 service. Explain.
- Should the Task Force recommend to the BLC Board of Directors that the ASK 24/7 service be continued beyond the initial two year pilot program? Why or why not?
We also included a grid requesting ranking on librarian preferred software features.
Results:
Summary of qualitative data:
We received thirty seven responses. The great majority of the responses were positive. Most librarians indicated that they believe that digital reference has become an integral and necessary reference service. A typical comment was: "I consider this an essential component of our service to our users and a great way to collaborate with colleagues across the country. We couldn't offer this much late night/weekend coverage any other way."
Another said: "… it is very difficult to withdraw a service from users once it has been introduced and accepted."
Many of those who were supportive had reservations. Several librarians reported "mixed feelings." One commented in part: "I like its immediacy, and I enjoy it when able to help someone effectively, ... I prefer dealing with student populations known to me (BLC) and am definitely more confused and less helpful dealing with those unknown to me because I need to learn more in the reference interview."
For the most part, concerns had to do with technical problems and staffing issues. Comments on the technology often took the form of suggestions that we look at simpler or easier to use products. There was widespread concern about technical problems causing disruption of transactions. A typical comment: "I find that the system has crashed or acted strangely so many times that one doesn't really have any trust that any given session will go without a serious technical problem."
With regard to staffing and user population, many librarians wished that the service could concentrate its efforts more on BLC or local libraries. Some had the sense that they were being stretched thin in serving so many libraries outside of the consortium or the geographic area. A representative comment was: "Now that there are so many libraries participating, it is extremely difficult to provide high quality service."(It should be noted that this was a departure from responses to a similar question last year when librarians were more likely to express dissatisfaction with service to our patrons by after hours librarians.) In response to this complaint, the Task Force has asked 24/7 Reference to consider breaking the academic queue, which we monitor as our obligation to the vendor in exchange for after-hours service, into smaller modules so that we would be monitoring fewer libraries outside of the consortium. They have indicated a willingness to have their steering committee look into the possibility.
By contrast with last year, several librarians commented on the improvement they have noticed in the quality of the after hours service and to a lesser extent in the technology. On librarian put it this way: "We have seen an enormous improvement in the service provided by the [vendor] librarians. [Vendor] librarians are now taking the time to carry out a thorough reference interview with patrons, and take them to database pages and research guides when it is appropriate… and the wait time for patrons is much shorter than it was last Spring. We have also seen a drop in the number of times that patrons disappear from sessions…"
Many librarians were unhappy that they seldom had the opportunity to serve BLC patrons during their shifts. The Task Force addressed this concern by asking 24/7 Reference to place BLC in the newly developed "overflow queue" discussed elsewhere in the report. Although it is too soon to report on the results of the change, it appears that it will ameliorate the problem.
Some comments indicated discomfort with the software and the chat environment in general, which may point to a need for additional training.
(See Appendix B below for a copy of the questionnaire and for a detailed list of librarian comments.)
C. Responses to Hours Library Could Staff the Service: We requested information from participating libraries about number of hours they could contribute and whether they could staff evenings and weekends.
Results:
Hours School/Library could Staff in a BLC/Regional Academic Virtual Reference Program
Six BLC participating libraries responded to a request for information on the maximum number of hours and evening/weekend hours they could staff the service per week. Three libraries were willing/able to expand into evening/weekend hours. Four libraries could cover more hours per week than the present commitment of four. This input indicates it is possible to expand the BLC hours of coverage if additional libraries joined the program.
(See Appendix C for the chart of responses.)
D. Publicity Survey: A publicity survey was sent out to the coordinators at participating BLC libraries with the following question:
How does your library publicize the BLC's ASK 24/7 service?
Results:
ASK 24/7 Reference Pilot Publicity Assessment Eight library coordinators responded to our survey of their publicity techniques in April 2004. Libraries used a variety of publicity techniques to inform users about the BLC ASK 24/7 Service. All responding libraries announce the service in their instructional sessions. A website spotlight, printed materials, and communication with faculty were also favorite publicity methods. The number of publicity techniques used did not relate to traffic. Undergraduate enrollments were more important factors. This indicates our publicity and webpage design was adequate, so students could find the service.
(See Appendix D for a chart of survey results.)
E. User Survey Data: We performed quantitative analysis on the BLC patron responses to a standardized user survey at the completion of a 24/7 Reference transaction.
Results: When a transaction is closed, the user is presented with a standardized user survey provided by 247ref.org. We summarized these for the months of April 2003, October 2003, March-December 2003, and April 2004. Some surveys were lost in April 2003, so the March-December 2003 summary is more accurate for 2003. The results for each snapshot were similar. We used the April 2004 (March-December 2003 in brackets) results as a basis for these comments because it had an acceptable 13% (a respectable 19%) response rate.
69% (61%) of users were satisfied
65% (59%) found the quality excellent
73% (69%) were very likely to use the service again
96% (92%) found the service very easy or easy to use
80% (81%) were college students
67% (80%) were first time users
76% (55%) heard about the service through the website
Although up to 20% had a negative reaction, users are generally satisfied with the service and respondents in April 2004 were significantly more positive and eager to use the service again than respondents in 2003. In April 2004 significantly more of the respondents were repeat users and found out about the service from the library's website.
(See Appendix E for survey results)
F. Questions to Library Directors Regarding Participation: In May, we sent the following questions to the BLC Directors:
- If the BLC Board decides to continue the 24/7 Reference program, the BLC 247 Evaluation Task Force would like to find out if your library, if it is not now participating in the program, might be interested in participating in the BLC Ask 24/7 Reference program. This is an interest poll and does not imply an obligation to participate.
- If the BLC Board decides to continue the 24/7 Reference program, the BLC 247 Evaluation Task Force would like to invite your comments in inviting non-BLC (affiliate), academic libraries in our region to participate in the BLC Ask 24/7 Reference program. The librarians who staff the service believe the high quality service would be maintained with the addition of other academic libraries in our region.
Results: BLC Director Survey Eleven BLC library directors responded to our inquiry on two points. The first question probed their interest in continuing in or joining ASK 24/7. Three of five current participants indicated their intention to continue with ASK 24/7 while the other two preferred a BLC-only service. Of the non-participating respondents, one would join if the hours of coverage by BLC librarians were extended. Two of the other four would continue with their present local chat reference service. On the question of whether we should invite non-BLC-academic/regional libraries to join ASK 24/7 to enable us to extend the hours of coverage by our librarians, six libraries supported asking other regional/BLC/academic libraries to join. Three libraries would consider it with further information, discussion, and clarification. One library doubted it would allow us to expand hours of coverage.
(There is no corresponding Appendix for section F.)
G. Survey of UMass LivePerson users: In May we queried libraries within the UMass system that use the LivePerson software on their satisfaction and experiences with that product:
- How is your virtual reference service is going now? Are the librarians satisfied? Are the users? Are you happy with LivePerson? Are you considering any changes, including hours, staffing model, or software?
- What version of Live Person are you presently using and what features it offers/ supports?
Results:Librarians at UMass Lowell and UMass Dartmouth find that LivePerson meets the needs of their librarians and users as their low desk traffic allows them to respond to chat questions from the Reference Desk during many of the hours it is staffed. The versions used by these libraries do not send a transcript to the user, support co-browsing or sending files, support transferring to another librarian, have an administrative tracking module, or interface with proxy servers-all features BLC librarians believe are important. However, we have since discovered (as discussed earlier) that the latest version of the LivePerson software includes a number of enhancements, including these aforementioned features.
(There is no corresponding Appendix for section G.)
- Recommendations:
We recommend that the BLC continue to offer virtual reference service. It is clear both from the steady increase in BLC questions and from the fact that 84% of the questions submitted in April 2004 were reference questions (catalog, database, ready reference and research), that this is a core reference service. There would be negative implications to withdrawing an established and expected service which has had increasing use and demand. (Appendix A: Quantitative Activity Analysis: Questions Submitted per Month listed below).
Because the past two years have been a pilot program, we should anticipate that some of the currently participating libraries may be withdrawing from the service in October 2004. We recommend allowing additional BLC libraries to join at that time, while keeping in mind that the current system is not Mac compatible.
Because we must have at least 10 libraries in order to maintain current levels of service without placing a strain on participating libraries, we recommend that enrollment be open to non-BLC academic libraries in our region. Additional libraries might enable us to extend our hours into the busier weeknight evenings to respond to BLC user demand patterns.
We recommend that the BLC:
- Continue service with the current vendor and software for a period of one year, while assessing alternative software.
- Monitor the proposed merger between OCLC QuestionPoint and our current service.
- Continue participating in the 24/7 academic cooperative in order to get the after-hours service our users require.
- Continue representation on the 24/7 Reference advisory board, as this gives us a very strong voice in changes and developments in the service and software.
Discussion
We looked at other options and dismissed them for the following reasons:
- BLC-only service: We surveyed the librarians who staff this service and library directors and discovered that most of them cannot expand their hours of coverage or staffing into the evening or at other high demand times beyond 9-5. Over 48% of the questions submitted by BLC users came in outside of the times when we were staffing the service (9-5 Monday through Friday). For the month of April 2004, only 32% of the BLC questions were submitted during the hours BLC staffs the service (9 am - 5pm Monday-Friday). Survey responses made it clear that BLC staff want a BLC-only (or regional) service, but that they also are unable to provide staff during the required hours. (Appendix B, part 3; appendix C)
- Changing vendors and software: Even though there have been a lot of technical problems, we found that other software packages which have all the features that our librarians feel are essential and desirable in a virtual reference software (see appendix B, part 2) are also plagued by technical problems. The vendor we use has been very responsive to our requests for software improvements, as can be seen in that 9% more transactions did not have technical problems in October 2003 when compared to April 2003 (Appendix A, section 1). There was also an increase of 6% in the number of transactions successfully completed between April 2003 and October 2003 (Appendix A, section 1), though the high number of lost transactions is still unacceptable. None of the comparable services have better results than this, which may be an argument for a simpler, more reliable software. For now, we recommend staying with our vendor both because they offer all the features our librarians want, and because we have found that the vendor is responsive and will likely work with OCLC to develop or create a new software that better fits the needs of libraries. We want to stay with this service and have a voice in this development, while also keeping an eye on the other services (i.e., LivePerson). If we were to switch to a different software at this point, negative repercussions could include the burden of software which local system support needs to maintain, as well as new and unfamiliar software for users and staff to learn. Docutek and LivePerson would both put a burden on system staff, whereas 24/7 does not because of vendor support.
Our current service's quality control has also shown significant improvement. Increased staffing, addition of academic librarians in after-hours service, and the immediate forwarding of BLC questions to the BLC coordinator for the patron's home institution for timely follow up when questions are not fully resolved are examples of this improvement. This is a very responsive vendor. Whenever they can make a change that we request, they do so, and quickly. The number of "too busy" responses for after-hours staffing has declined from 25% in April 2003 to 13 % in October 2003 to 11% in April 2004. (Appendix A, section 1)
The overflow queue system has been implemented and should further address some of the concerns regarding getting BLC patrons' questions answered by BLC librarians at the times when we are staffing the service. We contacted the coordinator of the statewide Maryland AskUsNow! service to ask what their experiences with the overflow queue system had been. His response was very positive, and we anticipate that it will also work well for the BLC's collaborative service.
The 24/7 Reference steering committee is also exploring the option of breaking up the academic queue into smaller groups, which would address our staff's frustration with having to answer questions for a large number of unfamiliar libraries (as well as the corresponding concern we have that staff at those libraries are unfamiliar with our institutions).
Appendices: (choose Section to open chart. All charts are in PDF format)
- Quantitative Activity Analysis
- Section 1: comparison data (April 03, October 03, April 04)
- Section 2: April 04 data
- Section 3: October 03 data
- Librarian Survey Questionnaire and Comments
- Section 1: Librarian Survey Questionnaire
- Section 2: Librarian Survey Questionnaire, chart of preferred software features
- Section 3: Librarian Survey Questionnaire, Representative Librarian Comments on the Questionnaire
- Responses to Hours Library Could Staff the Service
- Publicity Survey
- User Survey Data
|