501 FINAL REPORT UM Harlan Hatcher Graduate Library Instructional Services 13 December 2007
Written for: Darlene Nichols Coordinator of Instruction Harlan Hatcher Graduate Library dpn@umich.edu Written By: Andrea Mcvittie Ning Wang Julia Proctor Sergio Mendez Chris Webb
Key Contact: Julia Proctor pjulia@umich.edu
Table of Contents Executive Summary …………………………………………. pg 3 Introduction …………………………………………. pg 5 Methodology …………………………………………. pg 5 Data Collection Data Analysis Key Findings …………………………………………. pg 6 Recommendations …………………………………………. pg 7 Bulletin Board Master Calendar Mandate Public Folder Solution Integration of current IT Systems Social Tagging Recommender / collaborative filtering Conclusions …………………………………………. pg 11 Appendices…………………………………………. pg 12 Appendix A Intranet Screenshots RSS Popper Del.icio.us Appendix B Research Data Publications Summary of Findings Appendix C Consolidated Communication Model Cost/Time Implementation Graph Additional Recommendations Endnotes …………………………………………. pg 26
2
Executive Summary We were tasked with assessing the research activities through which the Hatcher Graduate Library Instructional librarians obtain information regarding best practices and pedagogical trends in preparation for instructional workshops. The Library provides workshops on a wide variety of subjects and research tools. The organizational structure in the Hatcher library is very flat, and each instructor is a specialist in his or her respective areas. The Coordinator of instruction was interested in “streamlining and economizing� preparation activity through possible centralization of resources and in facilitating the sharing of information obtained by the various instructors. Data was collected through the conduct of 8 interviews with key personnel. We also obtained artifacts from various instructors who were interviewed during the process. Using the data generated from those interviews, the group was able to construct models to interpret the data. Modeling was the key to our findings, as the focus of our project was on communication and collaboration. Several models were constructed, as was an affinity diagram. It was observed that various information systems exist within which this communication occurs. We examined the dynamics that exist between both groups and individuals. We also noted a large variety of communication systems exist within the organization (i.e. Exchange server, Intranet, Instructor College), and that there is no apparent protocol for the coordination of those systems. Our project focused on identifying solutions to these two problems.
Summary of Findings Communicative Environment The Hatcher Library Instruction environment is highly supportive of communication among staff. An Interest in Pedagogy is Present Staff members are interested in learning about new pedagogical methods and trends. Sharing Occurs but Could Increase Sharing of pedagogical research is occurring. However, this sharing is mostly limited to the smaller networks. Lack of Time is a Major Factor Lack of time was an issue for everyone. This lack of time led to prioritizing and limited the amount of time allocated for pedagogical research. Existing IT Systems are Not Facilitating Sharing Efficiently There are many different IT systems in use throughout the Hatcher, systems were either not facilitating sharing, or not doing so efficiently.
Summary of Recommendations Our recommendations to the client include a series of possible solutions listed as a function of cost versus time to implementation (Appendix C): Every recommendation addresses each of our findings by attempting to maximize the aspects of research that are working well and altering those that could be improved.
3
• • • • • • •
Bulletin Board (Low Cost/ Short Term) - Hung in one of the central spaces for instructors to post articles or ideas that they have come across in their research Master Calendar (Low Cost/Intermediate Term) - A master calendar listing the workshops, locations, and instructors so others could inquire about techniques used, or observe. Mandate (High Cost/Short Term) – Create a requirement for research & sharing protocol. Integration of current IT Systems (High Cost/Long Term) – Condense & refine systems Social Tagging (High Cost/Intermediate Term) - A social bookmarking system to flag and locate research of interest. Custom Recommender & Collaborative Filtering (High Cost/Long Term) - Scales independently and produces real-time recommendations based on massive data sets. Public Folder on the Exchange Server (Low Cost/Short Term) – A public folder integrated into the existing email system which would be available to all staff. This could serve as a repository for gathered information, including staff postings, RSS feeds, and other data sources.
4
Introduction The Hatcher Graduate Library offers workshops and seminars on a variety of subjects to the University of Michigan community. Among its units are; the Faculty Exploratory (providing individualized assistance to faculty and graduate students), and The Knowledge Navigation Center (providing individualized assistance and workshops to faculty, students and staff regarding multimedia tools and software). The subject specialists work with researchers in small groups and individually. “They can support instructors' efforts to develop and enhance the library research and information literacy skills of their students.” (* Client Provided Information) The organizational structure of the Instructional Services unit is very flat. The instructors work independently and without direct supervision. This reflects the organizational culture and our group was specifically instructed not to interfere with the way instructors conduct their research. The current coordinator inherited this culture, and refers to the instructors as "the lone ranger librarians." As a result of this flat structure, there exist a diverse set of habits and methods to conduct their pedagogical research. Some of the instructors use traditional resources and tactics in preparation for the workshops they conduct. Frequently they use highly specialized and targeted searches, such as literature reviews, specialized databases, or attending conferences and exchanging information through formal channels. Other researchers prefer informal information and ideas exchange methods, such as blogs, social networking web sites, or simply asking their colleague seated next to them, or calling another. Still other instructors report that they don't research at all, because they have established a system through teaching the same workshop over and over again, year after year. There is no standard process for conducting research, and there is no organizational standard, protocol, or mandate currently to place govern the conduct of research activity. The instructors are highly competent, and capable to conduct their own research independently, but they have encountered significant difficulty in efforts to access and benefit from the research conducted by their colleagues. Many instructors do share information with colleagues; however, the sharing occurs on a limited basis. Often times instructors only share with colleagues that they are certain will be interested. Some instructors freely share the information they obtain, while others do not. Some instructors don't share information because they don't have time or because there is not an effective or efficient way to share with the group at large. The result is a fragmented and diverse series of collaborative subsets wherein those whom are familiar with each other’s interests and research activity are collaborating freely, while others remain isolated from this store of information. Overall there is an interest in improving pedagogical practices and in sharing each other’s research findings. The Coordinator was interested in knowing more about approaches currently in use, and converting that tacit knowledge into a centralized store of organizational knowledge rather than creating a standardized process. To accomplish this, she hopes to facilitate the sharing of information obtained by the various instructors. The group was tasked with engaging this process of discovery, and to analyze the content, mode, and methods of communication currently in use by the staff as they design and implement instructional modules.
Methodology Data Collection and Analysis Our data was largely collected through the conduct of interviews with personnel of the Graduate Library Instruction team. They were asked questions regarding how often they conducted research in preparation for workshops, where they go to collect information, if there was any shared space available online for the
5
staff to extract or deposit resources, and if they use each other as an informational resource. We also gathered information regarding the culture of their workplace, for example, how often they communicate with each other both formally and informally, if it was an open work environment, and if collaboration occurs frequently. We also gathered specific information regarding the workshops; what types of workshops each individual teaches, how often they teach them, if they evaluate them, if they report to anyone regarding the workshops, if they teach workshops with colleagues, and if their preparation methods vary depending on which workshop they are teaching. We collected various artifacts in the conduct of our inquiry. These artifacts included screenshots of their intranet, articles that staff members had written regarding their independent research, copies of forms used to evaluate the workshops, and the results of an internal survey of instructors conducted by the library. Additionally we collected screen shots of the communication systems currently in use, including Microsoft Entourage & Communicator, The Instructor College, and the Intranet site maintained by the office, among others (Appendix A). We interviewed a total of 8 library instructors. Our data was derived almost exclusively from these interviews, and our data analysis took place during a series of interpretation sessions. We discussed the respondent’s replies to the interview, and the larger implications those responses had for the culture of the organization, as well for the scope of our project. We examined the available artifacts to obtain an idea of the systems in place and their limitation, in order to recommend improvements. Modeling The most useful model in terms of the scope of our project was the communication model (Appendix C) First, the communication model revealed a great deal of detail regarding the culture of the organization. It provided an abundance of information regarding the kinds of sharing currently taking place. By examining these models it was observed that while communication is an essential part of how they do their jobs, communication breakdowns persist. We also constructed a cultural model, artifact model, sequence model, physical model, and a series of consolidated models to examine various aspects of their activity. Each spoke to the research behavior of the instructors, the organizational structure, and the systems currently in place. They also helped us determine what areas were functioning well and where we could make suggestions for improvements. A total of 239 Affinity notes were extracted from the data generated. The Affinity Diagram helped us to navigate the copious amounts of information obtained, and to recognize trends in the data that were used to make our recommendations.
Key Findings Our research led us a series of findings, many of which bode well for the stated objective of increasing the sharing of pedagogical information. We grouped our findings into two major categories: research and communication, and problems and solutions. Sample data is provided after each finding, in italics. Communicative Environment The Hatcher Library Instruction environment is highly supportive of communication among staff. The group is large and the focus is on independent work, which may complicate communication and its wide dispersal. Many small networks exist within the larger social system where communication and collaboration occur frequently. “The Department is very open” “If I have a stupid question, I can say ‘Hey, I have a stupid question”
6
An Interest in Pedagogy is Present The data further indicated that most, but not all, of the staff members are interested in learning about new pedagogical methods and trends. Those who are do research are gathering it from a variety of sources, including blogs, literature reviews, colleagues, organizations, and other institutions (Appendix C). “People are very open to hearing new and interesting ways of teaching.” “If you teach you should always be learning about teaching.” Sharing Occurs but Could be Increased Sharing of pedagogical research findings is occurring. However, this sharing is mostly limited to the smaller networks. Most sharing appears to occur "point - to point," where one staff member will pass along information to another, whom they may think is interested. Comments such as this by U04 where not uncommon; “For the most part I just share with specific people...” Some staff members felt that sharing would increase if it were more formally encouraged. It was also noted that some may resist sharing their knowledge or experience for fear of criticism or exposing "failures". The current structure of communication within the organization is not conducive to the sharing and broad distribution of the information. Information that is circulated appears to only reach a small audience. For example, User 06 and User 05 communicate with one another, but User 06 rarely shares information with her other colleagues. “If I find resources that are of particular interest to someone, I contact them directly” Lack of Time is a Major Factor Lack of time is a major constraint for everyone. This leads to prioritizing and has had an impact both on the amount of research being done, and being shared. It was often noted that the staff felt they either had little time to do research, or not enough time to investigate findings that were shared by colleagues. When discussing possible systems to increase sharing of pedagological research, it was often noted by staff members that any solution should to be quick and easy. An inconvenient system, or one that required excessive additional time would go unused by staff who felt they could not budget time for this activity. “All the librarians are very time constrained.” “Instruction gets lost because we are dealing with other fires.” Existing IT Systems are Not Facilitating Sharing Efficiently There are numerous IT systems in place. The majority of respondents felt that these systems were not facilitating the sharing of pedagogy efficiently. Some systems were criticized as being difficult to use, while others were said have information that was irrelevant to the respondent's interests. It was also often noted that having one, centralized system, would greatly increase use and efficiency. In addition, we also noted that a large variety of communication systems exist within the organization (i.e. Exchange server, Intranet, Instructor College), and there is no apparent protocol for the coordination of those systems. For instance, not every user interacted with all of them. Furthermore, individual preferences appear to create a lack of continuity within which organizational knowledge remains largely tacit. Numerous breakdowns in the model appear to exist between users as a result. An example of this is the sense of indifference that was noted regarding the Organization's Intranet. In this case, users expressed a dislike for this system because it lacked relevant information. “It would be helpful if there was one location to report things that were tried that were successful”
Recommendations Our recommendations were based on efforts to identify solutions that are mitigated by the constraints we faced as set forth by the coordinator. Below we are providing 7 recommendations to the graduate library.
7
In light of our findings, it was decided to make recommendations based on an analysis of our key findings grouped into the themes and categories mentioned in the key findings section. They are listed here from least intrusive and most cost effective, to more complex and challenging to implement. Our ideal solution to match both a least intrusive and most time/cost effective implementation schedule is represented in Recommendation #5. This is the example of the Public Folder (Appendix A and C) which satisfies our agreed upon balance between the constraints of cost to the individual and organization, against our interest in offering a solution which is both easily implemented and practical in the short term. Criteria for Recommendations The coordinator asked us not to interfere with the way instructors conduct research – that was our primary constraint. A major constraint frequently mentioned by the respondents regarded a chronic lack of time. We took this into consideration when attempting to construct our set of recommendations. For this reason, we determined that among the costs we should consider, were the costs associated with tradeoffs between implementation of a solution set and the burden assumed by the instructors. We decided to consider this cost in terms similar to those outlined in the published article termed: The Cost Structure of Sensemaking (Russel, D.M., Stefik, M.J., Pirolli, P., Card, S.K.: 1993). With regard to cost, there are various ways to estimate its function practically, and informationally. We can speak of the cost in terms of cognitive resources, tradeoffs in navigating representational shifts, loss of time, and learning loop as well as the most (fiscally) cost effective implementation strategy. In terms of utility maximization and optimization, we took all of these considerations into concern, and formulated our recommendations accordingly: on the basis of time versus cost. Our observations and data indicate that the key findings can be grouped into the themes and categories detailed herein below. Recommendation #1 - Bulletin Board (Low Cost/ Short Term) One recommendation that would require very little effort, time, and resources would be a bulletin board placed in one of the central spaces of the office where instructors can post articles and ideas they have discovered. The bulletin board could be placed in an area where instructors pass by frequently or at least on a regular basis so that they might notice any new recommended readings. This option could take the form of a file drawer, bulletin or white board in the staff lounge. Advantages: It can be implemented quickly, it would be low cost, and it would provide a central place for instructors to store information. Disadvantages: This is an analog solution and would require the instructor to be physically present in the central location to take advantage of this solution. This solution may not be long term, and eventually another solution may have to be explored. Therefore, the library may be only prolonging the inevitable. Recommendation #2 - Master Calendar (Low Cost/Intermediate Term) It became evident while speaking with the library instruction team that simply knowing who was teaching which workshops and when would be helpful so that instructors could remain informed regarding which workshops are being taught, how frequently, and by whom. This would be helpful so they could know whom to contact regarding the research completed on that specific topic. One of the respondents suggested that a master calendar listing the workshops, locations, and instructors would be helpful for this specific purpose. This master calendar should be available online, possibly located within the intranet, or some other shared space.
8
Advantages: It would allow instructors to see what classes their colleagues are teaching, enabling them to contact that person regarding teaching or researching methods, perhaps even allowing them to attend the workshop. Disadvantages: Information regarding research and preparation activity would still need to be sought by the instructor, and she would still need to contact that person directly for the information regarding his or her research. Recommendation #3 - Mandate (High Cost/Short Term) The organizational structure is very flat. Based on our research, we discovered that instructors rarely report to the coordinator regarding the instruction process, and that the coordinator is not aware of specific details regarding research activity. It was discovered that there is currently no research requirement or protocol for sharing. An organizational mandate could address this. This “top-down� solution would motivate the instructors to invest more time and effort in both research and sharing activity. With a mandate intact, the goals of the coordinator would be addressed using the command and control structure currently in place. Advantages: A minimum requirement of additional research and sharing activity would be required to effect this change. It would induce a paradigm shift, wherein the staff communicates with others as a matter of priority. More emphasis will be place on instruction preparation activity with regard to research, which is currently neglected by some instructors. Standardizing instruction preparation activity would contribute to greater efficiency. Disadvantage: The change in the structure of the organization will take time for the staff to adjust to, and become comfortable operating within. There could be consequences to the moral of the staff, and organizational friction could result. With a standardized protocol the time constraints of the instructors will be complicated. The result could be loss of productivity elsewhere. Recommendation #4 - Integration of current IT Systems (High Cost/Intermediate to Long Term) The respondents frequently mentioned that there are too many IT systems in place and that there are usability issues. Our recommendation to solve this problem is to centralize the IT systems currently in use, adding a new interface that is user friendly. This recommendation would require a significant information architecture redesign. It would combine IT systems where possible, and a new design would increase usability. The ideal new system would contain additional capabilities and functionality such as filtering, search and tagging. Advantages: This is a long-term solution that needn’t be implemented immediately. This system could be built in a modular way, with future user needs and applications taken into consideration, allowing for scalability. Functionality could be added where required. This system would be convenient for staff to share information. It would also streamline the current IT environment. Disadvantages: The solution would be costly to implement. It would require new IT infrastructure and network architecture to be designed and implemented. Users will be required to learn, and adapt to the new system. Both sunk and switching costs will be significant, and could present barriers to implementation.
9
Recommendation # 5 - Public Folder (Intermediate Cost/Short Term) This is our ideal recommendation to satisfy both our time and cost constraint concerns. It involves simply the implementation of an additional folder to the Exchange Server that would allow a combination of RSS feeds as well as email on the basis of a dedicated list-serve, links to various organizations and databases mentioned as frequently in use by the staff. This could be combined with the RSS Popper program (Appendix A). Advantages: It is convenient - it appears in a place that instructors look at every day, and does not require them to go to many different websites to seek information. It is not obtrusive. The existence of the folder will not interrupt their daily routine nor does it clog up their main inbox Disadvantages: Although the information in the public folder would contain links, those links would be contained in an email. The public folder mainly organizes research-related emails. Recommendation #6 - Social Tagging (High Cost/Intermediate to Long Term) One possible is the use of a social tagging system, such as Del.icio.us. A "main" tag(s) could be created using a controlled vocabulary (such as "Hatcher-Instruction") to provide staff an efficient and unobtrusive way to locate pedagological research links bookmarked by colleagues. Additional tags would help staff narrow their browsing. (Appendix A) Advantages: The architecture for this solution is currently in place and it could be deployed, once the main tag was agreed upon. This system is easy to use, and when combined with the Firefox Del.icio.us toolbar, using the service is as easy as adding browser bookmarks. Staff may already be using this service, so switching costs for those users would be minimal. It would also be easily accessible from any computer. The system can easily be shared with, and added to by colleagues outside of the library Disadvantages: It can only be used to direct other staff members to resources that are web accessible. The use of De.licio.us is an additional service to some, and may be therefore seen as an imposition. Adoption may not be 100%, non “tech oriented� staff may resist. Recommendation #7 - Custom Recommender & Collaborative Filtering (High Cost/Long Term) Ideally the system would have an algorithm that scales independently and produces real time recommendations based on massive data sets. (Linden, Smith, and York: 2003). In this case the recommendation interface and technology are custom designed (Shafer, J.B., Konstan, J., Riedl, J. ) and provide the users an ability to collaboratively filter information based either on an internally derived tagging and taxonomy assumption basis, or standardized using semantic scheme widely accepted and understood within the librarian community. In this case, it could be a standard established by a national organization or professional organization. Advantages: This system would be proprietary, and could therefore be licensed to other institutions, creating a potential income stream and social capital for the library. It could also be designed to specifically interface with the network architecture already in place in the most unobtrusive of fashions. Users could request particular design features specific to their needs, and therefore microcustomization could mitigate internal resistance concerns. Disadvantages: Switching and sunk cost would be significant. Institutional resistance may be encountered to
10
such a radical recommendation. Implementing the system will not be enough to ensure its usage, it would require organizational leadership to commit to the implementation of a protocol governing its usage. Additional costs to the organization could be accumulated which may burden staff members in the short term.
Conclusions Our research revealed that the organization provides an excellent environment for the intended outcome of increased information sharing regarding pedagogical methods. Most of the respondents reported the belief that the library is an open and communicative environment though a few are reluctance to share for fear of judgment by colleagues, in the event that innovative methodologies were unsuccessful or "failed.� Most respondents did have a desire to improve their teaching methods, but a lack of time in the conduct of their present responsibilities combined with a lack of emphasis on this type of research as a matter of protocol by organizational leadership has resulted in it becoming of lower priority. We formulated a series of recommendations, which we believe are capable of increasing the sharing of information among staff members. A combination of various recommendations may be the ideal solution to interface the unique organizational culture represented by the Hatcher Library. We would like to encourage the staff to explore each of these options, instead of focusing on our ideal recommendation. We have attempted to take into account budget and time constraints, as well as varying comfort levels with technology among the staff while designing the possible recommendations detailed here. We strove to offer a variety of options that are adaptable to the needs of the organization per the data we possess. Lastly, the recommendation we would like to emphasize overall is that of the Public Folder as an addition to the Exchange Server. Our team strongly believes that this solution may be an ideal compromise between the costs and benefits of implementing any new system for the sharing of pedagogical research. While there are limitations to this option, many staff expressed variations on our concept during the course of our interviews with them, and we believe it to be one of the least disruptive solutions available as the infrastructure is already in place. This solution could offer the best "quick fix" for the library, while other longer term and higher cost solutions that offer greater benefits to the organization are being explored.
11
Appendices The following appendices are visual representations of the artifacts we acquired during are data collection and analysis and examples of how some the recommendations may appear if they are implemented. We have also included a chart that shows our recommendations and where those specific recommendations fall in terms of cost and time.
12
APPENDIX A Microsoft Entourage Folder Example Currently used as an office management software tool, including email access.
13
Microsoft Communicator Screen Shot Currently used as a consolidator of instant messaging chat software
14
The Instructor College Screen Shot The website for the Instructor College contains information regarding events, and information. There are 9 related links related to teaching and activities related to pedagogical improvement.
15
Selectors Space Screen Shot This is a common space available online for instructors.
16
User Activity Report Screenshot This is the form used after each workshop for statistical purposes. The only things required are the user’s name, email, whether or not the class should be reported and the name of the class
17
Library “Shared space� Library employees are given access to the areas of shared space that inform their work. Documents are shared in the various folders. Some are departmental, others are crossdepartmental. Examples of the types of documents found in shared space are reports, procedures, lists in the form of databases, etc.
(Provided by U04)
18
Selectors Shared Space Selectors purchase items for the collection and this is a place where we can find documents that help us with those activities.
19
RSS Popper: Public Folder Recommendation This is our ideal recommendation to satisfy both our time and cost constraint concerns. It involves simply the implementation of an additional folder to the Outlook (Entourage) window that would allow a combination of RSS feeds as well as email feed on the basis of a dedicated list-serve, links to various organizations and databases mentioned as frequently in use by the staff. The area of the public folder is highlighted in red here below.
20
Social Tagging Recommendation A "main" tag, or set of tags, could be created using a controlled vocabulary (such as "hatcherinstruction") to allow staff a simple way to locate pedagological research links that have been bookmarked by their colleagues. Additional tags would help staff narrow their browsing down to only items relevant to their interests.
21
APPENDIX B Internal Survey Results (Provided by U04) Information Needs of Graduate Library Subject Specialists Poll Results
Information Needs of Graduate Library Subject Specialists
30 25 20
Experienced Subject Specialists
15 New Subject Specialists
10 5
er Oth
or p ape r) Loc lec tion atin gd sta tist ata ics bas eu Tra ckin se sta gb tist udg ics et / Fin spe din ndi gp ng roc info edu Fin ral din gui gr del epo ine rts Ou s of o tlin eo t h er f th dep e fi ts. rm ord er p roc ess
atii
ng
col
line Loc
Loc
atin
g fo rms
(on
No
ans
we
r
0
Most common information needs. Easy access to: • Procedures/Policies/Reports • Statistics or Data • E-Resources Information Other areas of frustration: • Budget • Lack of time • Specific policies
22
track
I do not c
n vis
ollab
in W ord/E xc el
red t ools/ apps .
on co
ll dev
tasks
Othe r
Blog ging Tools
Chat or IM softw are
r sha
orate
Othe
ges
il
ema il
ema
the L ib. (M 5) se rver
chan
Shar ed sp ace o n
Use
ussio
Exch angin g file s via
Disc
No a nswe r
Internal Survey Results
(Provided by U04)
Use of Collaboration Methods and Technologies
25
20
15
10
5
0
23
APPENDIX C Cost/Time Key Recommendations Chart The X-axis is a representation of the relative time that would take to implement the recommendations presented. The Y-axis shows the relative cost that will take to implement our recommendations. It is important to note that this cost is not only represented on the basis of financial terms, but also based on the consumption of other resources. Those include the cognitive burden associated with the solution, organization and cultural shift required, time, learning curve, and related intangible costs.
24
Consolidated Communication Model This model demonstrates the communication flow between the Hatcher Graduate Library staff who teach workshops and their colleagues. In doing this, it represents the information systems within which this communication occurs, as well as the dynamics that exist between groups and individuals. This model only tracks communications that are relevant to the scope of our project, which is to investigate the research and sharing of new teaching methods and trends as reported by the instructional staff. This type of modeling was essential to our project due to its focus on communication. An essential objective for this project is to analyze the content, mode, and methods of communication currently in use by the staff as they design and implement instructional modules. Another objective is to obtain and articulate a "big picture" perspective of current organizational practices with regard to pedagogical research.
25
Endnotes Linden, G. Smith, B., York, J. (2003). Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEE Internet Computing (Industry Report). Jan-Feb 1089-7801/03 (pp.76-80) Russel, D.M., Stefik, M.J., Pirolli, P., Card, S.K. (1993). The Cost Structure of Sensemaking. INTERCHI ’93. ACM 0-89791-575-5/93/0004/0269 (pp. 269-276) Shafer, J.B., Konstan, J., Riedl, J. (2001). Electronic commerce recommender applications. Journal of Data Mining and Knowledge Discovery, 5(1/2):115--152 * Client provided information in original‌
26