Developing design principles for technology-based environments in education: considering the role of ethnography
Abstract This article takes the perception of ethnography as not being generalisable and questions whether such an understanding is correct and also whether a different perspective on generalisability might be helpful in developing formative approaches to educational technology research. The article explores how an ethnographic approach to exploring the failure of technology interventions to have the effect on education hoped for by their developers might provide some guidance to others. The aim of the article is to encourage discussion about ethnography as a tool for technology researchers, where institutions are treated as a socio‐technical system. It will be argued that such studies require localised understanding prior to interventions being developed, but that mutable design principles can provide a generalised starting point for formative research projects. This paper advances the argument that if effective implementation of education technologies depends on a deep understanding of the social world, this understanding can be reified into design principles using qualitative social science research traditions. In particular the paper seeks to apply a social informatics perspective to the existing body of design based research approaches – arguing that ethnographic methods have an important role to play in the design of well‐integrated tools, and that this aspect of ethnography’s usefulness is often overlooked by technology developers.
Overview Research on technologies for use in education environments is often predicated on a generalised systems development approach, where common solutions to problems are built in to the design. The strength of this approach is that while such systems start off with a core of generalised problem solving heuristics the design can be refined over time to meet the needs of particular user groups more closely. However, it could be argued that a weakness of this approach is its failure to engage with the socio‐technical system within which it operates. This lack of social sensitivity is increasingly recognised as an underlying factor in the failure of education technologies to noticeably transform classroom practices (see e.g. Selwyn 1999; Robertson 2003). Some attempts have been made at identifying general characteristics of a successful ICT intervention (see e.g. Tearle 2004). However it is not clear that such ideas have either been adopted or that they are capable of being so, due to the explanatory rather than developmental function they perform. Most studies continue to focus on effects of or with technology, independent of considerations of sustainability of the development. By focussing on Collins et al’s intention that ethnographic aspects of design research should form part of, “an infrastructure that can support researchers at different sites analyzing the large data sets ...being produced now” (Collins, Joseph et al. 2003) the article seeks to extend and deepen the role of design science within education research. Studies aimed at generating and refining design principles would provide evidence for other types of research to draw on and also promote a deeper understanding of the ways technology is interlaced with the other elements of an education institution.
The paper begins with an explanation of design principles. It then illustrates how an ethnographic approach can capture some of the tacit features of education environments which can help identify these design principles. These twin aspects of ethnography and design thinking are then drawn together in a discussion of the construction of a concrete design principle, illustrated using a current project related to a virtual research environment (VRE) in a large UK Further Education college. The paper concludes by discussing the issue of researchers avoiding the problem of discussing persistency of tool use after the research with a concluding sentence along the lines of, “X will take hold only if the A,B,C features identified become part of the education culture”. Such statements allow the researcher to focus on the potential for their intervention but they do little to effect those changes in practice which are seen as important. The research project discussed here aims to provide a mechanism where ethnography can help develop the ‘implementable validity’ described by Argyris (2004) and therefore improve the potential impact of formative research . Argyris describes implementable validity as the way in which the applied aspect of formative research can be judged as trustworthy; it allows the reader to judge whether it has been demonstrated during the research that the intervention is capable of being implemented within a real environment.
Design Principles within a mode of scientific enquiry A design principle is a statement which synthesises the knowledge gathered from designing an artefact. An essential feature of design principles is that they should be simple and should have enough of the contextual detail abstracted away to provide a general rule of behaviour that can be carried across contexts. An example of a universal design principle takes the form of , "to design X ...then you...are best advised to give that design the characteristics A, B, and C...and to do that via procedures K, L, and M , because of arguments P, Q, and R." (van den Akker 1999). Design principles developed along these lines should be : •
Short – they are a generalised principle not requiring lots of explanation
•
Memorable – they should describe something important in a way that resonates with the reader
•
Based on research of user experience – observation and description of behaviours rather than the declared goals or preferences of users.
•
Non‐conflicting – a group of related principles should form a coherent whole
•
Problem specific – a statement such as ‘strong leadership’ or ‘enthusiasm for ICT” are not design principles
As a concept design principles form part of a language for discussing the necessary architecture of a well‐functioning system. They are made up of the contextual, substantive and procedural features of a good design but do not, and should not be seen as providing a ‘foolproof recipe for success’. However, they do outline appropriate structures and processes around which a successful solution to a problem can be built whilst being sensitive to four key problems that Martin (2000)describes in terms of software development, but which I argue can be applied to any designed system:
•
Rigidity – change is difficult
•
Fragility – a change in one part of the system can have negative consequences elsewhere
•
Immobility – systems can be highly situated and not readily transferable
•
Viscosity – both the environment and the purpose of a system exist in a dynamic relationship with the world
Design principles then are mutable but changes to them require empirical justification. This can be achieved through the development of an alternative design principle which provides a better account of the idea in question and as Linn et al (2004) point out they are thus falsifiable, though not in a truly Popperian sense. As well as describing what ethnographically derived design principles are or how they might differ from such other concepts as ‘characteristics’ or ‘traits’ it is important to describe where they sit ontologically. This is because ethnography can be used to produce different but nevertheless valid types of account with the same data. This means that when using it within a design setting it is important to understand the types of question it is being used to address and how this differs from other types of question which ethnography helps answer. Design principles are an approach to scientific knowledge production in accord with the styles of reasoning discussed by Ian Hacking (1992). In this paper Hacking explains how reasoning styles can come to dominate in a field of scientific enquiry and gives examples of how and why this happens. He shows for example how statistical reasoning has spread across problem domains and produces stable new knowledge, partly as a result of its ability to self‐authenticate; the probability of something being statistically true is measurable in terms of probability. Design principles can be seen as one of what Hacking calls philosophical technologies; it is not that they are the revealers of objective truth but that they can become the measure of objectivity and the knowledge they encapsulate is objective in the sense that the reasoning style is accepted as the right way to reason. This places a heavy burden on design principles; they are mutable, not static ‘laws of nature’, and yet they must provide a stable and consistent way in which they can encapsulate knowledge. I would argue that this can be achieved if the design principles that researchers develop meet two criteria; firstly, that they are developed following in‐depth study of the behaviour of the people and artefacts touched by the design principle and secondly that they are capable of self‐authentication, meaning that they can be tested using other design principles. An inverted design principle for example can be developed to test the robustness of a newly developed one. I believe the design‐based approach to knowledge production is a candidate for being a different type of scientific reasoning style to others such as experimental study. The important issue however is not whether a ‘design style’ of reasoning exists separately from others. The important thing is that it is a trustworthy way of understanding that systematic enquiry into technology augmented education practices using ethnography and design principles is scientifically grounded, notwithstanding the naturalistic way that data is collected.
Ethnography and Design This paper concentrates on the role of ethnographic methods in order to stimulate debate on the ways in which deeper understandings of the full context within an individual institution can be used to develop more generally useful design principles. This lays emphasis on particular styles of ethnography that can be seen as directed rather than the view of traditional ethnography as ‘hanging about’. Often ethnography is used for examining situations from a critical standpoint such as Star’s work on infrastructure (1999) or from psychological perspectives such as the situatedness of cognitive activity (Suchman 1994). I am arguing that there is an additional use for ethnography in the area of design based research. I would suggest that the naturalistic approach to data collection places ethnographic methods at the heart of an analytical contradiction. If the traditional definition of ethnography is accepted then the outcomes of ethnographic work are highly situated and not generalisable. In his seminal critique of ethnography Hammersley points out that one of ethnography’s aims is to produce theoretical descriptions, but that such descriptions cannot be theoretical as, “they represent objects and events in particular space‐time locations; whereas theories are about types of phenomena, wherever their instances occur” (Hammersley 1992, p27). However, the use of such context‐dependent ethnographic methods in developing generalised design principles is a long recognised practice in socio‐technical systems analysis and computer systems development (see e.g. Clancey 1993). Design principles are the abstracted essence of a designed system and provide the mechanism whereby effective design can be carried across contexts without losing the elements that make it useful. A meta‐design‐principle would be that design principles must be applicable within a problem domain. It cannot be true that ethnography is not generalisable and merely provides interesting, though essentially useless (from a theory generation standpoint) ‘thick description’ while at the same time providing the material for design principles. In various fields of knowledge including economics (Ostrom 2000), library and information science (Crabtree, Nichols et al. 2000) , and architecture (Memmott, Hyde et al. 2009) ethnography is being used as a key component in the development of new knowledge in the form of design principles. Recent work in the field of health sciences research has also recognised the need to move beyond positivist approaches to studies of technology (Greenhalgh and Swinglehurst) . In the worlds of computer supported collaborative work and computer supported collaborative learning (CSCW/ CSCL) the use of ethnography is also widely mentioned (see e.g. Harper 2000) as are design based approaches (Scardamalia and Bereiter 1994) but rarely are the two combined in such a way as to assist researchers in ensuring persistency of their innovation even where the issue is recognised and the researchers are aware that, ”descriptive research is required to set the stage for principled interventions” (Hoadley and Pea 2002). ‘Principled’ is used here to mean an intervention that is appropriated and used by the community it is designed for, rather than being a research tool with no life beyond the project within which it is used. In her seminal paper on design experiments in education Ann Brown highlights how useful ethnographic methods are in understanding how and why her cognitive experiments have provided evidence of positive change. In the summing up of problems faced by design based researchers she
clearly sets out what sounds like a manifesto for the creation of design principles drawn from ethnographic studies, “The question becomes, what are the absolutely essential features that must be in place to cause change under conditions that one can reasonable hope to exist in normal school settings?.... I need to use ethnographic methodologies...to know the conditions for acceptance...to study the sociology of dissemination... ” (Brown 1992, p173). Some twenty years later it seems that this insight is still not fully accepted by many but, even where it is, there is a lack of research evidence that would provide the data store from which such principles could be drawn. One project that took the idea of design principles for cumulative knowledge building as its focus was Kali’s work on the Design Principles Database (Kali 2006). The strength of Kali’s approach is her understanding of how to engage stakeholders with the design process and structuring design principles to meet her project’s aim; that they should serve to “coalesce and synthesize emerging design knowledge about the use of technologies for education” (ibid, p188). However, because the study does not address the idea of design principles from the position of persistency of innovation the principles themselves relate only to the classroom and technological aspects of the research. As with most studies of this type therefore, despite identifying the importance of design thinking and developing a promising approach to practitioner input (where the term includes students, teachers and other actors) it is not clear how this database could be taken up on a large scale. This is not to criticise Kali’s design principles based approach; indeed, it is because such work has so much to offer the education system that I argue for what would essentially be a supporting technology for such projects. It is the role of design principles to provide guides to the development of new and innovative practices that can become part of the habitus of real environments, not as highlighted examples of ‘good practice’ or ‘innovative methods’ but as an embedded feature of a high quality education institution. An approach built on socio‐technical analysis drawn from ethnography can enrich the design research community and provide ethnography with a new purpose in addition to its existing uses. It would seem then that there is potential for ethnographic accounts to confound Hammersley’s assertion and produce theory capable of testing and refining in different places and times. It is important to address Hammersley’s well argued criticism as it is still made, including in the area of design based research (Collins, Joseph et al. 2003; Kelly 2003). The findings in such research about those aspects of social construction that ethnography is so strong at describing, innovation and development in technology augmented environments, is almost entirely overlooked because the quasi‐experimental approach is privileged. However if ethnography is to be an effective tool for developing knowledge building tools such as design principles it must take into account Walford’s advice on selection of sites (2001, p157). His advice that ethnography is most useful when it, “focuses on cases that are important in themselves because of their relationship to far wider concerns about schooling and the state” helped to focus this research by ensuring it was sensitive to the wider potential for the findings to be used.
The construction of a design principle The role of digital technologies in augmenting traditional education settings is often discussed in terms of how a particular technology‐based intervention has impacted on the learning of students and the practices of teachers. The problem with this is a failure of such studies to be translated into
lasting, embedded changes of institutional culture. Robertson does an effective job of stating the problem (2003). Disappointingly he resiles from explaining why ethnography will provide any answers to it, seeing it as only providing more deeply grounded explanations of the issues. In this section therefore I will describe how the design principle #keylink was developed, as it provides a useful example of how ethnographic methods can be used to move from behavioural observation to knowledge capture. The research project discussed here concerns how ideas from e‐science might be taken into the education field through the development of Virtual Research Environments (VRE). The research faces the problem of only being useful if it is persistent. Without implementable validity, that is to say a clear explanation of the basis on which the research can be translated into the practices of an education institution, it is nothing more than a conceptual conceit. It is formative research work and needs to be capable of appropriation by a range of communities; these include but are not limited to other education researchers, educators and education managers, policy makers, standards bodies and other groups. By taking an ethnographic approach and using that as the basis for developing some design principles the study benefits from an approach to design which is intrinsically coupled to the socio‐technical‐political context in which it is being trialled. A VRE is a collection of tools, practices and people that allow research work to be managed and “to support as much of the research process as appropriate for any given activity or role” (Fraser 2005). For this project it represents a technology where systematic enquiry into teacher performance can be designed, managed and run by teachers working in collaboration with senior managers and can be adopted as part of the formal teaching quality system. The project is within the context of the Further Education sector in England and the use of a VRE is being explored to see if it can form part of the quality management infrastructure within large colleges. Research into VREs within education is not common but some work has been done on using virtual learning environments as a basis for them (see e.g. Laterza, Carmichael et al. 2007). As a VRE is a major and disruptive change to the way a college supports staff development and records its quality improvement activity it was decided to adopt a social informatics perspective. This meant that instead of testing out how a particular design of VRE was received by teaching staff I focussed on the role of the VRE within the system of quality management. The idea of social informatics is that social sensitivity is the most effective method for ensuring information systems design will fulfil its potential (Kling 2007). Design principles are an essential element within sociotechnical systems analysis because they describe systems in terms of their essentials. The study therefore called for the use of design principles but in order to gather such principles a deep understanding of the system in question would need to be gathered. It was felt that the most legitimate and trustworthy method for this was ethnographic, as other methods are not so useful for drawing out the lived experience of college life. This is because ethnography can be used to measure a system’s activity in terms of behaviours as well as expressed intentions, such as policies and interview data.. A college where internal research activity was already an accepted practice and also where teaching and learning quality were independently judged as being of good quality was chosen. This was so that the focus would be on the development of design principles and a framework for effective VRE
development and to reduce the number of changes that would be expected to be made in the organisation of the college when testing the design principles in action. The college is both large, with student numbers in the many thousands and is also a member of the national Beacon College scheme which recognises high quality provision. OFSTED grades based on the 2009 Inspection framework (OFSTED 2009) were also used as a guide. A large college was selected because a VRE is of most use to research effort that is distributed over both time and space and having a college which is spread over multiple sites would be a good proxy for larger scale research involving multiple colleges as well as multiple sites. However, accepting ethnography as a research method carries risks for an institution because they do not know exactly what questions are being asked of them. The building of rapport takes time and requires constant care and attention. This was one reason why it was decided to adopt a top‐down approach to the research, counter to a lot of the literature which promotes bottom‐up change. Bottom‐up change has an important role to play in the adoption of specific technologies. However, the importance of this research rests on what it offers to other researchers, which is a clearer picture of the forces that need to be aligned for change to persist beyond the life of a research project. For this purpose, moving down through the hierarchy of control was more helpful.. The first design principles constructed were developed with a group of college governors and the clerk to the governors. The governing body of a college owns the buildings and the assets; it employs the staff both academic and support and is responsible for setting the strategy for a college and ensuring it is carried out. . The process of constructing the design principle took the form of a number of meetings and semi‐ structured interviews, document analysis and observations of activity. The resulting design principles will be reviewed during the summer of 2011 by the college’s quality and curriculum committee, which sets standards and suggested changes will then be fed into another iteration. At this point these principles will be subjected to third party evaluation by senior managers from a second college, with the social context of their development removed. This will have the effect of turning them into a synthesised version of abstracted tacit and explicit knowledge, a constructed object stripped of its social context but which serves as a measuring point for future findings (Latour and Woolgar 1986). A VRE is part of the technical infrastructure of a system and it was of interest to observe how some existing infrastructure is viewed. These observations were then discussed with the clerk to the governors, the chair of governors and the chair of the quality and curriculum committee. These interviews were undertaken to garner more detail about their experience of introducing disruptive technology into the college. From this it was possible to place the observations within a historical narrative but also within a discourse about the future of the college and the quality of the teaching and learning environment within it. A potentially controversial piece of infrastructure innovation, the use of CCTV monitoring in all non‐ classroom areas. was selected. It fulfilled the criteria as it had needed to be supported by the staff, students and outside stakeholders but it didn’t relate to computers in the classroom, e‐learning or anything directly related to ICTs. This is because it was important not to allow the view to take hold that the research would only relate to using ICTs in the classroom.This was something identified as
being an assumption of the reviewing panel when applying for access and which had been passed on to the governors. Observation activity began in the 6th form centre of the college. This was because it was a city centre location and is largely open to the general public, making security of staff and students more of an issue for the college managers than some of the more specialist centres which have more initial security such as swipe cards for doors. One incident which illustrates the subtle ways in which infrastructure can influence the behaviours of staff and students occurred during an observation of one of the public spaces. Students have access to various seating arrangements and each main teaching area on the site in question has a set of fixed sofa style seating around a central table. In addition to the seating students have access to cold drinks and snacks from vending machines and can use these social spaces between lessons and during study periods. Overlooking this space are two box‐type CCTV cameras and notices of the presence of CCTV is clearly marked in various locations. A small group of students were observed who were starting to become noisy and had also begun ‘play‐fighting’ and swearing in a way that could be heard by many of the people passing by the area on their way elsewhere. After approximately three minutes of this behaviour two members of teaching staff came through the area The two members of staff were both female and involved in a discussion of their own. When they saw the teachers the noisy group, which contained both boys and girls, exhibited no change in their behaviour and they were now being clearly avoided by other students. The two female teaching staff initially walked past the group but then one of them stopped and there followed a brief discussion which included hand gestures that implied they were discussing the group. One of them then approached two boys, who were now taking their play fighting outside the confines of the seating area and starting to collide into other people. The teacher pointed out that they were on CCTV and they couldn’t behave like that, because they had a duty under their student contract to be well mannered. One of the boys ignored the teacher but the other one said, “Yeah, you is on candid camera” and withdrew to the seating area collecting his bag. The other boy was then laughed at by two girls who told him he was going to be kicked out of college. This boy also then collected his bag and said ‘I don’t care, I couldn’t give a f*** it’s all boring I’m going home”. The two teachers then turned away and carried on walking and talking. Although this data could have been analysed in different ways, for this research it led to considerations of the issues related to the ethos of the college and how this is transmitted. The defining and maintenance of an ethos is an important part of the governors’ role as they are responsible for setting out what type of college the managers should be developing. This incident was used to look at other ways CCTV and the college’s mission and ethos were connected and how they were shared. I did this in order to develop questions for the governors about the introduction of this technology and the processes that led to it being seen as a supportive innovation which has now become part of the habitus; where teachers can use it to support their own authority and construct an environment where behaviour to agreed norms exists all over the college, not just the classroom. A set of questions relating to the quality of the experience of being at the college and how the governors were involved in measuring this were developed and three key people were interviewed in a semi structured interview style as described by Kvale (2009). The first of these was the clerk to
the governors and from him information on the history of the college and the various incarnations it had been through as it had grown was obtained. This was important because it became clear that the uncertain external environment in which Further Education colleges operate was a complex problem with a wide range of educational cultures and types of teaching belief system needing to be integrated into one. The introduction of CCTV was a case where the governors and senior management had responded to external pressure relating to safety concerns and the internal problem of persistent theft of expensive equipment but wanted to ensure that the change was managed in line with the mission of the college, to provide an open and participative atmosphere. The problem was viewed not only from a perspective of financial or managerial control but from how it might improve the overall quality of experience for staff and students. So far I had observational evidence that the college’s mission to provide a high quality experience for staff and students was visible in some of the infrastructure, and that even where such innovation required appropriation by staff and students to be successful this had been achieved. In addition I had learned that the history of the college was both complex and had on occasion been very challenging, especially where staff were being transferred from a failing institution. I used this information in my interviews with the chair of governors and the chair of the curriculum and quality committee to try and identify how they were able to retain a high degree of task certainty, knowing what the college was for and how it was going to achieve its mission whilst being flexible enough to deal with the frequent changes in operating landscape. Both felt that there were two key drivers, a large amount of transparency about their role and the boundaries to it, so that both they and managers knew who was responsible for what and secondly a strong link mechanism between the senior management team and the governors. They both separately specifically referred to the role of the clerk of governors and why it was so important to have an effective one. From observation of an incident involving the CCTV system ideas about how the college developed and maintained its particular ethos could be formulated. From this and other observational data an understanding of the complex socio‐political environment and how the college’s mission was both maintained and transmitted from the top level governors to the senior managers was becoming clearer. A design principle was proposed, related to creating a trustworthy environment, where difficult matters could be dealt with in a way that allowed change to be propagated both up and down the layers of management. I called this the #keylink principle. There is no standardised notation for a design principle but I have adopted the # marker to indicate that the term refers to a specific design principle rather than any general use of the term and I have placed the principle itself within angle brackets <<>> . The #keylink principle is an example of what Kali refers to as a pragmatic principle, it sets out the top level to which other specific principles provide more information (2006, p190). This principle states that: #keylink <<for clearly delineated boundaries between governors and senior managers to be maintained a senior manager trusted by both groups should have responsibility for ensuring clear information flows between them.>> In the format proposed by van den Akker it could be expressed in these terms
“to design a method for developing and maintaining an ethos of trust and mutual dependence in the college then give that design the characteristics of continuity of key linking individuals including a clerk to the governors, succession planning for senior officers of the corporation and clearly defined standards of transparency of information flow. This can be achieved via openly available procedures for recruitment, an active policy of preparing future volunteer candidates for senior governorship positions and regular exchanges of information between the two groups. Continuity and succession planning create a basis for trust amongst the leadership team, it increases the potential for identifying individuals capable of operating at the strategic level required in a large institution and longevity of service coupled with succession planning provide a trade off between the benefits of fresh thinking and the experience of having gone through similar situations.” This design principle has so far only gone through two iterations. It will be presented to the full governing body in late July 2011 and will then also be presented to the senior management team for their input. It may go through further revisions until it is agreed that it clearly expresses the idea that the link between the governors and the senior managers has a major impact on the development of a college’s mission and the translation of that mission into the daily life of the college, whether expressed through the people involved, the infrastructure or the documentation. As it is a design principle it might also prove to be a false understanding and could potentially then be replaced.
Conclusion I have argued that the study of how design principles can be successfully developed benefits from a choice of the ethnographic style that is widely used in education research but which has struggled to find a space in the twin streams of explanatory and developmental research that characterises design‐based studies. This argument is grounded in the suggestion that a new direction for ethnography is possible, one that connects it more closely to design and to the role of e‐science in providing education researchers with a richer starting point for design of technology based interventions. The use of design principles as a reification of tacit knowledge could be subject to a number of criticisms. Amongst these would be the interpretative nature of the principles. Can something genuinely be called a principle when it is based on the researcher‐interpreted observations of participants? If those principles are drawn from a larger scale data set then it could be argued that this layers misunderstandings on top of each other. The validity of such criticism however is undermined by the developmental and open nature of design principles. Whilst an individual principle might be developed based on a limited amount of case data those cases can be added to and the principle refined in light of new findings. Ethnography can serve many purposes, from the sociologically driven study of power and influence to the more psychologically concerned study of cognitive behaviour in individuals and groups. I believe that to this can be added the in‐depth description of complex environments from which patterns of behaviour can be mapped onto principles for the design of artefacts, which not only work well in terms of achieving their internal design goals but which exhibit the key feature of formative research which is implementable validity.
Bibliography Argyris, C. (2004). Reasons and Rationalizations: The Limits to Organizational Knowledge. Oxford, Oxford University Press. Brown, A. L. (1992). "Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings." The journal of the learning sciences 2(2): 141. Clancey, W. J. (1993). The knowledge level reinterpreted: Modeling socio‐technical systems, Wiley Subscription Services, Inc., A Wiley Company. 8: 33‐49. Collins, A., Joseph, D., et al. (2003). "Design research: theoretical and methodological issues." The journal of the learning sciences 13(1). Crabtree, A., Nichols, D. M., et al. (2000). "Ethnomethodologically informed ethnography and information system design." Journal of the American Society for Information Science 51(7): 666‐682. Fraser, M. (2005) Virtual Research Environments: Overview and Activity. Ariadne Greenhalgh, T. and Swinglehurst, D. "Studying technology use as social practice: the untapped potential of ethnography." BMC Medicine 9(1): 45. Hacking, I. (1992). ""Style" for Historians and Philosophers." Studies in history and philosophy of science 23(1): 1. Hammersley, M. (1992). What's wrong with ethnography?: Methodological explorations. London, Routledge. Harper, R. H. R. (2000). "The Organisation in Ethnography – A Discussion of Ethnographic Fieldwork Programs in CSCW." Computer Supported Cooperative Work (CSCW) 9(2): 239‐264. Hoadley, C. M. and Pea, R. (2002). Finding the Ties That Bind: Tools in support of knowledge building communities. Building virtual communities: Learning and change in cyberspace. K. Renninger and W. Shumar. Cambridge, Cambridge University Press. Kali, Y. (2006). "Collaborative knowledge building using the Design Principles Database." International Journal of Computer‐Supported Collaborative Learning 1(2): 187‐201.
Kelly, A. E. (2003). "Theme issue: The role of design in educational research." Educational Researcher 32(1): 3. Kling, R. (2007). "What is social informatics and why does it matter?" The Information Society 23(4): 205. Kvale, S. and Brinkmann, S. (2009). InterViews: Learning the craft of qualitative research interviewing. London, Sage Publications, Inc. Laterza, V., Carmichael, P., et al. (2007). "The Doubtful Guest? A Virtual Research Environment for Education." Technology Pedagogy and Education 16(3). Latour, B. and Woolgar, S. (1986). Laboratory Life: The Construction of Scientific Facts. Princeton, Princeton University Press. Linn, M. C., Bell, P., et al. (2004). Specific design principles: Elaborating the scaffolded knowledge integration framework. Internet environments for science education. M. C. Linn, P. Bell and E. A. Davis. Mahwah, New Jersey, Lawrence Erlbaum: 315–340. Martin, R. C. (2000). "Design principles and design patterns." Retrieved 12 May 2011, from www.objectmentor.com. Memmott, P., Hyde, R., et al. (2009). "Biomimetic theory and building technology: Use of Aboriginal and scientific knowledge of spinifex grass." Architectural Science Review 52(2): 117‐125. OFSTED (2009). Common Inspection Framework for Further Education and Skills. H. M. s. C. I. o. Schools. London, OFSTED. Ostrom, E. (2000). "Collective Action and the Evolution of Social Norms." Journal of Economic Perspectives 14(3): 137‐158. Robertson, J. W. (2003). "Stepping Out of the Box: Rethinking the Failure of ICT to Transform Schools." Journal of Educational Change 4(4): 323‐344. Scardamalia, M. and Bereiter, C. (1994). "Computer Support for Knowledge Building Communities." Journal of the Learning Sciences 3(3): 265.
Selwyn, N. (1999). "Why the Computer is not Dominating Schools: a failure of policy or a failure of practice?" Cambridge Journal of Education 29(1): 77‐91. Star, S. L. (1999). "The Ethnography of Infrastructure." American Behavioral Scientist 43(3): 377‐391. Suchman, L. (1994). Plans and situated actions: The problem of human‐machine communication. Cambridge, Cambridge University Press. Tearle, P. (2004). "ICT implementation: what makes the difference?" British Journal of Educational Technology 34(5): 567‐583. van den Akker, J. (1999). Principles and methods of development research. Design approaches and tools in education and training. J. van den Akker, R. Branch, K. Gustafson, N. Nieveen and T. Plomp. Dordrecht, Kluwer Academic Publishers. Walford, G. (2001). "Site selection within comparative case study and ethnographic research." Compare 31(2): 151.