Published on March 3, 2008
Slide1: Work Centered Evaluation of Collaborative Systems – the Collate Experience Hanne Albrechtsen Cognitive Systems Engineering Centre Risoe National Laboratory Roskilde Denmark Outline : Outline The Collate Collaboratory Evaluation Main results The Collate Collaboratory: The Collate Collaboratory The problem of cross-cultural film archive research: Valuable historic document collections exist but scattered in national archives Sources mostly not available online Lack of access to content Valuable expert domain knowledge exists but mostly inaccessible to externals Tacit knowledge, insufficiently documented Lack of technology support for collaborative research The Collate Collaboratory: The Collate Collaboratory Goals: To construct a “Collaboratory in use” Preserve historic documents in a distributed repository Content: European film documentation (20ies and 30ies) Metadata (cataloguing, indexing and annotation) Content providers / pilot users: Deutsches Filminstitut – DIF, Frankfurt, Germany Filmarchiv Austria, Vienna, Austria Národní Filmový Archiv, Prague, Czechia Technology developers: Fraunhofer IPSI, Darmstadt, Germany University of Bari, LACAM Lab, Bari, Italy Sword ICT S.r.l., Bari, Italy Evaluation partner: Risø National Laboratory, Systems Analysis Dept, Denmark Digital Repository: Digital Repository Data Historic film censorship documents (legal docs, application & decision forms, correspondence, etc.) Press material Photos (stills, portraits) Film posters Digital video fragments Metadata – value-added information by multi-national film experts Indexing & annotation Slide7: Censorship documents Cataloguing and Keyword Indexing: Cataloguing and Keyword Indexing Slide9: Evaluation of web-based collaboratory Evaluation framework: Evaluation framework 1. Conceptual framework and method: Cognitive work analysis Frameworks for work analysis and prototype evaluation Work centered approach a. Analytical evaluation: Is the design right? Input: evaluators examine interface Laboratory experiments - ‘think aloud’, eye-tracking b. Empirical evaluation: Is it the right design? Input: Empirical work studies and experience from field experiments Cognitive work analysis and empirical analysis of archival work: Cognitive work analysis and empirical analysis of archival work 1. Goals: To evaluate how the Collate prototypes support film archive research (work content), collaboration and knowledge sharing; acceptability 2. Inputs: Phase 1: Field studies of three film archives involved in Collate Input for empirical evaluation: identification of recurrent course of events in collaboratory film research (construction and mediation of knowledge) Phase 2: Field experiments on the course of events in collaborative research task scenario by use of Collate prototypes (at evaluation workshop) Cognitive work analysis: Levels of work analysis : Cognitive work analysis: Levels of work analysis Cognitive work analysis: Prototype evaluation: Cognitive work analysis: Prototype evaluation The means ends abstraction hierarchy: The means ends abstraction hierarchy Slide15: Empirical evaluation: Evaluation workshop Focus on the archivists’ (collaborative) work with the censorship material and the prototype’s support of: Common strategies Collaborative decisions Means and ends of common workspace Slide16: Design of evaluation workshop Participants: Two expert representatives from each of three Collate archives Evaluation team from Risø (four evaluators) Study method: Participant observation of collaborative research task scenario: To produce a censorship history of two films, using the Collate prototype (Film 1: Potemkin; film 2: Die drei von der Tankstelle) Settings and events: Introductory session for all Two laboratory sessions, each for one film Two groups of archive representatives and evaluators Concluding focus interview, including debriefing of group work Ethics: Aware participants (purpose of experiment known to all) Slide17: Design of collaborative task scenario for evaluation workshop Collaborative task: To produce a censorship history of two films Panzerkreuzer Potemkin, version 1930 Die Drei von der Tankstelle Suggested activities: Find censorship documents and materials (search) Coordinate the work Link documents and materials (cataloguing and indexing) Write your comments (annotation) Find keywords for comments Categorise comments Enter meta-information: comments on comments Slide18: Session at the evaluation workshop Empirical analysis: Empirical analysis Dimensions of the framework used for empirical analysis of collaborative knowledge integration: Dimensions of the framework used for empirical analysis of collaborative knowledge integration What are the archivists’ common strategies for problem solving and how does the prototype support them? What are the archivists’ collaborative decisions and how does the prototype support them? What are the means and ends of the common workspace for knowledge integration and how does the prototype support them? Slide21: Common Strategies: Choose a core document Slide22: Collaborative Decisions: Interpretation Collaborative decisions:2. Assign keywords: Collaborative decisions: 2. Assign keywords Common workspace: Common workspace Slide25: Main results from evaluation workshop User experiences Slide26: Main results from empirical evaluation - reflections Annotation module: Main advantages: Creativity and innovation of film archival research; new professional networking; innovations in research concepts and communication Main challenges: 1. Diverse conceptions of the value of making previously’ invisible’ work public; role allocations; ‘write comments-or-perish’; 2. The risk of implicit formalisation of informal work activities and professional networking Indexing module: Main advantages: Standardisation of categories and terminology for subject access to digital collection Main challenges: 1. Diverse understandings of concepts and terminology amongst archive staff; diverse professional and personal approaches to indexing work 2. (How) will end-users understand singular representations of collaboratory semantics? Slide27: Main results from empirical evaluation of knowledge integration by use of the Collate prototype Indexing feature in the collaboratory: Main advantages: Standardisation of categories and terminology for subject access to digital collection Main challenges: 1. The archivists have developed new strategies: from focus on singular documents to censorship history and case -> choose a core document 2. The archivists need to establish additional frames of reference for interpretation of single documents (cultural, historical, research perspectives) -> deeper semantics of work 3. The new deep semantics of knowledge integration goes across the divide between an indexing module on the one hand and an annotation module on the other hand Slide28: Future research avenues 1. Categorical complexity in computer-mediated collaborative research: Collaborating actors’ intersecting and diverse conceptions of the semantics of work (examples: annotation and indexing in Collate collaboratory) Aim: Models for work-based design of tools for knowledge exchange and organisation in (collaborative) research 2. Collaborative task situations as a main unit of analysis for design of research collaboratories: Prototypical task situations and the users’ need for computer-mediated support Aim: Construction of common work spaces for collaborative integration of knowledge and as test beds for technology development Censorship Keyword Map: Censorship Keyword Map COLLATE System Layers: COLLATE System Layers text docs photos annotations videos <XML metadata> <XML metadata> <XML metadata> <XML metadata> Metadata management cataloguing annotation interlinking indexing terminology development Structural manipulations internal external external Scientific discourse Interaction Collaboration Layer Task Layer Domain Metadata Layer Operational Layer Interface Layer
WETICE 2004 ECE Workshop - Final Report. 155-160. ... Hanne Albrechtsen, Hans H. K. Andersen, Bryan Cleal: Work Centered Evaluation of Collaborative ...