Processing Linked Data with Catmandu

100 %
0 %
Information about Processing Linked Data with Catmandu
Education

Published on March 13, 2014

Author: hochstenbach

Source: slideshare.net

Description

Introduction into library data processing with help of Catmandu

http://librecat.org

Processing Linked Data with Catmandu Patrick Hochstenbach | UGent http://librecat.org

Processing Linked Data with Catmandu | http://librecat.org LUND GHENT BIELEFELD

Processing Linked Data with Catmandu | http://librecat.org RATIONALE

Processing Linked Data with Catmandu | http://librecat.org KAHN-WILENSKI WEB HANDLE SERVICE PROVIDER REPOSITORY REPOSITORY I search a paper about...

Processing Linked Data with Catmandu | http://librecat.org Hypothesis 1: one network with a common schema Hypothesis 2: object-oriented design Hypothesis 3: the resource is the message

Processing Linked Data with Catmandu | http://librecat.org Hypothesis 1: one network with a common schema GOOGLE EUROPEANA OPENAIRE CRIS Videos Images Books Data sets

Processing Linked Data with Catmandu | http://librecat.org Hypothesis 2: object-oriented design Drive Race Park Economy Compact Minivan Convertible Wheel Half car Bicycle Zeppelin

Processing Linked Data with Catmandu | http://librecat.org Hypothesis 3: the resource researcher is the message DNS GOOGLE REPOSITORY CLOUD Dr. Müller

Processing Linked Data with Catmandu | http://librecat.org LIBRECAT/CATMANDU

Processing Linked Data with Catmandu | http://librecat.org CATMANDU PubMed MARC MODS EXCEL DSPACE Fedora SRU OAI-PMH DBI ISI Twitter DBI Atom EXCEL RDF JSON XML Solr ElasticSearch MongoDB Fedora Aleph Fix

Processing Linked Data with Catmandu | http://librecat.org FUNCTIONAL DESIGN JSON } each slice take group select map reduce add_field join_field lookup remove_field marc_mab count

Processing Linked Data with Catmandu | http://librecat.org LIBRECAT Institutional Repositories Search Engines Image Databases Archival Systems Data cleaning workbench Citation Style Processor

Processing Linked Data with Catmandu | http://librecat.org CATMANDU catmandu convert MARC to JSON < records.mrc catmandu convert OAI --url http://server/OAI to JSON catmandu convert SRU --url http://server/SRU --query dna to JSON catmandu convert DBI --query ‘SELECT * FROM table’ to JSON catmandu convert MARC to JSON < records.mrc catmandu convert OAI --url http://server/OAI to XML catmandu convert SRU --url http://server/SRU --query dna to YAML catmandu convert ArXiv --query ‘all:electron’ to CSV CONVERT

Processing Linked Data with Catmandu | http://librecat.org CATMANDU catmandu convert X to Y --fix ‘marc_map(“245”,”title”)’ catmandu convert X to Y --fix ‘prepend(“title”,”abcd-”)’ catmandu convert X toY --fix fixes.txt fixes.txt: remove_field(“_id”); marc_map(“001”, “merge.id”); prepend(“merge.id”, “author:”); add_field(“merge.source”,”author”); copy_field(“merge.id”,”_id”); FIX set_field add_field move_field copy_field remove_field upcase downcase capitalize trim substring prepend append lookup lookup_in_store count cmd split_field join_field retain_field replace_all collapse expand clone if_all_match if_any_match if_exists

Processing Linked Data with Catmandu | http://librecat.org CATMANDU catmandu import JSON to MongoDB --opt ... --opt ... catmandu import MARC to ElasticSearch catmandu import DC to FedoraCommons catmandu import CSV to DBI catmandu export MongoDB to JSON catmandu export Solr to YAML catmandu export DBI to CSV catmandu export FedoraCommons to Template --template test.tt test.tt: (TemplateToolKit) [%- FOREACH f IN record %] [% _id %] [% f.shift %][% f.shift %][% f.shift %][% f.join(“:”) %] [%- END %] IMPORT / EXPORT

Processing Linked Data with Catmandu | http://librecat.org CATMANDU https://metacpan.org/pod/Catmandu https://github.com/LibreCat/Catmandu http://librecat.org/tutorial/ http://librecat.org/catmandu/2013/06/21/catmandu-cheat-sheet.html

Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://biblio.ugent.be

Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://pub.uni-bielefeld.de/en

Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://adore.ugent.be

Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://libnew.ugent.be

Processing Linked Data with Catmandu | http://librecat.org Architecture FEDORA MEDIAMOSA VLE BIBLIO SCANNING RECEIVE ALEPH ABS CLOUD DEDUP/MERGE/AUGMENT BLACKLIGHT

Processing Linked Data with Catmandu | http://librecat.org LINKED DATA

Processing Linked Data with Catmandu | http://librecat.org PRODUCTION CATALOG MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML LINKED DATA

Processing Linked Data with Catmandu | http://librecat.org STAGE 1: CATALOG to MARC CATALOG MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... $ catmandu export ALEPH to MARC

Processing Linked Data with Catmandu | http://librecat.org STAGE 2: MARC to JSON MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML

Processing Linked Data with Catmandu | http://librecat.org MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML STAGE 2: MARC to JSON Tolstoj, Lev Nikolaevič, Author War and peace / Title 1952. Publication Year Napoleonic Wars, Subject

Processing Linked Data with Catmandu | http://librecat.org STAGE 2: MARC to JSON MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML Tolstoj, Lev Nikolaevič, War and peace / 1952. Napoleonic Wars, Author Title Year Subject Tolstoj, Lev Nikolaevič War and peace 1952 Napoleonic Wars Author Title Year Subject FIX

Processing Linked Data with Catmandu | http://librecat.org STAGE 3a: JSON to RDF JSON/YAML LINKED DATA Tolstoj, Lev Nikolaevič War and peace 1952 Napoleonic Wars Author Title Year Subject ? Tolstoj, Lev Nikolaevič War and peace 1952 Napoleonic Wars dc:creator dc:title dc:date dc:subject FIX

Processing Linked Data with Catmandu | http://librecat.org JSON/YAML LINKED DATA STAGE 3a: JSON to RDF <http://example.org/000000008> <http://purl.org/dc/elements/1.1/creator> “Tolstoj, Lev Nikolaevič”; <http://purl.org/dc/elements/1.1/title> “War and peace” ; <http://purl.org/dc/elements/1.1/date> “1952” ; <http://purl.org/dc/elements/1.1/subject> “Napoleonic Wars” ; a <http://www.europeana.eu/schemas/edm/Book> . FIX RDF/Turtle http://demo.librecat.org/

Processing Linked Data with Catmandu | http://librecat.org STAGE 3b: RDF to Linked Data JSON/YAML LINKED DATA <http://example.org/000000008> <http://purl.org/dc/elements/1.1/creator> “Tolstoj, Lev Nikolaevič”; <http://purl.org/dc/elements/1.1/title> “War and peace” ; <http://purl.org/dc/elements/1.1/date> “1952” ; <http://purl.org/dc/elements/1.1/subject> “Napoleonic Wars” ; a <http://www.europeana.eu/schemas/edm/Book> . <http://example.org/000000008> <http://purl.org/dc/elements/1.1/creator> <http://viaf.org/viaf/96987389>; <http://purl.org/dc/elements/1.1/title> “War and peace” ; <http://purl.org/dc/elements/1.1/date> “1952” ; <http://purl.org/dc/elements/1.1/subject> <http://dbpedia.org/page/Napoleonic_ Wars> ; a <http://www.europeana.eu/schemas/edm/Book> . FIX

Processing Linked Data with Catmandu | http://librecat.org THANK YOU Nicolas Steenlant Nicolas Franck Snorri Briem Jörgen Eriksson Maria Hedberg Dave Sheroman Friedrich Summann Najko Jahn Vitali Peil Petra Kohorst Christian Pietsch Mathias Lösch Johan RolschewskiJakob Voß UGENT LUND BIELEFELD GBV STAATSBIBLIOTHEK ZU BERLIN Wouter Willaert INUITS

Add a comment

Related presentations

Related pages

Processing Linked Data with Catmandu - Education

Introduction into library data processing with help of Catmandu http://librecat.org
Read more

Scaling data streams with Catmandu and Linked Data ...

... and best practices of library data processing. ... Catmandu imports and produces Linked Data. ... explore Catmandu and Linked Data Fragments and ...
Read more

Catmandu - LibreCat

... SPARQL and Linked Data Fragments. ... Internally all data processing by Catmandu is using a generic data format not unlike JSON. If one imports ...
Read more

LibreCat/Catmandu data processing toolkit

What is Catmandu? Catmandu is a command line tool to access and convert data from your digital library, research services or any other open data sets.
Read more

ELAG 2015 I Scaling data streams with Catmandu and Linked ...

Scaling data streams with Catmandu and Linked Data ... to make data stream processing ... Catmandu imports and produces Linked Data.
Read more

Tutorial: Catmandu – a (meta)data toolkit - TPDL 2015

Tutorial: Catmandu – a (meta)data ... The aim of this tutorial to generate linked data from ... export and transformation of data. With Catmandu we want ...
Read more

SPARQling Pig – Processing Linked Data with Pig Latin

SPARQling Pig – Processing Linked Data with Pig Latin Stefan Hagedorn 1, Katja Hose2, Kai-Uwe Sattler 1 Technische Universit¨at Ilmenau, Ilmenau, Germany
Read more

Data Processing | LinkedIn

... certain functions linked to the ... traditional ETL method be the best approach to the data? What type of data processing, we require on the data? ...
Read more

Processing Data | LinkedIn

Looking for job in Qualitative and Quantitative Data processing | Data analyst | Quantum | SPSS ... Data processing at GfK Custom Research North ...
Read more