Islandora CLAW

Latest CLAW News

7.x-2.x -- Sprint 001 planning

A new Islandora Community sprint is coming up November 2 - 14. In September we had great success working on maintenance tickets for Islandora 7.x-1.x. The focus this time will be Islandora 7.x-2.x and the future of the stack.

The goal for the sprint is to get sprinters up to speed on the project through knowledge sharing, project workflows, and documentation. It may be the case that little to no code is written during this two week sprint, and that is perfectly ok! We hope that this will be the first sprint of many on Islandora 7.x-2.x!

You do not have to be a developer to join and we have a number of non-dev or "newbie" tickets lined up for you to work on if you want to take part.

The sprint will kick off with a meeting on November 2 where Danny Lamb and Nick Ruest will provide a brief overview of the new stack, go over issues marked for the sprint, and work with sprinters on what they would like to do or get out of the sprint.

If you’d like to familiarize yourself with sprint issues, please check out this list of issues.

If you’d like to join the sprint, please add your information to this spreadsheet.

You can build your own site to sprint on using vagrant, or download a VM here.

It would be great if we could also use this time to discuss some outstanding use-cases that will have architectural impacts. You don’t have to be a committer to take part in the discussions. All opinions are welcome. Please see the list of use cases here. Highest priority:

Fedora 4 Project Update V

Open Repositories 2015 has always been bright line on our project calendar. It's been a deadline for most of our project goals because that's when we wanted to show the community what their funding, use cases, discussions, and development has wrought. Well, OR2015 has come and gone and there is plenty of good news. First, the official goals of the Islandora 7.x-2.x project that have been completed:

Update Tuque
Tuque is the repository agnostic API that allows Islandora to interact with Fedora repositories. Tuque will need to be updated to connect to and work with the Fedora 4 APIs.
Islandora (core)
Islandora (core) is the Drupal module that allows for browsing and managing assets in a Fedora repository. Islandora (core) provides core hook and API functionality for all Islandora solution pack and utility modules
Islandora Solution Pack Collection
Islandora Solution Pack Collection allows Islandora to view to view and manipulate objects as a collection. It also allows collections to be created, viewable, and manageable. Along with Tuque and Islandora (core), it is a requirement for all other solution packs.
Replace FedoraGSearch
Fedora 4 provides a pluggable framework for an external triplestore and search index. The same JMS indexer can be used to communicate with both the triplestore and search index (e.g. Solr). Any triplestore that supports SPARQL updates can be used with Fedora 4. Fuseki and Sesame have been tested.

But there's more. Since the project launched in January, it has generated tools and achievements that we didn't directly predict at the outset. We presented these at OR2015 under the concept of "Atonement," because so many of the improvements in Islandora 7.x-2.x derive from a conscious effort to consider the flaws in the current stack and how we can start fresh and just plain do it better. These improvements include:

Community driven and lead - With some guidance and funding from the Islandora Foundation, Islandora7.x-2.x has been designed, built, tested, and documented by the Islandora Community as a whole. In particular, the efforts of the Fedora 4 Interest Group (an open group welcome to anyone who wants to participate) have been integral to shaping the project and continue to drive the work forward.
Documentation from the onset - Rather than writing up documentation as a last step when Islandora 7.x-2.x is up and running, it has been a part of the process right from the design phase, making it easier for contributors to step in and make their own modifications to the heavily-commented code while its still being developed.
Encouraging contributions - Hand in hand with the previous improvement, encouragement to contribute to the project has been a goal at all levels, from developers helping with the code to end-users providing use cases and requesting features. By drawing on the opinions of the community as a whole, and not just the developers among us, Islandora 7.x-2.x is made to reflect the needs of a broad group of users. Our call: “All contributions are welcome: use-cases, documentation, code, patches, bug reports, feature requests, etc. You do not need to be a programmer to speak up!”
DevOps - A vagrant build was created to fire up a working copy of Islandora 7.x-2.x quickly and easily. Every update is reflected in the vagrant build. Anybody can run the environment and work with the stack, whether that be testing or developing.
Everything is in a single Git repository!
Portland Common Data Model - At long last, a common data model between Hydra and Islandora. Lots of collaboration between the Islandora, Hydra, and Fedora communities for the betterment of all our projects. Our Project Director, Nick Ruest, is a committer for PCDM and an active member of the Hydra Metadata Working Group, working with the Hydra community on base recommendations for technical metadata (Nick and Aaron Coburn co-lead that sub-group), structural metadata, rights metadata, and descriptive metadata.
Working with the Fedora team - Contributing to Fedora development, attending Fedora Tech meetings, being a stakeholder for the audit service integration.
migration-utils - The community is going to need a way to get content from Fedora 3 into Fedora 4. Our team collaborated with Mike Durbin (University of Virginia) on creating a migration utility, and with just a handful of minor changes, it will be ready for use very soon. Testing is even underway to adapt this tool to bring content out of legacy Fedora 2 repositories.
Scalable, Distributed, Service Oriented Architecture - no assumptions on the location of any component of the stack.
Apache Camel - adopted as a routing and mediation engine for integrating Drupal and Fedora. The fcrepo-camel component provided by the Fedora community has been utilized to replace Tuque as our primary means of interaction.
True Drupal integration - Fedora content is modelled as nodes. Due to this, many modules in the codebase have been replaced with third party drupal.org modules. We can finally use that 'Add Content' button in Drupal!
Asynchronous communication Between Drupal and Fedora.
Bi-directional synchronization Between Drupal Fields and Fedora RDF.
Drupal Field to RDF mappings - A handy user interface allowing end users to control field -> RDF mappings.
Map XML with XPath - User interface controlling extraction of xml metadata using XPath, allowing fragments to get mapped to RDF. Similar to Hydra “Terminologies” except with a user interface.
RDFa Enriched HTML Output - Google will love your site!
Solr Solr Solr - The entire Solr suite of modules from drupal.org is available, giving the end user control over how and when content is indexed in Solr.

The next steps and goals for the project include:

Refactoring the current Apache Camel processing implementation to utilize command-line PHP. This should be more comfortable for the Islandora community. The goal is to lower the barrier to contribution.
Starting serious work to make the project easier to share. Tickets are coming, some marked for "newbies" to take on as an entry point to the project. The Basic Image Solution Pack will be the first target. Contributors are very welcome!
WebAccessControl integration - The Fedora community has begun planning WebAccessControl integration. If you'd love to see XACML go away, we'd love to have you as part of the stakeholder team!
Islandora 7.x-2.x community sprints coordinated with Fedora community sprints.

Fedora 4 Project Update IV

As the project entered the fourth month, work continued on migration planning and mapping, migration-utils, and Drupal integration.

Migration work was split between working on migration-utils, migration mappings, data modeling (furthering Portland Common Data Model compliance), and working with the Islandora (Fedora 4 Interest Group), Fedora (Fedora Tech meetings), and Hydra (Hydra Metadata Working Group) communities on the preceding items. In addition, Audit Service-- a key requirement of an Islandora community fcrepo3 -> fcrepo4 migration -- finalized the second phase of the project. Community stakeholders are currently reviewing and providing feedback.

Work on migration-utils focused mainly applying a number of mappings (outlined here) to the utility, adding support for object-to-object linking, and providing documentation on how to use the utility. This work can be demonstrated by building the Islandora 7.x-2.x Vagrant Box, cloning the migration-utils repository, and pointing migration-utils at a fcrepo3 native filesystem or directory of exported FOXML.

As for object modeling and inter-community work, an example of this work is the below image of a sample Islandora Large Image object modeled in the Portland Common Data Model. This model will continue to evolve as the communities work together in the various Hydra Metadata Working Group sub-working groups.

On the Drupal side of things, work was started on Middleware Services, a middleware service that will utilize the Fedora 4 REST API and the Drupal Services modules, and create an API for the majority of interactions between the two systems. In addition, a few Drupal modules have been created to leverage this; islandora_basic_image, islandora_collection, islandora_dcterms.

In addition, the team has been exploring options with RDF integration and support in Drupal, as well as how to handle editing (Islandora XML Forms) the various descriptive metadata schemas the community uses. This is captured in a few issues in the issue queue; #27 & #28. Due to the importance of the issue, a special Fedora 4 Interest Group meeting was held to discuss how to proceed with this functionality in Islandora 7.x-2.x. The group's consensus was to solicit use cases from the community to better understand how to proceed with 7.x-2.x

Work will continue on the migration and Drupal sides of the project into May.

The Future of Forms in Islandora

If you have been following events in the Islandora community lately, you probably know that we are in the midst of an ambitious project to build an Islandora that works on top of Fedora 4. The project is going great, but it has raised some questions about how to proceed with one of the most crucial yet under-appreciated tools in the Islandora stack: XML Formbuilder.

Largely the work of a single developer (discoverygarden's awesome Nigel Banks), XML Form Builder is a powerful (and sometimes difficult-to-master) tool that allows you to leverage the ease of Drupal forms to edit metadata for your Fedora objects. All Islandora Solution Packs come with standard MODS forms, but XML Form Builder lets you go far beyond those basic building blocks to meet almost any use case. A simple form for students to add objects with just a few fields; a brand new form to manipulate esoteric metadata standards; a tiny tweak to an existing Solution Pack form that brings it in line with your institution's specific needs.

The Fedora 4 project team recognizes that this functionality needs to exist in the future version of Islandora. The question now is: How? Do we port over XML Forms and untangle the legacy of compromises that allows it to work in the current environment? Do we sidestep the issue and build a new tool that more closely leverages the underlying structure of Fedora 4? Do we build a new tool and make it look like XML Form Builder so it's comfortable to use, but still works completely differently? XML, RDF, Xpath, etc. It's a big task and it comes with a lot of big questions.

So, we turn to you, the Islandora Community, to get some answers. What tools do you need to work with metadata for your collections? How are you using XML Form Builder now? What parts of it don't you use? What parts are critical? To that end, Nick Ruest has put together a template to collect use cases. Please add yours to the list and help us shape Islandora's future.

Fedora 4 Project Update III

The Fedora 4 upgration is coming into its third month with a big focus on migration. Notes from the last project meeting are available here. Some highlights:

Jared Whiklo, Web Application Developer at University of Manitoba, has joined the project team and is working with Danny and Nick on code tasks.

Recent work has focused on a couple of areas. The first was collaborating with Mike Durbin (University of Virginia) on fcrepo4-labs/migration-utils, which will support an upgration in this order:

traversing the objectStore file system
archive export format
migrate export format

In order to provide a large set of test fixtures for use with this tool, Nick updated YUDL’s Fedora to 3.8.1-SNAPSHOT.

The second focus on was data modelling. Specifically, mapping fcrepo3 object properties to fcrepo4 container properties, fcrepo3 datastream properties to fcrepo4 file properties, mapping RELS-EXT properties, identifying standard audit trail events, and working towards bringing Islandora into compliance with the Portland Common Data Model. This work was shared with the community via a Large Image Solution Pack example object modelled in Fedora 4.

Related to the migration, work has also been done around Audit Service design in Fedora 4. Nick participated in all of the Audit Service design meetings, and led a discussion around PROV and PREMIS ontology usage in the service. A code sprint led by Esme Coles and devoted to the Audit Service began on March 30th. That work is outlined here.

The migration work will most likely continue through April. If you want to attend future meetings and keep an eye on development, please join the Islandora Fedora 4 Interest Group. Your ideas and use cases are also very welcome as issues is Islandora Labs. For anyone planning to attend the Open Repositories conference in Indianapolis this June, the upgration team will be giving a presentation on the upgration project called Islandora and Fedora 4: The Atonement.