Manifold v0.2.0 Released

Star
Watch

On behalf of the entire Manifold team, I’m super excited to announce the release of Manifold v0.2.0! The release is up on Github now, and we’ll be rolling it out to our staging site later today. This release contains a number of new features and bugfixes, listed below. For the full list of revisions and pull requests, please consult the changelog.

Remember, Manifold is open source and freely available to all who are interested. While we have plans to build docker containers and OS packages, we’ve also written up installation instructions for all you early adopters out there. Take it for a spin and let us know how it goes.

As always, we appreciate any feedback. To support our effort and motivate our development team, please take a moment to watch or star our repository on Github!

Features

  • All ingestion strategies can support a .zip or a directory #398
  • Backend users can edit text stylesheets #367
  • Manifold can ingest a single HTML file as a text #360
  • Manifold supports Node 6.10.x #343
  • Render a consistently styled confirmation box when leaving unsaved forms #309
  • Add keyboard inputs to all modal dialogs #302
  • Backend users can create, update, edit, and delete resource collections #281
  • Youtube and Vimeo resources import thumbnails from providers #278
  • React-router updated from 2.x to 4.x #276
  • Manifold renders sensible fallback fonts when Typekit is not enabled #274
  • Backend text ingestion user interface is more user friendly #273
  • Texts can be re-ingested #272
  • Readers can comment on resources #271
  • Backend users can configure a generic project avatar or upload a thumbnail #270
  • Readers can generate a citation (MLA, APA, Chicago, etc.) for the selected passage #269
  • Manifold utilizes Postgres JSONB field for storing text section content #258
  • Installation instructions are included in manifold/manifold-docs #245

Bugs

  • Markdown ingestion fails if there’s no stylesheet #399
  • HTML ingestion fails if there is no <style> tag in the document head. #397
  • If an API request triggers a fatal error (500 or 404), client app should return the correct status code. #396
  • Following page crashes when not logged in #391
  • Link to resource detail from activity feed is incorrect and 404s #380
  • Deleting a resource should trigger deletion of associated resource annotations #379
  • Project thumbnail is stretched on frontend #378
  • Removing text should destroy “TEXT_CREATED” event #374
  • Removing a resource annotation doesn’t remove it from margin until page reload. #373
  • Missing margin in backend project detail form #370
  • Project avatar thumbnail field is missing margin #369
  • Text in google doc ingestion is too small and not responsive #368
  • Manifold reader appears to apply default paragraph margin of 0 #366
  • Press logo doesn’t show in backend after its set. #365
  • Metadata header on project page should not appear if a project has no metadata. #364
  • Unable to update text title in backend #362
  • SSR mismatch when enabling download link in Slide.Caption #354
  • Empty collections slideshow should be adjusted or hidden #353
  • Inconsistent scale of social icons on project hero #341
  • Ingestion builder does not remove text sections during reingestion #338
  • ProjectDetail.Texts reordering isn’t working #336
  • Project images are converted to jpegs, don’t allow transparent backgrounds #335
  • Move annotation and resource fetching to refactored Section container #331
  • Improve responsive behavior of backend header delete/preview buttons #327
  • Improve responsive behavior of backend date field #326
  • Entering an invalid project or text URL should 404 #319
  • Updating metadata on texts (in backend) doesn’t save #317
  • Managing contributors on texts seems not to work. #316
  • Style issues on users list #313
  • Investigate why Manifold can’t ingest this google doc #311
  • Typekit fonts should load (if configured) when SSR is absent #310
  • Footer text nav has incorrect top margin #308
  • Tags in dropdowns should be sorted alphabetically #307
  • Component FetchData methods should not be triggered by URL hash changes #304
  • Vertical alignment of mobile backend secondary nav is off by a few pixels #303
  • Model filter loses scope after validating paginated results #299
  • Variance in backend resource icon size #294
  • Backend project nav clips on smaller screens #293
  • Modal dialog buttons are missing a hover state #291
  • Changing pages in resources list breaks filters #288
  • Collection resource count shows all resources in project #287
  • Visibility nav overlaps TOC in reader (on mobile screen) #284
  • Form upload component does not remove attachment #266
  • Improve scroll behavior on route change #264
  • Filtering resource collections doesn’t update URL params #257
  • Resource preview doesn’t update when resource cube is highlighted in Firefox #256
  • Remove excess left margin in backend entity header #254
  • Add &rel=0 to Youtube video URLs to prevent related videos from displaying #253
  • Marginal resource thumbnails aren’t displaying until hovering over the icon #252
  • Backend resource search input submit does not preventDefault #251
  • Issues parsing Manifold-Docs TOC #248
  • Ingesting Markdown Files fails if the sources include a zero k file. #247
  • CSS Validator throws an exception with invalid CSS #246

This Week in Manifold: Project Detail Improvements

Welcome to a new regular feature on the Building Manifold blog: “This Week in Manifold.” As we move toward a beta version of Manifold (scheduled for March, 2017), I’ll be releasing a Manifold build to our staging server every Friday afternoon. Each release will be accompanied by a blog post explaining what the development team accomplished in the sprint leading up to the release. This post comes a few days late. When the Internet broke last week, our ability to release builds from Github was impacted, which delayed things a bit. I expect we’ll be back on schedule this week, with our next build going out on Friday, October 28.

Nota Bene: Our staging instance of Manifold is currently on a private server. However, we’ll be opening it up to the public in a few weeks, as soon as we’re able to load an initial set of texts with open access rights onto the platform. Once that happens, it will be possible for readers of this blog to check out our weekly builds for themselves.

Our last development sprint focused on tightening up existing code, mainly around the project detail view. Notable changes included the following:

Refactoring

  • Refactored login overlay CSS for improved reusability
  • Refactored the authentication reducer
  • Significantly reorganized client frontend components for easier readability and reusability

Features and Improvements

  • Exposed “published” boolean attribute on Texts in the API
  • Stubbed markup and styles for the resource collection detail component
  • Stubbed markup and styles for the collection list component
  • Stubbed markup and styles for the resource list component
  • Improved client application exception handling
  • Upgraded React to 15.4.0-RC3
  • Implemented project importer that imports projects, texts, and resources from json manifest
  • Expanded project API to expose all data shown in the comps
  • Adjusted component to show project creator names below project thumbnails
  • Adjusted component to show actual project dates below project thumbnails
  • Stubbed markup and styles for project events (on project detail, and stand-alone), aka activity
  • Added mobile styles for event lists
  • Added metadata JSONB field type to projects and expose via API
  • Implemented actual project metadata in the project detail component

Tooling and Libraries

  • Switched Manifold client testing framework to Jest
  • Updated all client NPM modules
  • DRY’d up client application Webpack configuration
  • Reorganized and improved build and server scripts for client application
  • Improved API application Puma configuration
  • Switched client from NPM to Yarn for deterministic dependency resolution (and speed)
  • Removed redux-devtools component; replace with similar Chrome extension
  • Added moment module for handling dates and times
  • Updated global and API Rubies to version 2.3.1

Bug Fixes

  • Corrected login overlay view state
  • Removed randomness from stubbed resource components (breaks universal rendering)
  • Added code to prevent client application from requesting “page” entities on every route change

In addition to these changes, the Manifold team also touched base last week during our recurring (bi-monthly) meeting. The group discussed text and project metadata requirements, the overall project roadmap, the agenda for our in-person Portland meeting in mid-November, among other things. Joining that call for the first time was JoJo Karlin, a Doctoral student in English at the CUNY Graduate Center who is working with Matt Gold on this project. You’ll be hearing more from JoJo in the coming weeks as she works to share Manifold updates and progress via this blog.

This week’s development sprint will be focused on a few main goals. Naomi will be finishing up markup and UX for the resource collection slideshow. Zach will be focused on the initial modeling of project resources and project resource collections in the API, and wiring that data up to the client. He’ll also be improving the project importer—delivered in last week’s release—to pull in sample resources for our demonstration projects and texts.

All work and no play makes for dull developers, and so I will leave you with a screenshot of Manifold’s new error screen (only visible in development mode), inspired by the first computer Zach ever owned, the illustrious Commodore 64.

screen-shot-2016-10-24-at-6-33-08-am

 

A Technical Introduction to Manifold

Hello Manifoldians!

I’m Zach Davis, principle at Cast Iron Coding, a web development studio in Portland, Oregon. My team and I are responsible for UX design, visual design, and development of Manifold over the course of the grant, and it is my hope that I’ll be able to post regular updates here detailing development and design decisions, challenges, and milestones over the course of the project.

For now, however, I’d like to walk through the technology stack we’ve settled on, and the reasoning behind some of these choices. If anyone has any questions, please don’t hesitate to post a comment!

Where to find the source code

As you know, one of the requirements of the grant funding this development is that Manifold be released as open source, and our current plan is to release Manifold under the GPLv3. For the time being, Manifold source code is stored in the ManifoldScholar/manifold Github repository. If you look at the ManifoldScholar GitHub organization, you’ll see additional repositories for Manifold-API and Manifold-Client. Eventually, we hope to be able to provide the Manifold API as a stand-alone piece of software that can be used to parse, store, and expose ePub and Markdown documents over a RESTful API. In preparation for this, we’re using a git subtree-split strategy to break the Manifold project into separate API and Client packages. These split repositories are updated significantly less frequently than the main Manifold repository. If your goal is to follow our development, for now your best bet will be to just watch the primary repository.

We’ve already made significant progress on Manifold, although we’re still far from an alpha release. We’d love it if you’d show your support for our project by starring our GitHub repo and keeping an eye on development. As we get closer to a stable alpha release in 2016, we’ll invite community members to download and install their own instance of Manifold.

Server-side architecture

Manifold is split into the client application, and the server-side application. For now, these two applications share a repository to simplify development, but they are very distinct and meant to run independently of each other.

The server-side application is built in Ruby and Rails. Specifically, we’re using what is currently edge Rails, and what will soon be Rails 5. The development team thought long and hard about what server-side language to use, which was mainly a debate about the relative merits of using Ruby or Javascript on the server. While we have many years of experience working with PHP, and understand that it is still a popular, easy choice among academic developers, we felt strongly that the advantages of Ruby—strong OOP support, support for more functional styles, cleaner and more consistent syntax, and the maturity of Rails—outweighed the disadvantages of a more complex execution model and smaller (but by no means small) developer community. Laravel, the current leading PHP framework, for all its notable strengths, lacks the completeness and maturity of Rails and Active Record. Our choice to go with edge rails, rather than stable Rails 4.x, was driven by the fact that our server-side component is only serving an API, and Rails 5 will eventually ship with an api-only flag. This means that many aspects of Rails—the view layer, sprockets, etc—have been removed, making for a more lightweight framework overall. Another strong reason for choosing Rails was the presence of a strong HTML/XML parsing library in Nokogiri. Manifold deals heavily (but not exclusively) in EPUBs, which are HTML documents. To pull those documents apart and recreate them in the browser, we need a powerful parsing tool, which we’ve found in Nokogiri.

The server side application exposes a REST style API to be consumed by the client. This API conforms to the style and conventions laid out by Steve Klabnik, Yehuda Katz, Dan Gebhardt, Tyler Kellen, and Ethan Resnick in the JSON API Specification. While it’s important that developers expose all the things with APIs, it’s challenging when each API follows its own idiosyncratic conventions. One of our goals on this project is to make Manifold as easy as possible to integrate into a presses existing infrastructure and environment, and following an established API specification helps us with that goal. Because the Manifold client application consumed Manifold’s public API, integrators can be sure that the API is well tested and reliable.

Authentication against the API will be handled using JSON Web Tokens (JWT). It will also be possible for a user to generate an API token by logging in through an OAuth provider such as Twitter or Facebook.

Client-side architecture

If we made a safe, conservative choice in Rails on the server-side, we’ve taken more of a leap on the client-side. The Manifold client is a single page application (SPA) that is built on React and Redux. Both are relatively new Javascript libraries (or frameworks, depending on how you look at things) that are rapidly gaining mindshare and popularity among front-end developers. For those who are unfamiliar, React is a Javascript library that simplifies the process of creating reusable components. Unlike other popular libraries such as Angular or Knockout, React does not attempt to solve the problem of two-way data binding. Instead, it is meant to be used with a specific architecture called Flux, which is structured around one-way data flow and immutability. Both React and Redux have a relatively small surface area. Redux APIs are small, and the library as a whole is under 15k.

At Cast Iron, we’ve worked with many Javascript frameworks, in particular Angular and Backbone. Over time, client-side development has become quite complex, and it’s our feeling that while Angular can work well on some projects, it doesn’t do enough to reduce that complexity. By restricting the way in which data can flow through the application, and abstracting away the asynchronicity that can make client-side development challenging, we’ve found Redux and React to be a powerful combination that allows us to iterate quickly without sacrificing quality or maintainability.

There are some challenges that come with developing a single page javascript application. For our purposes, the biggest challenge is to ensure that there is content loaded on the page before the javascript executes. We feel strongly that Manifold needs to be accessible to users with disability, to search engine crawlers, to screen readers, etc, and that we can’t count on a javascript environment. For this reason, the Manifold client will be served by a small Node/Express application running on the server. Most of the javascript that executes in the browser is built to be isomorphic (now also known as universal), which means that care has been taken to ensure that it can be executed in a server-side context (node) and in a client-side context (the browser). The practical outcome of this approach is that the server-side component will be able to fully render the HTML before sending the page to the client, which solves the many accessibility and SEO problems.

The advantages to the SPA approach are, well, manifold. The application will feel very fast, and we’ll be vastly reducing the amount of data that’s sent over the network. Complex UI elements, like the annotation pieces, will be have a much more stable, testable framework backing them. There will be more room for reusable components, and the various state-changes that can occur in the client-side application will be well tested.

While we’re diving in head-first on React + Redux, we are not, for now, following the current trend of doing only inline-CSS. For the time being, we’ll be using the SASS preprocessor for all styles.

At the language level, we’re relying heavily on ES6 and upcoming ES7 features. We’re using Babel to transpile ES6+ code down to ES5  so that it will run in IE9+, Firefox, and Safari. By using edge Javascript language features, our hope is that to some degree we will future proof Manifold code. For now, we’re limiting ourselves to ES6+ features that are at stage 1 or greater, and acknowledge that there is some risk even with stage 1.

We rely heavily on NPM for package management, and Webpack for transpiling assets during development, and for building assets for production. At Cast Iron, we’ve used Grunt and Gulp on other projects, but it looks to us like the React community is gravitating toward Webpack, and for good reasons. On Manifold, it’s being used for automatically linting our code, for generating icon fonts from SVG icons, for transpiling ES6+ to ES5, and Sass to CSS.

Testing

Tests are important. They build confidence and help us to refactor without regressions. While we don’t always follow test-driven-development practices, we do feel that it’s important to have a robust test suite in place. On the API side, we’re using rspec for tests. On the client-side we will be using Mocha.

Server infrastructure and continuous integration

DevOps and infrastructure automation are essential to our workflow at Cast Iron. We use Boxen to setup development instances of projects, and we use Puppet to deploy applications to staging and production environments. Our manifests for deploying Manifold are still in progress, but they’re close. Soon, I hope to have a staging environment that’s updated regularly via Jenkins, our continuous integration server. Soon, we’ll also have Travis build notifications in place on the Github repository.

To run Manifold, you’ll need to have the following setup on a server:

  • Nginx for serving the Ruby app and the Node app
  • Puma for serving Manifold API (although you could use Unicorn or another app server)
  • Ruby 2.2.x
  • Postgres for the API database
  • A recent version of Node
  • Eventually, I expect you’ll need Elastic Search for the search component

We’ll be providing instructions on how to set this up, as well as sample config files. Eventually, I hope to provide apt-get packages for Ubuntu as well. Once things are a bit more stable, we’ll regularly release docker containers that you can use to take Manifold for a spin.

In Closing

Hopefully this gives you a better sense of what we’re using to build Manifold, and why. I’d love to have a conversation about our choices, so please don’t hesitate to leave feedback in the comments. In the coming weeks, I’m going to try to share some early artifacts from the UX and visual design process, to give you a better sense of what Manifold will look like and what it will be capable of.