Manifold v0.1.1 Released

Version 0.1.1 of Manifold is a bugfix release. It has been pushed to the Manifold staging instance. This release includes the following fixes and improvements.

[B] Reset default text color in dark reading mode
[B] Only show FE mobile nav when logged in
[C] Add spec test for resource event creation
[B] Fix issue with broken link on first click
[F] Create event when resource is added to project
[B] Remove top spacing from section-heading-utility-right
[C] Adjust sizing of event tiles
[F] Add start reading link to new text events
[B] Fix long file names overflowing dropzone
[C] Use sub kind in slideshow to determine video type
[C] Refactor resources to use sub kinds for better determination
[B] Fire reader color change with 1 click (iOS)
[B] Fix issue with resource markers not highlighting
[B] Require library in formatted attr concern
[B] Fix responsiveness of pagination controls
[B] Use formatted text for applicable resource fields
[C] Add disabled state to slideshow arrows
[C] Hide social logins and T&C from create form
[C] Change label on project backend form field
[F] Store formatted attributes in Redis

As always, B = Bugfix, F = Feature, and C = Chore.

This Week in Manifold: Markdown and Backend Development

Since I last posted, we’ve been making development progress on a few different fronts. Much of this work is still in feature branches, and has yet to be merged into the main development branch. Therefore, I’m going to skip the full list of revisions in this post, and instead offer a high level description of our recent efforts.

Markdown and GitBook Support

Fundamentally, Manifold has been conceived as a tool for university presses. When it comes to thinking about how it will be used, UMNP has guided our thinking about what problems presses are trying to solve, and how Manifold can help. That said, I think everyone on the project has also been interested in the ways in which Manifold can be a useful tool for the broader academic community. At the very least, we want to make it possible for individuals to find ways to use Manifold. One impediment to this, in my mind, has been the fact that to date Manifold only accepts finished EPUB documents as an input format, and independent scholars and self-publishers may find it difficult to come up with finished EPUBs.

EPUBs are essentially packages containing HTML, CSS, and Javascript. In a sense, an EPUB is a small, self-contained website, and EPUB readers are offline, limited browsers. HTML, therefore, is already an input format that Manifold understands and can work with. What this means is that any document type that can easily be converted to HTML can likely be consumed by Manifold and made available to readers.

We refer to the process of slurping a book up into Manifold as “Ingestion.” Manifold can be extended by adding ingestion strategies to it. An ingestion strategy tells Manifold how to map a source document—eg, an EPUB—to a format it can understand. In short, we’ve anticipated that we’ll want to ingest other kinds of documents and have left places where other developers can create their own strategies. With the aim of testing out this approach and also furthering our goal of increasing Manifold’s user base, we spent a couple days writing a Markdown strategy that supports the GitBook format. This means that users will be able to feed Manifold a set of nested Markdown documents, which Manifold will then be able to make available online, just as it does an EPUB. Manifold’s documentation, for example, is authored following the GitBook format, and it can now be read within Manifold itself. Three cheers for eating our own dog food!

Backend Development

Since I last posted, we’ve also been hard at work at building out the backend interfaces. Most of the basic backend HTML and CSS has been completed, and we’re now in the process of developing reusable patterns and components in React to wire up the backend forms to the API. This has been challenging, as there is a lot of functionality required in Manifold’s admin interface, and figuring out how to model the requisite state while keeping in our Flux architecture has required some thinking. We’re close on this, with some initial proof of concept work, which I’ll be discussing more in subsequent posts.

That’s it for today! We’ll be posting more updates (hopefully with screenshots and some design comps) later this week.

 

 

This Week in Manifold: DevOps, Infrastructure, and Social Activity

Thanks for joining me for another installation of “This Week in Manifold.”

First off, please accept my apologies for missing last week’s update. When last Friday rolled around, we had a number of updates ready to go out to our staging site. However, when it was time to deploy, we hit a few unexpected problems.

DevOps, Hosting Manifold, and Deployment

This release adds a new metadata field to projects. Because the exact metadata requirements are difficult to pin down and may change from press to press, we wanted a relatively flexible approach to storing project, text, and resource metadata. Back in the day, we might have used an EAV model for this, but there are many well-documented drawbacks to that model. Because Manifold uses PostgreSQL as its database backend, we have access to a variety of field types for storing arbitrary amounts of data. In the past, we’ve used HSTORE for storing key/value pairs, but in this case we wanted something more flexible that could store different data types (HSTORE stores strings, but we want to store integers, boolean values, strings, arrays, etc). Recent versions of PostgreSQL include the JSONB field type, which is exactly what we want in this case.

Alas, that meant that the October 28th release required some infrastructure updates. Sure, we could have shelled into the staging server and done a quick upgrade, but that’s not how we roll on this project. It’s really important to us that it be possible to easily provision hosts for Manifold, which means automating as much as possible. As Manifold grows, it relies on more and more services to run, which makes setting it up increasingly complex. With this release, Manifold needs all of the following configured on the server to run correctly:

  • Ruby 2.3 (for the API)
  • Node 4.5 or higher (for the client)
  • Postgres 9.4 or higher (for storage)
  • Redis (for queuing background jobs)
  • ImageMagick or GraphicsMagick (for resizing images)
  • Nginx or Apache (for serving the API and the Client)

Moreover, in addition to having all these requirements in place, the server also requires that a number of background processes are always running. As of right now, there are four custom services (excluding the webserver and the database server) that need to be running on the host: 1) manifold_api (Rails app served by Puma), 2) manifold_client (Node app), 3) manifold_scheduler (Ruby daemon), and 4) manifold_workers (Sidekiq). Keeping four services running on a server (or on multiple servers) is not without its complexities. If we run these on Ubuntu Trusty (v14), for example, we need Upstart scripts that manage the services. If we run Manifold on Ubuntu Xenial (v16), then we’re going to need Systemd scripts. If it runs on older versions of Centos or Redhat, then we’ll probably need some init.d scripts.

Running a complex app with separate parts is tricky. Internally at Cast Iron, we use puppet and foreman to manage these services. Needing to support Postgres 9.4 got us thinking about deploying Manifold to Ubuntu Xenial, which in turn led us to revisit our manifests. Once we went to Xenial, we had to rewrite our startup scripts for systemd, which in turn revealed problems with how we were storing secrets in the environment. In short, instead of spending the last week working on project resources and feature development, we spent it paying down some technical debt on the infrastructure side, and improving Manifold’s devOps and stability.

That’s done now, and we’re deploying Manifold to Xenial and fully managing its environment and all dependencies via Puppet. Revisiting this infrastructure has also given us an opportunity to reflect further on Manifold’s installation story and continue discussions about how we can make Manifold easy for other presses to adopt and install.

New Features

We did manage to slip a few important features into this week’s release. The full list is below, but there’s one in particular I want to highlight.

This release includes a scheduler that runs constantly in the background while Manifold is running. One of the things this scheduler does is trigger a background job for fetching tweets related to projects in Manifold. That job currently runs hourly.

Editors can use twitter search terms to describe what kind of tweets Manifold should pull in for a given project. This gives editors quite a bit of flexibility. For example, Manifold might watch the author’s twitter account for all tweets with a specific hash tag. Or, it might watch twitter in general for all tweets that include a given hashtag. Our tweet fetching service accepts multiple search terms, so it’s possible for a project to follow multiple accounts and hashtags.

These tweets show up on the project activity feed. Readers can browse through the historical activity for a project and see the events—texts being added, tweets about the project, new resources uploaded—that took place as the project became a published book (see screenshot, below). Our hope is that showing this activity helps make public the iterative aspects of authorship.

Project Activity

Well, that’s enough for today. I’ll be back next week with another update. Enjoy the weekend!

Refactoring

  • Display project “makers” (authors, editors, etc) on one line
  • Improve header/icon lockup on project detail component
  • Refactor ingestion resources into a new model that is distinct from project resources
  • Clean up API binstubs

Features and Improvements

  • Add JSONB metadata field to projects; adjust project import mechanism to include metadata
  • Expose project metadata in API
  • Render project metadata in the client
  • Incorporate Sidekiq for background job processing
  • Add the concept of model ownership in various places. Manifold will now track who creates a project.
  • See a system-level user for ownership purposes
  • First pass at project events subsystem including background event generation; create Event factory class
  • Expose project events via the API
  • Render project events in the “activity” component on the project detail view
  • Add job for queuing the fetching of project tweets
  • Add job for fetching project tweets
  • Add a background scheduler (Clockwork) for triggering recurring background jobs

Tooling and Libraries

  • Update Postgres requirement to 9.4 or 9.5 for JSONB
  • Adjust Travis CI build to use Yarn instead of NPM
  • Feed Travis build notifications to Slack
  • Numerous revisions to deployment configuration

Bug Fixes

  • PropType correction in EventList component
  • Address 1:N queries in the project controller
  • Correct text icon on project texts list
  • Fix padding on coverless project heroes
  • Adjust font size and padding on project makers

 

This Week in Manifold: Project Detail Improvements

Welcome to a new regular feature on the Building Manifold blog: “This Week in Manifold.” As we move toward a beta version of Manifold (scheduled for March, 2017), I’ll be releasing a Manifold build to our staging server every Friday afternoon. Each release will be accompanied by a blog post explaining what the development team accomplished in the sprint leading up to the release. This post comes a few days late. When the Internet broke last week, our ability to release builds from Github was impacted, which delayed things a bit. I expect we’ll be back on schedule this week, with our next build going out on Friday, October 28.

Nota Bene: Our staging instance of Manifold is currently on a private server. However, we’ll be opening it up to the public in a few weeks, as soon as we’re able to load an initial set of texts with open access rights onto the platform. Once that happens, it will be possible for readers of this blog to check out our weekly builds for themselves.

Our last development sprint focused on tightening up existing code, mainly around the project detail view. Notable changes included the following:

Refactoring

  • Refactored login overlay CSS for improved reusability
  • Refactored the authentication reducer
  • Significantly reorganized client frontend components for easier readability and reusability

Features and Improvements

  • Exposed “published” boolean attribute on Texts in the API
  • Stubbed markup and styles for the resource collection detail component
  • Stubbed markup and styles for the collection list component
  • Stubbed markup and styles for the resource list component
  • Improved client application exception handling
  • Upgraded React to 15.4.0-RC3
  • Implemented project importer that imports projects, texts, and resources from json manifest
  • Expanded project API to expose all data shown in the comps
  • Adjusted component to show project creator names below project thumbnails
  • Adjusted component to show actual project dates below project thumbnails
  • Stubbed markup and styles for project events (on project detail, and stand-alone), aka activity
  • Added mobile styles for event lists
  • Added metadata JSONB field type to projects and expose via API
  • Implemented actual project metadata in the project detail component

Tooling and Libraries

  • Switched Manifold client testing framework to Jest
  • Updated all client NPM modules
  • DRY’d up client application Webpack configuration
  • Reorganized and improved build and server scripts for client application
  • Improved API application Puma configuration
  • Switched client from NPM to Yarn for deterministic dependency resolution (and speed)
  • Removed redux-devtools component; replace with similar Chrome extension
  • Added moment module for handling dates and times
  • Updated global and API Rubies to version 2.3.1

Bug Fixes

  • Corrected login overlay view state
  • Removed randomness from stubbed resource components (breaks universal rendering)
  • Added code to prevent client application from requesting “page” entities on every route change

In addition to these changes, the Manifold team also touched base last week during our recurring (bi-monthly) meeting. The group discussed text and project metadata requirements, the overall project roadmap, the agenda for our in-person Portland meeting in mid-November, among other things. Joining that call for the first time was JoJo Karlin, a Doctoral student in English at the CUNY Graduate Center who is working with Matt Gold on this project. You’ll be hearing more from JoJo in the coming weeks as she works to share Manifold updates and progress via this blog.

This week’s development sprint will be focused on a few main goals. Naomi will be finishing up markup and UX for the resource collection slideshow. Zach will be focused on the initial modeling of project resources and project resource collections in the API, and wiring that data up to the client. He’ll also be improving the project importer—delivered in last week’s release—to pull in sample resources for our demonstration projects and texts.

All work and no play makes for dull developers, and so I will leave you with a screenshot of Manifold’s new error screen (only visible in development mode), inspired by the first computer Zach ever owned, the illustrious Commodore 64.

screen-shot-2016-10-24-at-6-33-08-am