Manifold Beta Now Available

The Manifold team is delighted to launch a public beta of its new publishing platform for interactive scholarly monographs: http://staging.manifoldapp.org/.

browse-splash_manifold-devices-1

Funded through the generous support of the Andrew W. Mellon Foundation, Manifold is a collaboration between University of Minnesota Press, the GC Digital Scholarship Lab at the Graduate Center, CUNY, and Cast Iron Coding.

We began work on the project two years ago, aiming to create a responsive platform for interactive books that would help university presses share long-form monographs through an appealing and elegant interface. After many meetings and planning discussions, and following 1300+ commits to our public code repository, the initial version of the platform is ready for review.

On the beta site, you will find a selection of projects from the University of Minnesota Press that may be read, annotated, highlighted, and shared through social media. These include two recently published full-length scholarly books, a selection from the Forerunners: Ideas First series, and four projects just beginning to take shape on the platform:

Published Books

The Perversity of Things: Hugo Gernsback on Media, Tinkering, and Scientification by Hugo Gernsback. Edited by Grant Wythoff

Metagaming: Playing, Competing, Spectating, Cheating, Trading, Making, and Breaking Videogames by Stephanie Boluk and Patrick LeMieux

Forerunners

The Uberfication of the University by Gary Hall

The Anthrobscene by Jussi Parikka

No Speed Limit: Three Essays on Accelerationism by Steven Shaviro

Mandela’s Dark Years by Sharon Sliwinski

Dark Deleuze by Andrew Culp

The Geek’s Chihuahua: Living with Apple by Ian Bogost

Ten Theses for an Aesthetics of Politics by Davide Panagia

Deep Mapping the Media City by Shannon Mattern

New Projects

The Lab Book: Situated Practices in Media Studies by Lori Emerson, Jussi Parikka, and Darren Wershler

In/visible Archives of the 1980s: Feminist Politics and Queer Platforms by Margaret Galvan

Cut/Copy/Paste: Echoes of Little Gidding by Whitney Trettien

Going the Rounds: Virality in Nineteenth-Century American Newspapers by Ryan Cordell, David Smith, Abby Mullen, and Jonathan Fitzgerald

 

Each project has a homepage that presents an overview of the text, provides a quick link to the text, aggregates recent activity, showcases the evolution of the project, and shares resources—images, videos, files, PDFs, image collections—that have been added to the text. Images that were part of the print version will appear in-line; resources that have been added for the Manifold edition will appear to the left of the text. Texts are responsive and may be read on any device, though the mobile versions are not yet fully featured. Please note that while we will make an effort to retain annotations and highlights left on the beta platform, we cannot guarantee their preservation.

Though the beta version only includes Minnesota publications, the platform is being designed so that any press or interested scholar can install Manifold, customize the platform with specific colors and logos, and publish work through the administrative dashboard. Manifold is capable of ingesting a variety of formats—ePub, HTML, Google Docs, Markdown, Microsoft Word—immediately transforming them into interactive web publications.

Manifold is proud to be an open-source project. You can find our code on Github, report bugs on our Github issue tracker, check out our progress on our public Pivotal tracker, join our mailing list, read our Building Manifold blog, and follow us on twitter at @manifoldscholar.

For a longer introduction to the project, please read “Building Manifold,” by Project Co-PIs Doug Armato and Matthew K. Gold. To learn more about the technology behind the platform, please read “A Technical Introduction to Manifold” by Manifold Lead Developer Zach Davis.

We hope you enjoy Manifold and we look forward to your feedback!

Manifold v0.1.3 released to staging.manifoldapp.org

Version 0.1.3 of Manifold introduces support for logins using Facebook, Twitter, and Google OAuth services. This version also incorporates a refactored approach to managing application settings, which can now be set via the backend, or managed by way of the dotenv file, which tends to be a more appropriate solution for automated provisioning and deployment.

Fixed bugs:

  • Recent event CSS changes broke mobile event layout #241
  • Settings are loaded even when SSR is unavailable
  • Fix upload display in FF/Safari
  • Fix popup position in Firefox

Features:

  • Refactor integration setttings; move to DB
  • Add styled 50x error page
  • Refactor service port configuration for easier install

Closed issues:

  • Missing Favicon in production env. #242

Merged pull requests:

Manifold v0.1.1 Released

Version 0.1.1 of Manifold is a bugfix release. It has been pushed to the Manifold staging instance. This release includes the following fixes and improvements.

[B] Reset default text color in dark reading mode
[B] Only show FE mobile nav when logged in
[C] Add spec test for resource event creation
[B] Fix issue with broken link on first click
[F] Create event when resource is added to project
[B] Remove top spacing from section-heading-utility-right
[C] Adjust sizing of event tiles
[F] Add start reading link to new text events
[B] Fix long file names overflowing dropzone
[C] Use sub kind in slideshow to determine video type
[C] Refactor resources to use sub kinds for better determination
[B] Fire reader color change with 1 click (iOS)
[B] Fix issue with resource markers not highlighting
[B] Require library in formatted attr concern
[B] Fix responsiveness of pagination controls
[B] Use formatted text for applicable resource fields
[C] Add disabled state to slideshow arrows
[C] Hide social logins and T&C from create form
[C] Change label on project backend form field
[F] Store formatted attributes in Redis

As always, B = Bugfix, F = Feature, and C = Chore.

Manifold Preview!

Manifold is excited to announce the public launch of our in-progress demo next Tuesday, April 4th! In advance of that launch, we’re happy to offer a brief preview of some of the features of this intuitive, collaborative publishing platform for scholars, made possible by a generous grant from the Andrew W. Mellon Foundation. Manifold provides an online, mobile-ready interface for reading and responding to texts. The demo version will be populated by University of Minnesota Press books and projects, but any press or other organization can install the open-source platform and upload their own texts for interactive reading and annotation. For technical details about the build, check out posts by our lead developer, Zach Davis of Cast Iron Coding.

Read on for your first sneak peek!

browse-splash_manifold-devices-1

Manifold

The Manifold homepage shows the Press’s library — the books and projects that it is offering through Manifold. Featured projects (those you have selected) stand beside the rest of the available titles, which you can browse or sort by category.

screen-shot-2017-03-30-at-4-41-00-pm

Projects

Manifold organizes around the idea of projects — a project includes both the text of the book and the array of materials the author chooses to publish with it. In addition to providing access to the reading interface and allowing readers to add projects to their reading lists, each project page links you to the Press’s print book, the project’s social media, and supplemental material.

Reading

The interface is made for readability and usability. You can change colors for higher-contrast, and adjust the margins of the text:

typograhy-view_optimized

Or adjust the typography.

typography-font_optimized

 

While reading a text, you also have the option to access supplementary materials and annotate both the text and ancillary materials, as well as add comments to other reader’s annotations that haven’t been made private. When you select text, a menu pops up with options for highlighting, annotation, citing, and sharing.

Annotating

You have the opportunity to comment on text in the reader interface right in your browser. While designing the platform, the Manifold team considered the expectations scholars have of digital texts. Blending personal reading habits, scholars’ wishlists, and successful aspects of comments in projects like the previous collaboration between the CUNY Graduate Center, Cast Iron Coding, the University of Minnesota Press — Debates in the Digital Humanities the team developed a user-centered annotation system. Moving beyond the sentence-level annotation of Debates in the Digital Humanities, Manifold lets you leave comments on any selection of text.

annotate02-optimized

 

Highlighting

As you read, you have the option to highlight passages with your cursor.highlight01-optimized

Sharing

Manifold also helps you share passages from the text via social media. Choosing “Share” from the dropdown links the passage in a post to your Facebook or Twitter accounts.

 

 share01-edited

These are only a few of the features of Manifold — get ready for filtering by categories, favorites, search, and more. We look forward to sharing the fruits of collaborative effort, thoughtful design, and hard work. 

Please check it out for yourself next Tuesday, April 4th! Stay tuned to the Manifold blog, sign up for our mailing list, and catch updates @ManifoldScholar.

 

 

Reading with Matthew K. Gold

mkgold-gc-15Before the holidays, I had the chance to chat with Manifold Co-PI Matthew K. Gold, who is Associate Professor of English and Digital Humanities and Executive Officer of the M.A. Program in Liberal Studies at the Graduate Center, CUNY. I have been lucky to work with Matt for a couple of years now, through seminars on digital humanities praxis and textual studies, on DH tool-building projects like DH Box, on the editorial collective of the Journal of Interactive Technology and Pedagogy, and with the digital fellows of the GC Digital Initiatives. His widespread commitment to textual scholarship and digital innovation, his seemingly tireless support of his students and digital experimentation, and his humility (which he may not even let me mention) never cease to amaze me. I have spent a lot of time discussing Manifold with Matt, but I was glad to get the chance to ask him more specifically about his personal motivations for developing a hybrid publishing platform. 

What matters most to you about Manifold? I think it’s the overall excellence of the user experience. Our goal is to create a platform that will rival the reading experience of a commercial site like Medium but that is free and open-source. I’m excited to see how various people will use it — certainly our primary audience of university presses, but also a wider set of authors and publishers of all types.

Cool. That sounds like a pretty noble mission to me. I’m also excited about our team — we have the University of Minnesota Press, which is doing exciting experimentation on both its editorial and production sides —  it’s a Press that’s willing to take risks with the way it thinks about the future of publishing. Working with a partner like that — and with Doug Armato and his team in particular — is a lot of fun.. And then we are lucky to be collaborating with Cast Iron Coding– Zach Davis and his team are extremely talented and very much aware of the latest trends in web development. Manifold won’t feel like a typical piece of academic technology, thanks to the work of Zach and his team.

What’s your ideal reading experience? For me, print is still most important as a reading experience. But I’m never satisfied when I buy a printed book from a bookstore and then “that’s it.” I can’t carry all of my books with me when I go on a trip, so I really want both print and digital at the point of purchase. For me, if I’m going to read a book of theory or criticism, I’m going to be reading it in print, but I also want to be able to easily pull it up on my laptop or phone when I’m traveling or just happen to think of it during a meeting with a student, or I’m doing some writing and I want to refer to it. I want the materiality of the print book, but for access and reference, the online version.

And Manifold is precisely about achieving that hybrid experience. So we are producing print books, but we are also creating web-based interactive texts that open up new collective and social reading experiences.

What are you reading now? I am reading (and loving) Barbarian Days by William Finnegan. Somehow, reading about waves feels like the right thing in this post-election moment–an escape from politics but also a reminder of how to make our way among larger and sometimes brutal forces.

Who inspires your work? I tend to be inspired by my academic mentors. My first academic mentor was Robert D. Richardson, Jr., who is a biographer of Thoreau and Emerson (his biography of Emerson remains one of my favorite books). I’m still in awe of his erudition and research and writing skills. I’m also inspired on a daily level by my colleagues at the Graduate Center and across CUNY, especially by the students I work with.

After Matt answered all the questions, we had the chance to speak more broadly about Manifold and the features that specifically address readers’ needs. The design of the platform — from the icons, to commenting and annotation, to the reading interface — is the result of careful consideration by a team of careful readers. Moving forward, even as I share more about the creators of Manifold and the reading practices that influence its development, I will be adding a features series to outline the thinking that goes into the various components of Manifold.

Reading with Zach Davis

Zach Davis

Zach Davis

Last week, calling Portland from the Digital Scholarship Lab at the Graduate Center, I had the pleasure of speaking with Manifold’s lead developer, Zach Davis at Cast Iron Coding. Yet again, my attempts to record the call came to naught, but I will conjure some of the conversation from my notes.

Zach is Principal Chief Technologist at Cast Iron Coding. His previous CIC projects include sites for the Hammer Museum in Los Angeles, The Edna McConnell Clark Foundation, the Jersey Give-Back Guide (check out the animation on the “generosity generator”) and Debates in Digital Humanities, a collaboration with The University of Minnesota Press and Matt Gold that served as a prototype for the Manifold project.

Zach has been working steadily and seems rejuvenated by the team’s recent Portland meeting. Talking through the next steps and considerations about how university presses will be able to customize Manifold installations seems to have crystallized the work that everyone has been doing in exciting ways.

First, I asked Zach what matters most to him about Manifold. He noted the two most important things to him at this stage, made even more clear after the meeting.

  1. That Manifold is open and open source. Zach says, “it is a pleasure to work on software that has to be open source, where it’s written into the very project.”
  2. That they are building something that solves a problem for a good number of people. He recognizes the audience, but beyond Manifold’s immediate uses for the press, the software has the potential to serve an even wider segment.

After that we spoke about ideal reading experiences. While his habits have changed since the days when he was a Ph.D. student in English at the CUNY Graduate Center, Zach says it’s “still a book most of the time. It depends on what kind of text. I don’t read theory and criticism the way I used to. For short forms, I will read on the web or my phone, but for reading, I still like the physical artifact.” We talked a bit about the different demands of different types of reading. When he was working on his dissertation, Zach categorized himself as a transcriber. He would cite passages as a way of thinking through the content. Now while working on Manifold, he imagines the sort of tools that would have facilitated this sort of reading. He says he also spent much of his academic life reading digital texts, which were often easier to track down. He thinks about the ways Manifold could simplify the lives of scholars.

What is he reading right now?

  1. A collection of Leonard Cohen poetry
  2. Margaret Atwood’s The Handmaid’s Tale

Both seem rather timely.

Lastly, I asked who inspires his work. Zach says lately he has been particularly inspired by the work of Dan Abramov, a JavaScript developer. He finds inspiration in Abramov’s career and particularly appreciates Abramov’s attitude about open source software and towards the community around his work. Now that he has transitioned to his developer role, he finds his book-world inspirations are much the same as they were ten years ago. He says he hopes he in any way resembles the thoughtful, careful, kind approach to work and students that he saw in Milton scholar and mentor Joseph Wittreich at the Graduate Center.

As the project moves on, I’m interested in the ways the underlying experience of the Manifold team influences the way they craft the reading experience.

Thanks, Zach!

Follow @zdavis

This Week in Manifold: Markdown and Backend Development

Since I last posted, we’ve been making development progress on a few different fronts. Much of this work is still in feature branches, and has yet to be merged into the main development branch. Therefore, I’m going to skip the full list of revisions in this post, and instead offer a high level description of our recent efforts.

Markdown and GitBook Support

Fundamentally, Manifold has been conceived as a tool for university presses. When it comes to thinking about how it will be used, UMNP has guided our thinking about what problems presses are trying to solve, and how Manifold can help. That said, I think everyone on the project has also been interested in the ways in which Manifold can be a useful tool for the broader academic community. At the very least, we want to make it possible for individuals to find ways to use Manifold. One impediment to this, in my mind, has been the fact that to date Manifold only accepts finished EPUB documents as an input format, and independent scholars and self-publishers may find it difficult to come up with finished EPUBs.

EPUBs are essentially packages containing HTML, CSS, and Javascript. In a sense, an EPUB is a small, self-contained website, and EPUB readers are offline, limited browsers. HTML, therefore, is already an input format that Manifold understands and can work with. What this means is that any document type that can easily be converted to HTML can likely be consumed by Manifold and made available to readers.

We refer to the process of slurping a book up into Manifold as “Ingestion.” Manifold can be extended by adding ingestion strategies to it. An ingestion strategy tells Manifold how to map a source document—eg, an EPUB—to a format it can understand. In short, we’ve anticipated that we’ll want to ingest other kinds of documents and have left places where other developers can create their own strategies. With the aim of testing out this approach and also furthering our goal of increasing Manifold’s user base, we spent a couple days writing a Markdown strategy that supports the GitBook format. This means that users will be able to feed Manifold a set of nested Markdown documents, which Manifold will then be able to make available online, just as it does an EPUB. Manifold’s documentation, for example, is authored following the GitBook format, and it can now be read within Manifold itself. Three cheers for eating our own dog food!

Backend Development

Since I last posted, we’ve also been hard at work at building out the backend interfaces. Most of the basic backend HTML and CSS has been completed, and we’re now in the process of developing reusable patterns and components in React to wire up the backend forms to the API. This has been challenging, as there is a lot of functionality required in Manifold’s admin interface, and figuring out how to model the requisite state while keeping in our Flux architecture has required some thinking. We’re close on this, with some initial proof of concept work, which I’ll be discussing more in subsequent posts.

That’s it for today! We’ll be posting more updates (hopefully with screenshots and some design comps) later this week.

 

 

This Week in Manifold: Team meeting, Numerous Fixes

This has been an exciting week for Manifold. The UMNP team—Terence, Dan, Susan, and Doug—and Matt all came out to Portland for one of our regular meetings. We met at the Cast Iron Coding office in the old Washington High School building and spent two days talking through issues, thinking about Manifold use cases, sorted through thorny metadata and DOI concerns, and generally reviewed how far we’ve come, where we’re at, and what’s left to do. I always find these meetings inspiring and invigorating, and am reminded of what a strong team we have in place for this project.

In the two weeks leading up to this meeting, the development and design team at Cast Iron has been working furiously to tighten up existing behavior and progress on the project resources. As a result, we don’t have a lot in the way of big new features to report for this sprint. We do, however, have a long list of fixes, small improvements, and tweaks, which I’m including below. Next week is a short week, and I likely won’t post an update until the following week. In the next sprint, we’ll be finalizing project resources, and then turning our attention back to the annotation user interface, and begin work on comment threads.

The release that went out to staging today includes the following revisions.

Backend

  • Add static markup/styles for backend text selector
  • Wire projects from API to backend views
  • Wire backend dashboard to project list
  • Add static upload form to backend
  • Add backend statics: maker select, textarea
  • Add radio group component to backend
  • Add static componet for backend panel/nav
  • Add static header to backend project detail
  • Add static component for backend activity
  • Add static for backend notifications
  • Add backend icons, secondary header style
  • Add static component for backend projects
  • Add placeholder avatar to makers

New Features

  • Add resource detail view
  • Slideshow loads resources one page at a time (batch loading)
  • Client can reduce one fetch to two locations in global state

UI Improvements

  • Close panels on click
  • Add graphic and messaging for users not following projects
  • Improve resource-related back links
  • Add marketing banner with signup button
  • Reduce input and button borders
  • Show actual annotation counts in text list
  • Refine resource handling on client
  • Animate project grid enter/leave (in some cases)
  • Improve following widget and project filtering
  • Mobilize mobile reader footer/appearance menu
  • Adjust reader header for mobile
  • Adjust positioning of user menu on mobile
  • Adjust resource collection icon on small sizes
  • Convert notification markup to use new styles
  • Add responsive font-styles to reader
  • Update twitter icon position on events

Bug fixes

  • Remove slide listeners on unmount
  • Correct resource detail link
  • Fix inconsistent text-header spacing
  • Adjust overlay header to match browse header
  • Fix svg grid padding on iOS
  • Prevent resource from showing blank slide
  • Fix resource link instances in card
  • Correct regression to collection detail
  • Remove underline from text title
  • Refactor resource scss files add dark overlay
  • Correct resource totals language
  • Show cards on collection resource list
  • Fix overlay on resource thumbnails
  • Fix slideshow transitions for demonstration
  • Prevent project subtitle wrapping on mobile
  • Fix text title weights on mobile
  • Restrict activity list on mobile
  • Fix sizing/wrapping on project metadata
  • Do not add resource to blank collection
  • Remove errant console.log message
  • Correct following transition style
  • Update user after edit profile
  • Correct home page header
  • Fix positioning and padding on header notifications
  • Scroll to top when changing sections
  • Correct annotation create
  • Adjust margin/positioning on thumbnails
  • Update primary nav for mobile
  • Fix duplicate project create events
  • Fix annotation popup position regression
  • Address improperly named project import method

API

  • Expose single collectionResource
  • Expose text age
  • Expose created month, day, and year on texts
  • Expose CollectionResources; add acts_as_list
  • Expose text annotation counts
  • Expose resource kinds on project
  • Move controllers to align with JSON API spec
  • Expose uncollected project resources in API
  • Expose user project favorites via API
  • Implement project subjects; adjust filtering
  • Ensure sources are associated with texts
  • Import resources when importing projects
  • Adjust resource serializers
  • Add collection controller to API
  • Generate resource thumbnail if it is an image
  • Expose date and kind data on collection
  • Expose project association counts
  • Add resource importer; build resource models
  • DRY up attachment validation; organize models

Misc

  • Refactor authentication; improve API actions
  • Refine frontend resource components
  • Correct resource and collection migrations
  • Clean up initializers

 

Reading with Susan Doerr

While the seismic ruptures of the election have unsettled the ground beneath us, it feels necessary to continue working on projects that look beyond the next four years, that address the possibility of connecting people through ideas and through constructive debate. Last week, in the midst of national turmoil, I had the pleasure of interviewing Susan Doerr, University of Minnesota Press Assistant Director, Digital Publishing and Operations. During our phone call, I asked Susan a number of questions about her thoughts on reading and the Manifold platform.

I started by asking Susan what matters most to her about Manifold. She talked about the process of collaborating with the Manifold team on writing the grant proposal, and thinking about publishing books in the browser. Susan believes the real revolution of an epub or pdf ebook is the speed of distribution — instant access for a reader. Susan said, “ebook reading devices are siloed — a reader is stuck in an app or device.” What the Manifold team wanted to do was “break free from that constraint. Ebooks today are replications of print — active media that don’t do much more than the print edition of a book. And we wanted to do more.”

Publishing a book in the browser allows it to be dynamic — vs static. The more they thought about it, the more opportunity they saw for readers to follow a project as it evolve because Manifold can publish a project iteratively, that is, in pieces over time.

We could transform the process of how the author might publish. We started our work with definitions: What is a version? What is an iteration? What is a resource? We broke from the word “book” and moved to the broader term “project,” which we define as a work that is a combination of texts and resources.

Susan noted that other publishing tools offer these features — but Manifold also aims to “create an efficient workflow process available to be replicated by other publishers.”

Next I asked, what’s your ideal reading experience? This one took a moment. Susan said she had no single answer. It all depends on “What is it that you’re reading for? Why are you coming to the text?” For leisure reading, Susan reads paperbacks or on her ebook reader. Reading for work, though–she’s a note taker–means reading reading parts of a work over time and marking up a text. Her reading demands “different interactions depending on why I’m reading this text. Reading for information, you want the apparatus for notes and asking questions and linking to other pieces.” For her, when reading for escapism she wants to lose herself in the story. Manifold is not meant to serve all the reasons you read, rather “it is addressing something specific. It’s not meant to be a panacea of all publishing.” Susan went on,

Print is a funny thing — we’re used to it historically being the solution for sharing and distributing published work. The book object — front cover, open it and read, turn the page, is familiar, and that medium for a mode of expressing text and images to share information is really sophisticated — it evolved over hundreds of years. Manifold is a baby — we’re developing its first version. A book, the object, meets so many needs — and we’ve had years to adapt it to meet so many needs — it feels perfect, but it may just be familiar.

When asked what she is reading now, Susan described a reading practice that spans formats. The books she reads circle around subjects she never studied in school, such as medieval central Asia. She is currently reading Empires of the Silk Road, by Christopher Beckwith, after finishing The Lost Enlightenment by S. Frederick Starr and Amazons: Lives and Legends of Warrior Women across the Ancient World by Adrienne Mayor. She also listens to audiobooks of cultural histories while she paints; she just finished The Art of Rivalry by Sebastian Smee, which she found after enjoying In Montmartre: Picasso, Matisse and Modernism in Paris 1900-1910, by Susan Roe. On her ebook reader, she says she reads mostly fiction — recently N. K. Jemisin’s The Obelisk Gate.

For the last question — who inspires your work? — Susan had quite an inspiring list.

  • Other university presses. She mentioned frontrunner work of National Academies Press. She follows publishing evolution — how they are adapting practices, not just around issues of open access. She appreciates learning from what works for others and what doesn’t.
  • Librarians. She mentioned the conversations and collaborations of librarians and publishers at the Charleston Conference.
  • Authors. By watching and talking to authors on social media, she sees how they figure out how to self-publish and work with publishers.

Ultimately, she finds inspiration in the process of art making.

Art – the process of art-making, creating and making; manifesting ideas as a physical thing — into text, fiber — if you’re making an art object — I find that riveting — that process, seeing the evolution of ideas through a work of art. If I’m able to go read about the ideas behind the artist’s work it changes my engagement with the art. Music is the same — one day while listening to Miles Davis’s Kind of Blue while in line at the bank in my car — I pulled out the pamphlet in the CD case and read about how the album was recorded, their process. It changed how I hear the music!

Throughout our conversation, I heard Susan’s excitement about the work she does and an attention to the ways she works and thinks. Susan gave me some good book recommendations, and we had the chance to discuss our hopes for scholarly communication and monographs. She generously shared her own experience, yet made clear that what matters most to her is serving her audience. A reassuring commitment in an unsteady time.

Thank you, Susan! 

 

Follow Susan @susanmpls.

This Week in Manifold: DevOps, Infrastructure, and Social Activity

Thanks for joining me for another installation of “This Week in Manifold.”

First off, please accept my apologies for missing last week’s update. When last Friday rolled around, we had a number of updates ready to go out to our staging site. However, when it was time to deploy, we hit a few unexpected problems.

DevOps, Hosting Manifold, and Deployment

This release adds a new metadata field to projects. Because the exact metadata requirements are difficult to pin down and may change from press to press, we wanted a relatively flexible approach to storing project, text, and resource metadata. Back in the day, we might have used an EAV model for this, but there are many well-documented drawbacks to that model. Because Manifold uses PostgreSQL as its database backend, we have access to a variety of field types for storing arbitrary amounts of data. In the past, we’ve used HSTORE for storing key/value pairs, but in this case we wanted something more flexible that could store different data types (HSTORE stores strings, but we want to store integers, boolean values, strings, arrays, etc). Recent versions of PostgreSQL include the JSONB field type, which is exactly what we want in this case.

Alas, that meant that the October 28th release required some infrastructure updates. Sure, we could have shelled into the staging server and done a quick upgrade, but that’s not how we roll on this project. It’s really important to us that it be possible to easily provision hosts for Manifold, which means automating as much as possible. As Manifold grows, it relies on more and more services to run, which makes setting it up increasingly complex. With this release, Manifold needs all of the following configured on the server to run correctly:

  • Ruby 2.3 (for the API)
  • Node 4.5 or higher (for the client)
  • Postgres 9.4 or higher (for storage)
  • Redis (for queuing background jobs)
  • ImageMagick or GraphicsMagick (for resizing images)
  • Nginx or Apache (for serving the API and the Client)

Moreover, in addition to having all these requirements in place, the server also requires that a number of background processes are always running. As of right now, there are four custom services (excluding the webserver and the database server) that need to be running on the host: 1) manifold_api (Rails app served by Puma), 2) manifold_client (Node app), 3) manifold_scheduler (Ruby daemon), and 4) manifold_workers (Sidekiq). Keeping four services running on a server (or on multiple servers) is not without its complexities. If we run these on Ubuntu Trusty (v14), for example, we need Upstart scripts that manage the services. If we run Manifold on Ubuntu Xenial (v16), then we’re going to need Systemd scripts. If it runs on older versions of Centos or Redhat, then we’ll probably need some init.d scripts.

Running a complex app with separate parts is tricky. Internally at Cast Iron, we use puppet and foreman to manage these services. Needing to support Postgres 9.4 got us thinking about deploying Manifold to Ubuntu Xenial, which in turn led us to revisit our manifests. Once we went to Xenial, we had to rewrite our startup scripts for systemd, which in turn revealed problems with how we were storing secrets in the environment. In short, instead of spending the last week working on project resources and feature development, we spent it paying down some technical debt on the infrastructure side, and improving Manifold’s devOps and stability.

That’s done now, and we’re deploying Manifold to Xenial and fully managing its environment and all dependencies via Puppet. Revisiting this infrastructure has also given us an opportunity to reflect further on Manifold’s installation story and continue discussions about how we can make Manifold easy for other presses to adopt and install.

New Features

We did manage to slip a few important features into this week’s release. The full list is below, but there’s one in particular I want to highlight.

This release includes a scheduler that runs constantly in the background while Manifold is running. One of the things this scheduler does is trigger a background job for fetching tweets related to projects in Manifold. That job currently runs hourly.

Editors can use twitter search terms to describe what kind of tweets Manifold should pull in for a given project. This gives editors quite a bit of flexibility. For example, Manifold might watch the author’s twitter account for all tweets with a specific hash tag. Or, it might watch twitter in general for all tweets that include a given hashtag. Our tweet fetching service accepts multiple search terms, so it’s possible for a project to follow multiple accounts and hashtags.

These tweets show up on the project activity feed. Readers can browse through the historical activity for a project and see the events—texts being added, tweets about the project, new resources uploaded—that took place as the project became a published book (see screenshot, below). Our hope is that showing this activity helps make public the iterative aspects of authorship.

Project Activity

Well, that’s enough for today. I’ll be back next week with another update. Enjoy the weekend!

Refactoring

  • Display project “makers” (authors, editors, etc) on one line
  • Improve header/icon lockup on project detail component
  • Refactor ingestion resources into a new model that is distinct from project resources
  • Clean up API binstubs

Features and Improvements

  • Add JSONB metadata field to projects; adjust project import mechanism to include metadata
  • Expose project metadata in API
  • Render project metadata in the client
  • Incorporate Sidekiq for background job processing
  • Add the concept of model ownership in various places. Manifold will now track who creates a project.
  • See a system-level user for ownership purposes
  • First pass at project events subsystem including background event generation; create Event factory class
  • Expose project events via the API
  • Render project events in the “activity” component on the project detail view
  • Add job for queuing the fetching of project tweets
  • Add job for fetching project tweets
  • Add a background scheduler (Clockwork) for triggering recurring background jobs

Tooling and Libraries

  • Update Postgres requirement to 9.4 or 9.5 for JSONB
  • Adjust Travis CI build to use Yarn instead of NPM
  • Feed Travis build notifications to Slack
  • Numerous revisions to deployment configuration

Bug Fixes

  • PropType correction in EventList component
  • Address 1:N queries in the project controller
  • Correct text icon on project texts list
  • Fix padding on coverless project heroes
  • Adjust font size and padding on project makers