TinaCMS Self Hosted Solution and Demo

[logan-anderson]

Self Hosted Solution and Demo

Just want to see the code? You can find the example here.

Goal

This demo provides an example of self hosting Tina. This allows you to own your own content and your own auth. It is also possible to self host and still use Tina Cloud for just authorization.

Caveats of Self Hosting

• If not using Tina Cloud's authorization, you must implement your own authentication.
  • Note: this means you will have to provide your own read-only token implementation.
• You are responsible for hosting your own database and LevelDB implementation (We have provided a MongoDB LevelDB implementation that can be used)
• You are responsible for hosting your TinaCMS API endpoint (like a Next.js api function)

Demo

We have recorded the following videos that explain how to self host Tina. The first video shows a demo of the final product. The second video demonstrates how to self host the content and API endpoint. The third video demonstrates how to implement authentication.

Self Hosting Overview

This video provides an overview of the self hosting process. It shows a demo as well as explain what the Data Layer is.

Self Hosting the "Data Layer"

This video shows how to turn a Next.js site into a self hosted Tina site.

Self Hosting Authentication

This video shows how to implement authentication for a self hosted Tina site.

Elements of Self Hosted Tina

Database File

The database is configured in .tina/database.{js,ts} .

This file is the main element of the self hosted solution. It exports an instance of the TinaCMS Database, which handles indexing, queries and CRUD operations. A database instance requires a LevelDB implementation. It is also configured with optional onPut/onDelete handlers which are used to make updates to your Git repository. The database acts as an ephemeral caching layer, so that when you query your content it is not necessary to retrieve it from the git provider.

import { createDatabase, TinaLevelClient } from '@tinacms/datalayer'
import { MongodbLevel } from 'mongodb-level'
import { Octokit } from '@octokit/rest'
import { Base64 } from 'js-base64'

// `isLocal` determines if the database is running in "Local Mode" or "Production Mode". You can set this value in your .env file or use a different method for determining the value. In this example we are using an environment variable.

// When in "Local Mode" a local levelDB server is used and data is saved to the file system
// When in "Production Mode" Your provided LevelDB implementation is used (MongoDB Level in this example) and data is written to the Git repository with "onPut" and "onDelete" callback functions
const isLocal = process.env.TINA_PUBLIC_IS_LOCAL === 'true'

if (isLocal) console.log('Running TinaCMS in local mode.')
else console.log('Running TinaCMS in production mode.')

const owner = process.env.GITHUB_OWNER as string
const repo = process.env.GITHUB_REPO as string
// Must create a personal access token
const token = process.env.GITHUB_PERSONAL_ACCESS_TOKEN as string
const branch = process.env.GITHUB_BRANCH as string

const octokit = new Octokit({
  auth: token,
})
const localLevelStore = new TinaLevelClient()
const mongodbLevelStore = new MongodbLevel({
  collectionName: 'tinacms',
  dbName: 'tinacms',
  mongoUri: process.env.MONGODB_URI as string,
})
if (isLocal) {
  localLevelStore.openConnection()
}

// When the data is updated this function is called
const githubOnPut = async (key, value) => {
  // See example for implementation
}
const localOnPut = async (key, value) => {
  // See example for implementation
}

// When the data is deelted this function is called
const githubOnDelete = async (key) => {
  // See example for implementation
}
const localOnDelete = async (key, value) => {
  // See example for implementation
}

export default createDatabase({
  level: isLocal ? localLevelStore : mongodbLevelStore,
  onPut: isLocal ? localOnPut : githubOnPut,
  onDelete: isLocal ? localOnDelete : githubOnDelete,
})

Level

You must provide an abstract-level database implementation. In the example above we have used mongodb-level which is a LevelDB implementation maintained by the TinaCMS team. You are free to use the mongodb-level implementation or make your own abstract-level implementation and use that instead.

onPut and onDelete

The onPut and onDelete functions are used to update the git repository when there are updates and deletes via the TinaCMS api. In the example above we show how to save data to Github, but feel free to swap our example for any git provider.

The onPut function takes key and value parameters. The key is the path to the file in the repository that was updated and the value is the file contents. The onDelete function takes a key parameter, which is the path to the file in the repository that was deleted.

Using the database on the server

Querying the database from the server works a bit different when using self hosted Tina. When using Tina, you normally use The Tina Client, but when self hosting, it is likely that that the GraphQL endpoint will not be available at build time (If you are using Next.js api endpoints, for example). So when querying content from the server, we recommend querying the database directly. We have created an example of what this looks like:

lib/databaseConnection.ts

import database from '../.tina/database'
import { queries } from '../.tina/__generated__/types'
import { resolve } from '@tinacms/datalayer'
import type { TinaClient } from 'tinacms/dist/client'

export async function databaseRequest({ query, variables }) {
  const config = {
    useRelativeMedia: true,
  } as any

  const result = await resolve({
    config,
    database,
    query,
    variables,
    verbose: true,
  })

  return result
}

export function getDatabaseConnection<GenQueries = Record<string, unknown>>({
  queries,
}: {
  queries: (client: {
    request: TinaClient<GenQueries>['request']
  }) => GenQueries
}) {
  const request = async ({ query, variables }) => {
    const data = await databaseRequest({ query, variables })
    return { data: data.data as any, query, variables, errors: data.errors }
  }
  const q = queries({
    request,
  })
  return { queries: q, request }
}

export const dbConnection = getDatabaseConnection({ queries })

With this, you can use dbConnection just like the Tina Client would be used. It will have all the generated queries and a request function for raw GraphQL requests.

Here is an example of using this:

import { dbConnection } from '../../lib/databaseConnection'

export const getStaticProps = async ({ params }) => {
  const tinaProps = await dbConnection.queries.blogPostQuery({
    relativePath: `${params.filename}.mdx`,
  })
  return {
    props: {
      ...tinaProps,
    },
  }
}

dbConnection has direct database access, so it can only be used on the server. If you want to fetch data on the client side, you must use the Tina Client).

The GraphQL Endpoint

When editing with TinaCMS, CRUD operations get sent to a GraphQL endpoint. Normally this is Tina Cloud, but when you self host you must provide this endpoint. The following examples show how this can be done in a Next.js API route, but it can adapted for use in any environment. You must add your own authorization function here or use Tina Cloud's auth server if you wish.

pages/api/gql.{ts,js}

import { databaseRequest } from '../../lib/databaseConnection'
import database from '../.tina/database'

export default async function handler(req, res) {
  // Add your own Auth here.
  // For example
  // If(!await isAuthorized(token: req.headers.authorization)){
  //   return a 401
  //  }
  const { query, variables } = req.body
  const result = await databaseRequest({ query, variables, database })
  return res.json(result)
}

Now you can configure this endpoint in the config:

.tina/config.{ts,js}

const config = defineConfig({
  contentApiUrlOverride: '/api/gql',
  admin: {
    auth: {
      useLocalAuth: process.env.TINA_PUBLIC_IS_LOCAL === 'true',
    }
  }
  //...

Authentication

Using Tina Cloud for Authentication

If you just wish to self host your content and you don't need to customize authentication, you can use Tina Cloud for authorization and authentication. This can be done by adding the following to your endpoint:

pages/api/gql.{ts,js}

import { isUserAuthorized } from "@tinacms/auth";

import { databaseRequest } from "../../lib/databaseConnection";
import createDatabase from "../../.tina/database";
const database = createDatabase();

export default async function handler(req, res) {
+  const isAuthorized = await isUserAuthorized({
+    token: req.headers.authorization,
+    clientID: "<YourClientIdFromTinaCloud>",
+  });
+  if (!isAuthorized) {
+    return res.status(401).json({ message: "Unauthorized" });
+  }

  const { query, variables } = req.body;
  const result = await databaseRequest({ query, variables, database });
  return res.json(result);
}

Self hosting your Authentication

To self host your own authentication, you must implement several functions. These functions are passed to the TinaCMS client when it is initialized. The following functions are available:

authenticate: This function is called when the user goes into /admin and they are not logged in (determined by getUser). This function should redirect the user to the login page or do whatever is necessary to authenticate the user.

getUser: This function is called when the user goes into /admin and is used to determine if the user is logged in. If it returns a truthy value, the user is logged in and if it returns a falsy value the user is not logged in.

getToken: This function is called when a request is made to the GraphQL endpoint. It should return an object with an id_token property. This will be passed as an Authorization header in the format Bearer <id_token>

logOut: This function is called when the user clicks the logout button in the admin.

Set customAuth to true in the config to enable this.

Add the functions to your config.{ts,js}

export default defineConfig({
  contentApiUrlOverride: '/api/gql',
  admin: {
    auth: {
      customAuth: true,
      // This is called when they want to authenticate a user. For a lot of implementations it just may be redirecting to the login page
      async authenticate() {
        console.log('Authenticating...')
        localStorage.setItem(
          'local_cache',
          JSON.stringify({ name: 'Logan', role: 'admin' })
        )
        return {}
      },
      // Get token this will be called when a request and will be passed as an `Authorization` header in the format `Bearer <id_token>`
      getToken: async () => {
        return {
          id_token: 'Foo',
        }
      },
      // Called to log the user out
      async logOut() {
        console.log('logOut...')
        localStorage.removeItem('local_cache')
        window.location.href = '/'
      },
      // The CMS uses this function to get info about the user. It also uses it to see if the user is logged in. Provide a truthy value if the user is logged in and a falsy value if the user is not
      async getUser() {
        console.log('getUser...')
        const userStr = localStorage.getItem('local_cache')
        if (!userStr) {
          return undefined
        } else {
          try {
            return JSON.parse(userStr)
          } catch {
            return null
          }
        }
      },
      //...

Next you can use the value passed from getToken in your backend function to make sure the user is authenticated:

pages/api/gql.{js,ts}

import { databaseRequest } from "../../lib/databaseConnection";
import createDatabase from "../../.tina/database";
const database = createDatabase();

export default async function handler(req, res) {
+  const isAuthorized = await myCustomIsAuthorizedFunction({
+       // This has the format of `Bearer <id_token>`
+       token: req.headers.authorization,
+  });

+  if (!isAuthorized) {
+    return res.status(401).json({ message: "Unauthorized" });
+  }

const { query, variables } = req.body;
const result = await databaseRequest({ query, variables, database });
return res.json(result);
}

Questions

This API is very new and is subject to changes based on feedback from the community. Please leave your feedback and questions below.

[mAAdhaTTah]

How is the database populated initially from the GitHub contents? The reason I ask is I have a personal blog, so I don't really need the Tina admin to run in production at all (I usually edit my posts locally rather than in Tina Cloud). I've been using Tina Cloud to fetch the content as needed to build the blog, but if I was able to fetch instead from a local db, that would (probably?) cut down on the build time (I also get 503s from Tina Cloud intermittently, plus some issues with attempting the build before the branch has been indexed). My thought would be to do something to generate the database into like a sqlite db on build before building the main site, then just use that local db connection to build & run the site. Does that sound feasible?

[mAAdhaTTah]

My intention is to disable editing in production and do all my editing locally, since it's my personal blog. Good to know RE the level server tho.

[logan-anderson]

If your plan is not to edit on production then just doing tiancms dev -c "next build" is your a viable solution.

EDIT: You might not even need self hosting or Tina Cloud in this case. tinacms dev does what you are doing in your database.ts by default.

[mAAdhaTTah]

@logan-anderson Part of the issue is Nextjs middleware isn't static, which I want to use to generate redirects, sitemaps, rss feeds, etc. I've also got some other things that need to revalidate periodically from non-Tina sources and that will fail when attempting to stitch that with the Tina data. Correct me if I'm wrong, but the approach you're suggesting would only work for a fully-static site, correct?

[logan-anderson]

Your entire site does not need to be static but you can only use the Tina client at build time with this approach (can only use Tina in a static context).

I believe this is also the case for your current setup?

[mAAdhaTTah]

Middleware is not static and would need to request Tina data at runtime. I'm not using middleware yet but would like to. Was hoping to use this self-hosted hooked up to make that work.

[coreyaus]

Massive congrats on the excellent work launching this self-hosting update Logan (and the rest of the Tina team!)

Similar to mAAdhaTTah's question, I'm also trying to understand the indexing process in more detail. I can see yarn build will run the indexing process on the self-hosted data layer via the indexIntoSelfHostedDatabase method (based on the default export in the .tina/database.ts file). So if our Tina content lives in the same repository as our Next.js site then the re-indexing should occur seamlessly whenever a commit is made (at which point yarn build will be run via an automatic deploy in Vercel).

My main questions are:

1. What would happen for projects that use Tina's new feature for separate content repositories? [Solved] Add separate repo support (Markdown content in one, Next.js in another) #2821

   • In that case, as far as I'm aware, there would be no automatic build process triggered by default when changes are made to the content repository.
   • What would be the best process to ensure re-indexing occurs in that case?
     • A) Would we essentially need to add a GitHub action to run the build process on every commit in the content repo? (and therefore the content repository would potentially need a database.ts file and a package.json with all Tina packages installed)
     • B) Are there any suggestions or alternatives based on what Tina Cloud does to implement the "Reset repository cache" and "Reindex" buttons (see example screenshot below)? Is there an example script you can share for explicitly triggering a reindex, similar to what's possible in the Tina Cloud portal under the "Configuration" tab for a project?
       • It seems the required components are the GraphQL schema, the Tina schema and the database.ts file, but there aren't any exported methods to run the indexing process directly as far as I can tell, so I can't see a way to do it without extracting a heap of code from the Tina internals (but obviously wiser heads than mine were able to deliver a neat solution to power those "reindex" features in Tina Cloud!)
       

2. Are configurations like client.referenceDepth still applied when using the self-hosted data layer?

3. When working with multiple branches, is the idea to create a separate collection for each branch in the same MongoDB database? Or how is this generally handled (e.g. what approach does Tina Cloud take)?

A huge thanks in advance for any help you can offer, and have a top week!

[logan-anderson]

1. What would happen for projects that use Tina's new feature for separate content repositories?

This is a great question. This was not really taken into consideration when doing our first version of this.

For context, behind the scene TinaCMS uses a "Bridge" to get the content for indexing. For self hosted we use the filesystem bridge by default but for this use case we would need a "Github Bridge" we had one at one point but I think it was removed from the codebase. We can look at adding it back to the codebase for those who want to use a separate content repo.

Although its not available right now I think that this would look something like this.

export default createDatabase({
  bridge: new GithubBridge({owner, ref, repo, token }),
  level: isLocal ? localLevelStore : mongodbLevelStore,
  onPut: isLocal ? localOnPut : githubOnPut,
  onDelete: isLocal ? localOnDelete : githubOnDelete,
});

This would index your content from another source but would still do it only at build time.

What would be the best process to ensure re-indexing occurs in that case?

A)

If you are editing from TinaCMS, this should not be an issue because the Datalayer should automatically be updated on save. But if you push commits to the content repo yourself (not through TinaCMS) that could cause an issue. The Datalayer would not know about these changes and would not be updated

Again this was not prioritized, so there won't be any examples of how to do it but I can provide an overview of how we could support this (Assuming we can add the GithubBridge).

I think the could look something like this

Commit in content repo -> Github action to get all changed files in content repo -> Sends updated files to a api route in your main repo.

The api route would look something like this.

import { NextApiHandler } from "next";

import database from "../../.tina/database";

const indexHandler: NextApiHandler = async (req, res) => {
  // use your own auth logic
  const isAuthorized = true;
  if (isAuthorized) {
    // get the paths from the request body instead of hardcoding them
    const paths: string[] = ["content/home.json"];
    await database.indexContentByPaths(paths);
    return res.status(200).json({ success: true });
  } else {
    res.status(401).json({ success: false });
  }
};

export default indexHandler;

This works right now but unfortunately currently looks at the filesystem which is not super helpful (and would not work for separate content repos)

I made a quick demo video that may help clarify: https://www.loom.com/share/c7d1c4f5bda04ec3a1dacff2a03efac1

B)

Reindexing the entire repo is supported but I was looking at the code and I think there is some cleanup required to make it easier for folks who are self hosting.

I think the API should look like this (not currently supported)

import { NextApiHandler } from "next";

import database from "../../.tina/database";
import config from "../../.tina/config";
import gqlSchema from "../../.tina/__generated__/_schema.json";

const indexHandler: NextApiHandler = async (req, res) => {
  //   use your own auth logic

  //  remove functions
  const schema = JSON.parse(JSON.stringify(config.schema));
  const result = await database.indexContent({
    tinaSchema: schema,
    gqlSchema,
  });
  console.log(result);
  res.status(200).json({ success: true });
};

export default indexHandler;

This will require some work on our end. I can make a Github Issue for this.

Update: Following along here for more updates on this.

2. Are configurations like client.referenceDepth still applied when using the self-hosted data layer?

Yup client.referenceDepth is support. I think almost everything would be work as expected except things like clientId, token, and branch are specific to Tina Cloud.

3 When working with multiple branches, is the idea to create a separate collection for each branch in the same MongoDB database? Or how is this generally handled (e.g. what approach does Tina Cloud take)?

We don't really have a best practice for this but separate collections make sense to me. This is similar to how we do it in Tina Cloud (we don't use MongoDB though)

[laicaro]

Hi,

Would it be possible to make the admin editor page inaccessible?
I'm planning to only have the admin editor page accessible in lower environments but have it totally inaccessible in production.

[kldavis4]

You can simply not build the admin in the production environment to make it inaccessible.

[GaspardCulis]

Have you tried to use the Media Manager with this setup ? Unfortunately I don't think the media manager can work with this setup.

[alsherif-khalaf]

It works but it doesn't upload

[SGiumelli]

Hi,

i know it's an old discussion but all the youtube videos are gone, did they get moved or is any replacement ?

[kldavis4]

which youtube videos specifically? This one, for example, is working: https://www.youtube.com/watch?v=DAJpnjkLBbM

[SGiumelli]

Self Hosting Overview

This video provides an overview of the self hosting process. It shows a demo as well as explain what the Data Layer is.

Self Hosting the "Data Layer"

This video shows how to turn a Next.js site into a self hosted Tina site.

Self Hosting Authentication

This video shows how to implement authentication for a self hosted Tina site.

Those 3 are unavailable to me

[kldavis4]

Those are from an earlier version of the self-hosted implementation. Please refer to these docs instead: https://tina.io/docs/self-hosted/overview/

[SGiumelli]

Thank you !