I Think the Docs Would be Clearer With a Central "Customizing Postgraphile" Page

[machineghost]

Summary

One thing I find frustrating about the Postgraphile docs is that the various ways of customizing Postgraphile are scattered throughout the docs in a way that's unfriendly to a new user. The closest place to a central explanatory page is https://www.graphile.org/postgraphile/functions/ ... but if you're new to the library, there's nothing to indicate to you that Functions === Customization.

That page also doesn't mention other ways to customize/extend a Postgraphile API, such as using views, smart comments, or plugins. Those last two are kept under a "Customizing" header ... but then there's no mention of the function stuff under that header.

And perhaps most crucially of all, none of this is user-oriented. As a user I'm looking for "how do I add a new endpoint?" or "how do I add a relation to an existing endpoint?" or "how do I rename an endpoint?" ... but I won't find any of that in the docs! What I'll find is stuff like "Computed Columns" or "makeExtendSchemaPlugin" ... but as a new user reading the docs, I have no idea that those things can answer my question.

Proposed Solution:
I propose a new page (under the "Customization" header, above "Smart Tags"), titled simply "Customizing Postgraphile" (or perhaps "Customizing a Postgraphile API"). This page would be user-oriented, and would have headings for the questions I just mentioned.

For instance, after some basic intro text, the first heading might be "Renaming Tables". It would have a basic example of using smart comments to rename a table, along with a link to https://www.graphile.org/postgraphile/smart-tags/ for further details. It would also offer the option of using a computed column instead (with a link to https://www.graphile.org/postgraphile/computed-columns/), as well as the option of simply making a view of the table.

Then it would go on to the next heading ("Renaming Columns"), and the next ("Adding a Relation"), and the next ("Hiding a Relation"), and so on. Every way that the user could want to change their API would be listed, and the technical solutions would flow from that user desire ... instead of being scattered around the documentation.

Additional context

While I am no Postgraphile expert, I would be happy to submit a PR of (at least a first draft of) this page, if it is desired.

[benjie]

Yep sounds great; we have had https://diataxis.fr/ on our todo list for ages, but all my work is going into the V5 codebase right now — we’d really appreciate the help. It would probably be better to focus your efforts on the PostGraphile.org website; once the V4 docs on there are equivalent to those at Graphile.org we’ll be setting the latter up as a redirect to the former; @jemgillam is currently working on this effort. You might need some assistance getting the V5 codebase including the websites running; do ask for guidance either here or on Discord if that’s the case.

[machineghost]

I was able to clone the repo and run yarn to install things, but when I try to start the dev server I got an error. I don't need to have a working site to add the page (I can just edit the MD file without seeing how it looks rendered), but if you have any thoughts, it'd be nice to see how the page will look in the site.

The error (must be superuser to create superusers) suggests I need to run as sudo ... but I wouldn't imagine sudo privileges would be necessary just to run a simple webserver?

$  yarn run develop
yarn run v1.22.19
$ ./scripts/update && gatsby develop
++ dirname ./scripts/update
+ cd ./scripts
+ ./update-cli
+ ./update-library-and-schema
+ ./update-examples
error: must be superuser to create superusers
    at Parser.parseErrorMessage (/home/me/Projects/graphile.github.io/node_modules/pg/node_modules/pg-protocol/dist/parser.js:278:15)
    at Parser.handlePacket (/home/me/Projects/graphile.github.io/node_modules/pg/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/home/me/Projects/graphile.github.io/node_modules/pg/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/home/me/Projects/graphile.github.io/node_modules/pg/node_modules/pg-protocol/dist/index.js:8:42)
    at Socket.emit (events.js:314:20)
    at addChunk (_stream_readable.js:297:12)
    at readableAddChunk (_stream_readable.js:272:9)
    at Socket.Readable.push (_stream_readable.js:213:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
  length: 91,
  severity: 'ERROR',
  code: '42501',
  detail: undefined,
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'user.c',
  line: '292',
  routine: 'CreateRole'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[benjie]

I think you might be running on the wrong branch; you should be on main of the crystal repo then it’s yarn && yarn watch and separately (once that tells you it’s waiting for changes or whatever) cd postgraphile/website && yarn start

[benjie]

Oh sorry yeah, this is the graphile.org site in this repo; the postgraphile site is now built into the crystal monorepo so that code changes can be synchronised with documentation changes; so you need to be on https://github.com/graphile/crystal

[machineghost]

Two things:

First, when I tried to yarn run docs:gen I got:

Debugger listening on ws://127.0.0.1:36715/ab5cf694-77ed-4c5f-862f-95b25553ad0c
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
Debugger listening on ws://127.0.0.1:43871/c6492e10-a970-4163-a7fd-560b2c03f067
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
[error] Tried to set an option (noMissingExports) that was not declared. You may have meant:
        sort
[error] Tried to set an option (pluginPages) that was not declared. You may have meant:
        plugin
Waiting for the debugger to disconnect...
Waiting for the debugger to disconnect...

Do I need to run something else before that command.

Second, I just saw https://postgraphile.org/postgraphile/current/ for the first time, and was totally confused because I've been using https://www.graphile.org/postgraphile/introduction/ for forever.

Having two different copies of the docs virtually guarantees users will wind up looking at the wrong one, so it really seems one of those should really be a redirect to the other.

[benjie]

Yes; that’s why I said above once we’ve finished porting the V4 docs to PostGraphile.org we will be setting graphile.org/postgraphile up as a redirect. For now, Graphile.org is the correct place for V4, postgraphile.org for v5; but postgraphile.org will support versioning of the documentation.

The docs:gen command is for a non-functional typedoc setup I was working on like 18 months ago; instead of running that, please follow the instructions in this comment: #407 (comment)

[machineghost]

I'm still just getting started on the page, but one quick note: the /postgraphile/computed-columns route seems to be broken.

I tried to figure out why, but there is a postgraphile-crystal/postgraphile/website/versioned_docs/version-4.12.0/computed-columns.md file, and that file isn't buggy (when I reduce it to just plain text, it still fails to show), so I'm not sure what's causing it.

[machineghost]

Here's a screenshot of what I've started for the page. It's incomplete of course, but I'd just like to get your take on the general direction (before I go too far down the wrong path).

[benjie]

I'm still just getting started on the page, but one quick note: the /postgraphile/computed-columns route seems to be broken.

Docusaurus uses versioning in the URLs, so it's /postgraphile/VERSION/page; i.e. computed-columns is at https://postgraphile.org/postgraphile/current/computed-columns/

[benjie]

Feedback on what you've started writing: my concern is this is going to become a mega page which'll be so long it'll be hard to navigate and lose its utility. How about something more along the lines of giving the very briefest of introductions into the various options in a task-oriented fashion, and then linking out to the relevant docs on how to do each thing. This might result in adding explicit examples in other parts of the documentation (e.g. we don't seem to have docs that give examples of primary key/unique constraints giving accessor fields) which would be good.

Another option might be to do a hybrid, where examples are provided inline using <details><summary>Bullet text here</summary> Example code here </details>; you can see this pattern in use on this page, look for the "click to expand" / blue triangles: https://postgraphile.org/postgraphile/next/migrating-from-v4/v5-new-feature-summary

(Note I scratched this stuff directly into the comment box, and I trailed off after the first few because it's more a demonstration of the kind of thing I'm thinking than an exhaustive list of what it would contain.)

---

Customization Summary

Though an instantly produced GraphQL API is exciting, PostGraphile is hugely flexible and we strongly encourage you to take some time to customize your API to your own needs. Time spent up front to customize the general shape of your API can pay dividends later, and help avoid having a huge sprawling API that is hard to manage.

The purpose of this page is to give you information at a glance about the options for customization in a task-oriented fashion, with links to the documentation for you to read about the specifics.

Adding fields

PostGraphile automatically reflects the tables, constraints, functions, etc in your DB, so you can add fields by adding such things, or by adding plugins:

Adding root Query fields

• Create a table or view to add fields to fetch all rows from this table
• Create a unique or primary key constraint on a table to get single-row accessor fields: "table row by id" or similar (for views, use a @primaryKey or @unique smart tag)
• Create a "custom query" database function to add a GraphQL field with the same name as the function; this can return a scalar, composite type, or a list or set of these
• Use makeExtendSchemaPlugin to add a field in JavaScript/TypeScript

Adding fields to GraphQL types

• Create a foreign key constraint to create fields in both directions between the types (for views, use the @foreignKey smart tag)
• Create a "computed column" database function, similar to a "custom query" database function except the result is added to a specific table's type rather than to the root Query
• Use makeExtendSchemaPlugin to add a field in JavaScript/TypeScript

Adding Mutation fields

• Grant permissions (if using ignoreRBAC: false)
• Create table
• Create "custom mutation" database function
• Use makeExtendSchemaPlugin
• Use one of the community plugins

Adding types

• Create a table, view, composite type, enum, ...
• Use makeExtendSchemaPlugin

Removing fields

• Use @omit smart tag to hone how things are exposed:
  • Add directly to the table to control common behaviors
  • Add to a specific column e.g. to disable filtering/ordering by a column
  • Add to a specific constraint e.g. to disable accessors or mutations for that constraint
• Manually with the plugin system; see: https://postgraphile.org/postgraphile/current/extending-raw/#removing-things-from-the-schema

Renaming fields

It's important to ensure that your fields are sensibly named; they don't have to have the same names as your database resources, nor do they have to have really long autogenerated names. Here are some resources for renaming fields:

• Use "simplify inflection" plugin for shorter default names if your database conforms to the expectations of the plugin, see the README
• Use @name/@fieldName/... smart tags, for function, view, etc etc
• Use @resultFieldName smart tag to rename the result of a mutation

Renaming types

• @name smart tag, etc

Filtering results

• Use makeAddPgTableConditionPlugin
• Use @omit's filter to choose which things get filtering
• Use @filterable on functions to enable filtering
• Use Matt's filter plugin
• Create custom query / computed column database functions
• Use plugins such as pg-omit-archived
• etc

Adding documentation

• COMMENT ON database resource to add documentation
• Alternatively use tags plugins to add descriptions:
  • postgraphile.tags.json5
  • makeAddPgSmartTagsJSONPluginOrWhateverItsCalledIDontRemember()

Etc

[machineghost]

Another option might be to do a hybrid, where examples are provided inline using

Bullet text here

Example code here

; you can see this pattern in use on this page, look for the "click to expand" / blue triangles: https://postgraphile.org/postgraphile/next/migrating-from-v4/v5-new-feature-summary

I really liked this approach for the SQL examples:

However, there was a problem: when I tried to use that pattern with the Javascript example ...

it failed, evidently because the markdown parser actually runs the Javascript code, and the first line of it is:

    const { postgraphile } = require("postgraphile");

For now I just removed examples entirely. We can always add them back in if the page isn't too "mega page"-y.

Speaking of that concern, I shortened the text, combined entries, and really just tried to focus almost entirely on routing the user (not educating them).

(Note I scratched this stuff directly into the comment box, and ...

This is super helpful.

Here's where I am so far: I have all of the root-level query field options, but still need to do nested fields and mutation.

Let me know if you'd prefer an actual PR, but I figured at this stage the screenshot conveys everything easily enough.

[benjie]

Excellent work!

it failed, evidently because the markdown parser actually runs the Javascript code, and the first line of it is:

You need to add the code using ```ts ... ``` backtick blocks, otherwise it gets interpreted as being part of the MDX (which is markdown with JS/React/etc mixed in).

really just tried to focus almost entirely on routing the user (not educating them).

I definitely think this is the right approach!

Postgraphile

Please be careful of captialization: PostGraphile. You can leave fixing this until the end, I just raise it in case doing it as you go along will be easier.

also

Remove extraneous words like "also", "just", "that", etc. when possible. In general, less is more. (I'm just as guilty of this as anyone.)

Learn more

Better for accessibility (and in general to know if you've been there before) to make the links more descriptive, for example: "Learn more in 'Inflection'." / "See example in 'Computed Columns'." / etc

The makeExtendSchemaPlugin plugin

To be clear, makeExtendSchemaPlugin makes a plugin that extends the schema; it is a plugin factory. Call it a function or a factory, it is definitely not a plugin itself. Also, drop the "the" :) makeExtendSchemaPlugin lets you create root level fields...

Simplify Inflection plugin

Yeah, there's two different (but similar) plugins with different names, one is for V4 and one for V5 but they're both referred to as this. V5's is here: https://github.com/graphile/crystal/tree/main/graphile-build/graphile-simplify-inflection; V4's is here: https://github.com/graphile/pg-simplify-inflector . They're both currently documented by their own READMEs. I support moving the READMEs into the (PostGraphile) documentation instead.

---

I like this. Couple general bits of feedback:

1. be sure that the levels of heading are expected (don't jump from H2 to H4 without an H3)
2. we're presenting the different options but not necessarily helping the user to choose which one they should do. For example: you want to rename one table, use a smart tag; you want to rename all tables, use inflection. You want to add a field, use a database function if it's suitable and you're comfortable doing so, or use makeExtendSchemaPlugin if you prefer; you want to add a field to all types, write a plugin.

Even without solving this, the work you have is valuable so I'd encourage you to get it merged, but factoring this in too would be great. I'd love to see it even more condensed, or even presented as some kind of decision tree. I was planning at one point to make a "choose your own adventure"-style documentation feature where you chose your route through the docs based on what you were trying to do and it'd take you to the relevant places.

[machineghost]

Please be careful of captialization: PostGraphile

Corrected

Remove extraneous words like "also",

Removed

make the links more descriptive

Done

be sure that the levels of heading are expected (don't jump from H2 to H4 without an H3)

Fixed

To be clear, makeExtendSchemaPlugin ...

Fixed, but ... after working with adding the nested field stuff, and realizing how much duplication there was, I thought that maybe I was taking wrong approach.

Going back to my core goal, I wanted to answer "I'm a user who doesn't know PostGraphile, and I want to get something done: how do I?". I don't think that requires spelling out every CRUD option, it just requires a short/simple page that I can skim in under a minute to figure out my options.

In that vein, what do you think of this restructuring? I swear this will be the last screenshot: whether you like it or want me to go back to what I had, my next step will be to push some code :)

[benjie]

Maybe something like this for the first three bullets:

• your database objects (tables, columns, constraints, grants, indexes, functions, views, ...)
• "smart tag" annotations (provided via database comments, a JSON tags file, or plugins)
• configuration parameters (including adding off the shelf or custom plugins)

This feels like it might be the right kind of direction, but I'd definitely want to do some heavy editorial on the text. For example, I don't agree that "the primary way to customize your API fields is to create or change the underlying database objects" - you should design a good database schema, one that will stand well in the stead of time, and then you should use PostGraphile customization facilities to customize the GraphQL schema to fit your needs, not the other way around. Of course, in a good database schema you'll already have indexes, foreign key and primary key constraints, GRANTs, etc, so a lot should just fall out of that automatically.

One other issue is that people get put off by "plugins", they think they're going to be really complex or abstract, so talking in terms of "schema extension and customization" in the headers might be preferred?

[machineghost]

I don't agree that "the primary way to customize your API fields is to create or change the underlying database objects" - you should design a good database schema ... in a good database schema you'll already have indexes, ...

Yeah, what I was trying to convey was just what you're saying: the whole point of Postgraphile is to build an API around your database. By definition then, the core way you're going to change that API (as your app grows over time) is by changing the underlying database. If you have a reviews endpoint, and you want to add a reviewers endpoint, you don't want to look at smart tags or extending the schema with JS ... you just want add a reviewers table (and a foreign key to the reviews table, and so on).

But if what I said didn't convey that, of course you can revise that or anything else I wrote: it's your site :) I just wrote this thing because you weren't writing it in the first place ;)

One other issue is that people get put off by "plugins", they think they're going to be really complex or abstract, so talking in terms of "schema extension and customization" in the headers might be preferred?

Plug-in is a pretty universal term; I have a hard time believing that software developers can't handle the concept of a plug-in.
I really think PostGraphile's documentation could be to blame.

Please, put the pitchfork down and hear me out! I'm going to be critical of the docs, but I truly do understand how much work and effort you put in to them. In many respects the PostGraphile docs are great, so please don't take this as me crapping on your work. But ... IMHO the plug-ins page is unintentionally hostile to users.

Try to imagine yourself not as Benjie, god of all things ending in -ile, but as a humble user trying to use a new library they have no context with. You just installed this thing, and now you're trying to accomplish something that wasn't in the tutorial. Already, you're one level of discomfort in, because there's nothing in the docs telling you how to do that (because this PR hasn't been merged).

There's also no "plug-in" page, but there is a Schema Plug-ins page ... is that the kind you want? You have no idea. But you're an intrepid dev, so you dive in ... and get to the very first sentence on the page:

PostGraphile's schema generator is built from a number of graphile-build plugins.

Great, so to understand what a plug-in is, I need to understand this entire other library called Graphile. I don't know what this Graphile thing is (I assume it's some related library because of the name) ... and now I need to go read its docs.

But again, our imaginary user presses ahead. They get to the page, scroll down an entire screen's worth, and finally get an answer:

A Graphile Build plugin is a simple object with a name, description, version and entries for any of the scopes it wishes to implement.

Well, they got an answer for "Graphile"; they're still not sure how exactly any of this relates to PostGraphile. So what do they do? They go back to the Postgraphile page, and just muddle through it.

It's no wonder people are:

put off by "plugins",

The first sentence of the docs tells them to go read another library, and then the third sentence tries to scare them away from using plug-ins.

If you're looking for an easy way to remove/rename things, check out [smart tags] https://postgraphile.org/postgraphile/next/smart-tags/).

So, I don't think you should abandon a universal concept like plug-ins. I just think you need wear your "I'm a new user" cap on that page. Imagine instead if the first sentence of the plug-ins page was:

A PostGraphile Build plugin is a simple object with basic fields, such as name, description, version, as well as function properties that allow it to interface with Postgraphile in various ways.

Then, it had the simplest example of a plug-in possible, maybe RootNamingPlugin (but something even simpler if possible).
Now, every dev smart enough to graduate a boot camp (ie. not a high bar) can instantly grok what a plug-in is (an object with meta data and hooks that change things) ... before they've even scrolled below the fold.

Then, you could say something to the effect of "there are these ____ hooks available: inflection (changes how API is generated from base objects), schema (manually control API schema), ..." and have sections for each type. But you could also open each session with something like

Most developers won't need to write schema plug-ins by hand, because the makeExtendSchemaPlugin is an easier approach ...

It's not that much a difference, but I think if you frame things properly, and provide basic info instead of sending users on a rabbit hunt, "plug-in" won't be such a dirty word ;)

With all that said, I'll update the page with the structure of:

• your database objects (tables, columns, constraints, grants, indexes, functions, views, ...)
• "smart tag" annotations (provided via database comments, a JSON tags file, or plugins)
• configuration parameters (including adding off the shelf or custom plugins)

and then get a PR in later today.

[benjie]

You're not wrong in what you say, but what I was referring to was that some people get put off by the term "plugins" before they even look at the documentation. For example, telling someone they can "extend the schema through plugins" is different to telling someone that they can "extend the schema with their own types and resolvers" even though under the hood the mechanics of the latter uses the former. With my "new user hat" on, I can see why: "plugin" sounds complicated, whereas types and resolvers are the things I was dealing with already in my previous solution, so I understand those. It's this "new user hat" that has inspired a lot of the changes in V5, for example unification of the plugins system (no more "schema plugins" vs "server plugins" - just "plugins" now), and later these improvements in core concepts will map to improvements of the documentation once the interfaces have settled. For example I'm planning to change makeExtendSchemaPlugin to just be called extendSchema, etc - the fact it's a plugin factory is irrelevant to most users, they just want to add stuff to their schema, and plugins: [extendSchema({typeDefs, plans})] likely makes more intuitive sense than plugins: [makeExtendSchemaPlugin({typeDefs, plans})] even though it's literally just a different name.

[benjie]

(By the way I really like the PR; I have some pending edits in it but got called into a meeting. Will update later today.)

[benjie]

Closing this in favour of graphile/crystal#2004