v0.4 Hosted Docs (WIP)

[matthewmueller]

Goals

1. You can effectively build documentation pages using Bud.

Deliverable

Bud has documentation live on https://livebud.com/ and a demo of the new Bud features that were built to make the documentation site possible.

Tasks

graph BT
classDef done fill:green,color:white,stroke:green;
classDef wip fill:#e3b241,stroke:#e3b241,color:#2e2000;
classDef x fill:red,stroke:red,color:white;
1(Publish Documentation)
2(Demo New Features)
3(Support Tailwind Plugin)
4(Support Custom Layouts)
5(Support Custom Error Pages)
6(Support Markdown Plugin)
8(Finish Documentation Design)
10(Build Documentation)
1-->2
10-->1
8-->10
4-->3
5-->3
4-->6
5-->6
6-->10
3-->10

[matthewmueller]

As I've been building out the documentation, I've been getting more real-world experience as a user of Bud. The following is a somewhat unstructured brain dump so please feel free to jump in anywhere and share your thoughts. I'm writing this primarily to share my thinking in the hopes that you will share your ideas too! I need your fresh perspective!

Before diving in, let me discuss the current Bud behavior.

Current UI Behavior

At a high-level, you define controllers that respond to requests and render views. You create controllers in the controller/ and views in the view/ directory. It looks like this:

controller/
  users/
    users.go
view/
  users/
    index.svelte
    show.svelte
    new.svelte
    edit.svelte

Bud uses file-based routing, so to render the view/users/show.svelte view, you'd issue a GET /users/:id request.

Svelte views are compiled by Bud into JS and loaded into a V8 VM. Then for each request, the controller's return value gets serialized into JSON and rendered to HTML inside the a V8 VM:

In psuedo-code that looks like this:

// on boot
const vm = new V8()
const show_post = readFile("posts/show.svelte.js") // svelte view compiled to server-side JS
vm.loadScript(show_post)

// per request
function show(request) {
  const post = controller.show(request.params.id)
  const html = vm.eval(`show_post.render(${JSON.stringify(post)})`)
  return response.html(html)
}

There are some perceived and real problems with this design. I'll start with the perceived problem:

Perceived Problem with Directory Structure

Rails and Laravel separate views and controllers into separate folders. This has the following benefits:

• Separation of concerns. A single controller could render many views.
• Frontend developers can focus on views, backend can focus on controllers.

But it also comes with a cost:

• More jumping around between folders, especially while iterating on a feature.

Most of the newer web frameworks choose to co-locate views with controllers and I'm wondering if I should change this in Bud. We could totally do:

routes/
  users/
    controller.go
    index.svelte
    show.svelte
    new.svelte
    edit.svelte

I tend to lean towards this for a couple reasons:

1. I haven't seen very many Rails codebases actually generate different views in a given controller action. They typically will redirect instead.
2. The directory structure of controllers and views mirror each other, yet are separate. In any other categorization system you'd merge those two folders together.
3. In my past Node.js projects, I tended to co-locate the controller route with the client-side JS. This required a custom build setup that pre-dates Webpack and even React!

What's holding me back is:

• I haven't worked on a large Rails or Laravel codebase, I might be missing some fundamental benefits of this separation.
• If I reflect on time spent making websites, I typically setup the controller, then spend a lot of time on views and occasionally hop back to controllers as data requirements come up. Since most of the time is spent in one or the other, maybe keeping them separate keeps the workspace slightly more tidy for the given task.

Real Problems with SSR in Go

1. The V8 VM is completely empty. There's no console.log, no fetch, etc. We're responsible for defining that ourselves and have minimally added important functionality so far. But what should go into this VM? Should we model it after the DOM? What about modeling it after the Cloudflare Worker or Deno runtime? We'll need to decide the runtime that Bud exposes to the V8 VM. All this is doable, but requires consideration.
2. Using a V8 VM to render views is computationally expensive. I did some basic benchmarks and while Go is about 2x faster than Node at serving requests, by using V8 within Go, Go is a tiny bit slower than Node.
3. All output data must be serializable before entering the VM. I'm not sure how you would have functions in your return object that call back into Go. This means everything needs to be computed upfront in the controller whether or not the code path is used in the view.
4. It's a bit strange to me that server-side computation first happens in Go, then gets passed off to Javascript for more server-side computation. Wouldn't it make more sense to do it all in Go or all in JS?

Possible Directions

Remix Integration

By having ownership over the runtime, we could offload views support to a framework like Remix. Remix's interface is an HTTP handler, so we could define a compatible runtime for Remix to run within similar to remix-cloudflare-workers, remix-deno or remix-node.

What I like about this idea is that I think Remix nailed the data flow. It's all in one file, it flows in one direction and there's clear boundaries between server-side and client-side.

Remix code example

// Load the component. Called server-side
export async function loader({ params })
  const { productId } = params;
  const product = await db.products.get(productid);
  return json(product)
}

// Rendered from params
export default function Route() {
  const product useLoaderData();
  return (
    <>
       <h2>Edit Product #{product.idk</h2>
       <Form method="post">
         <input type="text" ame="nme" defaultvalue={product.name} />
         <input type="number name="price" defaultValue={product.price} />
         <button type="submit">Save</button>
       </Form>
   </>
  )
}

// Respond to the form. Called server-side.
export async function action({ params, request }) {
  const { productid } parans;
  const form await request.formbata();
  const name = form.get("nane");
  const price form.get("price");
  const product = await db.products.edit({ id: productid, name, price: 3 });
  return redirect("/products/${product.id}");
}

And if you think about Go's core strength over Node.js, it's that Go has its own scheduler that takes advantage of multi-process architectures. Since Node is single-threaded, you can bring a Node.js server to its knees by doing anything computationally expensive in one of its handlers. This becomes increasingly difficult to avoid at scale and it's why packages like blocked were created at Segment. The industry has moved towards Serverless to remedy this design flaw, but then you're distributing your compute and you have additional problems you need to deal with like too many serverless functions hitting limited resources (like a database).

Bud could be one of many bases that power Remix. That's an interesting idea.

I have one major problem with this though. You're moving more and more code into Typescript/Javascript. The controllers would now be in JS and it's not clear why the models wouldn't be in JS too.

If we move Bud this way, Bud becomes more like the backend framework for a UI framework rather than a self-contained full-stack web framework.

The benefits:

• We narrow Bud's scope and build alongside the UI frameworks.

The cons:

• Not much control over the direction of the UI framework.
• Less clear boundaries between TS and Go. Likely more of the application logic will be in TS, while the infra dependencies will be in Go.

Stick with the Current Plan

Continue on the current course:

• Decide on the runtime API for the V8 VM and build out the runtime primarily in service of Bud.
• Add our own flavor of Svelte and React support. Add html/template support and HTML for simpler needs.
• Add support for custom layout.svelte and error.svelte pages.
• Support frame.svelte, which are like layouts but instead of overriding, they nest, so you could define your sidebar in a frame that is inherited by pages in subdirectories.

Problems:

• This is very much paving our own path. More time will be needed to experiment with what feels right.
• Unclear how to support data requirements in layouts, frames and errors. Maybe extend the controllers?

Re-focus on html/template support

SSR support has a lot of open questions and is going to require a lot of work to get just right. One thing I know is solid and is less likely to change will be the html/template support.

I've been wondering if I should pause SSR advancements, get html/template working, and first build out the framework using a simpler templating approach. Then when the framework is more broadly useful and the dust has settled a bit on the JS frameworks, we revisit the SSR support.

I've also been learning more about Turbo Streams and Stimulus and I must admit, while it's a strange idea at first, it really grows on you and fits quite well into the model of getting something working then incrementally improving with JS as time permits.

Pros:

• Will likely have a more complete framework faster
• Less design risk. html/template has been successfully used for websites like Gogs and go.dev
• Easy callbacks into Go for things like checking if a user is signed in or getting the primary key regardless of field name.
• Better performance

Cons:

• Probably less interesting to people. Svelte provides an impressive, productive environment for building UI.
• Not going to be as nice for building web apps. It'll build websites just fine.

Transform Svelte to html/template for the Server

This is the "out there" idea that I probably won't do now, but I wanted to share in case anyone gets a stroke of inspiration 😉

If you look at a Svelte view:

<script>
export let users = []
function onClick() {
  alert('hi')
}
</script>

<ul>
{#each users as user}
  <li on:click={onClick}>{user.name}</li>
{/each}
</ul>

It seems like there should be a way to render it without evaluating JS. Transform the view into a template render it, serve the HTML, and let the client-side Svelte attach the event handlers. Something like:

<ul>
{{range $user := $.Users}}
  <li>{{$user.Name}}</li>
{/range}
</ul>

I actually tried this at first. I had a project called Svelte to Go. The problem I ran into, is how do you transform variables that depend on the results of JS functions:

<script>
import translate from './german'
export let users = []
</script>

<ul>
{#each users as user}
  <li>{translate(user.name)}</li>
{/each}
</ul>

If we could come up with some reasonable heuristics for doing this transformation to Go's html/template templates that isn't overly restrictive, that would give us the benefits of Svelte without the costs of using V8 in Go.

---

Update: Many of these thoughts have been addressed in this update.

[barelyhuman]

my 2 cents ,

all are suggestions and can be ignored based on whatever makes sense to you

• vm needs to go
• use alpinejs instead, it's built for these kinda things
• folder structure can be left dynamic, and the controllers can take in a FS interface embedding the templates path would be the developers's decision, though the sensible default for it could be left as is (views folder)

These would be for the short term changes.

For the longer run, you could do a DSL for having the frontend and backend in the same file, but I'd pick this up a lot later

[austincollinpena]

I'd probably go with how most fullstack Go developers do things.

For example, if most fullstack Go developers already use React + Go, then that would probably have the biggest impact on the community.

Although Stimulus looks really cool!

[matthewmueller]

For example, if most fullstack Go developers already use React + Go, then that would probably have the biggest impact on the community.

That's a great point! I'm honestly not sure at the moment because there's not a whole lot of Go devs writing frontends, but if I had to guess it's about 50:50 text/template and client-side React. I could probably start getting a better picture using Github search though!

[omar391]

@matthewmueller Did you check https://inertiajs.com/? We could follow this approach!

[jfmario]

I'm in favor of the focus on html/template over SSR. Svelte is great even without SSR and can be easily incorporated by building the js and serving it. It might be helpful for bud to provide some opinionated scaffolding on how to do this.

I'm also looking into alpinejs as recommended by @barelyhuman . It looks like a good fit for this project too!

[vito]

My $.02: Svelte is a pretty killer part of Bud, and I'd be sad to see it replaced with html/template.

My biggest gripe with html/template is that it makes it hard to split templates into re-usable components. You can only pass one value to a {{template}} call, and you don't have a real language at your disposal so you can't construct arbitrary data structures to pass along. Svelte, to me, is a massive upgrade in keeping me unblocked and letting me refactor quickly or roll it back if it doesn't pan out.

Svelte also let me get rid of my Less CSS pipeline because having CSS scoped to the .svelte file by default eliminated so much complexity that I could just write much simpler CSS straight-up and trust the component to be portable. (I had to use :global here and there but the trade is worth it IMO.)

That being said, my frontend experience is somewhat limited. Bud was my intro to Svelte. This is more about Svelte > html/template and less about Svelte > *, though I do prefer it to React in my limited experience with both, since it feels much simpler.

re: Remix, I'm having a hard time imagining writing server code in JS. Part of the reason for using Bud in the first place is to have Go server-side. If it's not there, I'm not sure what Bud has to do with Go anymore. 🤔

[barelyhuman]

While I agree with most of this,
the main issue is the re-eval problem since that's pretty heavy for bigger apps and I doubt @matthewmueller can just ask users to keep their svelte components small and efficient,

if we can eliminate the dual evaluation of javascript, then I would also like to stick with svelte, but as it stands the easier one to implement while keeping speed on both sides would be something like alpinejs or If I can find something even lighter then that.

or a far fetched approach, how about connectors?

basically the connectors decide what the frontend tech would be, I mean it would make it modular and people could end of the day choose what works.

Though for proper optimization I'd like to keep the coupling since more context can be shared around but I guess connectors could be taken as an approach

[jfmario]

How do you envision alpine.js working? Would the bud server just cast the controller output to json and dump it into a top-level x-data tag, and let alpine take it from there on the client side?

[barelyhuman]

That's one way to do it but I'd instead prefer letting the developer decide on what goes where so the passed view is compiled like a normal go html template and the blocks can be replaced as needed.

https://github.com/barelyhuman/backend-go has a similar implementation

This gives control on both sides as to what works where.
it's not super client side so you still have enough stuff to handle interactions with whatever js you use on client
and the serialised data is accessible as normal.

so, for a serious go lang developer, nothing really changes.

might not be what you are working towards but just pointing it out

[matthewmueller]

Thanks for sharing your thoughts!

@vito

My $.02: Svelte is a pretty killer part of Bud, and I'd be sad to see it replaced with html/template.

Agreeeed.

My biggest gripe with html/template is that it makes it hard to split templates into re-usable components. You can only pass one value to a {{template}} call, and you don't have a real language at your disposal so you can't construct arbitrary data structures to pass along. Svelte, to me, is a massive upgrade in keeping me unblocked and letting me refactor quickly or roll it back if it doesn't pan out.

That's a great point! I haven't built any websites with text/template, but that makes perfect sense.

re: Remix, I'm having a hard time imagining writing server code in JS. Part of the reason for using Bud in the first place is to have Go server-side. If it's not there, I'm not sure what Bud has to do with Go anymore. 🤔

Bud would indeed play a much less significant role which would make me quite sad. I axed this idea :)

---

@barelyhuman

or a far fetched approach, how about connectors?

That's what I'm currently exploring in #225!

if we can eliminate the dual evaluation of javascript, then I would also like to stick with svelte, but as it stands the easier one to implement while keeping speed on both sides would be something like alpinejs or If I can find something even lighter then that.

See the update below, but Goja actually changes the equation entirely. Rendering JS to HTML essentially becomes a Go function and there's plenty of room to optimize the JS VM further.

https://github.com/barelyhuman/backend-go has a similar implementation

Thanks for sharing!

[matthewmueller]

Thank you so much for sharing your thoughts 🙌

To recap, the 3 things that were weighing on my mind:

1. Is it worth changing rendering solutions to simplify the internals and boost Bud's performance?
2. Where should we draw the line between the handoff from Go to JS?
3. Should we co-locate views and controllers?

Here's what I'm thinking now:

1. Is it worth changing rendering solutions to simplify the internals and boost Bud's performance?

I also find Svelte (or other virtual DOM frameworks) support to be one of Bud's killer features, but I understand that for simpler needs html and html/template are good choices.

I realized that most of my issues with supporting JS frameworks is that V8 is complex and CGO is slow, but there was an exciting new development last week. Goja got class support! I tested it out and it turns out to be the last ES2015 feature needed to render Svelte views into HTML. Using Goja for SSR is both operationally simpler and also faster! No more CGO! There's still going to be some JS gaps between what ESBuild transpiles down to and what Goja can evaluate, but those will close over time and we can help them there too. I'll be working on transitioning Bud to using Goja instead of V8 for v0.3.0.

As far as other rendering solutions, I'm trying to abstract out the view renderer right now and I'm thinking of exposing this publicly so you can bring your own view renderer. If this experiment works, I'm thinking of keeping Bud focused on just Svelte, but then adding text/template or React support is go get github.com/community/bud-react. I'm also personally excited to try out the HTML over the wire solutions, so I'd like to be able to experiment with them in the context of a Bud plugin.

2. Where should we draw the line between the handoff from Go to JS?

I've decided that the input needs to be serializable between Go and JS. JS to HTML is a pure transformation function. This keeps it simple & fast.

Then the JS VM remains focused on rendering and only implements a small set of global functions (e.g. fetch, setTimeout, console.log, etc). It does not have access to the request data, instead certain request data is provided as context to Svelte (and React). This is similar to what Sveltekit does with their $app/stores.

3. Should we co-locate views and controllers?

Still not sure about this one! Thoughts very much welcome!

---

Update: The roadmap above has been updated to reflect this direction. I also scoped it down to just having docs that maybe have a single top section for the landing page.

[aadog]

Maybe we can integrate it as a plugin in other frameworks, such as iris?

[matthewmueller]

Definitely aiming to be more like PHP development! Bud is very much inspired by Laravel, which was inspired by Rails.

Do not change the language of svelte

Yep, I'll be adding additional languages, not removing languages.

Maybe we can integrate it as a plugin in other frameworks, such as iris?

Definitely plan to use some of their libraries where it makes sense, but none of the other web frameworks in Go except for Buffalo aim to really be full-stack.

[aadog]

ssr is only responsible for rendering and compiling svelte pages, which can be changed to es6

[aadog]

Don't implement fetch, dom, render to html performance degrades too much, so it doesn't make sense

[aadog]

We just need the framework to be able to use the middleware and we don't need it to do anything else