New high-level architecture documentation

ARCHITECTURE.md

@@ -35,67 +35,3 @@ The web viewer can load `.rrd` files (just drag-drop them into the browser), or
 `.rrd` ("**R**e**r**un **D**ata") is just a bunch of log messages appended one after the other to a file.
 
 NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility. One version of Rerun will likely not be able to open an `.rrd` file generated by another Rerun version.
-
-
-## Crates
-In order to get an overview of our in-house crates and how they depend on each other, we recommend you run:
-
-```
-cargo install cargo-depgraph
-cargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > deps.png
-open deps.png
-```
-
-and:
-
-```
-cargo doc --no-deps --open
-```
-
-and then browse through the `re_*` crates.
-
-
-## Technologies we use
-### Apache Arrow
-[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md).
-
-In rust, we use the [`arrow2` crate](https://crates.io/crates/arrow2).
-
-### `wgpu`
-The Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web.
-
-We use the WebGL backend when compiling for web. Once WebGPU is available in most browsers, we can easily switch to it for a nice performance boost!
-
-We have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/re_renderer/README.md).
-
-### `egui`
-The GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode).
-
-We use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web.
-
-
-### Wasm
-Wasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.
-The Rerun Viewer can be compiled to Wasm and run in a browser.
-
-Threading support in Wasm is nascent, so care must we taken that we don't spawn any threads when compiling for `wasm32`.
-
-Wasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:
-* Access files
-* Read environment variables
-* Get the current time (use [`instant`](https://crates.io/crates/instant) instead)
-* Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)
-* etc
-
-
-## Immediate mode
-The Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch.
-
-In fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.
-
-The advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.
-
-Immediate mode is also a forcing function, forcing us to relentlessly optimize our code.
-This leads to a very responsive GUI, where there is no "hickups" when switching data source or doing time scrubbing.
-
-Of course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.

BUILD.md

@@ -4,7 +4,7 @@ This is a guide to how to build Rerun.
 
 ## See also
 * [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK
-* [`ARCHITECTURE.md`](ARCHITECTURE.md)
+* [Architecture](docs/content/architecture.md)
 * [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)
 * [`CODE_STYLE.md`](CODE_STYLE.md)
 * [`CONTRIBUTING.md`](CONTRIBUTING.md)

CODE_STYLE.md

@@ -1,7 +1,7 @@
 # Rerun code style
 
 ## See also
-* [`ARCHITECTURE.md`](ARCHITECTURE.md)
+* [Architecture](docs/content/architecture.md)
 * [`BUILD.md`](BUILD.md)
 * [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)
 * [`CONTRIBUTING.md`](CONTRIBUTING.md)

CONTRIBUTING.md

@@ -3,7 +3,7 @@ This is written for anyone who wants to contribute to the Rerun repository.
 
 
 ## See also
-* [`ARCHITECTURE.md`](ARCHITECTURE.md)
+* [Architecture](docs/content/architecture.md)
 * [`BUILD.md`](BUILD.md)
 * [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK
 * [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)
@@ -49,6 +49,14 @@ To get an overview of the crates, read their documentation with:
 cargo doc --no-deps --open
 ```
 
+You can see how they connect by plotting their dependency tree with:
+
+```
+cargo install cargo-depgraph
+cargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > deps.png
+open deps.png
+```
+
 To learn about the viewer, run:
 
 ```
@@ -74,3 +82,17 @@ You can set up [`sccache`](https://github.com/mozilla/sccache) to speed up re-co
 
 ### Other
 You can view higher log levels with `export RUST_LOG=debug` or `export RUST_LOG=trace`.
+
+## Wasm
+Wasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.
+The Rerun Viewer can be compiled to Wasm and run in a browser.
+
+Threading support in Wasm is nascent, so care must we taken that we don't spawn any threads when compiling for `wasm32`.
+
+Wasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:
+* Access files
+* Read environment variables
+* Get the current time (use [`instant`](https://crates.io/crates/instant) instead)
+* Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)
+* etc
+

README.md

@@ -75,7 +75,7 @@ The commercial product targets the needs specific to teams that build and run co
 
 
 # Development
-* [`ARCHITECTURE.md`](ARCHITECTURE.md)
+* [Architecture](docs/content/architecture.md)
 * [`BUILD.md`](BUILD.md)
 * [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK
 * [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)

RELEASES.md

@@ -3,7 +3,7 @@ This document describes the current release and versioning strategy. This strate
 
 
 ## See also
-* [`ARCHITECTURE.md`](ARCHITECTURE.md)
+* [Architecture](docs/content/architecture.md)
 * [`BUILD.md`](BUILD.md)
 * [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)
 * [`CODE_STYLE.md`](CODE_STYLE.md)

docs/README.md

@@ -1,4 +1,4 @@
-This is the high-level documentation for rerun that is hosted at https://www.rerun.io/docs
+This is the high-level documentation for rerun that is hosted at <https://www.rerun.io/docs>.
 
 ## Other documentation
 API-level documentation is built from source-code. Python lives at [rerun_py](https://github.com/rerun-io/rerun/tree/main/rerun_py) and Rust in the individual [crates](https://github.com/rerun-io/rerun/tree/main/crates).

docs/content/architecture.md

@@ -0,0 +1,69 @@
+---
+title: Architecture
+order: -1
+---
+
+## Overview
+
+Rerun provides a fast and extensible data visualization infrastructure adaptable to all kinds of data (2D, 3D, time series, etc.) and targets (native and web).
+Our stack is designed to be fast, flexible, extensible, and easily adaptable to any workflow, from local development to cloud-based deployment.
+
+Use one of our logging SDKs to produce log data that is then either live-streamed to the Rerun Viewer, or stored in a file for later viewing.
+
+<img src="https://github.com/rerun-io/rerun/assets/49431240/3a0e3b1f-aa71-4c19-84c4-f4626ce4499c" alt="overview" width="90%" />
+
+The Rerun Viewer is written in [Rust](https://www.rust-lang.org), which we chose for its performance, its memory safety, and cross-platform compatibility.
+
+For web targets, the Rerun Viewer is compiled to [WebAssembly](https://webassembly.org), allowing it to [run in a browser](https://demo.rerun.io/) with nearly no performeance compromise, and to be embedded anywhere you can put a web-view [(e.g. in Jupyter Notebook)](https://colab.research.google.com/drive/1R9I7s4o6wydQC_zkybqaSRFTtlEaked_).
+
+Rerun is open-source, being built in the open [on GitHub](https://github.com/rerun-io/rerun).
+
+## Architecture Overview
+
+
+<img src="https://github.com/rerun-io/rerun/assets/49431240/0b9cd101-e3a5-400f-acf4-d85ef72e2406" width="500px" alt="The Rerun stack"/>
+
+
+Data is ingested using our Python and/or Rust **SDKs** (C++ coming soon).
+The SDKs are code-generated from a simple [IDL](https://en.wikipedia.org/wiki/IDL_(programming_language)), making them easy to extend in the future.
+Data is serialized in [Apache Arrow](https://arrow.apache.org/overview/) format for language-independent, cross-platform compatibility.
+
+The **Communication Layer** abstract the transport layers and enables the Rerun Viewer to ingest data from various sources, including files, TCP, and WebSocket.
+
+The **Data Layer** provides a queryable store for arbitrary structured time-dependant data.
+Internally, the data model is implemented as a special purpose [entity-component-system](https://en.wikipedia.org/wiki/Entity_component_system) where time is a first-class citizen.
+
+The **Visualization Layer** is made of composable, extensible collection building blocks covering 2D and 3D renderers, plotting widgets, and textual data display.
+We have written our own high-level, purpose-built renderer on top of [wgpu](https://wgpu.rs), which offers high-performance cross-platform compatibility for native and web targets.
+
+Our **GUI** is built upon our CTO's [egui](https://www.egui.rs/) [immediate mode UI](https://github.com/emilk/egui#why-immediate-mode) framework.
+It's designed to be offer a reactive, user-friendly and customizable interface that's easy to deploy to both native and web targets.
+
+
+## Extensibility
+
+_TODO_
+
+Users can extend the builtin datatypes by logging arbitrary Arrow data.
+
+
+<!--
+## Immediate mode
+The Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/).
+This means that each frame the entire GUI is being laid out from scratch.
+
+In fact, the whole of the Rerun Viewer is written in an immediate mode style.
+Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.
+
+The advantage of immediate mode is that is removes all state management.
+There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.
+
+Immediate mode is also a forcing function, pressuring us to relentlessly optimize our code.
+This leads to a very responsive GUI, where there is no "hickups" when switching data source or doing time scrubbing.
+
+Of course, this will only take us so far.
+In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame.
+We also plan on doing larger operation in background threads.
+This will be necessary in order to support viewing large datasets, e.g. several million points.
+The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.
+-->

docs/content/development/architecture.md

@@ -1,5 +1,5 @@
 ---
 title: Architecture
 order: 0
-redirect: https://github.com/rerun-io/rerun/blob/main/ARCHITECTURE.md
+redirect: ../architecture.md
 ---

docs/content/reference/about.md

@@ -23,5 +23,5 @@ We depend on a number of third party libraries, most notably:
 * [PyO3](https://github.com/PyO3/pyo3) for python bindings
 
 If you want to learn more about the different parts of the SDK & viewer and how they work, check out
-[this architecture overview](https://github.com/rerun-io/rerun/blob/latest/ARCHITECTURE.md)
+[this architecture overview](../architecture.md)
 for an introduction.