Document file system object content addressing

In addition: - Take the opportunity to add a bunch more missing hyperlinks, too. - Remove some glossary entries that are now subsumed by dedicated pages. We used to not be able to do this without breaking link fragments, but now we can, so pick up where we left off. Co-authored-by: Robert Hensing <[email protected]>
NixOS · Apr 15, 2024 · 0a6c547 · 0a6c547
1 parent 03eb4f7
commit 0a6c547
Show file tree

Hide file tree

Showing 34 changed files with 230 additions and 105 deletions.
diff --git a/doc/manual/redirects.js b/doc/manual/redirects.js
@@ -247,7 +247,7 @@ const redirects = {
  "gloss-closure": "glossary.html#gloss-closure",
  "gloss-derivation": "glossary.html#gloss-derivation",
  "gloss-deriver": "glossary.html#gloss-deriver",
- "gloss-nar": "glossary.html#gloss-nar",
+ "gloss-nar": "store/file-system-object/content-address.html#serial-nix-archive",
  "gloss-output-path": "glossary.html#gloss-output-path",
  "gloss-profile": "glossary.html#gloss-profile",
  "gloss-reachable": "glossary.html#gloss-reachable",
@@ -362,6 +362,10 @@ const redirects = {
  "glossary.html": {
  "gloss-local-store": "store/types/local-store.html",
  "gloss-chroot-store": "store/types/local-store.html",
+ "gloss-file-system-object": "store/file-system-object.html",
+ "gloss-store-object": "store/file-system-object.html",
+ "gloss-store-path": "store/file-system-path.html",
+ "gloss-nar": "store/file-system-object/content-address.html#serial-nix-archive",
  },
 };
 

diff --git a/doc/manual/src/SUMMARY.md.in b/doc/manual/src/SUMMARY.md.in
@@ -18,6 +18,7 @@
  - [Uninstalling Nix](installation/uninstall.md)
 - [Nix Store](store/index.md)
  - [File System Object](store/file-system-object.md)
+ - [Content-Addressing File System Objects](store/file-system-object/content-address.md)
  - [Store Object](store/store-object.md)
  - [Store Path](store/store-path.md)
  - [Store Types](store/types/index.md)

diff --git a/doc/manual/src/command-ref/nix-collect-garbage.md b/doc/manual/src/command-ref/nix-collect-garbage.md
@@ -74,4 +74,4 @@ $ nix-collect-garbage -d
 ```
 
 [profiles]: @docroot@/command-ref/files/profiles.md
-[store objects]: @docroot@/glossary.md#gloss-store-object
+[store objects]: @docroot@/store/store-object.md
diff --git a/doc/manual/src/command-ref/nix-env/delete-generations.md b/doc/manual/src/command-ref/nix-env/delete-generations.md
@@ -49,7 +49,7 @@ Periodically deleting old generations is important to make garbage collection
 effective.
 The is because profiles are also garbage collection roots — any [store object] reachable from a profile is "alive" and ineligible for deletion.
 
-[store object]: @docroot@/glossary.md#gloss-store-object
+[store object]: @docroot@/store/store-object.md
 
 {{#include ./opt-common.md}}
 

diff --git a/doc/manual/src/command-ref/nix-env/install.md b/doc/manual/src/command-ref/nix-env/install.md
@@ -17,7 +17,7 @@
 The install operation creates a new user environment.
 It is based on the current generation of the active [profile](@docroot@/command-ref/files/profiles.md), to which a set of [store paths] described by *args* is added.
 
-[store paths]: @docroot@/glossary.md#gloss-store-path
+[store paths]: @docroot@/store/store-path.md
 
 The arguments *args* map to store paths in a number of possible ways:
 

diff --git a/doc/manual/src/command-ref/nix-hash.md b/doc/manual/src/command-ref/nix-hash.md
@@ -20,16 +20,21 @@ an example.
 The hash is computed over a *serialisation* of each path: a dump of
 the file system tree rooted at the path. This allows directories and
 symlinks to be hashed as well as regular files. The dump is in the
-*NAR format* produced by [`nix-store
+*[Nix Archive (NAR)][Nix Archive] format* produced by [`nix-store
 --dump`](@docroot@/command-ref/nix-store/dump.md). Thus, `nix-hash path`
 yields the same cryptographic hash as `nix-store --dump path |
 md5sum`.
 
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
+
 # Options
 
  - `--flat`\
- Print the cryptographic hash of the contents of each regular file
- *path*. That is, do not compute the hash over the dump of *path*.
+ Print the cryptographic hash of the contents of each regular file *path*.
+ That is, instead of computing
+ the hash of the [Nix Archive (NAR)](@docroot@/store/file-system-object/content-address.md#serial-nix-archive) of *path*,
+ just [directly hash]((@docroot@/store/file-system-object/content-address.md#serial-flat) *path* as is.
+ This requires *path* to resolve to a regular file rather than directory.
  The result is identical to that produced by the GNU commands
  `md5sum` and `sha1sum`.
 

diff --git a/doc/manual/src/command-ref/nix-store/dump.md b/doc/manual/src/command-ref/nix-store/dump.md
@@ -1,14 +1,14 @@
 # Name
 
-`nix-store --dump` - write a single path to a Nix Archive
+`nix-store --dump` - write a single path to a [Nix Archive]
 
 ## Synopsis
 
 `nix-store` `--dump` *path*
 
 ## Description
 
-The operation `--dump` produces a NAR (Nix ARchive) file containing the
+The operation `--dump` produces a [NAR (Nix ARchive)][Nix Archive] file containing the
 contents of the file system tree rooted at *path*. The archive is
 written to standard output.
 
@@ -33,6 +33,8 @@ but not other types of files (such as device nodes).
 A Nix archive can be unpacked using `nix-store
 --restore`.
 
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
+
 {{#include ./opt-common.md}}
 
 {{#include ../opt-common.md}}

diff --git a/doc/manual/src/command-ref/nix-store/export.md b/doc/manual/src/command-ref/nix-store/export.md
@@ -1,6 +1,6 @@
 # Name
 
-`nix-store --export` - export store paths to a Nix Archive
+`nix-store --export` - export store paths to a [Nix Archive]
 
 ## Synopsis
 
@@ -11,14 +11,16 @@
 The operation `--export` writes a serialisation of the specified store
 paths to standard output in a format that can be imported into another
 Nix store with `nix-store --import`. This is like `nix-store
---dump`, except that the NAR archive produced by that command doesn’t
+--dump`, except that the [Nix Archive (NAR)][Nix Archive] produced by that command doesn’t
 contain the necessary meta-information to allow it to be imported into
 another Nix store (namely, the set of references of the path).
 
 This command does not produce a *closure* of the specified paths, so if
 a store path references other store paths that are missing in the target
 Nix store, the import will fail.
 
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
+
 {{#include ./opt-common.md}}
 
 {{#include ../opt-common.md}}

diff --git a/doc/manual/src/command-ref/nix-store/import.md b/doc/manual/src/command-ref/nix-store/import.md
@@ -1,6 +1,8 @@
 # Name
 
-`nix-store --import` - import Nix Archive into the store
+`nix-store --import` - import [Nix Archive] into the store
+
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
 
 # Synopsis
 

diff --git a/doc/manual/src/command-ref/nix-store/optimise.md b/doc/manual/src/command-ref/nix-store/optimise.md
@@ -12,7 +12,7 @@ The operation `--optimise` reduces Nix store disk space usage by finding
 identical files in the store and hard-linking them to each other. It
 typically reduces the size of the store by something like 25-35%. Only
 regular files and symlinks are hard-linked in this manner. Files are
-considered identical when they have the same NAR archive serialisation:
+considered identical when they have the same [Nix Archive (NAR)][Nix Archive] serialisation:
 that is, regular files must have the same contents and permission
 (executable or non-executable), and symlinks must have the same
 contents.
@@ -38,3 +38,4 @@ hashing files in `/nix/store/qhqx7l2f1kmwihc9bnxs7rc159hsxnf3-gcc-4.1.1'
 there are 114486 files with equal contents out of 215894 files in total
 ```
 
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
diff --git a/doc/manual/src/command-ref/nix-store/realise.md b/doc/manual/src/command-ref/nix-store/realise.md
@@ -25,11 +25,11 @@ Each of *paths* is processed as follows:
 
 If no substitutes are available and no store derivation is given, realisation fails.
 
-[store paths]: @docroot@/glossary.md#gloss-store-path
+[store paths]: @docroot@/store/store-path.md
 [valid]: @docroot@/glossary.md#gloss-validity
 [store derivation]: @docroot@/glossary.md#gloss-store-derivation
 [output paths]: @docroot@/glossary.md#gloss-output-path
-[store objects]: @docroot@/glossary.md#gloss-store-object
+[store objects]: @docroot@/store/store-object.md
 [closure]: @docroot@/glossary.md#gloss-closure
 [substituters]: @docroot@/command-ref/conf-file.md#conf-substituters
 [content-addressed derivations]: @docroot@/contributing/experimental-features.md#xp-feature-ca-derivations

diff --git a/doc/manual/src/command-ref/nix-store/restore.md b/doc/manual/src/command-ref/nix-store/restore.md
@@ -8,9 +8,11 @@
 
 ## Description
 
-The operation `--restore` unpacks a NAR archive to *path*, which must
+The operation `--restore` unpacks a [Nix Archive (NAR)][Nix Archive] to *path*, which must
 not already exist. The archive is read from standard input.
 
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
+
 {{#include ./opt-common.md}}
 
 {{#include ../opt-common.md}}

diff --git a/doc/manual/src/contributing/documentation.md b/doc/manual/src/contributing/documentation.md
@@ -147,7 +147,7 @@ Please observe these guidelines to ease reviews:
  ```
  A [store object] contains a [file system object] and [references] to other store objects.
 
- [store object]: @docroot@/glossary.md#gloss-store-object
+ [store object]: @docroot@/store/store-object.md
  [file system object]: @docroot@/architecture/file-system-object.md
  [references]: @docroot@/glossary.md#gloss-reference
  ```

diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md
@@ -1,5 +1,25 @@
 # Glossary
 
+- [content address]{#gloss-content-address}
+
+ A
+ [*content address*](https://en.wikipedia.org/wiki/Content-addressable_storage)
+ is a secure way to reference immutable data.
+ The reference is calculated directly from the content of the data being referenced, which means the reference is
+ [*tamper proof*](https://en.wikipedia.org/wiki/Tamperproofing)
+ --- variations of the data should always calculate to distinct content addresses.
+
+ Content-addressing is a key technique used by build systems such as Nix to maintain purity and [cache coherence](https://en.wikipedia.org/wiki/Cache_coherence).
+ For how Nix uses content addresses, see:
+
+ - [Content-Addressing File System Objects](@docroot@/store/file-system-object/content-address.md)
+ - [content-addressed store object](#gloss-content-addressed-store-object)
+ - [content-addressed derivation](#gloss-content-addressed-derivation)
+
+ Software Heritage's writing on [*Intrinsic and Extrinsic identifiers*](https://www.softwareheritage.org/2020/07/09/intrinsic-vs-extrinsic-identifiers) is also a good introduction to the value of content-addressing over other referencing schemes.
+
+ Besides content addressing, the Nix store also uses [input addressing](#gloss-input-addressed-store-object).
+
 - [derivation]{#gloss-derivation}
 
  A description of a build task. The result of a derivation is a
@@ -74,36 +94,7 @@
  over the network, but cannot host builds. Examples of binary caches
  include S3 buckets and the [NixOS binary cache](https://cache.nixos.org).
 
-- [store path]{#gloss-store-path}
-
- The location of a [store object] in the file system, i.e., an immediate child of the Nix store directory.
-
- > **Example**
- >
- > `/nix/store/a040m110amc4h71lds2jmr8qrkj2jhxd-git-2.38.1`
-
- See [Store Path](@docroot@/store/store-path.md) for details.
-
- [store path]: #gloss-store-path
-
-- [file system object]{#gloss-file-system-object}
-
- The Nix data model for representing simplified file system data.
-
- See [File System Object](@docroot@/store/file-system-object.md) for details.
-
- [file system object]: #gloss-file-system-object
-
-- [store object]{#gloss-store-object}
-
- Part of the contents of a [store].
-
- A store object consists of a [file system object], [references][reference] to other store objects, and other metadata.
- It can be referred to by a [store path].
-
- See [Store Object](@docroot@/store/index.md#store-object) for details.
-
- [store object]: #gloss-store-object
+ [store path]: @docroot@/store/store-path.md
 
 - [IFD]{#gloss-ifd}
 
@@ -252,13 +243,6 @@
 
  See [installables](./command-ref/new-cli/nix.md#installables) for [`nix` commands](./command-ref/new-cli/nix.md) (experimental) for details.
 
-- [NAR]{#gloss-nar}
-
- A *N*ix *AR*chive. This is a serialisation of a path in the Nix
- store. It can contain regular files, directories and symbolic
- links. NARs are generated and unpacked using `nix-store --dump`
- and `nix-store --restore`.
-
 - [`∅`]{#gloss-emtpy-set}
 
  The empty set symbol. In the context of profile history, this denotes a package is not present in a particular version of the profile.

diff --git a/doc/manual/src/language/advanced-attributes.md b/doc/manual/src/language/advanced-attributes.md
@@ -199,19 +199,23 @@ Derivations can declare some infrequently used optional attributes.
  The `outputHashMode` attribute determines how the hash is computed.
  It must be one of the following two values:
 
- - `"flat"`\
- The output must be a non-executable regular file. If it isn’t,
- the build fails. The hash is simply computed over the contents
- of that file (so it’s equal to what Unix commands like
- `sha256sum` or `sha1sum` produce).
+ <!-- FIXME link to store object content-addressing not file system object content addressing once we have the page for that. -->
+
+ - `"flat"`
+
+ The output must be a non-executable regular file; if it isn’t, the build fails.
+ The hash is
+ [simply computed over the contents of that file](@docroot@/store/file-system-object/content-address.md#serial-flat)
+ (so it’s equal to what Unix commands like `sha256sum` or `sha1sum` produce).
 
  This is the default.
 
- - `"recursive"` or `"nar"`\
- The hash is computed over the [NAR archive](@docroot@/glossary.md#gloss-nar) dump of the output
- (i.e., the result of [`nix-store --dump`](@docroot@/command-ref/nix-store/dump.md)). In
- this case, the output can be anything, including a directory
- tree.
+ - `"recursive"` or `"nar"`
+
+ The hash is computed over the
+ [Nix Archive (NAR)](@docroot@/store/file-system-object/content-address.md#serial-nix-archive)
+ dump of the output (i.e., the result of [`nix-store --dump`](@docroot@/command-ref/nix-store/dump.md)).
+ In this case, the output can be anything, including a directory tree.
 
  `"recursive"` is the traditional way of indicating this,
  and is supported since 2005 (virtually the entire history of Nix).
@@ -303,7 +307,7 @@ Derivations can declare some infrequently used optional attributes.
  [`disallowedReferences`](#adv-attr-disallowedReferences) and [`disallowedRequisites`](#adv-attr-disallowedRequisites),
  the following attributes are available:
 
- - `maxSize` defines the maximum size of the resulting [store object](@docroot@/glossary.md#gloss-store-object).
+ - `maxSize` defines the maximum size of the resulting [store object](@docroot@/store/store-object.md).
  - `maxClosureSize` defines the maximum size of the output's closure.
  - `ignoreSelfRefs` controls whether self-references should be considered when
  checking for allowed references/requisites.

diff --git a/doc/manual/src/language/derivations.md b/doc/manual/src/language/derivations.md
@@ -17,7 +17,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
  A symbolic name for the derivation.
  It is added to the [store path] of the corresponding [store derivation] as well as to its [output paths](@docroot@/glossary.md#gloss-output-path).
 
- [store path]: @docroot@/glossary.md#gloss-store-path
+ [store path]: @docroot@/store/store-path.md
 
  > **Example**
  >
@@ -141,7 +141,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
 
  By default, a derivation produces a single output called `out`.
  However, derivations can produce multiple outputs.
- This allows the associated [store objects](@docroot@/glossary.md#gloss-store-object) and their [closures](@docroot@/glossary.md#gloss-closure) to be copied or garbage-collected separately.
+ This allows the associated [store objects](@docroot@/store/store-object.md) and their [closures](@docroot@/glossary.md#gloss-closure) to be copied or garbage-collected separately.
 
  > **Example**
  >

diff --git a/doc/manual/src/language/import-from-derivation.md b/doc/manual/src/language/import-from-derivation.md
@@ -2,9 +2,9 @@
 
 The value of a Nix expression can depend on the contents of a [store object].
 
-[store object]: @docroot@/glossary.md#gloss-store-object
+[store object]: @docroot@/store/store-object.md
 
-Passing an expression `expr` that evaluates to a [store path](@docroot@/glossary.md#gloss-store-path) to any built-in function which reads from the filesystem constitutes Import From Derivation (IFD):
+Passing an expression `expr` that evaluates to a [store path](@docroot@/store/store-path.md) to any built-in function which reads from the filesystem constitutes Import From Derivation (IFD):
 
 - [`import`](./builtins.md#builtins-import)` expr`
 - [`builtins.readFile`](./builtins.md#builtins-readFile)` expr`

diff --git a/doc/manual/src/language/operators.md b/doc/manual/src/language/operators.md
@@ -128,7 +128,7 @@ The result is a string.
 > The file or directory at *path* must exist and is copied to the [store].
 > The path appears in the result as the corresponding [store path].
 
-[store path]: @docroot@/glossary.md#gloss-store-path
+[store path]: @docroot@/store/store-path.md
 [store]: @docroot@/glossary.md#gloss-store
 
 [String and path concatenation]: #string-and-path-concatenation

diff --git a/doc/manual/src/language/string-interpolation.md b/doc/manual/src/language/string-interpolation.md
@@ -107,9 +107,9 @@ An expression that is interpolated must evaluate to one of the following:
 
 A string interpolates to itself.
 
-A path in an interpolated expression is first copied into the Nix store, and the resulting string is the [store path] of the newly created [store object](@docroot@/glossary.md#gloss-store-object).
+A path in an interpolated expression is first copied into the Nix store, and the resulting string is the [store path] of the newly created [store object](@docroot@/store/store-object.md).
 
-[store path]: @docroot@/glossary.md#gloss-store-path
+[store path]: @docroot@/store/store-path.md
 
 > **Example**
 >

diff --git a/doc/manual/src/language/values.md b/doc/manual/src/language/values.md
@@ -113,7 +113,7 @@
  For example, assume you used a file path in an interpolated string during a `nix repl` session.
  Later in the same session, after having changed the file contents, evaluating the interpolated string with the file path again might not return a new [store path], since Nix might not re-read the file contents.
 
- [store path]: @docroot@/glossary.md#gloss-store-path
+ [store path]: @docroot@/store/store-path.md
 
  Paths can include [string interpolation] and can themselves be [interpolated in other expressions].
 

diff --git a/doc/manual/src/protocols/json/store-object-info.md b/doc/manual/src/protocols/json/store-object-info.md
@@ -28,9 +28,9 @@ Info about a [store object].
 
  Content address of this store object's file system object, used to compute its store path.
 
-[store path]: @docroot@/glossary.md#gloss-store-path
+[store path]: @docroot@/store/store-path.md
 [file system object]: @docroot@/store/file-system-object.md
-[Nix Archive]: @docroot@/glossary.md#gloss-nar
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
 
 ## Impure fields
 

diff --git a/doc/manual/src/protocols/nix-archive.md b/doc/manual/src/protocols/nix-archive.md
@@ -1,9 +1,10 @@
 # Nix Archive (NAR) format
 
-This is the complete specification of the Nix Archive format.
+This is the complete specification of the [Nix Archive] format.
 The Nix Archive format closely follows the abstract specification of a [file system object] tree,
 because it is designed to serialize exactly that data structure.
 
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#nix-archive
 [file system object]: @docroot@/store/file-system-object.md
 
 The format of this specification is close to [Extended Backus–Naur form](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form), with the exception of the `str(..)` function / parameterized rule, which length-prefixes and pads strings.