Try out beta Composition Functions!

[negz]

In Crossplane v1.14 (coming late October) we've made big changes to the Composition Functions architecture, to make Functions more scalable. We're also investing in the developer experience to make them easier to build and use.

Leading up to the release, we'd love for you to try building a Function. This will help us identify and fix rough edges. The developer experience is still a rough MVP right now. In particular we've only built support for Go Functions right now - other languages such as Python will be supported after the v1.14 release.

The readme of https://github.com/crossplane/function-template-go is the best place to get started. If you'd like a more in-depth example you can watch me build a Function at https://youtu.be/otwg-bO757A?feature=shared&t=60. Here's the Functions we've built so far:

• https://github.com/crossplane-contrib/function-patch-and-transform
• https://github.com/crossplane-contrib/function-auto-ready
• https://github.com/crossplane-contrib/function-dummy
• https://github.com/stevendborrelli/function-add-k8s-labels-annotations
• https://github.com/turkenh/function-tee

If you write a Function, please link it here! If you struggle, please use this thread for help and I'll try respond. Please feel free also to suggest ideas for Functions. I'm unlikely to build them myself, but they may help inspire someone else in the community.

Thanks!

P.S. If you don't feel up to building a Function, another thing that would help a lot is testing out https://github.com/crossplane-contrib/function-patch-and-transform. We want this Function to be able to do everything "native" patch-and-transform style Composition can do. It would be great if you could help validate it with some of your more complex Compositions. See #4639 (comment) for info on how to try it out.

[negz]

Also, please join #sig-composition-functions on Crossplane Slack to follow along with our progress!

[negz]

Here's a proposed spec for how Functions should work: #4640

If this is accepted it will probably make sense to link from the template repo(s).

[humoflife]

CIDR subnet calculation is used by many Terraform users. There is 'old style' Crossplane XFN code for a HashiCorp compatible implementation that could be converted to the current way of writing functions.

[stevendborrelli]

Here is a function to add k8s labels and annotations to resources: https://github.com/stevendborrelli/function-add-k8s-labels-annotations

[negz]

Nice @stevendborrelli! How was writing your first beta Function? Any thoughts or feedback?

[stevendborrelli]

It was pretty easy, I think being able to test locally would help a lot. I ended up using https://github.com/phisco/function-runtime-exec/tree/main and turning off MTLS to do some local debugging. Having sample json/yaml files and an easy way to test locally could be helpful.

Also in setting up testing in fn_test.go, I haven't figured out yet the right data structures to properly return a state:

		Results: []*fnv1beta1.Result{
						{
							//State:    impl.MessageState{}, //Desired:  resource.MustStructObject{`{}`},
							Severity: fnv1beta1.Severity_SEVERITY_NORMAL,
							Message:  "Kubernetes Labels and Annotations Added Successfully",
						},

[stevendborrelli]

Figured it out, it was for an empty state:

rsp: &fnv1beta1.RunFunctionResponse{
					Desired: &fnv1beta1.State{},
					Meta:    &fnv1beta1.ResponseMeta{Tag: "hello", Ttl: durationpb.New(response.DefaultTTL)},
					Results: []*fnv1beta1.Result{
						{
							Severity: fnv1beta1.Severity_SEVERITY_NORMAL,
							Message:  "Kubernetes Labels and Annotations Added Successfully",
						},
					},
				},

[negz]

I ended up using https://github.com/phisco/function-runtime-exec/tree/main and turning off MTLS to do some local debugging.

You mean like using them in a pipeline - function-runtime-exec to return some composed resources, then pass that to your Function? I've been doing something similar using https://github.com/crossplane-contrib/function-dummy/ - see for example https://github.com/crossplane-contrib/function-auto-ready/tree/main/examples.

[negz]

One idea that comes up a lot is a Function that would allow you to express Composition logic using everyone's favorite hot new configuration language - CUE. 😉 No one is actively planning to work on this to my knowledge, but it would be a valuable contribution if someone wanted to try.

[negz]

This week I put together https://github.com/negz/xrender.

xrender is a little like helm template render, but for Crossplane. It can locally render any XR and Composition that uses Composition Functions. I hope this will help 'shift left' the feedback loop on Composition authoring.

xrender runs Composition Functions on your laptop to render a Composition without needing kind or any Kubernetes cluster. By default it runs the Functions in Docker, but you can also run a Function in "development mode" (i.e. using go run) to test out your Functions while you build them.

We intend to make xrender a subcommand of an official Crossplane CLI (e.g. xp render xr.yaml...) for Crossplane v1.14.

[turkenh]

I prototyped function-tee this morning :)

It is similar to the tee command in Unix. It outputs the input request it receives into a ConfigMap, and then passes it through to the next Function in the pipeline. I am imagining such a function would only be helpful for debugging purposes where you want to see what is going in and out at different stages of the pipeline.

It is also an interesting one, because it creates (actually desires) a ConfigMap wrapped as a provider-kubernetes Object. So the function has a dependency to provider-kubernetes but doesn't need any additional configuration to be able to write ConfigMaps since providers has already RBAC to write them by default.

Feedback on Development:

I used xrender with the development runtime throughout the whole implementation and it was a fantastic experience. Being able to build the function iteratively and ability to debug whenever needed was awesome. We should definitely replace debugging instructions here with xrender.

A huge shout to @negz making this available so quick but still flawless 👏

I experienced the following relatively small issues but none of them was blocking:

• xrender failed if image not available locally: It requires the function image to be available locally crossplane-contrib/xrender#8
• Some packaging issues when I want to test my function in cluster, which turned out to be due to lack of newline at the end of package.yaml 🤯 These are the errors I got from the package manager until I figure it out:
  • cannot parse package contents: yaml: line 8: did not find expected key
  • linting package contents failed: not exactly one package meta type
• Looks like we don't have multi-arch support with the build steps in the template. So my function built on my M1 Mac would not run on an x86_64 cluster I assume.

[negz]

I'm currently trying to put https://github.com/crossplane-contrib/function-patch-and-transform through its paces. This Function implements contemporary P&T style Composition... but as a Function. I'd love it if it could handle say 90%¹ of Compositions in time for v1.14. This way folks could port over if they found value in doing so (e.g. to use P&T together with other Functions).²

Something that would help a lot would be to test function-patch-and-transform with your hairier Compositions. You don't need Crossplane installed to to this, you can use https://github.com/crossplane-contrib/xrender to test them on your laptop.

Start by creating a functions.yaml like this:

apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-patch-and-transform
spec:
  # Unfortunately this image is only built for arm64 at the moment.
  # I haven't setup CI/CD so if you're not running ARM you might need to check out
  # the repo and `docker build` it locally. Sorry!
  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.1.4

You'll then just need to "nest" the resources array of your Composition. So adjust your Composition from:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: example
spec:
  resources:
    # Your resource templates go here.

To...

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: example
spec:
  mode: Pipeline
  pipeline:
  - step: patch-and-transform
    functionRef:
      # This needs to match the name of the Function in functions.yaml
      name: function-patch-and-transform
    input:
      apiVersion: pt.fn.crossplane.io/v1beta1
      kind: Resources
      resources:
        # Your resource templates go here.

You should then be able to run the following command, to have xrender print out the patched XR and the resources it would compose.

xrender some-xr.yaml composition.yaml functions.yaml

Both the Function and xrender are pretty early, so I'll watch this discussion and try to reply if you get stuck.

Thank you!

Footnotes

1. Notably it doesn't support EnvConfigs right now. ↩

2. See https://github.com/crossplane-contrib/function-patch-and-transform#differences-from-the-native-implementation for a few minor schema changes from the native implementation. ↩

[jbw976]

I'm seeing an error message with invalid resource, but I'm not quite sure yet why. The basic format of the error message is:

xrender: error: main.CLI.Run(): cannot render composite resource: pipeline step "patch-and-transform" returned a fatal result: invalid Function input: resources: Invalid value: v1beta1.ComposedTemplate{...<full dump of object>}: invalid resource

Questions:

• Any tips for debugging what exactly is invalid about my resource? That could then inform us, as we develop this exciting new feature, on how we show error messages that are easy for humans to parse and give straightforward guidance on what to fix 🤓
• Is a CompositeResourceDefinition file not needed to teach xrender about the schema of XR?
• Is it OK to be passing a claim instead of a XR like I'm doing?

Gist with all my files: https://gist.github.com/jbw976/ab0bd60dc84accbc9c69c3d279e4a543

Full error message details:

> xrender claim.yaml composition.yaml function.yaml
xrender: error: main.CLI.Run(): cannot render composite resource: pipeline step "patch-and-transform" returned a fatal result: invalid Function input: resources: Invalid value: v1beta1.ComposedTemplate{Name:"DBInstance", Base:(*runtime.RawExtension)(0x400002e000), Patches:[]v1beta1.Patch{v1beta1.Patch{Type:"", FromFieldPath:(*string)(0x400006f740), Combine:(*v1beta1.Combine)(nil), ToFieldPath:(*string)(0x400006f750), PatchSetName:(*string)(nil), Transforms:[]v1beta1.Transform{v1beta1.Transform{Type:"string", Math:(*v1beta1.MathTransform)(nil), Map:(*v1beta1.MapTransform)(nil), Match:(*v1beta1.MatchTransform)(nil), String:(*v1beta1.StringTransform)(0x400002e030), Convert:(*v1beta1.ConvertTransform)(nil)}}, Policy:(*v1beta1.PatchPolicy)(nil)}, v1beta1.Patch{Type:"", FromFieldPath:(*string)(0x400006f770), Combine:(*v1beta1.Combine)(nil), ToFieldPath:(*string)(0x400006f780), PatchSetName:(*string)(nil), Transforms:[]v1beta1.Transform(nil), Policy:(*v1beta1.PatchPolicy)(nil)}}, ConnectionDetails:[]v1beta1.ConnectionDetail{v1beta1.ConnectionDetail{Name:"", Type:"", FromConnectionSecretKey:(*string)(0x400006f700), FromFieldPath:(*string)(nil), Value:(*string)(nil)}, v1beta1.ConnectionDetail{Name:"", Type:"", FromConnectionSecretKey:(*string)(0x400006f710), FromFieldPath:(*string)(nil), Value:(*string)(nil)}, v1beta1.ConnectionDetail{Name:"", Type:"", FromConnectionSecretKey:(*string)(0x400006f720), FromFieldPath:(*string)(nil), Value:(*string)(nil)}, v1beta1.ConnectionDetail{Name:"port", Type:"FromValue", FromConnectionSecretKey:(*string)(nil), FromFieldPath:(*string)(nil), Value:(*string)(0x400006f730)}}, ReadinessChecks:[]v1beta1.ReadinessCheck(nil)}: invalid resource

[negz]

@jbw976 I think you're hitting crossplane-contrib/function-patch-and-transform#3, which I also ran into last night. The problem here is the low quality error being returned by function-patch-and-transform, so I think the solution will be to improve that to better surface what part of your resources array is invalid.

Apart from that:

• No need for an XRD in this context. xrender doesn't care about schemas.
• In most cases, yeah you can use a claim in place of an XR. The exception would be if your Composition was trying to patch from some 'machinery' field (e.g. spec.claimRef) that only exists on an XR, not on a claim.

[negz]

FYI, I fixed crossplane-contrib/function-patch-and-transform#3 today. If you use function-patch-and-transform v0.1.3 or newer it should give you a more useful error.

[stevendborrelli]

Some things I've been thinking about:

• We more examples of accessing and using the XR spec/status and dealing with Connection Secrets.
• Documentation referring to composed and composite is really confusing at times. Clarifying these terms early in the Function docs of what roles they play in Crossplane and the data structures would help. Currently docs are from a Crossplane perspective, what if we thought about what a function writer (who maybe isn't familiar with XP or Kubernetes) needs to know?
• Should input support references/selectors? I've been thinking about functions that do things on the operator side like pause all resources before a backup, or remove the Delete ManagementPolicy that we may not want to have set in the Claim. Another thought is an label/annotation is placed on the Claim by the operator and the functions trigger off that.
• I think it would help to clarify what is happening in the pipeline with the data and how it is round-tripped between the functions and Crossplane. For example, this PR 4687 discusses SSA and partial fields. What is the importance of this to a Platform Engineer who is writing functions?

[negz]

We more examples of accessing and using the XR spec/status and dealing with Connection Secrets.

@stevendborrelli Agreed. I added some examples today to https://github.com/crossplane/function-sdk-go that touch on using XR spec/status a little. Still need to document connection secrets, and readiness. xrender doesn't support either of those for testing yet, either.

Should input support references/selectors?

What do you think about the sketch at #3718 (comment)? I'm worried it might be a bit over the top.

[Piotr1215]

When trying to install a patch and transform function, I'm getting the following error on the function deployment:

function-patch-and-transform-5ee9b881186b-8657858768-mztxn exec /function: exec format error

Running Crossplane from master branch on a kind cluster. Here is some more environment info:

➜ ./__system_info.sh
-------------------
System Information:
-------------------
OS Details:
Linux pop-os 6.4.6-76060406-generic #202307241739~1694621917~22.04~ac5e1a8 SMP PREEMPT_DYNAMIC Wed S x86_64 x86_64 x86_64 GNU/Linux

----------------
Kind Version:
----------------
kind v0.20.0 go1.21.1 linux/amd64

-------------------
Kubectl Version:
-------------------
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3
{
  "major": "1",
  "minor": "27",
  "gitVersion": "v1.27.3",
  "gitCommit": "25b4e43193bcda6c7328a6d147b1fb73a33f1598",
  "gitTreeState": "clean",
  "buildDate": "2023-06-15T00:36:28Z",
  "goVersion": "go1.20.5",
  "compiler": "gc",
  "platform": "linux/amd64"
}
--------------------
Crossplane Version:
--------------------
crossplane/crossplane:v1.14.0-rc.0.387.g3dbe4b92

----------------------------------------
Kubernetes Node and Pod Information:
----------------------------------------
Node Information:
NAME                          STATUS   ROLES           AGE   VERSION
control-plane-control-plane   Ready    control-plane   72m   v1.27.3
control-plane-worker          Ready    <none>          72m   v1.27.3

Pod Information:

NAMESPACE            NAME                                                         READY   STATUS             RESTARTS       AGE
crossplane-system    crossplane-f9db6fc5d-pmhtx                                   1/1     Running            0              72m
crossplane-system    crossplane-rbac-manager-5449787c66-k2nxg                     1/1     Running            0              72m
crossplane-system    function-patch-and-transform-5ee9b881186b-8657858768-mztxn   0/1     CrashLoopBackOff   7 (4m1s ago)   14m
crossplane-system    provider-helm-59fc94cd939e-78f8f9c7d7-rtcl7                  1/1     Running            0              71m
crossplane-system    provider-kubernetes-a1a49ab74384-6776d5f6f8-76vw8            1/1     Running            0              71m
kube-system          coredns-5d78c9869d-6sh7x                                     1/1     Running            0              72m
kube-system          coredns-5d78c9869d-x5lxt                                     1/1     Running            0              72m
kube-system          etcd-control-plane-control-plane                             1/1     Running            0              72m
kube-system          kindnet-26gwp                                                1/1     Running            0              72m
kube-system          kindnet-gvzxv                                                1/1     Running            0              72m
kube-system          kube-apiserver-control-plane-control-plane                   1/1     Running            0              72m
kube-system          kube-controller-manager-control-plane-control-plane          1/1     Running            0              72m
kube-system          kube-proxy-8567c                                             1/1     Running            0              72m
kube-system          kube-proxy-qjznr                                             1/1     Running            0              72m
kube-system          kube-scheduler-control-plane-control-plane                   1/1     Running            0              72m
kube-system          metrics-server-7d75545b6c-29cs7                              1/1     Running            0              71m
local-path-storage   local-path-provisioner-6bc4bddd6b-fnx7b                      1/1     Running            0              72m

Kubernetes Status:

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/start-system-namespaces-controller ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-apiserver-identity-lease-controller ok
[+]poststarthook/start-deprecated-kube-apiserver-identity-lease-garbage-collector ok
[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok
[+]poststarthook/start-legacy-token-tracking-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
[+]poststarthook/apiservice-discovery-controller ok
healthz check passed

NAME                          STATUS   ROLES           AGE   VERSION
control-plane-control-plane   Ready    control-plane   72m   v1.27.3
control-plane-worker          Ready    <none>          72m   v1.27.3

Kubernetes control plane is running at https://127.0.0.1:45475
CoreDNS is running at https://127.0.0.1:45475/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

[phisco]

Looks like it got the wrong architecture for the image 🤔

[Piotr1215]

I think you are right. Maybe the images are only published for the M1 processor?

✗ docker image inspect xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.1.2 G -A3 arch
        "Architecture": "arm64",
        "Variant": "v8",
        "Os": "linux",
        "Size": 66341867,

Checking the published image shows only arm64 support.

[negz]

@Piotr1215 Yeah, there's no CI/CD right now. I'm just pushing them from my M1 laptop using docker build && docker push. 😬 I can try find something better tomorrow. In the meantime, you could probably check out the repo and build your own image locally if you felt really motivated. Sorry!

[Piotr1215]

That make sense and no problem, I can build/push the package or swap to laptop 👍

[haarchri]

#4706 Submit a pull request to address an issue related to the installation of functions with configurations. This problem currently results in an empty package and type in the lock lock

apiVersion: pkg.crossplane.io/v1beta1
kind: Lock
metadata:
  name: lock
packages:
- dependencies:
  - constraints: '>=v0.15.0'
    package: xpkg.upbound.io/crossplane-contrib/provider-helm
    type: Provider
  - constraints: '>=v0.9.0'
    package: xpkg.upbound.io/crossplane-contrib/provider-kubernetes
    type: Provider
  - constraints: '>=v0.1.3'
    package: ""
    type: ""
[...]

[haarchri]

Just a quick note: You can't package configurations with "dependsOn" using up-cli because it doesn't recognize the function type, and it quietly removes it during packaging.

up xpkg build --ignore="init/*.yaml,kind-config.yaml" --output=test.xpkg
up xpkg xp-extract --from-xpkg test.xpkg -o ./test.gz
gzip -d test.gz
cat test | grep -B 5 -A 5 ">=v0.1.3"
  dependsOn:
  - provider: xpkg.upbound.io/crossplane-contrib/provider-helm
    version: '>=v0.15.0'
  - provider: xpkg.upbound.io/crossplane-contrib/provider-kubernetes
    version: '>=v0.9.0'
  - version: '>=v0.1.3'
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  creationTimestamp: null

With the crossplane-cli (crank) included in this PR #4694, the packaging now works properly:

./crank xpkg build --ignore="init/*.yaml,kind-config.yaml" --output=test.xpkg
up xpkg xp-extract --from-xpkg test.xpkg -o ./test.gz
gzip -d test.gz
cat test | grep -B 5 -A 5 ">=v0.1.3"
  - provider: xpkg.upbound.io/crossplane-contrib/provider-helm
    version: '>=v0.15.0'
  - provider: xpkg.upbound.io/crossplane-contrib/provider-kubernetes
    version: '>=v0.9.0'
  - function: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform
    version: '>=v0.1.3'
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  creationTimestamp: null

[ytsarev]

During the hackathon today I created a simple if function https://github.com/ytsarev/function-if to check and demonstrate the capability of creating long-awaited conditionals in Crossplane :)

This function is intentionally simple and not designed to compete with #4573 which will probably be the main function around to provide loops and conditionals.

function-if has super simple if/then API . The support for else is planned for next year :D

  - step: if
    functionRef:
      name: function-if
    input:
      apiVersion: template.fn.crossplane.io/v1beta1
      kind: Input
      # conditional over the XR spec field
      if: spec.env == dev
      # Arbitrary YAML stream of MRs that will be composed
      # in case conditional is true
      then: |
        ---
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: Bucket
        spec:
          forProvider:
            region: us-east-2

xrender was the key to enable the local development loop no kind cluster or crossplane installation were used during development and it is totally awesome 👍

Example of xrender running with the examples in the repo, with go run . --insecure --debug function running in parallel shell tab

xrender examples/xr.yaml examples/composition.yaml examples/functions-dev.yaml
---
apiVersion: nopexample.org/v1
kind: XResource
metadata:
  name: test-resource
---
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/composition-resource-name: resource-from-if-0
  generateName: test-resource-
  labels:
    coolness: high
    crossplane.io/composite: test-resource
  ownerReferences:
  - apiVersion: nopexample.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: XResource
    name: test-resource
    uid: ""
spec:
  forProvider:
    region: us-east-2
---
apiVersion: s3.aws.upbound.io/v1beta1
kind: VPC
metadata:
  annotations:
    crossplane.io/composition-resource-name: resource-from-if-1
  generateName: test-resource-
  labels:
    coolness: high
    crossplane.io/composite: test-resource
  ownerReferences:
  - apiVersion: nopexample.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: XResource
    name: test-resource
    uid: ""
spec:
  forProvider:
    region: eu-central-1

Really nice, next-level DX, I was able to iterate quickly.

https://github.com/crossplane/function-sdk-go was super helpful. More examples there will definitely enable faster Functions crafting in the long run.

@negz thanks for the new beta functions and xrender, the first experience is just amazing 👍

[negz]

Awesome @ytsarev!

I love the idea of function-if. I wonder if it could be somehow used to wrap/gate entire other Functions. 🤔 So "only run this Function if X is true".

[bobh66]

Similar to a runtime-evaluated enabled or executeCondition attribute on the pipeline entry that resolves to a boolean - which would likely lead to someone wanting to be able to control the execution of the steps in the pipeline based on data in the Composite or environment, if it hasn't already.

[phisco]

During today's hackathon I built https://github.com/phisco/function-rego to allow specifying policies in rego, mostly to emit "results" based on some rule, e.g. fatal if some field is set, but I also managed to make it generate pretty complex responses directly.

Overall, the experience was quite nice, thanks to the unit tests already properly scaffolded and the local development setup using xrender, I didn't really had the need to deploy it until the end, and once I did, it just worked 🎉

It was not so straightforward to properly handle observed and desired state and build a full response from such a constrained language, but once found a few useful utilities it's doable even in rego, it's debatable whether one should actually do that, but that's another topic.

The deeply nested structures were a bit painful to handle without a proper IDE knowing the schema of the input resources. If needed custom functions written in Go could be injected into the query context, so one could think of exposing the SDK utilities directly. Few times I was hit because I forgot the additional resource inside of fnv1beta1.Resource, which was hard to spot.

I had some issues with the reading back the json response due to how the RunFunctionResponse.Meta.Ttl was serialised to json, so I had to work around it by getting the one created by response.To, unsetting it before marshaling and setting it back after the unmarshal.

In the template repository, I suggest making it easier to identify the items that need to be renamed by using more grep-friendly placeholders. Additionally, include an examples folder to facilitate the immediate usage of xrender. It might be beneficial to organize the repository into additional subdirectories, such as examples/local for setting xrender.crossplane.io/runtime to "Development," and examples/deploy for specifying the image. These subdirectories could utilize kustomize overlays. However, I prefer avoiding the introduction of another tool, even though kubectl now directly supports kustomize.

[phisco]

I was thinking to implement some function to retrieve resources from the outside world, but due to the lack of the environment I wouldn't have known where to put such information, as the observed state is not passed down the pipeline, while setting anything in the desired state would result in it being added to the composite resource references and applied later. So there is definitely a gap if we want to allow implementing functions able to retrieve information from the outside world to be used by other functions.

[negz]

In the template repository, I suggest making it easier to identify the items that need to be renamed by using more grep-friendly placeholders.

@phisco I could imagine us using the template repo combined with a crossplane function scaffold command that could maybe take care of this for you. So we could use names like %{FUNCTION_NAME} or whatever in the template and the CLI would do a find and replace for you after cloning.

[lsviben]

Today on the hackathon I wrote a simple metadata.name to crossplane.io/external-name annotation function, to enable users on setting the external resource names easily. For now it may not be super useful as it just takes a metadata.name but it could be expanded to have a custom Input which points to the field(or other resource field) from which to take the external name.

Here is the repo https://github.com/lsviben/function-external-namer.

The biggest help to get me started was the video @negz created, recommend it to all who want to give a shot at writing functions. Also took some examples from existing functions like https://github.com/stevendborrelli/function-add-k8s-labels-annotations.

Testing with xrender was much easier then testing with a local cluster. But this also got me as I forgot to put a spec.forProvider in the resources I was creating, which caused the creation of managed resource to fail, which I didnt notice in xrender. I guess we could think about validating the end xrender result. Not sure if default on xrender, or with a --validate flag or a separate command altogether.

Overall the experience was quite enjoyable. My other struggle was to think about which function could be really useful, and can be done during the hackathon. Most of my ideas ended up in: "Ah, probably you would be able to do that with @ezgidemirel's go-template function", so looking forward to seeing more of that in action.

[negz]

Not sure if default on xrender, or with a --validate flag or a separate command altogether.

Yeah, I like the idea of xrender accepting an XRD/CRDs that it could use to validate stuff optionally.

[avalanche123]

Today during hackathon I spent some time extending @ytsarev's function-if in ytsarev/function-if#1. I ended up using expr for evaluating the if condition. This prompted me to think of a general function-filter implementation that could be inserted in the middle or at the end of the function pipeline to conditionally filter previous steps, allowing conditionals purely in functions for undoing resources produced higher in the pipeline.

[negz]

Yeah, we should definitely build this. I could see folks potentially finding the UX a bit confusing (vs a more traditional "never do this unless" kind of approach) but I really like that it would essentially give "conditionals" implemented purely in a Function.

[jastang]

I wrote a simple function that "swaps" the Composition that will actuate your XR based on the input to the function. It may not be very useful, but I can imagine using it for things like a bulk migration (without patching all of your composites) or a preprocessing step of some kind in conjunction with other functions being contributed. There are still some TODOs here, for example making sure the Composition exists and is compatible with the XR (i.e. matches compositeTypeRef).

https://github.com/jastang/function-composition-swap

The SDK was fairly easy to acclimate to after a bit of use, and you can actually get reasonably far with unit testing - hence I will suggest some utilities in the SDK to help mock out test cases since the structures can get pretty complex when we inline them for test cases.

xrender, like others have said, makes the inner loop really smooth once you have it set up!

[stevendborrelli]

I tested porting platform-ref-aws to using functions upbound/platform-ref-aws#115 and was successful in my initial efforts after @negz put in a few fixes to https://github.com/crossplane-contrib/function-patch-and-transform.

Some findings:

• I had a couple string transforms that needed a type. This worked with the legacy patch&transform, but got caught here:

 - type: string
                  string:
                    type: Format # missing
                    fmt: "%s-ekscluster"

• ConnectionsecretKeys needed to be updated to have a name.

Overall it was a positive experience. I'm convinced that functions are a big leap in functionality and experience.

[haarchri]

I tried porting the configuration-caas and observed the same results as @stevendborrelli . In general, I had a great experience with it.
I appreciate the log outputs from P&T functions, especially the one:

{"level":"info","ts":1696146979.5705628,"caller":"fn/fn.go:215","msg":"Successfully processed patch-and-transform resources","tag":"","xr-version":"aws.caas.upbound.io/v1alpha1","xr-kind":"XNetwork","xr-name":"aws-spoke-02-8htxm-p2lmd","resource-templates":13,"existing-resources":0,"warnings":0}

Should we consider creating Prometheus metrics based on this log line in the future?

example:

              - fromFieldPath: spec.parameters.region
                toFieldPath: spec.forProvider.subnetIdSelector.matchLabels[zone]
                transforms:
                - type: string
                  string:
                    fmt: "%sb"

      connectionDetails:
        - fromConnectionSecretKey: kubeconfig

changed to:

              - fromFieldPath: spec.parameters.region
                toFieldPath: spec.forProvider.subnetIdSelector.matchLabels[zone]
                transforms:
                - type: string
                  string:
                    type: Format # <-- add type
                    fmt: "%sb"

            connectionDetails:
              - name: kubeconfig # <-- add name
                type: FromConnectionSecretKey # <-- add type
                fromConnectionSecretKey: kubeconfig

nested compositions not working - opened an issue: crossplane-contrib/function-patch-and-transform#13

[negz]

Should we consider creating Prometheus metrics based on this log line in the future?

Probably. See related issue: crossplane/function-sdk-go#20

[haarchri]

draft PR: upbound/configuration-caas#11

[stevendborrelli]

Another data point, I created crossplane-contrib/function-patch-and-transform#14 to add conditional rendering.

With patch-and-transform as a function is is mind-blowing we quickly can make such changes. Adding something to the core patch-and-transform engine is something that would probably take months from design to implementation to release. Here I was able to implement it and do some tests in xrender while waiting at the airport.

[haarchri]

Did we think about adding a conversion webhook to transition from embedded P&T to function P&T? I believe this could be the most effective way to make the switch to function P&T. What are your thoughts on this idea? We can activate this feature in version 1.15 and then eliminate the conversion webhook in version 1.17, along with the removal of embedded P&T.

[bobh66]

Did we think about adding a conversion webhook to transition from embedded P&T to function P&T? I believe this could be the most effective way to make the switch to function P&T. What are your thoughts on this idea? We can activate this feature in version 1.15 and then eliminate the conversion webhook in version 1.17, along with the removal of embedded P&T.

This would imply that the P&T Function is installed by the core Crossplane installation, as mentioned here: #4583 (comment)

[negz]

Adding breadcrumbs to #4728, which I think is a useful discussion around whether Functions should mostly be reading observed or desired state.

[tnm-]

Hi, very interested in trying out beta functions, especially the gotemplating one from crossplane-contrib.

1. gap: I noticed that unbound marketplace haven't been updated to allow (easy) searching for functions.
2. gap: After guessing the correct url ( https://marketplace.upbound.io/functions ), only P&T function is listed. When (if) other official crossplane team maintained functions will be published there?

Thanks

[jeanduplessis]

@tnm- The marketplace will be updated this week with support for functions.

[tnm-]

thx

[Peefy]

I developed the Crossplane KCL function using the Crossplane Go function template and KCL Go SDK, which means we can use KCL (a DSL) to write logic without the need for repeated packaging of crossover functions, and we support KCL package management tools and the KRM KCL specification, which allows for OCI/git source and the reuse of KCL's module ecosystem. The entire development process is very smooth because KCL has already done a lot of work in the early stages.

Reference

• Crossplane KCL Function: https://github.com/kcl-lang/crossplane-kcl
• KRM KCL Spec: https://github.com/kcl-lang/krm-kcl
• KCL Module Registry: https://artifacthub.io/packages/search?org=kcl&sort=relevance&page=1
• KCL Modules: https://github.com/kcl-lang/modules
• Related Issue: [Enhancement] Crossplane KCL Composition Functions kcl-lang/kcl#885

[jbw976]

That's very cool to see @Peefy, thank you for building that and sharing it here!

Do you know if there are some more helpful examples beyond the one in https://github.com/kcl-lang/crossplane-kcl/tree/main/examples/resources/basic? I would love to see more examples of how to use this 🤓

[Peefy]

I have just built the initial version of the Crossplane KCL function, and in the next one to two weeks, I will add more use cases. 😄

[jbw976]

awesome, looking forward to seeing those! thanks for sharing the work you've done so far 🙇‍♂️

in the meantime, maybe these examples can give folks more of a taste for what KCL in general can do: https://github.com/kcl-lang/kcl-lang.io/tree/main/examples

[Peefy]

I've added more examples for crossplane KCL functions including looping, condition, dynamic input parameters, etc. https://github.com/kcl-lang/crossplane-kcl/tree/main/examples

If possible, I am willing to contribute it to the Crossplane community and write a blog explaining the function development process and how to use the Crossplane kcl function.

[jbw976]

That sounds amazing @Peefy, both the contribution of the function to the Crossplane project, but also the blog post too!! I'll follow up with you on Slack to work through that process together 🏆