Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple InfluxDB and Fluentbit, allow configuration of multiple Fluentbit outputs. Rework logger config. #2781

Open
wants to merge 83 commits into
base: main
Choose a base branch
from

Conversation

timopulkkinen
Copy link

Description

This pull request decouples InfluxDB from the Logger as the sole persistence layer by adding the possibility to pass a customOutput config in values.xml and making InfluxDB dependency optional. It is possible to pass secrets and public env variables to Fluent Bit by using new attribute extraEnv.

Note: influxdb.enabled has been moved under logger.fluentbit, this will break existing configuration.

Example snippet from values.xml to use PostgreSQL as an log output target:

logger:
    enabled: true
    fluentbit:
      extraEnv:
        sensitive:
           DB_PASSWORD: x
        public:
           DB_TABLE: table   
      influxdb:
        enabled: false
      customOutput: |
        [OUTPUT]
          Name pgsql
          Match log.*
          Host ${DB_HOST}
          Port 5432
          Database ${DB_DATABASE}
          User ${DB_USER}
          Password ${DB_PASSWORD}
          Table ${DB_TABLE}

Which issue(s) this PR fixes:

Fixes no open issues that I know of.

Testing

helm lint
Installed chart with default + custom setup.

Checklist:

  • I ran tests as well as code linting locally to verify my changes.
  • I have done manual verification of my changes, changes working as expected.
  • I have added new tests to cover my changes.
  • My changes follow contributing guidelines of Fission.
  • I have signed all of my commits.

timopulkkinen and others added 30 commits April 11, 2023 13:59
…ues.xml. Make InfluxDB optional and move InfluxDB config under Fluent Bit as InfluxDB seems to be solely used by FluentBit for log collection.
…ogger, so it cannot be optional. Also, rename templates/fluentbit -> templates/logger to make the code navigation easier.
…ues.xml. Make InfluxDB optional and move InfluxDB config under Fluent Bit as InfluxDB seems to be solely used by FluentBit for log collection.
…ogger, so it cannot be optional. Also, rename templates/fluentbit -> templates/logger to make the code navigation easier.
- k8s.io/api v0.28.1
- k8s.io/apiextensions-apiserver v0.28.1
- k8s.io/apimachinery v0.28.1
- k8s.io/client-go v0.28.1
- k8s.io/metrics v0.28.1
- sigs.k8s.io/controller-runtime v0.16.1
- sigs.k8s.io/controller-tools v0.13.0

Signed-off-by: Sanket Sudake <[email protected]>
…sion#2828)

the FISSION_..._NAMESPACE environment variables are used by all services during init
to check if the CRDs are loaded.

Move them to the 'fission-resource-namespace' macro to ensure all services know the relavant namespaces.

Signed-off-by: Anthony King <[email protected]>
…#2830)

- added retainPods flag to take in the number of specialized pods to retain
- add retainPods in both the create function and update function command
- modify crd keys to be typed instead of string
- keep track of function generation in case of update function operation
- add delete handler function to make sure specialized pods are deleted in case function is deleted

---------

Signed-off-by: Sanket Sudake <[email protected]>
Signed-off-by: Pranoy Kundu <[email protected]>
Co-authored-by: Pranoy Kundu <[email protected]>
)

* Update the Kubernetes autoscaling API to v2 (fission#2722)
  - k8s.io/api/autoscaling/v2
* generate all files
* fix missing v2beta ref

---------

Signed-off-by: Lunik <[email protected]>
* Update CI actions and go modules
* Fix kind cluster name
* Fix upgrade cluster name

---

Signed-off-by: Sanket Sudake <[email protected]>
* Remove controller component from Fission
* Cleanup at few more places

---------

Signed-off-by: Sanket Sudake <[email protected]>
* added client generator inteface
* start router service asynchronously

Signed-off-by: Vardhaman Surana <[email protected]>
- Update kubernets version 1.23+
- Update post chart install notes with namespace info
- Update upgrade notes in chart

Signed-off-by: Sanket Sudake <[email protected]>
DadaGore and others added 30 commits December 6, 2023 10:11
* Update Go dependencies

---

github.com/go-git/go-git/v5 v5.10.1 [v5.11.0]
github.com/google/uuid v1.4.0 [v1.5.0]
google.golang.org/grpc v1.59.0 [v1.60.0]
k8s.io/api v0.28.4 [v0.29.0]
k8s.io/apiextensions-apiserver v0.28.4 [v0.29.0]
k8s.io/apimachinery v0.28.4 [v0.29.0]
k8s.io/client-go v0.28.4 [v0.29.0]
k8s.io/metrics v0.28.4 [v0.29.0]

Signed-off-by: Sanket Sudake <[email protected]>

* update deps

Signed-off-by: Sanket Sudake <[email protected]>

---------

Signed-off-by: Sanket Sudake <[email protected]>
* added fetcher test cases

* error message formatting fix

* error message formatting fix for e2e framework
Making typed common cache so that we don't use wrong types
across set/get methods and more higher-level methods can be
defined for cache.
Currently, we are not able to operate over all keys of the cache
due to generic types.
I also removed code comments around the cache.

Signed-off-by: Sanket Sudake <[email protected]>
If package has deployment already, we should set package status to none instead of pending.
If we have package has source, we set status to pending.
If both source and deployment are empty, package marked as failure.

Signed-off-by: Sanket Sudake <[email protected]>
Added configurable kubernetes client limits burst and QPS
Default QPS 200 and burst 500. Configurable via helm chart values.

Signed-off-by: Sanket Sudake <[email protected]>
- Update go modules
    + github.com/prometheus/client_golang v1.17.0 [v1.18.0]
   + golang.org/x/net v0.19.0 [v0.20.0]
    + sigs.k8s.io/controller-tools v0.13.0 [v0.14.0]
- Remove Kafka tests disabled.

Signed-off-by: Sanket Sudake <[email protected]>
Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.3.3 to 1.3.7.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](cloudflare/circl@v1.3.3...v1.3.7)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
github.com/google/uuid: v1.5.0 -> v1.6.0
github.com/prometheus/common: v0.45.0 -> v0.46.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.46.1 -> v0.47.0
go.opentelemetry.io/contrib/propagators/autoprop: v0.46.1 -> v0.47.0
go.opentelemetry.io/otel: v1.21.0 -> v1.22.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.21.0 -> v1.22.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.21.0 -> v1.22.0
go.opentelemetry.io/otel/sdk: v1.21.0 -> v1.22.0
go.opentelemetry.io/otel/trace: v1.21.0 -> v1.22.0
google.golang.org/grpc: v1.60.1 -> v1.61.0
k8s.io/api: v0.29.0 -> v0.29.1
k8s.io/apiextensions-apiserver: v0.29.0 -> v0.29.1
k8s.io/apimachinery: v0.29.0 -> v0.29.1
k8s.io/client-go: v0.29.0 -> v0.29.1
k8s.io/metrics: v0.29.0 -> v0.29.1
sigs.k8s.io/controller-runtime: v0.16.3 -> v0.17.0

Signed-off-by: Sanket Sudake <[email protected]>
Go depenencies updated
```
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.47.0 -> v0.48.0
go.opentelemetry.io/contrib/propagators/autoprop: v0.47.0 -> v0.48.0
go.opentelemetry.io/otel: v1.22.0 -> v1.23.1
go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.22.0 -> v1.23.1
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.22.0 -> v1.23.1
go.opentelemetry.io/otel/sdk: v1.22.0 -> v1.23.1
go.opentelemetry.io/otel/trace: v1.22.0 -> v1.23.1
golang.org/x/net: v0.20.0 -> v0.21.0
google.golang.org/grpc: v1.61.0 -> v1.61.1
k8s.io/api: v0.29.1 -> v0.29.2
k8s.io/apiextensions-apiserver: v0.29.1 -> v0.29.2
k8s.io/apimachinery: v0.29.1 -> v0.29.2
k8s.io/client-go: v0.29.1 -> v0.29.2
k8s.io/code-generator: v0.29.1 -> v0.29.2
k8s.io/metrics: v0.29.1 -> v0.29.2
sigs.k8s.io/controller-runtime: v0.17.0 -> v0.17.1
```

Signed-off-by: Sanket Sudake <[email protected]>
…ues.xml. Make InfluxDB optional and move InfluxDB config under Fluent Bit as InfluxDB seems to be solely used by FluentBit for log collection.
…ogger, so it cannot be optional. Also, rename templates/fluentbit -> templates/logger to make the code navigation easier.
…ues.xml. Make InfluxDB optional and move InfluxDB config under Fluent Bit as InfluxDB seems to be solely used by FluentBit for log collection.
…ogger, so it cannot be optional. Also, rename templates/fluentbit -> templates/logger to make the code navigation easier.
The logger and router templates have been updated to include the use of the release namespace when generating Kubernetes roles. This change enhances the accuracy and specificity of role assignments within the environment.
# Conflicts:
#	charts/fission-all/Chart.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants