Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flowcharts #4631

Draft
wants to merge 51 commits into
base: main
Choose a base branch
from
Draft

Flowcharts #4631

wants to merge 51 commits into from

Conversation

maskarb
Copy link
Member

@maskarb maskarb commented Aug 14, 2023

Jira Ticket

COST-####

Description

This change will ...

Testing

  1. Checkout Branch
  2. Restart Koku
  3. Hit endpoint or launch shell
    1. You should see ...
  4. Do more things...

Notes

...

maskarb and others added 30 commits July 15, 2023 10:29
* fix unleash flag for XL customers
* [COST-3956] others included in ordering by cost & limit

---------

Co-authored-by: Sam Doran <[email protected]>
…4552)

* Do not process AWS data if no billing data exists
* Add test
    - Test that no processing is done if bill info is missing
    - Use a non-naive datetime for the datehelper

* Update AWS get_bills_from_privider to return a list instead of a QuerySet
    An unevaluated QuerySet needs to be evaluated in the schema context, which is
    not obvious. This will always be a small number of items (~<=30) so it storing
    it in memory should not be expensive.

* Keep bill IDs as int
    I see no benefit to casting them to str. They eventually get passed through a
    Jinja template. The type in the model is int.

* Simplify and fix test
    Set attributes on Mock instantiation. Remove the patching of __iter__ and
    first() since the code uses __getitem__ now.

* Update get_bills_from_provider() functions to return a list of objects
    Instead of returning an unevaluated QuerySet, evaluate and return the QuerySet.
    This does cost memory to store the result, but it should be small. The benefit
    is not having to use the schema context on the result in order to avoid errors.

* Remove unnecessary ternary test
    There is now a test for aws_bills before this line is hit.

* Fix test
    - Mock out external calls.
    - Explicitly fail if exeuction gets farther than expected.
* [COST-3990] Add write check to the user access endpoint.



---------

Co-authored-by: Sam Doran <[email protected]>
…ission (#4582)

* [COST-3891] deploy worker for swatch data extraction

* [COST-3893] deploy worker for swatch data transmission

* add subs bucket to subs workers

* set subs ephemeral bucket name and add to object store

* add subs kafka topic

* define ENABLE_SUBS_DEBUG in base yml
Use PromQL query instead of a recording rule so that we have data in
until the recording rule is added to app-interface.

This can be changed to use the recording rule once it is available.

* Add all the XL queues
* Update legend
* Change hot-fix-smoke-tests to include crud provider tests
* Create model to hold all tags for all providers
* Only ignore unused imports

COST-3975
Victoremepunto and others added 19 commits August 9, 2023 10:42
- Set a default value for IQE_MARKER_EXPRESSION in Jenkinsfile and remove deprecated pr_check.sh script
…ission (#4597)

* [COST-3891] deploy worker for swatch data extraction

* [COST-3893] deploy worker for swatch data transmission

* add subs bucket to subs workers

* set subs ephemeral bucket name and add to object store

* add subs kafka topic

* define ENABLE_SUBS_DEBUG in base yml
* [COST-3996] Unleash flag to disable tag enablement on settings.

* Add one more check on the post to make sure.
…4576)

* Model and migration to switch to unified enabled tag key table
* Update a lot of AWSEnabledTagKeys to use EnabledTagKeys and code updates related
* Update to use unified enabled tags model everywhere and updating tests
* GCP and OCP keys are disabled by default
* Update settings endpoint
* Update post processor to create enabled tags using tag limit.
* Fix missed schema that may be causing failing smoke tests.
* Fix failing unittests.
* Improve tag limit logic.

* Delete all tags on data rollback
  Instead of deleting per-provider keys, just delete all the records.

* Use a single set for new tags
  Use a different object for storing the new records instead of modifying the
  batch being iterated over.

* Remove chained filters since a single filter will do

COST-3585

---------

Co-authored-by: Sam Doran <[email protected]>
Co-authored-by: Corey Goodfred <[email protected]>
Co-authored-by: Michael Skarbek <[email protected]>
* add subs data extraction

* add subs message transmission

* adjust kafka message and gate messaging on unleash

* update subs last processed time to be stored in postgres

* handle product_ids for subs processing and case insensitive match for required tag

---------

Co-authored-by: David Nakabaale <[email protected]>
* [COST-4000] Add an enabled tag limit.

* I did not mean to push that up.

* Improve test coverage.

* Apply suggestions from code review

Co-authored-by: Sam Doran <[email protected]>

---------

Co-authored-by: Sam Doran <[email protected]>
…ything (#4608)

* pin kombu so dependabot can update dependencies without breaking everything

* update test
* drop date override from OCP flow since its not required.
* adjust Azure queries and fix sqlsafe usage

* update aws and gcp hcs sql to remove sqlsafes
* cost-3991 Move Azure/AWS to daily archives during download processing
* fix unleash check for subs

* pragma no cover

---------

Co-authored-by: maskarb <[email protected]>
@codecov
Copy link

codecov bot commented Aug 28, 2023

Codecov Report

Merging #4631 (2c8437d) into main (d701221) will decrease coverage by 0.0%.
The diff coverage is n/a.

❗ Current head 2c8437d differs from pull request most recent head f3101b8. Consider uploading reports for the commit f3101b8 to get more accurate results

@@           Coverage Diff           @@
##            main   #4631     +/-   ##
=======================================
- Coverage   93.8%   93.8%   -0.0%     
=======================================
  Files        374     374             
  Lines      30389   30324     -65     
  Branches    3584    3572     -12     
=======================================
- Hits       28498   28433     -65     
  Misses      1225    1225             
  Partials     666     666             

@sonarcloud
Copy link

sonarcloud bot commented Aug 30, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants