Make pipelines aware of a timezone configuration #249
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roelschr
force-pushed
the
roelschr/add-timezone-config
branch
from
e3d4e24
to
082d134
Compare
|
Kudos, SonarCloud Quality Gate passed!
|
AlvaroMarquesAndrade
approved these changes
Sep 23, 2020
rafaelleinio
reviewed
Oct 6, 2020
Comment on lines
+21
to
+22
| timezone: timestamp feature transformations will assume this timezone | ||
| when they don't have a tz suffix. |
There was a problem hiding this comment.
I'd just inform here that spark con config (spark.sql.session.timeZone) and an env variable (TZ) will be set with this value.
lilbee101920
approved these changes
Feb 26, 2021
| @@ -1,4 +1,6 @@ | |||
| """FeatureSetPipeline entity.""" | |||
| import os | |||
lilbee101920
approved these changes
Feb 26, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why? 馃摉
While Spark's TimestampType timezone is controlled by the
spark.sql.session.timeZoneconfiguration option, python's datetime objects have their timezone controlled by the system's timezone (when they don't have a fixed tz suffix). This means some transformations can have their timestamps converted in different ways when running on different systems.An example of possible irregular results happens when we automatically set the
start_dateofAggregatedFeatureSets(here). Sometimes the spark and the system can have different timezones, meaning that the timestamp coming from the spark dataframe, when collected into plain python as a datetime object can change, generating astart_datedifferent then expected.What? 馃敡
This PR proposes to apply a timezone configuration that should be aware by each pipeline and that should be the same between spark and system. This timezone is configurable.
Type of change
Please delete options that are not relevant.
How everything was tested? 馃搹
TODO.
Checklist
bug,enhancement,feature, andreview.Attention Points鈿狅笍
Replace me for what the reviewer will need to pay attention to in the PR or just to cover any concerns after the merge.