Releases: mage-ai/mage-ai
0.9.71 | X-Men ❌
What's Changed
🎉 Exciting New Features
📡 Add Streaming Oracle Destination
This feature adds a new custom exporter option for OracleDB allowing developers to easily export their streaming data to Oracle DB.
👤 Multi Project Platform Support
This feature adds support for each user to have their own active project within a multi project platform that keeps track of every project for every user.
⚙️ Support Job Parameters in K8s Executor
Adds support to help manage environment variables, and control job execution and cleanup policies within the Kubernetes cluster.
k8s_executor_config:
container_config:
image: mageai/mageai:0.9.7
env:
- name: USER_CODE_PATH
value: /home/src/k8s_project
job:
active_deadline_seconds: 120
backoff_limit: 3
ttl_seconds_after_finished: 86400
🔑 Support OIDC_ROLES_MAPPING and user_roles
This feature is used to map roles obtained from an OpenID Connect (OIDC) identity provider to roles used within your application allowing for a standardized way to manage user roles across different identity providers.
@mihaivint in #5053 and #4899
💅 Enhancements & Polish
- Optimize CPU and memory usage
- [xy] Disable autoreload in non-dev environments to reduce CPU usage. by @wangxiaoyou1993 in #5136
- [xy] Reduce CPU usage in scheduler. by @wangxiaoyou1993 in #5137
- [td] Memory upgrades by @tommydangerous in #5092
- [td] Track resource usage and run block execution function in a context by @tommydangerous in #5105
- Improve page performance
- [xy] Some improvements on multi project by @wangxiaoyou1993 in #5063
- [xy] Cache project platform settings data in API calls. by @wangxiaoyou1993 in #5079
- [xy] Speed up pipeline update endpoint. by @wangxiaoyou1993 in #5125
- Improve dynamic blocks
- [td] Dynamic blocks use the new memory enhancements by @tommydangerous in #5106
- [td] Dynamic block stream mode by @tommydangerous in #5121
- [td] Dynamic block resilience by @tommydangerous in #5113
- [xy] Support configuring logs_dir_path in environment variable. by @wangxiaoyou1993 in #5018
- [dy] Add prune option for fetch by @dy46 in #5014
- [xy] Include server and scheduler process id in status response. by @wangxiaoyou1993 in #5027
- [xy] Add pipeline_run_id to kwargs. by @wangxiaoyou1993 in #5030
- [xy] Allow configuring authentication mode in mssql config. by @wangxiaoyou1993 in #5051
- Reload libraries when running blocks from edit pipeline by @hugabora in #4953
- [xy] Support configuring concurrency config via env vars. by @wangxiaoyou1993 in #5057
- [jk] Improve visibility of variable/secret names by @johnson-mage in #5072
- relax pytz version requirements by @alex-hunsaker in #5006
- upgrade deltalake for tableMerger by @oonyoontong in #5020
- [td] Turn off unoptimized images by @tommydangerous in #5082
- Add pipeline_schedule_description variable to notification sender by @hjhdaniel in #5118
🐛 Bug Fixes
- [xy] Exclude dropped columns from the postgres schema discovery query. by @wangxiaoyou1993 in #5002
- [xy] Fix None Bigquery database error. by @wangxiaoyou1993 in #5005
- [xy] Fix extracting table name from INSERT statement in SQL block. by @wangxiaoyou1993 in #5008
- [xy] Catch yaml interpolation error. by @wangxiaoyou1993 in #5012
- [xy] Not run insert query for empty dataframe. by @wangxiaoyou1993 in #5017
- [dy] Fix git and seed issues by @dy46 in #5025
- [xy] Fix custom template issue on server start. by @wangxiaoyou1993 in #5038
- [xy] Comment out flaky unit tests by @wangxiaoyou1993 in #5046
- [jk] Allow files to be selected for pipeline zip import by @johnson-mage in #5042
- [dy] Update git email field by @dy46 in #5045
- [dy] Fix circular dependency by @dy46 in #5050
- [td] Update charts and make a ton of bug fixes by @tommydangerous in #5075
- [td] Fix IDE outputs by @tommydangerous in #5088
- [td] Fix git issue by @tommydangerous in #5089
- Td fix git2 by @tommydangerous in #5090
- [td] Upgrade deltalake because of Docker image build error by @tommydangerous in #5091
- [xy] Fix websocket authentication. by @wangxiaoyou1993 in #5094
- [td] Fix interpolating dictionaries when adding custom code to ide execution by @tommydangerous in #5095
- [xy] Fix switching active kernel by @wangxiaoyou1993 in #5098
- [xy] Fix retry in streaming pipeline. by @wangxiaoyou1993 in #5099
- [xy] Fix performance issue of fetching variables. by @wangxiaoyou1993 in #5108
- [jk] Fix max concurrent runs value for backfill by @johnson-mage in #5115
- [xy] Fix circular dependency. by @wangxiaoyou1993 in #5122
- [td] Don’t raise error if can’t load by @tommydangerous in #5132
- [xy] Fix custom template creation. by @wangxiaoyou1993 in #5134
- [xy] Fix variables API and BaseEnum by @wangxiaoyou1993 in #5135
- [td] Fix after on pipeline dashboard by @tommydangerous in #5142
- [td] Fix after panel showing up everywhere by @tommydangerous in #5143
- [td] Fix charts when using Spark by @tommydangerous in #5028
- [xy] Not show locals in CLI exception. by @wangxiaoyou1993 in #5009
- [dy] Remove error pop up by @dy46 in #5015
- [td] Fix interactions, triggers, dynamic blocks, charts, and more by @tommydangerous in #5064
- [td] Fix infer variables by @tommydangerous in #5102
- Escaping Backslashes when interpolating code content by @jimilp7 in #5112
✨ New Contributors
- @alex-hunsaker made their first contribution in #5006
- @mihaivint made their first contribution in #4899
- @stefaan1o made their first contribution in #5034
- @richardlhughes made their first contribution in #5033
- @artche made their first contribution in #5044
- @CLHdevOps made their first contribution in #5077
- @jimilp7 made their first contribution in #5112
Full Changelog: 0.9.70...0.9.71
0.9.70 | Fallout ☢️
What's Changed
🎉 Exciting New Features
💥 Add "Last run failed" filter to Pipeline Runs table
This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.
by @johnson-mage in #4926
🛠️ Add workspace update features
This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.
🔄 Add upsert to MySQL, BQ, and Snowflake
Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:
unique_conflict_method='UPDATE',
unique_constraints=['col'],
⚠️ Support alerts and limited retries in streaming pipeline
Streaming pipeline used to be retried automatically and infinitely.
With this feature, when retry_config
is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config
retry_config:
# Number of retry times
retries: 3
# Initial delay (in seconds) before retry. If exponential_backoff is true,
# the delay time is multiplied by 2 for the next retry
delay: 5
# Maximum time between the first attempt and the last retry
max_delay: 60
# Whether to use exponential backoff retry
exponential_backoff: true
by @wangxiaoyou1993 in #4902
🔑 Add setting to update user roles on login
This feature adds the UPDATE_ROLES_ON_LOGIN
setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.
🐛 Bug Fixes
- [dy] Catch JSON decode errors in mage targets by @dy46 in #4838
- [dy] Fix permission seed by @dy46 in #4846
- [jk] Replace Block reference with BlockFactory by @johnson-mage in #4850
- [jk] Fix block search add block error by @johnson-mage in #4855
- [xy] Fix spark dataframe read error. by @wangxiaoyou1993 in #4856
- [jk] Fix filters from being reset when items fetched by @johnson-mage in #4859
- [xy] Support cancelling jobs when using multiple replicas. by @wangxiaoyou1993 in #4861
- [xy] Fix updating pipeline content with unicode. by @wangxiaoyou1993 in #4864
- [xy] Fix dangling process when reloading webserver. by @wangxiaoyou1993 in #4866
- [jk] Multi-project platform project settings updates by @johnson-mage in #4878
- [xy] Refresh block run statuses before fetching crashed block runs. by @wangxiaoyou1993 in #4885
- Fix: add template for oracle export by @matrixstone in #4892
- [xy] Catch pipeline error in scheduler. by @wangxiaoyou1993 in #4911
- [dy] Create metadata.yaml file if it doesn't exist by @dy46 in #4884
- [xy] Fix dynamic block with great expectation block. by @wangxiaoyou1993 in #4913
- [dy] Fix duckdb sql by @dy46 in #4915
- [xy] Fix log based sync with uppercase columns. by @wangxiaoyou1993 in #4922
- [jk] Prevent renamed triggers in code from being duplicated by @johnson-mage in #4929
- [td] Lots of fixes and enhancements by @tommydangerous in #4937
- [xy] Skip checking memory in Windows. by @wangxiaoyou1993 in #4944
- [xy] Check type key in anyOf array objects. by @wangxiaoyou1993 in #4948
- [dy] Avoid creating duplicate admin user by @dy46 in #4949
- [xy] Fix downloading xlsx file. by @wangxiaoyou1993 in #4957
- [jk] Bugfix - Dependency tree layout by @johnson-mage in #4961
- [xy] Fix logging tags for streaming pipeline run. by @wangxiaoyou1993 in #4963
- [xy] Catch notification send exceptions. by @wangxiaoyou1993 in #4964
- [dy] Remove git dependency by @dy46 in #4959
- [jk] Cancel unsaved modified file dialogue by @johnson-mage in #4975
- [xy] Always set logger in different blocks separately. by @wangxiaoyou1993 in #4974
- [dy] Fix more git issues by @dy46 in #4980
- [xy] Fix k8s affinity parsing. by @wangxiaoyou1993 in #4985
- [xy] Fix encoding for block action object. by @wangxiaoyou1993 in #4987
- [xy] Fix None job manager. by @wangxiaoyou1993 in #4995
- [dy] Remove unused submodule by @dy46 in #4984
- [dy] Fix some git bugs by @dy46 in #4973
- Update jupyter-server-proxy to fix CVE-2024-28179 by @lqc in #4967
- Trino iceberg and delta lake array issues fix along with issue 4895 and other improvements by @sumanshusamarora in #4952
- [edw] Fix typo in Kafka streaming source by @edmondwinston in #4918
- [dy] Cast ambiguous columns to strings by @dy46 in #4857
- Bugfix/okta authorization url discovery by @moseswynn in #4810
- Update ads_insights_age_and_gender.json by @sumanshusamarora in #4891
- [dy] Use project uuid for workspace default permissions by @dy46 in #4843
- [dy] Update oauth token column type by @dy46 in #4933
💅 Enhancements & Polish
- [xy] Use executor type from pipeline level when block level executor is not set by @wangxiaoyou1993 in #4968
- [jk] Replace blue font with cyan in terminal by @johnson-mage in #4848
- [xy] Kill all jobs on scheduler crash. by @wangxiaoyou1993 in #4851
- [td] Update GX and other libraries to support higher versions of Pydantic by @tommydangerous in #4860
- [xy] Update snowflake destination datetime to use TIMESTAMP type. by @wangxiaoyou1993 in #4875
- [jk] Block output view option by @johnson-mage in #4883
- [jk] Only make specific files request with ArcaneLibrary by @johnson-mage in #4888
- Caching pipeline_project outside of loop to avoid multple lookups by @hugabora in #4887
- [jk] Remove pipeline_runs request appearing on different pages by @johnson-mage in #4890
- [xy] Add measurement to influxdb source. by @wangxiaoyou1993 in #4893
- [jk] Add visibility into project config loading errors by @johnson-mage in #4894
- [jk] Hide header actions on sign-in page by @johnson-mage in #4904
- [jk] Disallow numbers as first character of block names by @johnson-mage in #4914
- [jk] Triggers table UI fixes by @johnson-mage in #4934
- [xy] Support interpolating trigger name in k8s job name. by @wangxiaoyou1993 in #4947
- [xy] Only send failure notification when pipeline run status is failed. by @wangxiaoyou1993 in #4950
- [jk] Backfills table improvements by @johnson-mage in #4942
- [xy] Improve the performance of Postgres source metadata query. by @wangxiaoyou1993 in #4962
- [jk] Allow disabling pipeline autosave by @johnson-mage in #4988
- [xy] Support query pipelines wi...
0.9.68 | Invincible 🦸♂️
What's Changed
🎉 Exciting New Features
🐍 Custom python source in Streaming pipeline
This feature allows users to use custom Python source code in their Streaming pipelines, which provides more flexibility and control for advanced users who need to incorporate custom logic or functionality into their streaming pipelines. Here is the code template:
from mage_ai.streaming.sources.base_python import BasePythonSource
from typing import Callable
if 'streaming_source' not in globals():
from mage_ai.data_preparation.decorators import streaming_source
@streaming_source
class CustomSource(BasePythonSource):
def init_client(self):
"""
Implement the logic of initializing the client.
"""
def batch_read(self, handler: Callable):
"""
Batch read the messages from the source and use handler to process the messages.
"""
while True:
records = []
# Implement the logic of fetching the records
if len(records) > 0:
handler(records)
by @wangxiaoyou1993 in #4763
📦 OracleDB exporter
Add OracleDB exporter template to standard batch pipeline.
by @matrixstone in #4785
🔧 Server logging format template overriding
This feature enables users to override the default server logging format template in Mage. This provides more flexibility in customizing the logging output, which can be useful for troubleshooting, monitoring, or integrating with external logging systems.
Here is an example setting:
export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'
🔄 Update pipeline name before pipeline creation
Allow user to edit the pipeline name and add a description/tags before creating a new pipeline. Previously, a randomized name was always used for new pipelines, and then users had to go to the pipeline's settings to update the name.
by @johnson-mage in #4790
🐛 Bug Fixes
- [jk] Add check to avoid rendering of invalid JSX element by @johnson-mage in #4755
- [dy] Fix GitLab auth issues by @dy46 in #4760
- [jk] Increase visibility of code block context menu by @johnson-mage in #4768
- [dy] Check status response for REQUIRE_USER_AUTHENTICATION by @dy46 in #4782
- [xy] Fix exporting dataframe to bigquery. by @wangxiaoyou1993 in #4783
- [xy] Catch job manager init error. by @wangxiaoyou1993 in #4784
- [dy] Add all parent block variables to the add on block by @dy46 in #4747
- [dy] Fix clickhouse destination by @dy46 in #4788
- [jk] Bugfixes - table render / rename destination block by @johnson-mage in #4797
- [dy] Update API trigger url by @dy46 in #4800
- [dy] Load pipeline tags from config by @dy46 in #4806
- [xy] Fix insert command for trino deltalake array type. by @wangxiaoyou1993 in #4819
- [jk] Fix unclickable dbt sql file in dbt v1 model select by @johnson-mage in #4822
- [xy] Raise ImportError when spark is not supported. by @wangxiaoyou1993 in #4825
- [xy] Fix mongodb source typo. by @wangxiaoyou1993 in #4833
- [jk] Configure dbt v2 yaml block and other dbt code block improvements by @johnson-mage in #4834
- [xy] Fix writing polars dataframe to gcs storage. by @wangxiaoyou1993 in #4839
- [xy] Create snowflake temp table like original table. by @wangxiaoyou1993 in #4802
- [jk] Bugfix pipeline tag cache by @johnson-mage in #4841
- [xh] Oracle fix issue: incremental datetime bookmark fetch fails by @matrixstone in #4816
- [jk] Clear block_uuid query param when selecting new block by @johnson-mage in #4844
- [dy] Add sign in check to index.tsx by @dy46 in #4781
- [dy] Fix SSO sign in for frontend build by @dy46 in #4794
- [dy] Put cache in try/except by @dy46 in #4796
- [dy] Fix slack notifications with stacktrace by @dy46 in #4826
- [dy] Update config_overwrite by @dy46 in #4811
💅 Enhancements & Polish
- [xh] OracleDB thick mode support in standard pipeline by @matrixstone in #4701
- [jk] Limit number of block output files read and included in pipeline response by @johnson-mage in #4758
- [xy] Only show sql logs in debug environment. by @wangxiaoyou1993 in #4789
- [xy] Interpolate global vars in k8s executor namespace by @wangxiaoyou1993 in #4792
- [enhancement] mongodb stream full_document config option by @moseswynn in #4793
- [xy] Improve the performance of loading dbt blocks. by @wangxiaoyou1993 in #4807
- [jk] Avoid N dbt config option requests for N blocks on Pipeline Editor by @johnson-mage in #4812
- [xy] Speed up backend requests. by @wangxiaoyou1993 in #4815
- [xy] Test kafka connection when initializing the consumer. by @wangxiaoyou1993 in #4832
- [xy] Make path_to_credential optional in GCS logger. by @wangxiaoyou1993 in #4837
- [dy] Expire certain pipeline runs by @dy46 in #4727
- [dy] Add pagination to workspace roles by @dy46 in #4799
- [dy] Improve elasticsearch speed by @dy46 in #4824
✨ New Contributors
- @afal007 made their first contribution in #4772
- @moseswynn made their first contribution in #4793
Full Changelog: 0.9.66...0.9.68
0.9.66 | Shogun 🎌
What's Changed
🎉 Exciting New Features
🗑️ Delete backfills
- Add delete button to backfills table that allows user to delete backfills individually.
- Prevent user from setting a backfill interval unit value less than 1, which could cause issues loading the backfills.
by @johnson-mage in #4726
🪪 Support AzureDevOps using Entra ID/AD authentication
If you have an Azure DevOps organization connected to your Active Directory/Microsoft Entra ID tenant, you can enable DevOps in the version control app.
- First, you will need to enable the Active Directory integration in order for Mage to authenticate your user.
- Once the Active Directory integration is enabled, you simply need to add your DevOps organization as an environment variable, and you should be able to authenticate with DevOps in the version control app
export AZURE_DEVOPS_ORGANIZATION=<organization_name>
📅 Scheduler status endpoint
Add /api/status
endpoint to scheduler when running scheduler in standalone mode.
This endpoint can be used to check the liveness of the scheduler.
by @wangxiaoyou1993 in #4736
🔐 Read authentication settings from AWS secrets manager
Add support for reading settings from AWS secrets manager. The existing authentication settings have been moved to using the new Settings class.
In order to set the settings backend, the user will need to add a section to the project metadata. Here is an example to set the setting backend to AWS secrets manager:
settings_backend:
backend_type: aws_secrets_manager
prefix: default_repo/settings/
Read the doc for details.
💅 Enhancements & Polish
- [jk] Limit block output for dynamic blocks in UI by @johnson-mage in #4693
- [xy] Add auto clean name flag to io classes. by @wangxiaoyou1993 in #4698
- [dy] Add interval variables for cron based schedules by @dy46 in #4699
- [dy] Add error stack trace to the message for pipeline run failure by @dy46 in #4697
- [dy] Add unique_conflict_method for mssql by @dy46 in #4712
- [jk] Scrollbar polish by @johnson-mage in #4732
- [xy] Remove Redshift stl insert query. by @wangxiaoyou1993 in #4735
- [edw] e2e test: load pages no errors by @edmondwinston in #4429
- [dy] Share main project logging config with workspaces by @dy46 in #4656
🐛 Bug Fixes
- [jk] Fix "filter is not a function" type error in schema props by @johnson-mage in #4686
- [xy] Fix list index out of range for dynamic blocks. by @wangxiaoyou1993 in #4688
- [xy] Set default domain for salesforce source by @wangxiaoyou1993 in #4689
- [dy] Handle JSON reading errors for elasticsearch by @dy46 in #4696
- [dy] Add try/except by @dy46 in #4683
- [jk] Update backfill policy to allow update on settings by @johnson-mage in #4706
- [dy] Fix logging for elasticsearch destination by @dy46 in #4709
- [xy] Add upstream_dbt_blocks method back to DBTBlockSQL. by @wangxiaoyou1993 in #4711
- [jk] Bugfix for pipeline interactions by @johnson-mage in #4710
- [xy] Catch sync schedules exception. by @wangxiaoyou1993 in #4717
- [xy] Fix pipeline metadata encoding issue. by @wangxiaoyou1993 in #4718
- [jk] Fix buggy pipeline dashboard by @johnson-mage in #4714
- Bugfix: Fix migration crash when using MSSQL as mage database by @tuokor in #4695
- [xy] Set repo_path when syncing trigger configs. by @wangxiaoyou1993 in #4719
- [jk] Allow file to be opened in Pipeline Editor without Command Center enabled by @johnson-mage in #4728
- [jk] Bugfix dock overlay by @johnson-mage in #4733
- [dy] Display error on oauth by @dy46 in #4734
- [jk] Avoid client error when making request before interactions are done loading by @johnson-mage in #4744
- Trino temp table location fix - Fixes issue #4738 by @sumanshusamarora in #4739
- [xy] Fix alter Snowflake table with lower_case=false by @wangxiaoyou1993 in #4750
- [jk] Fix block run sorting by @johnson-mage in #4751
- [dy] Fix sign in issue by @dy46 in #4753
- [dy] Fix elasticsearch destination by @dy46 in #4687
- fix the incorrect spelling by @ckfear in #4680
- [Bug] Resolve "parsing error" in the trino.mdx screen by @devharipragaz007 in #4702
- Pyarrow version conflicts from Snowflake connector by @Jawahars in #4707
- [dy] Update access token user queries and add tests by @dy46 in #4713
- bug fix with stripe sync by @sumanshusamarora in #4748
✨ New Contributors
- @devharipragaz007 made their first contribution in #4702
- @Jawahars made their first contribution in #4707
- @tuokor made their first contribution in #4695
Full Changelog: 0.9.65...0.9.66
0.9.65 | Demon Slayer 🐗 ⚔️
What's Changed
🎉 Exciting New Features
🔔 Telegram notification
Support for setting up Telegram notification channel for alerts is now available.
Check out the doc for details
by @luthfir-rahman in #4632
🔄 Runtime variables and concurrency config in Backfills
Backfills support configuring global variables override and concurrency limit is now available.
☁️ Export raw files to S3 and GCS
Using S3 and GCS data exporter templates, you can now directly upload raw files in addition to dataframes.
bucket_name = 'test-bucket'
object_key = 'test_file_copy.csv'
S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
'/home/src/test_file.csv',
bucket_name,
object_key,
)
GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
'/home/src/test_file.csv',
bucket_name,
object_key,
)
by @wangxiaoyou1993 in #4607
🐛 Bug Fixes
- Luthfir/discord notification by @luthfir-rahman in #4590
- [xy] Support exporting to mysql with selected columns. by @wangxiaoyou1993 in #4605
- [xy] Update the port check to use specified host. by @wangxiaoyou1993 in #4606
- [xy] Set default host when checking port. by @wangxiaoyou1993 in #4610
- [dy] Refactor duplicate git code by @dy46 in #4611
- [jk] Fix unclickable minimized windows by @johnson-mage in #4613
- [dy] Attempt to resolve network issues by @dy46 in #4620
- [jk] Fix block caching when renaming/detaching blocks by @johnson-mage in #4626
- [jk] Update tag and block cache when cloning pipeline by @johnson-mage in #4627
- [jk] Various bugfixes for pipeline search and filter by @johnson-mage in #4631
- [xy] Use milliseconds in parquet by default. by @wangxiaoyou1993 in #4634
- Fixed MongoDB reading input by @Luishfs in #4639
- Fixed random input inside singer destinations by @Luishfs in #4642
- [jk] Fix double scrollbars and draggable height by @johnson-mage in #4643
- [jk] UI bugfixes on Version Control page by @johnson-mage in #4644
- [dy] Fix unit test logging by @dy46 in #4630
- [dy] Add DB locking by @dy46 in #4629
- [jk] Bugfixes for Overview Dashboard by @johnson-mage in #4659
- [jk] Bugfix for save block shortcut by @johnson-mage in #4667
- Fixed postgreSQL reserved words by @Luishfs in #4640
- [dy] Minor workspace fixes by @dy46 in #4658
- [jk] Sync block contents for replica block when minimizing by @johnson-mage in #4672
- Incorrect spelling for the word level in documentation by @Sa1tyBiscuit in #4635
💅 Enhancements & Polish
- [xy] Support filtering google sheets by sheet names by @wangxiaoyou1993 in #4604
- [xy] Combine schedulers. by @wangxiaoyou1993 in #4614
- [xy] Separate scheduler logs and pipeline logs. by @wangxiaoyou1993 in #4621
- [xy] Update initial load of log_based replication to use incremental sync by @wangxiaoyou1993 in #4623
- [jk] Redirect to login page immediately by @johnson-mage in #4662
- [jk] Clean up chart block widths in Pipeline Editor by @johnson-mage in #4665
- [xy] Support extra connection arguments in mysql source and destination. by @wangxiaoyou1993 in #4668
- [xy] Fix image build by @wangxiaoyou1993 in #4671
- Add DEFAULT_EXECUTOR_TYPE by @danieldiamond in #4597
- [dy] Filter out sensitive config by @dy46 in #4594
- mage_ai/io MSSQL connection: add support for DATETIMEOFFSET by @hugabora in #4619
- [xy] Bump up more versions to resolve vulnerabilities. by @wangxiaoyou1993 in #4444
New Contributors
- @danieldiamond made their first contribution in #4597
- @carillpower made their first contribution in #4625
- @Sa1tyBiscuit made their first contribution in #4635
Full Changelog: 0.9.64...0.9.65
0.9.64 | Maestro 🎵
What's Changed
🎉 Exciting New Features
🔭 OpenTelemetry integration
- Added OpenTelemetry Libraries: Integrated OpenTelemetry Python libraries to instrument the application for telemetry data collection.
opentelemetry-api
: Provides the API for tracing and metrics.opentelemetry-sdk
: The core SDK for OpenTelemetry in Python.opentelemetry-instrumentation
: General instrumentation library.opentelemetry-instrumentation-sqlalchemy
: Specific for instrumenting SQLAlchemy operations.opentelemetry-instrumentation-tornado
: For monitoring Tornado-based HTTP server activities.
- Environment Variables: Added new environment variables (
OTEL_EXPORTER_OTLP_HTTP_ENDPOINT
andOTEL_EXPORTER_OTLP_ENDPOINT
) for configuring the OpenTelemetry Collector endpoints. - Database and HTTP Server Instrumentation: Configured instrumentation for both SQLAlchemy database operations and Tornado HTTP server activities to gather traces and metrics.
- Check out this doc for more details
by @mfreeman451 in #4476
🦊 Add GitLab authentication to version control
Support authenticating with GitLab in version control app.
The feature can be enabled by configuring the following environment variables:
GITLAB_HOST
GITLAB_CLIENT_ID
GITLAB_CLIENT_SECRET
🔔 Discord notification integration
Support setting up Discord notification channel for alerts.
Check out the doc for details
by @luthfir-rahman in #4478
🔑 Add OIDC generic provider
This feature allows users to hook up their OIDC server (e.g. Keycloak ) and use it to authenticate with Mage.
It can be enabled by configuring the following environment variables:
OIDC_CLIENT_ID
OIDC_CLIENT_SECRET
OIDC_DISCOVERY_URL
🐛 Bug Fixes
- [xy] Use utc time as the default time for monitor_stats. by @wangxiaoyou1993 in #4554
- Fix Google Ads Catalog to Data Integration Block by @Luishfs in #4565
- [jk] Load custom design logo by @johnson-mage in #4574
- [dy] Adjust alembic logger initialization by @dy46 in #4552
- [dy] Add support for AWS codecommit by @dy46 in #4568
- [dy] Fix frontend crashing issues by @dy46 in #4573
- [xy] Set max width for yaml file to inf. by @wangxiaoyou1993 in #4576
- [dy] Fix DBT logging by @dy46 in #4577
- [dy] Show the load balancer ingress host if there is no host in the ingress rules by @dy46 in #4575
- [xy] Match postgres logical replication records with stream name. by @wangxiaoyou1993 in #4588
💅 Enhancements & Polish
- Added reserved words settings to PostgreSQL by @Luishfs in #4142
- [xy] Improve google sheets source. by @wangxiaoyou1993 in #4561
- [jk] Reuse callback and conditional blocks by @johnson-mage in #4562
- [xy] Show pandas datetime in iso format in notebook. by @wangxiaoyou1993 in #4564
- [xy] Add case_sensitive param to the sql export method. by @wangxiaoyou1993 in #4569
- [xy] Create TEMP Snowflake table in data integration destination by @wangxiaoyou1993 in #4578
- [dy] Misc workspace/git improvements by @dy46 in #4583
- [dy] Update user authentication for subprojects by @dy46 in #4586
New Contributors
- @luthfir-rahman made their first contribution in #4478
Full Changelog: 0.9.63...0.9.64
0.9.63 | Halo 👾
What's Changed
🎉 Exciting New Features
🦆 MotherDuck Support
This one's for all the ducklings out there! In addition to supporting DuckDB, Mage now supports MotherDuck destinations!
By specifying a MOTHERDUCK_TOKEN
and adding a md:
prefix to your DuckDB database, you can read/write from/to MotherDuck locations! Check it out and get started here.
by @wangxiaoyou1993 in #4533
🤖 Support Thick
mode in OracleDB
We like our OracleDB connections like we like our pancakes, THICK 🥞. By default, Mage's Oracle client runs in a "Thin" mode which connects directly to Oracle Database— this mode does not need Oracle Client libraries. However, some additional functionality is available when they're used.
Now, you can use the "Thick" mode in Mage to connect to OracleDB using the Oracle Client libraries!
Check out our docs to get started or read more about the differences between "Thin" and "Thick" modes here.
by @matrixstone in #4421
🗄️ Show unused block files in file browser
This is one of our most requested features and we're excited to announce that it's finally here! 🎉
You can now see which files are not being used in your pipeline! This is a great way to clean up your projects and remove any unnecessary files. Check out the gif below to see it in action!
Head over to our docs to learn more!
by @johnson-mage in #4449
🤐 Import functionality for pipeline zips
Like to share? So do we! You can now import pipelines (via .zip
files) in your Mage projects! We're optimistic that this simple improvement will make it easier to share your pipelines or even borrow pipelines from your friends!
by @johnson-mage in #4453
🐛 Bug Fixes
- Prevent add block menu from disappearing by @johnson-mage in #4502
- Fix some minor bugs by @tommydangerous in #4505
- Remove table name helper by @tommydangerous in #4506
- Fix alter table column names cleaning (Postgresql integration exporter) by @arnetkachev in #4493
- Fixed Oauth connection on Salesforce Source by @Luishfs in #4402
- Fix unclickable vertical scrollbar and jumping before panel by @johnson-mage in #4512
- Catch unknown host error by @wangxiaoyou1993 in #4518
- Upgrade snowflake library version and fix datetime column type by @wangxiaoyou1993 in #4524
- Only show files with correct prefix by @dy46 in #4509
- Update submodule sync for ssh auth by @dy46 in #4522
- Update connection url by @dy46 in #4523
- Update how the stale pipeline message modal is displayed by @johnson-mage in #4536
- Fix running dynamic blocks with k8s executor by @wangxiaoyou1993 in #4543
- Widget policy update by @johnson-mage in #4545
- Catch exception of building cache key for block cache by @wangxiaoyou1993 in #4548
- Disable stale pipeline modal by @johnson-mage in #4525
💅 Enhancements & Polish
- MSSQL data integration source - add support for DATETIMEOFFSET type by @hugabora in #4499
- Fix: handle unprivileged user for postgres by @jdbranham in #4357
- Optimize pipeline schedule queries by @dy46 in #4188
- Remove pipeline's updated_at attribute by @johnson-mage in #4521
- Create and view workspaces in different namespaces by @dy46 in #4513
- Select status for pipeline runs that time out by @dy46 in #4519
New Contributors
- @arnetkachev made their first contribution in #4493
- @jdbranham made their first contribution in #4357
- @tanjibpa made their first contribution in #4528
Full Changelog: 0.9.62...0.9.63
0.9.62 | The Beekeeper 🐝
What's Changed
🎉 Exciting New Features
🧵 [Mage IO] Weaviate Integration
🤔 Building AI apps with Mage? Perfect! Now you can use Weaviate as a data source. Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI applications. With Mage, you can now read from and write to Weaviate databases! Read more in our docs here.
by @matrixstone in #4158
🔍 [Mage IO] Alogia Integration
Like search? Us too! That's why we've added support for Alogia as a data source in Mage. Alogia is a powerful search engine that helps you build fast and accurate search experiences. With Mage, you can now read from and write to Alogia! Read more in our docs here.
by @matrixstone in #4198
💥 Dynamic SQL Blocks
Another big improvement to dynamic blocks this week SQL Dynamic Blocks! That's right, you can now create dynamic outputs from SQL blocks in Mage. Previously, blocks had to be Python for dynamic outputs, but no more! This is a big step forward in making Mage more flexible and powerful. Give it a shot today. 🎉
by @tommydangerous in #4430
🌊 [Kafka Streaming Sources] Offset & Partitions
For our streaming aficionados, we have a few new Kafka features! You can now specify the offset and partitions for Kafka streaming sources. Offsets can be one of: Beginning
, End
, Int
, & Timestamp
. This allows users to set specific positions inside a topic to consume data.
⛴️ Configure Kubernetes Affinity & Tolerations
Last, but certainly not least, we've got some nitty Kubernetes configuration updates! You can now specify affinity
and tolerations
in your Kubernetes settings. ⚓
Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on nodes and label selectors specified in pods. Node affinity allows a pod to specify an affinity (or anti-affinity) towards a group of nodes it can be placed on.
Tolerations are applied to pods and indicate that the pod can be scheduled on nodes with specific taints.
These configurations should help our Kubernetes power users make the most of Mage! 🧙
by @wangxiaoyou1993 in #4407
🐛 Bug Fixes
- Fix LDAP unpacking by @dy46 in #4399
- Fix reduce output and triple layout saving by @tommydangerous in #4409
- Fix file browser bug and Git push bug by @tommydangerous in #4411
- Fix serializing list and dict when formatting output by @tommydangerous in #4412
- Prevent index out of bounds by @tommydangerous in #4425
- Remove test print statements by @tommydangerous in #4431
- Removing Draft7 validation from Clickhouse destination by @Luishfs in #4424
- Fix
global_vars
context in pipeline executor by @wangxiaoyou1993 in #4435 - Fix a few global data product bugs by @tommydangerous in #4440
- Fix dynamic blocks OOM round 2 by @tommydangerous in #4445
- Fix incremental sync in chargebee source by @wangxiaoyou1993 in #4450
- Don’t count values if is None by @tommydangerous in #4454
- Fix policy issue updating settings by @tommydangerous in #4456
- Fix keyboard shortcuts when its empty array by @tommydangerous in #4458
- Run submodule sync by @dy46 in #4457
- Fixing Snowflake
write_pandas
issue by @Luishfs in #4395 - Fix writing to Snowflake with mixed int and str types by @wangxiaoyou1993 in #4460
- Fix some bugs and improve the edit page by @tommydangerous in #4462
- Fix error logging in pipeline executor by @tommydangerous in #4468
- When clicking show file versions in arcane library, show right panel by @tommydangerous in #4472
- Catch BigQuery if it fails to fix table names by @tommydangerous in #4479
- Fix pipeline detail prop passed for fetching files by @johnson-mage in #4485
- Always show edit pipeline button by @dy46 in #4484
- Revert change to PG IO by @tommydangerous in #4486
- Fix io redshift by @wangxiaoyou1993 in #4487
- Fix multi project flag by @wangxiaoyou1993 in #4490
- Fix Bigquery clean column name by @wangxiaoyou1993 in #4500
- Convert datetime type for s3 data integration destination by @wangxiaoyou1993 in #4501
- Fix dynamic child block outputs by @tommydangerous in #4422
- Fix database missing and serializing QueryJob by @tommydangerous in #4428
- Fix incorrect spelling by @ckfear in #4438
- Fix kafka type and doc by @wangxiaoyou1993 in #4466
💅 Enhancements & Polish
- Workspace improvements by @dy46 in #4469
- Scheduler improvements by @wangxiaoyou1993 in #4467
- Prevent unnecessary initial pipeline run by @johnson-mage in #4291
- Speed up monitor stats and reduce calls on overview by @tommydangerous in #4408
- Bump up dependency versions to resolve vulnerabilities by @wangxiaoyou1993 in #4433
- Limit dynamic block output in notebook by @tommydangerous in #4436
- Improve bigquery name parsing to help fill in full name by @tommydangerous in #4447
- Add error logging by @tommydangerous in #4448
- Fix app slowness due to project platform check by @wangxiaoyou1993 in #4483
- Add PVC retention policy by @dy46 in #4491
- E2e test for
/pipelines
by @edmondwinston in #4306 - Add query decorator for data integration blocks by @tommydangerous in #4465
New Contributors
Full Changelog: 0.9.60...0.9.62
0.9.60 | Yusuke Urameshi
What's Changed
🎉 Exciting New Features
🌊 [Streaming] Google Cloud Storage Destination
🎉 Google Cloud users rejoice! Streaming pipelines just even got better— Mage now supports Google Cloud Storage as a streaming destination! Check out the docs here and get started today!
by @wangxiaoyou1993 in #4340
👷♂️ Overwrite SQL types
For anyone with a data warehouse, listen up! (We assume that's most of you 😅)
You can now specify custom column types when exporting to SQL destinations. This is useful when you want to export a dataframe with a column that has a type that is not supported by the default mapping. You can read more about overwriting types here.
Here's an example of an exporter that overwrites column types for a PostgreSQL destination:
@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
allow_reserved_words=True,
unique_conflict_method='UPDATE',
unique_constraints=['col'],
overwrite_types=overwrite_types,
)
This feature is currently supported for PostgreSQL, Redshift, Trino, MSSQL, MySQL, Clickhouse, and BigQuery.
👨💻 [Command Center] Version Control & Files
The Mage Command Center can now be used for version control commands! You can both configure git and run your favorite version control commands directly from the Mage UI. Additionally, you can manage your files via nav and Mage's new file explorer (🧙 Arcane Library)!
As a reminder, to enable the command center, you can do so by going to Settings
(the wizard icon) and click the Command Center
toggle. It can be accessed via ⌘ + .
(Mac) or Win + .
(Windows).
by @tommydangerous in #4273
👾 [Command Center] Terminal App
Mage LEGEND @tommydangerous is back at it again— he's implemented a full terminal app into the command center. For those of you with this beta feature enabled, you'll now have a terminal at your fingertips at all times.
Check out the following video for sample usage:
CleanShot.2024-01-14.at.10.09.43.mp4
As a reminder, to enable the command center, you can do so by going to Settings
(the wizard icon) and click the Command Center
toggle. It can be accessed via ⌘ + .
(Mac) or Win + .
(Windows).
by @tommydangerous in #4365
JSON Logging
A huge shoutout to @dy46 for adding JSON logging to Mage! This will make it easier to parse logs and integrate with other logging tools. Just specify SERVER_LOGGING_FORMAT=json
to change the output to something like the following:
🐛 Bug Fixes
- Fix caching issues with block cache and shared pipelines by @johnson-mage in #4338
- Fix SQL blocks by @tommydangerous in #4341
- Prevent error when searching for blocks by @johnson-mage in #4343
- Fix callbacks input data from dynamic child blocks by @tommydangerous in #4342
- Enable command center when user auth not required by @tommydangerous in #4346
- Fix bug when searching for block files by @johnson-mage in #4347
- Fix dynamic child block getting input data by @tommydangerous in #4349
- Fix cron expression conversion when using local midnight time by @johnson-mage in #4359
- Fix block search by @tommydangerous in #4360
- Fix command center hiding by @tommydangerous in #4361
- Fix GDP and add terminal colors by @tommydangerous in #4363
- Minor tweaks to existing apps by @tommydangerous in #4367
- Added
custom_fields
to freshdesk source by @Luishfs in #4354 - Fix creating widget by @wangxiaoyou1993 in #4375
- Lowercase
auth_type
enum by @dy46 in #4376 - Update cloud run workspace by @dy46 in #4377
- Fix terminal by @dy46 in #4389
- Fix pipeline run variable overwrite for sql block by @wangxiaoyou1993 in #4390
- Update dynamic block output and input data logic by @tommydangerous in #4388
- Fix dynamic block conditionals in runs and in notebook by @tommydangerous in #4397
- Remove terminal colors by @tommydangerous in #4398
- Fix reduce output block tests by @tommydangerous in #4400
- Removing modified
google-ads
lib by @Luishfs in #4330
💅 Enhancements & Polish
- Show multiple outputs and fix downstream dynamic child block inputs and outputs by @tommydangerous in #4382
- Improve command center shortcut wording and example by @johnson-mage in #4348
- Add mapping for active directory roles by @dy46 in #4345
- Make block type error more descriptive by @johnson-mage in #4353
- Upgrade app layout behavior by @tommydangerous in #4362
- Support overwriting column types in BigQuery by @wangxiaoyou1993 in #4374
- Add spark jar files to
emr_config
if using EMR cluster by @johnson-mage in #4379
Full Changelog: 0.9.59...0.9.60
0.9.59 | 🐲 🐉 Year of the Dragon
What's Changed
🎉 Exciting New Features
Note: many new features this week are in beta. You can enable them by navigating to your Mage settings and toggling the beta features there.
🎮 Multi-project Platform [BETA]
We've reworked our support for mulitple projects with a new multi-project platform! @tommydangerous is back at it again with this huge feature release, enabling nested projects, custom code paths, cross-project triggering, a split pipeline scheduler, and much more!
CleanShot.2024-01-08.at.02.33.54.mp4
If you'd like to try out the multi-project platform, you can check out this repo for a sample structure. Head over to your Mage settings to enable
- Support nested projects and custom code paths by @tommydangerous in #4161
- Trigger and run pipelines across projects by @tommydangerous in #4186
- Split pipeline scheduler and schedule models for project platform by @tommydangerous in #4233
- Configure root project preferences and settings by @tommydangerous in #4234
- Fix pipeline schedule creation and repo name by @tommydangerous in #4247
- Don't try to use file source if not exist by @tommydangerous in #4196
- Fix the way we store pipelines by type in the cache by @tommydangerous in #4278
🚀 Command Center [BETA]
Another new & exciting feature this week— the Mage Command Center. The command center is a floating search bar that can invoke actions like opening files & pages, perform actions within Mage, interacting with the page, and much more!
CleanShot.2024-01-08.at.03.15.15.mp4
Enable the Command Center in settings and give it a spin today!
- Command center by @tommydangerous in #4249
- Add command center models by @tommydangerous in #4254
🪣 Bitbucket Version Control
Shout out to @dy46 for continuing to crush the version control integrations!
You can now use Bitbucket as a version control provider! This is a great option for teams that use Bitbucket for their code repositories. To get started, navigate to the Mage Version Control app and select Bitbucket as your provider. You'll be prompted to authenticate with Bitbucket, and then you'll be able to select your repositories. Read more here.
◔ Qdrant integration
Mage now supports Qdrant, an open-source vector search engine. Qdrant is a great tool for similarity search, and it can be used for a variety of use cases, including product recommendations, image search, and more. With this update, you can load/export data from Qdrant sources in your batch pipelines! Read more here.
by @matrixstone in #4081
🧱 dbt DX v2 + dbt Upgrade [BETA]
This release contains a huge dbt overhaul 🤯
Alongside a much-awaited upgrade to dbt 1.7, the dbt developer experience has been completely rebuilt. @tommydangerous has been hard at work crafting a dbt experience that is more intuitive, powerful, and flexible!
Here's a quick demo:
CleanShot.2024-01-08.at.03.08.21.mp4
- Upgrade dbt to 1.7 by @tommydangerous in #4244
- dbt v2 browser UI by @tommydangerous in #4200
- Add block browser modal to notebook by @tommydangerous in #4246
- Add dbt cache by @tommydangerous in #4193
- Add fields to dbt code block 2.0 for manual entry by @tommydangerous in #4331
- Use absolute paths in dbt block by @tommydangerous in #4307
- Add custom code block tags by @tommydangerous in #4250
- Collapse or expand folders by @tommydangerous in #4203
🐛 Bug Fixes
SQL blocks
- Remove double quotes for postgres by @dy46 in #4170
- Escape BigQuery project name in SQL block by @wangxiaoyou1993 in #4294
- Pass in
query_vars
as a dict by @dy46 in #4280
Data integration
- Fix syncing MySQL
TIME
type by @wangxiaoyou1993 in #4275 - Bump Google Ads version from 14 to 15 by @Luishfs in #4289
- Fix incremental sync bug with missing arg by @tommydangerous in #4315
- Fix executing data integration block with ecs executor by @wangxiaoyou1993 in #4322
- Wrap MSSQL table name with double quotes in data integration pipeline by @wangxiaoyou1993 in #4290
Trigger and scheduling
- Use UTC date for trigger start date by @johnson-mage in #4283
- Display correct default start datetime when editing trigger/backfill by @johnson-mage in #4292
- Try preventing creating duplicate pipeline runs in scheduler by @wangxiaoyou1993 in #4311
- Wrap block run initialization logic with lock by @wangxiaoyou1993 in #4296
dbt
- Fix dbt profiles interpolation by @tommydangerous in #4225
- Lazy import for dbt files by @tommydangerous in #4251
- Re-work dbt project path for
yaml
files by @tommydangerous in #4207 - Error handling for "project not found" in dbt by @tommydangerous in #4334
- Adjust error behavior when adding dbt files by @tommydangerous in #4191
File browser
- Fix files page not opening files by @tommydangerous in #4222
- File browser bug bash by @tommydangerous in #4237
- Fix bug when deleting block from file browser by @tommydangerous in #4243
- Fix
requestIdleCallback
not supported on Safari by @tommydangerous in #4284 - Fix file browser not refreshing by @tommydangerous in #4223
Dynamic blocks
- Fix dynamic block + dynamic child blocks spawning other blocks by @tommydangerous in #4295
- Fix reduce output bug by @tommydangerous in #4326
Git
- Add actions to
GitBranchPolicy
by @dy46 in #4213 - Fix Git bugs by @tommydangerous in #4264
- Fix Git submodule sync by @dy46 in #4316
- Fix missing Git module by @tommydangerous in #4206
Other
- Fix silent errors from global hooks by @tommydangerous in #4173
- Fix block sorting bug by @johnson-mage in #4175
- Avoid saving error details in block run DB by @tommydangerous in #4179
- Reset page after applying pipeline filters by @johnson-mage in #4183
- Restrict opentelemetry package versions by @dy46 in #4208
- Fix dataframe validation by @tommydangerous in #4255
- Fix add new button tooltips by @tommydangerous in #4263
- Save
statistics.json
in correct execution partition folder by @wangxiaoyou1993 in #4271 - Fix dashboard resizing by @tommydangerous in #4282
- Only set the schema in the DB when the server is started by @dy46 in #4293
- Fix several bugs on Pipelines dashboard by @johnson-mage in #4300
- Improve UI in several areas by @tommydangerous in #4309
- Fix dragging and dropping file in browser by @tommydangerous in #4312
- Fix interpolating mage secret in project metadata.yaml by @wangxiaoyou1993 in https://github.com/mag...