-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[平台后续发展][DataOps][数智]任务调度及管理 #130
Labels
Comments
感谢对SREWorks数据服务相关能力关注,提出非常好的讨论点,当前SREWorks的做法如下: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
SREWorks多次提到运维方面的数仓和ETL,并且也有Flink这类应用集成,后续有没有规划在这方面深入,如集成企业业务数仓相关功能?
1)数仓案例:一个数仓,采用Flink CDC做数据抽取,Flink SQL(批处理或流处理)或数据库SQL(批处理,如Hologres SQL)做数据处理,相关任务是需要上线的,批处理任务涉及到定时调度,流处理任务涉及到状态监控,任务可能会失败报错,需要收集错误日志,以上也属于运维的范畴,有没有计划增加Flink SQL和数据库SQL的任务调度及管理功能(参考Airflow、DolphinScheduler)?
2)数据资产:有没有计划通过Flink Catalog等方式收集各数据源信息来做一个这方面的元数据管理,进而实现数据资产目录和数据血缘功能(会用到上述的数据调度任务依赖元信息)。
3)数据服务:用户只需要写SQL,平台生成相关的数据服务API,以供外部访问数据。
The text was updated successfully, but these errors were encountered: