dataflow
Ji Yu Spark Ren Wu Liu Zhi Xing Ping Tai
She Ji
Gai Nian
dataflow Zhong De Gai Nian :
-
executionId
Mei Ci Jiang dag Ren Wu Ti Jiao Dao dataflow Shi Hui Sheng Cheng Wei Yi Zhi Xing id
-
taskId
Zai Yi Ci dag Zhi Xing Zhong ,dag Zhong De Mei Ge node Zhi Xing Shi Du Hui Sheng Cheng Wei Yi Ren Wu id
spark Zhong De Gai Nian :
-
job
spark Mei Ci Yu Dao action Suan Zi Hui Ti Jiao Yi Ge job
-
stage
spark Jiang job Hua Fen Wei Duo Ge stages Zhi Xing (Mei Ci Yu Dao shuffle Du Hui Hua Fen Yi Ge stage)
-
task
stage Xia De Yi Ge Zhi Xing Dan Yuan ,Mei Ge task Zhi Chu Li Yi Ge partition Shang De Shu Ju
Xu Yao Zhu Yi De Shi ,dataflow Zhong De Yi Ge task Hui Sheng Cheng spark De Yi (Duo )Ge job Qu Wan Cheng Yi Ge Zu Jian De Cao Zuo .
Zu Jian Ding Yi
Ding Yi Ru Xia :
"id": "Zu Jian ID",
"name": "Zu Jian Lei Xing ",
"text": "Miao Shu ",
"argument": {
// Shu Ru
"input": {
},
// Shu Chu
"output": {
}
}
}
Zu Jian Jie Shao
Yi Lai
- Scala 2.11.12
- Spark 2.4.7
- Spring Boot 2.3.7.RELEASE
Mo Kuai Jie Shao
+-- dataflow-core Yin Qing
+-- dataflow-udf Spark SQLZi Ding Yi Han Shu
+-- dataflow-sdk SDKJi Cheng
+-- dataflow-launch Yin Qing Qi Dong
Kuai Su Kai Shi (Qian Ru Shi )
Pei Zhi
Xiu Gai src/main/resources/application.yaml Pei Zhi Wen Jian
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dataflow?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf8
username: root
password: 123456
Ben Di Qi Dong
- Qi Dong src/main/resources/k.fishfish.dataflow.Bootstrap Cheng Xu
Bu Shu
Da Bao (Zhi Ding embed Huan Jing ,Hui Jiang Spark Yi Lai Da Ru jar Bao ):
Yun Xing :
Ji Suan
Jiang Ren Wu Hui Zhi Cheng Liu Cheng Tu ,Li Yong DAG Suan Fa Jin Xing Jie Dian Yun Suan .
Li Ru ,Ru Xia Jiang test_user Biao Shu Ju Tong Bu Dao test_user_copy1 Biao Zhong
curl --location --request POST 'http://127.0.0.1:9090/v1/dag/run' \
--header 'Content-Type: application/json' \
--data-raw '{
"executionId": "1",
"context": {},
"graph": {
"nodes": [
{
"id": "1",
"name": "DATABASE_SOURCE",
"text": "Du ",
"argument": {
"input": {
"url": "jdbc:mysql://127.0.0.1:3306/test?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf8",
"user": "root",
"password": "123456",
"sql": "select * from test_user"
},
"output": {
"table": "t1"
}
}
},
{
"id": "2",
"name": "SQL_TRANSFORMER",
"text": "Zhuan Huan ",
"argument": {
"input": {
"sql": "select * from {namespace}.t1"
},
"output": {
"table": "t2"
}
}
},
{
"id": "3",
"name": "DATABASE_SINK",
"text": "Xie ",
"argument": {
"input": {
"table": "t2"
},
"output": {
"url": "jdbc:mysql://127.0.0.1:3306/test?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf8",
"user": "root",
"password": "123456",
"table": "test_user_copy",
"mode": "update"
}
}
}
],
"edges": [
{
"from": "1",
"to": "2"
},
{
"from": "2",
"to": "3"
}
]
},
"callback": "http://127.0.0.1:8080/callback"
}'
