Dark Mode

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Kazuhito00/Multimodal-Node-Editor

Repository files navigation

[Japanese/English]

Multimodal-Node-Editor

no-doedeita-be-sunomaruchimo-daruChu Li apurike-shiyondesu.
Hua Xiang *Yin Sheng *tekisutonadowono-dowoXi geChu Li surukotogaKe Neng de, Chu Li Jian Tao yaChu Li Bi Jiao denoLi Yong woXiang Ding shiteimasu.

Features

  • Hua Xiang *Yin Sheng *tekisuto*Shen Ceng Xue Xi nado, 100 Yi Shang nono-dowoBiao Zhun Da Zai
  • Web kamerayamaikuRu Li woYong itariarutaimuChu Li niDui Ying
  • TOML + Python niyoruXin Gui no-donoJian Dan naZhui Jia gaKe Neng (GUI woZi Zuo shinaiChang He )
  • Bao Cun shitagurahuwo GUI nashidehetsudoresuShi Xing Ke Neng
  • Google Colaboratory batsukuendodenoshisutemuQi Dong Ke Neng

Note

no-dohaZuo Cheng Zhe (Gao Qiao )gaBi Yao ninatsutaShun niZhui Jia shiteirutame,
Hua Xiang Chu Li , o-deioChu Li , tekisutoChu Li niokeruJi Ben De naChu Li woDan uno-dogaBu Zu shiteirukotogaarimasu.

Requirements

hurontoendo
Node.js v18 or later

react ^18.3.1
react-dom ^18.3.1
@xyflow/react ^12.3.6
dagre ^0.8.5
@types/dagre ^0.7.53
vite ^6.0.5
typescript ~5.6.2
@vitejs/plugin-react ^4.3.4
@types/react ^18.3.18
@types/react-dom ^18.3.5
batsukuendo
Python 3.10 or later

pydantic 2.12.5 or later
platformdirs 4.5.1 or later
fastapi 0.128.0 or later
python-multipart 0.0.21 or later
uvicorn[standard] 0.40.0 or later
opencv-python 4.11.0.86 or later
pillow 12.1.0 or later
motpy 0.0.10 or later
sahi 0.11.36 or later
onnx 1.20.0 or later
onnxruntime 1.23.2 or later *GPU woShi Yong suruChang He ha onnxruntime-gpu
mediapipe 0.10.31 or later
sounddevice 0.5.3 or later
soundfile 0.13.1 or later
webrtcvad-wheels 2.0.14 or later
scipy 1.16.3 or later
av 16.0.1 or later
openai 2.14.0 or later
aiortc 1.14.0 or later
websocket-client 1.9.0 or later
vosk 0.3.45 or later
google-cloud-speech 2.35.0 or later

Installation

Google ColaboratoryQi Dong woXing uFang ha, Yi Xia noZuo Ye haBu Yao desu. no-tobutsukunoChu Li niCong tsutekudasai.
mata, Yi Xia noShou Shun ha, Pythonyanode.jsgainsuto-rusareteiruQian Ti desu.

# ripozitorikuro-n
git clone https://github.com/Kazuhito00/Multimodal-Node-Editor
cd Multimodal-Node-Editor

# Python patsuke-ziinsuto-ru
pip install -r requirements.txt

# moderuueitodaunro-do
python download_weights.py # Quan huairuwodaunro-do(Ji nihuairugaaruChang He hasukitsupu)
# python download_weights.py --force # Quan huairuwoQiang Zhi De niShang Shu ki
# python download_weights.py --max-size 150 # Zhi Ding saizuMBYi Shang nohuairunodaunro-dohasukitsupu

# Node.js patsuke-ziinsuto-ru
cd src/gui/reactflow/frontend
npm install
cd ../../../../

# konhuiguwokopi-
cp config.example.json config.json

Yi Bu nono-dohaAPIki-nadowoShe Ding shinaitoShi Yong dekimasen.
Bi Yao niYing zite, config.json noYi Xia ki-woShe Ding shitekudasai.

{
"api_keys": {
"openai": "SET_YOUR_OPENAI_API_KEY",
"google_stt": "PATH_TO_GOOGLE_CREDENTIALS_JSON"
}
}
konhuiguXiang Xi
ki- Xing dehuoruto Shuo Ming
node_search_paths string[] ["src/nodes"] no-doDing Yi woJian Suo surudeirekutori
ui.theme string "light" te-ma(light / dark)
ui.sidebar.show_edit bool false saidoba-niandouridoumeniyu-woBiao Shi
ui.sidebar.show_file bool true saidoba-nigurahuBao Cun (json)meniyu-woBiao Shi
ui.sidebar.show_auto_layout bool true saidoba-niZi Dong reiautobotanwoBiao Shi
graph.interval_ms int 50 gurahuShi Xing Jian Ge (miriMiao )
audio.sample_rate int 16000 o-deioChu Li nosanpurure-to(Hz)
camera.max_scan_count int 2 kameradebaisunoZui Da sukiyanShu
auto_download.video bool false Dong Hua kiyapuchiyaShi noZi Dong daunro-do
auto_download.wav bool false Lu Yin Shi noZi Dong daunro-do
auto_download.capture bool false Hua Xiang kiyapuchiyaShi noZi Dong daunro-do
auto_download.text bool false tekisutoBao Cun Shi noZi Dong daunro-do
api_keys.openai string "" OpenAI APIki-
api_keys.google_stt string "" Google Speech-to-Text Yong kuredenshiyarujsonGe Na pasu

Launch the application

  • ro-karuPCdenoQi Dong
    Yi Xia nosukuriputowoShi Xing shitekudasai. Qi Dong niCheng Gong surutoburauzagaLi chiShang garimasu.

    python run_gui_reactflow.py
    opushiyon Shuo Ming
    --config She Ding huairunopasu(dehuoruto: config.json)

    mataha, Yi Xia wosorezoreBie konso-runiteShi Xing shi, burauzanitehttp://localhost:5173/niakusesushitekudasai.

    uvicorn src.gui.reactflow.backend.main:app --reload
    cd src/gui/reactflow/frontend
    npm run dev

  • Google ColaboratorydenoQi Dong


    Colaboratorydeno-tobutsukuwoKai ki, Shang karaShun niShi Xing shitekudasai.
    Zui Zhong serunoChu Li Jie Guo nihttps://localhost:8000/toBiao Shi sarerutame, sonorinkuwokuritsukushitekudasai

  • hetsudoresuShi Xing
    ReactFlowhurontoendowoQi Dong sezunikomandoraindegurahuShi Xing gaKe Neng desu

    python run_headless.py graph.json
    opushiyon Shuo Ming
    --config She Ding huairunopasu(dehuoruto: config.json)
    --count Shi Xing Hui Shu (0=Wu Xian ru-pu, 1=1Hui Shi Xing , dehuoruto: 0)

Usage

no-doPei Zhi & Shi Xing
  1. saidoba-karano-dowokiyanbasunidoratsugu&dorotsupu
  2. no-doJian nopo-towoJie Sok
    *Tong Se Tong Shi nopo-todeJie Sok Ke Neng
  3. Start botanwokuritsukushiteShi Xing (shiyo-tokatsutoki-:Ctrl + Enter)
    *Shi Xing Zhong hano-doPei Zhi yapo-toJie Sok noBian Geng haBu Ke
etsuziXue Chu , no-doXue Chu
  • Xue Chu shitaietsuziyano-dowoXuan Ze shiteDeleteki-
gurahujsonekusupo-to, inpo-to
  • gurahujsonekusupo-to:Save botanwokuritsuku(shiyo-tokatsutoki-:Ctrl + S)
  • gurahujsoninpo-to:Load botanwokuritsuku(shiyo-tokatsutoki-:Ctrl + L)
o-toreiauto
  • Auto Layout botanwokuritsuku(shiyo-tokatsutoki-:Ctrl + A)
no-dohenokomento
  • no-dowoYou kuritsukushite, Add Commentwokuritsuku

Keyboard Shortcuts

shiyo-tokatsuto Dong Zuo Bei Kao
Ctrl + Enter START/STOPQie riTi e Shi Xing Zhong naraSTOP, Ting Zhi Zhong naraSTART
Escape STOP gurahuShi Xing woTing Zhi
Ctrl + P Pause/ResumeQie riTi e Shi Xing Zhong naraPause, Yi Shi Ting Zhi Zhong naraResume
Ctrl + Z Undo Yuan niLi su
Ctrl + Y Redo yariZhi su
Ctrl + A Auto Layout no-dowoZi Dong Pei Zhi
Ctrl + S Save gurahuwoJSONdeBao Cun (ekusupo-to)
Ctrl + L Load gurahuJSONwoDu miIp mi(inpo-to)
Delete Xue Chu Xuan Ze Zhong nono-do/etsuziwoXue Chu (Shi Xing Zhong haWu Xiao )

Nodes

Image

Image > Input
no-doMing Shuo Ming
Image Jing Zhi Hua Xiang huairu(jpg, png, bmp, gif)woDu miIp mu
Webcam WebkamerakarariarutaimuYing Xiang woQu De
ColaboratorybatsukuendodeLi Yong Bu Ke
Webcam (WebSocket) burauzanogetUserMedia() APIJing You deWebkameraYing Xiang woQu De
ColaboratorybatsukuendodeLi Yong Ke
Video Dong Hua huairu(mp4, avi)woDu miIp ndehure-muZai Sheng
*Realtime Syncchietsukubotsukusu:Chu Li Shi Jian niTong Qi shitehure-muwoDu miChu suopushiyon
*Frame Step:Du miIp mihure-muJian Ge (realtime_sync=falseShi nomi)
*Preload All Framechietsukubotsukusu:Quan hure-muwopuriro-dosuru
*saidoba-no[Loop Playback] gaONnoChang He , ru-puZai Sheng woXing u
Video Frame Dong Hua noZhi Ding hure-muWei Zhi noHua Xiang woChu Li
RTSP netsutowa-kukameranoRTSPRu Li karaYing Xiang woQu De
Solid Color Dan Se Hua Xiang woSheng Cheng
*width:Hua Xiang Fu (1-4096, dehuoruto: 640)
*height:Hua Xiang Gao sa(1-4096, dehuoruto: 360)
*color:Se (kara-pitsuka-, dehuoruto: #ff0000)
URL Image URLkaraHua Xiang wodaunro-doshiteQu De
Image > Transform
no-doMing Shuo Ming
Crop Zheng Gui Hua Zuo Biao (0.0-1.0)deZhi Ding shitaLing Yu woQie riBa ku
Hua Xiang eriawodoratsugushiteLing Yu woZhi Ding Ke Neng
Flip Hua Xiang woShui Ping /Chui Zhi Fang Xiang niFan Zhuan suru
Resize Zhi Ding Jie Xiang Du *Bu Jian Fang Fa derisaizusuru
Rotate Zhi Ding Jiao Du deHua Xiang woHui Zhuan suru(90Du noBei Shu Yi Wai dehaYu Bai gaFa Sheng )
3D Rotate 3Ci Yuan Kong Jian denopitsuchi*yo-*ro-ruHui Zhuan woXing u
Click Perspective Hua Xiang kuritsukudeno4Dian Zhi Ding niyoruTou Shi Bian Huan woXing u
Image > Filter
no-doMing Shuo Ming
Apply Color Map gure-suke-ruHua Xiang niYi Si kara-woShi Yong suru
Background Subtraction Bei Jing Chai Fen deQian Jing woJian Chu suru
Blur Ge Zhong bokashihuirutawoShi Yong suru
Morphology moruhuorozi-Bian Huan woXing u
Brightness Hui Du woJia Suan Diao Zheng suru
Canny CannyFa niyoruetsuziJian Chu woXing u
Contrast kontorasutowoDiao Zheng suru
Equalize Hist HSVnoVchiyannerunihisutoguramuPing Tan Hua woShi Yong suru
Filter 2D (3x3) Ren Yi no3x3ka-neruniyoruDie miIp mihuirutawoShi Yong suru
Gamma ganmaBu Zheng woShi Yong suru(LUTte-buruShi Yong )
Grayscale Hua Xiang wogure-suke-runiBian Huan suru(3chiyanneruWei Chi )
RGB Extract Zhi Ding shitaRGBchiyanneruwoChou Chu suru
RGB Adjust RGBGe chiyanneruniZhi woJia Suan suru
HSV Adjust HSVSe Kong Jian deSe Xiang *Cai Du *Ming Du woDiao Zheng suru
Inpaint masukuwoYong iteinpeintosuru
Omnidirectional Viewer Zheng Ju Yuan Tong Tu Fa no360Du Hua Xiang woHui Zhuan Biao Shi suru
Hua Xiang nodoratsugudeShi Dian woBian Geng Ke Neng
Sepia sepiaDiao ehuekutowoShi Yong suru
Threshold Ge Zhong arugorizumude2Zhi Hua suru
Image > Marker Detection
no-doMing Shuo Ming
QR Code QRko-dowoJian Chu *deko-doshi, Jie Guo woJSONChu Li suru
ArUco Marker ArUcoma-ka-woJian Chu shi, ID*Si Yu Zuo Biao woJSONChu Li suru
AprilTag AprilTagwoJian Chu shi, ID*Si Yu Zuo Biao woJSONChu Li suru
Image > Deep Learning
no-doMing Shuo Ming
Image Classification ImageNet 1000kurasudeHua Xiang woFen Lei suru
*Model:Shi Yong surumoderuwoXuan Ze suru(dorotsupudaun)
Object Detection Wu Ti Jian Chu woXing u
motpyniyorumaruchiobuziekutotoratsukingu, SAHIniyorusuraisuChu Li Jian Chu nimoDui Ying
Face Detection Yan Jian Chu woXing u
motpyniyorumaruchiobuziekutotoratsukingu, SAHIniyorusuraisuChu Li Jian Chu nimoDui Ying
Low-Light Image Enhancement An Suo Hua Xiang noHua Xiang Qiang Diao woXing u
Depth Estimation Dan Yan Shen Du Tui Ding woXing u
Pose Estimation Ren Ti Zi Shi Tui Ding woXing u
Hand Pose Estimation Shou noZi Shi Tui Ding woXing u
Semantic Segmentation semanteitsukusegumente-shiyonwoXing u
OCR Guang Xue Wen Zi Ren Shi woXing u
Image > Analysis
no-doMing Shuo Ming
Color Histogram Ge chiyannerunohisutoguramuwogurahuBiao Shi suru
LBP Histogram Local Binary Patternhisutoguramuwoba-gurahuBiao Shi suru
FFT FFTmagunichiyu-dosupekutoramuwoDui Shu suke-rudeKe Shi Hua suru
Image > Draw
no-doMing Shuo Ming
Draw Text (ASCII) OpenCVdeASCIItekisutowoMiao Hua suru(Gai Xing Dui Ying )
Draw Canvas Ru Li Hua Xiang Shang nihuri-handoMiao Hua woXing u
Draw Mask Ru Li Hua Xiang Shang nihuri-handoMiao Hua woXing iEr Zhi masukuwoSheng Cheng suru
Simple Concat 2Mei noHua Xiang woLian Jie suru
Multi Image Concat Zui Da 9Mei noHua Xiang woguritsudoreiautodeLian Jie suru
Comparison Slider 2Mei noHua Xiang woBi Jiao suraida-deBiao Shi suru
Picture In Picture Image 2woImage 1noZhi Ding Ling Yu niZhong neHe waseru
Hua Xiang Shang nodoratsugudeLing Yu Zhi Ding gaKe Neng
Blend Ge Zhong burendomo-dodeHe Cheng suru
Alpha Blend Zhong miFu kiaruhuaburendowoXing u
Image > Output
no-doMing Shuo Ming
Image Display Ru Li Hua Xiang wono-doShang niBiao Shi suru(no-dorisaizuKe Neng )
Capture botanYa Xia deHua Xiang wokiyapuchiyashiBao Cun suru
Write Video Ru Li Hua Xiang woMP4Dong Hua toshiteBao Cun suru(STOPShi niBao Cun )
Image > Other
no-doMing Shuo Ming
Execute Python yu-za-Ru Li noPythonko-dowoShi Xing suru(input_image - output_image)
OpenAI APIki-woShe Ding shiteiruChang He , Sheng Cheng AIniyoruko-doSheng Cheng moKe Neng

Audio

Audio > Input
no-doMing Shuo Ming
Mic maikukarariarutaimuYin Sheng woQu De suru
Mic (WebSocket) burauzanogetUserMedia() APIJing You demaikuYin Sheng woQu De suru
Echo Cancellationha, Speaker (Browser)no-dodeYin Sheng Chu Li woXing tsutaChang He nomiYou Xiao
Audio File Yin Sheng huairu(wav, mp3, ogg)woZai Sheng suru
*saidoba-no[Loop Playback] gaONnoChang He , ru-puZai Sheng woXing u
Noise Ge Zhong noizuXin Hao woSheng Cheng suru
Zero Wu Yin (zerode-ta)woChu Li suru
Audio > Dynamics
no-doMing Shuo Ming
Volume (Hard Limit) Yin Liang wosuke-ruDiao Zheng shi, +-1.0deha-dokuritsupingusuru
Volume (Soft Limit : tanh) Yin Liang wosuke-ruDiao Zheng shi, tanhGuan Shu deHua rakanikuritsupingusuru
Dynamic Range Compression Yu Zhi woChao etaXin Hao woYa Suo suru
Expander Yu Zhi Wei Man noXin Hao woJian Shuai suru
Noise Gate Yu Zhi Wei Man noXin Hao wokatsutosuru
Audio > Filter
no-doMing Shuo Ming
Lowpass Filter katsutoohuZhou Bo Shu yoriGao iCheng Fen woChu Qu suru(Butterworth IIR)
Highpass Filter katsutoohuZhou Bo Shu yoriDi iCheng Fen woChu Qu suru(Butterworth IIR)
Bandpass Filter Zhi Ding Zhou Bo Shu Fan Wei nomiTong Guo saseru(Butterworth IIR)
Bandstop Filter Zhi Ding Zhou Bo Shu Fan Wei woChu Qu suru(Butterworth IIR)
Equalizer Zhi Ding Zhou Bo Shu Dai Yu wobu-suto/katsutosuru
Audio > Deep Learning
no-doMing Shuo Ming
Speech Enhancement Yin Sheng woQiang Diao suru(noizuChu Qu )
Audio Classification Yin Sheng ibentowoFen Lei suru
Audio > Recognition
no-doMing Shuo Ming
Google STT Google Cloud Speech-to-Text APIdesutori-minguYin Sheng Ren Shi woXing u
*konhuiguno api_keys.google_stt niGooglekuredenshiyarujsonwoZhi Ding shiteiruChang He nomiYou Xiao
Vosk STT Vosk APIdesutori-minguYin Sheng Ren Shi woXing u
Audio > Utility
no-doMing Shuo Ming
Delay Yin Sheng Xin Hao woZhi Ding Shi Jian Chi Yan saseru
Mixer 2tsunoYin Sheng Xin Hao woJia Suan mitsukususuru
Waveform to Image Yin Sheng Bo Xing karaue-buhuo-muHua Xiang woZuo Cheng suru
Audio > Analysis
no-doMing Shuo Ming
Spectrogram Yin Sheng Xin Hao nosupekutoroguramuwoBiao Shi suru
Power Spectrum Yin Sheng Xin Hao nopawa-supekutoruwoBiao Shi suru
VAD Yin Sheng Qu Jian Jian Chu woXing u
MSC 2Xin Hao Jian noMagnitude Squared Coherence(Zhou Bo Shu Bie Lei Si Du )woJi Suan suru
Audio > Output
no-doMing Shuo Ming
Speaker supi-ka-karaYin Sheng woZai Sheng suru
Speaker (Browser) burauzanoWeb Audio APIdeYin Sheng woZai Sheng suru
Write WAV Ru Li Yin Sheng woWAVhuairutoshiteJi Lu suru(STOPShi niBao Cun )

Text

Text > Input
no-doMing Shuo Ming
Text tekisutowoChu Li
Text > Process
no-doMing Shuo Ming
Text Replace Wen Zi Lie woZhi Huan suru
Text Join 2tsunotekisutowoJie He suru
Text Format tenpure-tonopure-suhoruda- {1}~ {10} woRu Li Zhi deZhi Huan suru
JSON Parse JSONWen Zi Lie wopa-sushiteki-/pasudeZhi woChou Chu suru
JSON Array Format JSONPei Lie karahui-rudowoChou Chu shitetekisutonihuo-matsutosuru
Text > Deep Learning
no-doMing Shuo Ming
Language Classification tekisutonoYan Yu woPan Ding suru
Text > Output
no-doMing Shuo Ming
Text Display tekisutoNei Rong wono-doShang niBiao Shi suru
Text Save tekisutowohuairuniBao Cun suru

OpenAI

OpenAI
no-doMing Shuo Ming
OpenAI LLM OpenAI LLM APIwoHu biChu su
ExecutebotanwoYa shitataimingudeShi Xing suru
*konhuiguno api_keys.openai niOpenAI APIki-woZhi Ding shiteiruChang He nomiYou Xiao
OpenAI VLM OpenAI LLM APIwoHu biChu su(Hua Xiang Ru Li )
ExecutebotanwoYa shitataimingudeShi Xing suru
*konhuiguno api_keys.openai niOpenAI APIki-woZhi Ding shiteiruChang He nomiYou Xiao
OpenAI STT OpenAI Realtime APIdeYin Sheng woWen Zi Qi koshisuru
*konhuiguno api_keys.openai niOpenAI APIki-woZhi Ding shiteiruChang He nomiYou Xiao
OpenAI Image Generation OpenAI Image Generation APIdeHua Xiang woSheng Cheng suru
*konhuiguno api_keys.openai niOpenAI APIki-woZhi Ding shiteiruChang He nomiYou Xiao

Math

Math > Value
no-doMing Shuo Ming
Int Zheng Shu Zhi woChu Li
Float Fu Dong Xiao Shu Dian Zhi woChu Li
Clamp Zhi woZhi Ding Fan Wei Nei niZhi Xian suru
Float2Int Fu Dong Xiao Shu Dian woZheng Shu niBian Huan
Math > Operation
no-doMing Shuo Ming
Add 2tsunoShu Zhi woJia Suan (a + b)
Sub 2tsunoShu Zhi woJian Suan (a - b)
Mul 2tsunoShu Zhi woCheng Suan (a x b)
Div 2tsunoShu Zhi woChu Suan (a / b), zeroChu Suan Shi ha0woFan su
Mod Sheng Yu woJi Suan (a % b)
Abs Jue Dui Zhi woJi Suan
Sin Du Shu Fa noJiao Du karasainZhi woJi Suan
*degree:Jiao Du (-360~ 360Du )
Math > Logic
no-doMing Shuo Ming
AND Lun Li Ji (Liang Fang ga0Yi Wai de1, soreYi Wai ha0)
OR Lun Li He (dochirakaga0Yi Wai de1, Liang Fang 0de0)
NOT Lun Li Fou Ding (0de1, 0Yi Wai de0)
XOR Pai Ta De Lun Li He (Yi Fang nomi0Yi Wai de1)

Utility

Utility
no-doMing Shuo Ming
Elapsed Time Start karanoJing Guo Shi Jian woChu Li
Timer Trigger Zhi Ding Jian Ge detoriga-(1)woChu Li , soreYi Wai ha0
Trigger Button botanYa Xia Shi nitoriga-(1)woChu Li , soreYi Wai ha0

Directory Structure

+-- src/
| +-- node_editor/ # koaraiburari
| | +-- core.py # gurahuShi Xing enzin
| | +-- models.py # de-tamoderu(Node, Port, Connection)
| | +-- node_def.py # no-doDing Yi shisutemu
| | +-- commands.py # Undo/Redo
| | +-- settings.py # She Ding Guan Li
| | +-- image_utils.py # Hua Xiang yu-teiritei
| +-- nodes/ # no-doShi Zhuang
| | +-- image/ # Hua Xiang no-do
| | +-- audio/ # Yin Sheng no-do
| | +-- math/ # Shu Zhi Yan Suan no-do
| | +-- text/ # tekisutono-do
| | +-- openai/ # OpenAILian Xi no-do
| | +-- utility/ # yu-teiriteino-do
| +-- gui/
| +-- reactflow/ # ReactFlow
| | +-- backend/ # FastAPI batsukuendo
| | +-- frontend/ # React hurontoendo
| +-- headless/ # hetsudoresuShi Xing
+-- config.example.json # apurike-shiyonShe Ding
+-- download_weights.py # moderudaunro-dosukuriputo
+-- run_gui_reactflow.py # GUIQi Dong sukuriputo
+-- run_gui_reactflow_colab.ipynb # ColaboratoryYong no-tobutsuku
+-- run_headless.py # hetsudoresuShi Xing sukuriputo
+-- requirements.txt # PythonYi Cun patsuke-zi
+-- requirements-gpu.txt # GPUYong Zhui Jia patsuke-zi

Custom Node Development

Xin shiino-dowoZuo Cheng suruChang He ha src/nodes/// niZuo Cheng shimasu.
Ge no-doha, node.tomltoimpl.pyno2tsunohuairudeGou Cheng saremasu.
Yi Xia ha, Image/Filter/Cannyno-donoGou Cheng Li desu.

src/nodes/
+-- image/
+-- category.toml # kategoriShe Ding
+-- filter/
+-- category.toml # sabukategoriShe Ding
+-- canny/
+-- node.toml # no-dometade-ta
+-- impl.py # Shi Zhuang
kategoriDing Yi (category.toml)

Ge kategorihuorudani category.toml woPei Zhi shitesaidoba-noBiao Shi woZhi Yu shimasu.

display_name = "Image" # saidoba-niBiao Shi suruMing Qian
order = 10 # Biao Shi Shun Xu (Xiao saihodoShang )
default_open = false # saidoba-dedehuorutodeZhan Kai suruka
hui-rudo Xing dehuoruto Shuo Ming
display_name string huorudaMing saidoba-niBiao Shi suruMing Qian
order int 100 Biao Shi Shun Xu (Xiao saihodoShang )
default_open bool true dehuorutodeZhan Kai suruka
requires_config string null Bi Yao naShe Ding ki-(nullZhi Ding noChang He haChang niBiao Shi )
no-doDing Yi (node.toml)
Ji Ben Gou Cheng
name = "image.filter.canny"
version = "1.0.0"
display_name = "Canny"
description = "Applies Canny edge detection to an image."
order = 50
gui = ["reactflow", "headless"]

[[ports]]
name = "image"
data_type = "image"
direction = "inout"

[[ports]]
name = "low_threshold"
data_type = "float"
direction = "in"

[[ports]]
name = "high_threshold"
data_type = "float"
direction = "in"

[[properties]]
name = "low_threshold"
display_name = "Low"
type = "int"
default = 50
widget = "slider"
min = 0
max = 255

[[properties]]
name = "high_threshold"
display_name = "High"
type = "int"
default = 150
widget = "slider"
min = 0
max = 255
no-doShe Ding opushiyon
hui-rudo Xing dehuoruto Shuo Ming
name string Bi Xu no-doID(category.subcategory.nameXing Shi )
version string Bi Xu
display_name string name saidoba-/no-doniBiao Shi suruMing Qian
description string "" no-donoShuo Ming
order int 100 kategoriNei denoBiao Shi Shun Xu
gui string[] [] Dui Ying GUI(Kong =Quan Dui Ying , reactflow, headless)
measure_time bool true Chu Li Shi Jian Ji Ce noDui Xiang ka
run_when_stopped bool false STOPZhong moShi Xing suruka
po-toDing Yi ([[ports]])
[[ports]]
name = "image"
data_type = "image"
direction = "inout"
preview = true
hui-rudo Xing dehuoruto Shuo Ming
name string Bi Xu po-toMing
data_type string Bi Xu de-taXing (Xia Ji Can Zhao )
direction string "in" in, out, inout
display_name string name UIniBiao Shi suruMing Qian
preview bool true purebiyu-Biao Shi suruka

de-taXing Yi Lan :

de-taXing Shuo Ming
image Hua Xiang (numpyPei Lie ). Jie Sok Ke Neng : image
audio Yin Sheng de-ta. Jie Sok Ke Neng : audio
int Zheng Shu . Jie Sok Ke Neng : int, float
float Fu Dong Xiao Shu Dian . Jie Sok Ke Neng : int, float
string Wen Zi Lie . Jie Sok Ke Neng : string
trigger toriga-Xin Hao (0/1). Jie Sok Ke Neng : trigger
any Ren Yi noXing . Jie Sok Ke Neng : subete
puropateiDing Yi ([[properties]])
Ji Ben Gou Cheng
[[properties]]
name = "low_threshold"
display_name = "Low"
type = "int"
default = 50
widget = "slider"
min = 0
max = 255

[[properties]]
name = "high_threshold"
display_name = "High"
type = "int"
default = 150
widget = "slider"
min = 0
max = 255

puropateiopushiyon

hui-rudo Xing dehuoruto Shuo Ming
name string Bi Xu puropateiMing
display_name string name Biao Shi Ming
type string "float" de-taXing (int, float, string, bool)
default any null dehuorutoZhi
widget string "input" UIuizietsuto(Xia Ji Can Zhao )
min float null Zui Xiao Zhi (slider/number_inputYong )
max float null Zui Da Zhi (slider/number_inputYong )
step float null sutetsupuZhi
options array [] dropdownYong noXuan Ze Zhi
options_source string null Dong De opushiyonso-su(cameras, audio_inputs)
accept string null file_pickerdeXu Ke suruhuairutaipu
button_label string null buttonuizietsutonoraberu
rows int null text_areanoXing Shu
visible_when object null Tiao Jian Fu kiBiao Shi
disabled_while_streaming bool false Shi Xing Zhong haBian Ji Bu Ke
requires_streaming bool false Shi Xing Zhong nomiYou Xiao (botanYong )
requires_gpu bool false GPULi Yong Ke Neng Shi nomiBiao Shi
requires_api_key string null Zhi Ding APIki-gaShe Ding sareteiruShi nomiBiao Shi
uizietsutoYi Lan

Biao Zhun uizietsuto:

uizietsuto Shuo Ming
slider suraida-. min, max, step
number_input Shu Zhi Ru Li . min, max, step
text_input tekisutoRu Li (1Xing )
text_area tekisutoeria(Fu Shu Xing ). rows
text_display tekisutoBiao Shi (Du miQu riZhuan Yong )
dropdown dorotsupudaun. options, options_source
checkbox chietsukubotsukusu
color_picker Se Xuan Ze
file_picker huairuXuan Ze . accept
button botan. button_label, requires_streaming
xy_input XYZuo Biao Ru Li
matrix3x3 3x3Xing Lie Ru Li
uizietsutoLi

dorotsupudaun:

[[properties]]
name = "mode"
display_name = "Mode"
type = "string"
default = "auto"
widget = "dropdown"
options = [
{ value = "auto", label = "Auto" },
{ value = "manual", label = "Manual" }
]

botan:

[[properties]]
name = "reset"
display_name = ""
type = "bool"
default = false
widget = "button"
button_label = "Reset"

Tiao Jian Fu kiBiao Shi :

[[properties]]
name = "custom_value"
display_name = "Custom Value"
type = "int"
default = 100
widget = "slider"
visible_when = { property = "mode", values = ["manual"] }

huairuXuan Ze :

[[properties]]
name = "file_path"
display_name = "File"
type = "string"
default = ""
widget = "file_picker"
accept = "image/*"

GPUYi Cun puropatei:

[[properties]]
name = "use_gpu"
display_name = "Use GPU"
type = "bool"
default = true
widget = "checkbox"
disabled_while_streaming = true
requires_gpu = true

Shi Xing Zhong nomiYou Xiao nabotan:

[[properties]]
name = "capture"
display_name = ""
type = "bool"
default = false
widget = "button"
button_label = "Capture"
requires_streaming = true
Shi Zhuang (impl.py)
Ji Ben Gou Cheng (Cannyno-donoLi )
from typing import Dict, Any
from node_editor.node_def import ComputeLogic
import cv2


class CannyNodeLogic(ComputeLogic):
"""
CannyetsuziJian Chu woShi Xing suruno-dorozitsuku.
Ru Li *Chu Li tomoniOpenCVHua Xiang (numpyPei Lie ). Base64Bian Huan hacore.pydeZi Dong Chu Li .
"""

def compute(
self,
inputs: Dict[str, Any],
properties: Dict[str, Any],
context: Dict[str, Any] = None,
) -> Dict[str, Any]:
img = inputs.get("image")
if img is None:
return {"image": None}

low_threshold = int(properties.get("low_threshold", 50))
high_threshold = int(properties.get("high_threshold", 150))

# gure-suke-runiBian Huan
if len(img.shape) == 2 or img.shape[2] == 1:
gray = img
else:
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

edges = cv2.Canny(gray, low_threshold, high_threshold)

# Chu Li woBGRniBian Huan (Ta no-dotonoHu Huan Xing notame)
edges_bgr = cv2.cvtColor(edges, cv2.COLOR_GRAY2BGR)

return {"image": edges_bgr}

pointo:

  • kurasuMing haRen Yi (ComputeLogicwoJi Cheng )
  • compute()mesotsudogaBi Xu
  • Ru Li hainputsCi Shu karaQu De (po-toMing gaki-)
  • puropateihapropertiesCi Shu karaQu De
  • Li riZhi haChu Li po-toMing woki-toshitaCi Shu
kontekisutoQing Bao

context Yin Shu nihaYi Xia noQing Bao gaHan maremasu:

ki- Xing Shuo Ming
is_streaming bool STARTZhong kadouka
preview bool purebiyu-mo-do(STOPZhuang Tai )kadouka
loop bool ru-puZai Sheng gaYou Xiao kadouka
interval_ms int Shi Xing Jian Ge (miriMiao )
node_id string Xian Zai nono-doID
encode_base64 bool Hua Xiang woBase64enko-dosuruka
era-handoringunoLi
from typing import Dict, Any
from node_editor.node_def import ComputeLogic

try:
import cv2
CV2_AVAILABLE = True
except ImportError:
CV2_AVAILABLE = False


class BlurNodeLogic(ComputeLogic):
"""gaushianbura-woShi Yong """

def compute(
self,
inputs: Dict[str, Any],
properties: Dict[str, Any],
context: Dict[str, Any] = None
) -> Dict[str, Any]:
# Yi Cun raiburarinochietsuku
if not CV2_AVAILABLE:
return {"image": None, "__error__": "opencv-python is not installed"}

image = inputs.get("image")
if image is None:
return {"image": None}

kernel_size = int(properties.get("kernel_size", 5))

# ka-nerusaizuhaQi Shu dearuBi Yao gaaru
if kernel_size % 2 == 0:
kernel_size += 1

try:
blurred = cv2.GaussianBlur(image, (kernel_size, kernel_size), 0)
return {"image": blurred}
except Exception as e:
return {"image": None, "__error__": str(e)}
Te Shu naChu Li ki-
ki- Shuo Ming
__error__ era-metsuse-zi(no-doniChi kuBiao Shi )
__is_busy__ bizi-Zhuang Tai (truedebotanwoWu Xiao Hua )
__update_property__ puropateiZhi noGeng Xin (puropateiMing -Zhi nodict)
__display_text__ tekisutoBiao Shi no-doYong noBiao Shi tekisuto

Author

Gao Qiao kazuhito(https://x.com/KzhtTkhs)

License

Multimodal-Node-Editor is under Apache-2.0 license.
Multimodal-Node-Editornoso-suko-doZi Ti haApache-2.0 licensedesuga,
Ge AImoderunoraisensuha, sorezorenoraisensuniCong imasu.

About

Hua Xiang , o-deio, tekisuto, LLM/VLM nomaruchimo-daru paipurainwoShi Yan surutamenono-doedeita-(Node-based editor to compose and experiment with multimodal pipelines across image, audio, text, and LLM/VLM)

Topics

Resources

Readme

License

Apache-2.0 license

Stars

Watchers

Forks

Packages

Contributors