Merge branch 'main' of https://github.com/ATLAS-PJK-GM007/ZeaVis-Edu into selly/frontend

This commit is contained in:
seriouselly
2026-06-12 11:29:45 +07:00
30 changed files with 4207 additions and 1011 deletions
+1 -6
View File
@@ -1,6 +1 @@
best_model/best_model.keras filter=lfs diff=lfs merge=lfs -text
model/saved_model/variables/variables.data-00000-of-00001 filter=lfs diff=lfs merge=lfs -text
model/tfjs_model/*.bin filter=lfs diff=lfs merge=lfs -text
**/best_model.keras filter=lfs diff=lfs merge=lfs -text
**/variables.data* filter=lfs diff=lfs merge=lfs -text
**/tfjs_model/*.bin filter=lfs diff=lfs merge=lfs -text
# Model artifacts are downloaded from Hugging Face at CI time — not stored in this repo.
+34 -7
View File
@@ -28,8 +28,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
lfs: true
- name: Set image prefix
run: echo "IMAGE_PREFIX=ghcr.io/${GITHUB_REPOSITORY,,}" >> "$GITHUB_ENV"
@@ -37,19 +35,48 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Set up Python for ONNX conversion
- name: Set up Python for model download & export
if: matrix.service.name == 'ml'
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Generate ONNX model artifact
- name: Download ONNX model from Hugging Face
if: matrix.service.name == 'ml'
working-directory: Machine_Learning
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
python -m pip install --upgrade pip
python -m pip install 'tensorflow>=2.13.0' 'tf2onnx>=1.16.1' 'onnx>=1.16.0'
python convert_onnx.py
set -eu
echo "::group::Install huggingface_hub"
python -m pip install --upgrade pip -q
python -m pip install huggingface_hub -q
echo "::endgroup::"
echo "::group::Check HF_TOKEN"
if [ -z "${HF_TOKEN:-}" ]; then
echo "ERROR: HF_TOKEN secret is not set."
echo "Add it: https://github.com/ATLAS-PJK-GM007/ZeaVis-Edu/settings/secrets/actions"
exit 1
fi
echo "HF_TOKEN is set (length: ${#HF_TOKEN})"
echo "::endgroup::"
echo "::group::Download model.onnx"
python -c "
from huggingface_hub import hf_hub_download
import os
os.makedirs('model', exist_ok=True)
path = hf_hub_download(
repo_id='MythEclipse2737/corn-leaf-disease-classifier',
filename='model/model.onnx',
token=os.environ['HF_TOKEN'],
local_dir='.',
)
print(f'Downloaded: {path}')
"
ls -lh model/model.onnx
echo "::endgroup::"
- name: Log in to GHCR
uses: docker/login-action@v3
+2
View File
@@ -9,6 +9,8 @@ dist/
build/
coverage/
*.tsbuildinfo
.venv/
venv/
.DS_Store
+2 -3
View File
@@ -14,11 +14,10 @@ The ML pipeline lives under `Machine_Learning/`. The inference service lives und
cd Machine_Learning
```
Set up a Python environment:
Activate the Python environment (already exists at repo root):
```bash
python -m venv venv
source venv/bin/activate
source ../.venv/bin/activate
pip install -r requirements.txt
```
+19
View File
@@ -0,0 +1,19 @@
node_modules/
.bun/
.moon/cache/
.env
.env.*
!.env.example
.codegraph/
dist/
build/
coverage/
*.tsbuildinfo
venv/
.DS_Store
dataset/
.claude/
dataset_split/
dataset_jagung.zip
best_model/
model/
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e8e3a4f86ae1594aeb221c40db3bc80e87c05b05f386bbb628ff1da16397424
size 112903200
Binary file not shown.
@@ -1 +0,0 @@
©©ءّ¸ث‡ق¬§¢¥ؤج™إأà’–‎سèز÷ئخ ’ڑھ¸¥ؤي—·(،وظ—ئƒ„÷ٌ2:10851638082828504866
Binary file not shown.
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e3831052cde64562f0a4e51d5f224c2c819dd625a1d89b35ef58d74b82f1858
size 53214594
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:613179718d653f027cf5e70a02f223b423a98abaffe7120ef626793aec23177f
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:20325958a36f2afdf149083a4a2079aee4725bb3d0c01f96694f617bb7c29a27
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:853bb38c854ff011b06ed98d7037b25501e02a0b718e65159acbbc62e97874ae
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bf9a4b64fb22a0e83a2862f119a61842a72ec8ba15981ece3291b049dd4db11a
size 2432372
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:84e7ddbfcf1fd5d55bf1198cc65fb024ff9110e544195779b42281245246b16d
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8d98aa415626d122e2a324d5ffd3a536bcd03f42957114bc23b3d4fc706c7eb8
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:30cb3863522eb0e1f1688249f58629ed1e1f03e82d723fde7a13763ade2d60da
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2f8fffbfc14379d113100374729ce0236306b6f1aa8b32808e07020adfb3f0a0
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:516a4af1cae907747b7dc91169579a29bd477a99ad4e638fbb205a9b0acaae9d
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:76757c0eb38bc7c38c12d8a8ab8ab68b83d17f377cceca985905e5c48dece798
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f5f1a2bd449370e6ffd13143d78e6d1b2803df4cbae968fee5ab4244349a64b5
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c79a0792c83c7bcbf8cb2768d3e6d12a928f95506c64c6818698cea0e3f2fec
size 4194304
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6c5a36d2a203ece90b2978f5295f42c7a0f9ff72a2edcdbcaa306e67c29b3d72
size 4194304
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+12 -7
View File
@@ -1,7 +1,12 @@
tensorflow>=2.13.0
tensorflowjs>=4.10.0
jupyter>=1.0.0
ipykernel>=6.25.0
tf2onnx>=1.16.1
onnx>=1.16.0
onnxruntime>=1.17.0
tensorflow==2.19.0 # CPU + Colab; for local GPU, install tensorflow[and-cuda]
tensorflowjs==4.22.0
gdown # download dataset from Google Drive
numpy
matplotlib
seaborn
pillow
split-folders
scikit-learn
tf2onnx
onnxruntime
huggingface_hub
+1 -1
View File
@@ -25,7 +25,7 @@ def build_clean_model(num_classes, img_size=(224, 224)):
x = layers.Dense(1024, activation='swish')(x)
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.4)(x)
outputs = layers.Dense(num_classes, activation='sigmoid', dtype='float32')(x)
outputs = layers.Dense(num_classes, activation='softmax', dtype='float32')(x)
return models.Model(inputs, outputs)
logging.info("=== EXPORT STARTED ===")
+258
View File
@@ -0,0 +1,258 @@
#!/usr/bin/env python3
"""Upload trained model artifacts to Hugging Face Hub.
Runs the full export pipeline (SavedModel → TFLite → ONNX → TFJS) then
pushes all artifacts to a Hugging Face model repository.
Requires ``HF_TOKEN`` environment variable to be set for authentication.
"""
import json
import logging
import os
import subprocess
import sys
from pathlib import Path
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
)
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
HF_REPO = "MythEclipse2737/corn-leaf-disease-classifier"
CLASS_NAMES = ["Bercak Daun", "Daun Sehat", "Hawar Daun", "Karat Daun"]
# Paths relative to this script's directory
SCRIPT_DIR = Path(__file__).resolve().parent
MODEL_DIR = SCRIPT_DIR / "model"
SAVED_MODEL_DIR = MODEL_DIR / "saved_model"
BEST_MODEL = SCRIPT_DIR / "best_model" / "best_model.keras"
TFLITE_PATH = MODEL_DIR / "model.tflite"
ONNX_PATH = MODEL_DIR / "model.onnx"
TFJS_DIR = MODEL_DIR / "tfjs_model"
LABELS_PATH = MODEL_DIR / "labels.json"
README_PATH = MODEL_DIR / "README.md"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def run_cmd(cmd: list[str], *, env: dict | None = None, cwd=None) -> None:
"""Run a subprocess command, logging and raising on failure."""
label = " ".join(str(p) for p in cmd)
logging.info("Running: %s", label)
run_env = os.environ.copy()
if env:
run_env.update(env)
subprocess.run(cmd, check=True, env=run_env, cwd=cwd)
# ---------------------------------------------------------------------------
# Export pipeline
# ---------------------------------------------------------------------------
def run_export_pipeline() -> None:
"""Execute save_model.py, convert_onnx.py, and the TFJS converter."""
# 1. SavedModel + TFLite
run_cmd([sys.executable, str(SCRIPT_DIR / "save_model.py")])
# 2. ONNX
run_cmd([sys.executable, str(SCRIPT_DIR / "convert_onnx.py")])
# 3. TensorFlow.js (non-blocking — known protobuf version issue)
logging.info("Converting SavedModel to TensorFlow.js format...")
try:
run_cmd(
[
"tensorflowjs_converter",
"--input_format=tf_saved_model",
"--output_format=tfjs_graph_model",
"--signature_name=serving_default",
"--saved_model_tags=serve",
str(SAVED_MODEL_DIR),
str(TFJS_DIR),
],
env={"PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION": "python"},
)
except subprocess.CalledProcessError:
logging.warning(
"TFJS conversion failed (likely protobuf version mismatch). "
"Skipping — SavedModel, TFLite, and ONNX are still available."
)
logging.info("Export pipeline completed.")
# ---------------------------------------------------------------------------
# Hugging Face upload
# ---------------------------------------------------------------------------
def generate_labels_json() -> None:
"""Write `labels.json` so downstream tools know the class order."""
MODEL_DIR.mkdir(parents=True, exist_ok=True)
with open(LABELS_PATH, "w") as fh:
json.dump(CLASS_NAMES, fh, ensure_ascii=False, indent=2)
logging.info("Labels written to %s", LABELS_PATH)
def generate_readme() -> None:
"""Write a minimal HF model card."""
content = """---
language:
- id
tags:
- agriculture
- corn
- leaf-disease
- efficientnet-v2
- tensorflow
- image-classification
license: mit
datasets:
- zeavis-edu/corn-leaf-dataset
---
# ZeaVis Edu — Corn Leaf Disease Classifier
Classifies corn leaf diseases into one of four categories:
- **Bercak Daun** — Gray Leaf Spot
- **Hawar Daun** — Northern / Southern Leaf Blight
- **Karat Daun** — Common Rust
- **Daun Sehat** — Healthy corn leaf
## Model
| Attribute | Detail |
| ------------------ | --------------------------------------------------- |
| Architecture | EfficientNetV2B0 (transfer learning) |
| Input | RGB image, 224×224 pixels |
| Output | Softmax probabilities over 4 classes |
| Framework | TensorFlow 2.x / Keras (float32) |
| Augmentation | Flip, Rotation, Zoom, MixUp, CutMix, RandomErasing |
| Optimizer | AdamW + EMA + Label Smoothing 0.2 |
| Training | 3-phase: Head → Partial FT → Full FT |
## Usage
```python
import tensorflow as tf
import numpy as np
from PIL import Image
model = tf.keras.models.load_model("best_model.keras")
img = Image.open("corn_leaf.jpg").resize((224, 224))
x = tf.keras.applications.efficientnet_v2.preprocess_input(
np.expand_dims(np.array(img), 0).astype("float32")
)
preds = model.predict(x)
print(["Bercak Daun", "Daun Sehat", "Hawar Daun", "Karat Daun"][np.argmax(preds)])
```
## Files
| File | Format | Use |
| ---------------------- | --------------- | ------------------------- |
| `best_model.keras` | Keras v3 | Training / fine-tuning |
| `model.saved_model/` | TF SavedModel | TensorFlow Serving |
| `model.tflite` | TFLite | Mobile / edge devices |
| `model.onnx` | ONNX | Cross-platform inference |
| `model.tfjs_model/` | TensorFlow.js | Browser / Node.js |
| `labels.json` | JSON | Class label mapping |
## Limitations
This model is intended for **educational and research purposes** only.
Always consult with agricultural experts before making crop management
decisions.
"""
with open(README_PATH, "w") as fh:
fh.write(content)
logging.info("README written to %s", README_PATH)
def upload_to_hub() -> None:
"""Upload all artifacts to the Hugging Face Hub repository."""
from huggingface_hub import HfApi, create_repo, login
login(token=os.environ["HF_TOKEN"])
api = HfApi()
# Ensure repo exists (public)
create_repo(HF_REPO, repo_type="model", exist_ok=True, private=False)
logging.info("Repo ready: https://huggingface.co/%s", HF_REPO)
# --- Single files ---
files_to_upload = [
(BEST_MODEL, "best_model.keras"),
(TFLITE_PATH, "model/model.tflite"),
(ONNX_PATH, "model/model.onnx"),
(LABELS_PATH, "model/labels.json"),
(README_PATH, "README.md"),
]
for local_path, repo_path in files_to_upload:
if not local_path.exists():
logging.warning("Skipping missing file: %s", local_path)
continue
logging.info("Uploading %s%s", local_path.name, repo_path)
api.upload_file(
path_or_fileobj=str(local_path),
path_in_repo=repo_path,
repo_id=HF_REPO,
repo_type="model",
)
# --- Folders ---
folders_to_upload = [
(SAVED_MODEL_DIR, "model/saved_model"),
]
if TFJS_DIR.exists():
folders_to_upload.append((TFJS_DIR, "model/tfjs_model"))
else:
logging.info("Skipping TFJS folder (not generated).")
for local_dir, repo_dir in folders_to_upload:
if not local_dir.exists():
logging.warning("Skipping missing folder: %s", local_dir)
continue
logging.info("Uploading folder %s%s", local_dir.name, repo_dir)
api.upload_folder(
folder_path=str(local_dir),
path_in_repo=repo_dir,
repo_id=HF_REPO,
repo_type="model",
)
logging.info(
"Upload complete! Visit https://huggingface.co/%s", HF_REPO
)
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def main() -> None:
token = os.environ.get("HF_TOKEN")
if not token:
logging.warning(
"HF_TOKEN environment variable is not set. "
"Skipping Hugging Face upload. "
"On Colab, set it via the Secrets manager (🔑 key icon in the left panel)."
)
return
logging.info("=== HUGGING FACE UPLOAD PIPELINE ===")
# 1. Run the export pipeline to generate all artifacts
run_export_pipeline()
# 2. Generate metadata files
generate_labels_json()
generate_readme()
# 3. Upload everything to Hugging Face Hub
upload_to_hub()
if __name__ == "__main__":
main()
+139
View File
@@ -0,0 +1,139 @@
# 📊 Laporan Perubahan — `ddced09` → `8137057` (HEAD)
**Periode:** 11 Juni 2026, 15:17 — 19:42 UTC
**Branch:** `main`
**Total commit:** 6 (ddced09 tidak termasuk, itu adalah base)
---
## 📜 Daftar Commit
| # | Hash | Tanggal | Deskripsi |
|---|------|---------|-----------|
| 1 | `b83b2e1` | 15:18 | **feat**: download dataset dari Google Drive saat running locally |
| 2 | `7a7e5b5` | 15:19 | **fix**: update Google Drive file ID ke link dataset yang benar |
| 3 | `8c65e33` | 16:31 | **chore**: clear notebook outputs dan tambah `.gitignore` |
| 4 | `c733f3b` | 16:39 | *WIP commit* |
| 5 | `4377be1` | 16:49 | **chore**: update notebook execution count dan outputs |
| 6 | `8137057` | 19:42 | *WIP commit — hasil kerja sesi Hugging Face integration* |
---
## 📁 File yang Berubah (dari `ddced09` → HEAD)
| File | Perubahan | Keterangan |
|------|-----------|------------|
| `Machine_Learning/notebook.ipynb` | +3,857/-348 | Notebook dual-compatible (Colab + local), section 118 lengkap |
| `Machine_Learning/.gitignore` | +19 (new) | Ignore `dataset/`, `dataset_split/`, `best_model/`, `model/`, dll |
| `Machine_Learning/requirements.txt` | +2 | Tambah `gdown` + `huggingface_hub` |
| `Machine_Learning/save_model.py` | 1 line | Fix: `sigmoid``softmax` di output layer |
| `Machine_Learning/upload_hf.py` | +258 (new) | Script export pipeline + upload ke Hugging Face |
| `model/model.tflite` | binary | Regenerated (softmax fix) |
| `model/saved_model/` | binary | Regenerated |
**Total:** 9 file, +3,792 insertions, -348 deletions
---
## 🔍 Detail Perubahan per Area
### 1. Dataset Download (`b83b2e1`, `7a7e5b5`)
- Tambah dependency `gdown` ke `requirements.txt`
- Notebook sekarang auto-download dataset dari Google Drive (`file_id: 1s0H2l...`) saat running locally jika file ZIP belum ada
- Colab path tetap pakai `drive.mount()`
- Drive file ID diupdate di commit `7a7e5b5`
### 2. Notebook Restructuring (`ddced09` → `8c65e33` → `4377be1`)
- Notebook direstruktur dari numbering 1/2/4 menjadi section bernomor rapi 117 (kemudian 18)
- Semua section punya header markdown yang deskriptif
- Hyperparameter (seed, IMG_SIZE, BATCH_SIZE) ditambahkan
- Import diperluas: `AdamW`, `compute_class_weight`, `preprocess_input`, `Counter`, `random`
- Mixed precision policy `float32` eksplisit
- Cell outputs cleared di commit `8c65e33`
- Execution count dan outputs diupdate di `4377be1`
### 3. `.gitignore` (`8c65e33`)
File baru `Machine_Learning/.gitignore` mengabaikan:
- `dataset/`, `dataset_split/`, `dataset_jagung.zip`
- `best_model/`, `model/`
- Path development lainnya (`node_modules/`, `.bun/`, `.moon/cache/`, `.env`, `dist/`, `build/`, `coverage/`, `venv/`, `.claude/`, dll)
### 4. `save_model.py` Fix (`8137057`)
```diff
- outputs = layers.Dense(num_classes, activation='sigmoid', dtype='float32')(x)
+ outputs = layers.Dense(num_classes, activation='softmax', dtype='float32')(x)
```
Perbaikan kritis: output 4 kelas harus softmax, bukan sigmoid.
### 5. Hugging Face Upload (`8137057`) 🔥 **Fitur Baru**
**`Machine_Learning/upload_hf.py`** — 258 lines script mandiri:
```
Pipeline:
save_model.py → convert_onnx.py → TFJS converter (optional)
→ labels.json + README.md
→ upload ke Hugging Face Hub
```
- Repo: `MythEclipse2737/corn-leaf-disease-classifier`
- Auth via `HF_TOKEN` env var
- TFJS conversion graceful-fail (protobuf version conflict known)
- Upload semua: `best_model.keras` (213MB) + TFLite + ONNX + SavedModel + TFJS (partial) + labels.json + README.md
- **Hasil:** 11 files terupload ke https://huggingface.co/MythEclipse2737/corn-leaf-disease-classifier
**Notebook Section 18** — cell markdown + code untuk call `upload_hf.py` otomatis.
---
## 📈 Working Tree (Uncommitted Changes)
Saat ini **tidak ada uncommitted changes**`8137057` adalah commit terakhir yang mencakup semua hasil kerja.
---
## ✅ Ringkasan Outcome
| Goal | Status |
|------|--------|
| Notebook dual-compatible (Colab + local) | ✅ |
| Auto-download dataset dari Google Drive | ✅ |
| `.gitignore` untuk artifacts besar | ✅ |
| Fix `sigmoid``softmax` | ✅ |
| Hugging Face integration (upload script + notebook section) | ✅ |
| Model artifacts uploaded to HF Hub | ✅ 11 files |
| Temporarily pinned to `MythEclipse2737/` namespace | ⚠️ not `zeavis-edu/` (no org access) |
### Files on Hugging Face
```
MythEclipse2737/corn-leaf-disease-classifier
├── .gitattributes
├── README.md
├── best_model.keras (213 MB)
├── model/
│ ├── labels.json
│ ├── model.onnx
│ ├── model.tflite
│ ├── saved_model/
│ │ ├── fingerprint.pb
│ │ ├── saved_model.pb
│ │ └── variables/
│ │ ├── variables.data-00000-of-00001
│ │ └── variables.index
│ └── tfjs_model/
│ └── model.json (partial — no weight shards)
```
---
## ⚠️ Catatan
1. **TFJS converter gagal** karena protobuf version mismatch (`tensorflow_decision_forests``yggdrasil_decision_forests`). Model TF.js di HF hanya berisi `model.json` (metadata saja, tidak ada weight shards). Issue ini pre-existing dan tidak blocking.
2. **HF repo namespace**: pakai `MythEclipse2737/` karena token tidak punya write access ke `zeavis-edu/` org.
3. **Token HF terekspos** di chat — perlu di-rotate di https://huggingface.co/settings/tokens.