MythEclipse/zeavis-edu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
7a7e5b59550f2fc0ec3999b6e153b0c83b1e7428
zeavis-edu/Machine_Learning/notebook.ipynb
T

33 KiB
Raw Blame History

  1. Persiapan Lingkungan
In [1]:
!pip install -r requirements.txt
Requirement already satisfied: tensorflow==2.19.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from -r requirements.txt (line 1)) (2.19.0)
Requirement already satisfied: tensorflowjs==4.22.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from -r requirements.txt (line 2)) (4.22.0)
Requirement already satisfied: numpy in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from -r requirements.txt (line 3)) (2.1.3)
Collecting matplotlib (from -r requirements.txt (line 4))
  Using cached matplotlib-3.10.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (52 kB)
Collecting seaborn (from -r requirements.txt (line 5))
  Using cached seaborn-0.13.2-py3-none-any.whl.metadata (5.4 kB)
Collecting pillow (from -r requirements.txt (line 6))
  Using cached pillow-12.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (8.8 kB)
Collecting split-folders (from -r requirements.txt (line 7))
  Using cached split_folders-0.6.1-py3-none-any.whl.metadata (11 kB)
Collecting scikit-learn (from -r requirements.txt (line 8))
  Using cached scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (11 kB)
Requirement already satisfied: tf2onnx in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from -r requirements.txt (line 9)) (1.17.0)
Requirement already satisfied: onnxruntime in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from -r requirements.txt (line 10)) (1.26.0)
Requirement already satisfied: absl-py>=1.0.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (2.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (1.6.3)
Requirement already satisfied: flatbuffers>=24.3.25 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (25.12.19)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (0.7.0)
Requirement already satisfied: google-pasta>=0.1.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (0.2.0)
Requirement already satisfied: libclang>=13.0.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (18.1.1)
Requirement already satisfied: opt-einsum>=2.3.2 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (3.4.0)
Requirement already satisfied: packaging in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (23.2)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.3 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (5.29.6)
Requirement already satisfied: requests<3,>=2.21.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (2.34.2)
Requirement already satisfied: setuptools in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (82.0.1)
Requirement already satisfied: six>=1.12.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (1.17.0)
Requirement already satisfied: termcolor>=1.1.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (3.3.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (4.15.0)
Requirement already satisfied: wrapt>=1.11.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (2.2.1)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (1.81.1)
Requirement already satisfied: tensorboard~=2.19.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (2.19.0)
Requirement already satisfied: keras>=3.5.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (3.14.1)
Requirement already satisfied: h5py>=3.11.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (3.14.0)
Requirement already satisfied: ml-dtypes<1.0.0,>=0.5.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow==2.19.0->-r requirements.txt (line 1)) (0.5.4)
Requirement already satisfied: flax>=0.7.2 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.12.7)
Requirement already satisfied: importlib_resources>=5.9.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflowjs==4.22.0->-r requirements.txt (line 2)) (7.1.0)
Requirement already satisfied: jax>=0.4.13 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.10.1)
Requirement already satisfied: jaxlib>=0.4.13 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.10.1)
Requirement already satisfied: tf-keras>=2.13.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflowjs==4.22.0->-r requirements.txt (line 2)) (2.19.0)
Requirement already satisfied: tensorflow-decision-forests>=1.5.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflowjs==4.22.0->-r requirements.txt (line 2)) (1.12.0)
Requirement already satisfied: tensorflow-hub>=0.16.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.16.1)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from requests<3,>=2.21.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from requests<3,>=2.21.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from requests<3,>=2.21.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from requests<3,>=2.21.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (2026.5.20)
Requirement already satisfied: markdown>=2.6.8 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorboard~=2.19.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (3.10.2)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorboard~=2.19.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorboard~=2.19.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (3.1.8)
Collecting contourpy>=1.0.1 (from matplotlib->-r requirements.txt (line 4))
  Using cached contourpy-1.3.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (5.5 kB)
Collecting cycler>=0.10 (from matplotlib->-r requirements.txt (line 4))
  Using cached cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB)
Collecting fonttools>=4.22.0 (from matplotlib->-r requirements.txt (line 4))
  Using cached fonttools-4.63.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (118 kB)
Collecting kiwisolver>=1.3.1 (from matplotlib->-r requirements.txt (line 4))
  Using cached kiwisolver-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (5.1 kB)
Collecting pyparsing>=3 (from matplotlib->-r requirements.txt (line 4))
  Using cached pyparsing-3.3.2-py3-none-any.whl.metadata (5.8 kB)
Requirement already satisfied: python-dateutil>=2.7 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from matplotlib->-r requirements.txt (line 4)) (2.9.0.post0)
Requirement already satisfied: pandas>=1.2 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from seaborn->-r requirements.txt (line 5)) (3.0.3)
Requirement already satisfied: scipy>=1.10.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from scikit-learn->-r requirements.txt (line 8)) (1.17.1)
Collecting joblib>=1.4.0 (from scikit-learn->-r requirements.txt (line 8))
  Using cached joblib-1.5.3-py3-none-any.whl.metadata (5.5 kB)
Collecting narwhals>=2.0.1 (from scikit-learn->-r requirements.txt (line 8))
  Using cached narwhals-2.22.1-py3-none-any.whl.metadata (15 kB)
Collecting threadpoolctl>=3.5.0 (from scikit-learn->-r requirements.txt (line 8))
  Using cached threadpoolctl-3.6.0-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: onnx>=1.14.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tf2onnx->-r requirements.txt (line 9)) (1.21.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from astunparse>=1.6.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (0.45.1)
Requirement already satisfied: msgpack in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (1.2.0)
Requirement already satisfied: optax in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.2.8)
Requirement already satisfied: orbax-checkpoint in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.12.0)
Requirement already satisfied: tensorstore in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.1.84)
Requirement already satisfied: rich>=11.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (15.0.0)
Requirement already satisfied: PyYAML>=5.4.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (6.0.3)
Requirement already satisfied: treescope>=0.1.7 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.1.10)
Requirement already satisfied: namex in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from keras>=3.5.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (0.1.0)
Requirement already satisfied: optree in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from keras>=3.5.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (0.19.1)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from rich>=11.1->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (4.2.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from rich>=11.1->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (2.20.0)
Requirement already satisfied: mdurl~=0.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=11.1->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.1.2)
Requirement already satisfied: wurlitzer in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow-decision-forests>=1.5.0->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (3.1.1)
Requirement already satisfied: ydf>=0.11.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from tensorflow-decision-forests>=1.5.0->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.16.0)
Requirement already satisfied: markupsafe>=2.1.1 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from werkzeug>=1.0.1->tensorboard~=2.19.0->tensorflow==2.19.0->-r requirements.txt (line 1)) (3.0.3)
Requirement already satisfied: etils[epath,epy] in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (1.14.0)
Requirement already satisfied: prometheus-client>=0.20.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.25.0)
Requirement already satisfied: aiofiles in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (25.1.0)
Requirement already satisfied: humanize in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (4.15.0)
Requirement already satisfied: simplejson>=3.16.0 in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (4.1.1)
Requirement already satisfied: psutil in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (7.2.2)
Requirement already satisfied: uvloop in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (0.22.1)
Requirement already satisfied: fsspec in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from etils[epath,epy]->orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (2026.4.0)
Requirement already satisfied: zipp in /home/asephs/ZeaVis-Edu/.venv/lib/python3.12/site-packages (from etils[epath,epy]->orbax-checkpoint->flax>=0.7.2->tensorflowjs==4.22.0->-r requirements.txt (line 2)) (4.1.0)
Using cached matplotlib-3.10.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (8.8 MB)
Using cached seaborn-0.13.2-py3-none-any.whl (294 kB)
Using cached pillow-12.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (7.1 MB)
Using cached split_folders-0.6.1-py3-none-any.whl (13 kB)
Using cached scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (9.1 MB)
Using cached contourpy-1.3.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (362 kB)
Using cached cycler-0.12.1-py3-none-any.whl (8.3 kB)
Using cached fonttools-4.63.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (5.0 MB)
Using cached joblib-1.5.3-py3-none-any.whl (309 kB)
Using cached kiwisolver-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (1.5 MB)
Using cached narwhals-2.22.1-py3-none-any.whl (454 kB)
Using cached pyparsing-3.3.2-py3-none-any.whl (122 kB)
Using cached threadpoolctl-3.6.0-py3-none-any.whl (18 kB)
Installing collected packages: threadpoolctl, split-folders, pyparsing, pillow, narwhals, kiwisolver, joblib, fonttools, cycler, contourpy, scikit-learn, matplotlib, seaborn
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/13 [seaborn]2/13 [seaborn]ib]n]
Successfully installed contourpy-1.3.3 cycler-0.12.1 fonttools-4.63.0 joblib-1.5.3 kiwisolver-1.5.0 matplotlib-3.10.9 narwhals-2.22.1 pillow-12.2.0 pyparsing-3.3.2 scikit-learn-1.9.0 seaborn-0.13.2 split-folders-0.6.1 threadpoolctl-3.6.0
In [ ]:
import os
import shutil
import zipfile
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import splitfolders
from PIL import Image

import tensorflow as tf
from tensorflow.keras import layers, models, callbacks
from tensorflow.keras.applications import EfficientNetV2B0
from sklearn.metrics import classification_report, confusion_matrix, multilabel_confusion_matrix

# Detect environment: Colab or local
try:
    from google.colab import drive
    IS_COLAB = True
    print("✅ Running on Google Colab")
except ModuleNotFoundError:
    IS_COLAB = False
    print(f"✅ Running locally (TF {tf.__version__}, GPU: {tf.config.list_physical_devices('GPU')})")

print(f"TensorFlow Version: {tf.__version__}")
  1. Download dan Ekstraksi Dataset
In [ ]:
if IS_COLAB:
    drive.mount('/content/drive')
    archive_path = '/content/drive/MyDrive/jagung/dataset_jagung.zip'
    destination_path = '/content/dataset_jagung.zip'
    extract_path = '/content/dataset'
else:
    import gdown
    base = os.getcwd()
    archive_path = os.path.join(base, 'dataset_jagung.zip')
    destination_path = archive_path
    extract_path = os.path.join(base, 'dataset')
    # If dataset doesn't exist locally, download from Google Drive
    DRIVE_FILE_ID = "1s0H2lDOQVCixywk5eZXJz2i9jj4JihxJ"
    if not os.path.exists(archive_path):
        print("Dataset not found locally. Downloading from Google Drive...")
        try:
            gdown.download(f"https://drive.google.com/uc?id={DRIVE_FILE_ID}", archive_path, quiet=False)
        except Exception as e:
            print(f"Download failed: {e}")

if os.path.exists(destination_path):
    if not os.path.exists(extract_path) or len(os.listdir(extract_path)) == 0:
        os.makedirs(extract_path, exist_ok=True)
        print("Extracting dataset...")
        try:
            with zipfile.ZipFile(destination_path, 'r') as zip_ref:
                zip_ref.extractall(path=extract_path)
            print("Extraction completed!")
        except Exception as e:
            print(f"Extraction failed: {e}")
    else:
        print("Dataset is already extracted and ready for use.")
  1. Data Cleaning dan Validasi Gambat (RGB)
In [ ]:
if IS_COLAB:
    dataset_path = "/content/dataset/dataset_jagung_v1"
else:
    dataset_path = os.path.join(extract_path, "dataset_jagung_v1")

def clean_image_data(directory):
    removed_count = 0
    for root, dirs, files in os.walk(directory):
        for file in files:
            file_path = os.path.join(root, file)
            try:
                img = Image.open(file_path)
                img.verify()

                img = Image.open(file_path)
                if img.mode != 'RGB':
                    img = img.convert('RGB')
                    img.save(file_path)
            except Exception as e:
                print(f"Removing corrupt/invalid file: {file_path}")
                os.remove(file_path)
                removed_count += 1
    return removed_count

print("Starting data cleanup process...")
removed = clean_image_data(dataset_path)
print(f"Cleanup completed. Total of {removed} problematic files removed.")
  1. Data Spliting (70/15/15/)
In [ ]:
if IS_COLAB:
    output_dir = "/content/dataset_split"
else:
    output_dir = os.path.join(os.getcwd(), "dataset_split")

if os.path.exists(output_dir):
    shutil.rmtree(output_dir)

print("Splitting dataset into Train, Validation, and Test...")
splitfolders.ratio(dataset_path, output=output_dir,
                   seed=42, ratio=(0.7, 0.15, 0.15),
                   group_prefix=None, move=False)
print("Dataset splitting completed.")

train_dir = os.path.join(output_dir, 'train')
val_dir = os.path.join(output_dir, 'val')
test_dir = os.path.join(output_dir, 'test')
  1. Data Loader dan Augmentasi
In [ ]:
BATCH_SIZE = 32
IMG_SIZE = (224, 224)

train_ds = tf.keras.utils.image_dataset_from_directory(
    train_dir, shuffle=True, batch_size=BATCH_SIZE, image_size=IMG_SIZE)

val_ds = tf.keras.utils.image_dataset_from_directory(
    val_dir, shuffle=True, batch_size=BATCH_SIZE, image_size=IMG_SIZE)

test_ds = tf.keras.utils.image_dataset_from_directory(
    test_dir, shuffle=False, batch_size=BATCH_SIZE, image_size=IMG_SIZE)

class_names = train_ds.class_names
NUM_CLASSES = len(class_names)
print(f"Number of classes: {NUM_CLASSES}")
print(f"Class names: {class_names}")

def to_one_hot(image, label):
    return image, tf.one_hot(label, NUM_CLASSES)

train_ds = train_ds.map(to_one_hot, num_parallel_calls=tf.data.AUTOTUNE)
val_ds = val_ds.map(to_one_hot, num_parallel_calls=tf.data.AUTOTUNE)
test_ds = test_ds.map(to_one_hot, num_parallel_calls=tf.data.AUTOTUNE)

AUTOTUNE = tf.data.AUTOTUNE
train_ds = train_ds.prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.prefetch(buffer_size=AUTOTUNE)
test_ds = test_ds.prefetch(buffer_size=AUTOTUNE)

data_augmentation = tf.keras.Sequential([
    layers.RandomFlip("horizontal_and_vertical"),
    layers.RandomRotation(0.2),
    layers.RandomZoom(0.2),
    layers.RandomTranslation(0.1, 0.1),
    layers.RandomContrast(0.2),
    layers.RandomBrightness(0.2)
], name="data_augmentation")
  1. Visualisasi Sample dan Hasil Augmentasi
In [ ]:
plt.figure(figsize=(12, 6))
for images, labels in train_ds.take(1):
    for i in range(4):
        ax = plt.subplot(2, 4, i + 1)
        plt.imshow(images[i].numpy().astype("uint8"))
        orig_class_idx = np.argmax(labels[i].numpy())

        plt.title(f"Ori: {class_names[orig_class_idx]}")
        plt.axis("off")

    augmented_images = data_augmentation(images, training=True)
    for i in range(4):
        ax = plt.subplot(2, 4, i + 5)
        plt.imshow(augmented_images[i].numpy().astype("uint8"))
        orig_class_idx = np.argmax(labels[i].numpy())

        plt.title(f"Aug: {class_names[orig_class_idx]}")
        plt.axis("off")
plt.tight_layout()
plt.show()
  1. Arsitektur Model CNN (EfficientNetv2B0)
In [ ]:
if IS_COLAB:
    checkpoint_path = '/content/best_model/best_model.keras'
else:
    checkpoint_path = os.path.join(os.getcwd(), 'best_model', 'best_model.keras')

if os.path.exists(checkpoint_path):
    print(f"Loading weights from model checkpoint: {checkpoint_path}...")
    try:
        model = models.load_model(checkpoint_path)
    except Exception as e:
        print(f"Failed to load checkpoint model: {e}.")
        model, base_model = build_model(NUM_CLASSES)
else:
    print("No checkpoint found.")
    model, base_model = build_model(NUM_CLASSES)

model.summary()
  1. Implementasi Callbacks
In [ ]:
checkpoint_cb = callbacks.ModelCheckpoint(
    "best_model/best_model.keras",
    save_best_only=True,
    monitor="val_loss",
    mode="min"
)

early_stopping_cb = callbacks.EarlyStopping(
    monitor="val_loss",
    patience=7,
    restore_best_weights=True,
    mode="min"
)

reduce_lr_cb = callbacks.ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.2,
    patience=3,
    min_lr=1e-6,
    verbose=1,
    mode="min"
)

callbacks_list = [checkpoint_cb, early_stopping_cb, reduce_lr_cb]
  1. Training Model
In [ ]:
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
    loss=tf.keras.losses.BinaryCrossentropy(),
    metrics=[tf.keras.metrics.BinaryAccuracy(name='accuracy')]
)

print("Training...")
EPOCHS_PHASE_1 = 15

history_1 = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=EPOCHS_PHASE_1,
    callbacks=callbacks_list
)
  1. Plot History
In [ ]:
def plot_history(history_1, initial_epochs):
    acc = history_1.history['accuracy']
    val_acc = history_1.history['val_accuracy']
    loss = history_1.history['loss']
    val_loss = history_1.history['val_loss']

    plt.figure(figsize=(16, 6))

    plt.subplot(1, 2, 1)
    plt.plot(acc, label='Training Accuracy')
    plt.plot(val_acc, label='Validation Accuracy')
    plt.legend(loc='lower right')
    plt.title('Training and Validation Accuracy')

    plt.subplot(1, 2, 2)
    plt.plot(loss, label='Training Loss')
    plt.plot(val_loss, label='Validation Loss')
    plt.legend(loc='upper right')
    plt.title('Training and Validation Loss')
    plt.show()

plot_history(history_1, EPOCHS_PHASE_1)
  1. Test Time Augmentation
In [ ]:
print("Evaluating model using TTA (Test-Time Augmentation) for Multi-Label...")

test_images = []
test_labels_true = []

for images, labels in test_ds.unbatch():
    test_images.append(images.numpy())
    test_labels_true.append(labels.numpy())

test_images = np.array(test_images)
test_labels_true = np.array(test_labels_true)

TTA_STEPS = 5
tta_predictions = []

for i in range(TTA_STEPS):
    print(f"TTA Step {i+1}/{TTA_STEPS}")
    aug_images = data_augmentation(test_images, training=True)
    preds = model.predict(aug_images, batch_size=BATCH_SIZE, verbose=0)
    tta_predictions.append(preds)

mean_tta_preds = np.mean(tta_predictions, axis=0)

THRESHOLD = 0.5
test_preds_binary = (mean_tta_preds > THRESHOLD).astype(int)

correct_predictions = np.equal(test_preds_binary, test_labels_true.astype(int))
tta_accuracy = np.mean(correct_predictions)

print(f"\nSubset Accuracy (Exact Match) on Test Set (with TTA): {np.mean(np.all(correct_predictions, axis=1)) * 100:.2f}%")
print(f"Binary Accuracy (Per-Class Average) on Test Set (with TTA): {tta_accuracy * 100:.2f}%")
  1. Metrik Evaluasi
In [ ]:
print("\nClassification Report (Multi-Label):\n")
print(classification_report(test_labels_true, test_preds_binary, target_names=class_names))

mcm = multilabel_confusion_matrix(test_labels_true, test_preds_binary)

fig, axes = plt.subplots(2, 2, figsize=(12, 10))
axes = axes.ravel()

for i, class_name in enumerate(class_names):
    sns.heatmap(mcm[i], annot=True, fmt='d', cmap='Blues', ax=axes[i], cbar=False,
                xticklabels=['Negative', 'Positive'], yticklabels=['Negative', 'Positive'])
    axes[i].set_title(f'Confusion Matrix: {class_name}')
    axes[i].set_xlabel('Predicted')
    axes[i].set_ylabel('True')

plt.tight_layout()
plt.show()
Reference in New Issue View Git Blame Copy Permalink