{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Gemma 4 E4B — Clone & Benchmark Notebook\n", "\n", "Notebook untuk meng-clone model [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) dari Hugging Face dan melakukan benchmark pada berbagai metrik:\n", "- Kecepatan loading & memory usage\n", "- Text generation throughput (tokens/sec)\n", "- Reasoning & knowledge QA\n", "- Coding capability\n", "- Multimodal understanding (image)\n", "- Long context retrieval\n", "\n", "**Model**: `google/gemma-4-E4B-it` (instruction-tuned, 4.5B effective params, 8B total, 128K context)\n", "\n", "**Cara pakai**: Runtime > Factory reset runtime, lalu Runtime > Run all" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 1. Environment Setup" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.7/60.7 MB 13.8 MB/s eta 0:00:00\n", "Install selesai\n" ] } ], "source": [ "# Install dependencies\n", "!pip install -qU \\\n", " 'transformers>=4.50.0' \\\n", " accelerate \\\n", " sentencepiece \\\n", " protobuf \\\n", " psutil \\\n", " 'pillow<11' \\\n", " requests \\\n", " matplotlib \\\n", " tabulate \\\n", " librosa \\\n", " soundfile \\\n", " einops \\\n", " bitsandbytes \\\n", " 2>&1 | tail -3\n", "print(\"Install selesai\")" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Python : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0]\n", "PyTorch : 2.11.0+cu128\n", "CUDA avail : True\n", "CUDA device : Tesla T4\n", "CUDA VRAM : 15.6 GB\n", "CUDA cap : (7, 5)\n" ] } ], "source": [ "import os, sys, json, time, gc, warnings\n", "from pathlib import Path\n", "from datetime import datetime\n", "from IPython.display import display\n", "\n", "import torch\n", "import psutil\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from tabulate import tabulate\n", "from PIL import Image\n", "import requests\n", "from io import BytesIO\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "print(f\"Python : {sys.version}\")\n", "print(f\"PyTorch : {torch.__version__}\")\n", "print(f\"CUDA avail : {torch.cuda.is_available()}\")\n", "if torch.cuda.is_available():\n", " print(f\"CUDA device : {torch.cuda.get_device_name(0)}\")\n", " print(f\"CUDA VRAM : {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB\")\n", " print(f\"CUDA cap : {torch.cuda.get_device_capability()}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 2. Clone Model from Hugging Face\n", "\n", "Model size ~16 GB dalam BF16. Karena T4 hanya 15.6GB VRAM, kita perlu 4-bit quantization." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model ID: google/gemma-4-E4B-it\n", "Loading (4-bit quantized)...\n", "Processor loaded in 6.2s\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "169a4541ec42498083b317e11576b863", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading weights: 0%| | 0/2076 [00:00" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Time: 23.8s\n", "This image displays a photograph featuring the **Golden Gate Bridge** spanning across a body of water toward a landmass in the distance.\n", "\n", "Here is a brief description:\n", "\n", "The dominant feature is the massive **red bridge structure** (clearly identifiable\n" ] } ], "source": [ "try:\n", " url = \"https://raw.githubusercontent.com/google-gemma/cookbook/main/apps/sample-data/GoldenGate.png\"\n", " img = Image.open(BytesIO(requests.get(url, timeout=30).content))\n", " display(img.resize((250, 180)))\n", " inputs = processor.apply_chat_template([{\"role\":\"user\",\"content\":[\n", " {\"type\":\"image\",\"image\":img},\n", " {\"type\":\"text\",\"text\":\"What is shown? Describe briefly.\"}\n", " ]}], tokenize=True, return_dict=True, return_tensors=\"pt\", add_generation_prompt=True).to(model.device)\n", " t0 = time.perf_counter()\n", " with torch.no_grad():\n", " out = model.generate(**inputs, max_new_tokens=128, temperature=0.7, do_sample=True)\n", " print(f\"Time: {time.perf_counter()-t0:.1f}s\")\n", " print(processor.decode(out[0][inputs[\"input_ids\"].shape[-1]:], skip_special_tokens=True)[:250])\n", "except Exception as e:\n", " print(f\"ERROR: {e}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 7. Long Context (Needle-in-Haystack)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Needle-in-Haystack:\n", "EARLY: 793 tok, correct=OK, 4.9s\n", "MIDDLE: 793 tok, correct=OK, 5.0s\n", "LATE: 793 tok, correct=OK, 5.0s\n" ] } ], "source": [ "torch.cuda.empty_cache()\n", "gc.collect()\n", "\n", "def needle(pos):\n", " needle_str = \"The secret code is BLUE-42-GREEN.\"\n", " filler = \"The quick brown fox jumps over the lazy dog. Python is versatile. \"\n", " sents = [filler] * 50 # shorter context for T4\n", " if pos == \"early\": sents.insert(0, needle_str)\n", " elif pos == \"middle\": sents.insert(len(sents)//2, needle_str)\n", " else: sents.append(needle_str)\n", " prompt = f\"Read the text and answer.\\n\\nText: {' '.join(sents)}\\n\\nQ: What is the secret code? Answer with code only.\"\n", " inputs = processor.apply_chat_template([{\"role\":\"user\",\"content\":prompt}],\n", " tokenize=True, return_dict=True, return_tensors=\"pt\", add_generation_prompt=True).to(model.device)\n", " t0 = time.perf_counter()\n", " with torch.no_grad():\n", " out = model.generate(**inputs, max_new_tokens=16, do_sample=False)\n", " resp = processor.decode(out[0][inputs[\"input_ids\"].shape[-1]:], skip_special_tokens=True).strip()\n", " return {\"pos\": pos, \"tokens\": inputs[\"input_ids\"].shape[-1], \"resp\": resp[:80],\n", " \"correct\": \"BLUE-42-GREEN\" in resp, \"time\": time.perf_counter()-t0}\n", "\n", "print(\"Needle-in-Haystack:\")\n", "for p in [\"early\", \"middle\", \"late\"]:\n", " r = needle(p)\n", " print(f\"{p.upper()}: {r['tokens']} tok, correct={'OK' if r['correct'] else 'NO'}, {r['time']:.1f}s\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 8. Summary" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "============================================================\n", " GEMMA 4 E4B — SUMMARY\n", "============================================================\n", "+------------+-----------------------+\n", "| Metric | Value |\n", "+============+=======================+\n", "| Model | google/gemma-4-E4B-it |\n", "+------------+-----------------------+\n", "| Parameters | 5.72B |\n", "+------------+-----------------------+\n", "| Device | cuda:0 |\n", "+------------+-----------------------+\n", "| VRAM | 10.26 GB |\n", "+------------+-----------------------+\n", "| RAM | 2.67 GB |\n", "+------------+-----------------------+\n", "| Load Time | 63.4s |\n", "+------------+-----------------------+\n", "| Throughput | 5.6 tok/s |\n", "+------------+-----------------------+\n", "| MMLU | 4/5 (80%) |\n", "+------------+-----------------------+\n", "============================================================\n" ] } ], "source": [ "rows = [\n", " [\"Model\", MODEL_ID],\n", " [\"Parameters\", f\"{total_params/1e9:.2f}B\"],\n", " [\"Device\", str(model.device)],\n", "]\n", "if torch.cuda.is_available(): rows.append([\"VRAM\", f\"{torch.cuda.memory_allocated()/1e9:.2f} GB\"])\n", "rows.append([\"RAM\", f\"{ram_used:.2f} GB\"])\n", "rows.append([\"Load Time\", f\"{load_time:.1f}s\"])\n", "if results: rows.append([\"Throughput\", f\"{np.mean([r['tps'] for r in results]):.1f} tok/s\"])\n", "rows.append([\"MMLU\", f\"{ok}/{len(mmlu)} ({ok/len(mmlu)*100:.0f}%)\"])\n", "\n", "print(\"=\"*60)\n", "print(\" GEMMA 4 E4B — SUMMARY\")\n", "print(\"=\"*60)\n", "print(tabulate(rows, headers=[\"Metric\",\"Value\"], tablefmt=\"grid\"))\n", "print(\"=\"*60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## 9. Cleanup" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "VRAM: 10.26 GB\n", "Done.\n" ] } ], "source": [ "del model, processor\n", "gc.collect()\n", "if torch.cuda.is_available():\n", " torch.cuda.empty_cache()\n", " print(f\"VRAM: {torch.cuda.memory_allocated()/1e9:.2f} GB\")\n", "print(\"Done.\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 4 }