zeavis-edu/docs/superpowers/plans/2026-05-22-backend-image-classification.md

Backend Image Classification Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add real backend image classification using the exported TensorFlow.js model and external uploader service.

Architecture: The web app sends an image to the Elysia API. The API uploads the original image to https://upload.asepharyana.tech/api/upload, runs the local TFJS graph model from Machine_Learning/model/tfjs_model/model.json, maps probabilities to ZeaVis Edu labels, stores the result in PostgreSQL, and returns disease education metadata. Shared types define the API contract for uploader metadata, prediction probabilities, and image classification history.

Tech Stack: Bun workspaces, TypeScript, Elysia, Drizzle ORM, PostgreSQL, TensorFlow.js graph model, React, TanStack Query, Tailwind CSS.

---

File structure

• Modify apps/api/package.json: add runtime dependencies for TensorFlow.js/image decoding if compatible with Bun.
• Modify packages/shared/src/classifications.ts: add image classification, probability, and uploader metadata types.
• Modify packages/shared/src/index.ts: export new image classification types.
• Modify apps/api/src/db/schema.ts: add imageClassifications table.
• Create/modify Drizzle migration under apps/api/drizzle/: create image_classifications table.
• Create apps/api/src/lib/disease-mappers.ts: map DB disease rows to shared disease records.
• Create apps/api/src/lib/uploader-client.ts: upload image files to external uploader.
• Create apps/api/src/lib/image-model.ts: load/cached TFJS graph model and classify image buffers.
• Modify apps/api/src/routes/classifications.ts: add image classification POST/GET routes.
• Modify apps/api/src/routes/dashboard.ts: optionally include image latest/count only if shared summary is extended.
• Modify apps/web/src/lib/api-client.ts: add image classification API methods.
• Create apps/web/src/components/image-classification-form.tsx: upload form and latest result display.
• Modify apps/web/src/pages/dashboard-page.tsx: show image classification form and image prediction history.

---

Task 1: Shared image classification contract

Files:

• Modify: packages/shared/src/classifications.ts

• Modify: packages/shared/src/index.ts

• Step 1: Extend shared classification types

Modify packages/shared/src/classifications.ts to:

import type { DiseaseCatalogItem, DiseaseSlug, RiskLevel } from './diseases';

export type ManualClassificationRequest = {
  diseaseSlug: DiseaseSlug;
  observation: string;
  location: string;
};

export type ManualClassificationRecord = {
  id: string;
  diseaseSlug: DiseaseSlug;
  observation: string;
  location: string;
  createdAt: string;
  disease: DiseaseCatalogItem;
};

export type PredictionProbability = {
  diseaseSlug: DiseaseSlug;
  label: DiseaseCatalogItem['label'];
  confidence: number;
};

export type UploaderMetadata = {
  public_id: string;
  file_name: string;
  mime_type: string;
  size_bytes: number;
  file_type: string;
  uploader_id?: number;
  created_at: string;
  telegram_file_id?: string;
  telegram_file_unique_id?: string;
  storage_chat_id?: number;
  storage_message_id?: number;
  download_url: string;
};

export type ImageClassificationRecord = {
  id: string;
  predictedDiseaseSlug: DiseaseSlug;
  confidence: number;
  probabilities: PredictionProbability[];
  imageUrl: string;
  originalFileName: string;
  uploaderPublicId: string;
  uploader: UploaderMetadata;
  createdAt: string;
  disease: DiseaseCatalogItem;
};

export type DashboardSummary = {
  diseaseCount: number;
  classificationCount: number;
  latestClassification: ManualClassificationRecord | null;
  riskDistribution: Record<RiskLevel, number>;
};

• Step 2: Export new types

Modify packages/shared/src/index.ts classification exports to include:

export type {
  DashboardSummary,
  ImageClassificationRecord,
  ManualClassificationRecord,
  ManualClassificationRequest,
  PredictionProbability,
  UploaderMetadata,
} from './classifications';

• Step 3: Verify shared typecheck

Run: bun run --cwd packages/shared typecheck

Expected: TypeScript exits with code 0.

---

Task 2: Backend dependencies, DB schema, and migrations

Files:

• Modify: apps/api/package.json

• Modify: apps/api/src/db/schema.ts

• Create/modify: apps/api/drizzle/*.sql

• Step 1: Add backend inference dependencies

Run: bun add --cwd apps/api @tensorflow/tfjs jpeg-js pngjs

Expected: apps/api/package.json and bun.lockb/lockfile update. Use pure @tensorflow/tfjs first because it avoids native Node bindings and is more likely to run under Bun.

• Step 2: Add image classification table to schema

Modify apps/api/src/db/schema.ts to import jsonb and real, then add:

export const imageClassifications = pgTable('image_classifications', {
  id: uuid('id').primaryKey().defaultRandom(),
  predictedDiseaseSlug: varchar('predicted_disease_slug', { length: 80 })
    .notNull()
    .references(() => diseaseCatalog.slug),
  confidence: real('confidence').notNull(),
  probabilities: jsonb('probabilities').notNull(),
  imageUrl: text('image_url').notNull(),
  originalFileName: varchar('original_file_name', { length: 240 }).notNull(),
  uploaderPublicId: varchar('uploader_public_id', { length: 160 }).notNull(),
  uploaderPayload: jsonb('uploader_payload').notNull(),
  createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
});

• Step 3: Generate migration

Run: bun run --cwd apps/api db:generate

Expected: a new SQL migration creates image_classifications with FK to disease_catalog.

• Step 4: Verify API typecheck

Run: bun run --cwd apps/api typecheck

Expected: TypeScript exits with code 0.

---

Task 3: Backend mapper, uploader client, and TFJS model service

Files:

• Create: apps/api/src/lib/disease-mappers.ts

• Create: apps/api/src/lib/uploader-client.ts

• Create: apps/api/src/lib/image-model.ts

• Modify: apps/api/src/lib/http-errors.ts

• Step 1: Add bad gateway helper

Modify apps/api/src/lib/http-errors.ts to add:

export function badGateway(message: string) {
  return new Response(JSON.stringify({ error: message }), {
    status: 502,
    headers: { 'content-type': 'application/json' },
  });
}

• Step 2: Add disease row mapper

Write apps/api/src/lib/disease-mappers.ts:

import type { DiseaseCatalogItem, DiseaseLabel, DiseaseSlug, RiskLevel } from '@zeavis/shared';
import { diseaseCatalog } from '../db/schema';

export function toDisease(row: typeof diseaseCatalog.$inferSelect): DiseaseCatalogItem {
  return {
    slug: row.slug as DiseaseSlug,
    label: row.label as DiseaseLabel,
    commonName: row.commonName,
    summary: row.summary,
    description: row.description,
    symptoms: row.symptoms,
    recommendations: row.recommendations,
    riskLevel: row.riskLevel as RiskLevel,
    accentColor: row.accentColor,
    displayOrder: row.displayOrder,
  };
}

• Step 3: Add uploader client

Write apps/api/src/lib/uploader-client.ts:

import type { UploaderMetadata } from '@zeavis/shared';

const uploaderUrl = 'https://upload.asepharyana.tech/api/upload';

export async function uploadImageToStorage(file: File): Promise<UploaderMetadata> {
  const formData = new FormData();
  formData.append('file', file, file.name);
  formData.append('fileName', file.name);

  const response = await fetch(uploaderUrl, {
    method: 'POST',
    body: formData,
  });

  const payload = (await response.json().catch(() => null)) as Partial<UploaderMetadata> & { error?: string } | null;
  if (!response.ok || !payload?.download_url || !payload.public_id) {
    throw new Error(payload?.error ?? 'Upload service failed');
  }

  return payload as UploaderMetadata;
}

• Step 4: Add TFJS image model service

Write apps/api/src/lib/image-model.ts:

import type { DiseaseLabel, DiseaseSlug, PredictionProbability } from '@zeavis/shared';
import * as tf from '@tensorflow/tfjs';
import jpeg from 'jpeg-js';
import { PNG } from 'pngjs';

const labels: Array<{ diseaseSlug: DiseaseSlug; label: DiseaseLabel }> = [
  { diseaseSlug: 'bercak-daun', label: 'Bercak Daun' },
  { diseaseSlug: 'hawar-daun', label: 'Hawar Daun' },
  { diseaseSlug: 'karat-daun', label: 'Karat Daun' },
  { diseaseSlug: 'daun-sehat', label: 'Daun Sehat' },
];

let modelPromise: Promise<tf.GraphModel> | undefined;

function modelUrl() {
  return `file://${process.cwd()}/../../Machine_Learning/model/tfjs_model/model.json`;
}

async function loadModel() {
  modelPromise ??= tf.loadGraphModel(modelUrl());
  return modelPromise;
}

function decodeImage(buffer: ArrayBuffer, mimeType: string) {
  const bytes = new Uint8Array(buffer);
  if (mimeType === 'image/png') {
    const png = PNG.sync.read(Buffer.from(bytes));
    return { width: png.width, height: png.height, data: png.data };
  }

  const jpegImage = jpeg.decode(Buffer.from(bytes), { useTArray: true });
  return { width: jpegImage.width, height: jpegImage.height, data: jpegImage.data };
}

function imageToTensor(image: { width: number; height: number; data: Uint8Array | Buffer }) {
  const rgb = new Uint8Array(image.width * image.height * 3);
  for (let source = 0, target = 0; source < image.data.length; source += 4, target += 3) {
    rgb[target] = image.data[source];
    rgb[target + 1] = image.data[source + 1];
    rgb[target + 2] = image.data[source + 2];
  }

  return tf.tidy(() =>
    tf.tensor3d(rgb, [image.height, image.width, 3], 'int32')
      .resizeBilinear([224, 224])
      .toFloat()
      .div(255)
      .expandDims(0),
  );
}

export async function classifyImage(file: File) {
  if (!['image/jpeg', 'image/png'].includes(file.type)) {
    throw new Error('Unsupported image type');
  }

  const model = await loadModel();
  const decoded = decodeImage(await file.arrayBuffer(), file.type);
  const input = imageToTensor(decoded);

  try {
    const output = model.predict(input) as tf.Tensor;
    const scores = Array.from(await output.data());
    output.dispose();

    const probabilities: PredictionProbability[] = scores.map((confidence, index) => ({
      ...labels[index],
      confidence,
    }));
    probabilities.sort((a, b) => b.confidence - a.confidence);

    return {
      predictedDiseaseSlug: probabilities[0].diseaseSlug,
      confidence: probabilities[0].confidence,
      probabilities,
    };
  } finally {
    input.dispose();
  }
}

• Step 5: Verify API typecheck

Run: bun run --cwd apps/api typecheck

Expected: TypeScript exits with code 0. If imports for jpeg-js or pngjs lack types, add minimal .d.ts declarations in apps/api/src/types/image-decoders.d.ts and include them through tsconfig include.

---

Task 4: Backend image classification routes

Files:

• Modify: apps/api/src/routes/classifications.ts

• Step 1: Add image row conversion helpers

In apps/api/src/routes/classifications.ts, import ImageClassificationRecord, PredictionProbability, UploaderMetadata, imageClassifications, badGateway, toDisease, classifyImage, and uploadImageToStorage. Add a helper:

function toImageRecord(row: {
  id: string;
  predictedDiseaseSlug: string;
  confidence: number;
  probabilities: unknown;
  imageUrl: string;
  originalFileName: string;
  uploaderPublicId: string;
  uploaderPayload: unknown;
  createdAt: Date;
  disease: typeof diseaseCatalog.$inferSelect;
}): ImageClassificationRecord {
  return {
    id: row.id,
    predictedDiseaseSlug: row.predictedDiseaseSlug as ImageClassificationRecord['predictedDiseaseSlug'],
    confidence: row.confidence,
    probabilities: row.probabilities as PredictionProbability[],
    imageUrl: row.imageUrl,
    originalFileName: row.originalFileName,
    uploaderPublicId: row.uploaderPublicId,
    uploader: row.uploaderPayload as UploaderMetadata,
    createdAt: row.createdAt.toISOString(),
    disease: toDisease(row.disease),
  };
}

• Step 2: Add GET image history route

Add to classificationRoutes before the manual POST route:

  .get('/classifications/image', async () => {
    try {
      const db = createDbClient();
      const rows = await db
        .select({
          id: imageClassifications.id,
          predictedDiseaseSlug: imageClassifications.predictedDiseaseSlug,
          confidence: imageClassifications.confidence,
          probabilities: imageClassifications.probabilities,
          imageUrl: imageClassifications.imageUrl,
          originalFileName: imageClassifications.originalFileName,
          uploaderPublicId: imageClassifications.uploaderPublicId,
          uploaderPayload: imageClassifications.uploaderPayload,
          createdAt: imageClassifications.createdAt,
          disease: diseaseCatalog,
        })
        .from(imageClassifications)
        .innerJoin(diseaseCatalog, eq(imageClassifications.predictedDiseaseSlug, diseaseCatalog.slug))
        .orderBy(desc(imageClassifications.createdAt))
        .limit(20);

      return rows.map(toImageRecord);
    } catch {
      return serviceUnavailable('Database unavailable');
    }
  })

• Step 3: Add POST image classification route

Add to classificationRoutes:

  .post('/classifications/image', async ({ body }) => {
    const payload = body as { file?: File } | undefined;
    const file = payload?.file;

    if (!(file instanceof File) || file.size === 0) {
      return badRequest('Image file is required');
    }

    if (!file.type.startsWith('image/')) {
      return badRequest('Only image files are supported');
    }

    let uploader: UploaderMetadata;
    try {
      uploader = await uploadImageToStorage(file);
    } catch {
      return badGateway('Image uploader is unavailable');
    }

    let prediction: Awaited<ReturnType<typeof classifyImage>>;
    try {
      prediction = await classifyImage(file);
    } catch {
      return serviceUnavailable('Image classification model is unavailable');
    }

    try {
      const db = createDbClient();
      const [disease] = await db
        .select()
        .from(diseaseCatalog)
        .where(eq(diseaseCatalog.slug, prediction.predictedDiseaseSlug))
        .limit(1);

      if (!disease) {
        return serviceUnavailable('Predicted disease is not available in the catalog');
      }

      const [created] = await db
        .insert(imageClassifications)
        .values({
          predictedDiseaseSlug: prediction.predictedDiseaseSlug,
          confidence: prediction.confidence,
          probabilities: prediction.probabilities,
          imageUrl: uploader.download_url,
          originalFileName: file.name || uploader.file_name,
          uploaderPublicId: uploader.public_id,
          uploaderPayload: uploader,
        })
        .returning();

      return toImageRecord({ ...created, disease });
    } catch {
      return serviceUnavailable('Database unavailable');
    }
  })

• Step 4: Verify API typecheck

Run: bun run --cwd apps/api typecheck

Expected: TypeScript exits with code 0.

---

Task 5: Web client, upload form, and dashboard integration

Files:

• Modify: apps/web/src/lib/api-client.ts

• Create: apps/web/src/components/image-classification-form.tsx

• Modify: apps/web/src/pages/dashboard-page.tsx

• Step 1: Extend API client

Modify apps/web/src/lib/api-client.ts imports to include ImageClassificationRecord. Add methods:

  getImageClassifications: () => request<ImageClassificationRecord[]>('/api/v1/classifications/image'),
  createImageClassification: (file: File) => {
    const body = new FormData();
    body.append('file', file, file.name);
    return request<ImageClassificationRecord>('/api/v1/classifications/image', { method: 'POST', body, headers: {} });
  },

Adjust request so it only sets content-type: application/json when init?.body is not a FormData.

• Step 2: Add image classification form

Write apps/web/src/components/image-classification-form.tsx:

import type { ImageClassificationRecord } from '@zeavis/shared';
import { useState } from 'react';
import { Button } from '@/components/ui/button';
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
import { RiskBadge } from '@/components/risk-badge';

export function ImageClassificationForm({ onSubmit, isSubmitting, latestResult }: { onSubmit: (file: File) => Promise<void>; isSubmitting: boolean; latestResult: ImageClassificationRecord | null }) {
  const [file, setFile] = useState<File | null>(null);
  const [error, setError] = useState<string | null>(null);

  return (
    <Card>
      <CardHeader>
        <CardTitle>Klasifikasi Gambar</CardTitle>
        <CardDescription>Unggah foto daun jagung untuk prediksi model ZeaVis Edu.</CardDescription>
      </CardHeader>
      <CardContent className="space-y-4">
        <form
          className="space-y-4"
          onSubmit={async (event) => {
            event.preventDefault();
            setError(null);
            if (!file) {
              setError('Pilih gambar daun terlebih dahulu');
              return;
            }
            await onSubmit(file);
            setFile(null);
          }}
        >
          {error && <div className="rounded-md bg-red-50 p-3 text-sm text-red-800">{error}</div>}
          <input
            accept="image/*"
            type="file"
            onChange={(event) => setFile(event.target.files?.[0] ?? null)}
            className="block w-full rounded-md border border-border bg-background px-3 py-2 text-sm"
          />
          <Button disabled={!file || isSubmitting} type="submit">
            {isSubmitting ? 'Mengklasifikasi...' : 'Klasifikasi gambar'}
          </Button>
        </form>

        {latestResult && (
          <div className="rounded-2xl border p-4">
            <img src={latestResult.imageUrl} alt={latestResult.originalFileName} className="mb-4 max-h-64 w-full rounded-xl object-cover" />
            <div className="flex items-start justify-between gap-4">
              <div>
                <p className="text-sm text-muted-foreground">Hasil prediksi</p>
                <h3 className="text-xl font-semibold">{latestResult.disease.label}</h3>
                <p className="text-sm text-muted-foreground">Confidence {(latestResult.confidence * 100).toFixed(1)}%</p>
              </div>
              <RiskBadge level={latestResult.disease.riskLevel} />
            </div>
            <ul className="mt-4 space-y-2 text-sm text-muted-foreground">
              {latestResult.disease.recommendations.slice(0, 3).map((recommendation) => (
                <li key={recommendation}>• {recommendation}</li>
              ))}
            </ul>
          </div>
        )}
      </CardContent>
    </Card>
  );
}

• Step 3: Wire dashboard image queries

Modify apps/web/src/pages/dashboard-page.tsx:

• Add ImageClassificationForm import.

• Add a fourth useQueries entry with queryKey: ['image-classifications'] and queryFn: () => apiClient.getImageClassifications().

• Add const imageClassifications = imageClassificationsQuery.data || [];.

• Add useMutation for apiClient.createImageClassification(file) and invalidate image-classifications on success.

• Render ImageClassificationForm above ManualClassificationForm with latestResult={imageClassifications[0] ?? null}.

• Add a history Card for image classifications showing thumbnail, label, confidence, and date.

• Step 4: Verify web typecheck

Run: bun run --cwd apps/web typecheck

Expected: TypeScript exits with code 0.

---

Task 6: Full verification and manual behavior check

Files:

• Modify only if verification reveals a concrete bug.

• Step 1: Run full typecheck and build

Run: bun run typecheck && bun run build

Expected: all workspace typecheck and build tasks pass.

• Step 2: Start API and web

Run API: bun run --cwd apps/api start

Run web: bun run --cwd apps/web dev

Expected: API logs ZeaVis Edu API running...; Vite serves the app.

• Step 3: Check API without database

Run: curl -s -w '\n%{http_code}' http://localhost:3000/api/v1/classifications/image

Expected without DATABASE_URL: JSON { "error": "Database unavailable" } with status 503, not a crash.

• Step 4: Check upload route validation

Run: curl -s -X POST -w '\n%{http_code}' http://localhost:3000/api/v1/classifications/image

Expected: JSON error with status 400.

• Step 5: Browser check dashboard

Open /dashboard. Confirm the image classification form renders. If no database is configured, confirm structured error state renders without crashing. If database is configured and migrated, submit a JPEG/PNG corn leaf image and verify the result card displays image, label, confidence, and recommendations.

---

Self-review notes

• Spec coverage: backend TFJS inference, uploader integration, DB persistence, API routes, shared types, frontend upload/result/history, and verification are covered.
• Placeholder scan: no TBD/TODO/fill-later placeholders remain; every file and route has explicit behavior.
• Type consistency: ImageClassificationRecord, PredictionProbability, and UploaderMetadata are defined once in shared and used consistently across API and web.