138 lines
24 KiB
Plaintext
138 lines
24 KiB
Plaintext
|
Ë
|
|||
|
|
�++j†G ã óV — d dl Z d dlZd dlZd dlZd dlZd dlZd dlmZ d dlm Z m
|
|||
|
|
Z
|
|||
|
|
ej ej d¬« ej e« Zg d¢ZdZdZd ZdZd
|
|||
|
|
d
|
|||
|
|
d d d d d d d
|
|||
|
|
œZg d¢Zd\ ZZdZdZd„ Zd„ Z d„ Z d„ Z d„ Z d„ Z d„ Z!d„ Z"d„ Z#d„ Z$d „ Z%d „ Z&d „ Z'd „ Z(d „ Z)ed!k( �rwejU d"« e d#e« e d$e « e d%e « e « e"« e$« e%« e&« e'« e(« e)« ejU d&« e jV jY e« ræd Z- e. e j^ e« « D ]³ Z0e jV jc ee0« Z2e jV jg e2« sŒ; e4 e j^ e2« D � cg c]: } e jV jk e jV jc e2| « « sŒ9| ‘Œ< c} « Z6e-e6z
|
|||
|
|
Z-ejU d'e0› d(e6› d)�« Œµ ejU d*e-› d)�« yyyc c} w )+é N)ÚPath)ÚImageÚUnidentifiedImageErrorz'%(asctime)s [%(levelname)s] %(message)s)ÚlevelÚformat)ú
|
|||
|
|
dataset_1.zipú
|
|||
|
|
dataset_2.zipú
|
|||
|
|
dataset_3.zipÚdatasetzndisan/corn-leaf-diseasez1smaranjitghose/corn-or-maize-leaf-disease-datasetú
|
|||
|
|
Hawar Daunú Bercak Daunú
|
|||
|
|
Karat Daun)u 大斑病u å°�æ–‘ç—…u è¤�æ–‘ç—…u 弯å¢éœ‰å�¶æ–‘ç—…u 圆斑病u ç�°æ–‘ç—…u å�—方锈病u 普通锈病)z CBS28.jpgzCorn_Common_Rust (1275).jpgzCorn_Common_Rust (1289).jpgzCorn_Common_Rust (1295).jpgzCorn_Gray_Spot (1).jpg)é r g @i c óh — ddl } t j dt › d�« | j t « }t j d|› �« |S # t
|
|||
|
|
$ r t j
|
|||
|
|
d« Y yt $ r7}t j
|
|||
|
|
d|› �« t j
|
|||
|
|
d« Y d}~yd}~ww xY w) z:Download dataset_1 (ndisan/corn-leaf-disease) from Kaggle.r Nz% Downloading dataset_1 from Kaggle: ú...z [OK] dataset_1 downloaded to úE [SKIP] kagglehub not installed. Install with: pip install kagglehubz$ [FAIL] dataset_1 download failed: zX Please download manually from https://www.kaggle.com/datasets/ndisan/corn-leaf-disease)Ú kagglehubÚlogÚinfoÚ
|
|||
|
|
KAGGLE_DS1Údataset_downloadÚ ImportErrorÚwarningÚ Exception©r ÚpathÚes ú9/home/asephs/ZeaVis-Edu/Machine_Learning/preprocessing.pyÚdownload_dataset_1r 6 s• € ð ÛÜ �‰Ð8¼¸ ÀCÐHÔIØ×)Ñ)¬*Ó5ˆÜ �‰Ð2°4°&Ð9Ô:؈ øÜ ò Ü � ‰ Ð[Ô\ÙÜ ò Ü � ‰ Ð:¸1¸#Ð>Ô?Ü � ‰ ÐnÔoÜûðúó ‚AA Á B1Á2B1Á:-B,Â,B1c óh — ddl } t j dt › d�« | j t « }t j d|› �« |S # t
|
|||
|
|
$ r t j
|
|||
|
|
d« Y yt $ r7}t j
|
|||
|
|
d|› �« t j
|
|||
|
|
d« Y d}~yd}~ww xY w) zSDownload dataset_2 (smaranjitghose/corn-or-maize-leaf-disease-dataset) from Kaggle.r Nz% Downloading dataset_2 from Kaggle: r z [OK] dataset_2 downloaded to r z$ [FAIL] dataset_2 download failed: zq Please download manually from https://www.kaggle.com/datasets/smaranjitghose/corn-or-maize-leaf-disease-dataset)r r r Ú
|
|||
|
|
KAGGLE_DS2r r r r r s r Údownload_dataset_2r# G s› € ð ÛÜ �‰Ð8¼¸ ÀCÐHÔIØ×)Ñ)¬*Ó5ˆÜ �‰Ð2°4°&Ð9Ô:؈ øÜ ò Ü � ‰ Ð[Ô\ÙÜ ò Ü � ‰ Ð:¸1¸#Ð>Ô?Ü � ‰ ð Hô IÜûðúr c ó — d} ddl }|j | « }t j d|› �« |S # t $ r Y nw xY wt j d« t j d« t j d« y)z¹Download dataset_3 (SciDB China Agricultural Dataset).
|
|||
|
|
|
|||
|
|
Attempt: Kaggle alternative host, then Google Drive, then warn.
|
|||
|
|
If all fail, user must download manually from SciDB.
|
|||
|
|
z0disease-identification/corn-leaf-disease-chineser Nz2 [OK] dataset_3 downloaded from Kaggle mirror to z9 [SKIP] dataset_3 could not be downloaded automatically.z Please download manually from:zM https://www.scidb.cn/en/detail?dataSetId=19536c73f6d74946a212719a94f53ab3)r r r r r r )Ú
|
|||
|
|
KAGGLE_DS3r r s r Údownload_dataset_3r& X sy € ð D€Jð
|
|||
|
|
ÛØ×)Ñ)¨*Ó5ˆÜ �‰ÐEÀdÀVÐLÔM؈ øÜ ò
|
|||
|
|
Ù ð
|
|||
|
|
úô ‡K�KÐKÔL܇K�KÐ2Ô3܇K�KÐ_Ô`Ø s „.3 ³ ?¾?c ó0 — | � t j j | « sy t j |j dd« d| « t j d| › d|› �« y# t $ r%}t j d | › d
|
|||
|
|
|› �« Y d}~yd}~ww xY w) zLCopy kagglehub download into a local ZIP so extraction step works unchanged.NFz.zipÚ Úzipz [OK] Packed ú -> Tz [FAIL] Could not pack ú: )
|
|||
|
|
Úosr ÚexistsÚshutilÚ make_archiveÚreplacer r r r )Úsrc_pathÚtarget_zip_namer s r Úcopy_kaggle_downloadr3 n sˆ € àÐœrŸw™wŸ~™~¨hÔ 7ØðÜ×јO× 3Ñ 3°F¸BÓ ?ÀÈÔQÜ �‰�> ( ¨4°Ð/@ÐAÔBØøÜ ò Ü � ‰ Ð.¨x¨j¸¸1¸#Ð>Ô?Üûðús ¤AA' Á' BÁ0BÂBc óÒ — t j j | « rt j d| › d�« yt j d| › d�« |« }|€yt || « S )z3Attempt auto-download if ZIP doesn't exist locally.z
|
|||
|
|
[CACHE] z# already exists, skipping download.Tz--- Downloading z ---F)r, r r- r r r3 )Úzip_nameÚ download_fnr s r Údownload_datasetr7 | s] € ä ‡w�w‡~�~�hÔ Ü �‰�:˜h˜ZÐ'JÐKÔLØ܇H�HÐ ˜z¨Ð
|
|||
|
|
.Ô/Ù ‹=€DØ €|Øä hÓ /Ð/ó c ó> — t j d« t D ]Á } t j j | « r‡t j j
|
|||
|
|
| « d }t j |d¬« t j | d« 5 }|j |« d d d « t j d| › d|› d�« Œ©t j d | › d �« ŒÃ y # 1 sw Y ŒBxY w# t j $ r t j d | › d
|
|||
|
|
�« Y Œÿw xY w)
|
|||
|
|
Nz%--- TAHAP 1: Mengekstrak File ZIP ---r T©Úexist_okÚrú [OK] r* ú/z
|
|||
|
|
[ERROR] z rusak.z [SKIP] File z0 tidak ditemukan. Beberapa kelas mungkin kosong.)r r Ú
|
|||
|
|
DAFTAR_ZIPr, r r- ÚsplitextÚmakedirsÚzipfileÚZipFileÚ
|
|||
|
|
extractallÚ
|
|||
|
|
BadZipFileÚerrorr )Úzip_fileÚ folder_nameÚzip_refs r Úekstrak_semua_ziprJ Œ sí € ܇H�HÐ
|
|||
|
|
4Ô5Ü ò eˆÜ
|
|||
|
|
�7‰7�>‰>˜(Ô #Ü Ÿ'™'×*Ñ*¨8Ó4°QÑ7ˆKÜ �K‰K˜ ¨dÕ 3ð
|
|||
|
|
:Ü —_‘_ X¨sÓ3ð 4°wØ×&Ñ& {Ô3÷4ä—‘˜7 8 *¨D°°
|
|||
|
|
¸QÐ?Õ@ô
|
|||
|
|
�K‰K˜.¨¨
|
|||
|
|
Ð2bÐcÕ dñ e÷
|
|||
|
|
4ð 4ûô ×%Ñ%ò
|
|||
|
|
:Ü— ‘ ˜J x j°Ð8Ö9ð
|
|||
|
|
:ús* Á8C-ÂC! $C-Ã!C* Ã&C-Ã-,D ÄD c ó ‡— t j j | « sy t j | « D ]H }|j « Št ˆfd„|D « « sŒ(t j j
|
|||
|
|
| |« c S y )Nc 3 ó& •K — | ] }|‰v –— Œ
|
|||
|
|
y w)N© )Ú.0ÚkÚf_lowers €r ú <genexpr>z"cari_folder_ds2.<locals>.<genexpr>¤ s øè ø€ Ò. ˆq�GŒ|Ñ.ùs ƒ)r, r r- ÚlistdirÚlowerÚanyÚjoin)Ú base_pathÚkeywordsÚfrP s @r Úcari_folder_ds2rY Ÿ sa ø€ Ü
|
|||
|
|
�7‰7�>‰>˜)Ô $ØÜ
|
|||
|
|
�Z‰Z˜ Ó
|
|||
|
|
"ò .ˆØ—'‘'“)ˆÜ Ó. XÔ.Õ .Ü—7‘7—<‘< ¨1Ó-Ò -ð.ð r8 c ó — t j d« t j t d¬« g d¢} | D ]™ }t j
|
|||
|
|
j
|
|||
|
|
d|« }t j
|
|||
|
|
j
|
|||
|
|
t |« }t j
|
|||
|
|
j |« sŒgt j ||d¬« t j d|› d|› �« Œ› t j
|
|||
|
|
j d « �r{d }t j
|
|||
|
|
j
|
|||
|
|
|d
|
|||
|
|
« }t j
|
|||
|
|
j |« r|}d d d
|
|||
|
|
œ}|j « D �] \ }}t ||« } t j
|
|||
|
|
j
|
|||
|
|
t |« }
|
|||
|
|
t j |
|
|||
|
|
d¬« | r¶t j
|
|||
|
|
j | « r—d} t j | « D ]] } t j
|
|||
|
|
j
|
|||
|
|
| | « }
|
|||
|
|
t j
|
|||
|
|
j |
|
|||
|
|
« sŒCt j |
|
|||
|
|
|
|
|||
|
|
« | dz
|
|||
|
|
} Œ_ t j d| › d| › d|
|
|||
|
|
› �« �Œt j d|› d�« �Œ! y t j d« y )Nz,--- TAHAP 2: Menggabungkan Dataset 1 & 2 ---Tr: )r
|
|||
|
|
ú
|
|||
|
|
Daun Sehatr Ú dataset_1)Ú
|
|||
|
|
dirs_exist_okz [OK] Menyalin folder z ke Ú dataset_2Údatar r[ ))Ú common_rustÚ commont_rust)Úhealthyr é z [OK] Menyalin z
|
|||
|
|
gambar dari z [SKIP] Folder untuk 'z ' tidak ditemukan di dataset_2z? [SKIP] Folder dataset_2/ tidak ada, dataset_2 tidak diproses.)r r r, rA Ú
|
|||
|
|
TARGET_DIRr rU r- r. ÚcopytreeÚitemsrY rR ÚisfileÚcopyr )Úfolder_dari_ds1ÚfolderÚsrcÚdstÚbase_ds2Údata_subÚ mapping_ds2rW Útarget_subfolderÚ
|
|||
|
|
src_folderÚ
|
|||
|
|
dst_folderÚ
|
|||
|
|
file_countÚ file_nameÚfull_file_names r Úgabungkan_dataset_1_dan_2rv © sí € ܇H�HÐ
|
|||
|
|
;Ô<܇K�K”
|
|||
|
|
TÕ*ò B€OØ!ò ?ˆÜ�g‰g�l‰l˜;¨Ó/ˆÜ�g‰g�l‰lœ: vÓ.ˆÜ
|
|||
|
|
�7‰7�>‰>˜#Õ Ü �O‰O˜C °DÕ 9Ü �H‰HÐ.¨s¨e°4¸°uÐ=Õ >ð ?ô
|
|||
|
|
‡w�w‡~�~�kÕ"à ˆä—7‘7—<‘< ¨&Ó1ˆÜ
|
|||
|
|
�7‰7�>‰>˜(Ô #Ø ˆHð .:Ø&ñ
|
|||
|
|
ˆ ð
|
|||
|
|
+6×*;Ñ*;Ó*=ó hÑ &ˆHÐ&Ü(¨°8Ó<ˆJÜŸ™Ÿ ™ ¤jÐ2BÓCˆJÜ �K‰K˜
|
|||
|
|
¨TÕ 2áœbŸg™gŸn™n¨ZÔ 8Ø �
|
|||
|
|
Ü!#§¡¨JÓ!7ò (�IÜ%'§W¡W§\¡\°*¸iÓ%H�NÜ—w‘w—~‘~ nÕ5Ü Ÿ ™ N°JÔ?Ø" a™™
|
|||
|
|
ð (ô
|
|||
|
|
—‘Ð+¨J¨<°}ÀZÀLÐPTÐU_ÐT`ÐaÖbä— ‘ Ð 5Ð6FÐ5GÐGeÐ fÖgñ hô � ‰ ÐUÕVr8 c ó
|
|||
|
|
— |j « }t j j | |« }t j j |« r|S |j « }t j | « D ]Š }|j « |k( sTt j j |« d j « t j j |« d k( sŒjt j j | |« c S y )Nr )Ústripr, r rU r- rS rR r@ )Ú
|
|||
|
|
folder_sumberÚnama_file_targetÚ
|
|||
|
|
path_langsungÚ target_lowerrX s r Úcari_gambar_fleksibelr} Ú sÄ € Ø'×-Ñ-Ó/ÐÜ—G‘G—L‘L Ð0@ÓA€MÜ ‡w�w‡~�~�mÔ$Ø Ð à#×)Ñ)Ó+€LÜ
|
|||
|
|
�Z‰Z˜
|
|||
|
|
Ó
|
|||
|
|
&ò 2ˆØ
|
|||
|
|
�G‰G‹I˜ Ò %Ü—7‘7×#Ñ# AÓ& qÑ)×/Ñ/Ó1´R·W±W×5EÑ5EÀlÓ5SÐTUÑ5VÓVÜ—7‘7—<‘<
|
|||
|
|
¨qÓ1Ò 1ð2ð r8 c ó´ — t j d« t j j dd« } t j j dd« }t j j |« st j
|
|||
|
|
d|› d�« y t |dd¬ « 5 }t j |« }d d d « d
|
|||
|
|
}D ]ê }|j d « }|j d d
|
|||
|
|
« j « }|sŒ7|t v sŒ@t | }t j j t |« } t j | d¬« t | |« }
|
|||
|
|
|
|
|||
|
|
sŒ“t j j! |
|
|||
|
|
« } t# j$ |
|
|||
|
|
t j j | | « « |dz
|
|||
|
|
}Œì t j d|› dt › d�« y # 1 sw Y �Œ xY w)Nz9--- TAHAP 3: Menggabungkan Dataset 3 berdasarkan JSON ---Ú dataset_3r_ z desc.jsonz [SKIP] z% tidak ditemukan. Dataset 3 dilewati.r< zutf-8)Úencodingr Ú
|
|||
|
|
image_nameÚlabelr( Tr: rc z [OK] Berhasil merutekan z gambar dari dataset_3 ke 'ú')r r r, r rU r- r ÚopenÚjsonÚloadÚgetrx ÚPEMETAAN_KATEGORIrd rA r} Úbasenamer. rh ) Ú folder_dataÚ file_jsonrX Ú
|
|||
|
|
data_labelÚberhasilÚitemr� r‚ Únama_folder_targetÚ
|
|||
|
|
folder_tujuanÚ path_sumberÚ nama_aslis r Úgabungkan_dataset_3r“ è so € ܇H�HÐ
|
|||
|
|
HÔIÜ—'‘'—,‘,˜{¨FÓ3€KÜ—‘— ‘ ˜[¨+Ó6€Iä
|
|||
|
|
�7‰7�>‰>˜)Ô $Ü � ‰ �i ˜{Ð*OÐPÔQØä
|
|||
|
|
ˆi˜ wÔ /ð "°1Ü—Y‘Y˜q“\ˆ
|
|||
|
|
÷"ð €HØò
|
|||
|
|
ˆØ—X‘X˜lÓ+ˆ
|
|||
|
|
Ø—‘˜ "Ó%×+Ñ+Ó-ˆâ ˜%Ô#4Ò4Ü!2°5Ñ!9Ð Ü ŸG™GŸL™L¬Ð5GÓ HˆMÜ �K‰K˜
|
|||
|
|
°Õ 5ä/° ¸ZÓHˆKÚÜ ŸG™G× ,Ñ ,¨[Ó 9� Ü— ‘ ˜K¬¯©¯ © °mÀYÓ)OÔPؘA‘
|
|||
|
|
‘ð
|
|||
|
|
ô ‡H�HÐ)¨(¨Ð3NÌzÈlÐZ[Ð
|
|||
|
|
\Õ]÷'"ñ "ús  G
|
|||
|
|
Ç
|
|||
|
|
Gc
|
|||
|
|
óÖ — t j d« t t « } d}t j
|
|||
|
|
j
|
|||
|
|
t « r’t j t « D ]v \ }}}|D ]k }|| v sŒt j
|
|||
|
|
j ||« } t j |« t j d|› �« | j |« |dz
|
|||
|
|
}Œm Œx t j d|› �« | rBt j d t | « › d
|
|||
|
|
�« | D ] }t j d |› �« Œ y y # t $ r&}t j d|› d|› d�« Y d }~Œ÷d }~ww xY w) Nz*--- TAHAP 4: Menghapus File Bermasalah ---r z
|