Merge branch 'develop' into fork_develop

Reviewed-on: https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/105
Co-authored-by: zaphkito <zaphkito@noreply.gitea.tendokyu.moe>
Co-committed-by: zaphkito <zaphkito@noreply.gitea.tendokyu.moe>

.gitattributes

@@ -0,0 +1,3 @@
+*.csv binary
+*.txt binary
+*.json binary

.gitignore

@@ -158,5 +158,6 @@ cert/*
 !cert/server.pem
 config/*
 deliver/*
+*.gz
 
 dbdump-*.json

Dockerfile

@@ -0,0 +1,21 @@
+FROM python:3.9.15-slim-bullseye
+
+RUN apt update && apt install default-libmysqlclient-dev build-essential libtk nodejs npm pkg-config -y
+
+WORKDIR /app
+COPY requirements.txt requirements.txt
+RUN pip3 install -r requirements.txt
+RUN npm i -g nodemon
+
+COPY entrypoint.sh entrypoint.sh
+RUN chmod +x entrypoint.sh
+
+COPY index.py index.py
+COPY dbutils.py dbutils.py
+COPY read.py read.py
+ADD core core
+ADD titles titles
+ADD logs logs
+ADD cert cert
+
+ENTRYPOINT [ "/app/entrypoint.sh" ]

LICENSE.txt

@@ -0,0 +1,13 @@
+           DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+                   Version 2, December 2004
+ 
+Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>
+
+Everyone is permitted to copy and distribute verbatim or modified
+copies of this license document, and changing it is allowed as long
+as the name is changed.
+ 
+           DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+  TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+ 0. You just DO WHAT THE FUCK YOU WANT TO.

changelog.md

@@ -0,0 +1,213 @@
+# Changelog
+Documenting updates to ARTEMiS, to be updated every time the master branch is pushed to.
+
+## 20240302
+### SAO
++ Fixing new profile creation with right heroes and start VP
++ Fix to the Unanalyzed Log responses returning the wrong rewards
++ Documentation revised
+
+## 20240226
+### CXB
++ Fixing paths for rev.py
++ Changed encoding for handle_data_item_list_icon_request
+
+## 20240202
+### SAO
++ Added reader assets and edited the game specific documentation
+
+## 20240118
+### System
++ Added game version names to the readme
+
+## 20240109
+### System
++ Removed `ADD config config` from dockerfile [#83](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/83) (Thanks zaphkito!)
+
+### Aimedb
++ Fixed an error that resulted from trying to scan a banned or locked card
+
+## 20240108
+### System
++ Change how the underlying system handles URLs
+  + This can now allow for things like version-specific, or even keychip-specific URLs
+  + Specific changes to games are noted below
++ Fix docker files [#60](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/60) (Thanks Rylie!)
++ Fix support for python 3.8 - 3.10
+
+### Aimedb
++ Add support for SegaAuth key in games that support it (for now only Chunithm)
+  + This is a JWT that is sent to games, by Aimedb, that the games send to their game server, to verify that the access code the game is sending to the server was obtained via aimedb.
+  + Requires a base64-encoded secret to be set in the `core.yaml`
+
+### Chunithm
++ Fix Air support
++ Add saving for userRecentPlayerList
++ Add support for SegaAuthKey
++ Fix a bug arising if a user set their name to be 'true' or 'false'
++ Add support for Sun+ [#78](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/78) (Thanks EmmyHeart!)
++ Add `matching` section to `chuni.yaml`
++ ~~Change `udpHolePunchUri` and `reflectorUri` to be STUN and TURN servers~~ Reverted
++ Imrpove `GetGameSetting` request handling for different versions
++ Fix issue where songs would not always return all scores [#92](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/92) (Thanks Kumubou!)
+
+### maimai DX
++ Fix user charges failing to save
+
+### maimai
++ Made it functional
+
+### CXB
++ Improvements to request dispatching
++ Add support for non-omnimix music lists
+
+
+### IDZ
++ Fix news urls in accordance with the system change to URLs
+
+### Initial D THE ARCADE
++ Added support for Initial D THE ARCADE S2
+  + Story mode progress added
+  + Bunta Challenge/Touhou Project modes added
+  + Time Trials added
+  + Leaderboards added, but doesn't refresh sometimes
+  + Theory of Street mode added (with CPUs)
+  + Play Stamp/Timetrial events added
+  + Frontend to download profile added
+  + Importer to import profiles added
+
+### ONGEKI
++ Now supports HTTPS on a per-version basis
++ Merg PR [#61](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/61) (Thanks phantomlan!)
+  + Add Ranking Event Support
+  + Add reward list support
+  + Add version segregation to Event Ranking, Tech Challenge, and Music Ranking
+  + Now stores ClientTestmode and ClientSetting data
++ Fix mission points not adding correctly [#68](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/68) (Thanks phantomlan!)
++ Fix tech challenge [#70](https://gitea.tendokyu.moe/Hay1tsme/artemis/pulls/70) (Thanks phantomlan!)
+
+### SAO
++ Change endpoint in accordance with the system change to URLs
++ Update request header class to be more accurate
++ Encrypted requests are now supported
++ Change to using handler classes instead of raw structs for simplicity
+
+### Wacca
++ Fix a server error causing a seperate error that casued issues
++ Add better error printing
++ Add better request validation
++ Fix HousingStartV2
++ Fix Lily's housing/get handler
+
+## 20231107
+### CXB
++  Hotfix `render_POST` sometimes failing to read the request body on large requests
+
+## 20231106
+### CXB
++ Hotfix `render_POST` function signature signature
++ Hotfix `handle_action_addenergy_request` hard failing if `get_energy` returns None
+
+## 20231015
+### maimai DX
++ Added support for FESTiVAL PLUS
+
+### Card Maker
++ Added support for maimai DX FESTiVAL PLUS
+
+## 20230716
+### General
++ Docker files added (#19)
++ Added support for threading
+  + This comes with the caviat that enabling it will not allow you to use Ctrl + C to stop the server.
+
+### Webui
++ Small improvements
++ Add card display
+
+### Allnet
++ Billing format validation
++ Fix naomitest.html endpoint
++ Add event logging for auths and billing
++ LoaderStateRecorder endpoint handler added
+
+### Mucha
++ Fixed log level always being "Info"
++ Add stub handler for DownloadState
+
+### Sword Art Online
++ Support added
+
+### Crossbeats
++ Added threading to profile loading
+  + This should cause a noticeable speed-up
+
+### Card Maker
++ DX Passes fixed
++ Various improvements
+
+### Diva
++ Added clear status calculation
++ Various minor fixes and improvements
+
+### Maimai
++ Added support for memorial photo uploads
++ Added support for the following versions
+  + Festival
+  + FiNALE
++ Various bug fixes and improvements
+
+### Wacca
++ Fixed an error that sometimes occoured when trying to unlock songs (#22)
+
+### Pokken
++ Profile saving added (loading TBA)
++ Use external STUN server for matching by default
+  + Matching still not working
+
+## 2023042300
+### Wacca
++ Time free now works properly
++ Fix reverse gate mission causing a fatal error
++ Other misc. fixes
++ Latest DB: 5
+
+### Pokken
++ Added preliminary support
+    + Nothing saves currently, but the game will boot and function properly.
+
+### Initial D Zero
++ Added preliminary support
+    + Nothing saves currently, but the game will boot and function for the most part.
+
+### Mai2
++ Added support for Festival
++ Lasted DB Version: 4
+
+### Ongeki
++ Misc fixes
++ Lasted DB Version: 4
+
+### Diva
++ Misc fixes
++ Lasted DB Version: 4
+
+### Chuni
++ Fix network encryption
++ Add `handle_remove_token_api_request` for event mode
+
+### Allnet
++ Added download order support
+    + It is up to the sysop to provide the INI file, and host the files.
+    + ONLY for use with cabs. It's not checked currently, which it's why it's default disabled
+    + YMMV, use at your own risk
++ When running develop mode, games that are not recognised will still be able to authenticate.
+
+### Database
++ Add autoupgrade command
+    + Invoke to automatically upgrade all schemas to their latest versions
+
++ `version` arg no longer required, leave it blank to update the game schema to latest if it isn't already
+
+### Misc
++ Update example nginx config file

core/__init__.py

@@ -1,6 +1,6 @@
 from core.config import CoreConfig
-from core.allnet import AllnetServlet
+from core.allnet import AllnetServlet, BillingServlet
-from core.aimedb import AimedbFactory
+from core.aimedb import AimedbServlette
 from core.title import TitleServlet
 from core.utils import Utils
 from core.mucha import MuchaServlet

core/adb_handlers/__init__.py

@@ -0,0 +1,6 @@
+from .base import ADBBaseRequest, ADBBaseResponse, ADBHeader, ADBHeaderException, PortalRegStatus, LogStatus, ADBStatus
+from .base import CompanyCodes, ReaderFwVer, CMD_CODE_GOODBYE, HEADER_SIZE
+from .lookup import ADBLookupRequest, ADBLookupResponse, ADBLookupExResponse
+from .campaign import ADBCampaignClearRequest, ADBCampaignClearResponse, ADBCampaignResponse, ADBOldCampaignRequest, ADBOldCampaignResponse
+from .felica import ADBFelicaLookupRequest, ADBFelicaLookupResponse, ADBFelicaLookup2Request, ADBFelicaLookup2Response
+from .log import ADBLogExRequest, ADBLogRequest, ADBStatusLogRequest, ADBLogExResponse

core/adb_handlers/base.py

@@ -0,0 +1,170 @@
+import struct
+from construct import Struct, Int16ul, Int32ul, PaddedString
+from enum import Enum
+import re
+from typing import Union, Final
+
+class LogStatus(Enum):
+    NONE = 0
+    START = 1
+    CONTINUE = 2
+    END = 3
+    OTHER = 4
+
+class PortalRegStatus(Enum):
+    NO_REG = 0
+    PORTAL = 1
+    SEGA_ID = 2
+
+class ADBStatus(Enum):
+    UNKNOWN = 0
+    GOOD = 1
+    BAD_AMIE_ID = 2
+    ALREADY_REG = 3
+    BAN_SYS_USER = 4
+    BAN_SYS = 5
+    BAN_USER = 6
+    BAN_GEN = 7
+    LOCK_SYS_USER = 8
+    LOCK_SYS = 9
+    LOCK_USER = 10
+
+class CompanyCodes(Enum):
+    NONE = 0
+    SEGA = 1
+    BAMCO = 2
+    KONAMI = 3
+    TAITO = 4
+
+class ReaderFwVer(Enum): # Newer readers use a singly byte value
+    NONE = 0
+    TN32_10 = 1
+    TN32_12 = 2
+    OTHER = 9
+    
+    def __str__(self) -> str:
+        if self == self.TN32_10:
+            return "TN32MSEC003S F/W Ver1.0"
+        elif self == self.TN32_12:
+            return "TN32MSEC003S F/W Ver1.2"
+        elif self == self.NONE:
+            return "Not Specified"
+        elif self == self.OTHER:
+            return "Unknown/Other"
+        else:
+            raise ValueError(f"Bad ReaderFwVer value {self.value}")
+    
+    @classmethod
+    def from_byte(self, byte: bytes) -> Union["ReaderFwVer", int]:
+        try:
+            i = int.from_bytes(byte, 'little')
+            try:
+                return ReaderFwVer(i)
+            except ValueError:
+                return i
+        except TypeError:
+            return 0
+
+class ADBHeaderException(Exception):
+    pass
+
+HEADER_SIZE: Final[int] = 0x20
+CMD_CODE_GOODBYE: Final[int] = 0x66
+
+# everything is LE
+class ADBHeader:
+    def __init__(self, magic: int, protocol_ver: int, cmd: int, length: int, status: int, game_id: Union[str, bytes], store_id: int, keychip_id: Union[str, bytes]) -> None:
+        self.magic = magic # u16
+        self.protocol_ver = protocol_ver # u16
+        self.cmd = cmd # u16
+        self.length = length # u16
+        try:
+            self.status = ADBStatus(status) # u16
+        except ValueError as e:
+            raise ADBHeaderException(f"Status is incorrect! {e}")
+        self.game_id = game_id # 4 char + \x00
+        self.store_id = store_id # u32
+        self.keychip_id = keychip_id# 11 char + \x00
+                
+        if type(self.game_id) == bytes:
+            self.game_id = self.game_id.decode()
+        
+        if type(self.keychip_id) == bytes:
+            self.keychip_id = self.keychip_id.decode()
+        
+        self.game_id = self.game_id.replace("\0", "")
+        self.keychip_id = self.keychip_id.replace("\0", "")
+        if self.cmd != CMD_CODE_GOODBYE: # Games for some reason send no data with goodbye
+            self.validate()
+    
+    @classmethod
+    def from_data(cls, data: bytes) -> "ADBHeader":
+        magic, protocol_ver, cmd, length, status, game_id, store_id, keychip_id = struct.unpack_from("<5H6sI12s", data)
+        head = cls(magic, protocol_ver, cmd, length, status, game_id, store_id, keychip_id)
+        
+        if head.length > len(data):
+            raise ADBHeaderException(f"Length is incorrect! Expect {head.length}, got {len(data)}")
+        
+        return head
+    
+    def validate(self) -> bool:
+        if self.magic != 0xa13e:
+            raise ADBHeaderException(f"Magic {self.magic} != 0xa13e")
+        
+        if self.protocol_ver < 0x1000:
+            raise ADBHeaderException(f"Protocol version {hex(self.protocol_ver)} is invalid!")
+
+        if re.fullmatch(r"^S[0-9A-Z]{3}[P]?$", self.game_id) is None:
+            raise ADBHeaderException(f"Game ID {self.game_id} is invalid!")
+        
+        if self.store_id == 0:
+            raise ADBHeaderException(f"Store ID cannot be 0!")
+
+        if re.fullmatch(r"^A[0-9]{2}[E|X][0-9]{2}[A-HJ-NP-Z][0-9]{4}$", self.keychip_id) is None:
+            raise ADBHeaderException(f"Keychip ID {self.keychip_id} is invalid!")
+        
+        return True
+
+    def make(self) -> bytes:
+        resp_struct = Struct(
+            "magic" / Int16ul,
+            "unknown" / Int16ul,
+            "response_code" / Int16ul,
+            "length" / Int16ul,
+            "status" / Int16ul,
+            "game_id" / PaddedString(6, 'utf_8'),
+            "store_id" / Int32ul,
+            "keychip_id" / PaddedString(12, 'utf_8'),
+        )
+
+        return resp_struct.build(dict(
+            magic=self.magic,
+            unknown=self.protocol_ver,
+            response_code=self.cmd,
+            length=self.length,
+            status=self.status.value,
+            game_id = self.game_id,
+            store_id = self.store_id,
+            keychip_id = self.keychip_id,
+        ))
+
+class ADBBaseRequest:
+    def __init__(self, data: bytes) -> None:
+        self.head = ADBHeader.from_data(data)
+
+class ADBBaseResponse:
+    def __init__(self, code: int = 0, length: int = 0x20, status: int = 1, game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", protocol_ver: int = 0x3087) -> None:
+        self.head = ADBHeader(0xa13e, protocol_ver, code, length, status, game_id, store_id, keychip_id)
+
+    @classmethod
+    def from_req(cls, req: ADBHeader, cmd: int, length: int = 0x20, status: int = 1) -> "ADBBaseResponse":
+        return cls(cmd, length, status, req.game_id, req.store_id, req.keychip_id, req.protocol_ver)
+
+    def append_padding(self, data: bytes):
+        """Appends 0s to the end of the data until it's at the correct size"""
+        padding_size = self.head.length - len(data)
+        data += bytes(padding_size)
+        return data
+
+    def make(self) -> bytes:
+        return self.head.make()

core/adb_handlers/campaign.py

@@ -0,0 +1,132 @@
+from construct import Struct, Int16ul, Padding, Bytes, Int32ul, Int32sl
+
+from .base import *
+
+class Campaign:
+    def __init__(self) -> None:
+        self.id = 0
+        self.name = ""
+        self.announce_date = 0
+        self.start_date = 0
+        self.end_date = 0
+        self.distrib_start_date = 0
+        self.distrib_end_date = 0
+    
+    def make(self) -> bytes:
+        name_padding = bytes(128 - len(self.name))
+        return Struct(
+            "id" / Int32ul,
+            "name" / Bytes(128),
+            "announce_date" / Int32ul,
+            "start_date" / Int32ul,
+            "end_date" / Int32ul,
+            "distrib_start_date" / Int32ul,
+            "distrib_end_date" / Int32ul,
+            Padding(8),
+        ).build(dict(
+            id = self.id,
+            name = self.name.encode() + name_padding,
+            announce_date = self.announce_date,
+            start_date = self.start_date,
+            end_date = self.end_date,
+            distrib_start_date = self.distrib_start_date,
+            distrib_end_date = self.distrib_end_date,
+        ))
+
+class CampaignClear:
+    def __init__(self) -> None:
+        self.id = 0
+        self.entry_flag = 0
+        self.clear_flag = 0
+    
+    def make(self) -> bytes:
+        return Struct(
+            "id" / Int32ul,
+            "entry_flag" / Int32ul,
+            "clear_flag" / Int32ul,
+            Padding(4),
+        ).build(dict(
+            id = self.id,
+            entry_flag = self.entry_flag,
+            clear_flag = self.clear_flag,
+        ))
+
+class ADBCampaignResponse(ADBBaseResponse):
+    def __init__(self, game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", code: int = 0x0C, length: int = 0x200, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id)
+        self.campaigns = [Campaign(), Campaign(), Campaign()]
+    
+    @classmethod
+    def from_req(cls, req: ADBHeader) -> "ADBCampaignResponse":
+        c = cls(req.game_id, req.store_id, req.keychip_id)
+        c.head.protocol_ver = req.protocol_ver
+        return c
+
+    def make(self) -> bytes:
+        body = b""
+        
+        for c in self.campaigns:
+            body += c.make()
+        
+        self.head.length = HEADER_SIZE + len(body)
+        return self.head.make() + body
+
+class ADBOldCampaignRequest(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        self.campaign_id = struct.unpack_from("<I", data, 0x20)
+
+class ADBOldCampaignResponse(ADBBaseResponse):
+    def __init__(self, game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", code: int = 0x0C, length: int = 0x30, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id)
+        self.info0 = 0
+        self.info1 = 0
+        self.info2 = 0
+        self.info3 = 0
+    
+    @classmethod
+    def from_req(cls, req: ADBHeader) -> "ADBCampaignResponse":
+        c = cls(req.game_id, req.store_id, req.keychip_id)
+        c.head.protocol_ver = req.protocol_ver
+        return c
+    
+    def make(self) -> bytes:
+        resp_struct = Struct(
+            "info0" / Int32sl,
+            "info1" / Int32sl,
+            "info2" / Int32sl,
+            "info3" / Int32sl,
+        ).build(
+            info0 = self.info0,
+            info1 = self.info1,
+            info2 = self.info2,
+            info3 = self.info3,
+        )
+
+        self.head.length = HEADER_SIZE + len(resp_struct)
+        return self.head.make() + resp_struct
+
+class ADBCampaignClearRequest(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        self.aime_id = struct.unpack_from("<i", data, 0x20)
+
+class ADBCampaignClearResponse(ADBBaseResponse):
+    def __init__(self, game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", code: int = 0x0E, length: int = 0x50, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id)
+        self.campaign_clear_status = [CampaignClear(), CampaignClear(), CampaignClear()]
+    
+    @classmethod
+    def from_req(cls, req: ADBHeader) -> "ADBCampaignResponse":
+        c = cls(req.game_id, req.store_id, req.keychip_id)
+        c.head.protocol_ver = req.protocol_ver
+        return c
+    
+    def make(self) -> bytes:
+        body = b""
+        
+        for c in self.campaign_clear_status:
+            body += c.make()
+        
+        self.head.length = HEADER_SIZE + len(body)
+        return self.head.make() + body

core/adb_handlers/felica.py

@@ -0,0 +1,85 @@
+from construct import Struct, Int32sl, Padding, Int8ub, Int16sl
+from typing import Union
+from .base import *
+
+class ADBFelicaLookupRequest(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        idm, pmm = struct.unpack_from(">QQ", data, 0x20)
+        self.idm = hex(idm)[2:].upper()
+        self.pmm = hex(pmm)[2:].upper()
+
+class ADBFelicaLookupResponse(ADBBaseResponse):
+    def __init__(self, access_code: str = None, game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", code: int = 0x03, length: int = 0x30, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id)
+        self.access_code = access_code if access_code is not None else "00000000000000000000"
+    
+    @classmethod
+    def from_req(cls, req: ADBHeader, access_code: str = None) -> "ADBFelicaLookupResponse":
+        c = cls(access_code, req.game_id, req.store_id, req.keychip_id)
+        c.head.protocol_ver = req.protocol_ver
+        return c
+    
+    def make(self) -> bytes:        
+        resp_struct = Struct(
+            "felica_idx" / Int32ul,
+            "access_code" / Int8ub[10],
+            Padding(2)
+        ).build(dict(
+            felica_idx = 0,
+            access_code = bytes.fromhex(self.access_code)
+        ))
+
+        self.head.length = HEADER_SIZE + len(resp_struct)
+
+        return self.head.make() + resp_struct
+
+class ADBFelicaLookup2Request(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        self.random = struct.unpack_from("<16s", data, 0x20)[0]
+        idm, pmm = struct.unpack_from(">QQ", data, 0x30)
+        self.card_key_ver, self.write_ct, self.maca, company, fw_ver, self.dfc = struct.unpack_from("<16s16sQccH", data, 0x40)
+        self.idm = hex(idm)[2:].upper()
+        self.pmm = hex(pmm)[2:].upper()
+        self.company = CompanyCodes(int.from_bytes(company, 'little'))
+        self.fw_ver = ReaderFwVer.from_byte(fw_ver)
+
+class ADBFelicaLookup2Response(ADBBaseResponse):
+    def __init__(self, user_id: Union[int, None] = None, access_code: Union[str, None] = None, game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", code: int = 0x12, length: int = 0x130, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id)
+        self.user_id = user_id if user_id is not None else -1
+        self.access_code = access_code if access_code is not None else "00000000000000000000"
+        self.company = CompanyCodes.SEGA
+        self.portal_status = PortalRegStatus.NO_REG
+        self.auth_key = [0] * 256
+
+    @classmethod
+    def from_req(cls, req: ADBHeader, user_id: Union[int, None] = None, access_code: Union[str, None] = None) -> "ADBFelicaLookup2Response":
+        c = cls(user_id, access_code, req.game_id, req.store_id, req.keychip_id)
+        c.head.protocol_ver = req.protocol_ver
+        return c
+
+    def make(self) -> bytes:        
+        resp_struct = Struct(
+            "user_id" / Int32sl,
+            "relation1" / Int32sl,
+            "relation2" / Int32sl,
+            "access_code" / Int8ub[10],
+            "portal_status" / Int8ub,
+            "company_code" / Int8ub,
+            Padding(8),
+            "auth_key" / Int8ub[256],
+        ).build(dict(
+            user_id = self.user_id,
+            relation1 = -1, # Unsupported
+            relation2 = -1, # Unsupported
+            access_code = bytes.fromhex(self.access_code),
+            portal_status = self.portal_status.value,
+            company_code = self.company.value,
+            auth_key = self.auth_key
+        ))
+
+        self.head.length = HEADER_SIZE + len(resp_struct)
+
+        return self.head.make() + resp_struct

core/adb_handlers/log.py

@@ -0,0 +1,56 @@
+from construct import Struct, Padding, Int8sl
+from typing import Final, List
+
+from .base import *
+NUM_LOGS: Final[int] = 20
+NUM_LEN_LOG_EX: Final[int] = 48
+
+class AmLogEx:
+    def __init__(self, data: bytes) -> None:
+        self.aime_id, status, self.user_id, self.credit_ct, self.bet_ct, self.won_ct, self.local_time, \
+            self.tseq, self.place_id = struct.unpack("<IIQiii4xQiI", data)
+        self.status = LogStatus(status)
+
+class ADBStatusLogRequest(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        self.aime_id, status = struct.unpack_from("<II", data, 0x20)
+        self.status = LogStatus(status)
+
+class ADBLogRequest(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        self.aime_id, status, self.user_id, self.credit_ct, self.bet_ct, self.won_ct = struct.unpack_from("<IIQiii", data, 0x20)
+        self.status = LogStatus(status)
+
+class ADBLogExRequest(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        self.logs: List[AmLogEx] = []
+
+        for x in range(NUM_LOGS):
+            self.logs.append(AmLogEx(data[0x20 + (NUM_LEN_LOG_EX * x): 0x50 + (NUM_LEN_LOG_EX * x)]))
+        
+        self.num_logs = struct.unpack_from("<I", data, 0x03E0)[0]
+
+class ADBLogExResponse(ADBBaseResponse):
+    def __init__(self, game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", protocol_ver: int = 12423, code: int = 20, length: int = 64, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id, protocol_ver)
+
+    @classmethod
+    def from_req(cls, req: ADBHeader) -> "ADBLogExResponse":
+        c = cls(req.game_id, req.store_id, req.keychip_id, req.protocol_ver)
+        return c
+    
+    def make(self) -> bytes:
+        resp_struct = Struct(
+            "log_result" / Int8sl[NUM_LOGS],
+            Padding(12)
+        )
+
+        body = resp_struct.build(dict(
+            log_result = [1] * NUM_LOGS
+        ))
+
+        self.head.length = HEADER_SIZE + len(body)
+        return self.head.make() + body

core/adb_handlers/lookup.py

@@ -0,0 +1,82 @@
+from construct import Struct, Int32sl, Padding, Int8sl
+from typing import Union
+
+from .base import *
+
+class ADBLookupException(Exception):
+    pass
+
+class ADBLookupRequest(ADBBaseRequest):
+    def __init__(self, data: bytes) -> None:
+        super().__init__(data)
+        self.access_code = data[0x20:0x2A].hex()
+        company_code, fw_version, self.serial_number = struct.unpack_from("<bbI", data, 0x2A)
+        
+        try:
+            self.company_code = CompanyCodes(company_code)
+        except ValueError as e:
+            raise ADBLookupException(f"Invalid company code - {e}")
+        
+        self.fw_version = ReaderFwVer.from_byte(fw_version)
+
+class ADBLookupResponse(ADBBaseResponse):
+    def __init__(self, user_id: Union[int, None], game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", code: int = 0x06, length: int = 0x30, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id)
+        self.user_id = user_id if user_id is not None else -1
+        self.portal_reg = PortalRegStatus.NO_REG
+
+    @classmethod
+    def from_req(cls, req: ADBHeader, user_id: Union[int, None]) -> "ADBLookupResponse":
+        c = cls(user_id, req.game_id, req.store_id, req.keychip_id)
+        c.head.protocol_ver = req.protocol_ver
+        return c
+
+    def make(self):
+        resp_struct = Struct(
+            "user_id" / Int32sl,
+            "portal_reg" / Int8sl,
+            Padding(11)
+        )
+
+        body = resp_struct.build(dict(
+            user_id = self.user_id,
+            portal_reg = self.portal_reg.value
+        ))
+
+        self.head.length = HEADER_SIZE + len(body)
+        return self.head.make() + body
+
+class ADBLookupExResponse(ADBBaseResponse):
+    def __init__(self, user_id: Union[int, None], game_id: str = "SXXX", store_id: int = 1, keychip_id: str = "A69E01A8888", 
+                code: int = 0x10, length: int = 0x130, status: int = 1) -> None:
+        super().__init__(code, length, status, game_id, store_id, keychip_id)
+        self.user_id = user_id if user_id is not None else -1
+        self.portal_reg = PortalRegStatus.NO_REG
+        self.auth_key = [0] * 256
+
+    @classmethod
+    def from_req(cls, req: ADBHeader, user_id: Union[int, None]) -> "ADBLookupExResponse":
+        c = cls(user_id, req.game_id, req.store_id, req.keychip_id)
+        c.head.protocol_ver = req.protocol_ver
+        return c
+
+    def make(self):
+        resp_struct = Struct(
+            "user_id" / Int32sl,
+            "portal_reg" / Int8sl,
+            Padding(3),
+            "auth_key" / Int8sl[256],
+            "relation1" / Int32sl,
+            "relation2" / Int32sl,
+        )
+
+        body = resp_struct.build(dict(
+            user_id = self.user_id,
+            portal_reg = self.portal_reg.value,
+            auth_key = self.auth_key,
+            relation1 = -1,
+            relation2 = -1
+        ))
+
+        self.head.length = HEADER_SIZE + len(body)
+        return self.head.make() + body

core/aimedb.py

@@ -1,273 +1,24 @@
-from twisted.internet.protocol import Factory, Protocol
 import logging, coloredlogs
 from Crypto.Cipher import AES
-import struct
+from typing import Dict, Tuple, Callable, Union, Optional
-from typing import Dict, Any
+import asyncio
 from logging.handlers import TimedRotatingFileHandler
 
 from core.config import CoreConfig
+from core.utils import create_sega_auth_key
 from core.data import Data
+from .adb_handlers import *
 
-
+class AimedbServlette():
-class AimedbProtocol(Protocol):
+    request_list: Dict[int, Tuple[Callable[[bytes, int], Union[ADBBaseResponse, bytes]], int, str]] = {}
-    AIMEDB_RESPONSE_CODES = {
-        "felica_lookup": 0x03,
-        "lookup": 0x06,
-        "log": 0x0A,
-        "campaign": 0x0C,
-        "touch": 0x0E,
-        "lookup2": 0x10,
-        "felica_lookup2": 0x12,
-        "log2": 0x14,
-        "hello": 0x65,
-    }
-
-    request_list: Dict[int, Any] = {}
-
     def __init__(self, core_cfg: CoreConfig) -> None:        
-        self.logger = logging.getLogger("aimedb")
         self.config = core_cfg        
         self.data = Data(core_cfg)
-        if core_cfg.aimedb.key == "":
-            self.logger.error("!!!KEY NOT SET!!!")
-            exit(1)
 
-        self.request_list[0x01] = self.handle_felica_lookup
+        self.logger = logging.getLogger("aimedb")
-        self.request_list[0x04] = self.handle_lookup
+        if not hasattr(self.logger, "initted"):
-        self.request_list[0x05] = self.handle_register
-        self.request_list[0x09] = self.handle_log
-        self.request_list[0x0B] = self.handle_campaign
-        self.request_list[0x0D] = self.handle_touch
-        self.request_list[0x0F] = self.handle_lookup2
-        self.request_list[0x11] = self.handle_felica_lookup2
-        self.request_list[0x13] = self.handle_log2
-        self.request_list[0x64] = self.handle_hello
-
-    def append_padding(self, data: bytes):
-        """Appends 0s to the end of the data until it's at the correct size"""
-        length = struct.unpack_from("<H", data, 6)
-        padding_size = length[0] - len(data)
-        data += bytes(padding_size)
-        return data
-
-    def connectionMade(self) -> None:
-        self.logger.debug(f"{self.transport.getPeer().host} Connected")
-
-    def connectionLost(self, reason) -> None:
-        self.logger.debug(
-            f"{self.transport.getPeer().host} Disconnected - {reason.value}"
-        )
-
-    def dataReceived(self, data: bytes) -> None:
-        cipher = AES.new(self.config.aimedb.key.encode(), AES.MODE_ECB)
-
-        try:
-            decrypted = cipher.decrypt(data)
-        except:
-            self.logger.error(f"Failed to decrypt {data.hex()}")
-            return None
-
-        self.logger.debug(f"{self.transport.getPeer().host} wrote {decrypted.hex()}")
-
-        if not decrypted[1] == 0xA1 and not decrypted[0] == 0x3E:
-            self.logger.error(f"Bad magic")
-            return None
-
-        req_code = decrypted[4]
-
-        if req_code == 0x66:
-            self.logger.info(f"goodbye from {self.transport.getPeer().host}")
-            self.transport.loseConnection()
-            return
-
-        try:
-            resp = self.request_list[req_code](decrypted)
-            encrypted = cipher.encrypt(resp)
-            self.logger.debug(f"Response {resp.hex()}")
-            self.transport.write(encrypted)
-
-        except KeyError:
-            self.logger.error(f"Unknown command code {hex(req_code)}")
-            return None
-
-        except ValueError as e:
-            self.logger.error(f"Failed to encrypt {resp.hex()} because {e}")
-            return None
-
-    def handle_campaign(self, data: bytes) -> bytes:
-        self.logger.info(f"campaign from {self.transport.getPeer().host}")
-        ret = struct.pack(
-            "<5H",
-            0xA13E,
-            0x3087,
-            self.AIMEDB_RESPONSE_CODES["campaign"],
-            0x0200,
-            0x0001,
-        )
-        return self.append_padding(ret)
-
-    def handle_hello(self, data: bytes) -> bytes:
-        self.logger.info(f"hello from {self.transport.getPeer().host}")
-        ret = struct.pack(
-            "<5H", 0xA13E, 0x3087, self.AIMEDB_RESPONSE_CODES["hello"], 0x0020, 0x0001
-        )
-        return self.append_padding(ret)
-
-    def handle_lookup(self, data: bytes) -> bytes:
-        luid = data[0x20:0x2A].hex()
-        user_id = self.data.card.get_user_id_from_card(access_code=luid)
-
-        if user_id is None:
-            user_id = -1
-
-        self.logger.info(
-            f"lookup from {self.transport.getPeer().host}: luid {luid} -> user_id {user_id}"
-        )
-
-        ret = struct.pack(
-            "<5H", 0xA13E, 0x3087, self.AIMEDB_RESPONSE_CODES["lookup"], 0x0130, 0x0001
-        )
-        ret += bytes(0x20 - len(ret))
-
-        if user_id is None:
-            ret += struct.pack("<iH", -1, 0)
-        else:
-            ret += struct.pack("<l", user_id)
-        return self.append_padding(ret)
-
-    def handle_lookup2(self, data: bytes) -> bytes:
-        self.logger.info(f"lookup2")
-
-        ret = bytearray(self.handle_lookup(data))
-        ret[4] = self.AIMEDB_RESPONSE_CODES["lookup2"]
-
-        return bytes(ret)
-
-    def handle_felica_lookup(self, data: bytes) -> bytes:
-        idm = data[0x20:0x28].hex()
-        pmm = data[0x28:0x30].hex()
-        access_code = self.data.card.to_access_code(idm)
-        self.logger.info(
-            f"felica_lookup from {self.transport.getPeer().host}: idm {idm} pmm {pmm} -> access_code {access_code}"
-        )
-
-        ret = struct.pack(
-            "<5H",
-            0xA13E,
-            0x3087,
-            self.AIMEDB_RESPONSE_CODES["felica_lookup"],
-            0x0030,
-            0x0001,
-        )
-        ret += bytes(26)
-        ret += bytes.fromhex(access_code)
-
-        return self.append_padding(ret)
-
-    def handle_felica_lookup2(self, data: bytes) -> bytes:
-        idm = data[0x30:0x38].hex()
-        pmm = data[0x38:0x40].hex()
-        access_code = self.data.card.to_access_code(idm)
-        user_id = self.data.card.get_user_id_from_card(access_code=access_code)
-
-        if user_id is None:
-            user_id = -1
-
-        self.logger.info(
-            f"felica_lookup2 from {self.transport.getPeer().host}: idm {idm} ipm {pmm} -> access_code {access_code} user_id {user_id}"
-        )
-
-        ret = struct.pack(
-            "<5H",
-            0xA13E,
-            0x3087,
-            self.AIMEDB_RESPONSE_CODES["felica_lookup2"],
-            0x0140,
-            0x0001,
-        )
-        ret += bytes(22)
-        ret += struct.pack("<lq", user_id, -1)  # first -1 is ext_id, 3rd is access code
-        ret += bytes.fromhex(access_code)
-        ret += struct.pack("<l", 1)
-
-        return self.append_padding(ret)
-
-    def handle_touch(self, data: bytes) -> bytes:
-        self.logger.info(f"touch from {self.transport.getPeer().host}")
-        ret = struct.pack(
-            "<5H", 0xA13E, 0x3087, self.AIMEDB_RESPONSE_CODES["touch"], 0x0050, 0x0001
-        )
-        ret += bytes(5)
-        ret += struct.pack("<3H", 0x6F, 0, 1)
-
-        return self.append_padding(ret)
-
-    def handle_register(self, data: bytes) -> bytes:
-        luid = data[0x20:0x2A].hex()
-        if self.config.server.allow_user_registration:
-            user_id = self.data.user.create_user()
-
-            if user_id is None:
-                user_id = -1
-                self.logger.error("Failed to register user!")
-
-            else:
-                card_id = self.data.card.create_card(user_id, luid)
-
-                if card_id is None:
-                    user_id = -1
-                    self.logger.error("Failed to register card!")
-
-            self.logger.info(
-                f"register from {self.transport.getPeer().host}: luid {luid} -> user_id {user_id}"
-            )
-
-        else:
-            self.logger.info(
-                f"register from {self.transport.getPeer().host} blocked!: luid {luid}"
-            )
-            user_id = -1
-
-        ret = struct.pack(
-            "<5H",
-            0xA13E,
-            0x3087,
-            self.AIMEDB_RESPONSE_CODES["lookup"],
-            0x0030,
-            0x0001 if user_id > -1 else 0,
-        )
-        ret += bytes(0x20 - len(ret))
-        ret += struct.pack("<l", user_id)
-
-        return self.append_padding(ret)
-
-    def handle_log(self, data: bytes) -> bytes:
-        # TODO: Save aimedb logs
-        self.logger.info(f"log from {self.transport.getPeer().host}")
-        ret = struct.pack(
-            "<5H", 0xA13E, 0x3087, self.AIMEDB_RESPONSE_CODES["log"], 0x0020, 0x0001
-        )
-        return self.append_padding(ret)
-
-    def handle_log2(self, data: bytes) -> bytes:
-        self.logger.info(f"log2 from {self.transport.getPeer().host}")
-        ret = struct.pack(
-            "<5H", 0xA13E, 0x3087, self.AIMEDB_RESPONSE_CODES["log2"], 0x0040, 0x0001
-        )
-        ret += bytes(22)
-        ret += struct.pack("H", 1)
-
-        return self.append_padding(ret)
-
-
-class AimedbFactory(Factory):
-    protocol = AimedbProtocol
-
-    def __init__(self, cfg: CoreConfig) -> None:
-        self.config = cfg
             log_fmt_str = "[%(asctime)s] Aimedb | %(levelname)s | %(message)s"
             log_fmt = logging.Formatter(log_fmt_str)
-        self.logger = logging.getLogger("aimedb")
 
             fileHandler = TimedRotatingFileHandler(
                 "{0}/{1}.log".format(self.config.server.log_dir, "aimedb"),
@@ -284,14 +35,335 @@ class AimedbFactory(Factory):
 
             self.logger.setLevel(self.config.aimedb.loglevel)
             coloredlogs.install(
-            level=cfg.aimedb.loglevel, logger=self.logger, fmt=log_fmt_str
+                level=core_cfg.aimedb.loglevel, logger=self.logger, fmt=log_fmt_str
             )
+            self.logger.initted = True
 
-        if self.config.aimedb.key == "":
+        if not core_cfg.aimedb.key:
-            self.logger.error("Please set 'key' field in your config file.")
+            self.logger.error("!!!KEY NOT SET!!!")
             exit(1)
 
-        self.logger.info(f"Ready on port {self.config.aimedb.port}")
+        self.register_handler(0x01, 0x03, self.handle_felica_lookup, 'felica_lookup')
+        self.register_handler(0x02, 0x03, self.handle_felica_register, 'felica_register')
+
+        self.register_handler(0x04, 0x06, self.handle_lookup, 'lookup')
+        self.register_handler(0x05, 0x06, self.handle_register, 'register')
+
+        self.register_handler(0x07, 0x08, self.handle_status_log, 'status_log')
+        self.register_handler(0x09, 0x0A, self.handle_log, 'aime_log')       
+
+        self.register_handler(0x0B, 0x0C, self.handle_campaign, 'campaign')
+        self.register_handler(0x0D, 0x0E, self.handle_campaign_clear, 'campaign_clear')
+
+        self.register_handler(0x0F, 0x10, self.handle_lookup_ex, 'lookup_ex')
+        self.register_handler(0x11, 0x12, self.handle_felica_lookup_ex, 'felica_lookup_ex')
+
+        self.register_handler(0x13, 0x14, self.handle_log_ex, 'aime_log_ex')
+        self.register_handler(0x64, 0x65, self.handle_hello, 'hello')
+
+    def register_handler(self, cmd: int, resp:int, handler: Callable[[bytes, int], Union[ADBBaseResponse, bytes]], name: str) -> None:
+        self.request_list[cmd] = (handler, resp, name)
+    
+    def start(self) -> None:
+        self.logger.info(f"Start on port {self.config.aimedb.port}")
+        addr = self.config.aimedb.listen_address if self.config.aimedb.listen_address else self.config.server.listen_address
+        asyncio.create_task(asyncio.start_server(self.dataReceived, addr, self.config.aimedb.port))
+    
+    async def dataReceived(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
+        self.logger.debug(f"Connection made from {writer.get_extra_info('peername')[0]}")
+        while True:
+            try:
+                data: bytes = await reader.read(4096)
+                if len(data) == 0:
+                    self.logger.debug("Connection closed")
+                    return
+                await self.process_data(data, reader, writer)
+                await writer.drain()
+            except ConnectionResetError as e:
+                self.logger.debug("Connection reset, disconnecting")
+                return
+
+    async def process_data(self, data: bytes, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> Optional[bytes]:
+        addr = writer.get_extra_info('peername')[0]
+        cipher = AES.new(self.config.aimedb.key.encode(), AES.MODE_ECB)
+
+        try:
+            decrypted = cipher.decrypt(data)
+        
+        except Exception as e:
+            self.logger.error(f"Failed to decrypt {data.hex()} because {e}")
+            return
+
+        self.logger.debug(f"{addr} wrote {decrypted.hex()}")
+
+        try:
+            head = ADBHeader.from_data(decrypted)
+        
+        except ADBHeaderException as e:
+            self.logger.error(f"Error parsing ADB header: {e}")
+            try:    
+                encrypted = cipher.encrypt(ADBBaseResponse().make())
+                writer.write(encrypted)
+                await writer.drain()
+                return
+
+            except Exception as e:
+                self.logger.error(f"Failed to encrypt default response because {e}")
+            
+            return
+
+        if head.keychip_id == "ABCD1234567" or head.store_id == 0xfff0:
+            self.logger.warning(f"Request from uninitialized AMLib: {vars(head)}")
+
+        if head.cmd == 0x66:
+            self.logger.info("Goodbye")
+            writer.close()
+            return
+
+        handler, resp_code, name = self.request_list.get(head.cmd, (self.handle_default, None, 'default'))
+
+        if resp_code is None:
+            self.logger.warning(f"No handler for cmd {hex(head.cmd)}")
+        
+        elif resp_code > 0:
+            self.logger.info(f"{name} from {head.keychip_id} ({head.game_id}) @ {addr}")
+        
+        resp = await handler(decrypted, resp_code)
+
+        if type(resp) == ADBBaseResponse or issubclass(type(resp), ADBBaseResponse):
+            resp_bytes = resp.make()
+
+        elif type(resp) == bytes:
+            resp_bytes = resp
+        
+        elif resp is None: # Nothing to send, probably a goodbye
+            self.logger.warn(f"None return by handler for {name}")
+            return
+        
+        else:
+            self.logger.error(f"Unsupported type returned by ADB handler for {name}: {type(resp)}")
+            raise TypeError(f"Unsupported type returned by ADB handler for {name}: {type(resp)}")
+
+        try:
+            encrypted = cipher.encrypt(resp_bytes)
+            self.logger.debug(f"Response {resp_bytes.hex()}")
+            writer.write(encrypted)
+
+        except Exception as e:
+            self.logger.error(f"Failed to encrypt {resp_bytes.hex()} because {e}")
+    
+    async def handle_default(self, data: bytes, resp_code: int, length: int = 0x20) -> ADBBaseResponse:
+        req = ADBHeader.from_data(data)
+        return ADBBaseResponse(resp_code, length, 1, req.game_id, req.store_id, req.keychip_id, req.protocol_ver)
+
+    async def handle_hello(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+        return await self.handle_default(data, resp_code)
+
+    async def handle_campaign(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+        h = ADBHeader.from_data(data)
+        if h.protocol_ver >= 0x3030:
+            req = h
+            resp = ADBCampaignResponse.from_req(req)
+
+        else:
+            req = ADBOldCampaignRequest(data)
+            
+            self.logger.info(f"Legacy campaign request for campaign {req.campaign_id} (protocol version {hex(h.protocol_ver)})")
+            resp = ADBOldCampaignResponse.from_req(req.head)
+        
+        # We don't currently support campaigns
+        return resp
+
+    async def handle_lookup(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+        req = ADBLookupRequest(data)
+        user_id = await self.data.card.get_user_id_from_card(req.access_code)
+        is_banned = await self.data.card.get_card_banned(req.access_code)
+        is_locked = await self.data.card.get_card_locked(req.access_code)
+        
+        ret = ADBLookupResponse.from_req(req.head, user_id)
+        if is_banned and is_locked:
+            ret.head.status = ADBStatus.BAN_SYS_USER
+        elif is_banned:
+            ret.head.status = ADBStatus.BAN_SYS
+        elif is_locked:
+            ret.head.status = ADBStatus.LOCK_USER
+        
+        self.logger.info(
+            f"access_code {req.access_code} -> user_id {ret.user_id}"
+        )
+        
+        if user_id and user_id > 0:
+            await self.data.card.update_card_last_login(req.access_code)
+        return ret
+
+    async def handle_lookup_ex(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+        req = ADBLookupRequest(data)
+        user_id = await self.data.card.get_user_id_from_card(req.access_code)
+
+        is_banned = await self.data.card.get_card_banned(req.access_code)
+        is_locked = await self.data.card.get_card_locked(req.access_code)
+
+        ret = ADBLookupExResponse.from_req(req.head, user_id)
+        if is_banned and is_locked:
+            ret.head.status = ADBStatus.BAN_SYS_USER
+        elif is_banned:
+            ret.head.status = ADBStatus.BAN_SYS
+        elif is_locked:
+            ret.head.status = ADBStatus.LOCK_USER
+
+        self.logger.info(
+            f"access_code {req.access_code} -> user_id {ret.user_id}"
+        )
+
+        if user_id and user_id > 0 and self.config.aimedb.id_secret:
+            auth_key = create_sega_auth_key(user_id, req.head.game_id, req.head.store_id, req.head.keychip_id, self.config.aimedb.id_secret, self.config.aimedb.id_lifetime_seconds)
+            if auth_key is not None:
+                auth_key_extra_len = 256 - len(auth_key)
+                auth_key_full = auth_key.encode() + (b"\0" * auth_key_extra_len)
+                self.logger.debug(f"Generated auth token {auth_key}")
+                ret.auth_key = auth_key_full
+
+        if user_id and user_id > 0:
+            await self.data.card.update_card_last_login(req.access_code)
+        return ret
+
+    async def handle_felica_lookup(self, data: bytes, resp_code: int) -> bytes:
+        """
+        On official, I think a card has to be registered for this to actually work, but 
+        I'm making the executive decision to not implement that and just kick back our
+        faux generated access code. The real felica IDm -> access code conversion is done
+        on the ADB server, which we do not and will not ever have access to. Because we can
+        assure that all IDms will be unique, this basic 0-padded hex -> int conversion will
+        be fine.
+        """
+        req = ADBFelicaLookupRequest(data)
+        ac = self.data.card.to_access_code(req.idm)
+        self.logger.info(
+            f"idm {req.idm} ipm {req.pmm} -> access_code {ac}"
+        )
+        return ADBFelicaLookupResponse.from_req(req.head, ac)
+
+    async def handle_felica_register(self, data: bytes, resp_code: int) -> bytes:
+        """
+        I've never seen this used.
+        """
+        req = ADBFelicaLookupRequest(data)
+        ac = self.data.card.to_access_code(req.idm)
+        
+        if self.config.server.allow_user_registration:
+            user_id = await self.data.user.create_user()
+
+            if user_id is None:
+                self.logger.error("Failed to register user!")
+                user_id = -1
+
+            else:
+                card_id = await self.data.card.create_card(user_id, ac)
+
+                if card_id is None:
+                    self.logger.error("Failed to register card!")
+                    user_id = -1
+
+            self.logger.info(
+                f"Register access code {ac} (IDm: {req.idm} PMm: {req.pmm}) -> user_id {user_id}"
+            )
+
+        else:
+            self.logger.info(
+                f"Registration blocked!: access code {ac} (IDm: {req.idm} PMm: {req.pmm})"
+            )
+
+        if user_id > 0:
+            await self.data.card.update_card_last_login(ac)
+        return ADBFelicaLookupResponse.from_req(req.head, ac)
+
+    async def handle_felica_lookup_ex(self, data: bytes, resp_code: int) -> bytes:
+        req = ADBFelicaLookup2Request(data)
+        access_code = self.data.card.to_access_code(req.idm)
+        user_id = await self.data.card.get_user_id_from_card(access_code=access_code)
+
+        if user_id is None:
+            user_id = -1
+
+        self.logger.info(
+            f"idm {req.idm} ipm {req.pmm} -> access_code {access_code} user_id {user_id}"
+        )
+
+        resp = ADBFelicaLookup2Response.from_req(req.head, user_id, access_code)
+
+        if user_id and user_id > 0 and self.config.aimedb.id_secret:
+            auth_key = create_sega_auth_key(user_id, req.head.game_id, req.head.store_id, req.head.keychip_id, self.config.aimedb.id_secret, self.config.aimedb.id_lifetime_seconds)
+            if auth_key is not None:
+                auth_key_extra_len = 256 - len(auth_key)
+                auth_key_full = auth_key.encode() + (b"\0" * auth_key_extra_len)
+                self.logger.debug(f"Generated auth token {auth_key}")
+                resp.auth_key = auth_key_full
+        
+        if user_id and user_id > 0:
+            await self.data.card.update_card_last_login(access_code)
+        return resp
+
+    async def handle_campaign_clear(self, data: bytes, resp_code: int) -> ADBBaseResponse:
+        req = ADBCampaignClearRequest(data)
+
+        resp = ADBCampaignClearResponse.from_req(req.head)
+
+        # We don't support campaign stuff
+        return resp
+
+    async def handle_register(self, data: bytes, resp_code: int) -> bytes:
+        req = ADBLookupRequest(data)
+        user_id = -1
+
+        if self.config.server.allow_user_registration:
+            user_id = await self.data.user.create_user()
+
+            if user_id is None:
+                self.logger.error("Failed to register user!")
+                user_id = -1
+
+            else:
+                card_id = await self.data.card.create_card(user_id, req.access_code)
+
+                if card_id is None:
+                    self.logger.error("Failed to register card!")
+                    user_id = -1
+
+            self.logger.info(
+                f"Register access code {req.access_code} -> user_id {user_id}"
+            )
+
+        else:
+            self.logger.info(
+                f"Registration blocked!: access code {req.access_code}"
+            )
+
+        resp = ADBLookupResponse.from_req(req.head, user_id)
+        if resp.user_id <= 0:
+            resp.head.status = ADBStatus.BAN_SYS # Closest we can get to a "You cannot register"
+
+        else:
+            await self.data.card.update_card_last_login(req.access_code)
+
+        return resp
+
+    # TODO: Save these in some capacity, as deemed relevant
+    async def handle_status_log(self, data: bytes, resp_code: int) -> bytes:
+        req = ADBStatusLogRequest(data)
+        self.logger.info(f"User {req.aime_id} logged {req.status.name} event")
+        return ADBBaseResponse(resp_code, 0x20, 1, req.head.game_id, req.head.store_id, req.head.keychip_id, req.head.protocol_ver)
+
+    async def handle_log(self, data: bytes, resp_code: int) -> bytes:
+        req = ADBLogRequest(data)
+        self.logger.info(f"User {req.aime_id} logged {req.status.name} event, credit_ct: {req.credit_ct} bet_ct: {req.bet_ct} won_ct: {req.won_ct}")
+        return ADBBaseResponse(resp_code, 0x20, 1, req.head.game_id, req.head.store_id, req.head.keychip_id, req.head.protocol_ver)
+
+    async def handle_log_ex(self, data: bytes, resp_code: int) -> bytes:
+        req = ADBLogExRequest(data)
+        strs = []
+        self.logger.info(f"Recieved {req.num_logs} or {len(req.logs)} logs")
+        
+        for x in range(req.num_logs):
+            self.logger.debug(f"User {req.logs[x].aime_id} logged {req.logs[x].status.name} event, credit_ct: {req.logs[x].credit_ct} bet_ct: {req.logs[x].bet_ct} won_ct: {req.logs[x].won_ct}")
+        return ADBLogExResponse.from_req(req.head)
 
-    def buildProtocol(self, addr):
-        return AimedbProtocol(self.config)

core/app.py

@@ -0,0 +1,92 @@
+import yaml
+import logging
+import coloredlogs
+from logging.handlers import TimedRotatingFileHandler
+from starlette.routing import Route
+from starlette.requests import Request
+from starlette.applications import Starlette
+from starlette.responses import PlainTextResponse
+from os import environ, path, mkdir, W_OK, access
+from typing import List
+
+from core import CoreConfig, TitleServlet, MuchaServlet, AllnetServlet, BillingServlet, AimedbServlette
+from core.frontend import FrontendServlet
+
+async def dummy_rt(request: Request):
+    return PlainTextResponse("Service OK")
+
+cfg_dir = environ.get("ARTEMIS_CFG_DIR", "config")
+cfg: CoreConfig = CoreConfig()
+if path.exists(f"{cfg_dir}/core.yaml"):
+    cfg.update(yaml.safe_load(open(f"{cfg_dir}/core.yaml")))
+
+if not path.exists(cfg.server.log_dir):
+    mkdir(cfg.server.log_dir)
+
+if not access(cfg.server.log_dir, W_OK):
+    print(
+        f"Log directory {cfg.server.log_dir} NOT writable, please check permissions"
+    )
+    exit(1)
+
+logger = logging.getLogger("core")
+log_fmt_str = "[%(asctime)s] Core | %(levelname)s | %(message)s"
+log_fmt = logging.Formatter(log_fmt_str)
+
+fileHandler = TimedRotatingFileHandler(
+    "{0}/{1}.log".format(cfg.server.log_dir, "core"), when="d", backupCount=10
+)
+fileHandler.setFormatter(log_fmt)
+
+consoleHandler = logging.StreamHandler()
+consoleHandler.setFormatter(log_fmt)
+
+logger.addHandler(fileHandler)
+logger.addHandler(consoleHandler)
+
+log_lv = logging.DEBUG if cfg.server.is_develop else logging.INFO
+logger.setLevel(log_lv)
+coloredlogs.install(level=log_lv, logger=logger, fmt=log_fmt_str)
+
+logger.info(f"Artemis starting in {'develop' if cfg.server.is_develop else 'production'} mode")
+
+title = TitleServlet(cfg, cfg_dir) # This has to be loaded first to load plugins
+mucha = MuchaServlet(cfg, cfg_dir)
+
+route_lst: List[Route] = [
+    # Mucha
+    Route("/mucha_front/boardauth.do", mucha.handle_boardauth, methods=["POST"]),
+    Route("/mucha_front/updatacheck.do", mucha.handle_updatecheck, methods=["POST"]),
+    Route("/mucha_front/downloadstate.do", mucha.handle_dlstate, methods=["POST"]),
+    # General
+    Route("/", dummy_rt),
+    Route("/robots.txt", FrontendServlet.robots)
+]
+
+if not cfg.billing.standalone:
+    billing = BillingServlet(cfg, cfg_dir)
+    route_lst += [
+        Route("/request", billing.handle_billing_request, methods=["POST"]),
+        Route("/request/", billing.handle_billing_request, methods=["POST"]),
+    ]
+
+if not cfg.allnet.standalone:
+    allnet = AllnetServlet(cfg, cfg_dir)
+    route_lst += [
+        Route("/sys/servlet/PowerOn", allnet.handle_poweron, methods=["GET", "POST"]),
+        Route("/sys/servlet/DownloadOrder", allnet.handle_dlorder, methods=["GET", "POST"]),
+        Route("/sys/servlet/LoaderStateRecorder", allnet.handle_loaderstaterecorder, methods=["GET", "POST"]),
+        Route("/sys/servlet/Alive", allnet.handle_alive, methods=["GET", "POST"]),
+        Route("/naomitest.html", allnet.handle_naomitest),
+    ]
+    
+    if cfg.allnet.allow_online_updates:
+        route_lst += [
+            Route("/report-api/Report", allnet.handle_dlorder_report, methods=["POST"]),
+            Route("/dl/ini/{file:str}", allnet.handle_dlorder_ini),
+        ]
+
+for code, game in title.title_registry.items():
+    route_lst += game.get_routes()
+
+app = Starlette(cfg.server.is_develop, route_lst)

core/config.py

@@ -1,17 +1,49 @@
 import logging, os
 from typing import Any
 
-
 class ServerConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
     def listen_address(self) -> str:
+        """
+        Address Artemis will bind to and listen on
+        """
         return CoreConfig.get_config_field(
             self.__config, "core", "server", "listen_address", default="127.0.0.1"
         )
     
+    @property
+    def hostname(self) -> str:
+        """
+        Hostname sent to games
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "hostname", default="localhost"
+        )
+    
+    @property
+    def port(self) -> int:
+        """
+        Port the game will listen on
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "port", default=80
+        )
+
+    @property
+    def ssl_key(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "ssl_key", default="cert/title.key"
+        )
+
+    @property
+    def ssl_cert(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "title", "ssl_cert", default="cert/title.pem"
+        )
+
     @property
     def allow_user_registration(self) -> bool:
         return CoreConfig.get_config_field(
@@ -36,12 +68,49 @@ class ServerConfig:
             self.__config, "core", "server", "is_develop", default=True
         )
 
+    @property
+    def is_using_proxy(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "is_using_proxy", default=False
+        )
+
+    @property
+    def proxy_port(self) -> int:
+        """
+        What port the proxy is listening on. This will be sent instead of 'port' if 
+        is_using_proxy is True and this value is non-zero
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "proxy_port", default=0
+        )
+
+    @property
+    def proxy_port_ssl(self) -> int:
+        """
+        What port the proxy is listening for secure connections on. This will be sent 
+        instead of 'port' if is_using_proxy is True and this value is non-zero
+        """
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "proxy_port_ssl", default=0
+        )
+
     @property
     def log_dir(self) -> str:
         return CoreConfig.get_config_field(
             self.__config, "core", "server", "log_dir", default="logs"
         )
 
+    @property
+    def check_arcade_ip(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "check_arcade_ip", default=False
+        )
+
+    @property
+    def strict_ip_checking(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "server", "strict_ip_checking", default=False
+        )
 
 class TitleConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
@@ -56,18 +125,17 @@ class TitleConfig:
         )
 
     @property
-    def hostname(self) -> str:
+    def reboot_start_time(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "title", "hostname", default="localhost"
+            self.__config, "core", "title", "reboot_start_time", default=""
         )
 
     @property
-    def port(self) -> int:
+    def reboot_end_time(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "title", "port", default=8080
+            self.__config, "core", "title", "reboot_end_time", default=""
         )
 
-
 class DatabaseConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
@@ -105,7 +173,7 @@ class DatabaseConfig:
     @property
     def protocol(self) -> str:
         return CoreConfig.get_config_field(
-            self.__config, "core", "database", "type", default="mysql"
+            self.__config, "core", "database", "protocol", default="mysql"
         )
 
     @property
@@ -123,13 +191,9 @@ class DatabaseConfig:
         )
 
     @property
-    def user_table_autoincrement_start(self) -> int:
+    def enable_memcached(self) -> bool:
         return CoreConfig.get_config_field(
-            self.__config,
+            self.__config, "core", "database", "enable_memcached", default=True
-            "core",
-            "database",
-            "user_table_autoincrement_start",
-            default=10000,
         )
 
     @property
@@ -138,13 +202,12 @@ class DatabaseConfig:
             self.__config, "core", "database", "memcached_host", default="localhost"
         )
 
-
 class FrontendConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
     @property
-    def enable(self) -> int:
+    def enable(self) -> bool:
         return CoreConfig.get_config_field(
             self.__config, "core", "frontend", "enable", default=False
         )
@@ -152,7 +215,7 @@ class FrontendConfig:
     @property
     def port(self) -> int:
         return CoreConfig.get_config_field(
-            self.__config, "core", "frontend", "port", default=8090
+            self.__config, "core", "frontend", "port", default=8080
         )
 
     @property
@@ -163,11 +226,28 @@ class FrontendConfig:
             )
         )
     
+    @property
+    def secret(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "frontend", "secret", default=""
+        )
 
 class AllnetConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
+    @property
+    def standalone(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "allnet", "standalone", default=False
+        )
+    
+    @property
+    def port(self) -> int:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "allnet", "port", default=80
+        )
+
     @property
     def loglevel(self) -> int:
         return CoreConfig.str_to_loglevel(
@@ -176,23 +256,36 @@ class AllnetConfig:
             )
         )
 
-    @property
-    def port(self) -> int:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "allnet", "port", default=80
-        )
-
     @property
     def allow_online_updates(self) -> int:
         return CoreConfig.get_config_field(
             self.__config, "core", "allnet", "allow_online_updates", default=False
         )
 
+    @property
+    def update_cfg_folder(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "allnet", "update_cfg_folder", default=""
+        )
 
 class BillingConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
     
+    @property
+    def standalone(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "billing", "standalone", default=True
+        )
+
+    @property
+    def loglevel(self) -> int:
+        return CoreConfig.str_to_loglevel(
+            CoreConfig.get_config_field(
+                self.__config, "core", "billing", "loglevel", default="info"
+            )
+        )
+
     @property
     def port(self) -> int:
         return CoreConfig.get_config_field(
@@ -217,11 +310,22 @@ class BillingConfig:
             self.__config, "core", "billing", "signing_key", default="cert/billing.key"
         )
 
-
 class AimedbConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
+    @property
+    def enable(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "enable", default=True
+        )
+
+    @property
+    def listen_address(self) -> bool:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "listen_address", default=""
+        )
+
     @property
     def loglevel(self) -> int:
         return CoreConfig.str_to_loglevel(
@@ -242,17 +346,22 @@ class AimedbConfig:
             self.__config, "core", "aimedb", "key", default=""
         )
 
+    @property
+    def id_secret(self) -> str:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "id_secret", default=""
+        )
+
+    @property
+    def id_lifetime_seconds(self) -> int:
+        return CoreConfig.get_config_field(
+            self.__config, "core", "aimedb", "id_lifetime_seconds", default=86400
+        )
 
 class MuchaConfig:
     def __init__(self, parent_config: "CoreConfig") -> None:
         self.__config = parent_config
 
-    @property
-    def enable(self) -> int:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "mucha", "enable", default=False
-        )
-
     @property
     def loglevel(self) -> int:
         return CoreConfig.str_to_loglevel(
@@ -261,13 +370,6 @@ class MuchaConfig:
             )
         )
 
-    @property
-    def hostname(self) -> str:
-        return CoreConfig.get_config_field(
-            self.__config, "core", "mucha", "hostname", default="localhost"
-        )
-
-
 class CoreConfig(dict):
     def __init__(self) -> None:
         self.server = ServerConfig(self)
@@ -290,6 +392,19 @@ class CoreConfig(dict):
         else:
             return logging.INFO
     
+    @classmethod
+    def loglevel_to_str(cls, level: int) -> str:
+        if level == logging.ERROR:
+            return "error"
+        elif level == logging.WARN:
+            return "warn"
+        elif level == logging.INFO:
+            return "info"
+        elif level == logging.DEBUG:
+            return "debug"
+        else:
+            return "notset"
+
     @classmethod
     def get_config_field(
         cls, __config: dict, module, *path: str, default: Any = ""

core/data/alembic/README

@@ -0,0 +1 @@
+Generic single-database configuration.

core/data/alembic/alembic.ini

@@ -0,0 +1,64 @@
+# A generic, single database configuration.
+
+[alembic]
+script_location=.
+
+# template used to generate migration files
+# file_template = %%(rev)s_%%(slug)s
+
+# max length of characters to apply to the
+# "slug" field
+#truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; this defaults
+# to migrations//versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path
+# version_locations = %(here)s/bar %(here)s/bat migrations//versions
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

core/data/alembic/env.py

@@ -0,0 +1,81 @@
+from __future__ import with_statement
+from alembic import context
+from sqlalchemy import engine_from_config, pool
+from logging.config import fileConfig
+
+from core.data.schema.base import metadata
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+fileConfig(config.config_file_name)
+
+# add your model's MetaData object here
+# for 'autogenerate' support
+# from myapp import mymodel
+# target_metadata = mymodel.Base.metadata
+target_metadata = metadata
+
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+def run_migrations_offline():
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL
+    and not an Engine, though an Engine is acceptable
+    here as well.  By skipping the Engine creation
+    we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    """
+    raise Exception('Not implemented or configured!')
+
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(
+        url=url, target_metadata=target_metadata, literal_binds=True)
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online():
+    """Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine
+    and associate a connection with the context.
+
+    """
+    ini_section = config.get_section(config.config_ini_section)
+    overrides = context.get_x_argument(as_dictionary=True)
+    for override in overrides:
+        ini_section[override] = overrides[override]
+
+    connectable = engine_from_config(
+        ini_section,
+        prefix='sqlalchemy.',
+        poolclass=pool.NullPool)
+
+    with connectable.connect() as connection:
+        context.configure(
+            connection=connection,
+            target_metadata=target_metadata,
+            compare_type=True,
+            compare_server_default=True,
+        )
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

core/data/alembic/script.py.mako

@@ -0,0 +1,24 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+branch_labels = ${repr(branch_labels)}
+depends_on = ${repr(depends_on)}
+
+
+def upgrade():
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade():
+    ${downgrades if downgrades else "pass"}

core/data/alembic/versions/835b862f9bf0_initial_migration.py

@@ -0,0 +1,24 @@
+"""Initial Migration
+
+Revision ID: 835b862f9bf0
+Revises: 
+Create Date: 2024-01-09 13:06:10.787432
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '835b862f9bf0'
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    pass
+
+
+def downgrade():
+    pass

core/data/alembic/versions/d8950c7ce2fc_remove_old_db_mgmt_system.py

@@ -0,0 +1,29 @@
+"""Remove old db mgmt system
+
+Revision ID: d8950c7ce2fc
+Revises: 835b862f9bf0
+Create Date: 2024-01-09 13:43:51.381175
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = 'd8950c7ce2fc'
+down_revision = '835b862f9bf0'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.drop_table("schema_versions")
+
+
+def downgrade():
+    op.create_table(
+        "schema_versions",
+        sa.Column("game", sa.String(4), primary_key=True, nullable=False),
+        sa.Column("version", sa.Integer, nullable=False, server_default="1"),
+        mysql_charset="utf8mb4",
+    )

core/data/cache.py

@@ -17,7 +17,7 @@ except ModuleNotFoundError:
 
 def cached(lifetime: int = 10, extra_key: Any = None) -> Callable:
     def _cached(func: Callable) -> Callable:
-        if has_mc:
+        if has_mc and (cfg and cfg.database.enable_memcached):
             hostname = "127.0.0.1"
             if cfg:
                 hostname = cfg.database.memcached_host

core/data/database.py

@@ -1,13 +1,14 @@
 import logging, coloredlogs
 from typing import Optional
 from sqlalchemy.orm import scoped_session, sessionmaker
-from sqlalchemy.exc import SQLAlchemyError
 from sqlalchemy import create_engine
 from logging.handlers import TimedRotatingFileHandler
-import importlib, os
+import os
 import secrets, string
 import bcrypt
 from hashlib import sha256
+import alembic.config
+import glob
 
 from core.config import CoreConfig
 from core.data.schema import *
@@ -15,31 +16,47 @@ from core.utils import Utils
 
 
 class Data:
+    engine = None
+    session = None
+    user = None
+    arcade = None
+    card = None
+    base = None
     def __init__(self, cfg: CoreConfig) -> None:
         self.config = cfg
 
         if self.config.database.sha2_password:
             passwd = sha256(self.config.database.password.encode()).digest()
-            self.__url = f"{self.config.database.protocol}://{self.config.database.username}:{passwd.hex()}@{self.config.database.host}/{self.config.database.name}?charset=utf8mb4"
+            self.__url = f"{self.config.database.protocol}://{self.config.database.username}:{passwd.hex()}@{self.config.database.host}:{self.config.database.port}/{self.config.database.name}?charset=utf8mb4"
         else:
-            self.__url = f"{self.config.database.protocol}://{self.config.database.username}:{self.config.database.password}@{self.config.database.host}/{self.config.database.name}?charset=utf8mb4"
+            self.__url = f"{self.config.database.protocol}://{self.config.database.username}:{self.config.database.password}@{self.config.database.host}:{self.config.database.port}/{self.config.database.name}?charset=utf8mb4"
 
-        self.__engine = create_engine(self.__url, pool_recycle=3600)
+        if Data.engine is None:
-        session = sessionmaker(bind=self.__engine, autoflush=True, autocommit=True)
+            Data.engine = create_engine(self.__url, pool_recycle=3600)
-        self.session = scoped_session(session)
+            self.__engine = Data.engine
 
-        self.user = UserData(self.config, self.session)
+        if Data.session is None:
-        self.arcade = ArcadeData(self.config, self.session)
+            s = sessionmaker(bind=Data.engine, autoflush=True, autocommit=True)
-        self.card = CardData(self.config, self.session)
+            Data.session = scoped_session(s)
-        self.base = BaseData(self.config, self.session)
+
-        self.schema_ver_latest = 4
+        if Data.user is None:
+            Data.user = UserData(self.config, self.session)
+        
+        if Data.arcade is None:
+            Data.arcade = ArcadeData(self.config, self.session)
+        
+        if Data.card is None:
+            Data.card = CardData(self.config, self.session)
+        
+        if Data.base is None:
+            Data.base = BaseData(self.config, self.session)
 
-        log_fmt_str = "[%(asctime)s] %(levelname)s | Database | %(message)s"
-        log_fmt = logging.Formatter(log_fmt_str)
         self.logger = logging.getLogger("database")
 
         # Prevent the logger from adding handlers multiple times
         if not getattr(self.logger, "handler_set", None):            
+            log_fmt_str = "[%(asctime)s] %(levelname)s | Database | %(message)s"
+            log_fmt = logging.Formatter(log_fmt_str)
             fileHandler = TimedRotatingFileHandler(
                 "{0}/{1}.log".format(self.config.server.log_dir, "db"),
                 encoding="utf-8",
@@ -60,220 +77,173 @@ class Data:
             )
             self.logger.handler_set = True  # type: ignore
 
+    def __alembic_cmd(self, command: str, *args: str) -> None:
+        old_dir = os.path.abspath(os.path.curdir)
+        base_dir = os.path.join(os.path.abspath(os.path.curdir), 'core', 'data', 'alembic')
+        alembicArgs = [
+            "-c",
+            os.path.join(base_dir, "alembic.ini"),
+            "-x",
+            f"script_location={base_dir}",
+            "-x",
+            f"sqlalchemy.url={self.__url}",
+            command,
+        ]
+        alembicArgs.extend(args)
+        os.chdir(base_dir)
+        alembic.config.main(argv=alembicArgs)
+        os.chdir(old_dir)
+
     def create_database(self):
         self.logger.info("Creating databases...")
-        try:
+        metadata.create_all(
-            metadata.create_all(self.__engine.connect())
+            self.engine,
-        except SQLAlchemyError as e:
+            checkfirst=True,
-            self.logger.error(f"Failed to create databases! {e}")
-            return
-
-        games = Utils.get_all_titles()
-        for game_dir, game_mod in games.items():
-            try:
-                if hasattr(game_mod, "database") and hasattr(game_mod, "current_schema_version"):
-                    game_mod.database(self.config)
-                metadata.create_all(self.__engine.connect())
-
-                self.base.set_schema_ver(
-                    game_mod.current_schema_version, game_mod.game_codes[0]
         )
 
-            except Exception as e:
+        for _, mod in Utils.get_all_titles().items():
-                self.logger.warning(
-                    f"Could not load database schema from {game_dir} - {e}"
-                )
-
-        self.logger.info(f"Setting base_schema_ver to {self.schema_ver_latest}")
-        self.base.set_schema_ver(self.schema_ver_latest)
-
-        self.logger.info(
-            f"Setting user auto_incrememnt to {self.config.database.user_table_autoincrement_start}"
-        )
-        self.user.reset_autoincrement(
-            self.config.database.user_table_autoincrement_start
-        )
-
-    def recreate_database(self):
-        self.logger.info("Dropping all databases...")
-        self.base.execute("SET FOREIGN_KEY_CHECKS=0")
-        try:
-            metadata.drop_all(self.__engine.connect())
-        except SQLAlchemyError as e:
-            self.logger.error(f"Failed to drop databases! {e}")
-            return
-
-        for root, dirs, files in os.walk("./titles"):
-            for dir in dirs:
-                if not dir.startswith("__"):
-                    try:
-                        mod = importlib.import_module(f"titles.{dir}")
-
-                        try:
             if hasattr(mod, "database"):
                 mod.database(self.config)
-                            metadata.drop_all(self.__engine.connect())
+                metadata.create_all(
-
+                    self.engine,
-                        except Exception as e:
+                    checkfirst=True,
-                            self.logger.warning(
-                                f"Could not load database schema from {dir} - {e}"
                 )
 
-                    except ImportError as e:
+        # Stamp the end revision as if alembic had created it, so it can take off after this.
-                        self.logger.warning(
+        self.__alembic_cmd(
-                            f"Failed to load database schema dir {dir} - {e}"
+            "stamp",
+            "head",
         )
-            break
 
-        self.base.execute("SET FOREIGN_KEY_CHECKS=1")
+    def schema_upgrade(self, ver: str = None):
-
+        self.__alembic_cmd(
-        self.create_database()
+            "upgrade",
-
+            "head" if not ver else ver,
-    def migrate_database(self, game: str, version: int, action: str) -> None:
-        old_ver = self.base.get_schema_ver(game)
-        sql = ""
-
-        if old_ver is None:
-            self.logger.error(
-                f"Schema for game {game} does not exist, did you run the creation script?"
         )
-            return
 
-        if old_ver == version:
+    def schema_downgrade(self, ver: str):
-            self.logger.info(
+        self.__alembic_cmd(
-                f"Schema for game {game} is already version {old_ver}, nothing to do"
+            "downgrade",
+            ver,
         )
-            return
 
-        if action == "upgrade":
+    async def create_owner(self, email: Optional[str] = None, code: Optional[str] = "00000000000000000000") -> None:
-            for x in range(old_ver, version):
-                if not os.path.exists(
-                    f"core/data/schema/versions/{game.upper()}_{x + 1}_{action}.sql"
-                ):
-                    self.logger.error(
-                        f"Could not find {action} script {game.upper()}_{x + 1}_{action}.sql in core/data/schema/versions folder"
-                    )
-                    return
-
-                with open(
-                    f"core/data/schema/versions/{game.upper()}_{x + 1}_{action}.sql",
-                    "r",
-                    encoding="utf-8",
-                ) as f:
-                    sql = f.read()
-
-                result = self.base.execute(sql)
-                if result is None:
-                    self.logger.error("Error execuing sql script!")
-                    return None
-        
-        else:
-            for x in range(old_ver, version, -1):
-                if not os.path.exists(
-                    f"core/data/schema/versions/{game.upper()}_{x - 1}_{action}.sql"
-                ):
-                    self.logger.error(
-                        f"Could not find {action} script {game.upper()}_{x - 1}_{action}.sql in core/data/schema/versions folder"
-                    )
-                    return
-
-                with open(
-                    f"core/data/schema/versions/{game.upper()}_{x - 1}_{action}.sql",
-                    "r",
-                    encoding="utf-8",
-                ) as f:
-                    sql = f.read()
-
-                result = self.base.execute(sql)
-                if result is None:
-                    self.logger.error("Error execuing sql script!")
-                    return None
-
-        result = self.base.set_schema_ver(version, game)
-        if result is None:
-            self.logger.error("Error setting version in schema_version table!")
-            return None
-
-        self.logger.info(f"Successfully migrated {game} to schema version {version}")
-
-    def create_owner(self, email: Optional[str] = None) -> None:
         pw = "".join(
             secrets.choice(string.ascii_letters + string.digits) for i in range(20)
         )
         hash = bcrypt.hashpw(pw.encode(), bcrypt.gensalt())
 
-        user_id = self.user.create_user(email=email, permission=255, password=hash)
+        user_id = await self.user.create_user(username="sysowner", email=email, password=hash.decode(), permission=255)
         if user_id is None:
             self.logger.error(f"Failed to create owner with email {email}")
             return
 
-        card_id = self.card.create_card(user_id, "00000000000000000000")
+        card_id = await self.card.create_card(user_id, code)
         if card_id is None:
             self.logger.error(f"Failed to create card for owner with id {user_id}")
             return
 
-        self.logger.warn(
+        self.logger.warning(
-            f"Successfully created owner with email {email}, access code 00000000000000000000, and password {pw} Make sure to change this password and assign a real card ASAP!"
+            f"Successfully created owner with email {email}, access code {code}, and password {pw} Make sure to change this password and assign a real card ASAP!"
         )
     
-    def migrate_card(self, old_ac: str, new_ac: str, should_force: bool) -> None:
+    async def migrate(self) -> None:
-        if old_ac == new_ac:
+        exist = await self.base.execute("SELECT * FROM alembic_version")
-            self.logger.error("Both access codes are the same!")
+        if exist is not None:
+            self.logger.warn("No need to migrate as you have already migrated to alembic. If you are trying to upgrade the schema, use `upgrade` instead!")
             return
         
-        new_card = self.card.get_card_by_access_code(new_ac)
+        self.logger.info("Upgrading to latest with legacy system")
-        if new_card is None:
+        if not await self.legacy_upgrade():
-            self.card.update_access_code(old_ac, new_ac)
+            self.logger.warn("No need to migrate as you have already deleted the old schema_versions system. If you are trying to upgrade the schema, use `upgrade` instead!")
             return
+        self.logger.info("Done")
         
-        if not should_force:
+        self.logger.info("Stamp with initial revision")
-            self.logger.warn(
+        self.__alembic_cmd(
-                f"Card already exists for access code {new_ac} (id {new_card['id']}). If you wish to continue, rerun with the '--force' flag."
+            "stamp",
-                f" All exiting data on the target card {new_ac} will be perminently erased and replaced with data from card {old_ac}."
+            "835b862f9bf0",
         )
-            return
 
-        self.logger.info(
+        self.logger.info("Upgrade")
-            f"All exiting data on the target card {new_ac} will be perminently erased and replaced with data from card {old_ac}."
+        self.__alembic_cmd(
+            "upgrade",
+            "head",
         )
-        self.card.delete_card(new_card["id"])
-        self.card.update_access_code(old_ac, new_ac)
     
-        hanging_user = self.user.get_user(new_card["user"])
+    async def legacy_upgrade(self) -> bool:
-        if hanging_user["password"] is None:
+        vers = await self.base.execute("SELECT * FROM schema_versions")
-            self.logger.info(f"Delete hanging user {hanging_user['id']}")
+        if vers is None:
-            self.user.delete_user(hanging_user["id"])
+            self.logger.warn("Cannot legacy upgrade, schema_versions table unavailable!")
+            return False
         
-    def delete_hanging_users(self) -> None:
+        db_vers = {}
-        """
+        vers_list = vers.fetchall()
-        Finds and deletes users that have not registered for the webui that have no cards assocated with them.
+        for x in vers_list:
-        """
+            db_vers[x['game']] = x['version']
-        unreg_users = self.user.get_unregistered_users()
-        if unreg_users is None:
-            self.logger.error("Error occoured finding unregistered users")
         
-        for user in unreg_users:
+        core_now_ver = int(db_vers['CORE']) + 1
-            cards = self.card.get_user_cards(user["id"])
+        while os.path.exists(f"core/data/schema/versions/CORE_{core_now_ver}_upgrade.sql"):
-            if cards is None:
+            with open(f"core/data/schema/versions/CORE_{core_now_ver}_upgrade.sql", "r") as f:
-                self.logger.error(f"Error getting cards for user {user['id']}")
+                result = await self.base.execute(f.read())
-                continue
                 
-            if not cards:
+                if result is None:
-                self.logger.info(f"Delete hanging user {user['id']}")
+                    self.logger.error(f"Invalid upgrade script CORE_{core_now_ver}_upgrade.sql")
-                self.user.delete_user(user["id"])
-
-    def autoupgrade(self) -> None:
-        all_games = self.base.get_all_schema_vers()
-        if all_games is None:
-            self.logger.warn("Failed to get schema versions")
-        
-        for x in all_games:
-            game = x["game"].upper()
-            update_ver = 1
-            for y in range(2, 100):
-                if os.path.exists(f"core/data/schema/versions/{game}_{y}_upgrade.sql"):
-                    update_ver = y
-                else:
                     break
 
-            self.migrate_database(game, update_ver, "upgrade")
+                result = await self.base.execute(f"UPDATE schema_versions SET version = {core_now_ver} WHERE game = 'CORE'")
+                if result is None:
+                    self.logger.error(f"Failed to update schema version for CORE to {core_now_ver}")
+                    break
+            
+            self.logger.info(f"Upgrade CORE to version {core_now_ver}")
+            core_now_ver += 1
+        
+        for _, mod in Utils.get_all_titles().items():
+            game_codes = getattr(mod, "game_codes", [])
+            for game in game_codes:
+                if game not in db_vers:
+                    self.logger.warn(f"{game} does not have an antry in schema_versions, skipping")
+                    continue
+
+                now_ver = int(db_vers[game]) + 1
+                while os.path.exists(f"core/data/schema/versions/{game}_{now_ver}_upgrade.sql"):
+                    with open(f"core/data/schema/versions/{game}_{now_ver}_upgrade.sql", "r") as f:
+                        result = await self.base.execute(f.read())
+                        
+                        if result is None:
+                            self.logger.error(f"Invalid upgrade script {game}_{now_ver}_upgrade.sql")
+                            break
+
+                        result = await self.base.execute(f"UPDATE schema_versions SET version = {now_ver} WHERE game = '{game}'")
+                        if result is None:
+                            self.logger.error(f"Failed to update schema version for {game} to {now_ver}")
+                            break
+
+                    self.logger.info(f"Upgrade {game} to version {now_ver}")
+                    now_ver += 1
+        
+        return True
+
+    async def create_revision(self, message: str) -> None:
+        if not message:
+            self.logger.info("Message is required for create-revision")
+            return
+        
+        self.__alembic_cmd(
+            "revision",
+            "-m",
+            message,
+        )
+
+    async def create_revision_auto(self, message: str) -> None:
+        if not message:
+            self.logger.info("Message is required for create-revision")
+            return
+        
+        self.__alembic_cmd(
+            "revision",
+            "--autogenerate",
+            "-m",
+            message,
+        )

core/data/schema/arcade.py

@@ -1,9 +1,10 @@
-from typing import Optional, Dict
+from typing import Optional, Dict, List
-from sqlalchemy import Table, Column
+from sqlalchemy import Table, Column, and_, or_
 from sqlalchemy.sql.schema import ForeignKey, PrimaryKeyConstraint
-from sqlalchemy.types import Integer, String, Boolean
+from sqlalchemy.types import Integer, String, Boolean, JSON
 from sqlalchemy.sql import func, select
 from sqlalchemy.dialects.mysql import insert
+from sqlalchemy.engine import Row
 import re
 
 from core.data.schema.base import BaseData, metadata
@@ -21,6 +22,7 @@ arcade = Table(
     Column("city", String(255)),
     Column("region_id", Integer),
     Column("timezone", String(255)),
+    Column("ip", String(39)),
     mysql_charset="utf8mb4",
 )
 
@@ -39,7 +41,9 @@ machine = Table(
     Column("country", String(3)),  # overwrites if not null
     Column("timezone", String(255)),
     Column("ota_enable", Boolean),
+    Column("memo", String(255)),
     Column("is_cab", Boolean),
+    Column("data", JSON),
     mysql_charset="utf8mb4",
 )
 
@@ -65,7 +69,7 @@ arcade_owner = Table(
 
 
 class ArcadeData(BaseData):
-    def get_machine(self, serial: str = None, id: int = None) -> Optional[Dict]:
+    async def get_machine(self, serial: str = None, id: int = None) -> Optional[Row]:
         if serial is not None:
             serial = serial.replace("-", "")
             if len(serial) == 11:
@@ -85,12 +89,12 @@ class ArcadeData(BaseData):
             self.logger.error(f"{__name__ }: Need either serial or ID to look up!")
             return None
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    def put_machine(
+    async def put_machine(
         self,
         arcade_id: int,
         serial: str = "",
@@ -106,13 +110,13 @@ class ArcadeData(BaseData):
             arcade=arcade_id, keychip=serial, board=board, game=game, is_cab=is_cab
         )
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    def set_machine_serial(self, machine_id: int, serial: str) -> None:
+    async def set_machine_serial(self, machine_id: int, serial: str) -> None:
-        result = self.execute(
+        result = await self.execute(
             machine.update(machine.c.id == machine_id).values(keychip=serial)
         )
         if result is None:
@@ -121,8 +125,8 @@ class ArcadeData(BaseData):
             )
         return result.lastrowid
 
-    def set_machine_boardid(self, machine_id: int, boardid: str) -> None:
+    async def set_machine_boardid(self, machine_id: int, boardid: str) -> None:
-        result = self.execute(
+        result = await self.execute(
             machine.update(machine.c.id == machine_id).values(board=boardid)
         )
         if result is None:
@@ -130,14 +134,21 @@ class ArcadeData(BaseData):
                 f"Failed to update board id for machine {machine_id} -> {boardid}"
             )
 
-    def get_arcade(self, id: int) -> Optional[Dict]:
+    async def get_arcade(self, id: int) -> Optional[Row]:
         sql = arcade.select(arcade.c.id == id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
     
-    def put_arcade(
+    async def get_arcade_machines(self, id: int) -> Optional[List[Row]]:
+        sql = machine.select(machine.c.arcade == id)
+        result = await self.execute(sql)
+        if result is None:
+            return None
+        return result.fetchall()
+
+    async def put_arcade(
         self,
         name: str,
         nickname: str = None,
@@ -160,60 +171,62 @@ class ArcadeData(BaseData):
             regional_id=regional_id,
         )
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    def get_arcade_owners(self, arcade_id: int) -> Optional[Dict]:
+    async def get_arcades_managed_by_user(self, user_id: int) -> Optional[List[Row]]:
+        sql = select(arcade).join(arcade_owner, arcade_owner.c.arcade == arcade.c.id).where(arcade_owner.c.user == user_id)
+        result = await self.execute(sql)
+        if result is None:
+            return False
+        return result.fetchall()
+    
+    async def get_manager_permissions(self, user_id: int, arcade_id: int) -> Optional[int]:
+        sql = select(arcade_owner.c.permissions).where(and_(arcade_owner.c.user == user_id, arcade_owner.c.arcade == arcade_id))
+        result = await self.execute(sql)
+        if result is None:
+            return False
+        return result.fetchone()
+
+    async def get_arcade_owners(self, arcade_id: int) -> Optional[Row]:
         sql = select(arcade_owner).where(arcade_owner.c.arcade == arcade_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    def add_arcade_owner(self, arcade_id: int, user_id: int) -> None:
+    async def add_arcade_owner(self, arcade_id: int, user_id: int) -> None:
         sql = insert(arcade_owner).values(arcade=arcade_id, user=user_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
-    def format_serial(
+    async def format_serial(
         self, platform_code: str, platform_rev: int, serial_num: int, append: int = 4152
     ) -> str:
         return f"{platform_code}{platform_rev:02d}A{serial_num:04d}{append:04d}"  # 0x41 = A, 0x52 = R
 
     def validate_keychip_format(self, serial: str) -> bool:
-        serial = serial.replace("-", "")
+        if re.fullmatch(r"^A[0-9]{2}[E|X][-]?[0-9]{2}[A-HJ-NP-Z][0-9]{4}([0-9]{4})?$", serial) is None:
-        if len(serial) != 11 or len(serial) != 15:
-            self.logger.error(
-                f"Serial validate failed: Incorrect length for {serial} (len {len(serial)})"
-            )
-            return False
-
-        platform_code = serial[:4]
-        platform_rev = serial[4:6]
-        const_a = serial[6]
-        num = serial[7:11]
-        append = serial[11:15]
-
-        if re.match("A[7|6]\d[E|X][0|1][0|1|2]A\d{4,8}", serial) is None:
-            self.logger.error(f"Serial validate failed: {serial} failed regex")
-            return False
-
-        if len(append) != 0 or len(append) != 4:
-            self.logger.error(
-                f"Serial validate failed: {serial} had malformed append {append}"
-            )
-            return False
-
-        if len(num) != 4:
-            self.logger.error(
-                f"Serial validate failed: {serial} had malformed number {num}"
-            )
             return False
         
         return True
+
+    async def get_arcade_by_name(self, name: str) -> Optional[List[Row]]:
+        sql = arcade.select(or_(arcade.c.name.like(f"%{name}%"), arcade.c.nickname.like(f"%{name}%")))
+        result = await self.execute(sql)
+        if result is None:
+            return None
+        return result.fetchall()
+
+    async def get_arcades_by_ip(self, ip: str) -> Optional[List[Row]]:
+        sql = arcade.select().where(arcade.c.ip == ip)
+        result = await self.execute(sql)
+        if result is None:
+            return None
+        return result.fetchall()

core/data/schema/base.py

@@ -15,14 +15,6 @@ from core.config import CoreConfig
 
 metadata = MetaData()
 
-schema_ver = Table(
-    "schema_versions",
-    metadata,
-    Column("game", String(4), primary_key=True, nullable=False),
-    Column("version", Integer, nullable=False, server_default="1"),
-    mysql_charset="utf8mb4",
-)
-
 event_log = Table(
     "event_log",
     metadata,
@@ -43,11 +35,11 @@ class BaseData:
         self.conn = conn
         self.logger = logging.getLogger("database")
 
-    def execute(self, sql: str, opts: Dict[str, Any] = {}) -> Optional[CursorResult]:
+    async def execute(self, sql: str, opts: Dict[str, Any] = {}) -> Optional[CursorResult]:
         res = None
 
         try:
-            self.logger.info(f"SQL Execute: {''.join(str(sql).splitlines())} || {opts}")
+            self.logger.debug(f"SQL Execute: {''.join(str(sql).splitlines())}")
             res = self.conn.execute(text(sql), opts)
 
         except SQLAlchemyError as e:
@@ -58,7 +50,7 @@ class BaseData:
             self.logger.error(f"UnicodeEncodeError error {e}")
             return None
 
-        except:
+        except Exception:
             try:
                 res = self.conn.execute(sql, opts)
 
@@ -70,7 +62,7 @@ class BaseData:
                 self.logger.error(f"UnicodeEncodeError error {e}")
                 return None
 
-            except:
+            except Exception:
                 self.logger.error(f"Unknown error")
                 raise
 
@@ -82,40 +74,7 @@ class BaseData:
         """
         return randrange(10000, 9999999)
 
-    def get_all_schema_vers(self) -> Optional[List[Row]]:
+    async def log_event(
-        sql = select(schema_ver)
-
-        result = self.execute(sql)
-        if result is None:
-            return None
-        return result.fetchall()
-
-    def get_schema_ver(self, game: str) -> Optional[int]:
-        sql = select(schema_ver).where(schema_ver.c.game == game)
-
-        result = self.execute(sql)
-        if result is None:
-            return None
-
-        row = result.fetchone()
-        if row is None:
-            return None
-
-        return row["version"]
-
-    def set_schema_ver(self, ver: int, game: str = "CORE") -> Optional[int]:
-        sql = insert(schema_ver).values(game=game, version=ver)
-        conflict = sql.on_duplicate_key_update(version=ver)
-
-        result = self.execute(conflict)
-        if result is None:
-            self.logger.error(
-                f"Failed to update schema version for game {game} (v{ver})"
-            )
-            return None
-        return result.lastrowid
-
-    def log_event(
         self, system: str, type: str, severity: int, message: str, details: Dict = {}
     ) -> Optional[int]:
         sql = event_log.insert().values(
@@ -125,7 +84,7 @@ class BaseData:
             message=message,
             details=json.dumps(details),
         )
-        result = self.execute(sql)
+        result = await self.execute(sql)
 
         if result is None:
             self.logger.error(
@@ -135,9 +94,9 @@ class BaseData:
 
         return result.lastrowid
 
-    def get_event_log(self, entries: int = 100) -> Optional[List[Dict]]:
+    async def get_event_log(self, entries: int = 100) -> Optional[List[Dict]]:
         sql = event_log.select().limit(entries).all()
-        result = self.execute(sql)
+        result = await self.execute(sql)
 
         if result is None:
             return None
@@ -145,6 +104,8 @@ class BaseData:
 
     def fix_bools(self, data: Dict) -> Dict:
         for k, v in data.items():
+            if k == "userName" or k == "teamName":
+                continue
             if type(v) == str and v.lower() == "true":
                 data[k] = True
             elif type(v) == str and v.lower() == "false":

core/data/schema/card.py

@@ -27,70 +27,101 @@ aime_card = Table(
 
 
 class CardData(BaseData):
-    def get_card_by_access_code(self, access_code: str) -> Optional[Row]:
+    async def get_card_by_access_code(self, access_code: str) -> Optional[Row]:
         sql = aime_card.select(aime_card.c.access_code == access_code)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    def get_card_by_id(self, card_id: int) -> Optional[Row]:
+    async def get_card_by_id(self, card_id: int) -> Optional[Row]:
         sql = aime_card.select(aime_card.c.id == card_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchone()
 
-    def update_access_code(self, old_ac: str, new_ac: str) -> None:
+    async def update_access_code(self, old_ac: str, new_ac: str) -> None:
         sql = aime_card.update(aime_card.c.access_code == old_ac).values(
             access_code=new_ac
         )
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             self.logger.error(
                 f"Failed to change card access code from {old_ac} to {new_ac}"
             )
 
-    def get_user_id_from_card(self, access_code: str) -> Optional[int]:
+    async def get_user_id_from_card(self, access_code: str) -> Optional[int]:
         """
         Given a 20 digit access code as a string, get the user id associated with that card
         """
-        card = self.get_card_by_access_code(access_code)
+        card = await self.get_card_by_access_code(access_code)
         if card is None:
             return None
 
         return int(card["user"])
 
-    def delete_card(self, card_id: int) -> None:
+    async def get_card_banned(self, access_code: str) -> Optional[bool]:
+        """
+        Given a 20 digit access code as a string, check if the card is banned
+        """
+        card = await self.get_card_by_access_code(access_code)
+        if card is None:
+            return None
+        if card["is_banned"]:
+            return True
+        return False
+    
+    async def get_card_locked(self, access_code: str) -> Optional[bool]:
+        """
+        Given a 20 digit access code as a string, check if the card is locked
+        """
+        card = await self.get_card_by_access_code(access_code)
+        if card is None:
+            return None
+        if card["is_locked"]:
+            return True
+        return False
+
+    async def delete_card(self, card_id: int) -> None:
         sql = aime_card.delete(aime_card.c.id == card_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             self.logger.error(f"Failed to delete card with id {card_id}")
 
-    def get_user_cards(self, aime_id: int) -> Optional[List[Row]]:
+    async def get_user_cards(self, aime_id: int) -> Optional[List[Row]]:
         """
         Returns all cards owned by a user
         """
         sql = aime_card.select(aime_card.c.user == aime_id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
 
-    def create_card(self, user_id: int, access_code: str) -> Optional[int]:
+    async def create_card(self, user_id: int, access_code: str) -> Optional[int]:
         """
         Given a aime_user id and a 20 digit access code as a string, create a card and return the ID if successful
         """
         sql = aime_card.insert().values(user=user_id, access_code=access_code)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.lastrowid
 
+    async def update_card_last_login(self, access_code: str) -> None:
+        sql = aime_card.update(aime_card.c.access_code == access_code).values(
+            last_login_date=func.now()
+        )
+        
+        result = await self.execute(sql)
+        if result is None:
+            self.logger.warn(f"Failed to update last login time for {access_code}")
+    
     def to_access_code(self, luid: str) -> str:
         """
         Given a felica cards internal 16 hex character luid, convert it to a 0-padded 20 digit access code as a string

core/data/schema/user.py

@@ -1,4 +1,3 @@
-from enum import Enum
 from typing import Optional, List
 from sqlalchemy import Table, Column
 from sqlalchemy.types import Integer, String, TIMESTAMP
@@ -24,15 +23,8 @@ aime_user = Table(
     mysql_charset="utf8mb4",
 )
 
-
-class PermissionBits(Enum):
-    PermUser = 1
-    PermMod = 2
-    PermSysAdmin = 4
-
-
 class UserData(BaseData):
-    def create_user(
+    async def create_user(
         self,
         id: int = None,
         username: str = None,
@@ -60,47 +52,71 @@ class UserData(BaseData):
             username=username, email=email, password=password, permissions=permission
         )
 
-        result = self.execute(conflict)
+        result = await self.execute(conflict)
         if result is None:
             return None
         return result.lastrowid
 
-    def get_user(self, user_id: int) -> Optional[Row]:
+    async def get_user(self, user_id: int) -> Optional[Row]:
         sql = select(aime_user).where(aime_user.c.id == user_id)
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return False
         return result.fetchone()
 
-    def check_password(self, user_id: int, passwd: bytes = None) -> bool:
+    async def check_password(self, user_id: int, passwd: bytes = None) -> bool:
-        usr = self.get_user(user_id)
+        usr = await self.get_user(user_id)
         if usr is None:
             return False
 
         if usr["password"] is None:
             return False
         
+        if passwd is None or not passwd:
+            return False
+
         return bcrypt.checkpw(passwd, usr["password"].encode())
 
-    def reset_autoincrement(self, ai_value: int) -> None:
+    async def delete_user(self, user_id: int) -> None:
-        # ALTER TABLE isn't in sqlalchemy so we do this the ugly way
-        sql = f"ALTER TABLE aime_user AUTO_INCREMENT={ai_value}"
-        self.execute(sql)
-
-    def delete_user(self, user_id: int) -> None:
         sql = aime_user.delete(aime_user.c.id == user_id)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             self.logger.error(f"Failed to delete user with id {user_id}")
 
-    def get_unregistered_users(self) -> List[Row]:
+    async def get_unregistered_users(self) -> List[Row]:
         """
         Returns a list of users who have not registered with the webui. They may or may not have cards.
         """
         sql = select(aime_user).where(aime_user.c.password == None)
 
-        result = self.execute(sql)
+        result = await self.execute(sql)
         if result is None:
             return None
         return result.fetchall()
+
+    async def find_user_by_email(self, email: str) -> Row:
+        sql = select(aime_user).where(aime_user.c.email == email)
+        result = await self.execute(sql)
+        if result is None:
+            return False
+        return result.fetchone()
+
+    async def find_user_by_username(self, username: str) -> List[Row]:
+        sql = aime_user.select(aime_user.c.username.like(f"%{username}%"))
+        result = await self.execute(sql)
+        if result is None:
+            return False
+        return result.fetchall()
+
+    async def change_password(self, user_id: int, new_passwd: str) -> bool:
+        sql = aime_user.update(aime_user.c.id == user_id).values(password = new_passwd)
+
+        result = await self.execute(sql)
+        return result is not None
+
+    async def change_username(self, user_id: int, new_name: str) -> bool:
+        sql = aime_user.update(aime_user.c.id == user_id).values(username = new_name)
+
+        result = await self.execute(sql)
+        return result is not None

core/data/schema/versions/CORE_4_rollback.sql

@@ -0,0 +1,3 @@
+ALTER TABLE machine DROP COLUMN memo;
+ALTER TABLE machine DROP COLUMN is_blacklisted;
+ALTER TABLE machine DROP COLUMN `data`;

core/data/schema/versions/CORE_5_rollback.sql

@@ -0,0 +1 @@
+ALTER TABLE arcade DROP COLUMN 'ip';

core/data/schema/versions/CORE_5_upgrade.sql

@@ -0,0 +1,3 @@
+ALTER TABLE machine ADD memo varchar(255) NULL;
+ALTER TABLE machine ADD is_blacklisted tinyint(1) NULL;
+ALTER TABLE machine ADD `data` longtext NULL;

core/data/schema/versions/CORE_6_upgrade.sql

@@ -0,0 +1 @@
+ALTER TABLE arcade ADD ip varchar(39) NULL;

core/data/schema/versions/SBZV_4_rollback.sql

@@ -0,0 +1,9 @@
+ALTER TABLE diva_profile
+    DROP cnp_cid,
+    DROP cnp_val,
+    DROP cnp_rr,
+    DROP cnp_sp,
+    DROP btn_se_eqp,
+    DROP sld_se_eqp,
+    DROP chn_sld_se_eqp,
+    DROP sldr_tch_se_eqp;

core/data/schema/versions/SBZV_5_rollback.sql

@@ -0,0 +1,2 @@
+ALTER TABLE diva_profile
+    DROP skn_eqp;

core/data/schema/versions/SBZV_5_upgrade.sql

@@ -0,0 +1,9 @@
+ALTER TABLE diva_profile
+    ADD cnp_cid INT NOT NULL DEFAULT -1,
+    ADD cnp_val INT NOT NULL DEFAULT -1,
+    ADD cnp_rr INT NOT NULL DEFAULT -1,
+    ADD cnp_sp VARCHAR(255) NOT NULL DEFAULT "",
+    ADD btn_se_eqp INT NOT NULL DEFAULT -1,
+    ADD sld_se_eqp INT NOT NULL DEFAULT -1,
+    ADD chn_sld_se_eqp INT NOT NULL DEFAULT -1,
+    ADD sldr_tch_se_eqp INT NOT NULL DEFAULT -1;

core/data/schema/versions/SBZV_6_upgrade.sql

@@ -0,0 +1,2 @@
+ALTER TABLE diva_profile
+    ADD skn_eqp INT NOT NULL DEFAULT 0;

core/data/schema/versions/SDBT_3_rollback.sql

@@ -0,0 +1,30 @@
+SET FOREIGN_KEY_CHECKS = 0;
+
+ALTER TABLE chuni_score_playlog
+    DROP COLUMN regionId,
+    DROP COLUMN machineType;
+
+ALTER TABLE chuni_static_events
+    DROP COLUMN startDate;
+
+ALTER TABLE chuni_profile_data
+    DROP COLUMN rankUpChallengeResults;
+
+ALTER TABLE chuni_static_login_bonus
+    DROP FOREIGN KEY chuni_static_login_bonus_ibfk_1;
+
+ALTER TABLE chuni_static_login_bonus_preset
+    DROP PRIMARY KEY;
+
+ALTER TABLE chuni_static_login_bonus_preset
+    CHANGE COLUMN presetId id INT NOT NULL;
+ALTER TABLE chuni_static_login_bonus_preset
+    ADD PRIMARY KEY(id);
+ALTER TABLE chuni_static_login_bonus_preset
+    ADD CONSTRAINT chuni_static_login_bonus_preset_uk UNIQUE(id, version);
+
+ALTER TABLE chuni_static_login_bonus
+    ADD CONSTRAINT chuni_static_login_bonus_ibfk_1 FOREIGN KEY(presetId)
+    REFERENCES chuni_static_login_bonus_preset(id) ON UPDATE CASCADE ON DELETE CASCADE;
+
+SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDBT_4_rollback.sql

@@ -0,0 +1,12 @@
+SET FOREIGN_KEY_CHECKS = 0;
+
+ALTER TABLE chuni_score_playlog
+  CHANGE COLUMN isClear isClear TINYINT(1) NULL DEFAULT NULL;
+
+ALTER TABLE chuni_score_best
+  CHANGE COLUMN isSuccess isSuccess TINYINT(1) NULL DEFAULT NULL ;
+
+ALTER TABLE chuni_score_playlog
+  DROP COLUMN ticketId;
+
+SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDBT_4_upgrade.sql

@@ -0,0 +1,29 @@
+SET FOREIGN_KEY_CHECKS = 0;
+
+ALTER TABLE chuni_score_playlog
+    ADD COLUMN regionId INT,
+    ADD COLUMN machineType INT;
+
+ALTER TABLE chuni_static_events
+    ADD COLUMN startDate TIMESTAMP NOT NULL DEFAULT current_timestamp();
+
+ALTER TABLE chuni_profile_data
+    ADD COLUMN rankUpChallengeResults JSON;
+
+ALTER TABLE chuni_static_login_bonus
+    DROP FOREIGN KEY chuni_static_login_bonus_ibfk_1;
+
+ALTER TABLE chuni_static_login_bonus_preset
+    CHANGE COLUMN id presetId INT NOT NULL;
+ALTER TABLE chuni_static_login_bonus_preset
+    DROP PRIMARY KEY;
+ALTER TABLE chuni_static_login_bonus_preset
+    DROP INDEX chuni_static_login_bonus_preset_uk;
+ALTER TABLE chuni_static_login_bonus_preset
+    ADD CONSTRAINT chuni_static_login_bonus_preset_pk PRIMARY KEY (presetId, version);
+
+ALTER TABLE chuni_static_login_bonus
+    ADD CONSTRAINT chuni_static_login_bonus_ibfk_1 FOREIGN KEY (presetId, version)
+    REFERENCES chuni_static_login_bonus_preset(presetId, version) ON UPDATE CASCADE ON DELETE CASCADE;
+
+SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDBT_5_upgrade.sql

@@ -0,0 +1,12 @@
+SET FOREIGN_KEY_CHECKS = 0;
+
+ALTER TABLE chuni_score_playlog
+  CHANGE COLUMN isClear isClear TINYINT(6) NULL DEFAULT NULL;
+
+ALTER TABLE chuni_score_best
+  CHANGE COLUMN isSuccess isSuccess INT(11) NULL DEFAULT NULL ;
+
+ALTER TABLE chuni_score_playlog
+  ADD COLUMN ticketId INT(11) NULL AFTER machineType;
+
+SET FOREIGN_KEY_CHECKS = 1;

core/data/schema/versions/SDDT_4_rollback.sql

@@ -0,0 +1,2 @@
+ALTER TABLE ongeki_static_events
+DROP COLUMN startDate;

core/data/schema/versions/SDDT_5_rollback.sql

@@ -0,0 +1,22 @@
+SET FOREIGN_KEY_CHECKS=0;
+
+ALTER TABLE ongeki_user_event_point DROP COLUMN version;
+ALTER TABLE ongeki_user_event_point DROP COLUMN `rank`;
+ALTER TABLE ongeki_user_event_point DROP COLUMN `type`;
+ALTER TABLE ongeki_user_event_point DROP COLUMN date;
+
+ALTER TABLE ongeki_user_tech_event DROP COLUMN version;
+
+ALTER TABLE ongeki_user_mission_point DROP COLUMN version;
+
+ALTER TABLE ongeki_static_events DROP COLUMN endDate;
+
+DROP TABLE ongeki_tech_event_ranking;
+DROP TABLE ongeki_static_music_ranking_list;
+DROP TABLE ongeki_static_rewards;
+DROP TABLE ongeki_static_present_list;
+DROP TABLE ongeki_static_tech_music;
+DROP TABLE ongeki_static_client_testmode;
+DROP TABLE ongeki_static_game_point;
+
+SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDDT_5_upgrade.sql

@@ -0,0 +1,2 @@
+ALTER TABLE ongeki_static_events
+ADD COLUMN startDate TIMESTAMP NOT NULL DEFAULT current_timestamp();

core/data/schema/versions/SDDT_6_upgrade.sql

@@ -0,0 +1,98 @@
+SET FOREIGN_KEY_CHECKS=0;
+
+ALTER TABLE ongeki_user_event_point ADD COLUMN version INTEGER NOT NULL;
+ALTER TABLE ongeki_user_event_point ADD COLUMN `rank` INTEGER;
+ALTER TABLE ongeki_user_event_point ADD COLUMN `type` INTEGER NOT NULL;
+ALTER TABLE ongeki_user_event_point ADD COLUMN date VARCHAR(25);
+
+ALTER TABLE ongeki_user_tech_event ADD COLUMN version INTEGER NOT NULL;
+
+ALTER TABLE ongeki_user_mission_point ADD COLUMN version INTEGER NOT NULL;
+
+ALTER TABLE ongeki_static_events ADD COLUMN endDate TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;
+
+CREATE TABLE ongeki_tech_event_ranking (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    user INT NOT NULL,
+    version INT NOT NULL,
+    date VARCHAR(25),
+    eventId INT NOT NULL,
+    `rank` INT,
+    totalPlatinumScore INT NOT NULL,
+    totalTechScore INT NOT NULL,
+    UNIQUE KEY ongeki_tech_event_ranking_uk (user, eventId),
+    CONSTRAINT ongeki_tech_event_ranking_ibfk1 FOREIGN KEY (user) REFERENCES aime_user(id) ON DELETE CASCADE ON UPDATE CASCADE
+);
+
+CREATE TABLE ongeki_static_music_ranking_list (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    musicId INT NOT NULL,
+    point INT NOT NULL,
+    userName VARCHAR(255),
+    UNIQUE KEY ongeki_static_music_ranking_list_uk (version, musicId)
+);
+
+CREATE TABLE ongeki_static_rewards (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    rewardId INT NOT NULL,
+    rewardName VARCHAR(255) NOT NULL,
+    itemKind INT NOT NULL,
+    itemId INT NOT NULL,
+    UNIQUE KEY ongeki_tech_event_ranking_uk (version, rewardId)
+);
+
+CREATE TABLE ongeki_static_present_list (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    presentId INT NOT NULL,
+    presentName VARCHAR(255) NOT NULL,
+    rewardId INT NOT NULL,
+    stock INT NOT NULL,
+    message VARCHAR(255),
+    startDate VARCHAR(25) NOT NULL,
+    endDate VARCHAR(25) NOT NULL,
+    UNIQUE KEY ongeki_static_present_list_uk (version, presentId, rewardId)
+);
+
+CREATE TABLE ongeki_static_tech_music (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    version INT NOT NULL,
+    eventId INT NOT NULL,
+    musicId INT NOT NULL,
+    level INT NOT NULL,
+    UNIQUE KEY ongeki_static_tech_music_uk (version, musicId, eventId)
+);
+
+CREATE TABLE ongeki_static_client_testmode (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    regionId INT NOT NULL,
+    placeId INT NOT NULL,
+    clientId VARCHAR(11) NOT NULL,
+    updateDate TIMESTAMP NOT NULL,
+    isDelivery BOOLEAN NOT NULL,
+    groupId INT NOT NULL,
+    groupRole INT NOT NULL,
+    continueMode INT NOT NULL,
+    selectMusicTime INT NOT NULL,
+    advertiseVolume INT NOT NULL,
+    eventMode INT NOT NULL,
+    eventMusicNum INT NOT NULL,
+    patternGp INT NOT NULL,
+    limitGp INT NOT NULL,
+    maxLeverMovable INT NOT NULL,
+    minLeverMovable INT NOT NULL,
+    UNIQUE KEY ongeki_static_client_testmode_uk (clientId)
+);
+
+CREATE TABLE ongeki_static_game_point (
+    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
+    `type` INT NOT NULL,
+    cost INT NOT NULL,
+    startDate VARCHAR(25) NOT NULL DEFAULT "2000-01-01 05:00:00.0",
+    endDate VARCHAR(25) NOT NULL DEFAULT "2099-01-01 05:00:00.0",
+    UNIQUE KEY ongeki_static_game_point_uk (`type`)
+);
+
+SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDEZ_4_rollback.sql

@@ -0,0 +1,3 @@
+ALTER TABLE mai2_item_card 
+    CHANGE COLUMN startDate startDate TIMESTAMP DEFAULT "2018-01-01 00:00:00.0",
+    CHANGE COLUMN endDate endDate TIMESTAMP DEFAULT "2038-01-01 00:00:00.0";

core/data/schema/versions/SDEZ_5_rollback.sql

@@ -0,0 +1,78 @@
+DELETE FROM mai2_static_event WHERE version < 13;
+UPDATE mai2_static_event SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_static_music WHERE version < 13;
+UPDATE mai2_static_music SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_static_ticket WHERE version < 13;
+UPDATE mai2_static_ticket SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_static_cards WHERE version < 13;
+UPDATE mai2_static_cards SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_profile_detail WHERE version < 13;
+UPDATE mai2_profile_detail SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_profile_extend WHERE version < 13;
+UPDATE mai2_profile_extend SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_profile_option WHERE version < 13;
+UPDATE mai2_profile_option SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_profile_ghost WHERE version < 13;
+UPDATE mai2_profile_ghost SET version = version - 13 WHERE version >= 13;
+
+DELETE FROM mai2_profile_rating WHERE version < 13;
+UPDATE mai2_profile_rating SET version = version - 13 WHERE version >= 13;
+
+DROP TABLE maimai_score_best;
+DROP TABLE maimai_playlog;
+DROP TABLE maimai_profile_detail;
+DROP TABLE maimai_profile_option;
+DROP TABLE maimai_profile_web_option;
+DROP TABLE maimai_profile_grade_status;
+
+ALTER TABLE mai2_item_character DROP COLUMN point;
+
+ALTER TABLE mai2_item_card MODIFY COLUMN cardId int(11) NOT NULL;
+ALTER TABLE mai2_item_card MODIFY COLUMN cardTypeId int(11) NOT NULL;
+ALTER TABLE mai2_item_card MODIFY COLUMN charaId int(11) NOT NULL;
+ALTER TABLE mai2_item_card MODIFY COLUMN mapId int(11) NOT NULL;
+
+ALTER TABLE mai2_item_character MODIFY COLUMN characterId int(11) NOT NULL;
+ALTER TABLE mai2_item_character MODIFY COLUMN level int(11) NOT NULL;
+ALTER TABLE mai2_item_character MODIFY COLUMN awakening int(11) NOT NULL;
+ALTER TABLE mai2_item_character MODIFY COLUMN useCount int(11) NOT NULL;
+
+ALTER TABLE mai2_item_charge MODIFY COLUMN chargeId int(11) NOT NULL;
+ALTER TABLE mai2_item_charge MODIFY COLUMN stock int(11) NOT NULL;
+
+ALTER TABLE mai2_item_favorite MODIFY COLUMN itemKind int(11) NOT NULL;
+
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN seasonId int(11) NOT NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN point int(11) NOT NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN `rank` int(11) NOT NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN rewardGet tinyint(1) NOT NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN userName varchar(8) NOT NULL;
+
+ALTER TABLE mai2_item_item MODIFY COLUMN itemId int(11) NOT NULL;
+ALTER TABLE mai2_item_item MODIFY COLUMN itemKind int(11) NOT NULL;
+ALTER TABLE mai2_item_item MODIFY COLUMN stock int(11) NOT NULL;
+ALTER TABLE mai2_item_item MODIFY COLUMN isValid tinyint(1) NOT NULL;
+
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN bonusId int(11) NOT NULL;
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN point int(11) NOT NULL;
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN isCurrent tinyint(1) NOT NULL;
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN isComplete tinyint(1) NOT NULL;
+
+ALTER TABLE mai2_item_map MODIFY COLUMN mapId int(11) NOT NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN distance int(11) NOT NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN isLock tinyint(1) NOT NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN isClear tinyint(1) NOT NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN isComplete tinyint(1) NOT NULL;
+
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN printDate timestamp DEFAULT current_timestamp() NOT NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN serialId varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN placeId int(11) NOT NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN clientId varchar(11) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN printerSerialId varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL;

core/data/schema/versions/SDEZ_5_upgrade.sql

@@ -0,0 +1,3 @@
+ALTER TABLE mai2_item_card 
+    CHANGE COLUMN startDate startDate TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    CHANGE COLUMN endDate endDate TIMESTAMP NOT NULL;

core/data/schema/versions/SDEZ_6_rollback.sql

@@ -0,0 +1 @@
+DROP TABLE aime.mai2_profile_consec_logins;

core/data/schema/versions/SDEZ_6_upgrade.sql

@@ -0,0 +1,62 @@
+UPDATE mai2_static_event SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_static_music SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_static_ticket SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_static_cards SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_profile_detail  SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_profile_extend  SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_profile_option  SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_profile_ghost  SET version = version + 13 WHERE version < 1000;
+
+UPDATE mai2_profile_rating  SET version = version + 13 WHERE version < 1000;
+
+ALTER TABLE mai2_item_character ADD point int(11) NULL;
+
+ALTER TABLE mai2_item_card MODIFY COLUMN cardId int(11) NULL;
+ALTER TABLE mai2_item_card MODIFY COLUMN cardTypeId int(11) NULL;
+ALTER TABLE mai2_item_card MODIFY COLUMN charaId int(11) NULL;
+ALTER TABLE mai2_item_card MODIFY COLUMN mapId int(11) NULL;
+
+ALTER TABLE mai2_item_character MODIFY COLUMN characterId int(11) NULL;
+ALTER TABLE mai2_item_character MODIFY COLUMN level int(11) NULL;
+ALTER TABLE mai2_item_character MODIFY COLUMN awakening int(11) NULL;
+ALTER TABLE mai2_item_character MODIFY COLUMN useCount int(11) NULL;
+
+ALTER TABLE mai2_item_charge MODIFY COLUMN chargeId int(11) NULL;
+ALTER TABLE mai2_item_charge MODIFY COLUMN stock int(11) NULL;
+
+ALTER TABLE mai2_item_favorite MODIFY COLUMN itemKind int(11) NULL;
+
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN seasonId int(11) NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN point int(11) NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN `rank` int(11) NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN rewardGet tinyint(1) NULL;
+ALTER TABLE mai2_item_friend_season_ranking MODIFY COLUMN userName varchar(8) NULL;
+
+ALTER TABLE mai2_item_item MODIFY COLUMN itemId int(11) NULL;
+ALTER TABLE mai2_item_item MODIFY COLUMN itemKind int(11) NULL;
+ALTER TABLE mai2_item_item MODIFY COLUMN stock int(11) NULL;
+ALTER TABLE mai2_item_item MODIFY COLUMN isValid tinyint(1) NULL;
+
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN bonusId int(11) NULL;
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN point int(11) NULL;
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN isCurrent tinyint(1) NULL;
+ALTER TABLE mai2_item_login_bonus MODIFY COLUMN isComplete tinyint(1) NULL;
+
+ALTER TABLE mai2_item_map MODIFY COLUMN mapId int(11) NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN distance int(11) NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN isLock tinyint(1) NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN isClear tinyint(1) NULL;
+ALTER TABLE mai2_item_map MODIFY COLUMN isComplete tinyint(1) NULL;
+
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN printDate timestamp DEFAULT current_timestamp() NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN serialId varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN placeId int(11) NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN clientId varchar(11) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL;
+ALTER TABLE mai2_item_print_detail MODIFY COLUMN printerSerialId varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL;

core/data/schema/versions/SDEZ_7_rollback.sql

@@ -0,0 +1,10 @@
+ALTER TABLE mai2_profile_detail
+    DROP COLUMN mapStock;
+
+ALTER TABLE mai2_profile_extend
+    DROP COLUMN selectResultScoreViewType;
+
+ALTER TABLE mai2_profile_option
+    DROP COLUMN outFrameType,
+    DROP COLUMN touchVolume,
+    DROP COLUMN breakSlideVolume;

core/data/schema/versions/SDEZ_7_upgrade.sql

@@ -0,0 +1,9 @@
+CREATE TABLE `mai2_profile_consec_logins` (
+  `id` int(11) NOT NULL AUTO_INCREMENT,
+  `user` int(11) NOT NULL,
+  `version` int(11) NOT NULL,
+  `logins` int(11) DEFAULT NULL,
+  PRIMARY KEY (`id`),
+  UNIQUE KEY `mai2_profile_consec_logins_uk` (`user`,`version`),
+  CONSTRAINT `mai2_profile_consec_logins_ibfk_1` FOREIGN KEY (`user`) REFERENCES `aime_user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

core/data/schema/versions/SDEZ_8_upgrade.sql

@@ -0,0 +1,10 @@
+ALTER TABLE mai2_profile_detail
+    ADD mapStock INT NULL AFTER playCount;
+
+ALTER TABLE mai2_profile_extend
+    ADD selectResultScoreViewType INT NULL AFTER selectResultDetails;
+
+ALTER TABLE mai2_profile_option
+    ADD outFrameType INT NULL AFTER dispCenter,
+    ADD touchVolume INT NULL AFTER slideVolume,
+    ADD breakSlideVolume INT NULL AFTER slideVolume;

core/data/schema/versions/SDFE_3_rollback.sql

@@ -0,0 +1,2 @@
+SET FOREIGN_KEY_CHECKS=0;
+SET FOREIGN_KEY_CHECKS=1;

core/data/schema/versions/SDFE_4_rollback.sql

@@ -0,0 +1 @@
+ALTER TABLE wacca_profile DROP COLUMN playcount_time_free;

core/data/schema/versions/SDFE_4_upgrade.sql

@@ -0,0 +1 @@
+DELETE FROM wacca_item WHERE type=17 AND item_id=312002;

core/data/schema/versions/SDFE_5_upgrade.sql

@@ -0,0 +1 @@
+ALTER TABLE wacca_profile ADD playcount_time_free int(11) DEFAULT 0 NULL AFTER playcount_stageup;

core/frontend.py

@@ -1,49 +1,63 @@
 import logging, coloredlogs
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Union, Optional
-from twisted.web import resource
+from starlette.requests import Request
-from twisted.web.util import redirectTo
+from starlette.routing import Route, Mount
-from twisted.web.http import Request
+from starlette.responses import Response, PlainTextResponse, RedirectResponse
+from starlette.applications import Starlette
 from logging.handlers import TimedRotatingFileHandler
-from twisted.web.server import Session
-from zope.interface import Interface, Attribute, implementer
-from twisted.python.components import registerAdapter
 import jinja2
 import bcrypt
+import re
+import jwt
+import yaml
+from base64 import b64decode
+from enum import Enum
+from datetime import datetime, timezone
+from os import path, environ, mkdir, W_OK, access
 
 from core import CoreConfig, Utils
 from core.data import Data
 
+class PermissionOffset(Enum):
+    USER = 0 # Regular user
+    USERMOD = 1 # Can moderate other users
+    ACMOD = 2 # Can add arcades and cabs
+    SYSADMIN = 3 # Can change settings
+    # 4 - 6 reserved for future use
+    OWNER = 7 # Can do anything
 
-class IUserSession(Interface):
+class ShopPermissionOffset(Enum):
-    userId = Attribute("User's ID")
+    VIEW = 0 # View info and cabs
-    current_ip = Attribute("User's current ip address")
+    BOOKKEEP = 1 # View bookeeping info
-    permissions = Attribute("User's permission level")
+    EDITOR = 2 # Can edit name, settings
+    REGISTRAR = 3 # Can add cabs
+    # 4 - 6 reserved for future use
+    OWNER = 7 # Can do anything
 
+class ShopOwner():
+    def __init__(self, usr_id: int = 0, usr_name: str = "", perms: int = 0) -> None:
+        self.user_id = usr_id
+        self.username = usr_name
+        self.permissions = perms
 
-@implementer(IUserSession)
+class UserSession():
-class UserSession(object):
+    def __init__(self, usr_id: int = 0, ip: str = "", perms: int = 0, ongeki_ver: int = 7):
-    def __init__(self, session):
+        self.user_id = usr_id
-        self.userId = 0
+        self.current_ip = ip
-        self.current_ip = "0.0.0.0"
+        self.permissions = perms
-        self.permissions = 0
+        self.ongeki_version = ongeki_ver
-
-
-class FrontendServlet(resource.Resource):
-    def getChild(self, name: bytes, request: Request):
-        self.logger.debug(f"{Utils.get_ip_addr(request)} -> {name.decode()}")
-        if name == b"":
-            return self
-        return resource.Resource.getChild(self, name, request)
 
+class FrontendServlet():
     def __init__(self, cfg: CoreConfig, config_dir: str) -> None:
         self.config = cfg
         log_fmt_str = "[%(asctime)s] Frontend | %(levelname)s | %(message)s"
         log_fmt = logging.Formatter(log_fmt_str)
-        self.logger = logging.getLogger("frontend")
         self.environment = jinja2.Environment(loader=jinja2.FileSystemLoader("."))
-        self.game_list: List[Dict[str, str]] = []
+        self.game_list: Dict[str, Dict[str, Any]] = {}
-        self.children: Dict[str, Any] = {}
+        self.sn_cvt: Dict[str, str] = {}
         
+        self.logger = logging.getLogger("frontend")
+        if not hasattr(self.logger, "inited"):
             fileHandler = TimedRotatingFileHandler(
                 "{0}/{1}.log".format(self.config.server.log_dir, "frontend"),
                 when="d",
@@ -61,183 +75,671 @@ class FrontendServlet(resource.Resource):
             coloredlogs.install(
                 level=cfg.frontend.loglevel, logger=self.logger, fmt=log_fmt_str
             )
-        registerAdapter(UserSession, Session, IUserSession)
 
-        fe_game = FE_Game(cfg, self.environment)
+            self.logger.inited = True
+        
         games = Utils.get_all_titles()
         for game_dir, game_mod in games.items():
-            if hasattr(game_mod, "frontend"):
+            if hasattr(game_mod, "frontend") and hasattr(game_mod, "index") and hasattr(game_mod, "game_codes"):
                 try:
+                    if game_mod.index.is_game_enabled(game_mod.game_codes[0], self.config, config_dir):
                         game_fe = game_mod.frontend(cfg, self.environment, config_dir)
-                    self.game_list.append({"url": game_dir, "name": game_fe.nav_name})
+                        self.game_list[game_fe.nav_name] = {"url": f"/{game_dir}", "class": game_fe }
-                    fe_game.putChild(game_dir.encode(), game_fe)
-                except:
-                    raise
                         
-        self.environment.globals["game_list"] = self.game_list
+                        if hasattr(game_fe, "SN_PREFIX") and hasattr(game_fe, "NETID_PREFIX"):
-        self.putChild(b"gate", FE_Gate(cfg, self.environment))
+                            if len(game_fe.SN_PREFIX) == len(game_fe.NETID_PREFIX):
-        self.putChild(b"user", FE_User(cfg, self.environment))
+                                for x in range(len(game_fe.SN_PREFIX)):
-        self.putChild(b"game", fe_game)
+                                    self.sn_cvt[game_fe.SN_PREFIX[x]] = game_fe.NETID_PREFIX[x]
 
-        self.logger.info(
+                except Exception as e:
-            f"Ready on port {self.config.frontend.port} serving {len(fe_game.children)} games"
+                    self.logger.error(
+                        f"Failed to import frontend from {game_dir} because {e}"
                     )
 
-    def render_GET(self, request):
-        self.logger.debug(f"{Utils.get_ip_addr(request)} -> {request.uri.decode()}")
-        template = self.environment.get_template("core/frontend/index.jinja")
-        return template.render(
-            server_name=self.config.server.name,
-            title=self.config.server.name,
-            game_list=self.game_list,
-            sesh=vars(IUserSession(request.getSession())),
-        ).encode("utf-16")
 
+        self.environment.globals["game_list"] = self.game_list
+        self.environment.globals["sn_cvt"] = self.sn_cvt
+        self.base = FE_Base(cfg, self.environment)
+        self.gate = FE_Gate(cfg, self.environment)
+        self.user = FE_User(cfg, self.environment)
+        self.system = FE_System(cfg, self.environment)
+        self.arcade = FE_Arcade(cfg, self.environment)
+        self.machine = FE_Machine(cfg, self.environment)
     
-class FE_Base(resource.Resource):
+    def get_routes(self) -> List[Route]:        
+        g_routes = []
+        for nav_name, g_data in self.environment.globals["game_list"].items():
+            g_routes.append(Mount(g_data['url'], routes=g_data['class'].get_routes()))
+        return [
+            Route("/", self.base.render_GET, methods=['GET']),
+            Mount("/user", routes=[
+                Route("/", self.user.render_GET, methods=['GET']),
+                Route("/{user_id:int}", self.user.render_GET, methods=['GET']),
+                Route("/update.pw", self.user.render_POST, methods=['POST']),
+                Route("/update.name", self.user.update_username, methods=['POST']),
+                Route("/edit.card", self.user.edit_card, methods=['POST']),
+                Route("/add.card", self.user.add_card, methods=['POST']),
+                Route("/logout", self.user.render_logout, methods=['GET']),
+            ]),
+            Mount("/gate", routes=[
+                Route("/", self.gate.render_GET, methods=['GET', 'POST']),
+                Route("/gate.login", self.gate.render_login, methods=['POST']),
+                Route("/gate.create", self.gate.render_create, methods=['POST']),
+                Route("/create", self.gate.render_create_get, methods=['GET']),
+            ]),
+            Mount("/sys", routes=[
+                Route("/", self.system.render_GET, methods=['GET']),
+                Route("/lookup.user", self.system.lookup_user, methods=['GET']),
+                Route("/lookup.shop", self.system.lookup_shop, methods=['GET']),
+            ]),
+            Mount("/shop", routes=[
+                Route("/", self.arcade.render_GET, methods=['GET']),
+                Route("/{shop_id:int}", self.arcade.render_GET, methods=['GET']),
+            ]),
+            Mount("/cab", routes=[
+                Route("/", self.machine.render_GET, methods=['GET']),
+                Route("/{machine_id:int}", self.machine.render_GET, methods=['GET']),
+            ]),
+            Mount("/game", routes=g_routes),            
+            Route("/robots.txt", self.robots)
+        ]
+    
+    def startup(self) -> None:
+        self.config.update({
+            "frontend": {
+                "standalone": True, 
+                "loglevel": CoreConfig.loglevel_to_str(self.config.frontend.loglevel), 
+                "secret": self.config.frontend.secret
+            }
+        })
+        self.logger.info(f"Serving {len(self.game_list)} games")
+    
+    @classmethod    
+    async def robots(cls, request: Request) -> PlainTextResponse:
+        return PlainTextResponse("User-agent: *\nDisallow: /\n\nUser-agent: AdsBot-Google\nDisallow: /")
+
+class FE_Base():
     """
     A Generic skeleton class that all frontend handlers should inherit from
     Initializes the environment, data, logger, config, and sets isLeaf to true
     It is expected that game implementations of this class overwrite many of these
     """
-
-    isLeaf = True
-
     def __init__(self, cfg: CoreConfig, environment: jinja2.Environment) -> None:
         self.core_config = cfg
         self.data = Data(cfg)
         self.logger = logging.getLogger("frontend")
         self.environment = environment
-        self.nav_name = "nav_name"
+        self.nav_name = "index"
     
+    async def render_GET(self, request: Request):
+        self.logger.debug(f"{Utils.get_ip_addr(request)} -> {request.url}")
+        template = self.environment.get_template("core/templates/index.jinja")
+        sesh = self.validate_session(request)
+        resp = Response(template.render(
+            server_name=self.core_config.server.name,
+            title=self.core_config.server.name,
+            game_list=self.environment.globals["game_list"],
+            sesh=vars(sesh) if sesh is not None else vars(UserSession()),
+        ), media_type="text/html; charset=utf-8")
+        
+        if sesh is None:
+            resp.delete_cookie("DIANA_SESH")
+        return resp
+    
+    def get_routes(self) -> List[Route]:
+        return []
+    
+    @classmethod
+    def test_perm(cls, permission: int, offset: Union[PermissionOffset, ShopPermissionOffset]) -> bool:
+        logging.getLogger('frontend').debug(f"{permission} vs {1 << offset.value}")
+        return permission & 1 << offset.value == 1 << offset.value
+    
+    @classmethod
+    def test_perm_minimum(cls, permission: int, offset: Union[PermissionOffset, ShopPermissionOffset]) -> bool:
+        return permission >= 1 << offset.value
+    
+    def decode_session(self, token: str) -> UserSession:
+        sesh = UserSession()
+        if not token: return sesh
+        try:
+            tk = jwt.decode(token, b64decode(self.core_config.frontend.secret), options={"verify_signature": True}, algorithms=["HS256"])
+            sesh.user_id = tk['user_id']
+            sesh.current_ip = tk['current_ip']
+            sesh.permissions = tk['permissions']
+            
+            if sesh.user_id <= 0:
+                self.logger.error("User session failed to validate due to an invalid ID!")
+                return UserSession()
+            return sesh
+        except jwt.ExpiredSignatureError:
+            self.logger.error("User session failed to validate due to an expired signature!")
+            return sesh
+        except jwt.InvalidSignatureError:
+            self.logger.error("User session failed to validate due to an invalid signature!")
+            return sesh
+        except jwt.DecodeError as e:
+            self.logger.error(f"User session failed to decode! {e}")
+            return sesh
+        except jwt.InvalidTokenError as e:
+            self.logger.error(f"User session is invalid! {e}")
+            return sesh
+        except KeyError as e:
+            self.logger.error(f"{e} missing from User session!")
+            return UserSession()
+        except Exception as e:
+            self.logger.error(f"Unknown exception occoured when decoding User session! {e}")
+            return UserSession()
+    
+    def validate_session(self, request: Request) -> Optional[UserSession]:
+        sesh = request.cookies.get('DIANA_SESH', "")
+        if not sesh:
+            return None
+        
+        usr_sesh = self.decode_session(sesh)
+        req_ip = Utils.get_ip_addr(request)
+        
+        if usr_sesh.current_ip != req_ip:
+            self.logger.error(f"User session failed to validate due to mismatched IPs! {usr_sesh.current_ip} -> {req_ip}")
+            return None
+
+        if usr_sesh.permissions <= 0 or usr_sesh.permissions > 255:
+            self.logger.error(f"User session failed to validate due to an invalid permission value! {usr_sesh.permissions}")
+            return None
+        
+        return usr_sesh
+    
+    def encode_session(self, sesh: UserSession, exp_seconds: int = 86400) -> str:
+        try:
+            return jwt.encode({ "user_id": sesh.user_id, "current_ip": sesh.current_ip, "permissions": sesh.permissions, "ongeki_version": sesh.ongeki_version, "exp": int(datetime.now(tz=timezone.utc).timestamp()) + exp_seconds }, b64decode(self.core_config.frontend.secret), algorithm="HS256")
+        except jwt.InvalidKeyError:
+            self.logger.error("Failed to encode User session because the secret is invalid!")
+            return ""
+        except Exception as e:
+            self.logger.error(f"Unknown exception occoured when encoding User session! {e}")
+            return ""
 
 class FE_Gate(FE_Base):
-    def render_GET(self, request: Request):
+    async def render_GET(self, request: Request):
-        self.logger.debug(f"{Utils.get_ip_addr(request)} -> {request.uri.decode()}")
+        self.logger.debug(f"{Utils.get_ip_addr(request)} -> {request.url.path}")
-        uri: str = request.uri.decode()
 
-        sesh = request.getSession()
+        usr_sesh = self.validate_session(request)
-        usr_sesh = IUserSession(sesh)
+        if usr_sesh and usr_sesh.user_id > 0:
-        if usr_sesh.userId > 0:
+            return RedirectResponse("/user/", 303)
-            return redirectTo(b"/user", request)
         
-        if uri.startswith("/gate/create"):
-            return self.create_user(request)
         
-        if b"e" in request.args:
+        if "e" in request.query_params:
             try:
-                err = int(request.args[b"e"][0].decode())
+                err = int(request.query_params.get("e", ["0"])[0])
-            except:
+            except Exception:
                 err = 0
 
         else:
             err = 0
 
-        template = self.environment.get_template("core/frontend/gate/gate.jinja")
+        template = self.environment.get_template("core/templates/gate/gate.jinja")
-        return template.render(
+        resp = Response(template.render(
             title=f"{self.core_config.server.name} | Login Gate",
             error=err,
-            sesh=vars(usr_sesh),
+            sesh=vars(UserSession()),
-        ).encode("utf-16")
+        ), media_type="text/html; charset=utf-8")
+        resp.delete_cookie("DIANA_SESH")
+        return resp
     
-    def render_POST(self, request: Request):
+    async def render_login(self, request: Request):
-        uri = request.uri.decode()
         ip = Utils.get_ip_addr(request)
+        frm = await request.form()
+        access_code: str = frm.get("access_code", None)
+        if not access_code:
+            return RedirectResponse("/gate/?e=1", 303)
         
-        if uri == "/gate/gate.login":
+        passwd: bytes = frm.get("passwd", "").encode()
-            access_code: str = request.args[b"access_code"][0].decode()
-            passwd: bytes = request.args[b"passwd"][0]
         if passwd == b"":
             passwd = None
 
-            uid = self.data.card.get_user_id_from_card(access_code)
+        uid = await self.data.card.get_user_id_from_card(access_code)
         if uid is None:
-                return redirectTo(b"/gate?e=1", request)
+            self.logger.debug(f"Failed to find user for card {access_code}")
+            return RedirectResponse("/gate/?e=1", 303)
+
+        user = await self.data.user.get_user(uid)
+        if user is None:
+            self.logger.error(f"Failed to load user {uid}")
+            return RedirectResponse("/gate/?e=1", 303)
 
         if passwd is None:
-                sesh = self.data.user.check_password(uid)
+            sesh = await self.data.user.check_password(uid)
 
             if sesh is not None:
-                    return redirectTo(
+                return RedirectResponse(f"/gate/create?ac={access_code}", 303)
-                        f"/gate/create?ac={access_code}".encode(), request
-                    )
-                return redirectTo(b"/gate?e=1", request)
             
-            if not self.data.user.check_password(uid, passwd):
+            return RedirectResponse("/gate/?e=1", 303)
-                return redirectTo(b"/gate?e=1", request)
+
+        if not await self.data.user.check_password(uid, passwd):
+            self.logger.debug(f"Failed password for access code {access_code}")
+            return RedirectResponse("/gate/?e=1", 303)
 
         self.logger.info(f"Successful login of user {uid} at {ip}")
 
-            sesh = request.getSession()
+        sesh = UserSession()
-            usr_sesh = IUserSession(sesh)
+        sesh.user_id = uid
-            usr_sesh.userId = uid
+        sesh.current_ip = ip
-            usr_sesh.current_ip = ip
+        sesh.permissions = user['permissions']
         
-            return redirectTo(b"/user", request)
+        usr_sesh = self.encode_session(sesh)
+        self.logger.debug(f"Created session with JWT {usr_sesh}")
+        resp = RedirectResponse("/user/", 303)
+        resp.set_cookie("DIANA_SESH", usr_sesh)
 
-        elif uri == "/gate/gate.create":
+        return resp
-            access_code: str = request.args[b"access_code"][0].decode()
-            username: str = request.args[b"username"][0]
-            email: str = request.args[b"email"][0].decode()
-            passwd: str = request.args[b"passwd"][0]
 
-            uid = self.data.card.get_user_id_from_card(access_code)
+    async def render_create(self, request: Request):
+        ip = Utils.get_ip_addr(request)
+        frm = await request.form()
+        access_code: str = frm.get("access_code", "")
+        username: str = frm.get("username", "")
+        email: str = frm.get("email", "")
+        passwd: bytes = frm.get("passwd", "").encode()
+
+        if not access_code or not username or not email or not passwd:
+            return RedirectResponse("/gate/?e=1", 303)
+
+        uid = await self.data.card.get_user_id_from_card(access_code)
         if uid is None:
-                return redirectTo(b"/gate?e=1", request)
+            return RedirectResponse("/gate/?e=1", 303)
 
         salt = bcrypt.gensalt()
         hashed = bcrypt.hashpw(passwd, salt)
 
-            result = self.data.user.create_user(
+        result = await self.data.user.create_user(
-                uid, username, email, hashed.decode(), 1
+            uid, username, email.lower(), hashed.decode(), 1
         )
         if result is None:
-                return redirectTo(b"/gate?e=3", request)
+            return RedirectResponse("/gate/?e=3", 303)
 
-            if not self.data.user.check_password(uid, passwd.encode()):
+        if not await self.data.user.check_password(uid, passwd):
-                return redirectTo(b"/gate", request)
+            return RedirectResponse("/gate/", 303)
         
-            return redirectTo(b"/user", request)
+        sesh = UserSession()
+        sesh.user_id = uid
+        sesh.current_ip = ip
+        sesh.permissions = 1
         
-        else:
+        usr_sesh = self.encode_session(sesh)
-            return b""
+        self.logger.debug(f"Created session with JWT {usr_sesh}")
+        resp = RedirectResponse("/user/", 303)
+        resp.set_cookie("DIANA_SESH", usr_sesh)
 
-    def create_user(self, request: Request):
+        return resp
-        if b"ac" not in request.args or len(request.args[b"ac"][0].decode()) != 20:
-            return redirectTo(b"/gate?e=2", request)
 
-        ac = request.args[b"ac"][0].decode()
+    async def render_create_get(self, request: Request):
+        ac = request.query_params.get("ac", "")
+        if len(ac) != 20:
+            return RedirectResponse("/gate/?e=2", 303)
 
-        template = self.environment.get_template("core/frontend/gate/create.jinja")
+        card = await self.data.card.get_card_by_access_code(ac)
-        return template.render(
+        if card is None:
+            return RedirectResponse("/gate/?e=1", 303)
+        
+        user = await self.data.user.get_user(card['user'])
+        if user is None:
+            self.logger.warning(f"Card {ac} exists with no/invalid associated user ID {card['user']}")
+            return RedirectResponse("/gate/?e=0", 303)
+
+        if user['password'] is not None:
+            return RedirectResponse("/gate/?e=1", 303)
+
+        template = self.environment.get_template("core/templates/gate/create.jinja")
+        return Response(template.render(
             title=f"{self.core_config.server.name} | Create User",
             code=ac,
-            sesh={"userId": 0},
+            sesh={"user_id": 0, "permissions": 0},
-        ).encode("utf-16")
+        ), media_type="text/html; charset=utf-8")
-
 
 class FE_User(FE_Base):
-    def render_GET(self, request: Request):
+    async def render_GET(self, request: Request):
-        template = self.environment.get_template("core/frontend/user/index.jinja")
+        uri = request.url.path
+        user_id = request.path_params.get('user_id', None)
+        self.logger.debug(f"{Utils.get_ip_addr(request)} -> {uri}")
+        template = self.environment.get_template("core/templates/user/index.jinja")
 
-        sesh: Session = request.getSession()
+        usr_sesh = self.validate_session(request)
-        usr_sesh = IUserSession(sesh)
+        if not usr_sesh:
-        if usr_sesh.userId == 0:
+            return RedirectResponse("/gate/", 303)
-            return redirectTo(b"/gate", request)
         
-        return template.render(
+        if user_id:
-            title=f"{self.core_config.server.name} | Account", sesh=vars(usr_sesh)
+            if not self.test_perm(usr_sesh.permissions, PermissionOffset.USERMOD) and user_id != usr_sesh.user_id:
-        ).encode("utf-16")
+                self.logger.warn(f"User {usr_sesh.user_id} does not have permission to view user {user_id}")
+                return RedirectResponse("/user/", 303)
+        
+        else:
+            user_id = usr_sesh.user_id
+        
+        user = await self.data.user.get_user(user_id)
+        if user is None:
+            self.logger.debug(f"User {user_id} not found")
+            return RedirectResponse("/user/", 303)
+        
+        cards = await self.data.card.get_user_cards(user_id)
+        
+        card_data = []
+        arcade_data = []
+
+        for c in cards:
+            if c['is_locked']:
+                status = 'Locked'
+            elif c['is_banned']:
+                status = 'Banned'
+            else:
+                status = 'Active'
+            
+            #idm = c['idm']
+            ac = c['access_code']
+
+            if ac.startswith("5"): #or idm is not None:
+                c_type = "AmusementIC"
+            elif ac.startswith("3"):
+                c_type = "Banapass"
+            elif ac.startswith("010"):                
+                c_type = "Aime" # TODO: Aime verification
+            elif ac.startswith("0008"):
+                c_type = "Generated AIC"
+            else:
+                c_type = "Unknown"
+            
+            card_data.append({
+                'access_code': ac, 
+                'status': status, 
+                'chip_id': "", #None if c['chip_id'] is None else f"{c['chip_id']:X}", 
+                'idm': "", 
+                'type': c_type, 
+                "memo": ""
+            })
+
+        if "e" in request.query_params:
+            try:
+                err = int(request.query_params.get("e", 0))
+            except Exception:
+                err = 0
+
+        else:
+            err = 0
+        
+        if "s" in request.query_params:
+            try:
+                succ = int(request.query_params.get("s", 0))
+            except Exception:
+                succ = 0
+
+        else:
+            succ = 0
+
+        return Response(template.render(
+            title=f"{self.core_config.server.name} | Account", 
+            sesh=vars(usr_sesh), 
+            cards=card_data,
+            error=err,
+            success=succ,
+            username=user['username'],
+            arcades=arcade_data
+        ), media_type="text/html; charset=utf-8")
+    
+    async def render_logout(self, request: Request):
+        resp = RedirectResponse("/gate/", 303)
+        resp.delete_cookie("DIANA_SESH")
+        return resp
+    
+    async def edit_card(self, request: Request) -> RedirectResponse:
+        return RedirectResponse("/user/", 303)
+    
+    async def add_card(self, request: Request) -> RedirectResponse:
+        return RedirectResponse("/user/", 303)
+
+    async def render_POST(self, request: Request):
+        frm = await request.form()
+        usr_sesh = self.validate_session(request)
+        if not usr_sesh or not self.test_perm(usr_sesh.permissions, PermissionOffset.USERMOD):
+            return RedirectResponse("/gate/", 303)
+        
+        old_pw: str = frm.get('current_pw', None)
+        pw1: str = frm.get('password1', None)
+        pw2: str = frm.get('password2', None)
+
+        if old_pw is None or pw1 is None or pw2 is None:
+            return RedirectResponse("/user/?e=4", 303)
+        
+        if pw1 != pw2:
+            return RedirectResponse("/user/?e=6", 303)
+        
+        if not await self.data.user.check_password(usr_sesh.user_id, old_pw.encode()):
+            return RedirectResponse("/user/?e=5", 303)
+        
+        if len(pw1) < 10 or not any(ele.isupper() for ele in pw1) or not any(ele.islower() for ele in pw1) \
+            or not any(ele.isdigit() for ele in pw1) or not any(not ele.isalnum() for ele in pw1):
+            return RedirectResponse("/user/?e=7", 303)
+        
+        salt = bcrypt.gensalt()
+        hashed = bcrypt.hashpw(pw1.encode(), salt)
+        if not await self.data.user.change_password(usr_sesh.user_id, hashed.decode()):
+            return RedirectResponse("/gate/?e=1", 303)
+        
+        return RedirectResponse("/user/?s=1", 303)
+        
+    async def update_username(self, request: Request):
+        frm = await request.form()
+        new_name: bytes = frm.get('new_name', "")
+        usr_sesh = self.validate_session(request)
+        if not usr_sesh or not self.test_perm(usr_sesh.permissions, PermissionOffset.USERMOD):
+            return RedirectResponse("/gate/", 303)
+        
+        if new_name is None or not new_name:
+            return RedirectResponse("/user/?e=4", 303)
+        
+        if len(new_name) > 10:
+            return RedirectResponse("/user/?e=8", 303)
+
+        if not await self.data.user.change_username(usr_sesh.user_id, new_name):
+            return RedirectResponse("/user/?e=8", 303)
+        
+        return RedirectResponse("/user/?s=2", 303)
+
+class FE_System(FE_Base):
+    async def render_GET(self, request: Request):
+        template = self.environment.get_template("core/templates/sys/index.jinja")
+        self.logger.debug(f"{Utils.get_ip_addr(request)} -> {request.url.path}")
+
+        usr_sesh = self.validate_session(request)
+        if not usr_sesh or not self.test_perm_minimum(usr_sesh.permissions, PermissionOffset.USERMOD):
+            return RedirectResponse("/gate/", 303)
+        
+        return Response(template.render(
+            title=f"{self.core_config.server.name} | System", 
+            sesh=vars(usr_sesh), 
+            usrlist=[],
+        ), media_type="text/html; charset=utf-8")
+        
+    async def lookup_user(self, request: Request):
+        template = self.environment.get_template("core/templates/sys/index.jinja")
+        usrlist: List[Dict] = []
+        usr_sesh = self.validate_session(request)
+        if not usr_sesh or not self.test_perm(usr_sesh.permissions, PermissionOffset.USERMOD):
+            return RedirectResponse("/gate/", 303)
+        
+        uid_search = request.query_params.get("usrId",  None)
+        email_search = request.query_params.get("usrEmail", None)
+        uname_search = request.query_params.get("usrName", None)
+
+        if uid_search:
+            u = await self.data.user.get_user(uid_search)
+            if u is not None:
+                usrlist.append(u._asdict())
+
+        elif email_search:
+            u = await self.data.user.find_user_by_email(email_search)
+            if u is not None:
+                usrlist.append(u._asdict())
+
+        elif uname_search:
+            ul = await self.data.user.find_user_by_username(uname_search)
+            for u in ul:
+                usrlist.append(u._asdict())
+
+        return Response(template.render(
+            title=f"{self.core_config.server.name} | System", 
+            sesh=vars(usr_sesh), 
+            usrlist=usrlist,
+            shoplist=[],
+        ), media_type="text/html; charset=utf-8")
+
+    async def lookup_shop(self, request: Request):
+        shoplist = []
+        template = self.environment.get_template("core/templates/sys/index.jinja")
+        
+        usr_sesh = self.validate_session(request)
+        if not usr_sesh or not self.test_perm(usr_sesh.permissions, PermissionOffset.ACMOD):
+            return RedirectResponse("/gate/", 303)
+        
+        shopid_search = request.query_params.get("shopId",  None)
+        sn_search = request.query_params.get("serialNum", None)
+        
+        if shopid_search:
+            if shopid_search.isdigit():
+                shopid_search = int(shopid_search)
+                try:
+                    sinfo = await self.data.arcade.get_arcade(shopid_search)
+                except Exception as e:
+                    self.logger.error(f"Failed to fetch shop info for shop {shopid_search} in lookup_shop - {e}")
+                    sinfo = None
+                if sinfo:
+                    shoplist.append({
+                        "name": sinfo['name'],
+                        "id": sinfo['id']
+                    })
+            
+            else:
+                return Response(template.render(
+                    title=f"{self.core_config.server.name} | System", 
+                    sesh=vars(usr_sesh),
+                    usrlist=[],
+                    shoplist=shoplist,
+                    error=4
+                ), media_type="text/html; charset=utf-8")
+        
+        if sn_search:
+            sn_search = sn_search.upper().replace("-", "").strip()
+            if sn_search.isdigit() and len(sn_search) == 12:
+                prefix = sn_search[:4]
+                suffix = sn_search[5:]
+                
+                netid_prefix = self.environment.globals["sn_cvt"].get(prefix, "")                
+                sn_search = netid_prefix + suffix
+            
+            if re.match(r"^AB[DGL]N\d{7}$", sn_search) or re.match(r"^A\d{2}[EX]\d{2}[A-Z]\d{4,8}$", sn_search):
+                cabinfo = await self.data.arcade.get_machine(sn_search)
+                if cabinfo is None: sinfo = None
+                else:
+                    sinfo = await self.data.arcade.get_arcade(cabinfo['arcade'])
+                if sinfo:
+                    shoplist.append({
+                        "name": sinfo['name'],
+                        "id": sinfo['id']
+                    })
+            
+            else:
+                return Response(template.render(
+                    title=f"{self.core_config.server.name} | System", 
+                    sesh=vars(usr_sesh),
+                    usrlist=[],
+                    shoplist=shoplist,
+                    error=10
+                ), media_type="text/html; charset=utf-8")
             
         
-class FE_Game(FE_Base):
+        return Response(template.render(
-    isLeaf = False
+            title=f"{self.core_config.server.name} | System", 
-    children: Dict[str, Any] = {}
+            sesh=vars(usr_sesh), 
+            usrlist=[],
+            shoplist=shoplist,
+        ), media_type="text/html; charset=utf-8")
 
-    def getChild(self, name: bytes, request: Request):
+class FE_Arcade(FE_Base):
-        if name == b"":
+    async def render_GET(self, request: Request):
-            return self
+        template = self.environment.get_template("core/templates/arcade/index.jinja")
-        return resource.Resource.getChild(self, name, request)
+        shop_id = request.path_params.get('shop_id', None)
         
-    def render_GET(self, request: Request) -> bytes:
+        usr_sesh = self.validate_session(request)
-        return redirectTo(b"/user", request)
+        if not usr_sesh or not self.test_perm(usr_sesh.permissions, PermissionOffset.ACMOD):
+            self.logger.warn(f"User {usr_sesh.user_id} does not have permission to view shops!")
+            return RedirectResponse("/gate/", 303)
+        
+        if not shop_id:
+            return Response(template.render(
+                title=f"{self.core_config.server.name} | Arcade", 
+                sesh=vars(usr_sesh),
+            ), media_type="text/html; charset=utf-8")
+        
+        sinfo = await self.data.arcade.get_arcade(shop_id)
+        if not sinfo:
+            return Response(template.render(
+                title=f"{self.core_config.server.name} | Arcade", 
+                sesh=vars(usr_sesh),
+            ), media_type="text/html; charset=utf-8")
+        
+        cabs = await self.data.arcade.get_arcade_machines(shop_id)
+        cablst = []
+        if cabs:
+            for x in cabs:
+                cablst.append({
+                    "id": x['id'],
+                    "serial": x['serial'],
+                    "game": x['game'],
+                })
+        
+        return Response(template.render(
+            title=f"{self.core_config.server.name} | Arcade", 
+            sesh=vars(usr_sesh),
+            arcade={
+                "name": sinfo['name'],
+                "id": sinfo['id'],
+                "cabs": cablst
+            }
+            
+        ), media_type="text/html; charset=utf-8")
+
+class FE_Machine(FE_Base):
+    async def render_GET(self, request: Request):
+        template = self.environment.get_template("core/templates/machine/index.jinja")
+        cab_id = request.path_params.get('cab_id', None)
+        
+        usr_sesh = self.validate_session(request)
+        if not usr_sesh or not self.test_perm(usr_sesh.permissions, PermissionOffset.ACMOD):
+            self.logger.warn(f"User {usr_sesh.user_id} does not have permission to view shops!")
+            return RedirectResponse("/gate/", 303)
+        
+        if not cab_id:
+            return Response(template.render(
+                title=f"{self.core_config.server.name} | Machine", 
+                sesh=vars(usr_sesh),
+            ), media_type="text/html; charset=utf-8")
+        
+        return Response(template.render(
+            title=f"{self.core_config.server.name} | Machine", 
+            sesh=vars(usr_sesh),
+            arcade={}
+        ), media_type="text/html; charset=utf-8")
+
+cfg_dir = environ.get("DIANA_CFG_DIR", "config")
+cfg: CoreConfig = CoreConfig()
+if path.exists(f"{cfg_dir}/core.yaml"):
+    cfg.update(yaml.safe_load(open(f"{cfg_dir}/core.yaml")))
+
+if not path.exists(cfg.server.log_dir):
+    mkdir(cfg.server.log_dir)
+
+if not access(cfg.server.log_dir, W_OK):
+    print(
+        f"Log directory {cfg.server.log_dir} NOT writable, please check permissions"
+    )
+    exit(1)
+
+fe = FrontendServlet(cfg, cfg_dir)
+app = Starlette(cfg.server.is_develop, fe.get_routes(), on_startup=[fe.startup])

core/frontend/user/index.jinja

@@ -1,4 +0,0 @@
-{% extends "core/frontend/index.jinja" %}
-{% block content %}
-<h1>testing</h1>
-{% endblock content %}

core/frontend/widgets/err_banner.jinja

@@ -1,14 +0,0 @@
-{% if error > 0 %}
-<div class="err-banner">
-<h3>Error</h3>
-{% if error == 1 %}
-Card not registered, or wrong password
-{% elif error == 2 %}
-Missing or malformed access code
-{% elif error == 3 %}
-Failed to create user
-{% else %}
-An unknown error occoured
-{% endif %}
-</div>
-{% endif %}

core/frontend/widgets/topbar.jinja

@@ -1,18 +0,0 @@
-<div style="background: #333; color: #f9f9f9; width: 10%; height: 50px; line-height: 50px; text-align: center; float: left;">
-    Navigation
-</div>
-<div style="background: #333; color: #f9f9f9; width: 80%; height: 50px; line-height: 50px; padding-left: 10px; float: left;">
-    <a href=/><button class="btn btn-primary">Home</button></a>&nbsp;
-    {% for game in game_list %}    
-    <a href=game/{{ game.url }}><button class="btn btn-success">{{ game.name }}</button></a>&nbsp;
-    {% endfor %}
-</div>
-</div>
-<div style="background: #333; color: #f9f9f9; width: 10%; height: 50px; line-height: 50px; text-align: center; float: left;">
-    {% if sesh is defined and sesh["userId"] > 0 %}
-    <a href="/user"><button class="btn btn-primary">Account</button></a>
-    {% else %}
-    <a href="/gate"><button class="btn btn-primary">Gate</button></a>
-    {% endif %}
-    
-</div>

core/mucha.py

@@ -1,20 +1,23 @@
-from typing import Dict, Any, Optional, List
+from typing import Dict, Any, Optional
 import logging, coloredlogs
 from logging.handlers import TimedRotatingFileHandler
-from twisted.web import resource
+from starlette.requests import Request
-from twisted.web.http import Request
+from starlette.responses import PlainTextResponse
 from datetime import datetime
+from Crypto.Cipher import Blowfish
 import pytz
 
-from core import CoreConfig
+from .config import CoreConfig
-from core.utils import Utils
+from .utils import Utils
-
+from .title import TitleServlet
+from .data import Data
+from .const import *
 
 class MuchaServlet:
+    mucha_registry: Dict[str, Dict[str, str]] = {}
     def __init__(self, cfg: CoreConfig, cfg_dir: str) -> None:
         self.config = cfg
         self.config_dir = cfg_dir
-        self.mucha_registry: List[str] = []
 
         self.logger = logging.getLogger("mucha")
         log_fmt_str = "[%(asctime)s] Mucha | %(levelname)s | %(message)s"
@@ -33,78 +36,150 @@ class MuchaServlet:
         self.logger.addHandler(fileHandler)
         self.logger.addHandler(consoleHandler)
 
-        self.logger.setLevel(logging.INFO)
+        self.logger.setLevel(cfg.mucha.loglevel)
-        coloredlogs.install(level=logging.INFO, logger=self.logger, fmt=log_fmt_str)
+        coloredlogs.install(level=cfg.mucha.loglevel, logger=self.logger, fmt=log_fmt_str)
         
-        all_titles = Utils.get_all_titles()
+        self.data = Data(cfg)
 
-        for _, mod in all_titles.items():
+        for _, mod in TitleServlet.title_registry.items():
-            if hasattr(mod, "index") and hasattr(mod.index, "get_mucha_info"):
+            enabled, game_cds, netids = mod.get_mucha_info(self.config, self.config_dir)
-                enabled, game_cd = mod.index.get_mucha_info(
-                    self.config, self.config_dir
-                )
             if enabled:
-                    self.mucha_registry.append(game_cd)
+                for x in range(len(game_cds)):
+                    self.mucha_registry[game_cds[x]] = { "netid_prefix": netids[x] }
 
-        self.logger.info(
+        self.logger.info(f"Serving {len(self.mucha_registry)} games")
-            f"Serving {len(self.mucha_registry)} games"
-        )
 
-    def handle_boardauth(self, request: Request, _: Dict) -> bytes:
+    async def handle_boardauth(self, request: Request) -> bytes:
-        req_dict = self.mucha_preprocess(request.content.getvalue())
+        bod = await request.body()
+        req_dict = self.mucha_preprocess(bod)
         client_ip = Utils.get_ip_addr(request)
 
         if req_dict is None:
             self.logger.error(
-                f"Error processing mucha request {request.content.getvalue()}"
+                f"Error processing mucha request {bod}"
             )
-            return b""
+            return PlainTextResponse("RESULTS=000")
 
         req = MuchaAuthRequest(req_dict)
         self.logger.debug(f"Mucha request {vars(req)}")
-        self.logger.info(
-            f"Boardauth request from {client_ip} for {req.gameVer}"
-        )
         
-        if req.gameCd not in self.mucha_registry:
+        if not req.gameCd or not req.gameVer or not req.sendDate or not req.countryCd or not req.serialNum:
-            self.logger.warn(f"Unknown gameCd {req.gameCd}")
+            self.logger.warn(f"Missing required fields - {vars(req)}")
-            return b""
+            return PlainTextResponse("RESULTS=000")
 
-        # TODO: Decrypt S/N
+        minfo = self.mucha_registry.get(req.gameCd, {})
+
+        if not minfo:
+            self.logger.warning(f"Unknown gameCd {req.gameCd} from {client_ip}")
+            return PlainTextResponse("RESULTS=000")
+
+        b_key = b""
+        for x in range(8):
+            b_key += req.sendDate[(x - 1) & 7].encode()
+        
+        b_iv = b_key # what the fuck namco
+
+        cipher = Blowfish.new(b_key, Blowfish.MODE_CBC, b_iv)
+        try:
+            sn_decrypt = cipher.decrypt(bytes.fromhex(req.serialNum))[:12].decode()
+        except Exception as e:
+            self.logger.error(f"Decrypt SN {req.serialNum} failed! - {e}")
+            return PlainTextResponse("RESULTS=000")
+
+        self.logger.info(f"Boardauth request from {sn_decrypt} ({client_ip}) for {req.gameVer}")
 
         resp = MuchaAuthResponse(
-            f"{self.config.mucha.hostname}{':' + str(self.config.allnet.port) if self.config.server.is_develop else ''}"
+            f"{self.config.server.hostname}{':' + str(self.config.server.port) if not self.config.server.is_using_proxy else ''}"
         )
 
+        netid = minfo.get('netid_prefix', "ABxN") + sn_decrypt[5:]
+
+        cab = await self.data.arcade.get_machine(netid)
+        if cab:
+            arcade = await self.data.arcade.get_arcade(cab['id'])
+            if not arcade:
+                self.logger.error(f"Failed to get arcade with id {cab['id']}")
+                return PlainTextResponse("RESULTS=000")
+
+            resp.AREA_0 = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            resp.AREA_0_EN = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            resp.AREA_FULL_0 = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            resp.AREA_FULL_0_EN = arcade["region_id"] or AllnetJapanRegionId.AICHI.name
+            
+            resp.AREA_1 = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.AREA_1_EN = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.AREA_FULL_1 = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.AREA_FULL_1_EN = arcade["country"] or cab['country'] or AllnetCountryCode.JAPAN.value
+
+            resp.AREA_2 = arcade["city"] if arcade["city"] else ""
+            resp.AREA_2_EN = arcade["city"] if arcade["city"] else ""
+            resp.AREA_FULL_2 = arcade["city"] if arcade["city"] else ""
+            resp.AREA_FULL_2_EN = arcade["city"] if arcade["city"] else ""
+
+            resp.AREA_3 = ""
+            resp.AREA_3_EN = ""
+            resp.AREA_FULL_3 = ""
+            resp.AREA_FULL_3_EN = ""
+
+            resp.PREFECTURE_ID = arcade['region_id']
+            resp.COUNTRY_CD = arcade['country'] or cab['country'] or AllnetCountryCode.JAPAN.value
+            resp.PLACE_ID = req.placeId if req.placeId else f"{arcade['country'] or cab['country'] or AllnetCountryCode.JAPAN.value}{arcade['id']:04X}"
+            resp.SHOP_NAME = arcade['name']
+            resp.SHOP_NAME_EN = arcade['name']
+            resp.SHOP_NICKNAME = arcade['nickname']
+            resp.SHOP_NICKNAME_EN = arcade['nickname']
+
+        elif self.config.server.allow_unregistered_serials:
+            self.logger.info(f"Allow unknown serial {netid} ({sn_decrypt}) to auth")
+        
+        else:
+            self.logger.warn(f'Auth failed for NetID {netid}')
+            return PlainTextResponse("RESULTS=000")
+
         self.logger.debug(f"Mucha response {vars(resp)}")
 
-        return self.mucha_postprocess(vars(resp))
+        return PlainTextResponse(self.mucha_postprocess(vars(resp)))
 
-    def handle_updatecheck(self, request: Request, _: Dict) -> bytes:
+    async def handle_updatecheck(self, request: Request) -> bytes:
-        req_dict = self.mucha_preprocess(request.content.getvalue())
+        bod = await request.body()
+        req_dict = self.mucha_preprocess(bod)
         client_ip = Utils.get_ip_addr(request)
 
         if req_dict is None:
             self.logger.error(
-                f"Error processing mucha request {request.content.getvalue()}"
+                f"Error processing mucha request {bod}"
             )
-            return b""
+            return PlainTextResponse("RESULTS=000")
 
         req = MuchaUpdateRequest(req_dict)
+        self.logger.info(f"Updatecheck request from {req.serialNum} ({client_ip}) for {req.gameVer}")        
         self.logger.debug(f"Mucha request {vars(req)}")
-        self.logger.info(
-            f"Updatecheck request from {client_ip} for {req.gameVer}"
-        )
 
         if req.gameCd not in self.mucha_registry:
-            self.logger.warn(f"Unknown gameCd {req.gameCd}")
+            self.logger.warning(f"Unknown gameCd {req.gameCd}")
-            return b""
+            return PlainTextResponse("RESULTS=000")
 
-        resp = MuchaUpdateResponseStub(req.gameVer)
+        resp = MuchaUpdateResponse(req.gameVer, f"{self.config.server.hostname}{':' + str(self.config.server.port) if not self.config.server.is_using_proxy else ''}")
 
         self.logger.debug(f"Mucha response {vars(resp)}")
 
-        return self.mucha_postprocess(vars(resp))
+        return PlainTextResponse(self.mucha_postprocess(vars(resp)))
+
+    async def handle_dlstate(self, request: Request) -> bytes:
+        bod = await request.body()
+        req_dict = self.mucha_preprocess(bod)
+        client_ip = Utils.get_ip_addr(request)
+
+        if req_dict is None:
+            self.logger.error(
+                f"Error processing mucha request {bod}"
+            )
+            return PlainTextResponse("RESULTS=000")
+        
+        req = MuchaDownloadStateRequest(req_dict)
+        self.logger.info(f"DownloadState request from {req.serialNum} ({client_ip}) for {req.gameCd} -> {req.updateVer}")        
+        self.logger.debug(f"request {vars(req)}")
+        return PlainTextResponse("RESULTS=001")
 
     def mucha_preprocess(self, data: bytes) -> Optional[Dict]:
         try:
@@ -117,19 +192,17 @@ class MuchaServlet:
 
             return ret
 
-        except:
+        except Exception:
             self.logger.error(f"Error processing mucha request {data}")
             return None
 
     def mucha_postprocess(self, data: dict) -> Optional[bytes]:
         try:
-            urlencode = ""
+            urlencode = "&".join(f"{k}={v}" for k, v in data.items())
-            for k, v in data.items():
-                urlencode += f"{k}={v}&"
 
             return urlencode.encode()
 
-        except:
+        except Exception:
             self.logger.error("Error processing mucha response")
             return None
 
@@ -156,7 +229,7 @@ class MuchaAuthResponse:
         self.RESULTS = "001"
         self.AUTH_INTERVAL = "86400"
         self.SERVER_TIME = datetime.strftime(datetime.now(), "%Y%m%d%H%M")
-        self.UTC_SERVER_TIME = datetime.strftime(datetime.now(pytz.UTC), "%Y%m%d%H%M")
+        self.SERVER_TIME_UTC = datetime.strftime(datetime.now(pytz.UTC), "%Y%m%d%H%M")
 
         self.CHARGE_URL = f"https://{mucha_url}/charge/"
         self.FILE_URL = f"https://{mucha_url}/file/"
@@ -209,21 +282,120 @@ class MuchaUpdateRequest:
 class MuchaUpdateResponse:
     def __init__(self, game_ver: str, mucha_url: str) -> None:
         self.RESULTS = "001"        
+        self.EXE_VER = game_ver
+
         self.UPDATE_VER_1 = game_ver
-        self.UPDATE_URL_1 = f"https://{mucha_url}/updUrl1/"
+        self.UPDATE_URL_1 = f"http://{mucha_url}/updUrl1/"
-        self.UPDATE_SIZE_1 = "0"
+        self.UPDATE_SIZE_1 = "20"
-        self.UPDATE_CRC_1 = "0000000000000000"
+
-        self.CHECK_URL_1 = f"https://{mucha_url}/checkUrl/"
+        self.CHECK_CRC_1 = "0000000000000000"
-        self.EXE_VER_1 = game_ver
+        self.CHECK_URL_1 = f"http://{mucha_url}/checkUrl/"
+        self.CHECK_SIZE_1 = "20"
+
         self.INFO_SIZE_1 = "0"
         self.COM_SIZE_1 = "0"
         self.COM_TIME_1 = "0"
         self.LAN_INFO_SIZE_1 = "0"
+
         self.USER_ID = ""
         self.PASSWORD = ""
 
+"""
+RESULTS
+EXE_VER
 
+UPDATE_VER_%d
+UPDATE_URL_%d
+UPDATE_SIZE_%d
+
+CHECK_CRC_%d
+CHECK_URL_%d
+CHECK_SIZE_%d
+
+INFO_SIZE_1
+COM_SIZE_1
+COM_TIME_1
+LAN_INFO_SIZE_1
+
+USER_ID
+PASSWORD
+"""
 class MuchaUpdateResponseStub:
     def __init__(self, game_ver: str) -> None:
         self.RESULTS = "001"
         self.UPDATE_VER_1 = game_ver
+
+class MuchaDownloadStateRequest:
+    def __init__(self, request: Dict) -> None:        
+        self.gameCd = request.get("gameCd", "")
+        self.updateVer = request.get("updateVer", "")
+        self.serialNum = request.get("serialNum", "")
+        self.fileSize = request.get("fileSize", "")
+        self.compFileSize = request.get("compFileSize", "")
+        self.boardId = request.get("boardId", "")
+        self.placeId = request.get("placeId", "")
+        self.storeRouterIp = request.get("storeRouterIp", "")
+
+class MuchaDownloadErrorRequest:
+    def __init__(self, request: Dict) -> None:        
+        self.gameCd = request.get("gameCd", "")
+        self.updateVer = request.get("updateVer", "")
+        self.serialNum = request.get("serialNum", "")
+        self.downloadUrl = request.get("downloadUrl", "")
+        self.errCd = request.get("errCd", "")
+        self.errMessage = request.get("errMessage", "")
+        self.boardId = request.get("boardId", "")
+        self.placeId = request.get("placeId", "")
+        self.storeRouterIp = request.get("storeRouterIp", "")
+
+class MuchaRegiAuthRequest:
+    def __init__(self, request: Dict) -> None:        
+        self.gameCd = request.get("gameCd", "")
+        self.serialNum = request.get("serialNum", "") # Encrypted
+        self.countryCd = request.get("countryCd", "")
+        self.registrationCd = request.get("registrationCd", "")
+        self.sendDate = request.get("sendDate", "")
+        self.useToken = request.get("useToken", "")
+        self.allToken = request.get("allToken", "")
+        self.placeId = request.get("placeId", "")
+        self.storeRouterIp = request.get("storeRouterIp", "")
+
+class MuchaRegiAuthResponse:
+    def __init__(self) -> None:
+        self.RESULTS = "001" # 001 = success, 099, 098, 097 = fail, others = fail
+        self.ALL_TOKEN = "0" # Encrypted
+        self.ADD_TOKEN = "0" # Encrypted
+
+class MuchaTokenStateRequest:
+    def __init__(self, request: Dict) -> None:        
+        self.gameCd = request.get("gameCd", "")
+        self.serialNum = request.get("serialNum", "")
+        self.countryCd = request.get("countryCd", "")
+        self.useToken = request.get("useToken", "")
+        self.allToken = request.get("allToken", "")
+        self.placeId = request.get("placeId", "")
+        self.storeRouterIp = request.get("storeRouterIp", "")
+
+class MuchaTokenStateResponse:
+    def __init__(self) -> None:
+        self.RESULTS = "001"
+
+class MuchaTokenMarginStateRequest:
+    def __init__(self, request: Dict) -> None:        
+        self.gameCd = request.get("gameCd", "")
+        self.serialNum = request.get("serialNum", "")
+        self.countryCd = request.get("countryCd", "")
+        self.placeId = request.get("placeId", "")
+        self.limitLowerToken = request.get("limitLowerToken", 0)
+        self.limitUpperToken = request.get("limitUpperToken", 0)
+        self.settlementMonth = request.get("settlementMonth", 0)
+
+class MuchaTokenMarginStateResponse:
+    def __init__(self) -> None:
+        self.RESULTS = "001"
+        self.LIMIT_LOWER_TOKEN = 0
+        self.LIMIT_UPPER_TOKEN = 0
+        self.LAST_SETTLEMENT_MONTH = 0
+        self.LAST_LIMIT_LOWER_TOKEN = 0
+        self.LAST_LIMIT_UPPER_TOKEN = 0
+        self.SETTLEMENT_MONTH = 0

core/templates/arcade/index.jinja

@@ -0,0 +1,19 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+{% if arcade is defined %}
+<h1>{{ arcade.name }}</h1>
+<h2>PCBs assigned to this arcade <button class="btn btn-success" id="btn_add_cab" onclick="toggle_add_cab_form()">Add</button></h2>
+{% if success is defined and success == 3 %}
+<div style="background-color: #00AA00; padding: 20px; margin-bottom: 10px; width: 15%;">
+Cab added successfully
+</div>
+{% endif %}
+<ul style="font-size: 20px;">
+{% for c in arcade.cabs %}
+<li><a href="/cab/{{ c.id }}">{{ c.serial }} ({{ c.game }})</a>&nbsp;<button class="btn btn-secondary" onclick="prep_edit_form()">Edit</button>&nbsp;<button class="btn-danger btn">Delete</button></li>
+{% endfor %}
+</ul>
+{% else %}
+<h3>Arcade Not Found</h3>
+{% endif %}
+{% endblock content %}

core/templates/gate/create.jinja

@@ -1,4 +1,4 @@
-{% extends "core/frontend/index.jinja" %}
+{% extends "core/templates/index.jinja" %}
 {% block content %}
 <h1>Create User</h1>
 <form id="create" style="max-width: 240px; min-width: 10%;" action="/gate/gate.create" method="post">

core/templates/gate/gate.jinja

@@ -1,7 +1,7 @@
-{% extends "core/frontend/index.jinja" %}
+{% extends "core/templates/index.jinja" %}
 {% block content %}
 <h1>Gate</h1>
-{% include "core/frontend/widgets/err_banner.jinja" %}
+{% include "core/templates/widgets/err_banner.jinja" %}
 <style>
     /* Chrome, Safari, Edge, Opera */
     input::-webkit-outer-spin-button,

core/templates/index.jinja

@@ -4,6 +4,7 @@
         <title>{{ title }}</title>
         <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.2.3/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-rbsA2VBKQhggwzxH7pPCaAqO46MgnOM80zW1RWuH61DGLwZJEdK2Kadq2F9CUG65" crossorigin="anonymous">
         <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.2.3/dist/js/bootstrap.bundle.min.js" integrity="sha384-kenU1KFdBIe4zVF0s0G1M5b4hcpxyD9F7jL+jjXkk+Q2h455rYXK/7HAuoJl+0I4" crossorigin="anonymous"></script>
+        <script src="https://cdn.jsdelivr.net/npm/jquery@3.2.1/dist/jquery.min.js" integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4=" crossorigin="anonymous"></script>
         <style>
             html {
                 background-color: #181a1b !important;
@@ -77,10 +78,13 @@
                 margin-bottom: 10px;
                 width: 15%;
             }
+            .modal-content {
+                background-color: #181a1b;
+            }
         </style>
     </head>
     <body>
-    {% include "core/frontend/widgets/topbar.jinja" %}
+    {% include "core/templates/widgets/topbar.jinja" %}
     {% block content %}
     <h1>{{ server_name }}</h1>
     {% endblock content %}

core/templates/machine/index.jinja

@@ -0,0 +1,4 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+<h1>Machine Management</h1>
+{% endblock content %}

core/templates/sys/index.jinja

@@ -0,0 +1,69 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+<h1>System Management</h1>
+{% if error is defined %}
+{% include "core/templates/widgets/err_banner.jinja" %}
+{% endif %}
+<div class="row" id="rowForm">
+    {% if "{:08b}".format(sesh.permissions)[6] == "1" %}
+    <div class="col-sm-6" style="max-width: 25%;">
+        <form id="usrLookup" name="usrLookup" action="/sys/lookup.user" class="form-inline">
+            <h3>User Search</h3>
+            <div class="form-group">
+                <label for="usrId">User ID</label>
+                <input type="number" class="form-control" id="usrId" name="usrId">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="usrName">Username</label>
+                <input type="text" class="form-control" id="usrName" name="usrName">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="usrEmail">Email address</label>
+                <input type="email" class="form-control" id="usrEmail" name="usrEmail" aria-describedby="emailHelp">
+            </div>
+            <br />
+            <button type="submit" class="btn btn-primary">Search</button>
+        </form>
+    </div>
+    {% endif %}
+    {% if "{:08b}".format(sesh.permissions)[5] == "1" %}
+    <div class="col-sm-6" style="max-width: 25%;">
+        <form id="shopLookup" name="shopLookup" action="/sys/lookup.shop" class="form-inline">
+            <h3>Shop search</h3>
+            <div class="form-group">
+                <label for="shopId">Shop ID</label>
+                <input type="number" class="form-control" id="shopId" name="shopId">
+            </div>
+            OR
+            <div class="form-group">
+                <label for="serialNum">Serial Number</label>
+                <input type="text" class="form-control" id="serialNum" name="serialNum" maxlength="15">
+            </div>
+            <br />
+            <button type="submit" class="btn btn-primary">Search</button>
+        </form>
+    </div>
+    {% endif %}
+</div>
+<div class="row" id="rowResult" style="margin: 10px;">
+    {% if "{:08b}".format(sesh.permissions)[6] == "1" %}
+    <div id="userSearchResult" class="col-sm-6" style="max-width: 25%;">
+        {% for usr in usrlist %}
+        <a href=/user/{{ usr.id }}><pre>{{ usr.username if usr.username is not none else "<i>No Name Set</i>"}}</pre></a>
+        {% endfor %}
+    </div>
+    {% endif %}
+    {% if "{:08b}".format(sesh.permissions)[5] == "1" %}
+    <div id="shopSearchResult" class="col-sm-6" style="max-width: 25%;">
+        {% for shop in shoplist %}
+        <a href="/shop/{{ shop.id }}"><pre>{{ shop.name if shop.name else "<i>No Name Set</i>"}}</pre></a>
+        {% endfor %}
+    </div>
+    {% endif %}
+</div>
+<div class="row" id="rowAdd">
+
+</div>
+{% endblock content %}

core/templates/user/index.jinja

@@ -0,0 +1,175 @@
+{% extends "core/templates/index.jinja" %}
+{% block content %}
+<script type="text/javascript">
+function toggle_new_name_form() {
+  let frm = document.getElementById("new_name_form");
+  let btn = document.getElementById("btn_toggle_form");
+
+  if (frm.style['display'] != "") {
+    frm.style['display'] = "";
+    frm.style['max-height'] = "";
+    btn.innerText = "Cancel";
+  } else {
+    frm.style['display'] = "none";
+    frm.style['max-height'] = "0px";
+    btn.innerText = "Edit";
+  }
+}
+function toggle_add_card_form() {
+  let btn = document.getElementById("btn_add_card");
+  let dv = document.getElementById("add_card_container")
+
+  if (dv.style['display'] != "") {
+    btn.innerText = "Cancel";
+    dv.style['display'] = "";
+  } else {
+    btn.innerText = "Add";
+    dv.style['display'] = "none";
+  }
+}
+
+function prep_edit_form(access_code, chip_id, idm, card_type, u_memo) {
+  ac = document.getElementById("card_edit_frm_access_code");
+  cid = document.getElementById("card_edit_frm_chip_id");
+  fidm = document.getElementById("card_edit_frm_idm");
+  memo = document.getElementById("card_edit_frm_memo");
+
+  if (chip_id == "None" || chip_id == undefined) {
+    chip_id = ""
+  }
+  if (idm == "None" || idm == undefined) {
+    idm = ""
+  }
+  if (u_memo == "None" || u_memo == undefined) {
+    u_memo = ""
+  }
+
+  ac.value = access_code;
+  cid.value = chip_id;
+  fidm.value = idm;
+  memo.value = u_memo;
+
+  if (card_type == "AmusementIC") {
+    cid.disabled = true;
+    fidm.disabled = false;
+  } else {
+    cid.disabled = false;
+    fidm.disabled = true;
+  }
+}
+</script>
+<h1>Management for {{ username }}&nbsp;<button onclick="toggle_new_name_form()" class="btn btn-secondary" id="btn_toggle_form">Edit</button></h1>
+{% if error is defined %}
+{% include "core/templates/widgets/err_banner.jinja" %}
+{% endif %}
+{% if success is defined and success == 2 %}
+<div style="background-color: #00AA00; padding: 20px; margin-bottom: 10px; width: 15%;">
+Update successful
+</div>
+{% endif %}
+<form style="max-width: 33%; display: none; max-height: 0px;" action="/user/update.name" method="post" id="new_name_form">
+  <div class="mb-3">
+    <label for="new_name" class="form-label">New Nickname</label>
+    <input type="text" class="form-control" id="new_name" name="new_name" aria-describedby="new_name_help">
+    <div id="new_name_help" class="form-text">Must be 10 characters or less</div>
+  </div>
+  <button type="submit" class="btn btn-primary">Submit</button>
+</form>
+<p></p>
+<h2>Cards <button class="btn btn-success" id="btn_add_card" onclick="toggle_add_card_form()">Add</button></h2>
+{% if success is defined and success == 3 %}
+<div style="background-color: #00AA00; padding: 20px; margin-bottom: 10px; width: 15%;">
+Card added successfully
+</div>
+{% endif %}
+<div id="add_card_container" style="display: none; max-width: 33%;">
+  <form action="/user/add.card" method="post", id="frm_add_card">
+    <label class="form-label" for="card_add_frm_access_code">Access Code:</label>
+    <input class="form-control" name="add_access_code" id="card_add_frm_access_code" maxlength="20" type="text" required aria-describedby="ac_help">
+    <div id="ac_help" class="form-text">20 digit code on the back of the card.</div>
+    <button type="submit" class="btn btn-primary">Add</button>
+  </form>
+  <br>
+</div>
+<ul style="font-size: 20px;">
+{% for c in cards %}
+<li>{{ c.access_code }} ({{ c.type}}): {{ c.status }}&nbsp;<button onclick="prep_edit_form('{{ c.access_code }}', '{{ c.chip_id}}', '{{ c.idm }}', '{{ c.type }}', '{{ c.memo }}')" data-bs-toggle="modal" data-bs-target="#card_edit" class="btn btn-secondary" id="btn_edit_card_{{ c.access_code }}">Edit</button>&nbsp;{% if c.status == 'Active'%}<button class="btn-warning btn">Lock</button>{% elif c.status == 'Locked' %}<button class="btn-warning btn">Unlock</button>{% endif %}&nbsp;<button class="btn-danger btn">Delete</button></li>
+{% endfor %}
+</ul>
+
+<h2>Reset Password</h2>
+{% if success is defined and success == 1 %}
+<div style="background-color: #00AA00; padding: 20px; margin-bottom: 10px; width: 15%;">
+Update successful
+</div>
+{% endif %}
+<form style="max-width: 33%;" action="/user/update.pw" method="post">
+  <div class="mb-3">
+    <label for="current_pw" class="form-label">Current Password</label>
+    <input type="password" class="form-control" id="current_pw" name="current_pw">
+  </div>
+  <div class="mb-3">
+    <label for="password1" class="form-label">New Password</label>
+    <input type="password" class="form-control" id="password1" name="password1" aria-describedby="password_help">
+    <div id="password_help" class="form-text">Password must be at least 10 characters long, contain an upper and lowercase character, number, and special character</div>
+  </div>
+  <div class="mb-3">
+    <label for="password2" class="form-label">Retype New Password</label>
+    <input type="password" class="form-control" id="password2" name="password2">
+  </div>
+  <button type="submit" class="btn btn-primary">Submit</button>
+</form>
+
+{% if arcades is defined and arcades|length > 0 %}
+<h2>Arcades</h2>
+  <ul>
+  {% for a in arcades %}
+    <li><h3>{{ a.name }}</h3>
+      {% if a.machines|length > 0 %}
+        <table>
+          <tr><th>Serial</th><th>Game</th><th>Last Seen</th></tr>
+          {% for m in a.machines %}
+            <tr><td>{{ m.serial }}</td><td>{{ m.game }}</td><td>{{ m.last_seen }}</td></tr>
+          {% endfor %}
+        </table>
+      {% endif %}
+    </li>
+  {% endfor %}
+  </ul>
+{% endif %}
+
+<div class="modal fade" id="card_edit" tabindex="-1" aria-labelledby="card_edit_label" aria-hidden="true">
+  <div class="modal-dialog">
+    <div class="modal-content">
+      <div class="modal-header">
+        <h1 class="modal-title fs-5" id="card_edit_label">Edit Card</h1>
+        <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="Close"></button>
+      </div>
+      <div class="modal-body">
+        <form action="/user/edit.card" method="post" id="frm_edit_card">
+          <label class="form-label" for="card_edit_frm_access_code">Access Code:</label>
+          <input class="form-control" readonly name="add_access_code" id="card_edit_frm_access_code" maxlength="20" type="text" required aria-describedby="ac_help">
+          <div id="ac_help" class="form-text">20 digit code on the back of the card. If this is incorrect, contact a sysadmin.</div>
+
+          <label class="form-label" for="card_edit_frm_memo" id="card_edit_frm_memo_lbl">Memo:</label>
+          <input class="form-control" aria-describedby="memo_help" name="add_memo" id="card_edit_frm_memo" maxlength="16" type="text">
+          <div id="memo_help" class="form-text">Must be 16 characters or less.</div>
+
+          <label class="form-label" for="card_edit_frm_idm" id="card_edit_frm_idm_lbl">FeliCa IDm:</label>
+          <input class="form-control" aria-describedby="idm_help" name="add_felica_idm" id="card_edit_frm_idm" maxlength="16" type="text">
+          <div id="idm_help" class="form-text">8 bytes that uniquly idenfites a FeliCa card. Obtained by reading the card with an NFC reader.</div>
+        
+          <label class="form-label" for="card_edit_frm_chip_id" id="card_edit_frm_chip_id_lbl">Mifare UID:</label>
+          <input class="form-control" aria-describedby="chip_id_help" name="add_mifare_chip_id" id="card_edit_frm_chip_id" maxlength="8" type="text">
+          <div id="chip_id_help" class="form-text">4 byte integer that uniquly identifies a Mifare card. Obtained by reading the card with an NFC reader.</div>
+        </form>
+      </div>
+      <div class="modal-footer">
+        <button type="submit" class="btn btn-primary" form="frm_edit_card">Edit</button>
+        <button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
+      </div>
+    </div>
+  </div>
+</div>
+
+{% endblock content %}

core/templates/widgets/err_banner.jinja

@@ -0,0 +1,28 @@
+{% if error > 0 %}
+<div class="err-banner">
+<h3>Error</h3>
+{% if error == 1 %}
+Card not registered, or wrong password
+{% elif error == 2 %}
+Missing or malformed access code
+{% elif error == 3 %}
+Failed to create user
+{% elif error == 4 %}
+Required field not filled or invalid
+{% elif error == 5 %}
+Incorrect old password
+{% elif error == 6 %}
+Passwords don't match
+{% elif error == 7 %}
+New password not acceptable
+{% elif error == 8 %}
+New Nickname too long
+{% elif error == 9 %}
+You must be logged in to preform this action
+{% elif error == 10 %}
+Invalid serial number
+{% else %}
+An unknown error occoured
+{% endif %}
+</div>
+{% endif %}

core/templates/widgets/topbar.jinja

@@ -0,0 +1,22 @@
+<div style="background: #333; color: #f9f9f9; width: 10%; height: 50px; line-height: 50px; text-align: center; float: left;">
+    Navigation
+</div>
+<div style="background: #333; color: #f9f9f9; width: 80%; height: 50px; line-height: 50px; padding-left: 10px; float: left;">
+    <a href=/><button class="btn btn-primary">Home</button></a>&nbsp;
+    {% for game, data in game_list|items %}    
+    <a href=/game{{ data.url }}/><button class="btn btn-success">{{ game }}</button></a>&nbsp;
+    {% endfor %}
+</div>
+</div>
+<div style="background: #333; color: #f9f9f9; width: 10%; height: 50px; line-height: 50px; text-align: center; float: left;">
+    {% if sesh is defined and sesh["permissions"] >= 2 %}
+    <a href="/sys/"><button class="btn btn-primary">System</button></a>
+    {% endif %}
+    {% if sesh is defined and sesh["user_id"] > 0 %}
+    <a href="/user/"><button class="btn btn-primary">Account</button></a>
+    <a href="/user/logout"><button class="btn btn-danger">Logout</button></a>
+    {% else %}
+    <a href="/gate/"><button class="btn btn-primary">Gate</button></a>
+    {% endif %}
+    
+</div>

core/title.py

@@ -1,20 +1,104 @@
-from typing import Dict, Any
+from typing import Dict, List, Tuple, Any
+import json
 import logging, coloredlogs
 from logging.handlers import TimedRotatingFileHandler
-from twisted.web.http import Request
+from starlette.requests import Request
+from starlette.responses import Response
+from starlette.routing import Route
 
 from core.config import CoreConfig
 from core.data import Data
 from core.utils import Utils
 
+class JSONResponseNoASCII(Response):
+    media_type = "application/json"
+
+    def render(self, content: Any) -> bytes:
+        return json.dumps(
+            content,
+            ensure_ascii=False,
+        ).encode("utf-8")
+
+class BaseServlet:
+    def __init__(self, core_cfg: CoreConfig, cfg_dir: str) -> None:
+        self.core_cfg = core_cfg
+        self.game_cfg = None
+        self.logger = logging.getLogger("title")
+
+    @classmethod
+    def is_game_enabled(cls, game_code: str, core_cfg: CoreConfig, cfg_dir: str) -> bool:
+        """Called during boot to check if a specific game code should load.
+
+        Args:
+            game_code (str): 4 character game code
+            core_cfg (CoreConfig): CoreConfig class
+            cfg_dir (str): Config directory
+
+        Returns:
+            bool: True if the game is enabled and set to run, False otherwise
+        
+        """
+        return False
+    
+    def get_routes(self) -> List[Route]:
+        """Called during boot to get all matcher endpoints this title servlet handles
+
+        Returns:
+            List[Route]: A list of Routes, WebSocketRoutes, or similar classes
+        """
+        return [
+            Route("/{game}/{version}/{endpoint}", self.render_POST, methods=["POST"]),
+            Route("/{game}/{version}/{endpoint}", self.render_GET, methods=["GET"]),
+        ]
+    
+    def setup(self) -> None:
+        """Called once during boot, should contain any additional setup the handler must do, such as starting any sub-services
+        """
+        pass
+
+    def get_allnet_info(self, game_code: str, game_ver: int, keychip: str) -> Tuple[str, str]:
+        """Called any time a request to PowerOn is made to retrieve the url/host strings to be sent back to the game
+
+        Args:
+            game_code (str): 4 character game code
+            game_ver (int): version, expressed as an integer by multiplying by 100 (1.10 -> 110)
+            keychip (str): Keychip serial of the requesting machine, can be used to deliver specific URIs to different machines
+
+        Returns:
+            Tuple[str, str]: A tuple where offset 0 is the allnet uri field, and offset 1 is the allnet host field
+        """
+        if not self.core_cfg.server.is_using_proxy and Utils.get_title_port(self.core_cfg) != 80:
+            return (f"http://{self.core_cfg.server.hostname}:{Utils.get_title_port(self.core_cfg)}/{game_code}/{game_ver}/", "")
+
+        return (f"http://{self.core_cfg.server.hostname}/{game_code}/{game_ver}/", "")
+
+    def get_mucha_info(self, core_cfg: CoreConfig, cfg_dir: str) -> Tuple[bool, List[str], List[str]]:
+        """Called once during boot to check if this game is a mucha game
+
+        Args:
+            core_cfg (CoreConfig): CoreConfig class
+            cfg_dir (str): Config directory
+
+        Returns:
+            Tuple[bool, str]: Tuple where offset 0 is true if the game is enabled, false otherwise, and offset 1 is the game CD
+        """
+        return (False, [], [])
+
+    async def render_POST(self, request: Request) -> bytes:
+        self.logger.warn(f"Game Does not dispatch POST")
+        return Response()
+
+    async def render_GET(self, request: Request) -> bytes:
+        self.logger.warn(f"Game Does not dispatch GET")
+        return Response()
 
 class TitleServlet:
+    title_registry: Dict[str, BaseServlet] = {}
     def __init__(self, core_cfg: CoreConfig, cfg_folder: str):
         super().__init__()
         self.config = core_cfg
         self.config_folder = cfg_folder
         self.data = Data(core_cfg)
-        self.title_registry: Dict[str, Any] = {}
 
         self.logger = logging.getLogger("title")
         if not hasattr(self.logger, "initialized"):
@@ -43,17 +127,14 @@ class TitleServlet:
         plugins = Utils.get_all_titles()
 
         for folder, mod in plugins.items():
-            if hasattr(mod, "game_codes") and hasattr(mod, "index"):
+            if hasattr(mod, "game_codes") and hasattr(mod, "index") and hasattr(mod.index, "is_game_enabled"):
                 should_call_setup = True
+                game_servlet: BaseServlet = mod.index
+                game_codes: List[str] = mod.game_codes
                 
-                if hasattr(mod.index, "get_allnet_info"):
+                for code in game_codes:
-                    for code in mod.game_codes:
+                    if game_servlet.is_game_enabled(code, self.config, self.config_folder):
-                        enabled, _, _ = mod.index.get_allnet_info(
+                        handler_cls = game_servlet(self.config, self.config_folder)
-                            code, self.config, self.config_folder
-                        )
-
-                        if enabled:
-                            handler_cls = mod.index(self.config, self.config_folder)
 
                         if hasattr(handler_cls, "setup") and should_call_setup:
                             handler_cls.setup()
@@ -62,43 +143,46 @@ class TitleServlet:
                         self.title_registry[code] = handler_cls
 
             else:
-                    self.logger.warn(f"Game {folder} has no get_allnet_info")
+                self.logger.error(f"{folder} missing game_code or index in __init__.py, or is_game_enabled in index")
-
-            else:
-                self.logger.error(f"{folder} missing game_code or index in __init__.py")
 
         self.logger.info(
-            f"Serving {len(self.title_registry)} game codes {'on port ' + str(core_cfg.title.port) if core_cfg.title.port > 0 else ''}"
+            f"Serving {len(self.title_registry)} game codes {'on port ' + str(core_cfg.server.port) if core_cfg.server.port > 0 else ''}"
         )
 
     def render_GET(self, request: Request, endpoints: dict) -> bytes:
-        code = endpoints["game"]
+        code = endpoints["title"]
+        subaction = endpoints['subaction']
+        
         if code not in self.title_registry:
-            self.logger.warn(f"Unknown game code {code}")
+            self.logger.warning(f"Unknown game code {code}")
             request.setResponseCode(404)
             return b""
 
         index = self.title_registry[code]
-        if not hasattr(index, "render_GET"):
+        handler = getattr(index, f"{subaction}", None)
-            self.logger.warn(f"{code} does not dispatch GET")
+        if handler is None:
-            request.setResponseCode(405)
+            self.logger.error(f"{code} does not have handler for GET subaction {subaction}")
+            request.setResponseCode(500)
             return b""
 
-        return index.render_GET(request, endpoints["version"], endpoints["endpoint"])
+        return handler(request, code, endpoints)
 
     def render_POST(self, request: Request, endpoints: dict) -> bytes:
-        code = endpoints["game"]
+        code = endpoints["title"]
+        subaction = endpoints['subaction']
+
         if code not in self.title_registry:
-            self.logger.warn(f"Unknown game code {code}")
+            self.logger.warning(f"Unknown game code {code}")
             request.setResponseCode(404)
             return b""
 
         index = self.title_registry[code]
-        if not hasattr(index, "render_POST"):
+        handler = getattr(index, f"{subaction}", None)
-            self.logger.warn(f"{code} does not dispatch POST")
+        if handler is None:
-            request.setResponseCode(405)
+            self.logger.error(f"{code} does not have handler for POST subaction {subaction}")
+            request.setResponseCode(500)
             return b""
 
-        return index.render_POST(
+        endpoints.pop("title")
-            request, int(endpoints["version"]), endpoints["endpoint"]
+        endpoints.pop("subaction")
-        )
+        return handler(request, code, endpoints)

core/utils.py

@@ -1,12 +1,18 @@
-from typing import Dict, Any
+from typing import Dict, Any, Optional
 from types import ModuleType
-from twisted.web.http import Request
+from starlette.requests import Request
 import logging
 import importlib
 from os import walk
+import jwt
+from base64 import b64decode
+from datetime import datetime, timezone
 
+from .config import CoreConfig
 
 class Utils:
+    real_title_port = None
+    real_title_port_ssl = None
     @classmethod
     def get_all_titles(cls) -> Dict[str, ModuleType]:
         ret: Dict[str, Any] = {}
@@ -16,6 +22,9 @@ class Utils:
                 if not dir.startswith("__"):
                     try:
                         mod = importlib.import_module(f"titles.{dir}")
+                        if hasattr(mod, "game_codes") and hasattr(
+                            mod, "index"
+                        ):  # Minimum required to function
                             ret[dir] = mod
 
                     except ImportError as e:
@@ -25,4 +34,52 @@ class Utils:
 
     @classmethod
     def get_ip_addr(cls, req: Request) -> str:
-        return req.getAllHeaders()[b"x-forwarded-for"].decode() if b"x-forwarded-for" in req.getAllHeaders() else req.getClientAddress().host
+        return req.headers.get("x-forwarded-for", req.client.host)
+    
+    @classmethod
+    def get_title_port(cls, cfg: CoreConfig):
+        if cls.real_title_port is not None: return cls.real_title_port
+
+        cls.real_title_port = cfg.server.proxy_port if cfg.server.is_using_proxy and cfg.server.proxy_port else cfg.server.port
+        
+        return cls.real_title_port
+    
+    @classmethod
+    def get_title_port_ssl(cls, cfg: CoreConfig):
+        if cls.real_title_port_ssl is not None: return cls.real_title_port_ssl
+
+        cls.real_title_port_ssl = cfg.server.proxy_port_ssl if cfg.server.is_using_proxy and cfg.server.proxy_port_ssl else Utils.get_title_port(cfg)
+        
+        return cls.real_title_port_ssl
+
+def create_sega_auth_key(aime_id: int, game: str, place_id: int, keychip_id: str, b64_secret: str, exp_seconds: int = 86400, err_logger: str = 'aimedb') -> Optional[str]:
+    logger = logging.getLogger(err_logger)
+    try:
+        return jwt.encode({ "aime_id": aime_id, "game": game, "place_id": place_id, "keychip_id": keychip_id, "exp": int(datetime.now(tz=timezone.utc).timestamp()) + exp_seconds }, b64decode(b64_secret), algorithm="HS256")
+    except jwt.InvalidKeyError:
+        logger.error("Failed to encode Sega Auth Key because the secret is invalid!")
+        return None
+    except Exception as e:
+        logger.error(f"Unknown exception occoured when encoding Sega Auth Key! {e}")
+        return None
+
+def decode_sega_auth_key(token: str, b64_secret: str, err_logger: str = 'aimedb') -> Optional[Dict]:
+    logger = logging.getLogger(err_logger)
+    try:
+        return jwt.decode(token, "secret", b64decode(b64_secret), algorithms=["HS256"], options={"verify_signature": True})
+    except jwt.ExpiredSignatureError:
+        logger.error("Sega Auth Key failed to validate due to an expired signature!")
+        return None
+    except jwt.InvalidSignatureError:
+        logger.error("Sega Auth Key failed to validate due to an invalid signature!")
+        return None
+    except jwt.DecodeError as e:
+        logger.error(f"Sega Auth Key failed to decode! {e}")
+        return None
+    except jwt.InvalidTokenError as e:
+        logger.error(f"Sega Auth Key is invalid! {e}")
+        return None
+    except Exception as e:
+        logger.error(f"Unknown exception occoured when decoding Sega Auth Key! {e}")
+        return None
+    

dbutils.py

@@ -1,8 +1,12 @@
-import yaml
+#!/usr/bin/env python3
 import argparse
-from core.config import CoreConfig
+import logging
+from os import mkdir, path, access, W_OK
+import yaml
+import asyncio
+
 from core.data import Data
-from os import path, mkdir, access, W_OK
+from core.config import CoreConfig
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="Database utilities")
@@ -15,24 +19,17 @@ if __name__ == "__main__":
         type=str,
         help="Version of the database to upgrade/rollback to",
     )
-    parser.add_argument(
-        "--game",
-        "-g",
-        type=str,
-        help="Game code of the game who's schema will be updated/rolled back. Ex. SDFE",
-    )
     parser.add_argument("--email", "-e", type=str, help="Email for the new user")
-    parser.add_argument("--old_ac", "-o", type=str, help="Access code to transfer from")
+    parser.add_argument("--access_code", "-a", type=str, help="Access code for new/transfer user", default="00000000000000000000")
-    parser.add_argument("--new_ac", "-n", type=str, help="Access code to transfer to")
+    parser.add_argument("--message", "-m", type=str, help="Revision message")
-    parser.add_argument("--force", "-f", type=bool, help="Force the action to happen")
+    parser.add_argument("action", type=str, help="create, upgrade, downgrade, create-owner, migrate, create-revision, create-autorevision")
-    parser.add_argument(
-        "action", type=str, help="DB Action, create, recreate, upgrade, or rollback"
-    )
     args = parser.parse_args()
 
     cfg = CoreConfig()
     if path.exists(f"{args.config}/core.yaml"):
-        cfg.update(yaml.safe_load(open(f"{args.config}/core.yaml")))
+        cfg_dict = yaml.safe_load(open(f"{args.config}/core.yaml"))
+        cfg_dict.get("database", {})["loglevel"] = "info"
+        cfg.update(cfg_dict)
 
     if not path.exists(cfg.server.log_dir):
         mkdir(cfg.server.log_dir)
@@ -45,35 +42,33 @@ if __name__ == "__main__":
 
     data = Data(cfg)
 
-
     if args.action == "create":
         data.create_database()
     
-    elif args.action == "recreate":
+    elif args.action == "upgrade":
-        data.recreate_database()
+        data.schema_upgrade(args.version)
 
-    elif args.action == "upgrade" or args.action == "rollback":
+    elif args.action == "downgrade":
-        if args.version is None:
+        if not args.version:
-            data.logger.error("Must set game and version to migrate to")
+            logging.getLogger("database").error(f"Version argument required for downgrade")
-            exit(0)
+            exit(1)
-
+        data.schema_downgrade(args.version)
-        if args.game is None:
-            data.logger.info("No game set, upgrading core schema")
-            data.migrate_database("CORE", int(args.version), args.action)
-
-        else:
-            data.migrate_database(args.game, int(args.version), args.action)
-
-    elif args.action == "autoupgrade":
-        data.autoupgrade()
 
     elif args.action == "create-owner":
-        data.create_owner(args.email)
+        loop = asyncio.get_event_loop()
+        loop.run_until_complete(data.create_owner(args.email, args.access_code))
 
-    elif args.action == "migrate-card":
+    elif args.action == "migrate":
-        data.migrate_card(args.old_ac, args.new_ac, args.force)
+        loop = asyncio.get_event_loop()
+        loop.run_until_complete(data.migrate())
 
-    elif args.action == "cleanup":
+    elif args.action == "create-revision":
-        data.delete_hanging_users()
+        loop = asyncio.get_event_loop()
+        loop.run_until_complete(data.create_revision(args.message))
 
-    data.logger.info("Done")
+    elif args.action == "create-autorevision":
+        loop = asyncio.get_event_loop()
+        loop.run_until_complete(data.create_revision_auto(args.message))
+
+    else:
+        logging.getLogger("database").info(f"Unknown action {args.action}")

docker-compose.yml

@@ -0,0 +1,66 @@
+version: "3.9"
+services:
+  app:
+    hostname: ma.app
+    build: .
+    volumes:
+      - ./aime:/app/aime
+      - ./configs/config:/app/config
+
+    environment:
+      CFG_DEV: 1
+      CFG_CORE_SERVER_HOSTNAME: 0.0.0.0
+      CFG_CORE_DATABASE_HOST: ma.db
+      CFG_CORE_MEMCACHED_HOSTNAME: ma.memcached
+      CFG_CORE_AIMEDB_KEY: <INSERT AIMEDB KEY HERE>
+      CFG_CHUNI_SERVER_LOGLEVEL: debug
+
+    ports:
+      - "80:80"
+      - "8443:8443"
+      - "22345:22345"
+
+      - "8080:8080"
+      - "8090:8090"
+
+    depends_on:
+      db:
+        condition: service_healthy
+
+  db:
+    hostname: ma.db
+    image: yobasystems/alpine-mariadb:10.11.5
+    environment:
+      MYSQL_DATABASE: aime
+      MYSQL_USER: aime
+      MYSQL_PASSWORD: aime
+      MYSQL_ROOT_PASSWORD: AimeRootPassword
+      MYSQL_CHARSET: utf8mb4
+      MYSQL_COLLATION: utf8mb4_general_ci
+    ##Note: expose port 3306 to allow read.py importer into database, comment out when not needed
+    #ports:
+    #  - "3306:3306"
+    ##Note: uncomment to allow mysql to create a persistent database, leave commented if you want to rebuild database from scratch often
+    #volumes:
+    #  - ./AimeDB:/var/lib/mysql
+    healthcheck:
+      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost", "-pAimeRootPassword"]
+      timeout: 5s
+      retries: 5
+
+  memcached:
+    hostname: ma.memcached
+    image: memcached:1.6.22-alpine3.18
+    command: [ "memcached", "-m", "1024", "-I", "128m" ]
+
+  phpmyadmin:
+    hostname: ma.phpmyadmin
+    image: phpmyadmin:latest
+    environment:
+      PMA_HOSTS: ma.db
+      PMA_USER: root
+      PMA_PASSWORD: AimeRootPassword
+      APACHE_PORT: 8080
+    ports:
+    - "9090:8080"
+

docs/INSTALL_DOCKER.md

@@ -0,0 +1,246 @@
+# ARTEMiS - Docker Installation Guide
+
+This step-by-step guide will allow you to install a Contenerized Version of ARTEMiS inside Docker, some steps can be skipped assuming you already have pre-requisite components and modules installed.
+
+This guide assumes using Debian 12(bookworm-stable) as a Host Operating System for most of packages and modules.
+
+## Pre-Requisites:
+
+- Linux-Based Operating System (e.g. Debian, Ubuntu)
+- Docker (https://get.docker.com)
+- Python 3.9+
+- (optional) Git
+
+## Install Python3.9+ and Docker
+
+```
+(if this is a fresh install of the system)
+sudo apt update && sudo apt upgrade
+
+(installs python3 and pip)
+sudo apt install python3 python3-pip
+
+(installs docker)
+curl -fsSL https://get.docker.com -o get-docker.sh
+sh get-docker.sh
+
+(optionally install git)
+sudo apt install git
+```
+
+## Get ARTEMiS
+
+If you installed git, clone into your choice of ARTEMiS git repository, e.g.:
+```
+git clone <ARTEMiS Repo> <folder>
+```
+If not, download the source package, and unpack it to the folder of your choice.
+
+## Prepare development/home configuration
+
+To build our Docker setup, first we need to create some folders and copy some files around
+- Create 'aime', 'configs', 'AimeDB', and 'logs' folder in ARTEMiS root folder (where all source files exist)
+- Inside configs folder, create 'config' folder, and copy all .yaml files from example_config to config (thats all files without nginx_example.conf)
+- Edit .yaml files inside configs/config to suit your server needs
+- Edit core.yaml inside configs/config:
+```
+set server.listen_address: to "0.0.0.0"
+set title.hostname: to machine's IP address, e.g. "192.168.x.x", depending on your network, or actual hostname if your configuration is already set for dns resolve
+set database.host: to "ma.db"
+set database.memcached_host: to "ma.memcached"
+set aimedb.key: to "<actual AIMEDB key>"
+```
+
+## Running Docker Compose
+
+After configuring, go to ARTEMiS root folder, and execute:
+
+```
+docker compose up -d
+```
+
+("-d" argument means detached or daemon, meaning you will regain control of your terminal and Containers will run in background)
+
+This will start pulling and building required images from network, after it's done, a development server should be running, with server accessible under machine's IP, frontend with port 8090, and PHPMyAdmin under port 9090.
+
+- To turn off the server, from ARTEMiS root folder, execute:
+
+```
+docker compose down
+```
+
+- If you changed some files around, and don't see your changes applied, execute:
+
+```
+(turn off the server)
+docker compose down
+(rebuild)
+docker compose build
+(turn on)
+docker compose up -d
+```
+
+- If you need to see logs from containers running, execute:
+
+```
+docker compose logs
+```
+
+- add '-f' to the end if you want to follow logs.
+
+## Running commands
+
+If you need to execute python scripts supplied with the application, use `docker compose exec app python3 <script> <command>`, for example `docker compose exec app python3 dbutils.py version`
+
+## Persistent DB
+
+By default, in development mode, ARTEMiS database is stored temporarily, if you wish to keep your database saved between restarts, we need to bind the database inside the container to actual storage/folder inside our server, to do this we need to make a few changes:
+
+- First off, edit docker-compose.yml, and uncomment 2 lines:
+
+```
+(uncomment these two)
+#volumes:
+#  - ./AimeDB:/var/lib/mysql
+```
+
+- After that, start up the server, this time Database will be saved in AimeDB folder we created in our configuration steps.
+- If you wish to save it in another folder and/or storage device, change the "./AimeDB" target folder to folder/device of your choice
+
+NOTE (NEEDS FIX): at the moment running development mode with persistent DB will always run database creation script at the start of application, while it doesn't break database outright, it might create some issues, a temporary fix can be applied:
+
+- Start up containers with persistent DB already enabled, let application create database
+- After startup, `docker compose down` the instance
+- Edit entrypoint.sh and remove the `python3 dbutils.py create` line from Development mode statement
+- Execute `docker compose build` and `docker compose up -d` to rebuild the app and start the containers back
+
+## Adding importer data
+
+To add data using importer, we can do that a few ways:
+
+### Use importer locally on server
+
+For that we need actual GameData and Options supplied somehow to the server system, be it wsl2 mounting layer, a pendrive with data, network share, or a direct copy to the server storage
+With python3 installed on system, install requirements.txt directly to the system, or through python3 virtual-environment (python3-venv)
+Default mysql/mariadb client development packages will also be required
+
+- In the system:
+
+```
+sudo apt install default-libmysqlclient-dev build-essential pkg-config libmemcached-dev
+sudo apt install mysql-client
+OR
+sudo apt install libmariadb-dev
+```
+
+- In the root ARTEMiS folder
+
+```
+python3 -m pip install -r requirements.txt
+```
+
+- If we wish to layer that with python3 virtual-environment, install required system packages, then:
+
+```
+sudo apt install python3-venv
+python3 -m venv /path/to/venv
+cd /path/to/venv/bin
+python3 -m pip install -r /path/to/artemis/requirements.txt
+```
+
+- Depending on how you installed, now you can run read.py using:
+- For direct installation, from root ARTEMiS folder:
+
+```
+python3 read.py <args>
+```
+
+- Or from python3 virtual environment, from root ARTEMiS folder:
+
+```
+/path/to/python3-venv/bin/python3 /path/to/artemis/read.py <args>
+```
+
+- We need to expose database container port, so that read.py can communicate with the database, inside docker-compose.yml, uncomment 2 lines in the database container declaration (db):
+
+```
+#ports:
+#  - "3306:3306"
+```
+
+- Now, `docker compose down && docker compose build && docker compose up -d` to restart containers
+
+Now to insert the data, by default, docker doesn't expose container hostnames to root system, when trying to run read.py against a container, it will Error that hostname is not available, to fix that, we can add database hostname by hand to /etc/hosts:
+
+```
+sudo <editor of your choice> /etc/hosts
+add '127.0.0.1  ma.db' to the table
+save and close
+```
+
+- You can remove the line in /etc/hosts and de-expose the database port after successful import (this assumes you're using Persistent DB, as restarting the container without it will clear imported data).
+
+### Use importer on remote Linux system
+
+Follow the system and python portion of the guide, installing required packages and python3 modules, Download the ARTEMiS source.
+
+- Edit core.yaml and insert it into config catalog:
+
+```
+database:
+  host: "<hostname of target system>"
+```
+
+- Expose port 3306 from database docker container to system, and allow port 3306 through system firewall to expose port to the system from which you will be importing data. (Remember to close down the database ports after finishing!)
+
+- Import data using read.py
+
+### Use importer on remote Windows system
+
+Follow the [windows](docs/INSTALL_WINDOWS.md) guide for installing python dependencies, download the ARTEMiS source.
+
+- Edit core.yaml and insert it into config catalog:
+
+```
+database:
+  host: "<hostname of target system>"
+```
+
+- Expose port 3306 from database docker container to system, and allow port 3306 through system firewall to expose port to the system from which you will be importing data.
+- For Windows, also allow port 3306 outside the system so that read.py can communicate with remote database. (Remember to close down the database ports after finishing!)
+
+# Troubleshooting
+
+## Game does not connect to ARTEMiS Allnet Server
+Double check your core.yaml if all addresses are correct and ports are correctly set and/or opened.
+
+## Game does not connect to Title Server
+Title server hostname requires your actual system hostname, from which you set up the Containers, or it's IP address, you can get the IP by using command `ip a` which will list all interfaces, and one of them should be your system IP (typically under eth0).
+
+## Unhandled command in AimeDB
+Make sure you have a proper AimeDB Key added to configuration.
+
+## Memcached Error in ARTEMiS application causes errors in loading data
+Currently when running ARTEMiS from master branch, there is a small bug that causes app to always configure memcached service to 127.0.0.1, to fix that, locate cache.py file in core/data, and edit:
+
+```
+memcache = pylibmc.Client([hostname]), binary=True)
+```
+
+to:
+
+```
+memcache = pylibmc.Client(["ma.memcached"], binary=True)
+```
+
+And build the containers again.
+This will fix errors loading data from server.
+(This is fixed in development branch)
+
+## read.py "Can't connect to local server through socket '/run/mysqld/mysqld.sock'"
+
+sqlalchemy by default reads any ip based connection as socket, thus trying to connect locally, please use a hostname (such as ma.db as in guide, and do not localhost) to force it to use a network interface.
+
+### TODO:
+
+- Production environment

docs/INSTALL_LINUX.md

@@ -0,0 +1,107 @@
+# Installing ARTEMiS on Linux
+This guide assumes a fresh install of Debian 12 or Rasperry Pi OS. If you're using a different distrubution, your package manager commands and package names may be different then what's listed below. Please check with your repository's package manager for package names.
+
+## Install prerequisits
+### Python
+Some installs may come with python already installed. You can verify this by trying the following commands:
+- `python --version`
+- `python3 --version`
+- `python3.<minor version> --version` where `<minor version>` is a python 3 release (eg 11, 10)
+
+If your python version is at least 3.7, you can move to the next step
+
+### Libraries and other software
+ARTEMiS depends on mysql and memcached. As stated above, package names may vary by distrubution, but this is generally what you should expect to install.
+#### Rasperry Pi OS
+`sudo apt install git mariadb-server python3-pip memcached libmemcached-dev `
+
+#### Debian 12
+`sudo apt install git mariadb-server python3-pip memcached libmemcached-dev default-libmysqlclient-dev pkg-config`
+
+### Optional: Install proxy
+If you intend to use a proxy (recomended for public-facing production setups), we recomend nginx
+`sudo apt install nginx`
+
+## Database setup
+### mysql_secure_installation
+If you already have your database installed and configured, and are able to log in, skip down to the [Creating the database](#creating-the-database) section below. Otherwise, setup your newly installed database.
+
+`sudo mysql_secure_installation`
+
+Leave the root password blank, do not switch to unix socket, do reset the root password to something secure, and answer yes to the rest of the prompts. You can then log into your database with `sudo mysql`
+
+### Creating the database
+Once you're logged in, run the following commands, as root, to set up our database. Make sure you note down whatever you decide to make the password for the aime account, as you will need it to configure artemis.
+
+```sql
+CREATE USER 'aime'@'localhost' IDENTIFIED BY '<password>';
+CREATE DATABASE aime;
+GRANT Alter,Create,Delete,Drop,Index,Insert,References,Select,Update ON aime.* TO 'aime'@'localhost';
+quit
+```
+We have now set up our new user, `aime`, created a database called `aime` and given our user all the permissions it needs on every table of that database.
+
+### Configure memcached
+Under the file /etc/memcached.conf, please make sure the following parameters are set:
+
+```
+# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default
+# Note that the daemon will grow to this size, but does not start out holding this much
+# memory
+
+-I 128m
+-m 1024
+```
+
+** This is mandatory to avoid memcached overload caused by Crossbeats or by massive profiles
+
+Restart memcached using:  sudo systemctl restart memcached
+
+## Getting ARTEMiS
+### Clone from gitea
+use `git clone https://gitea.tendokyu.moe/Hay1tsme/artemis.git` to pull down ARTEMiS into a folder called `artemis` created at wherever your current working directory is. `cd` into `artemis`.
+
+### Optional: Create a venv
+Python venvs are a way to install and manage packages on a per-project basis and are recomended on systems that will have multiple python scripts running on them to avoid dependancy issues. If this server will be running ARTEMiS and ONLY ARTEMiS, then it is possible to get away without creating one. If you do want to create one, you will have to install an additional package:
+
+`sudo apt install python3-venv` (like above, package name may vary depending on distro and python version)
+
+Now, simply run `python -m venv .venv` (may have to use python3 or python 3.11 instead of python) to create your virtual environment in the folder `.venv`. In order to install packages and run scripts in this environment, you have to 'activate' it by running `source .venv/bin/activate`. Your terminal should now have (venv) appended to it.
+
+### Optional: Use the develop branch
+By default, pulling down ARTEMiS from gitea will pull the `master` branch. This branch is updated less frequently, but is considered stable and ready for production use. If you'd rather have more updates, but a possibility for instability or bugs, you can switch to the develop branch by running `git checkout develop`. You can run `git checkout master` to switch back to stable.
+
+## Install python libraries
+Run `pip install -r requirements.txt` to install all of ARTEMiS' dependencies. If any installs fail, you may have missed a step in the [Install prerequisits](#install-prerequisits) section above. If you're absolutly sure you didn't, submit an issue on gitea.
+
+## Configuration
+### Copy example configs
+From the `artemis` directory, run `cp -r example_config config` to copy the example configuration files to a new folder called `config`. All of the config changes you make will be done in the `config` folder.
+
+### Optional: Generate AimeDB and Frontend JWT Secrets
+AimeDB and the frontend utalize JSON Web Tokens (JWT) for card authentication and session cookies respectivly. While generating a secret for AimeDB is optional, if you intend to run the frontend, a secret is required. You can generate a secret easily by running:
+
+`openssl rand --base64 64`
+
+With 64 being the number of bytes. You shouldn't need to go higher then 64, but you can if desired. **NOTE: When pasting secrets into the config file, make sure you remove any newlines!**
+
+### Edit `core.yaml`
+Before editing `core.yaml`, you should familiarize yourself with the name and function of each of the config options. You can find a full list in [config.md](config.md)
+
+Open `core.yaml` in the `config` folder in your prefered text editor. The only configuration option that it is absolutly mandatory to change is `aimedb`->`key`. This key must be set for the server to start, and the key must be correct, otherwise you will not be able to process aimedb requests. The correct key is floating around online, and finding it is left as an excersie to the reader. 
+
+Another option that should be changed is `database`->`password` to be the password you set when you created your database user. You did write it down somewhere, right?
+
+Since you are presumably not running the games on the same computer you're installing this server on, you're going to want to change `server`->`hostname` to be whatever hostname or IP address other PCs can reach this server by. Note that some games reject IPs and require hostnames, so setting a hostname is always recomended over an IP.
+
+### Edit game configs
+Every game has their own yaml file with settings that you may want to tweek. `InitialD Zero` and `Pokken` both have `hostname` fields in their config file that you should edit, and some games support encryption, if supplied with proper keys.
+
+### A note about IDZ
+InitialD Zero is currently the only game where it is required to specify encryption information (the AES key and at least one RSA key) for the game to start. These keys are, like the aimedb key, floating around online and will not be provided. If you don't have the keys, and don't plan on anybody connecting to your server playing InitialD Zero, it's best to set `enabled` to `False` in idz.yaml to disable the game.
+
+## Create database tables
+ARTEMiS uses alembic to manage datbase versioning. `dbutils.py` acts as a wrapper for alembic, and can execute some necessassary database functions. To create the database tables, run `python dbutils.py create`. Confirm that there are no errors, and you're good to go. If you intend to use the frontend, you may also want to run `python dbutils.py create-owner -a <your 20 digit access code here>` to create a superuser account to log in with.
+
+## Run ARTEMiS
+Once you have everything configured properly, simply run `python index.py` to start ARTEMiS. Verify that clients can connect to all services (allnet, billing, aimedb, and game servers) and setup is complete.

docs/INSTALL_WINDOWS.md

@@ -1,83 +1,77 @@
-# ARTEMiS - Windows 10/11 Guide
+# Installing ARTEMiS on Windows
-This step-by-step guide assumes that you are using a fresh install of Windows 10/11 without MySQL installed, some of the steps can be skipped if you already have an installation with MySQL 8.0 or even some of the modules already present on your environment
+This guide assumes a fresh install of Windows 10. Please be aware that due to the lack of memcached and the general woes of running a server on Windows, this is only recommended for local setups or small hosting-for-the-homies type servers.
 
-# Setup
+## Install prerequisites
-## Install Python Python 3.9 (recommended) or 3.10
+### Python
-1. Download Python 3.9 : [Link](https://www.python.org/ftp/python/3.9.13/python-3.9.13-amd64.exe)
+- Python versions from 3.8 to 3.11 work with ARTEMiS. We recommend 3.11.
-2. Install python-3.9.13-amd64.exe
+    - https://www.python.org/ftp/python/3.11.7/python-3.11.7-amd64.exe
-    1. Select Customize installation
+- Install using whichever options best suit your environment, making sure that the Python executable is on path, such that you can open CMD, type `python --version` and see the version of Python you have installed.
-    2. Make sure that pip, tcl/tk, and the for all users are checked and hit Next
+- If you already have a working version of Python installed, skip this step.
-    3. Make sure that you enable "Create shortcuts for installed applications" and "Add Python to environment variables" and hit Install
 
-## Install MySQL 8.0
+### MariaDB
-1. Download MySQL 8.0 Server : [Link](https://cdn.mysql.com//Downloads/MySQLInstaller/mysql-installer-web-community-8.0.31.0.msi)
+- It is always recommended to use MariaDB over MySQL because Oracle is a terrible company.
-2. Install mysql-installer-web-community-8.0.31.0.msi
+- While the latest release of v10 is recommended, as it is an LTS release, v11 should work fine.
-    1. Click on "Add ..." on the side
+    - https://ftp.osuosl.org/pub/mariadb//mariadb-10.11.6/winx64-packages/mariadb-10.11.6-winx64.msi
-    2. Click on the "+" next to MySQL Servers
+- REMEMBER YOUR ROOT PASSWORD SO YOU CAN LOG IN IN FUTURE STEPS.
-    3. Make sure MySQL Server 8.0.29 - X64 is under the products to be installed.
+
-    4. Hit Next and Next once installed
+### Git
-    5. Select the configuration type "Development Computer"
+- While technically optional, it is strongly recommended to obtain ARTEMiS via git clone instead of just downloading it.
-    6. Hit Next
+    - https://git-scm.com/download/win
-    7. Select "Use Legacy Authentication Method (Retain MySQL 5.x compatibility)" and hit Next
+- It is recommended to use Notepad++ as the default editor (if you have it installed), other than that, the default settings should be fine.
-    8. Enter a root password and then hit Next >
+
-    9. Leave everything under Windows Service as default and hit Next >
+### Optional: GUI database viewer
-    10. Click on Execute and for it to finish and hit Next> and then Finish
+- Having a GUI database editor is recommended but not required.
-3. Open MySQL 8.0 Command Line Client and login as your root user
+- MariaDB will try to install HeidiSQL, but we recommend DBeaver.
-4. Type those commands to create your user and the database
+    - https://dbeaver.io/download/
+
+## Obtain ARTEMiS
+### Via git (recommended)
+- `git clone https://gitea.tendokyu.moe/Hay1tsme/artemis.git` via cmd in whatever folder you want to install ARTEMiS.
+    - You can switch to the develop branch for latest changes via `git checkout develop`.
+
+### Via http download
+- Download [here](https://gitea.tendokyu.moe/Hay1tsme/artemis/archive/master.zip).
+    - Develop branch can be found [here](https://gitea.tendokyu.moe/Hay1tsme/artemis/archive/develop.zip).
+- Extract the zip file somewhere.
+
+## Database setup
+- Log into your server as root, either via GUI (recommended) or CMD
+- Create the `aime` user, replace `<password>` with a password you choose. Remember it!
 ```
-CREATE USER 'aime'@'localhost' IDENTIFIED BY 'MyStrongPass.';
+CREATE USER 'aime'@'localhost' IDENTIFIED BY '<password>';
 CREATE DATABASE aime;
 GRANT Alter,Create,Delete,Drop,Index,Insert,References,Select,Update ON aime.* TO 'aime'@'localhost';
-FLUSH PRIVILEGES;
-exit;
 ```
+- If you create the database via a GUI, make sure you grant all the above permissions.
 
-## Install Python modules
+## Create a venv
-1. Change your work path to the artemis-master folder using 'cd' and install the requirements:
+- Python virtual environments are a good way to manage packages and make dealing with python and pip easier.
-> pip install -r requirements.txt
+- `python -m pip venv venv`
+- `venv\Scripts\activate.bat` to activate the venv whenever you need to interact with ARTEMiS.
+- All the rest of the steps assume your venv is activated.
 
-## Copy/Rename the folder example_config to config
+## Install pip modules
+- `pip install -r requirements.txt`
 
-## Adjust /config/core.yaml
+## Setup configuration
+- Create a new `config` folder and copy the files in `example_config` over.
+- edit `core.yaml`
+    - Put the password you created for the aime user into the `database` section.
+    - Put in the aimedb key (YOU DO NOT GENERATE THIS KEY, FIND IT SOMEWHERE).
+    - Set your hostname to be whatever hostname or IP address games can reach your server at (many games reject localhost and 127.0.0.1).
+    - Optional: generate base64-encoded secrets for aimedb and frontend.
+    - See [config.md](docs/config.md) for a full list of options.
+- edit `idz.yaml`
+    - If you don't plan on anyone using your server to play Initial D Zero, it is best to disable it to cut down on console spam on boot.
+- Edit other game yamls
+    - Add keys, set hostnames, ports, etc. Specific settings will depend on the game. See [game_specific_info](docs/game_specific_info.md).
 
-1. Make sure to change the server listen_address to be set to your local machine IP (ex.: 192.168.1.xxx) 
+## Create Database Tables
-    - In case you want to run this only locally, set the following values:
+- `python dbutils.py create`
-```
-server:
-    listen_address: 0.0.0.0
-title: 
-    hostname: localhost
-```
-2. Adjust the proper MySQL information you created earlier
-3. Add the AimeDB key at the bottom of the file
-4. If the webui is needed, change the flag from False to True
 
-## Create the database tables for ARTEMiS
+## Firewall
-> python dbutils.py create
+- If you're planning on serving games not on your PC, open at least ports 80, 8443, and 22345 in windows firewall
+    - Also set `listen_address` to either your local IP to serve on your LAN, or `0.0.0.0` for all interfaces, to accept connections from other places.
 
-## Firewall Adjustements 
+## Start ARTEMiS
-Make sure the following ports are open both on your router and local Windows firewall in case you want to use this for public use (NOT recommended):
+- `python index.py`
-> Port 80 (TCP), 443 (TCP), 8443 (TCP), 22345 (TCP), 8080 (TCP), 8090 (TCP) **webui, 8444 (TCP) **mucha
-
-## Running the ARTEMiS instance
-> python index.py
-
-# Troubleshooting
-
-## Game does not connect to ARTEMiS Allnet server
-1. Double-check your core.yaml, the listen_address is most likely either not binded to the proper IP or the port is not opened
-
-## Game does not connect to Title Server
-1. Verify that your core.yaml is setup properly for both the server listen_address and title hostname
-2. Boot your game and verify that an AllNet response does show and if it does, attempt to open the URI that is shown under a browser such as Edge, Chrome & Firefox.
-3. If a page is shown, the server is working properly and if it doesn't, double check your port forwarding and also that you have entered the proper local IP under the Title hostname in core.yaml.
-
-## Unhandled command under AimeDB
-1. Double check your AimeDB key under core.yaml, it is incorrect.
-
-## AttributeError: module 'collections' has no attribute 'Hashable'
-1. This means the pyYAML module is obsolete, simply rerun pip with the -U (force update) flag, as shown below.
-    - Change your work path to the artemis-master (or artemis-develop) folder using 'cd' and run the following commands:
-```
-pip install -r requirements.txt -U
-```

docs/INSTALL_WINDOWS_old.md

@@ -0,0 +1,102 @@
+# ARTEMiS - Windows 10/11 Guide
+This step-by-step guide assumes that you are using a fresh install of Windows 10/11 without MySQL installed, some of the steps can be skipped if you already have an installation with MySQL 8.0 or even some of the modules already present on your environment
+
+# Setup
+## Install Python Python 3.9 (recommended) or 3.10
+1. Download Python 3.9 : [Link](https://www.python.org/ftp/python/3.9.13/python-3.9.13-amd64.exe)
+2. Install python-3.9.13-amd64.exe
+    1. Select Customize installation
+    2. Make sure that pip, tcl/tk, and the for all users are checked and hit Next
+    3. Make sure that you enable "Create shortcuts for installed applications" and "Add Python to environment variables" and hit Install
+
+## Install MySQL 8.0
+1. Download MySQL 8.0 Server : [Link](https://dev.mysql.com/get/Downloads/MySQLInstaller/mysql-installer-community-8.0.34.0.msi)
+2. Install mysql-installer-web-community-8.0.34.0.msi
+    1. Click on "Add ..." on the side
+    2. Click on the "+" next to MySQL Servers
+    3. Make sure MySQL Server 8.0.34 - X64 is under the products to be installed.
+    4. Hit Next and Next once installed
+    5. Select the configuration type "Development Computer"
+    6. Hit Next
+    7. Select "Use Legacy Authentication Method (Retain MySQL 5.x compatibility)" and hit Next
+    8. Enter a root password and then hit Next >
+    9. Leave everything under Windows Service as default and hit Next >
+    10. Click on Execute and for it to finish and hit Next> and then Finish
+3. Open MySQL 8.0 Command Line Client and login as your root user
+4. Change `<Enter Password Here>` to a new password for the user aime, type those commands to create your user and the database
+
+```sql
+CREATE USER 'aime'@'localhost' IDENTIFIED BY '<Enter Password Here>';
+CREATE DATABASE aime;
+GRANT Alter,Create,Delete,Drop,Index,Insert,References,Select,Update ON aime.* TO 'aime'@'localhost';
+FLUSH PRIVILEGES;
+exit;
+```
+
+## Install Python modules
+1. Change your work path to the artemis-master folder using 'cd' and install the requirements:
+
+```shell
+pip install -r requirements.txt
+```	
+
+## Copy/Rename the folder `example_config` to `config`
+
+## Adjust `config/core.yaml`
+
+1. Make sure to change the server `hostname` to be set to your local machine IP (ex.: 192.168.xxx.xxx) 
+    - In case you want to run this only locally, set the following values:
+
+```yaml
+server:
+    listen_address: 0.0.0.0
+title: 
+    hostname: 192.168.xxx.xxx
+```
+
+1. Adjust the proper MySQL information you created earlier
+```yaml
+database:
+  host: "localhost"
+  username: "aime"
+  password: "<Enter Password Here>"
+  name: "aime"
+```
+3. Add the AimeDB key at the bottom of the file
+4. If the webui is needed, change the flag from False to True
+
+## Create the database tables for ARTEMiS
+
+```shell
+python dbutils.py create
+```
+
+## Firewall Adjustements 
+Make sure the following ports are open both on your router and local Windows firewall in case you want to use this for public use (NOT recommended):
+> Port 80 (TCP), 443 (TCP), 8443 (TCP), 22345 (TCP), 8080 (TCP), 8090 (TCP) **webui, 8444 (TCP) **mucha
+
+## Running the ARTEMiS instance
+```shell
+python index.py
+```
+
+# Troubleshooting
+
+## Game does not connect to ARTEMiS Allnet server
+1. Double-check your core.yaml, the listen_address is most likely either not binded to the proper IP or the port is not opened
+
+## Game does not connect to Title Server
+1. Verify that your core.yaml is setup properly for both the server listen_address and title hostname
+2. Boot your game and verify that an AllNet response does show and if it does, attempt to open the URI that is shown under a browser such as Edge, Chrome & Firefox.
+3. If a page is shown, the server is working properly and if it doesn't, double check your port forwarding and also that you have entered the proper local IP under the Title hostname in core.yaml.
+
+## Unhandled command under AimeDB
+1. Double check your AimeDB key under core.yaml, it is incorrect.
+
+## AttributeError: module 'collections' has no attribute 'Hashable'
+1. This means the pyYAML module is obsolete, simply rerun pip with the -U (force update) flag, as shown below.
+    - Change your work path to the artemis-master (or artemis-develop) folder using 'cd' and run the following commands:
+
+```shell
+pip install -r requirements.txt -U
+```

docs/config.md

@@ -1,15 +1,24 @@
 # ARTEMiS Configuration
 ## Server
 - `listen_address`: IP Address or hostname that the server will listen for connections on. Set to 127.0.0.1 for local only, or 0.0.0.0 for all interfaces. Default `127.0.0.1`
+- `hostname`: Hostname that gets sent to clients to tell them where to connect. Games must be able to connect to your server via the hostname or IP you spcify here. Note that most games will reject `localhost` or `127.0.0.1`. Default `localhost`
+- `port`: Port that the server will listen for connections on. Default `80`
+- `ssl_key`: Location of the ssl server key for the secure title server. Ignored if you don't use SSL. Default `cert/title.key`
+- `ssl_cert`: Location of the ssl server certificate for the secure title server. Must not be a self-signed SSL. Ignored if you don't use SSL. Default `cert/title.pem`
 - `allow_user_registration`: Allows users to register in-game via the AimeDB `register` function. Disable to be able to control who can use cards on your server. Default `True`
 - `allow_unregistered_serials`: Allows games that do not have registered keychips to connect and authenticate. Disable to restrict who can connect to your server. Recomended to disable for production setups. Default `True`
 - `name`: Name for the server, used by some games in their default MOTDs. Default `ARTEMiS`
-- `is_develop`: Flags that the server is a development instance without a proxy standing in front of it. Setting to `False` tells the server not to listen for SSL, because the proxy should be handling all SSL-related things, among other things. Default `True`
+- `is_develop`: Flags that the server is a development instance, and enables some useful development features. Disable for production setups. Default `True`.
+- `is_using_proxy`: Flags that you'll be using some other software, such as nginx, to proxy requests, and to send `proxy_port` or `proxy_port_ssl` to games instead of `port`. Default `False`
+- `proxy_port`: Which port your front-facing proxy will be listening on. Ignored if `is_using_proxy` is `False` or if set to `0`. Default `0`
+- `proxy_port`: Which port your front-facing proxy will be listening for ssl connections on. Ignored if `is_using_proxy` is `False` or if set to `0`. Default `0`
 - `log_dir`: Directory to store logs. Server MUST have read and write permissions to this directory or you will have issues. Default `logs`
+- `check_arcade_ip`: Checks IPs against the `arcade` table in the database, if one is defined. Default `False`
+- `strict_ip_checking`: Rejects clients if there is no IP in the `arcade` table for the respective arcade. Default `False`
 ## Title
 - `loglevel`: Logging level for the title server. Default `info`
-- `hostname`: Hostname that gets sent to clients to tell them where to connect. Games must be able to connect to your server via the hostname or IP you spcify here. Note that most games will reject `localhost` or `127.0.0.1`. Default `localhost`
+- `reboot_start_time`: 24 hour JST time that clients will see as the start of maintenance period, ex `04:00`. Leave blank for no maintenance time. Default: `""`
-- `port`: Port that the title server will listen for connections on. Set to 0 to use the Allnet handler to reduce the port footprint. Default `8080`
+- `reboot_end_time`: 24 hour JST time that clients will see as the end of maintenance period, ex `05:00`. Leave blank for no maintenance time. Default: `""`
 ## Database
 - `host`: Host of the database. Default `localhost`
 - `username`: Username of the account the server should connect to the database with. Default `aime`
@@ -17,24 +26,32 @@
 - `name`: Name of the database the server should expect. Default `aime`
 - `port`: Port the database server is listening on. Default `3306`
 - `protocol`: Protocol used in the connection string, e.i `mysql` would result in `mysql://...`. Default `mysql`
-- `sha2_password`: Weather or not the password in the connection string should be hashed via SHA2. Default `False`
+- `sha2_password`: Whether or not the password in the connection string should be hashed via SHA2. Default `False`
-- `loglevel`: Logging level for the database. Default `warn`
+- `loglevel`: Logging level for the database. Default `info`
-- `user_table_autoincrement_start`: What the `aime_user` table ID autoincrememnt should start with. Default `10000`
 - `memcached_host`: Host of the memcached server. Default `localhost`
 ## Frontend
-- `enable`: Weather or not the frontend should be enabled. Default `False`
+- `enable`: Whether or not the frontend servlet should run. Frontend can still be run via `python -m uvicorn core.frontend:app` even if this is set to `False`. Default `False`
-- `port`: Port the frontend should listen for connections on. Default `8090`
+- `port`: Port the frontend should listen on. Default `8080`
 - `loglevel`: Logging level for the frontend server. Default `info`
+- `secret`: Base64-encoded JWT secret for session cookies, generated by you. Default `""`
 ## Allnet
+- `standalone`: Whether allnet should launch it's own servlet on it's own port, or be part of the main servlet on the default port. Disable if you either have something proxying `naominet.jp` requests to port 80, or have port 80 set in `server` -> `port`
+- `port`: Port the allnet server should listen for connections on if it's running standalone. Games are hardcoded to ask for port `80` so only change if you have a proxy redirecting properly. Ignored if `standalone` is `False`. Default `80`
 - `loglevel`: Logging level for the allnet server. Default `info`
-- `port`: Port the allnet server should listen for connections on. Games are hardcoded to ask for port `80` so only change if you have a proxy redirecting properly. Default `80`
 - `allow_online_updates`: Allow allnet to distribute online updates via DownloadOrders. This system is currently non-functional, so leave it disabled. Default `False`
+- `update_cfg_folder`: Folder where delivery INI files will be checked for. Ignored if `allow_online_updates` is `False`. Default `""`
 ## Billing
-- `port`: Port the billing server should listen for connections on. Games are hardcoded to ask for port `8443` so only change if you have a proxy redirecting properly. Set to 0 to use the allnet handler to reduce the number of ports the server eats up. Default `8443`
+- `standalone`: Whether the billing server should launch it's own servlet on it's own port, or be part of the main servlet on the default port. Setting this to `True` requires that you have `ssl_key` and `ssl_cert` set. Default `False`
-- `ssl_key`: Location of the ssl server key for the billing server. Ignored if `port` is set to `0` or `is_develop` set to `False`. Default `cert/server.key`
+- `loglevel`: Logging level for the billing server. Default `info`
-- `ssl_cert`: Location of the ssl server certificate for the billing server. Must match the CA distributed to users or the billing server will not connect. Ignored if `port` is set to `0` or `is_develop` is set to `False`. Default `cert/server.pem`
+- `port`: Port the billing server should listen for connections on. Games are hardcoded to ask for port `8443` so only change if you have a proxy redirecting properly. Ignored if `standalone` is `False`. Default `8443`
+- `ssl_key`: Location of the ssl server key for the billing server. Ignored if `standalone` is `False`. Default `cert/server.key`
+- `ssl_cert`: Location of the ssl server certificate for the billing server. Ignored if `standalone` is `False`.  Must match the CA distributed to users or the billing server will not connect. Default `cert/server.pem`
 - `signing_key`: Location of the RSA Private key used to sign billing requests. Must match the public key distributed to users or the billing server will not connect. Default `cert/billing.key`
 ## Aimedb
+- `enable`: Whether or not aimedb should run. Default `True`
+- `listen_address`: IP Address or hostname that the aimedb server will listen for connections on. Leave this blank to use the listen address under `server`. Default `""`
 - `loglevel`: Logging level for the aimedb server. Default `info`
 - `port`: Port the aimedb server should listen for connections on. Games are hardcoded to ask for port `22345` so only change if you have a proxy redirecting properly. Default `22345`
 - `key`: Key to encrypt/decrypt aimedb requests and responses. MUST be set or the server will not start. If set incorrectly, your server will not properly handle aimedb requests. Default `""`
+- `id_secret`: Base64-encoded JWT secret for Sega Auth IDs. Leaving this blank disables this feature. Default `""`
+- `id_lifetime_seconds`: Number of secons a JWT generated should be valid for. Default `86400` (1 day)

docs/game_specific_info.md

@@ -6,45 +6,55 @@ the corresponding importer and database upgrades.
 **Important: The described database upgrades are only required if you are using an old database schema, f.e. still
 using the megaime database. Clean installations always create the latest database structure!**
 
+To upgrade the core database and the database for every game, execute:
+
+```shell
+python dbutils.py autoupgrade
+```
+
 # Table of content
 
 - [Supported Games](#supported-games)
-    - [Chunithm](#chunithm)
+    - [CHUNITHM](#chunithm)
     - [crossbeats REV.](#crossbeats-rev)
     - [maimai DX](#maimai-dx)
     - [O.N.G.E.K.I.](#o-n-g-e-k-i)
     - [Card Maker](#card-maker)
     - [WACCA](#wacca)
+    - [Sword Art Online Arcade](#sao)
+	- [Initial D THE ARCADE](#initial-d-the-arcade)
 
 
 # Supported Games
 
 Games listed below have been tested and confirmed working.
 
-## Chunithm
+## CHUNITHM
 
 ### SDBT
 
 | Version ID | Version Name          |
-|------------|--------------------|
+| ---------- | --------------------- |
-| 0          | Chunithm           |
+| 0          | CHUNITHM              |
-| 1          | Chunithm+          |
+| 1          | CHUNITHM PLUS         |
-| 2          | Chunithm Air       |
+| 2          | CHUNITHM AIR          |
-| 3          | Chunithm Air +     |
+| 3          | CHUNITHM AIR PLUS     |
-| 4          | Chunithm Star      |
+| 4          | CHUNITHM STAR         |
-| 5          | Chunithm Star +    |
+| 5          | CHUNITHM STAR PLUS    |
-| 6          | Chunithm Amazon    |
+| 6          | CHUNITHM AMAZON       |
-| 7          | Chunithm Amazon +  |
+| 7          | CHUNITHM AMAZON PLUS  |
-| 8          | Chunithm Crystal   |
+| 8          | CHUNITHM CRYSTAL      |
-| 9          | Chunithm Crystal + |
+| 9          | CHUNITHM CRYSTAL PLUS |
-| 10         | Chunithm Paradise  |
+| 10         | CHUNITHM PARADISE     |
 
 ### SDHD/SDBT
 
 | Version ID | Version Name        |
-|------------|-----------------|
+| ---------- | ------------------- |
-| 11         | Chunithm New!!  |
+| 11         | CHUNITHM NEW!!      |
-| 12         | Chunithm New!!+ |
+| 12         | CHUNITHM NEW PLUS!! |
+| 13         | CHUNITHM SUN        |
+| 14         | CHUNITHM SUN PLUS   |
 
 
 ### Importer
@@ -52,28 +62,90 @@ Games listed below have been tested and confirmed working.
 In order to use the importer locate your game installation folder and execute:
 
 ```shell
-python read.py --series SDBT --version <version ID> --binfolder /path/to/game/folder --optfolder /path/to/game/option/folder
+python read.py --game SDBT --version <version ID> --binfolder /path/to/game/folder --optfolder /path/to/game/option/folder
 ```
 
 The importer for Chunithm will import: Events, Music, Charge Items and Avatar Accesories.
 
+### Config
+
+Config file is located in `config/chuni.yaml`.
+
+| Option           | Info                                                                                                           |
+|------------------|----------------------------------------------------------------------------------------------------------------|
+| `news_msg`       | If this is set, the news at the top of the main screen will be displayed (up to Chunithm Paradise Lost)        |
+| `name`           | If this is set, all players that are not on a team will use this one by default.                               |
+| `use_login_bonus`| This is used to enable the login bonuses                                                                       |
+| `crypto`         | This option is used to enable the TLS Encryption                                                               |
+
+
+**If you would like to use network encryption, the following will be required underneath but key, iv and hash are required:**
+
+```yaml
+crypto:
+  encrypted_only: False
+  keys:
+    13: ["0000000000000000000000000000000000000000000000000000000000000000", "00000000000000000000000000000000", "0000000000000000"]
+```
+
 ### Database upgrade
 
-Always make sure your database (tables) are up-to-date, to do so go to the `core/data/schema/versions` folder and see
+Always make sure your database (tables) are up-to-date:
-which version is the latest, f.e. `SDBT_3_upgrade.sql`. In order to upgrade to version 3 in this case you need to
-perform all previous updates as well:
 
 ```shell
-python dbutils.py --game SDBT --version 2 upgrade
+python dbutils.py --game SDBT upgrade
-python dbutils.py --game SDBT --version 3 upgrade
 ```
 
+### Online Battle
+
+**Only matchmaking (with your imaginary friends) is supported! Online Battle does not (yet?) work!**
+
+The first person to start the Online Battle (now called host) will create a "matching room" with a given `roomId`, after that max 3 other people can join the created room.
+Non used slots during the matchmaking will be filled with CPUs after the timer runs out.
+As soon as a new member will join the room the timer will jump back to 60 secs again.
+Sending those 4 messages to all other users is also working properly.
+In order to use the Online Battle every user needs the same ICF, same rom version and same data version!
+If a room is full a new room will be created if another user starts an Online Battle.
+After a failed Online Battle the room will be deleted. The host is used for the timer countdown, so if the connection failes to the host the timer will stop and could create a "frozen" state.
+
+#### Information/Problems:
+
+- Online Battle uses UDP hole punching and opens port 50201?
+- `reflectorUri` seems related to that?
+- Timer countdown should be handled globally and not by one user
+- Game can freeze or can crash if someone (especially the host) leaves the matchmaking
+
+### Rivals
+
+You can configure up to 4 rivals in Chunithm on a per-user basis. There is no UI to do this currently, so in the database, you can do this:
+```sql
+INSERT INTO aime.chuni_item_favorite (user, version, favId, favKind) VALUES (<user1>, <version>, <user2>, 2);
+INSERT INTO aime.chuni_item_favorite (user, version, favId, favKind) VALUES (<user2>, <version>, <user1>, 2);
+```
+Note that the version **must match**, otherwise song lookup may not work.
+
+### Teams
+
+You can also configure teams for users to be on. There is no UI to do this currently, so in the database, you can do this:
+```sql
+INSERT INTO aime.chuni_profile_team (teamName) VALUES (<teamName>);
+```
+Team names can be regular ASCII, and they will be displayed ingame.
+
+### Favorite songs
+You can set the songs that will be in a user's Favorite Songs category using the following SQL entries:
+```sql
+INSERT INTO aime.chuni_item_favorite (user, version, favId, favKind) VALUES (<user>, <version>, <songId>, 1);
+```
+The songId is based on the actual ID within your version of Chunithm.
+
+
 ## crossbeats REV.
 
 ### SDCA
 
 | Version ID | Version Name                       |
-|------------|------------------------------------|
+| ---------- | ---------------------------------- |
 | 0          | crossbeats REV.                    |
 | 1          | crossbeats REV. SUNRISE            |
 | 2          | crossbeats REV. SUNRISE S2         |
@@ -84,7 +156,7 @@ python dbutils.py --game SDBT --version 3 upgrade
 In order to use the importer you need to use the provided `Export.csv` file:
 
 ```shell
-python read.py --series SDCA --version <version ID> --binfolder titles/cxb/data
+python read.py --game SDCA --version <version ID> --binfolder titles/cxb/data
 ```
 
 The importer for crossbeats REV. will import Music.
@@ -93,56 +165,67 @@ The importer for crossbeats REV. will import Music.
 
 Config file is located in `config/cxb.yaml`.
 
-| Option                 | Info                                                       |
-|------------------------|------------------------------------------------------------|
-| `hostname`             | Requires a proper `hostname` (not localhost!) to run       |
-| `ssl_enable`           | Enables/Disables the use of the `ssl_cert` and `ssl_key`   |
-| `port`                 | Set your unsecure port number                              |
-| `port_secure`          | Set your secure/SSL port number                            |
-| `ssl_cert`, `ssl_key`  | Enter your SSL certificate (requires not self signed cert) |
-
-
 ## maimai DX
 
-### SDEZ
+### Versions
 
-| Version ID | Version Name            |
+| Game Code | Version ID | Version Name            |
-|------------|-------------------------|
+| --------- | ---------- | ----------------------- |
-| 0          | maimai DX               |
+| SBXL      | 0          | maimai                  |
-| 1          | maimai DX PLUS          |
+| SBXL      | 1          | maimai PLUS             |
-| 2          | maimai DX Splash        |
+| SBZF      | 2          | maimai GreeN            |
-| 3          | maimai DX Splash PLUS   |
+| SBZF      | 3          | maimai GreeN PLUS       |
-| 4          | maimai DX Universe      |
+| SDBM      | 4          | maimai ORANGE           |
-| 5          | maimai DX Universe PLUS |
+| SDBM      | 5          | maimai ORANGE PLUS      |
-| 6          | maimai DX Festival      |
+| SDCQ      | 6          | maimai PiNK             |
+| SDCQ      | 7          | maimai PiNK PLUS        |
+| SDDK      | 8          | maimai MURASAKi         |
+| SDDK      | 9          | maimai MURASAKi PLUS    |
+| SDDZ      | 10         | maimai MiLK             |
+| SDDZ      | 11         | maimai MiLK PLUS        |
+| SDEY      | 12         | maimai FiNALE           |
+| SDEZ      | 13         | maimai DX               |
+| SDEZ      | 14         | maimai DX PLUS          |
+| SDEZ      | 15         | maimai DX Splash        |
+| SDEZ      | 16         | maimai DX Splash PLUS   |
+| SDEZ      | 17         | maimai DX UNiVERSE      |
+| SDEZ      | 18         | maimai DX UNiVERSE PLUS |
+| SDEZ      | 19         | maimai DX FESTiVAL      |
+| SDEZ      | 20         | maimai DX FESTiVAL PLUS |
 
 ### Importer
 
 In order to use the importer locate your game installation folder and execute:
-
+DX:
 ```shell
-python read.py --series SDEZ --version <version ID> --binfolder /path/to/game/folder --optfolder /path/to/game/option/folder
+python read.py --game <Game Code> --version <Version ID> --binfolder /path/to/StreamingAssets --optfolder /path/to/game/option/folder
+```
+Pre-DX:
+```shell
+python read.py --game <Game Code> --version <Version ID> --binfolder /path/to/data --optfolder /path/to/patch/data
 ```
-
 The importer for maimai DX will import Events, Music and Tickets.
 
-**NOTE: It is required to use the importer because the game will
+The importer for maimai Pre-DX will import Events and Music. Not all games will have patch data. Milk - Finale have file encryption, and need an AES key. That key is not provided by the developers. For games that do use encryption, provide the key, as a hex string, with the `--extra` flag. Ex `--extra 00112233445566778899AABBCCDDEEFF`
-crash without Events!**
+
+**Important: It is required to use the importer because some games may not function properly or even crash without Events!**
 
 ### Database upgrade
 
-Always make sure your database (tables) are up-to-date, to do so go to the `core/data/schema/versions` folder and see which version is the latest, f.e. `SDEZ_2_upgrade.sql`. In order to upgrade to version 2 in this case you need to perform all previous updates as well:
+Always make sure your database (tables) are up-to-date:
 
 ```shell
-python dbutils.py --game SDEZ --version 2 upgrade
+python dbutils.py --game SDEZ upgrade
 ```
 
+Pre-Dx uses the same database as DX, so only upgrade using the SDEZ game code!
+
 ## Hatsune Miku Project Diva
 
 ### SBZV
 
 | Version ID | Version Name                    |
-|------------|---------------------------------|
+| ---------- | ------------------------------- |
 | 0          | Project Diva Arcade             |
 | 1          | Project Diva Arcade Future Tone |
 
@@ -152,7 +235,7 @@ python dbutils.py --game SDEZ --version 2 upgrade
 In order to use the importer locate your game installation folder and execute:
 
 ```shell
-python read.py --series SBZV --version <version ID> --binfolder /path/to/game/data/diva --optfolder /path/to/game/data/diva/mdata
+python read.py --game SBZV --version <version ID> --binfolder /path/to/game/data/diva --optfolder /path/to/game/data/diva/mdata
 ```
 
 The importer for Project Diva Arcade will all required data in order to use
@@ -163,21 +246,20 @@ the Shop, Modules and Customizations.
 Config file is located in `config/diva.yaml`.
 
 | Option               | Info                                                                                            |
-|----------------------|-------------------------------------------------------------------------------------------------|
+| -------------------- | ----------------------------------------------------------------------------------------------- |
 | `unlock_all_modules` | Unlocks all modules (costumes) by default, if set to `False` all modules need to be purchased   |
 | `unlock_all_items`   | Unlocks all items (customizations) by default, if set to `False` all items need to be purchased |
 
+### Custom PV Lists (databanks)
+
+In order to use custom PV Lists, simply drop in your .dat files inside of /titles/diva/data/ and make sure they are called PvList0.dat, PvList1.dat, PvList2.dat, PvList3.dat and PvList4.dat exactly.
 
 ### Database upgrade
 
-Always make sure your database (tables) are up-to-date, to do so go to the `core/data/schema/versions` folder and see
+Always make sure your database (tables) are up-to-date:
-which version is the latest, f.e. `SBZV_4_upgrade.sql`. In order to upgrade to version 4 in this case you need to
-perform all previous updates as well:
 
 ```shell
-python dbutils.py --game SBZV --version 2 upgrade
+python dbutils.py --game SBZV upgrade
-python dbutils.py --game SBZV --version 3 upgrade
-python dbutils.py --game SBZV --version 4 upgrade
 ```
 
 ## O.N.G.E.K.I.
@@ -185,15 +267,15 @@ python dbutils.py --game SBZV --version 4 upgrade
 ### SDDT
 
 | Version ID | Version Name               |
-|------------|----------------------------|
+| ---------- | -------------------------- |
 | 0          | O.N.G.E.K.I.               |
 | 1          | O.N.G.E.K.I. +             |
-| 2          | O.N.G.E.K.I. Summer        |
+| 2          | O.N.G.E.K.I. SUMMER        |
-| 3          | O.N.G.E.K.I. Summer +      |
+| 3          | O.N.G.E.K.I. SUMMER +      |
-| 4          | O.N.G.E.K.I. Red           |
+| 4          | O.N.G.E.K.I. R.E.D.        |
-| 5          | O.N.G.E.K.I. Red +         |
+| 5          | O.N.G.E.K.I. R.E.D. +      |
-| 6          | O.N.G.E.K.I. Bright        |
+| 6          | O.N.G.E.K.I. bright        |
-| 7          | O.N.G.E.K.I. Bright Memory |
+| 7          | O.N.G.E.K.I. bright MEMORY |
 
 
 ### Importer
@@ -201,7 +283,7 @@ python dbutils.py --game SBZV --version 4 upgrade
 In order to use the importer locate your game installation folder and execute:
 
 ```shell
-python read.py --series SDDT --version <version ID> --binfolder /path/to/game/folder --optfolder /path/to/game/option/folder
+python read.py --game SDDT --version <version ID> --binfolder /path/to/game/folder --optfolder /path/to/game/option/folder
 ```
 
 The importer for O.N.G.E.K.I. will all all Cards, Music and Events.
@@ -213,44 +295,118 @@ The importer for O.N.G.E.K.I. will all all Cards, Music and Events.
 Config file is located in `config/ongeki.yaml`.
 
 | Option           | Info                                                                                                           |
-|------------------|----------------------------------------------------------------------------------------------------------------|
+| ---------------- | -------------------------------------------------------------------------------------------------------------- |
 | `enabled_gachas` | Enter all gacha IDs for Card Maker to work, other than default may not work due to missing cards added to them |
+| `crypto`         | This option is used to enable the TLS Encryption                                                               |
 
 Note: 1149 and higher are only for Card Maker 1.35 and higher and will be ignored on lower versions.
 
+**If you would like to use network encryption, the following will be required underneath but key, iv and hash are required:**
+
+```yaml
+crypto:
+  encrypted_only: False
+  keys:
+    7: ["0000000000000000000000000000000000000000000000000000000000000000", "00000000000000000000000000000000", "0000000000000000"]
+```
+
 ### Database upgrade
 
-Always make sure your database (tables) are up-to-date, to do so go to the `core/data/schema/versions` folder and see
+Always make sure your database (tables) are up-to-date:
-which version is the latest, f.e. `SDDT_4_upgrade.sql`. In order to upgrade to version 4 in this case you need to
-perform all previous updates as well:
 
 ```shell
-python dbutils.py --game SDDT --version 2 upgrade
+python dbutils.py --game SDDT upgrade
-python dbutils.py --game SDDT --version 3 upgrade
-python dbutils.py --game SDDT --version 4 upgrade
 ```
 
+### Controlling Events (Ranking Event, Technical Challenge Event, Mission Event)
+
+Events are controlled by 2 types of enabled events:
+- RankingEvent (type 6), TechChallengeEvent (type 17)
+- AcceptRankingEvent (type 7), AcceptTechChallengeEvent (type 18)
+
+Both Ranking and Accept must be enabled for event to function properly
+
+Event will run for the time specified in startDate and endDate
+
+AcceptRankingEvent and AcceptTechChallengeEvent are reward period for events, which specify from what startDate until endDate you can collect the rewards for attending the event, so the reward period must start in the future, e.g. :
+
+- RankingEvent startDate 2023-12-01 - endDate 2023-12-30 - period in which whole event is running
+- AcceptRankingEvent startDate 2023-12-23 - endDate 2023-12-30 - period in which you can collect rewards for the event
+
+If player misses the AcceptRankingEvent period - ranking will be invalidated and receive lowest reward from the event (typically 500x money)
+
+Technical Challenge Song List:
+
+Songs that are used for Technical Challenge are not stored anywhere in data files, so you need to fill the database table by yourself, you can gather all songs that should be in Technical Challenges from ONGEKI japanese wikis, or, you can create your own sets:
+
+Database table : `ongeki_static_tech_music`
+
+```
+id: Id in table, just increment for each entry
+version: version of the game you want the tech challenge to be in (from RED and up)
+eventId: Id of the event in ongeki_static_events, insert the Id of the TechChallengeEvent (type 17) you want the song be assigned to
+musicId: Id of the song you want to add, use songId from ongeki_static_music table
+level: Difficulty of the song you want to track during the event, from 0(basic) to 3(master)
+
+```
+
+Current implementation of Ranking and Technical Challenge Events are updated on every profile save to the Network, and Ranked on each player login, in official specification, calculation for current rank on the network should be done in the maintenance window
+
+Mission Event (type 13) is a monthly type of event, which is used when another event doesn't have it's own Ranking or Technical Challenge Event running, only one Mission Event should be running at a time, so enable only the specific Mission you want to run currently on the Network
+
+If you're often trying fresh cards, registering new profiles etc., you can also consider disabling all Announcement Events (type 1), as it will disable all the banners that pop up on login (they show up only once though, so if you click through them once they won't show again)
+
+Event type 2 in Database are Advertisement Movies, enable only 1 you want to currently play, and disable others
+
+
+Present and Reward List - populate reward list using read.py
+
+Create present for players by adding an entry in `ongeki_static_present_list`
+```
+id: unique for each entry
+version: game version you want the present be in
+presentId: id of the present - starts with 1001 and go up from that, must be unique for each reward(don't set multiple rewardIds with same presentId)
+presentName: present name which will be shown on the bottom when received
+rewardId: ID of item from ongeki_static_rewards
+stock: how many you want to give (like 5 copies of same card, or 10000 money, etc.)
+message: no idea, can be left empty
+startDate: date when to start giving out
+endDate: date when ends
+```
+
+After inserting present to the table, add the presentId into players `ongeki_static_item`, where itemKind is 9, itemId is the presentId, and stock set 1 and isValid to 1
+
+After that, on next login the present should be received (or whenever it supposed to happen)
+
+
+
 ## Card Maker
 
 ### SDED
 
 | Version ID | Version Name    |
-|------------|-----------------|
+| ---------- | --------------- |
-| 0          | Card Maker 1.34 |
+| 0          | Card Maker 1.30 |
 | 1          | Card Maker 1.35 |
 
 
 ### Support status
 
-* Card Maker 1.34:
+#### Card Maker 1.30:
-  * Chunithm New!!: Yes
+* CHUNITHM NEW!!: Yes
-  * maimai DX Universe: Yes
+* maimai DX UNiVERSE: Yes
-  * O.N.G.E.K.I. Bright: Yes
+* O.N.G.E.K.I. bright: Yes
 
-* Card Maker 1.35:
+#### Card Maker 1.35:
-  * Chunithm New!!+: Yes
+* CHUNITHM: 
-  * maimai DX Universe PLUS: Yes
+  * NEW!!: Yes
-  * O.N.G.E.K.I. Bright Memory: Yes
+  * NEW PLUS!!: Yes (added in A028)
+  * SUN: Yes (added in A032)
+* maimai DX:
+  * UNiVERSE PLUS: Yes
+  * FESTiVAL: Yes (added in A031)
+  * FESTiVAL PLUS: Yes (added in A035)
+* O.N.G.E.K.I. bright MEMORY: Yes
 
 
 ### Importer
@@ -259,28 +415,63 @@ In order to use the importer you need to use the provided `.csv` files (which ar
 option folders:
 
 ```shell
-python read.py --series SDED --version <version ID> --binfolder titles/cm/cm_data --optfolder /path/to/cardmaker/option/folder
+python read.py --game SDED --version <version ID> --binfolder titles/cm/cm_data --optfolder /path/to/cardmaker/option/folder
 ```
 
 **If you haven't already executed the O.N.G.E.K.I. importer, make sure you import all cards!**
 
 ```shell
-python read.py --series SDDT --version <version ID> --binfolder /path/to/game/folder --optfolder /path/to/game/option/folder
+python read.py --game SDDT --version <version ID> --binfolder /path/to/game/folder --optfolder /path/to/game/option/folder
 ```
 
-Also make sure to import all maimai and Chunithm data as well:
+Also make sure to import all maimai DX and CHUNITHM data as well:
 
 ```shell
-python read.py --series SDED --version <version ID> --binfolder /path/to/cardmaker/CardMaker_Data
+python read.py --game SDED --version <version ID> --binfolder /path/to/cardmaker/CardMaker_Data
 ```
 
-The importer for Card Maker will import all required Gachas (Banners) and cards (for maimai/Chunithm) and the hardcoded
+The importer for Card Maker will import all required Gachas (Banners) and cards (for maimai DX/CHUNITHM) and the hardcoded
 Cards for each Gacha (O.N.G.E.K.I. only).
 
 **NOTE: Without executing the importer Card Maker WILL NOT work!**
 
 
-### O.N.G.E.K.I. Gachas
+### Config setup
+
+Make sure to update your `config/cardmaker.yaml` with the correct version for each game. To get the current version required to run a specific game, open every opt (Axxx) folder descending until you find all three folders:
+
+- `MU3`: O.N.G.E.K.I.
+- `MAI`: maimai DX
+- `CHU`: CHUNITHM
+
+Inside each folder is a `DataConfig.xml` file, for example:
+
+`MU3/DataConfig.xml`:
+```xml
+  <cardMakerVersion>
+    <major>1</major>
+    <minor>35</minor>
+    <release>3</release>
+  </cardMakerVersion>
+```
+
+Now update your `config/cardmaker.yaml` with the correct version number, for example:
+
+```yaml
+version:
+  1: # Card Maker 1.35
+    ongeki: 1.35.03
+```
+
+For now you also need to update your `config/ongeki.yaml` with the correct version number, for example:
+
+```yaml
+version:
+  7: # O.N.G.E.K.I. bright MEMORY
+    card_maker: 1.35.03
+```
+
+### O.N.G.E.K.I.
 
 Gacha "無料ガチャ" can only pull from the free cards with the following probabilities: 94%: R, 5% SR and 1% chance of
 getting an SSR card
@@ -293,26 +484,30 @@ and 3% chance of getting an SSR card
 All other (limited) gachas can pull from every card added to ongeki_static_cards but with the promoted cards
 (click on the green button under the banner) having a 10 times higher chance to get pulled
 
-### Chunithm Gachas
+### CHUNITHM
 
-All cards in Chunithm (basically just the characters) have the same rarity to it just pulls randomly from all cards
+All cards in CHUNITHM (basically just the characters) have the same rarity to it just pulls randomly from all cards
 from a given gacha but made sure you cannot pull the same card twice in the same 5 times gacha roll.
 
+### maimai DX
+
+Printed maimai DX cards: Freedom (`cardTypeId=6`) or Gold Pass (`cardTypeId=4`) can now be selected during the login process. You can only have ONE Freedom and ONE Gold Pass active at a given time. The cards will expire after 15 days.
+
+Thanks GetzeAvenue for the `selectedCardList` rarity hint!
+
 ### Notes
 
-Card Maker 1.34 will only load an O.N.G.E.K.I. Bright profile (1.30). Card Maker 1.35 will only load an O.N.G.E.K.I.
+Card Maker 1.30-1.34 will only load an O.N.G.E.K.I. Bright profile (1.30). Card Maker 1.35+ will only load an O.N.G.E.K.I.
 Bright Memory profile (1.35).
-The gachas inside the `ongeki.yaml` will make sure only the right gacha ids for the right CM version will be loaded.
+The gachas inside the `config/ongeki.yaml` will make sure only the right gacha ids for the right CM version will be loaded.
 Gacha IDs up to 1140 will be loaded for CM 1.34 and all gachas will be loaded for CM 1.35.
 
-**NOTE: There is currently no way to load/use the (printed) maimai DX cards!**
-
 ## WACCA
 
 ### SDFE
 
 | Version ID | Version Name  |
-|------------|---------------|
+| ---------- | ------------- |
 | 0          | WACCA         |
 | 1          | WACCA S       |
 | 2          | WACCA Lily    |
@@ -325,7 +520,7 @@ Gacha IDs up to 1140 will be loaded for CM 1.34 and all gachas will be loaded fo
 In order to use the importer locate your game installation folder and execute:
 
 ```shell
-python read.py --series SDFE --version <version ID> --binfolder /path/to/game/WindowsNoEditor/Mercury/Content
+python read.py --game SDFE --version <version ID> --binfolder /path/to/game/WindowsNoEditor/Mercury/Content
 ```
 
 The importer for WACCA will import all Music data.
@@ -335,7 +530,7 @@ The importer for WACCA will import all Music data.
 Config file is located in `config/wacca.yaml`.
 
 | Option             | Info                                                                        |
-|--------------------|-----------------------------------------------------------------------------|
+| ------------------ | --------------------------------------------------------------------------- |
 | `always_vip`       | Enables/Disables VIP, if disabled it needs to be purchased manually in game |
 | `infinite_tickets` | Always set the "unlock expert" tickets to 5                                 |
 | `infinite_wp`      | Sets the user WP to `999999`                                                |
@@ -344,9 +539,229 @@ Config file is located in `config/wacca.yaml`.
 
 ### Database upgrade
 
-Always make sure your database (tables) are up-to-date, to do so go to the `core/data/schema/versions` folder and see which version is the latest, f.e. `SDFE_3_upgrade.sql`. In order to upgrade to version 3 in this case you need to perform all previous updates as well:
+Always make sure your database (tables) are up-to-date:
 
 ```shell
-python dbutils.py --game SDFE --version 2 upgrade
+python dbutils.py --game SDFE upgrade
-python dbutils.py --game SDFE --version 3 upgrade
 ```
+
+### VIP Rewards
+Below is a list of VIP rewards. Currently, VIP is not implemented, and thus these are not obtainable. These 23 rewards were distributed once per month for VIP users on the real network.
+
+	Plates:
+		211004 リッチ
+		211018 特盛えりざべす
+		211025 イースター
+		211026 特盛りりぃ
+		311004 ファンシー
+		311005 インカンテーション
+		311014 夜明け
+		311015 ネイビー
+		311016 特盛るーん
+	
+	Ring Colors:
+		203002 Gold Rushイエロー
+		203009 トロピカル
+		303005 ネイチャー
+	
+	Icons:
+		202020 どらみんぐ
+		202063 ユニコーン
+		202086 ゴリラ
+		302014 ローズ
+		302015 ファラオ
+		302045 肉球
+		302046 WACCA
+		302047 WACCA Lily
+		302048 WACCA Reverse
+	
+	Note Sound Effect:
+		205002 テニス
+		205008 シャワー
+		305003 タンバリンMk-Ⅱ
+
+## SAO
+
+### SDEW
+
+| Version ID | Version Name |
+| ---------- | ------------ |
+| 0          | SAO          |
+
+
+### Importer
+
+In order to use the importer locate your game installation folder and execute:
+
+```shell
+python read.py --game SDEW --version 0 --binfolder /titles/sao/data/
+```
+
+The importer for SAO will import all items, heroes, support skills and titles data.
+
+### Config
+
+Config file is located in `config/sao.yaml`.
+
+| Option          | Info                                                              |
+| --------------- | ----------------------------------------------------------------- |
+| `hostname`      | Changes the server listening address for Mucha                    |
+| `port`          | Changes the listing port                                          |
+| `auto_register` | Allows the game to handle the automatic registration of new cards |
+
+
+### Database upgrade
+
+Always make sure your database (tables) are up-to-date:
+
+```shell
+python dbutils.py --game SDEW upgrade
+```
+
+### Notes
+- Defrag Match will crash at loading
+- Co-Op Online is not supported
+- Shop is displayed but cannot purchase heroes or items
+- Player title is currently static and cannot be changed in-game
+- QR Card Scanning currently only load a static hero
+- Ex-quests progression not supported yet
+- Daily Missions not implemented
+- EX TOWER 1,2 & 3 are not yet supported
+- Daily Yui coin not yet fixed
+
+### Credits for SAO support:
+
+- Midorica - Network Support
+- Dniel97 - Helping with network base
+- tungnotpunk - Source
+
+## Initial D THE ARCADE
+
+### SDGT
+
+| Version ID | Version Name                  |
+| ---------- | ----------------------------- |
+| 0          | Initial D THE ARCADE Season 1 |
+| 1          | Initial D THE ARCADE Season 2 |
+
+**Important: Only version 1.50.00 (Season 2) is currently working and actively supported!**
+
+### Profile Importer
+
+In order to use the profile importer download the `idac_profile.json` file from the frontend
+and either directly use the folder path with `idac_profile.json` in it or specify the complete
+path to the `.json` file
+
+```shell
+python read.py --game SDGT --version <Version ID> --optfolder /path/to/game/download/folder
+```
+
+The importer for SDGT will import the complete profile data with personal high scores as well.
+
+### Config
+
+Config file is located in `config/idac.yaml`.
+
+| Option                        | Info                                                                                                        |
+| ----------------------------- | ----------------------------------------------------------------------------------------------------------- |
+| `ssl`                         | Enables/Disables the use of the `ssl_cert` and `ssl_key` (currently unsuported)                             |
+| `matching_host`               | IPv4 address of your PC for the Online Battle (currently unsupported)                                       |
+| `port_matching`               | Port number for the Online Battle Matching                                                                  |
+| `port_echo1/2`                | Port numbers for Echos                                                                                      |
+| `port_matching_p2p`           | Port number for Online Battle (currently unsupported)                                                       |
+| `stamp.enable`               | Enables/Disabled the play stamp events                                                                      |
+| `stamp.enabled_stamps`        | Define up to 3 play stamp events (without `.json` extension, which are placed in `titles/idac/data/stamps`) |
+| `timetrial.enable`           | Enables/Disables the time trial event                                                                       |
+| `timetrial.enabled_timetrial` | Define one! trial event (without `.json` extension, which are placed in `titles/idac/data/timetrial`)       |
+
+
+### Database upgrade
+
+Always make sure your database (tables) are up-to-date:
+
+```shell
+python dbutils.py --game SDGT upgrade
+```
+
+### Notes
+- Online Battle is not supported
+- Online Battle Matching is not supported
+
+### Item categories
+
+| Category ID | Category Name            |
+| ----------- | ------------------------ |
+| 1           | D Coin                   |
+| 3           | Car Dressup Token        |
+| 5           | Avatar Dressup Token     |
+| 6           | Tachometer               |
+| 7           | Aura                     |
+| 8           | Aura Color               |
+| 9           | Avatar Face              |
+| 10          | Avatar Eye               |
+| 11          | Avatar Mouth             |
+| 12          | Avatar Hair              |
+| 13          | Avatar Glasses           |
+| 14          | Avatar Face accessories  |
+| 15          | Avatar Body              |
+| 18          | Avatar Background        |
+| 21          | Chat Stamp               |
+| 22          | Keychain                 |
+| 24          | Title                    |
+| 25          | FullTune Ticket          |
+| 26          | Paper Cup                |
+| 27          | BGM                      |
+| 28          | Drifting Text            |
+| 31          | Start Menu BG            |
+| 32          | Car Color/Paint          |
+| 33          | Aura Level               |
+| 34          | FullTune Ticket Fragment |
+| 35          | Underneon Lights         |
+
+### TimeRelease Chapter:
+
+1. Story: 1, 2, 3, 4, 5, 6, 7, 8, 9, 19 (Chapter 10), (29 Chapter 11?)
+2. MF Ghost: 10, 11, 12, 13, 14, 15
+3. Bunta: 15, 16, 17, 18, 19, 20, (21, 21, 22?)
+4. Special Event: 23, 24, 25, 26, 27, 28 (Touhou Project)
+
+### TimeRelease Courses:
+
+
+| Course ID | Course Name               | Direction                |
+| --------- | ------------------------- | ------------------------ |
+| 0         | Akina Lake(秋名湖)        | CounterClockwise(左周り) |
+| 2         | Akina Lake(秋名湖)        | Clockwise(右周り)        |
+| 52        | Hakone(箱根)              | Downhill(下り)           |
+| 54        | Hakone(箱根)              | Hillclimb(上り)          |
+| 36        | Usui(碓氷)                | CounterClockwise(左周り) |
+| 38        | Usui(碓氷)                | Clockwise(右周り)        |
+| 4         | Myogi(妙義)               | Downhill(下り)           |
+| 6         | Myogi(妙義)               | Hillclimb(上り)          |
+| 8         | Akagi(赤城)               | Downhill(下り)           |
+| 10        | Akagi(赤城)               | Hillclimb(上り)          |
+| 12        | Akina(秋名)               | Downhill(下り)           |
+| 14        | Akina(秋名)               | Hillclimb(上り)          |
+| 16        | Irohazaka(いろは坂)       | Downhill(下り)           |
+| 18        | Irohazaka(いろは坂)       | Reverse(逆走)            |
+| 56        | Momiji Line(もみじライン) | Downhill(下り)           |
+| 58        | Momiji Line(もみじライン) | Hillclimb(上り)          |
+| 20        | Tsukuba(筑波)             | Outbound(往路)           |
+| 22        | Tsukuba(筑波)             | Inbound(復路)            |
+| 24        | Happogahara(八方ヶ原)     | Outbound(往路)           |
+| 26        | Happogahara(八方ヶ原)     | Inbound(復路)            |
+| 40        | Sadamine(定峰)            | Downhill(下り)           |
+| 42        | Sadamine(定峰)            | Hillclimb(上り)          |
+| 44        | Tsuchisaka(土坂)          | Outbound(往路)           |
+| 46        | Tsuchisaka(土坂)          | Inbound(復路)            |
+| 48        | Akina Snow(秋名雪)        | Downhill(下り)           |
+| 50        | Akina Snow(秋名雪)        | Hillclimb(上り)          |
+| 68        | Odawara(小田原)           | Forward(順走)            |
+| 70        | Odawara(小田原)           | Reverse(逆走)            |
+
+### Credits
+- Bottersnike: For the HUGE Reverse Engineering help
+- Kinako: For helping with the timeRelease unlocking of courses and special mode
+
+A huge thanks to all people who helped shaping this project to what it is now and don't want to be mentioned here.
+

docs/prod.md

@@ -1,41 +1,34 @@
 # ARTEMiS Production mode
-Production mode is a configuration option that changes how the server listens to be more friendly to a production environment. This mode assumes that a proxy (for this guide, nginx) is standing in front of the server to handle port mapping and TLS. In order to activate production mode, simply change `is_develop` to `False` in `core.yaml`. Next time you start the server, you should see "Starting server in production mode".
+ARTEMiS is designed to run in one of two ways. Developmen/local mode, which assumes you're just trying to set up something to save your scores and make the games work, and have patched your games to disable SSL and cert checks and encryption and the like, and production mode. In production mode, artemis assumes you have a proxy server, such as nginx or apache, standing in front of artemis doing HTTPS and port management. This document will cover how to properly set up a production instance of ARTEMiS.
+
+## ARTEMiS configuration
+Step 1 is to edit your artemis configuration. Some recomended changes:
+### `server`
+- `listen_address` -> `127.0.0.1`
+- `is_develop` -> `False`
+- `is_using_proxy` -> `True`
+- `port` -> The port nginx will send proxied requests to. If you're using the example config, set this to 8080.
+- `proxy_port` -> The port your proxy will be accepting title server connections on. If you're using the example config, set this to 80.
+- `proxy_port_ssl` -> The port your proxy will be accepting secure title server connections on. If you're using the example config, set this to 443.
+- `allow_unregistered_serials` -> `False`
+### `billing`
+- `standalone` -> `False`
+### `allnet`
+- `standalone` -> `False`
+### `frontend`
+- `enable` -> `True` if you want the frontend
+- `port` -> `8090` if you're using the default nginx config, otherwise whatever port your proxy will be sending requests to
+### `aimedb`
+- `listen_address` -> `0.0.0.0` unless you're proxying aimedb requests (not recomended at this time), in which case, leave this option unchanged
+
+If you plan to serve artemis behind a VPN, these additional settings are also recomended
+- `check_arcade_ip` -> `True`
+- `strict_ip_checking` -> `True`
 
 ## Nginx Configuration
-### Port forwarding
+For most cases, the config in `example_config` will suffice. It makes the following assumptions
-Artemis requires that the following ports be forwarded to allow internet traffic to access the server. This will not change regardless of what you set in the config, as many of these ports are hard-coded in the games.
+- ARTEMiS is running on port 8080
-`tcp:80` all.net, non-ssl titles
+- Billing is set to not be standalone
-`tcp:8443` billing
+- You're not using cloudflare in front of your frontend
-`tcp:22345` aimedb
-`tcp:443` frontend, SSL titles
 
-### A note about external proxy services (cloudflare, etc)
+If this describes you, your only configuration needs are to edit the `server_name` and `certificate_*` directives. Otherwise, please see nginx configuration documentation to configure it to best suit your setup.
-Due to the way that artemis functions, it is currently not possible to put the server behind something like Cloudflare. Cloudflare only proxies web traffic on the standard ports (80, 443) and, as shown above, this does not work with artemis. Server administrators should seek other means to protect their network (VPS hosting, VPN, etc)
-
-### SSL Certificates
-You will need to generate SSL certificates for some games. The certificates vary in security and validity requirements. Please see the general guide below
-- General Title: The certificate for the general title server should be valid, not self-signed and match the CN that the game will be reaching out to (e.i if your games are reaching out to titles.hostname.here, your ssl certificate should be valid for titles.hostname.here, or *.hostname.here)
-- CXB: Same requires as the title server. It must not be self-signed, and CN must match. Recomended to get a wildcard cert if possible, and use it for both Title and CXB
-- Pokken: Pokken can be self-signed, and the CN doesn't have to match, but it MUST use 2048-bit RSA. Due to the games age, andthing stronger then that will be rejected.
-
-### Port mappings
-An example config is provided in the `config` folder called `nginx_example.conf`. It is set up for the following:
-`naominet.jp:tcp:80` -> `localhost:tcp:8000` for allnet
-`ib.naominet.jp:ssl:8443` -> `localhost:tcp:8444` for the billing server
-`your.hostname.here:ssl:443` -> `localhost:tcp:8080` for the SSL title server
-`your.hostname.here:tcp:80` -> `localhost:tcp:8080` for the non-SSL title server
-`cxb.hostname.here:ssl:443` -> `localhost:tcp:8080` for crossbeats (appends /SDCA/104/ to the request)
-`pokken.hostname.here:ssl:443` -> `localhost:tcp:8080` for pokken
-`frontend.hostname.here:ssl:443` -> `localhost:tcp:8090` for the frontend, includes https redirection
-
-If you're using this as a guide, be sure to replace your.hostname.here with the hostname you specified in core.yaml under `titles->hostname`. Do *not* change naominet.jp, or allnet/billing will fail. Also remember to specifiy certificate paths correctly, as in the example they are simply placeholders.
-
-### Multi-service ports
-It is possible to use nginx to redirect billing and title server requests to the same port that all.net uses. By setting `port` to 0 under billing and title server, you can change the nginx config to serve the following (entries not shown here should be the same)
-`ib.naominet.jp:ssl:8443` -> `localhost:tcp:8000` for the billing server
-`your.hostname.here:ssl:443` -> `localhost:tcp:8000` for the SSL title server
-`your.hostname.here:tcp:80` -> `localhost:tcp:8000` for the non-SSL title server
-`cxb.hostname.here:ssl:443` -> `localhost:tcp:8000` for crossbeats (appends /SDCA/104/ to the request)
-`pokken.hostname.here:ssl:443` -> `localhost:tcp:8000` for pokken
-
-This will allow you to only use 3 ports locally, but you will still need to forward the same internet-facing ports as before.

entrypoint.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+if [[ -z "${CFG_DEV}" ]]; then
+    echo Production mode
+    python3 index.py
+else
+    echo Development mode
+    python3 dbutils.py create
+    nodemon -w aime --legacy-watch index.py
+fi
+

example_config/cardmaker.yaml

@@ -1,3 +1,13 @@
 server:
   enable: True
   loglevel: "info"
+
+version:
+  0:
+    ongeki: 1.30.01
+    chuni: 2.00.00
+    maimai: 1.20.00
+  1:
+    ongeki: 1.35.03
+    chuni: 2.10.00
+    maimai: 1.30.00

example_config/chuni.yaml

@@ -1,9 +1,10 @@
 server:
   enable: True
   loglevel: "info"
+  news_msg: ""
 
 team:
-  name: ARTEMiS
+  name: ARTEMiS # If this is set, all players that are not on a team will use this one by default.
 
 mods:
   use_login_bonus: True
@@ -15,6 +16,17 @@ version:
   12:
     rom: 2.05.00
     data: 2.05.00
+  13:
+    rom: 2.10.00
+    data: 2.10.00
+  14:
+    rom: 2.15.00
+    data: 2.15.00
 
 crypto:
   encrypted_only: False
+
+matching:
+  enable: False
+  match_time_limit: 60
+  match_error_limit: 9999

example_config/core.yaml

@@ -1,15 +1,24 @@
 server:
   listen_address: "127.0.0.1"  
+  hostname: "localhost"  
+  port: 80
+  ssl_key: "cert/title.key"
+  ssl_cert: "cert/title.crt"
   allow_user_registration: True
   allow_unregistered_serials: True
   name: "ARTEMiS"
   is_develop: True
+  is_using_proxy: False
+  proxy_port: 0
+  proxy_port_ssl: 0
   log_dir: "logs"
+  check_arcade_ip: False
+  strict_ip_checking: False
 
 title:
   loglevel: "info"
-  hostname: "localhost"
+  reboot_start_time: "04:00"
-  port: 8080
+  reboot_end_time: "05:00"
 
 database:
   host: "localhost"
@@ -19,32 +28,39 @@ database:
   port: 3306
   protocol: "mysql"
   sha2_password: False
-  loglevel: "warn"
+  loglevel: "info"
-  user_table_autoincrement_start: 10000
+  enable_memcached: True
   memcached_host: "localhost"
 
 frontend:
   enable: False
-  port: 8090
+  port: 8080
   loglevel: "info"
+  secret: ""
 
 allnet:
-  loglevel: "info"
+  standalone: False
   port: 80
+  loglevel: "info"
   allow_online_updates: False
+  update_cfg_folder: ""
 
 billing:
+  standalone: True
+  loglevel: "info"
   port: 8443
   ssl_key: "cert/server.key"
   ssl_cert: "cert/server.pem"
   signing_key: "cert/billing.key"
 
 aimedb:
+  enable: True
+  listen_address: ""
   loglevel: "info"
   port: 22345
   key: ""
+  id_secret: ""
+  id_lifetime_seconds: 86400
 
 mucha:
-  enable: False
-  hostname: "localhost"
   loglevel: "info"

example_config/cxb.yaml

@@ -1,9 +1,4 @@
 server:
   enable: True
   loglevel: "info"
-  hostname: "localhost"
+  use:https: True
-  ssl_enable: False
-  port: 8082
-  port_secure: 443
-  ssl_cert: "cert/title.crt"
-  ssl_key: "cert/title.key"

example_config/idac.yaml

@@ -0,0 +1,22 @@
+server:
+  enable: True
+  loglevel: "info"
+  ssl: False
+  ssl_key: "cert/idac.key"
+  ssl_cert: "cert/idac.crt"
+  matching_host: "127.0.0.1"
+  port_matching: 20000
+  port_echo1: 20001
+  port_echo2: 20002
+  port_matching_p2p: 20003
+
+stamp:
+  enable: True
+  enabled_stamps: # max 3 play stamps
+    - "touhou_remilia_scarlet"
+    - "touhou_flandre_scarlet"
+    - "touhou_sakuya_izayoi"
+
+timetrial:
+  enable: True
+  enabled_timetrial: "touhou_remilia_scarlet"

example_config/idz.yaml

@@ -0,0 +1,11 @@
+server:
+  enable: True
+  loglevel: "info"
+  hostname: ""
+  news: ""
+  aes_key: ""
+
+ports:
+  userdb: 10000
+  match: 10010
+  echo: 10020

example_config/mai2.yaml

@@ -1,3 +1,14 @@
 server:
   enable: True
   loglevel: "info"
+
+deliver:
+  enable: False
+  udbdl_enable: False
+  content_folder: ""
+
+uploads:
+  photos: False
+  photos_dir: ""
+  movies: False
+  movies_dir: ""

example_config/nginx_example.conf

@@ -6,7 +6,7 @@ server {
 	location / {
 		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 		proxy_pass_request_headers on;
-		proxy_pass http://localhost:8000/;
+		proxy_pass http://127.0.0.1:8080/;
 	}
 }
 
@@ -18,14 +18,13 @@ server {
 	location / {
 		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 		proxy_pass_request_headers on;
-		proxy_pass http://localhost:8080/;
+		proxy_pass http://127.0.0.1:8080/;
 	}
 }
 
 # SSL titles, comment out if you don't plan on accepting SSL titles
 server {
-	listen 443 ssl default_server;
+	listen 443 ssl;
-	listen [::]:443 ssl default_server;	
 	server_name your.hostname.here;
 
 	ssl_certificate /path/to/cert/title.crt;
@@ -39,11 +38,13 @@ server {
 	ssl_prefer_server_ciphers off;
 
 	location / {
-		proxy_pass http://localhost:8080/;
+		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+		proxy_pass_request_headers on;
+		proxy_pass http://127.0.0.1:8080/;
 	}
 }
 
-# Billing
+# Billing, comment this out if running billing standalone
 server {
 	listen 8443 ssl;	
 	server_name ib.naominet.jp;
@@ -55,55 +56,13 @@ server {
 	ssl_session_tickets off;
 
 	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
-	ssl_ciphers "ALL:@SECLEVEL=1";
+	ssl_ciphers "ALL:@SECLEVEL=0";
-	ssl_prefer_server_ciphers off;
-
-	location / {
-		proxy_pass http://localhost:8444/;
-	}
-}
-
-# Pokken, comment this out if you don't plan on serving pokken.
-server {
-	listen 443 ssl;	
-	server_name pokken.hostname.here;
-	
-	ssl_certificate /path/to/cert/pokken.pem;
-	ssl_certificate_key /path/to/cert/pokken.key;
-	ssl_session_timeout 1d;
-	ssl_session_cache shared:MozSSL:10m;
-	ssl_session_tickets off;
-
-	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
-	ssl_ciphers "ALL:@SECLEVEL=1";
 	ssl_prefer_server_ciphers off;
 
 	location / {
 		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 		proxy_pass_request_headers on;
-		proxy_pass http://localhost:8080/;
+		proxy_pass http://127.0.0.1:8080/;
-	}
-}
-
-# CXB, comment this out if you don't plan on serving crossbeats.
-server {
-	listen 443 ssl;	
-	server_name cxb.hostname.here;
-	
-	ssl_certificate /path/to/cert/cxb.pem;
-	ssl_certificate_key /path/to/cert/cxb.key;
-	ssl_session_timeout 1d;
-	ssl_session_cache shared:MozSSL:10m;
-	ssl_session_tickets off;
-
-	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
-	ssl_ciphers "ALL:@SECLEVEL=1";
-	ssl_prefer_server_ciphers off;
-
-	location / {
-		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-		proxy_pass_request_headers on;
-		proxy_pass http://localhost:8080/SDBT/104/;
 	}
 }
 
@@ -114,12 +73,12 @@ server {
 
     location / {
         return 301 https://$host$request_uri;
-		# If you don't want https redirection, comment the line above and uncomment the line below
+		# If you don't want https redirection, or are using something like cloudflare to manage HTTPS, comment out the line above and uncomment the line below
-		# proxy_pass http://localhost:8090/;
+		# proxy_pass http://127.0.0.1:8090/;
     }
 }
 
-# Frontend HTTPS. Comment out if you on't intend to use the frontend
+# Frontend HTTPS. Comment out if you on't intend to use the frontend, or have cloudflare or something managing https for you.
 server {
     listen 443 ssl;
 	server_name frontend.hostname.here;
@@ -141,6 +100,6 @@ server {
 	location / {
 		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 		proxy_pass_request_headers on;
-		proxy_pass http://localhost:8090/;
+		proxy_pass http://127.0.0.1:8090/;
 	}
 }

example_config/ongeki.yaml

@@ -1,6 +1,7 @@
 server:
   enable: True
   loglevel: "info"
+  use_https: False
 
 gachas:
   enabled_gachas:
@@ -29,3 +30,12 @@ gachas:
     - 1156
     - 1163
     - 1164
+
+version:
+  6:
+    card_maker: 1.30.01
+  7:
+    card_maker: 1.35.03
+
+crypto:
+  encrypted_only: False

example_config/pokken.yaml

@@ -2,8 +2,11 @@ server:
   hostname: "localhost"
   enable: True
   loglevel: "info"
-  port: 9000
-  port_stun: 9001
-  port_turn: 9002
-  port_admission: 9003
   auto_register: True
+  enable_matching: False
+  stun_server_host: "stunserver.stunprotocol.org"
+  stun_server_port: 3478
+
+ports:
+  game: 9000
+  admission: 9001

example_config/sao.yaml

@@ -0,0 +1,13 @@
+server:
+  enable: True
+  loglevel: "info"
+  auto_register: True
+
+crypt:
+  enable: False
+  key: ""
+  iv: ""
+
+hash:
+  verify_hash: False
+  hash_base: ""

example_config/wacca.yaml

@@ -29,5 +29,8 @@ gates:
     - 17
     - 18
     - 19
+    - 20
     - 21
     - 22
+    - 23
+    - 24

index.py

@@ -1,243 +1,117 @@
 #!/usr/bin/env python3
 import argparse
-import logging, coloredlogs
-from logging.handlers import TimedRotatingFileHandler
-from typing import Dict
 import yaml
-from os import path, mkdir, access, W_OK
+from os import path, environ
-from core import *
+import uvicorn
+import logging
+import asyncio
 
-from twisted.web import server, resource
+from core import CoreConfig, AimedbServlette
-from twisted.internet import reactor, endpoints
-from twisted.web.http import Request
-from routes import Mapper
 
-
+async def launch_main(cfg: CoreConfig, ssl: bool) -> None:
-class HttpDispatcher(resource.Resource):
+    if ssl:
-    def __init__(self, cfg: CoreConfig, config_dir: str):
+        server_cfg = uvicorn.Config(
-        super().__init__()
+            "core.app:app", 
-        self.config = cfg
+            host=cfg.server.listen_address, 
-        self.isLeaf = True
+            port=cfg.server.port if args.port == 0 else args.port, 
-        self.map_get = Mapper()
+            reload=cfg.server.is_develop,
-        self.map_post = Mapper()
+            log_level="info" if cfg.server.is_develop else "critical",
-        self.logger = logging.getLogger("core")
+            ssl_version=3,
-
+            ssl_certfile=cfg.server.ssl_cert,
-        self.allnet = AllnetServlet(cfg, config_dir)
+            ssl_keyfile=cfg.server.ssl_key
-        self.title = TitleServlet(cfg, config_dir)
-        self.mucha = MuchaServlet(cfg, config_dir)
-
-        self.map_post.connect(
-            "allnet_ping",
-            "/naomitest.html",
-            controller="allnet",
-            action="handle_naomitest",
-            conditions=dict(method=["GET"]),
         ) 
-        self.map_post.connect(
-            "allnet_poweron",
-            "/sys/servlet/PowerOn",
-            controller="allnet",
-            action="handle_poweron",
-            conditions=dict(method=["POST"]),
-        )
-        self.map_post.connect(
-            "allnet_downloadorder",
-            "/sys/servlet/DownloadOrder",
-            controller="allnet",
-            action="handle_dlorder",
-            conditions=dict(method=["POST"]),
-        )
-        self.map_post.connect(
-            "allnet_billing",
-            "/request",
-            controller="allnet",
-            action="handle_billing_request",
-            conditions=dict(method=["POST"]),
-        )
-        self.map_post.connect(
-            "allnet_billing",
-            "/request/",
-            controller="allnet",
-            action="handle_billing_request",
-            conditions=dict(method=["POST"]),
-        )
-
-        self.map_post.connect(
-            "mucha_boardauth",
-            "/mucha/boardauth.do",
-            controller="mucha",
-            action="handle_boardauth",
-            conditions=dict(method=["POST"]),
-        )
-        self.map_post.connect(
-            "mucha_updatacheck",
-            "/mucha/updatacheck.do",
-            controller="mucha",
-            action="handle_updatecheck",
-            conditions=dict(method=["POST"]),
-        )
-
-        self.map_get.connect(
-            "title_get",
-            "/{game}/{version}/{endpoint:.*?}",
-            controller="title",
-            action="render_GET",
-            conditions=dict(method=["GET"]),
-            requirements=dict(game=R"S..."),
-        )
-        self.map_post.connect(
-            "title_post",
-            "/{game}/{version}/{endpoint:.*?}",
-            controller="title",
-            action="render_POST",
-            conditions=dict(method=["POST"]),
-            requirements=dict(game=R"S..."),
-        )
-
-    def render_GET(self, request: Request) -> bytes:
-        self.logger.debug(request.uri)
-        test = self.map_get.match(request.uri.decode())
-        client_ip = Utils.get_ip_addr(request)
-
-        if test is None:
-            self.logger.debug(
-                f"Unknown GET endpoint {request.uri.decode()} from {client_ip} to port {request.getHost().port}"
-            )
-            request.setResponseCode(404)
-            return b"Endpoint not found."
-
-        return self.dispatch(test, request)
-
-    def render_POST(self, request: Request) -> bytes:
-        test = self.map_post.match(request.uri.decode())
-        client_ip = Utils.get_ip_addr(request)
-
-        if test is None:
-            self.logger.debug(
-                f"Unknown POST endpoint {request.uri.decode()} from {client_ip} to port {request.getHost().port}"
-            )
-            request.setResponseCode(404)
-            return b"Endpoint not found."
-
-        return self.dispatch(test, request)
-
-    def dispatch(self, matcher: Dict, request: Request) -> bytes:
-        controller = getattr(self, matcher["controller"], None)
-        if controller is None:
-            self.logger.error(
-                f"Controller {matcher['controller']} not found via endpoint {request.uri.decode()}"
-            )
-            request.setResponseCode(404)
-            return b"Endpoint not found."
-
-        handler = getattr(controller, matcher["action"], None)
-        if handler is None:
-            self.logger.error(
-                f"Action {matcher['action']} not found in controller {matcher['controller']} via endpoint {request.uri.decode()}"
-            )
-            request.setResponseCode(404)
-            return b"Endpoint not found."
-
-        url_vars = matcher
-        url_vars.pop("controller")
-        url_vars.pop("action")
-        ret = handler(request, url_vars)
-
-        if type(ret) == str:
-            return ret.encode()
-        elif type(ret) == bytes:
-            return ret
     else:
-            return b""
+        server_cfg = uvicorn.Config(
+            "core.app:app", 
+            host=cfg.server.listen_address, 
+            port=cfg.server.port if args.port == 0 else args.port, 
+            reload=cfg.server.is_develop,
+            log_level="info" if cfg.server.is_develop else "critical"
+        )
+    server = uvicorn.Server(server_cfg)
+    await server.serve()
 
+async def launch_billing(cfg: CoreConfig) -> None:
+    server_cfg = uvicorn.Config(
+        "core.allnet:app_billing", 
+        host=cfg.server.listen_address, 
+        port=cfg.billing.port, 
+        reload=cfg.server.is_develop,
+        log_level="info" if cfg.server.is_develop else "critical",
+        ssl_version=3,
+        ssl_certfile=cfg.billing.ssl_cert,
+        ssl_keyfile=cfg.billing.ssl_key,
+        ssl_ciphers="DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK",
+    )
+    server = uvicorn.Server(server_cfg)
+    await server.serve()
+
+async def launch_frontend(cfg: CoreConfig) -> None:
+    server_cfg = uvicorn.Config(
+        "core.frontend:app", 
+        host=cfg.server.listen_address, 
+        port=cfg.frontend.port, 
+        reload=cfg.server.is_develop,
+        log_level="info" if cfg.server.is_develop else "critical",
+    )
+    server = uvicorn.Server(server_cfg)
+    await server.serve()
+
+async def launch_allnet(cfg: CoreConfig) -> None:
+    server_cfg = uvicorn.Config(
+        "core.allnet:app_allnet", 
+        host=cfg.server.listen_address, 
+        port=cfg.allnet.port, 
+        reload=cfg.server.is_develop,
+        log_level="info" if cfg.server.is_develop else "critical",
+    )
+    server = uvicorn.Server(server_cfg)
+    await server.serve()
+
+
+async def launcher(cfg: CoreConfig, ssl: bool) -> None:
+    task_list = [asyncio.create_task(launch_main(cfg, ssl))]
+    
+    if cfg.billing.standalone:
+        task_list.append(asyncio.create_task(launch_billing(cfg)))
+    if cfg.frontend.enable:
+        task_list.append(asyncio.create_task(launch_frontend(cfg)))
+    if cfg.allnet.standalone:
+        task_list.append(asyncio.create_task(launch_allnet(cfg)))
+    if cfg.aimedb.enable:
+        AimedbServlette(cfg).start()
+    
+    done, pending = await asyncio.wait(
+        task_list,
+        return_when=asyncio.FIRST_COMPLETED,
+    )
+    
+    logging.getLogger("core").info("Shutdown")
+    for pending_task in pending:
+        pending_task.cancel("Another service died, server is shutting down")
 
 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="ARTEMiS main entry point")
+    parser = argparse.ArgumentParser(description="Artemis main entry point")
     parser.add_argument(
         "--config", "-c", type=str, default="config", help="Configuration folder"
     )
+    parser.add_argument(
+        "--port", "-p", type=int, default=0, help="Port override"
+    )
+    parser.add_argument(
+        "--ssl", "-s", type=bool, help="Launch with SSL"
+    )
     args = parser.parse_args()
 
     if not path.exists(f"{args.config}/core.yaml"):
         print(
-            f"The config folder you specified ({args.config}) does not exist or does not contain core.yaml.\nDid you copy the example folder?"
+            f"The config folder you specified ({args.config}) does not exist or does not contain core.yaml. Defaults will be used.\nDid you copy the example folder?"
         )
-        exit(1)
     
     cfg: CoreConfig = CoreConfig()
     if path.exists(f"{args.config}/core.yaml"):
         cfg.update(yaml.safe_load(open(f"{args.config}/core.yaml")))
 
-    if not path.exists(cfg.server.log_dir):
+    environ["ARTEMIS_CFG_DIR"] = args.config
-        mkdir(cfg.server.log_dir)
 
-    if not access(cfg.server.log_dir, W_OK):
+    asyncio.run(launcher(cfg, args.ssl))
-        print(
-            f"Log directory {cfg.server.log_dir} NOT writable, please check permissions"
-        )
-        exit(1)
-
-    logger = logging.getLogger("core")
-    log_fmt_str = "[%(asctime)s] Core | %(levelname)s | %(message)s"
-    log_fmt = logging.Formatter(log_fmt_str)
-
-    fileHandler = TimedRotatingFileHandler(
-        "{0}/{1}.log".format(cfg.server.log_dir, "core"), when="d", backupCount=10
-    )
-    fileHandler.setFormatter(log_fmt)
-
-    consoleHandler = logging.StreamHandler()
-    consoleHandler.setFormatter(log_fmt)
-
-    logger.addHandler(fileHandler)
-    logger.addHandler(consoleHandler)
-
-    log_lv = logging.DEBUG if cfg.server.is_develop else logging.INFO
-    logger.setLevel(log_lv)
-    coloredlogs.install(level=log_lv, logger=logger, fmt=log_fmt_str)
-
-    if not cfg.aimedb.key:
-        logger.error("!!AIMEDB KEY BLANK, SET KEY IN CORE.YAML!!")
-        exit(1)
-
-    logger.info(
-        f"ARTEMiS starting in {'develop' if cfg.server.is_develop else 'production'} mode"
-    )
-
-    allnet_server_str = f"tcp:{cfg.allnet.port}:interface={cfg.server.listen_address}"
-    title_server_str = f"tcp:{cfg.title.port}:interface={cfg.server.listen_address}"
-    adb_server_str = f"tcp:{cfg.aimedb.port}:interface={cfg.server.listen_address}"
-    frontend_server_str = (
-        f"tcp:{cfg.frontend.port}:interface={cfg.server.listen_address}"
-    )
-
-    billing_server_str = f"tcp:{cfg.billing.port}:interface={cfg.server.listen_address}"
-    if cfg.server.is_develop:
-        billing_server_str = (
-            f"ssl:{cfg.billing.port}:interface={cfg.server.listen_address}"
-            f":privateKey={cfg.billing.ssl_key}:certKey={cfg.billing.ssl_cert}"
-        )
-
-    dispatcher = HttpDispatcher(cfg, args.config)
-
-    endpoints.serverFromString(reactor, allnet_server_str).listen(
-        server.Site(dispatcher)
-    )
-    endpoints.serverFromString(reactor, adb_server_str).listen(AimedbFactory(cfg))
-
-    if cfg.frontend.enable:
-        endpoints.serverFromString(reactor, frontend_server_str).listen(
-            server.Site(FrontendServlet(cfg, args.config))
-        )
-
-    if cfg.billing.port > 0:
-        endpoints.serverFromString(reactor, billing_server_str).listen(
-            server.Site(dispatcher)
-        )
-
-    if cfg.title.port > 0:
-        endpoints.serverFromString(reactor, title_server_str).listen(
-            server.Site(dispatcher)
-        )
-
-    reactor.run()  # type: ignore

read.py

@@ -1,4 +1,4 @@
-# vim: set fileencoding=utf-8
+#!/usr/bin/env python3
 import argparse
 import re
 import os
@@ -6,6 +6,7 @@ import yaml
 from os import path
 import logging
 import coloredlogs
+import asyncio
 
 from logging.handlers import TimedRotatingFileHandler
 from typing import List, Optional
@@ -39,15 +40,18 @@ class BaseReader:
 
         return ret
     
+    async def read(self) -> None:
+        pass
+
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="Import Game Information")
     parser.add_argument(
-        "--series",
+        "--game",
         action="store",
         type=str,
         required=True,
-        help="The game series we are importing.",
+        help="The game we are importing.",
     )
     parser.add_argument(
         "--version",
@@ -109,7 +113,7 @@ if __name__ == "__main__":
     logger.setLevel(log_lv)
     coloredlogs.install(level=log_lv, logger=logger, fmt=log_fmt_str)
 
-    if args.series is None or args.version is None:
+    if args.game is None or args.version is None:
         logger.error("Game or version not specified")
         parser.print_help()
         exit(1)
@@ -134,9 +138,10 @@ if __name__ == "__main__":
     titles = Utils.get_all_titles()
 
     for dir, mod in titles.items():
-        if args.series in mod.game_codes:
+        if args.game in mod.game_codes:
-            handler = mod.reader(config, args.version,
+            handler = mod.reader(config, args.version, bin_arg, opt_arg, args.extra)
-                                 bin_arg, opt_arg, args.extra)
+            loop = asyncio.get_event_loop()
-            handler.read()
+            loop.run_until_complete(handler.read())
+            
 
     logger.info("Done")

readme.md

@@ -2,31 +2,70 @@
 A network service emulator for games running SEGA'S ALL.NET service, and similar.
 
 # Supported games
-Games listed below have been tested and confirmed working. Only game versions older then the current one in active use in arcades (n-0) or current game versions older then a year (y-1) are supported.
+Games listed below have been tested and confirmed working. Only game versions older then the version currently active in arcades, or games versions that have not recieved a major update in over one year, are supported.
-+ Chunithm
-    + All versions up to New!! Plus
-
-+ Crossbeats Rev
-    + All versions + omnimix
-
-+ maimai DX
-    + All versions up to Festival
-
-+ Hatsune Miku Arcade
-    + All versions
 
 + Card Maker
-    + 1.34.xx
+    + 1.30
-    + 1.35.xx
+    + 1.35
 
-+ Ongeki
++ CHUNITHM INTL
-    + All versions up to Bright Memory
+    + SUPERSTAR
+    + NEW
+    + NEW PLUS
+    + SUN
+    + SUN PLUS
 
-+ Wacca
++ CHUNITHM JP
+    + AIR
+    + AIR PLUS
+    + AMAZON
+    + AMAZON PLUS
+    + CRYSTAL
+    + CRYSTAL PLUS
+    + PARADISE
+    + PARADISE LOST
+    + NEW
+    + NEW PLUS
+    + SUN
+    + SUN PLUS
+
++ crossbeats REV.
+    + Crossbeats REV.
+    + Crossbeats REV. SUNRiSE S1
+    + Crossbeats REV. SUNRiSE S2 + omnimix
+
++ Hatsune Miku: Project DIVA Arcade
+    + Future Tone Arcade - All versions
+
++ Initial D THE ARCADE
+    + Season 2
+
++ maimai DX
+    + Splash
+    + Splash Plus
+    + UNiVERSE
+    + UNiVERSE PLUS
+    + FESTiVAL
+    + FESTiVAL PLUS
+
++ O.N.G.E.K.I.
+    + SUMMER
+    + SUMMER PLUS
+    + R.E.D.
+    + R.E.D. PLUS
+    + bright
+    + bright MEMORY
+
++ POKKÉN TOURNAMENT
+    + Final Online
+
++ Sword Art Online Arcade
+    + Final (Single player only)
+
++ WACCA
     + Lily R
     + Reverse
 
-
 ## Requirements
 - python 3 (tested working with 3.9 and 3.10, other versions YMMV)
 - pip
@@ -34,7 +73,7 @@ Games listed below have been tested and confirmed working. Only game versions ol
 - mysql/mariadb server
 
 ## Setup guides
-Follow the platform-specific guides for [windows](docs/INSTALL_WINDOWS.md) and [ubuntu](docs/INSTALL_UBUNTU.md) to setup and run the server.
+Follow the platform-specific guides for [windows](docs/INSTALL_WINDOWS.md), [linux (Debian 12 or Rasperry Pi OS recomended, but anything works)](docs/INSTALL_LINUX.md) or [docker](docs/INSTALL_DOCKER.md) to setup and run the server.
 
 ## Game specific information
 Read [Games specific info](docs/game_specific_info.md) for all supported games, importer settings, configuration option and database upgrades.

requirements.txt

@@ -1,6 +1,5 @@
 mypy
 wheel
-twisted
 pytz
 pyyaml
 sqlalchemy==1.4.46
@@ -12,7 +11,13 @@ inflection
 coloredlogs
 pylibmc; platform_system != "Windows"
 wacky
-Routes
 bcrypt
 jinja2
 protobuf
+pillow
+pyjwt==2.8.0
+websockets
+starlette
+asyncio
+uvicorn
+alembic

titles/chuni/__init__.py

@@ -1,10 +1,11 @@
-from titles.chuni.index import ChuniServlet
+from .index import ChuniServlet
-from titles.chuni.const import ChuniConstants
+from .const import ChuniConstants
-from titles.chuni.database import ChuniData
+from .database import ChuniData
-from titles.chuni.read import ChuniReader
+from .read import ChuniReader
+from .frontend import ChuniFrontend
 
 index = ChuniServlet
 database = ChuniData
 reader = ChuniReader
-game_codes = [ChuniConstants.GAME_CODE, ChuniConstants.GAME_CODE_NEW]
+frontend = ChuniFrontend
-current_schema_version = 3
+game_codes = [ChuniConstants.GAME_CODE, ChuniConstants.GAME_CODE_NEW, ChuniConstants.GAME_CODE_INT]