Teachable learnset helper mechanics (#3856)

* Teachable learnset helper mechanics

* Rename folder and python script

* Some teachable learnset work

* Update PoryMoves file labels

* Add header and make custom json

* Include found moves in output file

* Update SV file to latest version

* Don't run if there are no jsons to be found

* Add Basculin duplication in json

* Add universal move support to

* Ignore and skip Mew

* Integrate tool in Makefile

* Condense Basculin learnsets

* Split Oinkologne for easier generation

* Add Deoxys' XD move tutor data

* Add missing Darumaka/Yamask Galarian SwSh TMs

* Add TID species to sv.json

* Update sv.json to The Indigo Disk data

* Add Python install instructions

* Fix Makefile

* Expand header with more information

* Add config to allow disabling the learnset helper

* Update include/config/pokemon.h

Co-authored-by: Eduardo Quezada D'Ottone <eduardo602002@gmail.com>

* Don't crash if the config is missing

---------

Co-authored-by: Eduardo Quezada D'Ottone <eduardo602002@gmail.com>

INSTALL.md

@@ -118,6 +118,17 @@ Some tips before proceeding:
 
 devkitARM is now installed.
 
+### Installing Python on WSL1
+
+To install Python on WSL1, simply run the following commands:
+
+    ```bash
+    sudo apt update && upgrade
+    sudo apt install python3
+    ```
+
+Python is now installed.
+
 ### Choosing where to store pokeemerald Expansion (WSL1)
 WSL has its own file system that's not natively accessible from Windows, but Windows files *are* accessible from WSL. So you're going to want to store pokeemerald Expansion within Windows.
 
@@ -200,6 +211,16 @@ Note that in msys2, Copy is Ctrl+Insert and Paste is Shift+Insert.
     cd
     ```
 
+### Installing Python on msys2
+
+To install Python on WSL1, simply run the following commands:
+
+    ```bash
+    pacman -S mingw-w64-x86_64-python3
+    ```
+
+Python is now installed.
+
 ### Choosing where to store pokeemerald Expansion (msys2)
 At this point, you can choose a folder to store pokeemerald Expansion into. If you're okay with storing pokeemerald Expansion in the user profile folder, then proceed to [Installation](#installation). Otherwise, you'll need to account for where pokeemerald Expansion is stored when changing directory to the pokeemerald-expansion folder.
 
@@ -334,6 +355,12 @@ If this works, then proceed to [Installation](#installation). Otherwise, ask for
     echo "if [ -f ~/.bashrc ]; then . ~/.bashrc; fi" >> ~/.bash_profile
     ```
 
+### Installing Python (macOS)
+1. Download the latest Python package from [here](https://www.python.org/downloads/).
+2. Open the package to install Python.
+
+Python is now installed.
+
 ### Choosing where to store pokeemerald Expansion (macOS)
 At this point, you can choose a folder to store pokeemerald Expansion into. If you're okay with storing pokeemerald Expansion in the user folder, then proceed to [Installation](#installation). Otherwise, you'll need to account for where pokeemerald Expansion is stored when changing directory to the pokeemerald-expansion folder.
 
@@ -434,6 +461,9 @@ _(Specific instructions for other distributions would be greatly appreciated!)_
 
     The last command will ask for the selection of packages to install. Just press Enter to install all of them, followed by entering Y to proceed with the installation.
 
+### Installing Python in Linux
+Installing Python depends on your distribution, please refere to the instructions [here](https://docs.python-guide.org/starting/install3/linux/).
+
 ### Choosing where to store pokeemerald Expansion (Linux)
 At this point, you can choose a folder to store pokeemerald Expansion into. If so, you'll have to account for the modified folder path when changing directory to the pokeemerald-expansion folder.
 
@@ -576,4 +606,4 @@ Note that this is not necessary for a non-modern (agbcc) build since those are b
 
 * [porymap](https://github.com/huderlem/porymap) for viewing and editing maps
 * [poryscript](https://github.com/huderlem/poryscript) for scripting ([VS Code extension](https://marketplace.visualstudio.com/items?itemName=karathan.poryscript))
-* [Tilemap Studio](https://github.com/Rangi42/tilemap-studio) for viewing and editing tilemaps
+* [Tilemap Studio](https://github.com/Rangi42/tilemap-studio) for viewing and editing tilemaps

Makefile

@@ -461,6 +461,10 @@ $(OBJ_DIR)/sym_common.ld: sym_common.txt $(C_OBJS) $(wildcard common_syms/*.txt)
 $(OBJ_DIR)/sym_ewram.ld: sym_ewram.txt
 	$(RAMSCRGEN) ewram_data $< ENGLISH > $@
 
+# NOTE: Depending on event_scripts.o is hacky, but we want to depend on everything event_scripts.s depends on without having to alter scaninc
+$(DATA_SRC_SUBDIR)/pokemon/teachable_learnsets.h: $(DATA_ASM_BUILDDIR)/event_scripts.o
+	python3 tools/learnset_helpers/teachable.py
+
 # NOTE: Based on C_DEP above, but without NODEP and KEEP_TEMPS handling.
 define TEST_DEP
 $1: $2 $$(shell $(SCANINC) -I include -I tools/agbcc/include -I gflib $2)

include/config/pokemon.h

@@ -45,6 +45,9 @@
 #define P_SHOW_TERA_TYPE            GEN_LATEST  // Since Gen 9, the Tera Type is shown on the summary screen.
 #define P_TM_LITERACY               GEN_LATEST  // Since Gen 6, TM illiterate Pokémon can learn TMs that teach moves that are in their level-up learnsets.
 
+// Learnset helper toggles
+#define P_LEARNSET_HELPER_TEACHABLE TRUE        // If TRUE, teachable_learnsets.h will be populated by tools/learnset_helpers/teachable.py using the included JSON files based on available TMs and tutors.
+
 // Flag settings
 // To use the following features in scripting, replace the 0s with the flag ID you're assigning it to.
 // Eg: Replace with FLAG_UNUSED_0x264 so you can use that flag to toggle the feature.

tools/learnset_helpers/porymoves_files/xd.json

@@ -0,0 +1,20 @@
+{
+    "DEOXYS_NORMAL": {
+      "LevelMoves": [],
+      "PreEvoMoves": [],
+      "TMMoves": [],
+      "EggMoves": [],
+      "TutorMoves": [
+        "MOVE_BODY_SLAM",
+        "MOVE_DOUBLE_EDGE",
+        "MOVE_DREAM_EATER",
+        "MOVE_ICY_WIND",
+        "MOVE_MIMIC",
+        "MOVE_NIGHTMARE",
+        "MOVE_SEISMIC_TOSS",
+        "MOVE_SUBSTITUTE",
+        "MOVE_SWAGGER",
+        "MOVE_THUNDER_WAVE"
+      ]
+    }
+}

tools/learnset_helpers/teachable.py

@@ -0,0 +1,205 @@
+import glob
+import re
+import json
+import os
+
+# before all else, abort if the config is off
+with open("./include/config/pokemon.h", "r") as file:
+    learnset_config = re.findall("#define P_LEARNSET_HELPER_TEACHABLE *([^ ]*)", file.read())
+    if len(learnset_config) != 1:
+        quit()
+    if learnset_config[0] != "TRUE":
+        quit()
+
+def parse_mon_name(name):
+    return re.sub('(?!^)([A-Z]+)', r'_\1', name).upper()
+    
+tm_moves = []
+tutor_moves = []
+
+# scan incs
+incs_to_check =  glob.glob('./data/scripts/*.inc') # all .incs in the script folder
+incs_to_check += glob.glob('./data/maps/*/scripts.inc') # all map scripts
+
+if len(incs_to_check) == 0: # disabled if no jsons present
+    quit()
+
+for file in incs_to_check:
+    with open(file, 'r') as f2:
+        raw = f2.read()
+    if 'special ChooseMonForMoveTutor' in raw:
+        for x in re.findall('setvar VAR_0x8005, (MOVE_.*)', raw):
+            if not x in tutor_moves:
+                tutor_moves.append(x)
+
+# scan TMs and HMs
+with open("./include/constants/tms_hms.h", 'r') as file:
+    for x in re.findall('F\((.*)\)', file.read()):
+        if not 'MOVE_' + x in tm_moves:
+            tm_moves.append('MOVE_' + x)
+
+# look up universal moves to exclude them
+universal_moves = []
+with open("./src/pokemon.c", "r") as file:
+    for x in re.findall("static const u16 sUniversalMoves\[\] =(.|\n)*?{((.|\n)*?)};", file.read())[0]:
+        x = x.replace("\n", "")
+        for y in x.split(","):
+            y = y.strip()
+            if y == "":
+                continue
+            universal_moves.append(y)
+
+# get compatibility from jsons
+def construct_compatibility_dict(force_custom_check):
+    dict_out = {}
+    for pth in glob.glob('./tools/learnset_helpers/porymoves_files/*.json'):
+        f = open(pth, 'r')
+        data = json.load(f)
+        for mon in data.keys():
+            if not mon in dict_out:
+                dict_out[mon] = []
+            for move in data[mon]['LevelMoves']:
+                if not move['Move'] in dict_out[mon]:
+                    dict_out[mon].append(move['Move'])
+            #for move in data[mon]['PreEvoMoves']:
+            #    if not move in dict_out[mon]:
+            #        dict_out[mon].append(move)
+            for move in data[mon]['TMMoves']:
+                if not move in dict_out[mon]:
+                    dict_out[mon].append(move)
+            for move in data[mon]['EggMoves']:
+                if not move in dict_out[mon]:
+                    dict_out[mon].append(move)
+            for move in data[mon]['TutorMoves']:
+                if not move in dict_out[mon]:
+                    dict_out[mon].append(move)
+
+    # if the file was not previously generated, check if there is custom data there that needs to be preserved
+    with open("./src/data/pokemon/teachable_learnsets.h", 'r') as file:
+        raw = file.read()
+        if not "// DO NOT MODIFY THIS FILE!" in raw and force_custom_check == True:
+            custom_teachable_compatibilities = {}
+            for entry in re.findall("static const u16 s(.*)TeachableLearnset\[\] = {\n((.|\n)*?)\n};", raw):
+                monname = parse_mon_name(entry[0])
+                if monname == "NONE":
+                    continue
+                compatibility = entry[1].split("\n")
+                if not monname in custom_teachable_compatibilities:
+                    custom_teachable_compatibilities[monname] = []
+                if not monname in dict_out:
+                    # this mon is unknown, so all data needs to be preserved
+                    for move in compatibility:
+                        move = move.replace(",", "").strip()
+                        if move == "" or move == "MOVE_UNAVAILABLE":
+                            continue
+                        custom_teachable_compatibilities[monname].append(move)
+                else:
+                    # this mon is known, so check if the moves in the old teachable_learnsets.h are not in the jsons
+                    for move in compatibility:
+                        move = move.replace(",", "").strip()
+                        if move == "" or move == "MOVE_UNAVAILABLE":
+                            continue
+                        if not move in dict_out[monname]:
+                            custom_teachable_compatibilities[monname].append(move)
+            # actually store the data in custom.json
+            if os.path.exists("./tools/learnset_helpers/porymoves_files/custom.json"):
+                f2 = open("./tools/learnset_helpers/porymoves_files/custom.json", "r")
+                custom_json = json.load(f2)
+                f2.close()
+            else:
+                custom_json = {}
+            for x in custom_teachable_compatibilities:
+                if len(custom_teachable_compatibilities[x]) == 0:
+                    continue
+                if not x in custom_json:
+                    custom_json[x] = {"LevelMoves": [], "PreEvoMoves": [], "TMMoves": [], "EggMoves": [], "TutorMoves": []}
+                for move in custom_teachable_compatibilities[x]:
+                    custom_json[x]["TutorMoves"].append(move)
+                f2 = open("./tools/learnset_helpers/porymoves_files/custom.json", "w")
+                f2.write(json.dumps(custom_json, indent=2))
+                f2.close()
+            print("FIRST RUN: Updated custom.json with teachable_learnsets.h's data")
+            # rerun the process
+            dict_out = construct_compatibility_dict(False)
+    return dict_out
+
+compatibility_dict = construct_compatibility_dict(True)
+
+# actually prepare the file
+with open("./src/data/pokemon/teachable_learnsets.h", 'r') as file:
+    out = file.read()
+    list_of_mons = re.findall('static const u16 s(.*)TeachableLearnset', out)
+for mon in list_of_mons:
+    mon_parsed = parse_mon_name(mon)
+    tm_learnset = []
+    tutor_learnset = []
+    if mon_parsed == "NONE" or mon_parsed == "MEW":
+        continue
+    if not mon_parsed in compatibility_dict:
+        print("Unable to find %s in json" % mon)
+        continue
+    for move in tm_moves:
+        if move in universal_moves:
+            continue
+        if move in tm_learnset:
+            continue
+        if move in compatibility_dict[mon_parsed]:
+            tm_learnset.append(move)
+            continue
+    for move in tutor_moves:
+        if move in universal_moves:
+            continue
+        if move in tutor_learnset:
+            continue
+        if move in compatibility_dict[mon_parsed]:
+            tutor_learnset.append(move)
+            continue
+    tm_learnset.sort()
+    tutor_learnset.sort()
+    tm_learnset += tutor_learnset
+    repl = "static const u16 s%sTeachableLearnset[] = {\n    " % mon
+    if len(tm_learnset) > 0:
+        repl += ",\n    ".join(tm_learnset) + ",\n    "
+    repl += "MOVE_UNAVAILABLE,\n};"
+    newout = re.sub('static const u16 s%sTeachableLearnset\[\] = {[\s\S]*?};' % mon, repl, out)
+    if newout != out:
+        out = newout
+        print("Updated %s" % mon)
+
+# add/update header
+header = "//\n// DO NOT MODIFY THIS FILE! It is auto-generated from tools/learnset_helpers/teachable.py\n//\n\n"
+longest_move_name = 0
+for move in tm_moves + tutor_moves:
+    if len(move) > longest_move_name:
+        longest_move_name = len(move)
+longest_move_name += 2 # + 2 for a hyphen and a space
+if longest_move_name < len("Found near-universal moves:"):
+    longest_move_name = len("Found near-universal moves:")
+
+def header_print(str):
+    global header
+    header += "// " + str + " " * (longest_move_name - len(str)) + " //\n"
+
+header += "// " + longest_move_name * "*" + " //\n"
+header_print("Found TM/HM moves:")
+for move in tm_moves:
+    header_print("- " + move)
+header += "// " + longest_move_name * "*" + " //\n"
+header_print("Found tutor moves:")
+tutor_moves.sort() # alphabetically sort tutor moves for easier referencing
+for move in tutor_moves: 
+    header_print("- " + move)
+header += "// " + longest_move_name * "*" + " //\n"
+header_print("Found near-universal moves:")
+universal_moves.sort() # alphabetically sort near-universal moves for easier referencing
+for move in universal_moves:
+    header_print("- " + move)
+header += "// " + longest_move_name * "*" + " //\n\n"
+
+if not "// DO NOT MODIFY THIS FILE!" in out:
+    out = header + out
+else:
+    out = re.sub("\/\/\n\/\/ DO NOT MODIFY THIS FILE!(.|\n)*\* \/\/\n\n", header, out)
+
+with open("./src/data/pokemon/teachable_learnsets.h", 'w') as file:
+    file.write(out)