This commit is contained in:
Alex Bates 2021-01-15 16:28:05 +00:00
parent c5d4a5a05c
commit 2eae138610
10 changed files with 105 additions and 402 deletions

6
.vscode/tasks.json vendored
View File

@ -2,9 +2,9 @@
"version": "2.0.0",
"tasks": [
{
"label": "make",
"label": "ninja",
"type": "shell",
"command": "PM_HEADER_REBUILD=1 make all",
"command": "ninja",
"problemMatcher": [
{
"fileLocation": ["relative", "${workspaceFolder}"],
@ -35,7 +35,7 @@
{
"label": "diff",
"type": "shell",
"command": "PM_HEADER_REBUILD=1 ./diff.py -mwo ${input:funcName}",
"command": "./diff.py -mwo ${input:funcName}",
"isBackground": true,
"problemMatcher": [
{

View File

@ -1,20 +1,22 @@
## Contributing
# Contributing
### Dependencies
Thank you for your interest in contributing to this project!
## Dependencies
There are a few additional dependencies needed when contributing to this project. You can install them with `./install.sh --extra`.
### WSL
We provide [windows_terminal.bat](tools/windows_terminal.bat) to open a [Windows Terminal](https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701) with a recommended layout. Make sure a default distro (e.g. `wsl --set-default ubuntu`) is set beforehand.
## Build System
### Rebuilding
You will need to re-run `./configure.py` whenever `splat.yaml` or the number of source files changes (e.g. a function is matched and its `.s` file is removed). If you `git pull` and `ninja` breaks, you probably need to re-run `./configure.py`.
If you use Visual Studio Code, you can use _Run Build Task_ (Ctrl+Shift+B) to run `make`. Any errors or warnings generated by the compiler will show up in the _Problems_ tab.
If you use Visual Studio Code, you can use _Run Build Task_ (Ctrl+Shift+B) to run `ninja`. Any errors or warnings generated by the compiler will show up in the _Problems_ tab.
### Matching a function
#### Setup
## Tutorial: Matching a function
### Setup
Once you've created a successful (`OK`) build, copy `build/` to `expected/build/`:
@ -23,7 +25,7 @@ $ mkdir -p expected
$ cp -r build expected
```
#### Roughly converting assembly to C
### Roughly converting assembly to C
Decide on a function to match. These can be found in the subdirectories of `asm/nonmatchings/`. Currently, functions which use float constants, data sections, or jump tables are unmatchable.
@ -31,9 +33,7 @@ Take the relevant `.s` file and pass it to [mips_to_c](https://github.com/matt-k
You can also use mips_to_c locally installed to a destination of your choice. Then register a function in `~/.bashrc` that calls `path/to/mips_to_c.py (with args)`:
```
sudo apt update
sudo apt install python3-pip
sudo git clone https://github.com/matt-kempster/mips_to_c.git path/to/mips_to_c
git clone https://github.com/matt-kempster/mips_to_c /path/to/mips_to_c
```
Here's a starter function you can use:
@ -62,13 +62,13 @@ Fix any errors and rerun `diff.py`. This will involve typing the function signat
Once a successful build is made, `diff.py` will show you the difference between the original game's assembly (on the left) and what your C code generated (on the right).
#### Matching the function
### Matching the function
You're on your own now. Get your C code compiling to match the original assembly! `diff.py`, when running, will automatically recompile your code whenever you save the `.c` file.
If you use Visual Studio Code, you can use _Run Test Task_ to run `diff.py` and show you errors and warnings from the compiler inline. You might want to attach _Run Test Task_ to a keybinding, as you'll be using it often.
#### After matching
### After matching
Once you've matched a function, run the following scripts:
@ -78,3 +78,5 @@ $ ./format.sh
```
If `format.sh` has any problems with your code, go and fix the issues. If you can't fix a warning without making the function not match anymore, append `// NOLINT` to the offending line.
Then, please [create a pull request](https://github.com/pmret/nananian/pulls)!

62
INSTALL.md Normal file
View File

@ -0,0 +1,62 @@
# Setup
This repository supports:
- [Linux](#unix)
- (macOS)(#unix)
- [Windows Subsystem for Linux 2](#wsl-2)
- [Docker](#docker) (any host OS)
If you encounter any issues setting up the repo, please feel free to [reach out to us on Discord](https://discord.gg/urUm3VG).
## Unix
Clone the repository:
```sh
git clone https://github.com/pmret/papermario
cd papermario
```
Install build dependencies:
```sh
./install.sh
```
Copy a Paper Mario (USA) ROM into the root directory of the repository with the name `baserom.z64`. If you're using WSL, you can enter the Linux filesystem by opening `\\wsl$` in File Explorer; e.g. `\\wsl$\Ubuntu\home\<your username>\papermario`.
Configure the build and extract assets from the base ROM:
```sh
./configure.py
```
Compile the game:
```
ninja
```
If you get `papermario.z64: OK` at the end, the build succeeded!
The output ROM is `papermario.z64` - you can run this in an emulator such as Project64.
## WSL 2
1. Install or upgrade to **WSL 2** following [these instructions](https://aka.ms/wsl2-install) (Ubuntu is recommended)
2. Open a WSL terminal
3. Run the following command: `sudo apt update && sudo apt upgrade && cd ~`
4. Continue with [the instructions for Linux](#unix)
If you have Visual Studio Code, you can type code . to open the repo within it.
`Ctrl + J` opens up a Linux terminal within VS Code.
## Docker
A Docker image containing all dependencies can be built and ran as follows:
```sh
docker build . -t pm
docker run --rm -ti -v $(pwd):/papermario pm
```
Then continue with [the instructions for Linux](#unix).

1
Jenkinsfile vendored
View File

@ -18,6 +18,7 @@ pipeline {
}
steps {
sh 'python3 progress.py --csv >> /var/www/papermar.io/html/reports/progress.csv'
sh 'python3 progress.py --shield-json >> /var/www/papermar.io/html/reports/progress_shield.json'
}
}
}

313
Makefile
View File

@ -1,313 +0,0 @@
### Build Options ###
# Override these options in settings.mk or with `make SETTING=value'.
BASEROM = baserom.z64
TARGET = papermario
COMPARE = 1
NON_MATCHING = 0
WATCH_INCLUDES = 1
WSL_ELEVATE_GUI = 1
# Fail early if baserom does not exist
ifeq ($(wildcard $(BASEROM)),)
$(error Baserom `$(BASEROM)' not found.)
endif
# NON_MATCHING=1 implies COMPARE=0
ifeq ($(NON_MATCHING),1)
override COMPARE=0
endif
# PERMUTER=1 implies WATCH_INCLUDES=0
ifeq ($(PERMUTER),1)
override WATCH_INCLUDES=0
endif
### Output ###
BUILD_DIR := build
ROM := $(TARGET).z64
ELF := $(BUILD_DIR)/$(TARGET).elf
LD_SCRIPT := $(TARGET).ld
LD_MAP := $(BUILD_DIR)/$(TARGET).map
ASSETS_BIN := $(BUILD_DIR)/bin/assets/assets.bin
MSG_BIN := $(BUILD_DIR)/msg.bin
NPC_BIN := $(BUILD_DIR)/sprite/npc.bin
### Tools ###
PYTHON := python3
N64CKSUM := tools/n64crc
SPLAT_YAML := tools/splat.yaml
SPLAT = $(PYTHON) tools/n64splat/split.py $(BASEROM) $(SPLAT_YAML) .
YAY0COMPRESS = tools/Yay0compress
EMULATOR = mupen64plus
CROSS := mips-linux-gnu-
AS := $(CROSS)as
OLD_AS := tools/mips-nintendo-nu64-as
CC := tools/cc1
CPP := cpp
LD := $(CROSS)ld
OBJCOPY := $(CROSS)objcopy
WSL := 0
JAVA := java
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S),Linux)
OS=linux
ICONV := iconv --from UTF-8 --to SHIFT-JIS
ifeq ($(findstring microsoft,$(shell cat /proc/sys/kernel/osrelease)),microsoft)
WSL := 1
ifeq ($(WSL_ELEVATE_GUI),1)
JAVA := powershell.exe -command java
endif
endif
endif
ifeq ($(UNAME_S),Darwin)
OS=mac
ICONV := tools/iconv.py UTF-8 SHIFT-JIS
endif
OLD_AS=tools/$(OS)/mips-nintendo-nu64-as
CC=tools/$(OS)/cc1
### Compiler Options ###
CPPFLAGS := -Iinclude -Isrc -D _LANGUAGE_C -D _FINALROM -ffreestanding -DF3DEX_GBI_2 -D_MIPS_SZLONG=32 -Wundef -Wcomment
ASFLAGS := -EB -Iinclude -march=vr4300 -mtune=vr4300
OLDASFLAGS := -EB -Iinclude -G 0
CFLAGS := -O2 -quiet -G 0 -mcpu=vr4300 -mfix4300 -mips3 -mgp32 -mfp32 -Wimplicit -Wuninitialized -Wshadow
LDFLAGS := -T undefined_syms.txt -T undefined_syms_auto.txt -T undefined_funcs.txt -T undefined_funcs_auto.txt -T $(BUILD_DIR)/$(LD_SCRIPT) -Map $(LD_MAP) --no-check-sections
ifeq ($(WATCH_INCLUDES),1)
CPPMFLAGS = -MP -MD -MF $@.mk -MT $(BUILD_DIR)/$*.d
MDEPS = $(BUILD_DIR)/%.d
endif
ifeq ($(NON_MATCHING),1)
CPPFLAGS += -DNON_MATCHING
endif
-include settings.mk
### Sources ###
include sources.mk
ifeq ($(PERMUTER),1)
override OBJECTS:=$(filter %.c.o, $(OBJECTS))
endif
%.d: ;
ifeq ($(WATCH_INCLUDES),1)
-include $(foreach obj, $(OBJECTS), $(obj).mk)
endif
NPC_DIRS := $(foreach npc, $(NPC_SPRITES), sprite/npc/$(npc))
GENERATED_HEADERS := include/ld_addrs.h $(foreach dir, $(NPC_DIRS), include/$(dir).h)
### Targets ###
clean:
rm -rf $(BUILD_DIR) $(LD_SCRIPT)
clean-all:
rm -rf $(BUILD_DIR) bin msg img sprite .splat_cache $(LD_SCRIPT)
clean-code:
rm -rf $(BUILD_DIR)/src
tools:
setup: clean-all tools
@make split
split:
make $(LD_SCRIPT) -W $(SPLAT_YAML)
split-%:
$(SPLAT) --modes ld $* --verbose
split-all:
$(SPLAT) --modes all
test: $(ROM)
$(EMULATOR) $<
%.bin: $(LD_SCRIPT)
# Compressed files
%.Yay0: %
@mkdir -p $(shell dirname $@)
$(YAY0COMPRESS) $< $@
# $(BUILD_DIR)/%.bin.Yay0: %.bin
# @mkdir -p $(shell dirname $@)
# $(YAY0COMPRESS) $< $@
# Data objects
# $(BUILD_DIR)/%.bin.o: %.bin
# @mkdir -p $(shell dirname $@)
# $(LD) -r -b binary -o $@ $<
# Compressed data objects
# $(BUILD_DIR)/%.Yay0.o: $(BUILD_DIR)/%.bin.Yay0
# @mkdir -p $(shell dirname $@)
# $(LD) -r -b binary -o $@ $<
# Compile C files
# $(BUILD_DIR)/%.c.o: %.c $(MDEPS) | $(GENERATED_HEADERS)
# @mkdir -p $(shell dirname $@)
# $(CPP) $(CPPFLAGS) -o - $(CPPMFLAGS) $< | $(ICONV) | $(CC) $(CFLAGS) -o - | $(OLD_AS) $(OLDASFLAGS) -o $@ -
# # Compile C files (with DSL macros)
# $(foreach cfile, $(DSL_C_FILES), $(BUILD_DIR)/$(cfile).o): $(BUILD_DIR)/%.c.o: %.c $(MDEPS) tools/compile_dsl_macros.py | $(GENERATED_HEADERS)
# @mkdir -p $(shell dirname $@)
# $(CPP) $(CPPFLAGS) -o - $< $(CPPMFLAGS) | $(PYTHON) tools/compile_dsl_macros.py | $(ICONV) | $(CC) $(CFLAGS) -o - | $(OLD_AS) $(OLDASFLAGS) -o $@ -
# Assemble handwritten ASM
# $(BUILD_DIR)/%.s.o: %.s
# @mkdir -p $(shell dirname $@)
# $(AS) $(ASFLAGS) -o $@ $<
# # Data
# $(BUILD_DIR)/data/%.data.o: asm/data/%.data.s
# @mkdir -p $(shell dirname $@)
# $(AS) $(ASFLAGS) -o $@ $<
# # Rodata
# $(BUILD_DIR)/rodata/%.rodata.o: asm/data/%.rodata.s
# @mkdir -p $(shell dirname $@)
# $(AS) $(ASFLAGS) -o $@ $<
# Images
# $(BUILD_DIR)/%.png.o: $(BUILD_DIR)/%.png
# $(LD) -r -b binary -o $@ $<
# $(BUILD_DIR)/%.rgba16.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py rgba16 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.rgba32.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py rgba32 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.ci8.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py ci8 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.ci4.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py ci4 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.palette.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py palette $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.ia4.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py ia4 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.ia8.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py ia8 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.ia16.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py ia16 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.i4.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py i4 $< $@ $(IMG_FLAGS)
# $(BUILD_DIR)/%.i8.png: %.png
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/convert_image.py i8 $< $@ $(IMG_FLAGS)
# Assets
# ASSET_FILES := $(foreach asset, $(ASSETS), $(BUILD_DIR)/bin/assets/$(asset))
# YAY0_ASSET_FILES := $(foreach asset, $(filter-out %_tex, $(ASSET_FILES)), $(asset).Yay0)
# $(BUILD_DIR)/bin/assets/%: bin/assets/%.bin
# @mkdir -p $(shell dirname $@)
# @cp $< $@
# $(ASSETS_BIN): $(ASSET_FILES) $(YAY0_ASSET_FILES) sources.mk
# @mkdir -p $(shell dirname $@)
# @echo "building $@"
# @$(PYTHON) tools/build_assets_bin.py $@ $(ASSET_FILES)
# $(ASSETS_BIN:.bin=.o): $(ASSETS_BIN)
# $(LD) -r -b binary -o $@ $<
# Messages
# $(MSG_BIN): $(MESSAGES)
# @mkdir -p $(shell dirname $@)
# @echo "building $@"
# @$(PYTHON) tools/compile_messages.py $@ /dev/null $(MESSAGES)
# $(MSG_BIN:.bin=.o): $(MSG_BIN)
# $(LD) -r -b binary -o $@ $<
# Sprites
# $(foreach npc, $(NPC_SPRITES), $(eval $(BUILD_DIR)/sprite/npc/$(npc):: $(shell find sprite/npc/$(npc) -type f 2> /dev/null))) # dependencies
# NPC_YAY0 := $(foreach npc, $(NPC_SPRITES), $(BUILD_DIR)/sprite/npc/$(npc).Yay0)
# $(BUILD_DIR)/sprite/npc/%:: sprite/npc/% tools/compile_npc_sprite.py
# @mkdir -p $(shell dirname $@)
# $(PYTHON) tools/compile_npc_sprite.py $@ $<
# $(NPC_BIN): $(NPC_YAY0) tools/compile_npc_sprites.py
# @mkdir -p $(shell dirname $@)
# @echo "building $@"
# @$(PYTHON) tools/compile_npc_sprites.py $@ $(NPC_YAY0)
# $(NPC_BIN:.bin=.o): $(NPC_BIN)
# $(LD) -r -b binary -o $@ $<
# include/sprite/npc/%.h: sprite/npc/%/SpriteSheet.xml tools/gen_sprite_animations_h.py
# @mkdir -p $(shell dirname $@)
# @echo "building $@"
# @$(PYTHON) tools/gen_sprite_animations_h.py $@ sprite/npc/$* $(NPC_DIRS)
### Linker ###
# $(LD_SCRIPT): $(SPLAT_YAML)
# $(SPLAT) --modes ld bin Yay0 PaperMarioMapFS PaperMarioMessages img PaperMarioNpcSprites --new
# $(BUILD_DIR)/$(LD_SCRIPT): $(LD_SCRIPT)
# @mkdir -p $(shell dirname $@)
# $(CPP) -P -DBUILD_DIR=$(BUILD_DIR) -o $@ $<
# $(ROM): $(BUILD_DIR)/$(TARGET).bin
# @cp $< $@
# ifeq ($(COMPARE),1)
# @sha1sum -c checksum.sha1 || (echo 'The build succeeded, but did not match the base ROM. This is expected if you are making changes to the game. To skip this check, use "make COMPARE=0".' && false)
# endif
# $(BUILD_DIR)/$(TARGET).elf: $(BUILD_DIR)/$(LD_SCRIPT) $(OBJECTS)
# $(LD) $(LDFLAGS) -o $@
# $(BUILD_DIR)/$(TARGET).bin: $(BUILD_DIR)/$(TARGET).elf
# $(OBJCOPY) $< $@ -O binary
# include/ld_addrs.h: $(BUILD_DIR)/$(LD_SCRIPT)
# grep -E "[^\. ]+ =" $< -o | sed 's/^/extern void* /; s/ =/;/' > $@
### Star Rod (optional) ###
STAR_ROD := cd tools/star-rod && $(JAVA) -jar StarRod.jar
sprite/SpriteTable.xml: tools/star-rod sources.mk
$(PYTHON) tools/star-rod/spritetable.xml.py $(NPC_SPRITES) > $@
editor: tools/star-rod sprite/SpriteTable.xml
$(STAR_ROD)
### Make Settings ###
.PHONY: clean tools test setup split editor $(ROM)
.DELETE_ON_ERROR:
.SECONDARY:
.PRECIOUS: $(ROM) %.Yay0
.DEFAULT_GOAL := $(ROM)
# Remove built-in implicit rules to improve performance
MAKEFLAGS += --no-builtin-rules
# Fail targets if any command in the pipe exits with error
SHELL = /bin/bash -e -o pipefail

View File

@ -1,75 +1,22 @@
# Paper Mario
This is a WIP decompilation of Paper Mario (USA). It builds the following ROM:
[![Build Status][jenkins-badge]][jenkins] [![Progress][progress-badge]][progress] [![Discord Channel][discord-badge]][discord]
* papermario.z64 `md5: a722f8161ff489943191330bf8416496`
[jenkins]: https://jenkins.zelda64.dev/job/papermario/job/master
[jenkins-badge]: https://img.shields.io/jenkins/build?jobUrl=https%3A%2F%2Fjenkins.zelda64.dev%2Fjob%2Fpapermario%2Fjob%2Fmaster
Discord: [Paper Mario Modding](https://discord.gg/urUm3VG)
[progress]: https://papermar.io/progress
[progress-badge]: https://img.shields.io/endpoint?url=https://papermar.io/reports/progress_shield.json
## Setup
[discord]: https://discord.gg/urUm3VG
[discord-badge]: https://img.shields.io/discord/279322074412089344?color=%237289DA&logo=discord&logoColor=ffffff
You'll need Linux, a Linux VM, or [Windows 10 (WSL2)](#wsl) to work on this project.
This is a work-in-progress decompilation of Paper Mario (USA).
#### Clone the repository
```sh
$ git clone https://github.com/ethteck/papermario.git
$ cd papermario
```
It builds the following ROM:
#### Install build dependencies
* [**papermario.z64**](https://datomatic.no-intro.org/index.php?page=show_record&s=24&n=0537) `sha1: 3837f44cda784b466c9a2d99df70d77c322b97a0`
```sh
$ ./install.sh
```
To set up the repository, see [INSTALL.md](INSTALL.md).
Our install script does not yet support distros other than Ubuntu, Arch, and their derivatives. Please consider contributing to the script if you use another distro!
##### Docker
A Docker image containing all dependencies can be built and ran as follows:
```sh
# build image
$ docker build . -t pm
# spin up container, mounting current directory inside
$ docker run --rm -ti -v $(pwd):/papermario pm
```
#### Base ROM
You'll need a Paper Mario (USA) ROM to work on this project. Copy it into the root directory of the repository with the name `baserom.z64`.
#### Install tools and extract ROM
```sh
$ make setup
```
### Compile the game
```sh
$ make
```
If you get `OK`, you're all set! Otherwise, please feel free to reach out to us in [our Discord channel](https://discord.gg/urUm3VG).
## Star Rod GUI
You can open [our modified version of Star Rod](https://github.com/nanaian/star-rod-for-decomp) with `make editor`.
## Contributing
Please see [CONTRIBUTING.md](CONTRIBUTING.md) for information on how to contribute to the project. Any and all help is welcome!
## Troubleshooting
* If you are using Windows, you may encounter the following when you run `make`:
```
sha1sum -c checksum.sha1
sha1sum: 'papermario.z64'$'\r': No such file or directory
: FAILED open or read
sha1sum: WARNING: 1 listed file could not be read
Makefile:118: recipe for target 'verify' failed
make: *** [verify] Error 1
```
> 💡 Solution
> Run `git checkout checksum.sha1` and retry building. Windows has different line endings than Linux, causing some of our tools to break.

View File

@ -153,12 +153,6 @@ async def main():
cpp = args.cpp or "cpp"
"""
# update submodules
await shell("git submodule init tools/n64splat")
await shell("git submodule update --recursive")
"""
# compile n64splat dependencies
await shell("make -C tools/n64splat")

View File

@ -79,6 +79,16 @@ def main(args):
csv_list = [str(version), timestamp, git_hash, str(len(all_funcs)), str(len(nonmatching_funcs)),
str(len(matching_funcs)), str(total_size), str(nonmatching_size), str(matching_size)]
print(",".join(csv_list))
elif args.shield_json:
import json
# https://shields.io/endpoint
print(json.dumps({
"schemaVersion": 1,
"label": "progress",
"message": f"{matching_ratio:.2f}%",
"color": "yellow",
}))
else:
if matching_size + nonmatching_size != total_size:
print("Warning: category/total size mismatch!\n")
@ -89,6 +99,7 @@ def main(args):
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Reports progress for the project")
parser.add_argument("--csv", action="store_true")
parser.add_argument("--shield-json", action="store_true")
args = parser.parse_args()
main(args)

View File

@ -3,3 +3,4 @@ cxxfilt
python-Levenshtein
stringcase
watchdog
gitpython

View File

@ -1,2 +0,0 @@
REM first set your default distro using: wsl --set-default <Distro>
wt --title "diff.py" -d "../" ; split-pane -d "../" -V ; new-tab --title "mips_to_c.py" -d "../" ; focus-tab -t 0