Alright everyone, here's the guide as promised! It turned out a little long, but I think as a result it should be detailed enough to actually convey basically everything I myself currently know and think of this workflow. Please take your time and let me know if anything is wrong or you run into problems, thank you!
Prerequisites
Firstly, please make sure you are familiar with using Lunar Magic, GPS, PIXI, AddMusicK, UberASMTool, FLIPS, Asar and just hacking in general at least a little bit. If you aren't, I'm sure you can find plenty of useful resources in the
tutorials section. I would really only recommend this sort of setup if you are already pretty familiar with the basic tools and tasks involved in hacking.
Brief intro to git
This will be a short introduction to what I think are the most important parts of git to get you started, but you should feel free to check out other resources on git, since there are plenty out there (
here's just one for example and a
cheatsheet Atari recommended) and probably a lot that are better and more thorough. If you're already familiar with git or have consulted some other source for the basics you can safely skip this section, I will talk about using git specifically for hacking later on!
As previously stated, git in its essence is a program that allows you to save snapshots of your files. It's actually a little smarter than that even, instead of saving an entire backup of your work, it will only save the changes you made since the last snapshot. If you go back to a previous snapshot, it merely reverts the changes between your current snapshot and the one you want to revert to. In this way, you can think of these "snapshots" not as separate things, but little checkpoints that connect to form a path from the start of your project towards the current state of your project, each containing little changes that together add up to the entirety of your work!
Here are some terms that are pretty essential to git's workflow:
commit: What I previously referred to as "snapshot" or "checkpoint", it's a set of changes you've made to files since your last commit. You can add a title and description to your commits to state and explain the changes you made, which makes it very easy to tell when and why what was changed[1].
repository (often abbreviated to repo): A structure git creates which contains the history of all changes you ever make to your project, basically a collection of all your commits.
push: A repository can be "pushed", meaning essentially "uploaded" to some remote location, usually a website that specializes in hosting git repositories like GitHub, GitLab, Bitbucket, etc., from which it can later be restored.
pull: You can "pull" or "download" changes made to your "remote" repository (the one hosted on GitHub or a similiar site) to integrate them into your "local" repository (the one you have on your own computer).
clone: You can "clone" a remote repository to create a copy of it (a local repository) on your own computer. The difference between pull and clone is basically that clone creates the repository again from the ground up (for example if you somehow lost your local repository), while pull only applies new changes it finds in the remote repository to your local repository (for example if someone you're working with made some changes since you last pushed).
staging area: The staging area is where you add your changes to before you commit them, its use is basically to let you split your changes into multiple smaller commits and to make it easier for you to tell what you are actually committing in the first place. Changes that are currently in the staging area and not yet committed are often said to be "staged".
branch: I previously explained how your commits form a path from the start of your project towards the current state of the project, this was a bit of a simplification, with git you can actually have multiple such "paths" or branches, that can split, merge or end at any point where there is a commit, forming something similar to an actual road network. Branches are useful if you want to work on different aspects of your project in isolation or if you're working together with others, but there are some pitfalls specific to using branches while hacking that you should be aware of that I will explain later! As you may expect, you must always have at least one such branch, or your project would not even have a path to walk along, this is called the "main"[2] branch.
conflict: A (merge) conflict occurs when two or more conflicting changes are made to the same file. In a solo project you should normally never run into a conflict if you're only using one (the main) branch. Resolving such conflicts is a relatively simple affair with text files, you simply pick the lines you want to keep or discard from each set of changes, but unfortunately basically all the files Lunar Magic exports are binary files, where "lines" to keep or discard don't really exist, which complicates matters. I will discuss specifics about merge conflicts and the problems with binary files in the context of hacking later.
.gitignore file: An incredibly important feature of git, .gitignore is a file you can create that contains specifications about which files or folders you want git to ignore completely. Any changes you make to files specified in .gitignore will be ignored by git, it will basically fully disregard their existance and never add them to the repository. Obviously this is very important in our context and I will talk about which files we probably want to ignore later on (spoiler: ROMs!!).
Some basic git commands
I will now list some important git commands, you can find a list of every command git has
here.
git clone <url>
Command used to create a local copy of a remote repository. You can obtain the <url> to the specificy repository you want to clone from the website the remote repository is hosted on.
git init
Command used to create a new repository in the current folder.
git add <path>
The command we use to add changes we made to the staging area. The <path> specifies the file(s) we would like to add. You can specify the path to a folder instead of single files too and git will add all changes to files it can find in that folder or the folders it contains to the staging area.
For example, if I'm in the root of my project and I do 'git add levels' git will stage all changes it can find in the "levels" folder. 'git add .' is very useful (note the dot), it will add all changes it can find in the current directory (denoted by the dot) or any folders inside it. Using this we can quickly add all changes in our whole project to the staging area if we want to!
Note that you can use git add as often as you want to, you don't have to add all your changes in one go.
git status
This command will show you which changes git currently sees and which ones are already in the staging area.
git commit
The command we use to commit the changes that are currently in the staging area. By default, if you use just "git commit" you will be prompted via some text editor to enter a commit message, if you want to just write a quick little note instead, you can do 'git commit -m <your commit message here>' for example 'git commit -m "Remove a Goomba"' will commit your changes with the message "Remove a Goomba". I would generally recommend using the text editor though, since you ideally want to be as descriptive as possible for the sake of organization.
git log
This command will show you the history of your repository, meaning your commits and commit messages as well as which branch they were created in, when they were created and by whom, as well as their hash (basically an ID that can be used to refer to the commit). The default git log format is not the prettiest so see [3] if you want to learn how to make it a little easier to look at.
git branch <name>
This command lets you create a new branch at the current position with the given <name>. For example, I could create a new branch called "rework_world_1" via 'git branch rework_world_1'.
git checkout <branch or commit hash>
This command lets you "check out" or "switch" the state of your project to the state it is/was in in the specified branch or commit. To checkout a commit, get its commit hash using git log and then do 'git checkout <hash here>' for example, if I wanted to check out a commit with the hash "283a94f" I would write 'git checkout 283a94f' and to check out a branch with the name "rework_world_1" I would write 'git checkout rework_world_1'.
The latest commit in a branch has a special name called "HEAD". To switch from a previous commit back to the latest state of your branch, let's say it's the "main" branch you can thus do either 'git checkout HEAD' or 'git checkout main'.
Note that if you're in a previous commit git will tell you that you're in a "detached head", this is nothing to worry about as long as you do not make any changes until you return to the latest (aka HEAD) commit as described earlier, see [4] for more info on "detached head".
git restore <path>
Inverse of git add, lets you remove files from the staging area if you accidentally added them.
git push
Command used to push changes to a remote repository.
git pull
Command used to pull changes from a remote repository. This command can cause merge conflicts under some circumstances.
git merge <branch>
Command used to merge <branch> into the current branch. Basically means git will try to automatically integrate the changes from the <branch> into your current branch. This command can cause merge conflicts under some circumstances.
Here are a few more useful commands which I use sometimes but will not explain in detail here (you can find plenty of resources on them elsewhere if you're interested):
- git stash
- git rebase
- git tag
- git reset
- git bisect
The actual setup
I hope I didn't scare you off with all the git commands and such. They may look scary at first but in my experience you get used to the workflow pretty quickly, especially because to start out with you don't actually need that many commands.
Let's now actually set up our example project as promised!
Required resources
Now, I'm going to list a bunch of stuff you will need to follow this guide from start to finish, you may either download these all right now and keep them handy for later in the guide, or you can come back and download them as soon as we actually need them, up to you:
-
git (I would recommend installing it immediately if you don't have it yet)
- Lunar Magic
3.31 or
3.30
-
AddMusicK
-
Asar
-
GPS
-
PIXI
-
UberASMTool
-
FLIPS (you may get the version from this comment for now, the version SMWC is hosting is outdated by 6 years and has a bug that makes Lunar Helper not work, note that I cannot guarantee that the version in this comment is safe so you'll have to download and use it at your own risk, wait for SMWC to update FLIPS or compile an up-to-date version yourself!)
-
Lunar Helper + Lunar Monitor
-
initial_patch.bps (will be explained later!)
-
GitKraken (optional), offers a nice GUI for git but is only free for public repositories (or if you're a student), so I will be sticking to using git via the command line for this guide
Folder structure
For the purposes of this guide, I will create a new folder and call it "example_project". I will then create two empty folders in it and call one "my_hack" and the other "tools". "my_hack" will be our actual project folder which we will version control with git and "tools" will contain all our tools. It's also totally fine to keep your tools in your project folder and to version control them too, I'm just leaving them out of the project for simplicity. I will talk about pros and cons of keeping your tools inside/outside your project a bit at the end of this guide.
This is what your "example_project" folder should look like now:
Now I'm going to open up my "tools" folder and just dump all the tools I need in there. It should look like this afterwards:
Here are links to screenshots of the contents of each folder just for completeness (nothing exciting, just extracted the files from the links above):
-
AddmuscK_1.0.8
-
asar181
-
FLIPS
-
GPS (V1.4.21)
-
Lunar Magic 3.31
-
pixi_v1.32
-
UberASMTool14
Note that I did add a (for now) empty "list.txt" file in PIXI's folder and you should do so too, because that's the sprite list file Lunar Helper will tell PIXI to use later!
At this point, open the Lunar Helper + Lunar Monitor bundle's "lunar-monitor-v0.3.1" folder. Take the "lunar-monitor-injector.exe", "usertoolbar.txt" and one of the two DLLs in the "LM3.30" or "LM3.31" folders (depends on which Lunar Magic version you're using, I'm using 3.31 for this guide) and copy them into your Lunar Magic folder. Afterwards, it should look like this:
While you're here, also take the "lunar-monitor-config.txt" file and copy it to your "my_hack" folder.
Let's make a "Lunar Helper" folder inside "my_hack", then take the "LunarHelper.exe", "config_project.txt" and "config_user.txt" files from the Lunar Helper + Lunar Monitor bundle's "lunar-helper-v1.0.0-LMC" folder and put them in this folder.
Afterwards, your "Lunar Helper" folder inside your "my_hack" folder should look like this:
and your "my_hack" folder like this:
Let's add a few more folders, a "Levels" folder and a "Other" folder which will contain our extracted resources later, as well as a "Patches" folder for our patches to go:
While we're at it, let's also add a "clean.smc" and "my_hack.smc" like this ("clean.smc" should obviously be a clean ROM, "my_hack.smc" will also be a clean ROM for the purposes of this tutorial but if you were porting an existing project this should be your hacked ROM):
To finish our folder setup, please take the "initial_patch.bps" file I linked above and copy it to the "Other" folder.
And that's it for our folder setup, next we will be configuring Lunar Helper and Lunar Monitor!
Configuration
Let's start with the "config_user.txt" file in your "Lunar Helper" folder. Open it in a text editor of your choice and you should see something like this:
Most of this should be pretty self-explanatory, just change the paths to the paths of the tools we set up (most of them should already fit mostly), you can delete the retroarch variables for the purposes of this tutorial, or, if you have retroarch and know where it is, you can change those paths too! (Note if you're not that familiar with relative paths: ".." denotes the parent folder of the current folder, so in our case, ".." would refer to our "example_project" folder, since our "dir" is pointing to our "my_hack" folder, which is inside our "example_project" folder, so this way we can refer to our "tools" folder, which is not actually inside our "my_hack" folder, using "../tools"!)
This is what my "config_user.txt" looks like after I edit it, your "dir" will very likely be different, because the path on your local system to your project folder is going to be different:
Let's now switch over to "config_project.txt". There are a few more configuration variables here but most should be pretty self-explanatory again, just change the paths to match our folder structure. I'm going to remove the "title_moves", "test_level" and "test_level_dest" lines for the sake of simplicity. Since we don't have any patches yet, the "patches" list will be empty for now (I removed some comments from the file for brevity, but you can keep them if you want):
Note that the initial_patch variable specifies the path to a .bps patch that will be applied during the build process before anything else. This is so you can apply Lunar Magic hijacks that other patches and tools might rely on before those are applied. There is a full guide on creating such an initial_patch in the readme, I only included one here to make the guide more convenient to follow. Depending on what you need (FastROM, SA-1, compression options, etc.) you may want your initial_patch to be different, please consult the readme for details on this when you start a real project!
If you have a keen eye, you might have noticed that the "gps_path", "pixi_path", "addmusick_path" and "uberasm_path" are now pointing to folders
outside our project folder, you can tell this is the case because of the ".." referring to the parent directory as described earlier! This should cause you to wonder if you should be specifying these paths in the "config_project.txt" file, since it's the config file of the project, but we're referring to something outside of our project. Generally, I would highly recommend moving any variables that refer to paths outside your project from the "config_project.txt" file to the "config_user.txt" file. We will later make git ignore the "config_user.txt" file, since we do
not want to push configuration files that depend on folders that are not a part of the project (imagine if you were working together with someone and they had their tools installed somewhere else, the paths wouldn't match up with the ones in "config_user.txt" because they match
your folder structure but not theirs!).
After I move those variables from "config_project.txt" to "config_user.txt", mine look like this:
"config_project.txt":
"config_user.txt":
That's it for Lunar Helper's configuration, let's switch over to Lunar Monitor's now.
Open the "lunar-monitor-config.txt" file we copied over to our "my_hack" folder earlier, you should see something like this:
There's some obvious similarities between this and Lunar Helper's configuration files, just adjust the paths to match those we had for Lunar Helper (all paths are relative to the folder the config file is in!). I have this afterwards:
The last two variables, "log_level" and "log_path" specify where Lunar Monitor will log what it does to, so you can figure out what's wrong if it's not working and see what it exported when. The way they're set up is fine for our purposes here, if you want more attention-grabbing errors or a different log location you can check out the readme for additional details.
That should be it as far as configuration goes!
Finalizing and verifying our setup
It's finally time, open your Lunar Magic from your "tools" folder. First things first, let's turn of restore points (we don't need those were we're going). Under Options -> Restore Point Options... just uncheck all options, note that I believe this might apply to
all Lunar Magic executables, so if you actually want to keep Lunar Magic creating restore points for different hacks that might be kind of a problem (not much I can do about that right this second, I'll think about it though, sorry!).
Now open the "my_hack.smc" ROM inside Lunar Magic. In good ol' level 105, grab something, move it and move it back, just so we can save the level real quick without actually altering it. If saving takes a second and you have a "level 105.mwl" in your "Levels" directory afterwards, Lunar Monitor is working:
Next, open the palette editor, uncheck "Auto-Enable custom palette on edit", click on a color and just click OK (you don't need to alter the color). This will let you save shared palettes using the little save icon in the top right of the palette editor:
Click it, saving should take a second again and you should have a "shared.pal" file in your "Other" folder afterwards!
Now open the map16 editor, move stuff around and return it to its original position (just to be allowed to save, again).
Press the save icon on the map16 editor and you should have an "all.map16" file in your "Other" folder.
Finally, open the overworld editor. Again, just move stuff around and return it and click the save icon in the overworld editor. You should have a "global_data.bps" file in your "Other" folder afterwards.
Before closing Lunar Magic, make sure to export GFX and ExGFX, and then close it for now. Your "Other" folder should now look like this:
You can open "lunar-monitor-log.txt" to see what Lunar Monitor did, if there were any errors you should also see them here and be able to fix them (hopefully!).
Go ahead and open up "LunarHelper.exe" from your "Lunar Helper" folder, you should be greeted by a window containing this friendly menu:
Let's try building our ROM using the "B - Build" option, just press "B" on your keyboard and Lunar Helper will try to build our ROM from the (vanilla) resources we just extracted with Lunar Monitor.
You should see Lunar Helper applying all of the tools we specified and inserting our admittedly currently pretty uninteresting exported data.
If Lunar Helper tells you "ROM patched successfully to 'my_hack.smc'!", everything went well and our setup works!
Git repository setup
Now it's finally time to set up our git repository, exciting!
Open a command prompt (or PowerShell) in your "my_hack" folder. I believe on newer Windows versions you can do so by holding shift and right-clicking in/on the folder and selecting "Open command window here", if that does not work for you
here are a bunch of other ways to do it.
In this command prompt, simply type "git init" to create a new git repository in your project folder. If successful, you should get something like this (your folder path may differ, that's normal!):
If you get an error about "git" not being found or something, make sure you have git installed!
If you enter "git status", you'll get a nice little list of what files git is currently aware of:
But what's that, there are ROMs in there! Not good, git by default will try to track all files it can find in the whole project, but we don't want it to track our ROMs ("clean.smc" and "my_hack.smc") as well as any other formats similar to the .smc format that may contain ROM data. This is where .gitignore comes to the rescue! Create a ".gitignore" file in your "my_hack" folder, like this:
In your ".gitignore" file, add the following content:
"*" simply means that the files we want to ignore could have any name, we only care about their types (generally, "*" is a wildcard for any text/path). We're ignoring the Lunar Monitor config file, because it's not really actually part of our project, remember that our Lunar Magic executable is in the "tools" folder and so is Lunar Monitor. Since those aren't a part of our project, neither should the config file. I would always recommend ignoring Lunar Monitor's log file, because it's not something we want to track (after all, what's the point in restoring a log file?). We're also ignoring the user specific configuration for Lunar Helper since it's referring to stuff outside our "my_hack" folder as well.
Do "git status" again and voila, as you can see, no more ROMs, Lunar Monitor config or log files:
Go ahead and add use "git add ." to add all of your changes (in this case, all these new "untracked" files) to the staging area.
You can see all files that are in the staging area if you do "git status" again, it's a very useful command!
Now that they're staged, you can use "git commit" to create a commit for these changes like so:
Do "git status" again and you can see that there are no current changes. We just created the starting point for our project, give yourself a pat on the back!
Pushing our repository
To actually push our repository, I will create a new remote repository on
GitHub (you can also use GitLab, Bitbucket, etc. instead of course). An account is required to create repositories, but it's totally free! I would recommend making a private repository (like
this). Afterwards, you will be redirected to your new remote repo with instructions on how to push to it. We have an existing local repository, so we are interested in this:
Perform these steps in your command prompt (with your own repo URL of course!):
If you see something like this, it worked. Refresh your remote repository's page and you should see your repository was uploaded:
Working on our hack
That's all fine and good, but we haven't even actually worked on our hack yet. Let's do something about that!
Open up the Lunar Magic from the "tools" folder again and open "my_hack.smc" in it. In level 105, I never liked that first ledge very much, let's get rid of it (and the Koopa):
Save the level and Lunar Monitor will automatically export your changes.
If we do "git status" now, we will see a new change, that's the level file Lunar Monitor exported because we changed it:
We could make more changes and keep saving them and Lunar Monitor would keep exporting them, but let's commit our changes for now (as a rule of thumb, you should make commits whenever it feels logical to you, but I will talk about upsides and downsides of many/few commits later). Do "git add ." and "git commit":
If we want, we can also immediately push our change to the remote repository, so that it's stored there too:
For illustrative purposes, let's try "accidentally" deleting our ROM. Just go ahead delete your "my_hack.smc". Let's now rebuild it with Lunar Helper, just do what we did earlier and use its "B - Build" option. If everything goes well, you should get your ROM back exactly in the state we had it in. Open it up, the ledge is still gone, our changes were saved:
Right now you might be thinking "So what? Lunar Magic could have restored a ROM to this state again too", which is true, so let's kick it up a notch.
Make a copy of the "lunar-monitor-config.txt" and "config_user.txt" files somewhere outside your project folder (not necessary, but typing those up again would be slightly repetitive so we're going to avoid that, even though it would obviously be very easy). Now delete your entire "my_hack" folder (only do this if you actually pushed your repository or it'll be gone for real!).
We will now simply restore our entire project, including our git repository, like it's nothing. Copy your repository's URL from GitHub from the box that pops up when you click on "Code":
Now open a command prompt (or PowerShell) in your "example_project" folder and type "git clone <url>" (replace <url> with the URL you just copied) in my case:
Rename the cloned folder to what it was called before, in this case, since I called the remote repository "example_project" but the folder was called "my_hack", I had to rename it from "example_project" to "my_hack". Open the newly cloned folder and you should see most files we had earlier with the exception of the ROMs and the config files, which we specifically told git to ignore.
Take the config files we backed up earlier and move them back to their original locations (or recreate them, if you feel like it, it's no harder than first setting them up).
Afterwards, we need a clean ROM again, so create a "clean.smc" in "my_hack" again.
Now just run Lunar Helper's "B - Build" option again like earlier and you'll have your entire project, not just your ROM, back in the exact state it was in before we "lost" it, pretty sick, right?
This is the basic idea behind this workflow. Make changes, commit them and push them at some point. Not that hard really, is it?
Advanced workflow concepts
GitHub issues
Many repository hosting sites have a very handy feature called "issues". They're kind of similar to threads in forums, you can use issues to keep track of "issues" in your project (bugs, features you want to implement, etc.). Let's create one for our example project:
We can do a bunch of things with issues, you can assign them to people, you can label them, make them part of a milestone, comment on them, etc., very useful! Note also that every issue is assigned a number, you can see it next to its title, in our case it's #1, since it's our very first. Once the bug/feature/whatever that's associated with an issue is taken care of (i.e. there is a commit or a set of commits that addressed it) an issue is typically closed, a button for closing an issue can be seen on the comment box when you have an issue open.
Let's actually address our issue now, 105 needs to be diamond-dirtified and quick!
That's better, beautiful even, some might say.
Save your changes, "git status" will once again show us that level 105 is modified. Let's commit our changes, but let's learn a really cool feature we can use to close an issue via commit message!
We always used "git commit -m <commit message here>" before, let's use "git commit" alone this time, so git will let us create a commit message in a text editor instead, by default this text editor is vim, which is not very nice to work with (at least imo), so see
this for how to make git use common text editors instead, if you want to.
I have it set up to open Visual Studio Code, so after "git add ." and "git commit":
git pops up Visual Studio Code for me:
and I can type my commit message (it's convention to type a subject with at most 50 characters on one line and then a body explaining in more detail below):
Note the "Closes #1" at the end, this is the secret sauce that will close the issue we made on GitHub once we push our changes.
After I close the commit text file in Visual Studio Code, git finishes the commit and adds it to our history.
If we do "git status", we can see that our branch is "ahead of origin/main by 1 commit". Meaning we once again have changes we can push to our repository, let's do a "git push" again to do so:
and now let's check on our issue:
It's closed as expected, pretty cool, no? This workflow of opening issues and closing them like this feels very organized and intuitive for many, I've been really enjoying using it for hacking personally!
Note that you can also use phrases like "close #1", "closed #1", "fixes #1", "fix #1", "resolve #1", "resolves #1" instead, for a full list see
here.
Branches and conflicts
As I explained briefly at the start, branches are essentially sequences of commits in their own little separate location. Believe it or not, you've already used a branch, the main branch! You can make a branch for any reason, but usually a branch serves some specific purpose. For example, you may have a branch where you work on a specific feature and only once you're confident it's fully implemented and working you merge it back into your main branch. Likewise, some projects use their main branch only when they release their work to the public, using a development branch in the meantime and only merging it back into the main branch once it's ready to be released. In team projects, sometimes people prefer having their own branch to work on stuff. There are a lot more uses for branches, these are just some common ones I've encountered myself!
There are currently some pitfalls when using branches in our context, but that doesn't necessarily mean you can't use them, you just have to be aware of these pitfalls and know how to deal with them!
Let's create a branch so we can see what can go wrong (and right!). Let's say it's a "rework_level_106" branch, we can just create it with "git branch rework_level_106" and then check it out with "git checkout rework_level_106" or we can also do "git checkout -b rework_level_106", which does both in one command:
Let's also create a new issue for this on GitHub (you can have cool little checklists like this in issues too!):
As you can see, I already got the first step covered:
I'll save the level and commit my changes like usual:
Now let's add the diamond dirt, I'll just create a map16 tile for it and save it like this:
And the sprinkle a bit of it across the level:
And commit my changes again (note that "Other/all.map16" was modified because we added the diamond dirt tile to the map16!):
All the tasks in our little list are done! We could now try to merge this branch back into our main branch with "git checkout main" and then "git merge rework_level_106", but let's take advantage of another GitHub feature: pull requests!
First, we will have to push our branch, you cannot just "git push" on branches you create locally, because they do not yet exist in the remote repository, but luckily if we try to, git will tell us exactly how to do what we wanted to do in the first place:
After we take this advice:
We're even told about how to create a pull request, how nice!
Note that you will only have to do this "--set-upstream" stuff once for a new branch, afterwards you can just do "git push", since the branch has now been set up in the remote repository.
If we go to our GitHub page, it will also suggest that our branch had recent pushes and gives us the option to create a pull request:
But what is a pull request actually? It's really not that special, basically, instead of just merging our changes into the main branch immediately (i.e. copying our commits from the rework_level_106 into it), we first submit a
request to do so. This is useful if you want someone else to verify that whatever you were working on actually works before merging it into a branch you want to ensure stays bug-free, for example. You can also set your project up so that only certain people can approve pull requests and nobody can just go ahead and merge stuff on their own, there are a lot of options when it comes controlling what gets merged where!
Let's open such a pull request by clicking on "Compare & pull request". We can write a little comment explaining what's contained in the pull request if we want to, I'll just write "Closes #2" so that when our pull request is approved and merged, it will close the issue we created earlier (just like we did with the commit message earlier!):
Note also the "Able to merge." message. This is a great sign, it means our branches have no conflicts and can be merged without us having to do anything special!
Click on "Create pull request" and you will see a page created for it on GitHub. Here you can request reviews, add labels, etc. In this case, we could simply click "Merge pull request" and have our changes be added to the main branch and it would be totally fine, but I want to show you something instead.
Let's go back and do a "git checkout main", we can totally work on two different branches if we want to! First thing you should do after checking out a new branch is running Lunar Helper's build option, this will make your ROM actually match the state of the project in the branch you newly checked out. If you don't do this, things could get a little wonky [6]! You should also reload the ROM in Lunar Magic if you had it open during the build (just to make it refresh map16 and such).
Now, let's make a change in this branch. Go back to our trusty level 105 and remove the first Yoshi coin, it's kinda hard to reach now after all:
Save the level and commit your changes again, then push them:
Go back to your pull request page on GitHub, you'll see that it can still be merged automatically, even though we made changes in both branches. Nice, right?
Well, let's now create a cool new (totally unique) block:
And add it to level 105:
We were even smart about it and created the new map16 tile one tile to the right of the one in the other branch, I'm sure that will totally make it so there's no conflict (famous last words).
Let's commit and push these changes:
Refresh your pull request page again, uh oh:
Why did this happen? Well, unfortunately "all.map16" is a binary file, the way git handles these is simply that it keeps a full copy of each new version it encounters in the repository so that it can later be restored. Since we made changes to this file in both our branches, git has two conflicting copies that do not share a history, i.e. one is not a descendant of the other, they are on separate paths in our repository's "roadmap" that are about to collide.
For regular text files (and code), resolving conflicts is simple, git shows you which lines differ between the files and tells you to figure out which ones to keep or (re)move. But for binary files, git doesn't know what it is that changed, it doesn't understand the file format!
When this happens we have to really get our hands dirty and handle the merge even more manually than if two text files collided. We essentially have to merge the two different map16 versions by hand. It's "lucky" for me that I documented which map16 tile I changed in our most recent commit, otherwise, in an actual project, I may have to sift through dozens of pages and compare them by eye, not great!
GitHub also will no longer give us a nice button to merge automatically (how could it, there are conflicts!) so we will have to use the command prompt (fine by me).
Let's try to merge our branch, we're already in the main branch so it's just "git merge rework_level_106":
As we expected, git cannot merge our binary files for us, "git status" will give us:
Since we need to resolve this by hand, let's stop our merge for now with "git merge --abort", now let's switch back to our other branch with "git checkout rework_level_106". Remember, first thing after doing a checkout, always re-build and reload your ROM!
Opening our ROM again, we will be greeted by our good old tiny diamond dirt in the map16 editor, we need this fella for our merge, so we need to somehow grab a hold of him. We could try copying it with Ctrl+V, but I'm not sure if that actually works, so I will export the tile in a map16 file (do
not export
all map16 pages, that is exactly the problem we're addressing!) so I can later import it manually during our merge. I'll just put the file somewhere outside our repository so git won't try to track it.
Now switch back to the main branch with "git checkout main", re-build your ROM and try "git merge rework_level_106" again. Obviously you'll get the same error, after all, we didn't change anything in either branch! But we do now have the tile(s) we need from the branch we're trying to merge and we're currently merging, so let's now import the map16 file we saved of our beautiful diamond dirt:
Look at the two, so cute, what a power couple! Let's save map16 and then "git add ." to stage our resulting "all.map16" file (git tells you to stage conflicting files to mark the conflict as resolved):
Git is giving us the go ahead, "All conflicts fixed but you are still merging. (use "git commit" to conclude merge)", let's do so (your text editor will pop up to let you change the commit message, the default of "Merge branch 'rework_level_106'" is totally fine for our purposes so you can just leave it unchanged):
"git status" will now tell us that our branch is ahead of 'origin/main' by 3 commits, two of those are our commits from the rework_level_106 branch, the third is the merge commit we just finished, let's push:
Look at your pull request page:
Yay, we did it, you can even see our issue was closed (on the right!).
As GitHub tells you, you can now delete the branch if you want, since all the commits are integrated into the main branch and will not be lost.
As you can see, resolving binary file merge conflicts can be a real pain in the ass, which is why you'll generally want to avoid them as much as possible. Given that we're stuck with "all.map16" and its binary format for now, avoiding conflicts like this is relatively difficult if you're using multiple branches or working with other people (the same conflict could have happened if someone had changed "all.map16" while we were working on our own version of it in the same branch and they pushed it to GitHub before us!).
Note also that conflicts like this could happen the same way with shared palettes, levels, global data, even title screen moves, and they would be just as annoying to resolve, because those are also all stored in a binary format. You'd have to use a similar approach like we did, somehow get your hands on the conflicting versions from each branch and merge them by hand (i.e. getting conflicting MWL files and merging them side-by-side in two Lunar Magic windows, opening two OWs side-by-side and merging them, etc.). Sadly, that's currently the state of things, but we can mitigate it somewhat by writing clear and descriptive commit message (i.e. me documenting that I only altered tile 201 in our main branch's map16 commit so I could find that tile manually later, just in case).
That's pretty much it for branches and conflicts. Personally, I stick to using branches for highly self-contained parts of hacks, so that I can avoid conflicts as much as possible. If I can tell there might be a conflict at some point (map16 is often a culprit because it changes often and is not split up at all), I write very detailed commit messages, so I can find the actual tiles I need to merge easily later.
Considerations and common problems
Tools in or outside of repository?
In our example project we kept our tools outside of our repository for simplicity. The reason it is simpler this way is that these tools are very noise and some of them produce a seemingly endless amount of irrelevant (for our purposes) files that will clog up your commit history with completely useless changes if you don't ignore all of them via your .gitignore file.
So if you want to keep your tools in your repository, make sure to build your ROM with Lunar Helper and then take note of all the irrelevant files git spots and tries to track. Make sure to add any that are just build artifacts and not necessary to be kept as backup to your .gitignore or you will waste time and space on them.
Another, but negligible, downside is that your repository will have to contain these tools, taking up a slight amount of space (not a big deal at all, git is good at compression).
The upside of keeping tools in your repo is pretty simple, you can ensure when you check out your project at a certain point, it will be built with
exactly the tools and versions of them you were using at that point.
Make sure to follow the general idea I mentioned when configuring Lunar Helper of keeping paths referring to tools that are part of the project in "config_project.txt" and paths to tools that are outside of the project folder in "config_user.txt" to keep your project portable!
Many/few large/small commits?
In our case, in theory, fewer commits with more changes would save space, since git has to store our binary files as full copies instead of just storing the difference between them and the less commits we make with them, the less copies it will have to store. However, we really don't need to worry that much about space, the binary files actually take up surprisingly little space in the repository (thanks git compression!), and being able to restore our work at important points if we need to is much more important, so in my opinion you should be striving to make your commits as small and commit as often as you find sensible without worrying about space too much!
I tried to push my commit, but someone pushed to the same branch in the meantime, what can I do?
This often occurs, you're working with other people and they push changes in the same branch as you before you pushed yours, usually you'd just "git pull" to add their changes to your local repository, but you have work committed already, what now? Luckily there is "git pull --rebase", this command will first pull the changes and then add your commits after them, very convenient! Note that this will only work automatically if they did not change the same files as you did. In that case, you'd have a merge conflict to resolve (good luck!).
Can I make my repository public?
DISCLAIMER: I'm not a lawyer!
If there are no ROMs in there, you might think it would be okay to make your repository public. Maybe it would be, but in theory it could still be illegal, but for different reasons. If you have
any sort of resource (text, code, graphics, tools, etc.) in your repository that you did not create (pretty much a 100% chance), you in theory have to respect that resource's license (if it even has one!) or re-distributing it (which is what you're doing by making your repository public) could still be technically illegal! Is anybody going to care? I don't know, I wish nobody did and we could all just have a good time, but unfortunately I cannot guarantee that that's the case. I would thus recommend keeping your repository private and only inviting collaborators to it, if you want to work in a team. Might still be illegal (I do not know), but at least almost nobody will know about it or be able to prove it!
Final thoughts
If you've made it this far, I do not believe you read the whole thing (if you actually did, I'm very impressed). This turned out a lot longer and more detailed than I thought it would, I hope it's actually useful. I did not want to cut too much, since there are a lot of specifics with this system.
One thing I will mention is that git to me is very "learn as you go". I still encounter situations I don't entirely know how to handle sometimes, but luckily so many people use git that you can almost always find advice on your specific situation online, no matter what's going on.
That's pretty much it from me, thanks again for the kind words everyone, I'm glad people actually enjoy this release, thanks again to everyone involved (especially Maddy, of course)! If you think anything is missing from this guide or just plain wrong, please let me know and I'll look into it!
See you around.
Footnotes
[1]
Here is a guide on writing "good" commit messages if that is something you're interested in
[2] Historically and problematically also sometimes still called the "master" branch, thankfully most important tools will either use "main" by default now or at least allow you to switch the default name from "master" to "main" easily
[3]
Here is a short guide on how to create your own "command" that will let you log changes in a much prettier format
[4]
https://www.git-tower.com/learn/git/faq/detached-head-when-checkout-commit/
[5]
https://dev.to/biancapower/how-to-change-your-default-text-editor-for-git-and-avoid-vim-fk0
[6] Very advanced concept: see
this for "Git Hooks" which let you run programs whenever you perform an action in git, you could theoretically create a post-checkout hook which could run Lunar Helper's build option for you!
