Lunar Helper & Lunar Monitor

Lunar Helper (originally created by Maddy Thorson, modified by me (underway)) and Lunar Monitor (created by me, logging functionality added by Atari2.0), are two tools that together allow you to automatically extract resources from your ROM while you edit it in Lunar Magic and to then rebuild your ROM from these extracted resources as well as making it easier to use version control systems on these extracted files.
Additionally, Lunar Helper also allows you to insert resources from any tool with the press of a single key as well as allowing you to build your ROM in different ways depending on your goals.

Source code for Lunar Helper is available here (my repo) and here (Maddy's repo).
Source code for Lunar Monitor is available here (the DLL part) and here (injector part).
Both tools use the MIT license in case you're interested in re-distributing or modifying them.

Project template including Lunar Helper and Lunar Monitor
Lunar Helper Only Download
Lunar Monitor Only Download

I submitted a tool (github repo) the other day that can convert Lunar Magic's map16 files to a text format and from that text format back to a map16 file.

I also updated the bundle and Lunar Helper versions linked in the first post so that they can use this new tool to automatically use this new format.

This should drastically cut down on the amount of merge conflicts when working with map16 changes. Basically any changes made should be merged automatically, unless there are two or more changes to exactly the same map16 tile(s). Even in this case, you should be able to easily resolve the problem, since the file format is text based and human readable. All you should need is a text editor (even notepad would do the trick).

This new method takes a bit more space starting out. The size of the output directory that contains all the generated text files is about 1.1 MB for a vanilla ROM, which is about twice the size of a binary map16 file for the same data.

The directory will grow as you use more and more tiles, though it should not exceed ~5 MB, even if all tiles are in use.

Overall, if you're using this format in a git repo, it will actually use less space in the long run. With the binary map16 files git has to store a copy of the whole file whenever it changes. With this text format it only actually stores the individual lines that changed, which is much more space efficient for a repository.

Note that if you are using this in a git repo, you will want to ignore the binary map16 file that Lunar Monitor exports and Lunar Helper imports, since it's not deleted automatically at the moment.

I just made an update to Lunar Monitor that adds an "Export all" button (similar to Lunar Helper's Save function) to Lunar Magic and also shows the most recent log message in a new panel I added to the status bar at the bottom of Lunar Magic's main editor window. This should make it easier to tell when Lunar Monitor is/isn't running and whether exports are working correctly if that's something you were having trouble with in the past.

I followed your C3 guide, and everything works pretty great. I feel like there's something you skimmed over though, which is how to set up PIXI/UberASM/GPS/AddMusicK in a way that works well for a version controlled repository.

These tools are different from flips and asar in that the sprites/blocks/music/uberasm go directly in the folder of the tool itself. In your guide, you recommended putting these tools outside of the version controlled environment, but that kind of presents a problem if you want to track your sprite/block/music/uberasm additions through version control.

I'm at least fairly certain that's why these 4 tools start in config_project.txt as opposed to config_user.txt. Ultimately, I could just move these 4 tools into the main project directory, and version control everything; that's one way to do it. It will bother me a bit that some tools are version controlled and some aren't, but I can deal with it. I'm sure one could also write a .gitignore that ignores anything that's part of the base download for these tools, but I think it's a little weird that anyone downloading the repo would be required to fill in the blanks, so to speak.

I think the ideal scenario is something like how Asar patches are set up in config_project.txt; similarly, you could perhaps have a folder that has everything it needs to send a payload to AddMusicK, etc., while using a user-provided path to these tools that points outside of the repository.

I'm mostly curious if you have a personal preference for managing this, since I'm sure you've done a lot more experimentation than I have (I'm also not super familiar with these tools to begin with, since these tools are newer than the last rom hack I worked on, or at least it's been long enough that I've forgotten a lot of it.)

I actually don't think I recommend keeping them outside the repository in the guide, but I might just be misremembering.

There are a few paragraphs regarding the pros and cons of keeping them outside your repository at the end of the guide.

I decided to keep them outside of the repo for the purposes of the guide, because keeping them inside is a slightly more involved process, as you have to gitignore a bunch of useless files that tools generate that will clog up your version control otherwise.

You do bring up some good additional downsides that I failed to mention though. Theoretically, I think you could hack around this by creating a symlink such that the tools outside the project can sort of "indirectly" work with the structure inside the project, but it's not exactly ideal.

As for my personal preference, I actually now highly recommend keeping as many tools as you can inside your project and version controlling them, ideally including Asar and Lunar Magic. It's much better for reproducibility since you'll always be building a given commit with the exact tools it was originally created with. The space you gain by keeping the tools outside is also so negligible that it's basically irrelevant compared to the potential downside of making old commits much harder to rebuild accurately.

Note that keeping Lunar Magic inside the project structure could be kinda annoying, because if you're working on multiple projects with different Lunar Magic versions it could be quite easy to get mixed up if you have it on your taskbar or something. I would still recommend it if possible, just to make sure each commit includes the Lunar Magic version that was used at the time.

Really, the only case where I might recommend keeping tools outside or gitignoring them would be if you wanted to host a public repo and were scared of getting sued over licensing. Though, given that essentially no resources on this site come with a license anyway, you'd probably have more than the tools to worry about. I believe all sprites, blocks, etc. that don't come with a license are technically not legal to redistribute, even just as part of a repo, though I highly doubt anyone would ever sue you over it.

Hope this helps a bit. It's been a little while since I wrote the guide and I've been working on both tools since then so there's a good chance parts of it are already outdated (merge conflicts are a lot less scary now, for example) and there are probably also things I changed my mind about, so feel free to ask if anything else comes up.

EDIT: I might actually supply a sort of generic .gitignore file and/or a repo skeleton at some point. I assume that's something people might be interested in.

That all makes sense. I guess you didn't really recommend it, but since that's what was done in the guide, I figured that was sort of the expected setup, although I understand that it's meant to be a simplified as well. I do agree with your recommendation, since I think that not version controlling your pixi/etc folders would be a huge mistake, greatly outweighing the negatives of keeping your tools in the repo. From the perspective of someone following the guide, it sort of made it seem like a viable option, but I don't think it's very viable in practice, and I wouldn't recommend anyone follow that approach unless they're not using those tools in the first place.

For the sake of other people who may get the wrong idea like I did, I think it may be worth adding an amendment somewhere, or at least your comment in this thread might answer someone's questions the same way it did mine.

I still think this is a great way to make a hack, and honestly I don't even think I would be interested in revisiting ROM hacking if it weren't for this workflow, so I'm super happy that you've been working on improving it. The human readable map16 is also awesome, and that alone will probably make a huge difference in making collaborations easier.

So, I decided to whip up a project template today. It's basically a fully functional setup right out the gate including Lunar Helper, Lunar Monitor and all the tools they rely on, the only thing you have to do to actually get started with a fresh project is providing a clean ROM and double-clicking on an initialization script to circumvent any potential licensing issues.
You can use it with or without git (some people might be excited about that) and I have included instructions on how to do both.
For any further details, please consult the readme I've included, as I said, getting it set up should be very simple.
Link (briefly had the repo private, it's public again now and should be working fine)

Underway, I'm afraid you're violating Lunar Magic's license by distributing it that way (check the "Legal Notice" in the readme.txt file or help file). It can only be distributed as "the stand-alone original zip file provided by the author, with no files added, removed, or modified."

I believe the way it is handled at this time should be to your liking.

Alright, thanks!

Just pushed updates for Lunar Helper, Lunar Monitor and the project template.
I'm very happy with this update, it adds a new "Quick Build" option to Lunar Helper's menu that will attempt to reuse a previously built ROM during the build process instead of rebuilding from scratch each time. This is really useful for inserting small changes, like if you're working on a single sprite. Lunar Helper will automatically determine which tools/resources should be reinserted by checking whether the folder structure of the tool or the data in the resource file has changed since the last build. There's more logic it uses to determine what to insert, but the basic gist of it is that if you change anything in a tool's folder, Quick Build will reapply that tool. If you think Lunar Helper missed a change, you can still always rebuild your ROM from scratch with the default Build option.
The new Lunar Helper and Lunar Monitor versions together also automatically make Lunar Magic reload the ROM if Lunar Helper rebuilds it while it's open, which is super useful since you no longer have to remember to do it by hand (I've previously lost map16 changes because I forgot to reload so this was a personal grievance of mine).
I also added config variables that let you pass additional command line options to tools during the build process if that's something you need, personally I like being able to tell Lunar Magic not to auto-set the screen amount when importing levels.
I also fixed a hard to reproduce bug that could cause Lunar Magic to freeze after it was injected with Lunar Monitor, I'm still not sure what caused this as I've only ever heard one report of this, but the changes I made fixed it for that person in particular, so I hope it won't crop up for other people.

EDIT: I have now also updated the project template's readme to mention these new features.

Just updated Lunar Monitor and the project template again. It should now be able to communicate to Lunar Helper that it doesn't need to reinsert resources that were saved to the ROM and exported using Lunar Monitor since the last build.

So, I realized today that Lunar Monitor and the text-based map16 format converter depend on the C++ redistributable and if it's not installed, they will just not work. I have since remedied this and pushed updates for all of them to github. In addition, I noticed that the FLIPS executable I included in the project template actually will not work unless a specific GCC DLL is present, so I recompiled it with some additional flags and it should now work just fine no matter what.

I have updated the project template to include these adaptations and verified that everything actually works by setting up a clean windows virtual machine and testing it on that, so if you were having issues previously those may have been solved now.

I will probably submit updates for Lunar Monitor and the map16 format converter to the tools section at some point since having to have the redistributable installed is kind of annoying, even though I assume many people already do anyway.

If by "maintain" you mean updating Lunar Monitor to work with the newest Lunar Magic version, that sort of depends. If changes in future versions are similar to the ones from 3.30 to 3.31 then it shouldn't take me more than a day or two of actual work to support those versions, but if there are changes to many function signatures it may take longer and longer still if fusoya adds mechanisms to discourage DLL injection, which would be a shame.

I will also say, I actually do not recommend using Lunar Magic 3.31 with Lunar Helper at the moment unless you can tolerate longer build times due to having to apply PIXI twice to work around the incompatibility that was introduced between it and Lunar Magic in 3.31. Personally I stick to 3.30 for this reason and it's also the version my project template uses by default (you can switch it out manually of course, don't forget to also switch out Lunar Monitor's DLL for an up-to-date one from its github!)

Just pushed more updates:
- Maddy made some changes to Lunar Helper which I pulled into my version which should result in better error reporting for incorrect config variables. The size of the executable and overall zip archive has also decreased by about half which is pretty sweet.
- I updated my text based map16 format to insert a blank line between each tile's specification which should ensure git won't report merge conflicts unless the same exact tile(s) was/were edited in two different commits being merged. Previously, you could actually get merge conflicts when changing neighboring map16 tiles, because git will report merge conflicts if neighboring lines in text files change for safety. In this case, there is no reason to report a conflict when adjacent tiles are changed, so inserting a blank line is the most pragmatic way I could come up with to get around this problem. You can just update to the new version if you've used previous ones, it's backwards compatible so it should parse any previously generated files just fine.
Both updates are already included in my project template as well.

Just submitted an updated Lunar Helper + Lunar Monitor bundle to the tools section. It's been a while since I first submitted the bundle and the frequency of updates to both tools has decreased drastically as they've become more tested and stable so I figure it may be a good time.

The next step would arguably be .mwl <-> text conversion, however this is going to be a very slow and painful process given that the mwl documentation is outdated, so probably don't expect too many updates outside of potential bug fixes, performance improvements and adjustments in case of new Lunar Magic versions for the time being.

Just kidding I guess, just added a profile feature and command-line options to Lunar Helper. The profile feature lets you easily switch between different sets of config files, meaning you can for example have a Debug and Release profile with different emulators, patches, sprites, blocks, etc. that you can easily switch between depending on what you need at the moment.
The command-line options right now are --help, --quickbuild, --build and --package. Each of them (except --help) should behave exactly the same as if you had picked the corresponding option from Lunar Helper's menu, making it easier to run Lunar Helper builds from a script, a git hook or (potentially) a Lunar Magic toolbar button (not making any promises on that last one yet).
Obviously just a small overview, so if you want the full picture please check the release notes/readme.

Just updated Lunar Monitor, Lunar Helper, my project template and the bundle waiting in the tools section (phew).

Headliners:
- Lunar Monitor should now be able to handle Lunar Magic 3.32
- Lunar Monitor can automatically determine which Lunar Magic version you're using and handle 3.30, 3.31 and 3.32 without you having to manually switch out or choose DLL files anymore
- Lunar Helper & Lunar Monitor have a new safety mechanism that should ensure you can no longer accidentally overwrite unexported data in your ROM by building with Lunar Helper (this used to happen when people forgot to use Lunar Monitor instead of "normal" Lunar Magic sometimes)