Banner
Views: 784,946,748
Time:
16 users online: Atomic-Citrus,  Dan, Darolac,  Eevee, faloppa, Fermín Acosta Jr.,  Giftshaven,  Lazy,  musicalman, N450,  Sayuri, SquidMan_Raps,  Tahixham, TCgamerboy2002,  Teyla, Yung Gotenks - Guests: 48 - Bots: 63 Users: 41,044 (1,547 active)
Latest: CoffeeMugs
Tip: If you edit level 0, make sure you also edit level 100. Both of them are used for the bonus game.Not logged in.
JPS: web-based IPS patcher - status: released (also:Alcaro and ZPedro have a conversation with teachable moments about software)
Forum Index - SMW Hacking - Resource & Tool Releases - JPS: web-based IPS patcher - status: released (also:Alcaro and ZPedro have a conversation with teachable moments about software)
Pages: « 1 2 »
JPS (stands for Javascript Patching System) is a web app that applies patches, currently IPS patches. It differs from previous patchers in that it is web based: you use it the same way regardless of your platform, and you don't so much install it as just navigate to an URL:

http://wanderingcoder.net/projects/JPS/ (EDIT: replaced beta URL with public release URL)

The benefits are obvious: someone new to using ROM hacks no longer needs to be told to also download a patcher (typically Lunar IPS)… with additional instructions in case he may be using a non-Windows platform, and with the potential problem of the program no longer running on current systems, especially in the case of the Mac: this happened for PowerPC-compiled patchers, it could happen again. Just point him at JPS, and he will be good to go (ideally, anyway; there are currently limitations, as we'll see). Furthermore, being web-based means you are, by default, using the latest version without any additional effort. And you do not lose the benefit of being able to use the software while offline: after having been loaded once, your browser will be able to reload it while cut off from the Net (of course, you should put it in a bookmark to make sure you can recall the URL without needing to go through this page); in fact, even if I were to let my domain expire and no one could access the URL anymore, your browser would keep the latest version it obtained and allow you to keep using JPS.

All this is made possible by recent advances in browsers, particularly in the domain of file manipulation and offline use support. So recent, in fact, that unfortunately not all browsers have the necessary support, and I am sad to say that, regardless of my efforts, Safari and Internet Explorer are not supported*. Any reasonably recent version of Firefox or Chrome (both of which update automatically anyway), as well as any version of Opera starting with Opera 15 is supported; Safari is supported starting with Technology Preview 19 or Safari 10.1, and Edge starting with version 13 normally.

Also, while I have subjected JPS to about 100 patches downloaded from romhacking.net to validate its behavior**, and fixed the most egregious issues (one of which crashed Firefox. Yes, taking down all your other Firefox tabs), there remains a few know issues. The most notable one is that, on Firefox, some patches (just one in the ~100 I tested), once applied, cannot be downloaded: that is, clicking the download link does absolutely nothing***. The only other known limitation is truncation support: it is not implemented yet.****

But even with the remaining issues, it is in good enough state that I now declare it in open beta. Use it, prod it, test it, abuse it, send feedback, or not, go ahead. Obviously, new beta versions will be deployed at the same URL and you will automatically get them.

And for the anecdote, yes, this is what I asked IPS test files for in the other topic. I intend to release its source code once it is out of beta.

------

*the reason being that neither browser supports downloading generated files (called blobs) with the download attribute. In the case of Safari, I requested the functionality in http://www.openradar.me/23550189; no answer so far the request was marked as a duplicate of a previous request, so at least they are not rejecting the idea outright.
**including checking the output is identical to what Lunar IPS gives for the same patch
***logged in the Firefox bug tracker at https://bugzilla.mozilla.org/show_bug.cgi?id=1224870
****what is not a limitation, however, is the lack of support for creating patches; creation of IPS patches will never be supported

(cross-posted from the RHDN forums)

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Haven't actually tested it, but I glanced at the source code and it looks like it was probably a nightmare to develop. Good job on your first JS app, though!

I could see this being useful on things like mobile devices, where the other option would be to download an app for patching. This seems a lot easier than that.
idk what to put here
Originally posted by K3fka
Haven't actually tested it, but I glanced at the source code and it looks like it was probably a nightmare to develop. Good job on your first JS app, though!


Thanks! A nightmare? I wouldn't say that, really, because I did enjoy the programming and problem solving of it. Especially not a nightmare when working on it for these last few days provided me with an escape from an actual nightmare. But a pain in the neck and the source of many a late night coding session over the last few weeks, yes; in part because of constraints I put upon myself, in part because of the wonderful ambiguities of the IPS format (overlapping directives are real, people. I have seen them.), and in part because of the immaturity of the features I use (viz. aforementioned Firefox crash; even if my program might not have been entirely responsible here, it was better for me make sure to avoid the situation anyway…).

Quote
I could see this being useful on things like mobile devices, where the other option would be to download an app for patching. This seems a lot easier than that.


Up to now, mobile browsers were unsupported (there is a check for that), but I do need to revisit that decision, at least for Android (AFAIK it is still not possible to download through the browser on iOS anyway).

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Small update (same URL): Chrome running on Android is now supported, and I lifted the check that would make JPS refuse to proceed on it (JPS still refuses to proceed on other mobile browsers). You can now try it out, though it's not necessarily straightforward, because Chrome on Android will not offer the option to load a file from the filesystem by default (only a few sources such as captured images are offered); the workaround is to install a file manager, that will act as a file source as far as Chrome is concerned and allow you to pick files from the filesystem.

(Ironically, the built-in browser of my Android handset does support loading a file from the filesystem, but will not run JPS because it does not have the necessary JS API support; so I've only been able to test Chrome.)

EDIT: forgot to add: if you are loading JPS on a browser on which it was previously loaded, you will likely need to reload again in order for changes to be visible. This is because the version that was downloaded the last time is loaded (so that it loads faster), while the download of the new version is occurring in the background.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
This is pretty cool.

I think an option to load IPS files from URLs should exist, also with the capability to extract them from ZIP folders :P
Originally posted by Nox
This is pretty cool.


Thanks!

Keep suggestions coming, people (for those who may still be hesitating), I can't guarantee that I will get to them, but I do log them all in a feature enhancements and bugs tracker. Here your suggestions may not be obvious (or even possible) to implement because of the same origin policy (not to mention most sites distributing patches have anti-hotlinking protections, which I would run into), but I'll investigate them at some point.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Hello, and happy new year!
Big update (same URL; as usual, reload twice to make sure you are actually running it):

* Support for truncation. I actually deployed that bit in December, but felt it was too small to specifically announce.
* Progress indication. This was made necessary by the slowness of some supported browsers (namely, Google Chrome, even on a current desktop), where even the spinner was insufficient feedback given that particularly complex patches could take more than 30 seconds to apply (and a few minutes on mobile!); in my opinion, even a complex patch ought to only take a fraction of a second to execute, and not even require a spinner for feedback, and yet here I was, implementing a whole progress bar for that purpose… Anyway, it's done.
* Support for cancellation. If a patching operation was started by mistake, you can stop it to avoid having to reload the page (or wait for the operation to complete) in order to be able to start the operation you actually wanted.
* Removed "apply" button. Patching now starts automatically when both a base and patch file are provided; this was made reasonable to provide with the support for cancellation, given that it is easy to start patching by mistake when doing so occurs automatically.

This update was a long time coming not only because of these features, but because I went a few dead-ends as well; most notably I planned to integrate FileSaver.js, and was pretty far in the design, until I realized from the online demos that it would buy me exactly nothing with regard to Safari (it still wouldn't work on it), and not much elsewhere.

For a feature and interface standpoint at least, this is the version I plan to officially release; only fixes for functional issues (inaccessible functionality, wrong output, wrong user feedback, etc.) will happen prior to the release. In other words, we're in final beta testing stage. Thanks in advance for helping test it so that any such issue can be caught prior to the release!

--------------------
Tired of IPS patching with a desktop app? Try JPS!
JPS has now been publicly released at http://wanderingcoder.net/projects/JPS/ (the beta URL will keep working, but the version at this URL should be more stable). Its source code is now available on Bitucket, should you want to tinker with it.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Important update: I released a massive performance improvement of JPS at the usual URL, http://wanderingcoder.net/projects/JPS/. We're talking 10x performance improvement here, people. Now even the most complex patches should apply reasonably fast (in a few seconds at most) in desktop Chrome, and it should be even faster in the other desktop browsers. Performance improved similarly for Chrome for Android.

To ensure you are using the latest version, load the page, wait a bit, then reload it, until you see the donation links (new to this version).

I put the version without the cache at http://wanderingcoder.net/projects/JPS-dev/nocache-demonstrator/, if you want too see for yourself how far JPS has come.

I pushed the code to BitBucket, you can check it out if you're interested. https://bitbucket.org/Pierre_Lebeaupin/jps

(also, if you missed it I previously added support for Safari)

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Now you'd only need to add support for BPS, since we switched to it as our standard patching format.

--------------------
Anime statistic on MyAnimeList:
400 animes completed ✓
6000 episodes completed ✓
100 Days completed ✓
... what even am I doing with my life?
Originally posted by JackTheSpades
Now you'd only need to add support for BPS, since we switched to it as our standard patching format.


It's definitely in the roadmap; but I'm taking things one at a time and cutting my teeth on UPS before tackling BPS.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
In case it helps, I've got a JS BPS patcher here, feel free to steal it.

--------------------
<blm> zsnes users are the flatearthers of emulation
Originally posted by Alcaro
In case it helps, I've got a JS BPS patcher here, feel free to steal it.


Very interesting, always glad to see other people experimenting in that space, let me take a look at it…

A few minutes pass. Then, you hear audible gasps and several expletives. Then more silence.

So, I am surprised (and a little bit jealous) that your approach works as well as it does! Indeed, for JPS I considered structuring the IPS patching code so that performing a single function call would apply the whole patch, but decided against it. The issue I foresaw was that the interface would become unresponsive while patching was taking place, which is not acceptable, even for just one second (even if the browser offloads JS to another thread, that just means other tabs remain responsive, yours is still stuck). But it turns out I could have, as processing is near-instant in your approach even with a particularly complex patch (about 10000 directives)!

Instead, easily half the complexity of the IPS engine in JPS, not to mention the entirety of the cache code I had to add to get acceptable performance, is a direct consequence of it working asynchronously. The only redeeming aspect of my approach is that it should scale better, but that's useless for IPS, which has a 16MB limitation (and likely useless for BPS: isn't the encoder unusably slow for larger file sizes?).

Anyway, thanks for the pointer. I don't know if I will be able to use it as is, but it taught me things anyways.

Side note: also, I thought I read somewhere that at least some browsers did not allow simulated clicks to trigger downloads, but I must have been mistaken, as I tried CPS with a few browsers, and none of them complained.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Originally posted by ZPedro
The issue I foresaw was that the interface would become unresponsive while patching was taking place, which is not acceptable, even for just one second

I never felt that was an issue. It's not like you're running a game in the same tab, or even at the same time. Finish time is more important, and I feel your approach likely adds a lot of overhead.

Either way, I believe there is a way to get it responsive, while keeping the simplicity of the straightforward approach: WebWorkers. Didn't look into it very closely, but it should work.

Quote
isn't the [BPS] encoder unusably slow for larger file sizes?

I've made BPS patches for 560MB files. It took about 90 seconds. I'd count that as usable.

Maybe I should take a look at how exactly xdelta manages to be so fast. I suspect it somehow only looks at parts of the input file.

--------------------
<blm> zsnes users are the flatearthers of emulation
Originally posted by Alcaro
I never felt that was an issue. It's not like you're running a game in the same tab, or even at the same time. Finish time is more important, and I feel your approach likely adds a lot of overhead.

Either way, I believe there is a way to get it responsive, while keeping the simplicity of the straightforward approach: WebWorkers. Didn't look into it very closely, but it should work.


It does cause overhead, though (I just profiled it, to be sure) less than my insistence on generating the output file by concatenating blobs rather than performing writes in an array.

I am still on the fence between using WebWorkers and using ES7 async functions (which can be used on all browsers by transpiling to ES5-level JS) in the future, because user interface responsiveness matters to me (why, yes, I am a mobile developer, why do you ask?), so I want to keep experimenting in that space. Doing it the way I did in JPS was in large part as an experiment to see if it could be done as manually asynchronous code. The answer is: yes, but there is no way I'm doing it that way for anything more complicated.

Quote
I've made BPS patches for 560MB files. It took about 90 seconds. I'd count that as usable.


Oooooh, I missed that happening (I researched the status of the various patch formats in early 2015). So BPS patches for huge files are in the wild for sure. And sorry I wasn't clearer, I didn't mean to diss Floating IPS, because I meant to refer to the reference encoder, which by byuu's own admission has quadratic complexity.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Originally posted by ZPedro
It does cause overhead, though (I just profiled it, to be sure) less than my insistence on generating the output file by concatenating blobs rather than performing writes in an array.

Yeah, that's another part that confused me in JPS. Appending blobs works fine if there's only a few, but with patchers, you end up with 20000 blobs, most of which are five bytes.

And it breaks on IPSes that are shuffled in silly ways for no reason, or even writing a non-RLE record on top of a RLE one (a perfectly legitimate optimization, though no known patcher does it). Such patches aren't common, but as evidenced by that guy noticing my warning, they exist.

But on the other hand, appending blobs has the advantage that you don't need to know the output size beforehand, which you don't because IPS is a shitty format. But back on the first hand again, IPS is a shitty format, so you can just allocate 16MB+64KB and not worry about it.

Lunar IPS uses a physical file as output, not a memory blob, so it lets the OS deal with resizing. Floating IPS does two passes to find the size, which is, quite honestly, just stupid.

Quote
(which can be used on all browsers by transpiling to ES5-level JS)

Things do get a lot easier if you ignore old browsers, like I did. But yeah, some don't have that luxury...

Quote
So BPS patches for huge files are in the wild for sure.

There are 48MB BPSes right on this website. No doubt even bigger ones exist somewhere.

Quote
And sorry I wasn't clearer, I didn't mean to diss Floating IPS

I was unclear as well; I mean I don't know if 90 seconds is reasonable, not a sarcastic "of course it's reasonable, so let's word it as a question to make me feel smart". xdelta's five seconds is reasonable (I still want to know its secrets), Flips121's nine hours is not, where's the cutoff?

Quote
I meant to refer to the reference encoder, which by byuu's own admission has quadratic complexity.

Yeah, that one sucks. In fact, byuu pretty much surrendered the field of BPS patching to me.

--------------------
<blm> zsnes users are the flatearthers of emulation
Originally posted by Alcaro
stuff!


Ah, thanks a lot for the test case, I've been looking for stuff like this. Let's try it… Yup, JPS applies it all fine already! I did in fact foresee that possibility from the IPS "spec", and took it into account, though at the time more as a sense of anal-retentive completeness than anything else.

How? If you look at line 1141 in JPS, I specifically detect that a record position doesn't follow the previous one, and create an intermediate blob from the blobs appended so far, and re-slice that to be able to support the misplaced directive, pretty much emulating the behavior of fseek+fwrite. If you look at the JPS repo, you'll see I have a number of test cases of about every IPS oddity I could think of, and a test page that runs the engine against these test inputs.

And yeah, don't get me started on the patchers who break a perfectly good record just because one byte happens to be identical in the original and the hack, resulting in a bunch of small records which take up more space and take longer to apply…

And yes, while I got a lot of respect for what it historically enabled, people need to stop using IPS already. Support for decoding it in JPS is only for historical preservation.

Re: async, even current browsers other than Chrome don't support it, though I just checked and was pleasantly surprised, as it seems to be just around the corner: literally all next versions of the main browsers will support it.

Re: reasonable, heh. This was from the viewpoint of me as someone intending to develop a decoder, and who wonders "OK, the format itself has no theoretical size limitation, but there is no need for me to support up to the theoretical limits, what is about the maximum size of patches and hacks I can expect to encounter in the wild?" So reasonable in this context is "Less than the time the most patient hack developer is willing to wait for the result of his encoder". I'd peg it at 24 hours. So if, for instance, there is at least one BPS encoder than can generate patches for hacks of 1GB size, and can do so in less than a day, then I need to architect my BPS decoder so that it can support these patches without needing to rearchitect it.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Originally posted by ZPedro
How? If you look at line 1141 in JPS, I specifically detect that a record position doesn't follow the previous one, and create an intermediate blob from the blobs appended so far, and re-slice that to be able to support the misplaced directive, pretty much emulating the behavior of fseek+fwrite.

That sounds about fifty kinds of overcomplicated. Just go for a byte array, it's a lot easier for everyone and works on everything except IE9.

Quote
I have a number of test cases of about every IPS oddity I could think of, and a test page that runs the engine against these test inputs.

Neat, I'll check what Flips does with them.

...It rejects EOF_in_middle_patch, missing_EOF_patch and not_truncation completely. Judging by the filenames, that's the correct behavior, though judging by the existence of _expected files, you're more permissive than me.
The rest match your expected output. directives_out_of_sequence and overlapping_out_of_sequence_directives are reported as malformed, which also matches the names.

These testcases seem to be missing:
- Patch doesn't start with the bytes PATCH; testing error handlers is important
- Patch contains a truncation instruction, but the file is smaller than that. Does this expand the file? Should it?
- Patch contains truncation instruction, but has written bytes after this truncation. I doubt anything would ever hit this one, but why not?

(and why are some of the files executable?)

Quote
And yeah, don't get me started on the patchers who break a perfectly good record just because one byte happens to be identical in the original and the hack, resulting in a bunch of small records which take up more space and take longer to apply…

Well, it is easier to implement that way. I guess the first patchers were intended for changes so small the patch size and application time was neglible.

Fun[citation needed] fact: The UPS format represents writes as 'XOR the original file from this point with this 00-terminated byte sequence', so it has to split records on every unchanged byte. At least that only wastes one byte, rather than the 5 an IPS record requires.

Quote
So reasonable in this context is "Less than the time the most patient hack developer is willing to wait for the result of his encoder". I'd peg it at 24 hours. So if, for instance, there is at least one BPS encoder than can generate patches for hacks of 1GB size, and can do so in less than a day, then I need to architect my BPS decoder so that it can support these patches without needing to rearchitect it.

Even Flips 1.21, with its quadratic complexity, can do a 560MB file in """only""" nine hours. Give it fifteen more hours and it'd go up to 900MB.

Flips 1.30 takes a severe hit if you go above 2GB (it uses 32bit math if it can, to save RAM and memory bandwidth, and the 64bit path is a LOT less optimized), but 24 hours is huge. I'd guess 30GB. (You have 540GB RAM, right?)

But the native JS bitshift operator is limited to 2GB, so I vote use that as limit. You could emulate it with multiplications and get to 2^53 bytes, but that's a fair bit more effort than it's worth.

--------------------
<blm> zsnes users are the flatearthers of emulation
Originally posted by Alcaro

That sounds about fifty kinds of overcomplicated. Just go for a byte array, it's a lot easier for everyone and works on everything except IE9.


Well, it's not that complicated; at any rate, I wanted to see if it was possible to do it that way even for formats that implicitly assume fseek()+fwrite() (or equivalent), and it is.

I may switch to a big byte array for IPS, but I will need blob concatenation for the other formats, for scalability: to generate client files, you have to create a Blob for the whole file and make the client "download" that, it's the only way (and there is no alternative in sight, e.g. FileWriter and the filesystem API are deader than dead). But with Blob concatenation, I can do so in a way that the whole file need not be brought in browser memory, which will be useful to support decoding the at least theoretically possible 2GB hacks. Instead, during the "download" operation the browser fetches the parts corresponding to a File object slice directly from the source files, and browser memory is only needed for the data structures and the new parts (e.g. RLE). Blob concatenation is the solution for larger-than-memory file manipulation in the browser.

Plus, with BPS and VCDiff, this will fit naturally as those at least are defined in terms of instructions telling how to build the output file block by block.

I'm indeed probably going to hit JS limitations at some point, though in the worst case I can always multiply my values by 256 instead of shifting, which I see no reason why it shouldn't work, as long as my values remain within the range of integers exactly representable by doubles. And I am safe from hitting the 2^53 limit (who would create a 8 petabyte hack?!)… for now.

P.S.: yes, I need better testing of error handlers. Some of these test files have the executable flag only as an artifact side effect of them having been generated by a different hex editor, if I recall correctly. Haven't bothered to clean that yet.

--------------------
Tired of IPS patching with a desktop app? Try JPS!
Originally posted by ZPedro
Well, it's not that complicated; at any rate, I wanted to see if it was possible to do it that way even for formats that implicitly assume fseek()+fwrite() (or equivalent), and it is.

Figuring out whether something is doable, feasible and/or convenient is a valid reason to do weird stuff. I once wrote a GUI program in C, for that reason. (Verdict: 500 kinds of painful, but perfectly doable. I switched to C++.)

Quote
but I will need blob concatenation for the other formats

Yes, you can concatenate blobs for BPS, they just copy stuff around from the three files (in, out, patch) - but only if the BPS only uses a few large records.

The 48MB BPS I found is 1.5 million records, an average record size of 32 bytes. I think the memory overhead for a slice is somewhere around 64 bytes, so that'd cost more memory than it saves.

But you could use a hybrid approach: write to byte arrays until you find a >192 byte record, then append the array and slice the file. That'd give about 15MB of stuff in the byte arrays and about 8000 slices. (Feel free to replace the number 192; I assume a slice or byte array costs 64 bytes each, so two slices and one array equals 192, but that probably varies per browser. Maybe you can save the extra array somehow, not sure how exactly slices work.)

Quote
and the new parts (e.g. RLE)

New parts can be implemented as 'slice the patch file'.

But RLE is "copy 100 bytes from the output file, starting at the output position minus 1". Doubt you can slice that.

Quote
Plus, with BPS and VCDiff, this will fit naturally as those at least are defined in terms of instructions telling how to build the output file block by block.

As is UPS, and probably every other format, with possible exception of the human readable Unix diff/patch.

Quote
I'm indeed probably going to hit JS limitations at some point, though in the worst case I can always multiply my values by 256 instead of shifting

Works, yes. Sane, not sure.

But sanity is overrated, anyways. Cthulhu fhtagn!

Most people on desktops will probably download Flips instead, and I'm not sure which 2GB files would be useful on a phone. (Flips 2.0 will mmap the files into virtual memory without using any RAM, which is probably equivalent to that filesystem API you said died.)

But if you want to push the limits, sure, why not? Limits are overrated as well.

Quote
And I am safe from hitting the 2^53 limit (who would create a 8 petabyte hack?!)… for now.

And once anything goes anywhere near that, you can switch to WebAssembly, C++ and uint64_t.

And once anything goes anywhere near THAT, replace the u64s with a u128 type with overloaded math operators.

And nothing will ever go anywhere near 2^128. The universe isn't that big.

--------------------
<blm> zsnes users are the flatearthers of emulation
Pages: « 1 2 »
Forum Index - SMW Hacking - Resource & Tool Releases - JPS: web-based IPS patcher - status: released (also:Alcaro and ZPedro have a conversation with teachable moments about software)

The purpose of this site is not to distribute copyrighted material, but to honor one of our favourite games.

Copyright © 2005 - 2019 - SMW Central
Legal Information - Privacy Policy - Link To Us


Total queries: 7

Menu

Follow Us On

  • YouTube
  • Twitch
  • Twitter

Affiliates

  • Talkhaus
  • SMBX Community
  • GTx0
  • Super Luigi Bros
  • ROMhacking.net
  • MFGG
  • Gaming Reinvented