Big Changes in FTL 1.6.1+

Discuss and distribute tools and methods for modding. Moderator - Grognak
Vhati
Posts: 792
Joined: Thu Oct 25, 2012 12:01 pm

Big Changes in FTL 1.6.1+

Postby Vhati » Sat Nov 25, 2017 12:45 am

Hi everyone. I know I haven't been on the forum in ages, but was I revisiting the SavedGame editor this month for a new release... and today, I got an email from Matthew!

Matthew:
FTL has been very static for awhile, but we're going to be pushing out some big changes on the backend that will break some of your work. I'm concerned about the Slipstream Mod Manager, as the structure of the games data files is changing.
[...]
It'd be good to be able to let the modding community know what's happening and for us to be able to help out making any changes that may be necessary.

lolsob

I'll add replies as I learn more.
Last edited by Vhati on Sun Dec 17, 2017 11:10 am, edited 1 time in total.
User avatar
fdagpigj
Posts: 84
Joined: Sat Apr 25, 2015 3:14 pm

Re: Big FTL changes are coming!

Postby fdagpigj » Sat Nov 25, 2017 11:27 am

Welp, guess I might as well shoehorn in here that I forked SMM and added a few new convenience features related to the advanced SMM tags for modders, even though I know for fact there's still at least one bug left (specifically, the negation operators seem to be order sensitive). I haven't gotten around to trying to fix those (or to add further features such as math operations on numbers) since there hasn't really been any hurry (how could I possibly have gotten the community to start using my fork of SMM?) until yesterday that Matthew said in the FTL Discord server that SMM will break for sure (meaning basically every modded FTL player will be forced to upgrade SMM), and now that you're active again.

I did also find a bug related to panicking in 1.6; specificially, the code below (when placed in events_rebel.xml.append as the sole file in a mod) fails because of the panic even though if you remove the panic, it clearly works:

Code: Select all

<mod:findLike type="event">
    <mod:findLike type="choice">
        <mod:findComposite>
            <mod:par op="OR">
                <mod:findWithChildLike child-type="ship" panic="true">
                    <mod:selector hostile="true"/>
                </mod:findWithChildLike>
            </mod:par>
            <mod-append:stupid>YESMAN</mod-append:stupid>
        </mod:findComposite>
    </mod:findLike>
</mod:findLike>


But yeah, this all is relatively low priority, although I guess until he releases a (testing?) version of the new patch or explains in detail how it loads the data files, we won't really be able to do anything about it anyway, and I think some advanced modders could find these features useful.
Vhati
Posts: 792
Joined: Thu Oct 25, 2012 12:01 pm

Re: Big FTL changes are coming!

Postby Vhati » Sat Nov 25, 2017 12:49 pm

fdagpigj wrote:I did also find a bug related to panicking in 1.6; specificially, the code below (when placed in events_rebel.xml.append as the sole file in a mod) fails because of the panic even though if you remove the panic, it clearly works
If I read that right, you're telling SMM...

For each event, find a choice. For each choice, find children containing a hostile ship. If *any* choice lacks a [child containing a hostile ship], then panic. Working as intended. If you remove the panic, choices lacking desired children will be silently unaffected, and SMM will continue on, altering any other choice's children that DO match on successive iterations of the outer event or choice loops.

That innermost findWithChildLike is running multiple times, independently, and panic quits whenever an instance fails to match.
User avatar
fdagpigj
Posts: 84
Joined: Sat Apr 25, 2015 3:14 pm

Re: Big FTL changes are coming!

Postby fdagpigj » Sat Nov 25, 2017 1:39 pm

Vhati wrote:
fdagpigj wrote:I did also find a bug related to panicking in 1.6; specificially, the code below (when placed in events_rebel.xml.append as the sole file in a mod) fails because of the panic even though if you remove the panic, it clearly works
If I read that right, you're telling SMM...

For each event, find a choice. For each choice, find children containing a hostile ship. If *any* choice lacks a [child containing a hostile ship], then panic. Working as intended. If you remove the panic, choices lacking desired children will be silently unaffected, and SMM will continue on, altering any other choice's children that DO match on successive iterations of the outer event or choice loops.

That innermost findWithChildLike is running multiple times, independently, and panic quits whenever an instance fails to match.

So what would be the "correct" way of achieving the same thing?
Vhati
Posts: 792
Joined: Thu Oct 25, 2012 12:01 pm

Re: Big FTL changes are coming!

Postby Vhati » Sat Nov 25, 2017 2:14 pm

Oh another possible misunderstanding: findComposite delegates its criteria to *two or more* nested find tags, to act on the results of both. A composite/par with only a single nested find doesn't do anything special.

fdagpigj wrote:So what would be the "correct" way of achieving the same thing?
IIRC mod tags act at the scope of their immediate parent. Each find tag further restricts the scope, allowing action on deeper elements, each match processed iteratively. They can't see the entire file at once. An inner tag will not know what's beyond the currently examined event/choice/etc context it inherited. You can drill down, but if you want to know that *all* attempts to drill everywhere in the file had failed, that isn't possible. A particular find instance can only panic when *it* fails.

Edit: I suppose you could do two passes. Make scattered modifications in the first set of nested finds, and insert a dummy tag in the process, wherever they succeed. Then after all that, do a find for all such dummy tags, remove them, and panic if none were found? Unless finds only shallowly scan immediate children, in which case no? I'm rusty.

Edit: Ah, looks like finds only use jdom's shallow getContent(filter). Had I written an arg to switch to finding via a deep getDescendants(filter) call, that trick would've worked.
User avatar
fdagpigj
Posts: 84
Joined: Sat Apr 25, 2015 3:14 pm

Re: Big FTL changes are coming!

Postby fdagpigj » Sat Nov 25, 2017 3:16 pm

Vhati wrote:Oh another possible misunderstanding: findComposite delegates its criteria to *two or more* nested find tags, to act on the results of both. A composite/par with only a single nested find doesn't do anything special.
Yeah, I know findComposite is intended for acting on results that satisfy all or any nested finds, I was just simplifying it for this specific case to better demonstrate the issue.
IIRC mod tags act at the scope of their immediate parent. Each find tag further restricts the scope, allowing action on deeper elements, each match processed iteratively. They can't see the entire file at once. An inner tag will not know what's beyond the currently examined event/choice/etc context it inherited. You can drill down, but if you want to know that *all* attempts to drill everywhere in the file had failed, that isn't possible. A particular find instance can only panic when *it* fails.
Yeah, I know find tags don't find nested elements unless the find tag is nested at the same level as the element to be found. But I didn't realise that the panic will be tested for every time the find runs, I guess this just means there is no easy way to panic if there is no matching element at any level other than the top level, you can only panic if not every single matched parent element contains such a child element, or if no top-level matching parent element is found (could you nest a crapload of findWithChildLikes to test if it has the desired children and only on a match on that, do "real" finds for the children? Hmmm).
Edit: I suppose you could do two passes. Make scattered modifications in the first set of nested finds, and insert a dummy tag in the process, wherever they succeed. Then after all that, do a find for all such dummy tags, remove them, and panic if none were found? Unless finds only shallowly scan immediate children, in which case no? I'm rusty.
I don't think a find ever matches grandchildren, you would need several nested finds for that. Which is actually quite burdenous at times since if you want to match any elements of a type at any level, you have to make a massive number of similar but differently deeply nested finds, but I guess it can also be desirable sometimes - I was actually thinking of maybe adding some kind of feature to have some mod attribute that tells the patcher to look at the children of non-matching elements - maybe even children of matching elements, too. But it might be tricky to program that feature. And yeah, the by far biggest problem with advanced SMM tags is that basically nobody really understands and remembers exactly how they behave, which leads to people avoiding them or struggling for hours just to use them for something simple. But to be honest, I don't really know how to change that.
Vhati
Posts: 792
Joined: Thu Oct 25, 2012 12:01 pm

Re: Big FTL changes are coming!

Postby Vhati » Wed Nov 29, 2017 11:49 am

More details from Matthew.

Matthew wrote:The big change that hurts mods is that resource.dat and data.dat no longer exist, and instead they're packaged together in ftl.dat which is packaged in a new way. We're using the custom engine that the iPad version of FTL runs for PC now, which means new data files. The (limited) documentation for this is here.

http://achurch.org/SIL/current/src/resource/
http://achurch.org/SIL/

I know a lot of XML typos were fixed in this update, but there still might be some. FTL XML data is also wrapped in an <FTL> tag now, which adds a minor complication to generating, editing, and inserting XML files.
Matthew wrote:the new version of FTL is available on Steam on a beta branch. You can read more about it here:

http://www.subsetgames.com/ftl_changelog.html
Vhati
Posts: 792
Joined: Thu Oct 25, 2012 12:01 pm

Re: Big FTL changes are coming!

Postby Vhati » Wed Nov 29, 2017 12:25 pm

Okay... Installed Steam (~870MB), redeemed a key from Humble Bundle, got the beta FTL (200MB).
Installed C compiling stuff: msys/mingw-w64/yasm (920MB). Downloaded SIL source.

Edited SIL to build with mingw (I'm not a C person)...
- Set an environment var: "export CC=gcc".
- "SIL/tools/Makefile" to work with mingw (I defined a MKDIR_TAKES_ONE_ARG macro).
- "SIL/tools/extract-pkg.c" (an #ifdef to switch "mkdir(path, 0777)" to "mkdir(path)")
- I didn't fetch all the SIL tools' library dependencies, so the build fails eventually, but it compiles the one tool I really want.

So now I've got a 130KB Windows binary of "extract-pkg.exe" to peek inside "ftl.dat".

Good news, images are still individual png files!
I worried they might've been packed together into texture atlas collages in SIL's obscure tex format.

I haven't diff'd, but the extracted contents basically look like "data.dat" and "resource.dat" had been extracted to the same place. XML has <FTL> root tags enclosing everything. The "ftl.dat" format itself is different, so I'll need to write a whole new Java (un)packer class to drop into Slipstream.

From SIL's package-pkg.h

Code: Select all

 * The basic format of a PKG file is as follows:
 *   - Package header (struct PKGHeader).
 *   - File index (struct PKGIndexEntry * 1 for each file in the package),
 *     sorted by pathname hash, then by lowercased (y/A-Z/a-z/) pathname
 *     for files with the same pathname hash.
 *   - File pathname buffer, containing the pathname of each file (in
 *     arbitrary order) as a null-terminated string.
 *   - File data.  The start of each file may be freely adjusted to any
 *     desired alignment as long as the file's index entry points to the
 *     correct offset.
So the general structure is analogous to the 1.5.13 dat format.
User avatar
Sleeper Service
Posts: 2305
Joined: Sun Mar 24, 2013 8:49 pm

Re: Big FTL changes are coming!

Postby Sleeper Service » Wed Nov 29, 2017 1:32 pm

Can you gauge if modders will be required to make certain changes on their end to in order to make their mods compatible with the new system? Like will the new <FTL> root tag be handled by the mod manager or will people need to add this? Just asking for the MML sakes, really any changes required would mean that a lot of old mods with no one around to update them would be lost. :(
User avatar
mr_easy_money
Posts: 625
Joined: Fri May 29, 2015 9:05 pm

Re: Big FTL changes are coming!

Postby mr_easy_money » Wed Nov 29, 2017 6:28 pm

Vhati wrote:I haven't diff'd, but the extracted contents basically look like "data.dat" and "resource.dat" had been extracted to the same place. XML has <FTL> root tags enclosing everything.

I don't know much about this unpacking business, but I noticed the names of all the files are no longer at the tops of each file, but are instead all on one line (1298 right now), and they seem to be in the order they're listed in the entire ftl.dat file.
Sleeper Service wrote:Like will the new <FTL> root tag be handled by the mod manager or will people need to add this? Just asking for the MML sakes, really any changes required would mean that a lot of old mods with no one around to update them would be lost. :(

I'm hoping that it will be up to the modder to decide whether or not they want to include the <FTL> root tag. This would make old mods still compatible (which I presume is one of the goals by subset, enable modding to function correctly again), but also let modders put a proper root tag if they want to and not have SMM break because they put one in. I'd like to have a root tag if I can, makes any generated mods much easier to search through, instead having to leave out the tag because I can't put it in.