Slipstream Mod Manager v1.9.1 (2018-01-07)

[Sleeper Service]

Hey there Vhati

I guess 37983 new blueprints put a little to much strain on the mod manager:

Code: Select all

INFO  ModPatchThread - Installing mod: CE Endless Loot Addon test.ftl
Exception in thread "Thread-4" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2367)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
	at java.lang.StringBuffer.append(StringBuffer.java:237)
	at java.io.StringWriter.write(StringWriter.java:101)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.write(AbstractXMLOutputProcessor.java:342)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.textRaw(AbstractXMLOutputProcessor.java:419)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.printContent(AbstractXMLOutputProcessor.java:932)
	at net.vhati.modmanager.core.SloppyXMLOutputProcessor.printElement(SloppyXMLOutputProcessor.java:113)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.printContent(AbstractXMLOutputProcessor.java:946)
	at net.vhati.modmanager.core.SloppyXMLOutputProcessor.printElement(SloppyXMLOutputProcessor.java:113)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.printContent(AbstractXMLOutputProcessor.java:946)
	at net.vhati.modmanager.core.SloppyXMLOutputProcessor.printElement(SloppyXMLOutputProcessor.java:113)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.printContent(AbstractXMLOutputProcessor.java:946)
	at net.vhati.modmanager.core.SloppyXMLOutputProcessor.printElement(SloppyXMLOutputProcessor.java:113)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.printDocument(AbstractXMLOutputProcessor.java:533)
	at org.jdom2.output.support.AbstractXMLOutputProcessor.process(AbstractXMLOutputProcessor.java:191)
	at org.jdom2.output.XMLOutputter.output(XMLOutputter.java:822)
	at net.vhati.modmanager.core.SloppyXMLOutputProcessor.sloppyPrint(SloppyXMLOutputProcessor.java:165)
	at net.vhati.modmanager.core.ModUtilities.patchXMLFile(ModUtilities.java:228)
	at net.vhati.modmanager.core.ModPatchThread.patch(ModPatchThread.java:236)
	at net.vhati.modmanager.core.ModPatchThread.run(ModPatchThread.java:88)

The mod verifies correctly though. Can this be fixed somehow on my or your end? Is some kind of heavy duty version of the SMM possible for this stuff?

We can always go back to single weapon prefixes, that would reduce the amount of blueprints exponentially. But I still would like to see how this behaves ingame first. Help possible?

/EDIT: I was informed that this is purely related to my java version, sorry to have bothered you.

//EDIT: The usuall fixe would be this apparently: '/home/sleeper/Games/FTL/Slipstream Mod Manager v1.4-Unix/modman.command' -Xmx2068M -Xms1024M

Which did absolutely nothing. I also increased Javas memory usage directly in the preference, but it still produces the same error. Any suggestions? RannI says loading the mod works in windows, so its either related to the Linux version of SMM or to my Java I guess?

[Sleeper Service]

So, I allocated even more memory to Java. Htop says slipstream uses only 1600 of the 3000m it has available when it crashes. I don't understand whats happening, but I'd really like to get this mod to work on my Linux rig to test and release it.

Modfile, in case someone wants to have a look.

[kartoFlane]

It doesn't look like the .command script passes arguments over to the .jar, so you'd have to run SMM via Terminal directly. Also, you need to run the memory command on the JVM, not the program itself -- so even if the script passed the args, it's already disqualified.

Either way, the following command should do it (remember to cd to SMM's directory first):
java -Xmx1024M -jar modman.jar

Alternatively, if you don't want to mess with cd, put the absolute path in there:
java -Xmx1024M -jar '/home/sleeper/Games/FTL/Slipstream Mod Manager v1.4-Unix/modman.jar'

You don't really need to set the initial heap space. 1024M max (~1GB of RAM) should be more than enough, I think.

Trivia: depending on how much RAM you have available, default max is 64MB or 256MB (the former if you have less than 1GB, latter otherwise)

[Sleeper Service]

Nope, still does not work. RannI suggested this is some problem SMM has on 32b system, cause when I try using the windows version of SMM via Wine I still get the same error. But on his 64b Windows it works without problems.

[Vhati]

you really should take a look at your spam settings, Vhati

Thanks for the heads up. I think it's a quirk of the forum that it only notifies for the last thread you reply to, and even then, not reliably. It's definitely not arriving at my end.

I'm getting [the same thing] again. It also [removed my ship mod from the mods list] but it [didn't remove it from the game] please help.
Some reason if I open it from the desktop [it] works, not through [documents]. WEIRD

Could you rephrase... all of that?

I guess 37983 new blueprints put a little to much strain on the mod manager

I doubt there's anything I can do to fix this error, but I may be able to rein in the mod's excesses...

Why were 37983 being added?
Is it necessary to literally add every possile combination of stats in a single patch?
Even if you're worried about reapplying the mod later to continue a saved game, what about seeding the mod-generator with a fixed number?

What's the most you've successfully patched? And also not killed FTL with.

[kartoFlane]

Thanks for the heads up. I think it's a quirk of the forum that it only notifies for the last thread you reply to, and even then, not reliably. It's definitely not arriving at my end.

You're welcome. Though it's weird, 'cause it works fine for me...
That said, I've had an issue with gmail several months ago where it marked the sites I frequently receive notifications from (this board among them) as spam. I had to manually whitelist the ones I wanted to keep...

I suppose it's because the service sees that you consistently only delete the messages, and never reply to them, it guesses that they're spam *shrug*

I guess 37983 new blueprints put a little to much strain on the mod manager

I doubt there's anything I can do to fix this error, given where in the code it complained, but I may be able to reign in the mod's excesses...

I agree though, I don't think much can be done in regards to this. It pretty much comes down to 32-bit memory limitations, which is outside of the manager's influence.

Looking at the stack trace again now, however... The program crashed basically because it tried to create a huge string to contain the contents of the patched file. Perhaps this could be avoided by splitting big documents into smaller chunks that are handled separately one after another?

What's the most you've successfully patched? And also not killed FTL with.

Funny thing, FTL responds pretty well to this, at least on a powerful enough machine. I've successfuly patched this on my PC (64-bit Win7), and there was no noticeable lag, or dramatically longer load times.

edit:
For the record (and context), here's a link to the thread where the idea of randomly generated blueprints originated: http://www.ftlgame.com/forum/viewtopic.php?f=12&t=23292

[rannl]

I guess 37983 new blueprints put a little to much strain on the mod manager

I doubt there's anything I can do to fix this error, given where in the code it complained, but I may be able to reign in the mod's excesses...

I agree though, I don't think much can be done in regards to this. It pretty much comes down to 32-bit memory limitations, which is outside of the manager's influence.

Looking at the stack trace again now, however... The program crashed basically because it tried to create a huge string to contain the contents of the patched file. Perhaps this could be avoided by splitting big documents into smaller chunks that are handled separately one after another?

What's the most you've successfully patched? And also not killed FTL with.

Funny thing, FTL responds pretty well to this, at least on a powerful enough machine. I've successfuly patched this on my PC (64-bit Win7), and there was no noticeable lag, or dramatically longer load times.

Since the append files used are 10-50Mb files, it seems like SMM is rather memory hungry - reaching 1.6 GB of mem usage. Other then standard practice memory reduction solutions, kartoFlane is right: large char buffers should dump/append their data to a file, before getting to big...

[Sleeper Service]

"Procedurally Generated Content - so much stuff that you need a new SMM version for it."

[Vhati]

The program crashed basically because it tried to create a huge string to contain the contents of the patched file. Perhaps this could be avoided by splitting big documents into smaller chunks that are handled separately one after another?

At least, that was the last allocation it attempted, however much memory was already in use by that point.

kartoFlane, would you mind running a profiler against it to see what's definitely eating the most memory before I bother micro-optimizing in the face of exponential usage? That single character buffer of a single text file can't seriously be the culprit.

Had it not crashed then, there are a couple other copies of that buffer that would have been made. I could write a special replacement for the StringWriter object (currently collecting baked XML text) to substitute newlines and encode to bytes in one step. There'd still be one full copy in memory... unless I also rewrite ftldat's add() method. Ftldat currently takes a complete buffer and sends it to disk; it'd have to instead provide a custom stream of its own that could be incrementally flushed (with special cleanup in a close()) - during the XML-printing method.

But all that assumes it's specifically the text at fault, and not for instance, the tree of abstract XML objects it's based on.

[kartoFlane]

Well, that explains it...
Before installing the mod, fresh start...
...and right after the installation was completed, before GC kicked in

Almost 3 million instances of char arrays, taking over 500 MB of memory. Nice.

The total count at the bottom is pretty interesting. From ~18 MB in standby, to over 1GB after installing the mod.
For the record, installing CE barely makes it reach 50MB, though most of CE's size comes from new images / sounds, so that's not particularly surprising.