Slipstream Mod Manager v1.9.1 (2018-01-07)

[Vhati]

Almost 3 million instances of char arrays, taking over 500 MB of memory. Nice.

Can you narrow down the source of the majority/biggest of the arrays?

[kartoFlane]

I'm not really familiar with JProfiler, but I'll try. It seems there are appropriate options... Though I think I may need to modify SMM's code a bit, as it is now GC cleans everything up before the profiler can take a memory snapshot.

edit
Nevermind, found a way to dynamically record where the allocations are occuring. The problem now is, this increases overhead significantly, and the manager started crashing for me, too... The VM hosting the manager has 1024MB of max heap space -- I've tried giving it more, but then the profiler is unable to reserve that much space, despite my having plenty of spare RAM :/

Not sure if that's the level of precision you've been hoping for, but here's what I came up with so far:
http://i.imgur.com/G1rUiYq.png
http://i.imgur.com/ekIQXPm.png
If I'm reading this correctly, that's just 4 allocations of toString()'d StringBuilders accounting for 175MB. Both of these pictures seem to be indicating that most of the memory gets hogged by patchXMLFile()

It appears that in Sleeper's case, the printing of XML elements was just the straw that broke the camel's back.

[Vhati]

I pushed some minor optimizations to git.

Nulled a few vars immediately after I was done with 'em. There's a slim chance that'll serve as a hint to the garbage collector; strings were discarded after being parsed into trees of xml objects; then the trees are discarded after a combined tree is created from them; the combined tree gets baked back into text; then the patch method ends where those local vars would've gone out of scope anyway.

As mentioned earlier, I removed a couple intermediate copies going from:
baking the xml tree to a string, regexing that to create a new string, and encoding that into a buffer;
to
baking the xml tree to a stream, passed through a newline-substituting filter (inspecting individual chars that go by), then passed through an encoding filter into a buffer.

[spacecowboy3]

I can't run the mod manager. :c Im using llnux

From what i can tell is that the manager cannot detect the mods folder. There are no mod options at all.

EDIT: If i try patching an error message says:

"Patching failed:java.io.FileNotFoundException:./back/data.dat.bak(no such file or directory)

Also if i look at the file drop down menu the option "rescan mods/" is greyed out.

[Vhati]

I can't run the mod manager. :c Im using llnux

From what i can tell is that the manager cannot detect the mods folder.

Double-click "modman.command", not the jar.

[kartoFlane]

I have downloaded the updated code from git, compiled it and ran it against the profiler... I haven't noticed any improvement - it's still crashing :/
However, I didn't see any more of those 4 huge StringBuffers -- there were many more smaller arrays instead. I suppose that could be qualified as a minor improvement?

[Vhati]

However, I didn't see any more of those 4 huge StringBuffers -- there were many more smaller arrays instead. I suppose that could be qualified as a minor improvement?

*Shrug* I figured as much.
Nevertheless, I think I took care of some of the smaller arrays with another git push.

I had introduced some unnecessary small-scale buffering, goin' on mid-stream. It's standard practice for streams to be accumulated into intermittent bursts when headed for the filesystem/network, but pointless when the stream's ending in a big buffer anyway. So that's gone.

[Sleeper Service]

Well it looks like the mod manager in the current state will create a bottle neck for our little project. It's your call of course, but I would appreciate an update that fixes this a whole lot (and I'm sure the users that will play the result would as well.)


Of course, we could also decrease the amount of blueprints to make the mod work. But then we would probably have to sacrifice some of the more advance features of the content generation. No more gear that has multiple prefixes applied. (No Radioactive Shredder Missiles and such stuff )

I really can't judge if it the mod manager can be enabled to deal with our blueprint-rich mods, for any users I mean, even the ones that have 32b systems. But I'd like to hear your opinion on this. Can it be done? Would it be worth the hassle? Would you rather recommend we decrease the amount of blueprints?

[Vhati]

Can it be done? Would it be worth the hassle?

Ehhh. Not in a practical way. In order to have both random access to ALL parts of a file (for mod tags) and NOT have the entire file in memory (for gigantic files), it'd have to constantly re-read the file from the beginning to 'rewind' the stream for every little change - which is seriously inefficient and slow. Not to mention being a verry ugly total rewrite of the xml handling code.

I can't accomodate the exponential usage that comes from having every possible combination of every variant.

Would you rather recommend we decrease the amount of blueprints?

Yes. But you CAN still have multiple prefixes: use a seed to randomly omit a portion of the resulting blueprints (or equivalently: decide randomly for each blueprint whether it will be left out).
If a blueprint depends on others, be sure to check that they exist.

You'd have "Radioactive Shredder Missiles" that do 1, 2, or 4 damage, for instance (omitting 3).

[Sleeper Service]

Yes. But you CAN still have multiple prefixes: use a seed to randomly omit a portion of the resulting blueprints (or equivalently: decide randomly for each blueprint whether it will be left out).
If a blueprint depends on others, be sure to check that they exist.

You'd have "Radioactive Shredder Missiles" that do 1, 2, or 4 damage, for instance (omitting 3).

Alright, we'll check that possibility. Thanks for looking into the whole matter.

/Edit: How would it effect performance if we would separate the mod into smaller mod packages? SMM would process these one after each other, right? Could that be a possible workaround? Or does this get problematic if it has to append stuff to a file that is already very big from previously appended packages?