[Tool] ftldat r7

[ZahaianGhost]

Edit: Okay, seems the forum ate my post. Here's what got eaten, an attempt at thorough documentation for this tool and some notes on manual installer compilation issues I've been working to resolve the past few days.

Hey guys, I hope this is alright to post to despite the OP being inactive and the last post being around five months ago, but I've been working my way through this as a little of a project to acquaint myself with how these tools are built, their installers configured, and etc. Unfortunately, I don't have much to show for it yet, as I've mostly been trying to decipher the OP's instructions here and there, as well crash course myself through all of the aforementioned stuff; however, I do think I could at least provide somewhat improved documentation for those interested in using this for whatever reason (different platform/dislike Java/just interested).

That being said, here we go, may have to split into multiple posts for this:

For Windows:

1. Run the installer found here Bas' original link or rehosted here. (You do not need to install Python anymore.)
2. Open up a command prompt (Start -> Run... -> "cmd" -> "OK")
3. Navigate to the folder with the data.dat and resource.dat. E.g. C:\Program Files (x86)\FTL\resources; C:\Program Files (x86)\Steam\steamapps\common\FTL\resources; etc.
4. To unpack, run ftldat unpack data.dat
5. This will create a data.dat-unpacked folder in the resources folder.
6. WARNING: Before packing/repacking with file alterations, make a backup of data.dat. To pack, run ftldat pack data.dat.
7. To show the contents and fingerprints of a .dat, run ftldat info --hashes data.dat.

In case you don't want to be troubled with false-positives from the installer, you may also do the following, in the form of compiling it yourself:

1. Download the repository from Github by clicking the 'Download ZIP' option on the righthand side of the page.
2. Visit the instructions page and try to follow them. Otherwise continue on with my instructions.
3. Download Python 2.7.6, NSIS, PyInstaller, and PyWin32. Make sure to get the appropriate PyWin32 version for your version of Python, in this case, Pywin32-218.win32-py2.7.exe.
4. Once you've all that downloaded, open your terminal or command prompt and navigate to the win-installer folder wherever you downloaded the above repository/zip file. E.g. cd c:\users\download\ftldat-master\win-installer
5. Then type in the following command: c:\python27\python.exe [filepath to downloaded PyInstaller]\PyInstaller-2.1\pyinstaller.py ftldat.spec
   
   What your command line should look like:
   
   What the output will be (click image for higher quality shot, if legibility is an issue):
   
   
6. After this, right click the ftldat NSIS script file and select 'Compile NSIS Script'. Look to the next image if you get mixed up as to where it's located. Hint: it's back in the main win-installer folder.
   
   Which should produce this (click image for higher quality shot, if legibility is an issue):
   
   
7. The ftldat r7 file is the newly compiled installer file. Click it to install, then you should be able to run ftldat from the command line.*

*Ideally. I've gotten to this point but am completely jammed by NSIS script errors. The file produced above does install ftldat to a point, you'll get it in your programs folder and the like, but you won't be able to call it from a command window, which defeats the entire point of it.

I'm still not sure what the core issue at hand here is. I think it has something to do with modifying the environment variables in the installer scripts, but the modifications I've tried so far haven't resolved anything. If anything they've served to bug out the compiled installer further, but I'm still researching the code to see if I can figure it out.

For others interested in fixing it what I've tried so far:
Adding calls to the main ftldat.nsi file for the AddEnvVar and un.RemoveEnvVar functions, with no luck.
Adding an !include of the EnvVarUpdate.nsh to the ftldat.nsi file, still no luck. If memory serves it produces a strstr function call bug. Commenting these (strstr related things in EnvVarUpdate) out puts you back at square one with the aforementioned environment functions not being used.
Commenting out the strstr function in AddToPath completely breaks the installer. Produces an opcode error and generally produces useless clutter folders in the start menu and program files folder.


For Mac/Linux(?):
"On Mac it is a bit easier, because Python is already installed.

Open Terminal and go to the/path/to/ftldat/src. Then run, for instance:

Code: Select all

python main.py info /Applications/FTL.app/Contents/Resources/data.dat

"-Bas.

Wherever you downloaded ftldat to. Navigate to that exact file path via the command 'cd' in the Terminal window. E.g. cd ~/Desktop/ftldat on Mac/Linux (last I checked they share similar Terminal commands); cd c:\users\[your username]\Desktop\ftldat on Windows.

More specifically within the Terminal:

Code: Select all

cd /Applications/FTL.app/Contents/Resources

Once there:

Code: Select all

bla@bla /Applications/FTL.app/Contents/Resources $ python ~/Downloads/ftldat/src/main.py unpack data.dat

Again, with the filepath after python being replaced by wherever you may have the ftldat folder located.

Command listing:

• ftldat [command] -h: For more help.
• ftldat pack -h: For more help.
• add: Adds a single file to the dat file. E.g. ftldat add data.dat my-ship.xml data/my_ship.xml
• append: Appends a file to an existing file in the dat file. E.g. ftldat append data.dat data/blueprints.xml my-extra-blueprints.txt
• replace: Replace a single file in a dat file. E.g. ftldat replace data.dat data/blueprints.xml fully-new-blueprints.xml
• extract: Extract a single file to the dat file. E.g. ftldat extract data.dat data/blueprints.xml out.xml
• remove: Remove a single file from the dat file. E.g. ftldat remove data.dat data/events_mantis.xml
• hashes: Prints an alphabetic list of filename and md5 hash. Useful for integrity checking. E.g. ftldat hashes data.dat
• list: Lists the files in the dat file.
• repack: Efficiently repacks a dat file. E.g. ftldat repack C:\...\FTL\resources.
• pack: Overwrites and loads in the contents of data.dat-unpacked folders all at once. Only use after backing up original .dat files. E.g. ftldat pack "[your filepath to...]\FTL\resources\data.dat" "[your filepath to...]\FTL\resources\data.dat-unpacked"
• unpack: Unloads the content of the specified dat file. E.g. ftldat unpack data.dat removes all .xml and .txt files.

Command parameters:

• --hashes: This will show the MD5 fingerprint of each entry. For use with the info command. E.g. ftldat info --hashes data.dat
• --bytes: This displays sizes in bytes instead of rounding them to KB/KiB. For use with info command.
• --indexsize: For matching index sizes of LTF (for Mac, you would use --indexsize 2027 for data.dat and
• --indexsize 1918 for resource.dat). For use with the pack command. E.g. ftldat pack --indexsize 2027 data.dat

Additional info:
"These commands are all in-place and thus as fast as they can be. One could use ftldat.exe with these commands to create an installer of a patch. It would be as simple as writing a .bat.

If anyone is interested, I could also add support for patches like XDelta or Uniform Diff. If you script in Python, these commands are fairly easy to use by importing the "FTLPack" class."-Bas.

Noted issues:
Some users experience a strange mixup of functionality when attempting to run the unpack command on their dat files, resulting in a listing of its contents instead of extracting of them.

Currently no noted fix.

Some users' antivirus software detects the Windows installer as a virus. This is a confirmed false-positive as a result of the method used to compile ftldat into an easily dispersed installer.

Currently no noted intent on OP's part to recompile installer using an alternative method to bypass false-positive.

Translation attempts via saving .xml files using different character encodings (e.g. UTF-8, Big5, GBK) results in square blocks/boxes where text should be instead. Not a ftldat issue, but worth being aware of in case you wish to attempt translation.

[The Sober Fox]

I was in the process of writing to ask how to get it all up and running for Mac users but then worked it out whilst waiting for the membership validation email for come through. Having seen a few comments through the pages (have read most, not all of the 25 pages, at time of writing) I thought I'd put up a little tutorial for those who are new to this and on a Mac, like myself. Apologies to OP if this seems un-sporting, theres no maliciousness intended!

Mac Users Initial Setup (in the simplest of terms)
1. Download the GitHub folder at https://github.com/bwesterb/ftldat, not the .exe installer for obvious reasons.
2. Move the downloaded folder to the desired location.
3. Right click on the folder and select "Get Info"
4. In the "General" section of the window copy the location of the folder, listed after "Where:"
5. Open up Terminal
6. To unpack the data.dat file type the following in

Code: Select all

python /[Paste in the location you copied in step 4]/ftldat/src/main.py unpack

7. Wait for it to finish extracting/unpacking. If worked successfully then your Terminal window will list lots of files like these:

Code: Select all

 data/rock_cruiser.xml
 data/rock_cruiser_2.txt
 data/rock_cruiser_2.xml
 data/rock_fight.txt
 data/rock_fight.xml
 data/rock_scout.txt
 data/rock_scout.xml
 data/sector_data.xml

8. Go to your Applications folder and locate FTL
9. Right click on the file and select "Show package contents"
10. Go through Contents > Resources. Here it should now have a folder called data.dat-unpacked with a folder called data inside.
11. Open this folder and you'll have all of the .txt and .xml files you wish to mod

Hope this helps the confusion for some people! Big thanks to bas and everyone that has worked on this CLI tool, it really is amazing!! (Oh and it most definitely isn't a virus. So shush.)

[ZahaianGhost]

Hello everyone! Finally back after a bit of real life this and that as well as simply trying to pin down where some of the issues with this tool's manual compilation were, and I'm happy to say after much abashing of my head out of my own stupidity and unfamiliarity with the tools used, I managed to figure it out. As a result, I come bearing some gifts that will hopefully prove useful.

First and foremost, I initially incidentally forked Bas' tool on GitHub as I stumbled about figuring that lot out, but it has proven to our benefit, as I've revised his ReadMe file if only ever so slightly, and somewhat overhauled the contents of the Win-Installer folder, to the point that I feel it appropriate to call it v2 as I believe this is roughly what Bas himself made when providing us with his installer yet never got around to updating in the old Win-Installer folder, leaving us with a slightly rough to manually compile v1.

That being said, here's the fork of the tool, with cleaned up Win-Installer and slightly clarified readme to boot: https://github.com/ZahaianGhost/ftldat

Alongside this, I looked into what was causing the virus false-positives with the installer, and found that as others here had noted, it was a known issue to the guys behind the tool. Looking a little further on, I also found that it sounded like maybe using the dev version of the tool would resolve the issue, so I tried repacking and recompiling the installer using that to see if it might work. Unfortunately I don't have any systems that haven't been exposed to the ftldat installers in working through all this, so I can't verify whether or not it really worked, leaving me to just hope you guys might try it and tell me. I'm not expecting it to honestly, given some of what I read, but if it does, hey, that's awesome!

Here's the link to that if you didn't already catch it in the above link. I really hope this helps, and if not, I hope cleaning up the Win-Installer folder and revising some of the files there helps to make manually compiling it easier.

Thanks for giving me something fun to tinker on if you come back around Bas, and hopefully now I can get around to actually working on some little mods here and there for the game!

[Limoster]

I managed to install the ftldat, but when I pack an archive it tells me this.:

Code: Select all

c:\Program Files (x86)\ftldat\dist>ftldat.exe pack "c:\users\andreialexandru\desktop\resource.dat-unpacked" -f
Listing files to pack...
Create datafile...
Traceback (most recent call last):
  File "<string>", line 736 in <module>
  File "<string>", line 733, in main
  File "<string>", line 615, in main
  File "<string>", line 494, in cmd_pack
  File "<string>", line 164, in __init__
IOError: [Errno 13] Permiision denied: "c:\\users\\andreialexandru\\desktop\resource.dat-unpacked" -f



By the way: I program in Windows 8.1 and your FTLDAT r7 Installer without python contains malware. My computer found some Trojans and keyloggers. I scanned-it with MalwareBytes PRO and BitDefender 2014 PRO.

[kartoFlane]

@Limoster
ftldat needs to be run with admin permissions, otherwise it can't unpack the files (or rather, write the unpacked ones)

By the way: I program in Windows 8.1 and your FTLDAT r7 Installer without python contains malware. My computer found some Trojans and keyloggers. I scanned-it with MalwareBytes PRO and BitDefender 2014 PRO.

viewtopic.php?f=12&t=1920&p=43997#p43997

Either way, you might want to use another tool, seeing as ftldat has become a bit dated and problematic. For one, SMM has a feature that allows you to unpack the game's archives without having to play around with command line - just go to File > Extract Dats.

[ZahaianGhost]

I'd have to agree with Karto considering that that's the case. Thought I'd give it another shot to make it less bothersome but seems the situation remains as some of the PyInstaller guys saw it, a matter of constantly working around the antivirus guys rather than with that ensures this stuff will repeatedly get flagged by their detection no matter how much they patch up PyInstaller. That being the situation, you're pretty much better off working with the other tools available, such as the SMM mentioned.

[Limoster]

@kartoFlame

I always run Command Prompt (CMD) but it tells me [Errno 13] ACCESS DENIED.
I have installed ftldat r7 correctly and it took me 4 hours to be able to install it because I was getting errors and program incompatibilities but now it's finally working. I've got SMM and it works but I do not want to extract the .dat, I want to compress-it again so FTL can read the files. Oh! Now I remember . I can pack the files with a ship edditor, it's called Superluminal. When you finish a project you can save it like a folder or a .dat file with more folders inside so when I finish my small mod I can compress all the archives again.
Let's see if it works. I will give updates about it.

[kartoFlane]

@Limoster
Why not use the .ftl format, then? That's what Superluminal does during the exportation process, it basically packs the mod for you. You can read about how to create .ftl files here.
Either way, whatever your reason, you might be interested in KuroSaru's FTL Dat Manager

[DragonMaster2962]

my computer says its malicious and i don't know what to do

[kartoFlane]

viewtopic.php?f=12&t=1920&p=43997#p43997

Either way, you might want to use another tool, seeing as ftldat has become a bit dated and problematic. For one, SMM has a feature that allows you to unpack the game's archives without having to play around with command line - just go to File > Extract Dats.

If for whatever reason you want to pack dats directly, without SMM as an intermediary, you might be interested in Dat Manager