PHO - Pokémon Hackers Online
Go Back   PHO - Pokémon Hackers Online > Other Generations Hacking > Tools, Utilities & Resources

Tools, Utilities & Resources Get tools that help you create and ease the process of creating and modifying. Share resources for people to use in their hack.

Reply
 
Thread Tools Display Modes
Old 30th January 2014, 04:32 AM   #1
Full Metal ★
Unstable?
Ex-StaffPHO VIP
 
Full Metal ★'s Avatar
 
Join Date: Oct 2012
Posts: 326
Full Metal ★ Full Metal ★
Lightbulb Patch Builder

Patch Builder



What is it?


Patch Builder is a completely new methodology to hacking that focuses on files and data structures rather than offsets, pointers, and bytes. In other words, when you're mapping, you save a map file -- not your ROM file. Later, this tool takes that map file, finds free space in a ROM, turns the map file into appropriate bytes, and then saves an IPS patch for it. This tool is designed to run with Python 3.3, but is 2.7 compatible ( to my knowledge ). You can get Python at the official site.



Why Use It?


As it stands, ROM Hacking involves a certain level of maintaining knowledge of where things are inserted, tracking changes, dealing with various bugs in tools, etc. This tool circumvents this by keeping the tools separate from the ROM. An other advantage to using this methodology is the ability to collaborate as a team on the same ROM Hack Project.



How Does It Work?


It's a lot more simple than you might think. First - this program depends on descriptions of in-rom structures. It looks for these in a few places:
  • A semicolon seperated list of paths in the environment variable "RomStructureSearch".
  • A subfolder "RomStructureSearch" in the location of the script.
  • A subfolder "RomStructureSearch" in the current directory.
  • The current directory.
.

Rom Structure definitions need to be named with the extension ".RomStructure.json". Since this software still has some engineering left to be done, the exact structure is not presently defined or set in stone. I plan to add support for specialization according to each game code. ( You can specify different structure definitions on a "per-game-code" basis, in other words ). These structures will also specify any enumerations ( "bulbasaur" is pokemon 0x00, for example), as well as any constants that need to be determined ( the map bank table, for example ). To notate a field as having a reference, prefix it with an "@" symbol in the fields list ( see the Git folders RomStructureSearch for an up-to-date example of a RomStructure file ). If it points to a table, then you can specify that like so "@[field_saying_how_large_the_table_is]@[field_name]". ( I need to add support for x-terminated tables, I just haven't yet ).

Entity's are what make up a project. Entities are named like so: entity_name.[entity_type].json. So if you made a map, the correct filename would be farmtown.pokemonmap.json. This would indicate to Patch Builder that the entity is named "farmtown" and that it is a "pokemonmap". Like Rom Structure's, you can also specialize on a per-game-code basis. These are the files that editors will work with. To refer to an internal rom structure in a referring field ( "@connection_header" ) then just set this field in the entity data area to be an integer set to the offset of the structure in-game. If you want to refer to an entity file, then you can simply enter "farmtown.connection_header" in the appropriate data field.

You'll notice I keep mentioning "game-code-specialization" -- for good reason! One of the huge benefits to this project is that you work on the meta-level. In other words, a map you make can be inserted to both Ruby, and Saphire, or even Fire Red. If you want to make two versions of your game ( like Ruby and Saphire ) then you can do that, too!

As it stands, there's not any existing tools for this type of hacking. But lo! Being as these are JSON documents, editing them manually is simple, and intuitive ( when compared to using a hex editor ).



Where Can I Find Out More?


This project is currently hosted on Github at this repository. Fee free to fork and pull away!

You can also clone it using Github For Windows or from this .git url using your console.

If you don't have git and don't feel like installing it, you can also download the latest revision at this url.



Who All Is Involved?


So far, this has mostly been a one-man-show. However, this is a big show and I'm looking for some help. Whether it's Rom Structure ideas, Rom Structure information, Table offsets, or anything, I'm interested in hearing what you have to contribute. If you're interested in helping, you can use Git, or Skype me through my signature.
  • Karate Kid - LZ77 Algorithms
  • Full Metal - Main Coder



__________________

★ full metal.

#busy

Last edited by Full Metal ★; 14th April 2014 at 02:37 PM.
Full Metal ★ is offline   Reply With Quote
Likes DarkPsychic, circaoffire liked this post
Sponsored Links
Old 30th January 2014, 12:50 PM   #2
Elsa
Let it go!
PHO VIP
 
Elsa's Avatar
 
Join Date: Apr 2010
Location: United States
Age: 21
Posts: 384
Elsa
Default

Holy, man Full... This really sounds like the dream of hacking. The whole idea of patching it would make hacking as a team so much nicer (That free space finder is a nice touch). Could repatching on a single rom many times run into issues, though? Well, now that I think of it, it's probably safer than what we currently do... Anyway, spectacular job, FM!
__________________

- - - - - - - Elsa - - - - - - -
Twitter | YouTube | Steam | Twitch
- - - - - - - - - - - - - - - - -
Elsa is offline   Reply With Quote
Likes circaoffire liked this post
Old 30th January 2014, 01:35 PM   #3
karatekid552
What does this button do?.....
Ex-Staff
 
karatekid552's Avatar
 
Join Date: Feb 2013
Location: Stalker.......
Posts: 242
karatekid552 karatekid552
Default

I finished both LZ decompression and compression. I still need to make sure the functions work perfectly, but when I do, you can use them. You may need to adjust them for python 3.3 as I wrote them in 2.7. My functions work on byte strings (ie, what you get when you read a from a file in binary mode) so they should be pretty portable.

If you want to write your own, just ask me and I'll give you the structure.
karatekid552 is offline   Reply With Quote
Old 30th January 2014, 03:52 PM   #4
Full Metal ★
Unstable?
Ex-StaffPHO VIP
 
Full Metal ★'s Avatar
 
Join Date: Oct 2012
Posts: 326
Full Metal ★ Full Metal ★
Default

Quote:
Originally Posted by Rocka View Post
Holy, man Full... This really sounds like the dream of hacking. The whole idea of patching it would make hacking as a team so much nicer (That free space finder is a nice touch). Could repatching on a single rom many times run into issues, though? Well, now that I think of it, it's probably safer than what we currently do... Anyway, spectacular job, FM!
You won't re-apply the same patch multiple times. That's the point. :P
[ the included ips patcher makes a copy of the ROM that has been patched, rather than in-place modifications ].

Quote:
Originally Posted by karatekid552 View Post
I finished both LZ decompression and compression. I still need to make sure the functions work perfectly, but when I do, you can use them. You may need to adjust them for python 3.3 as I wrote them in 2.7. My functions work on byte strings (ie, what you get when you read a from a file in binary mode) so they should be pretty portable.

If you want to write your own, just ask me and I'll give you the structure.
I'd be glad to have a look at them!
__________________

★ full metal.

#busy
Full Metal ★ is offline   Reply With Quote
Likes Elsa liked this post
Old 1st February 2014, 07:24 AM   #5
bcrobert
n00b
 
Join Date: Nov 2013
Posts: 6
bcrobert
Default

My brain hurts. So basically this means we wouldn't have to choose between (A)tediously hex editing every structure or (B)juggling tool after tool, forced to trust that they're all compatible, because with this we could (C)produce and edit safe, easy to understand json docs?

That's oversimplified but is that the gist of it? Cuz it sounds magical.
__________________
If all data could be organized into tables...that would be Heaven.
bcrobert is offline   Reply With Quote
Old 1st February 2014, 03:30 PM   #6
Full Metal ★
Unstable?
Ex-StaffPHO VIP
 
Full Metal ★'s Avatar
 
Join Date: Oct 2012
Posts: 326
Full Metal ★ Full Metal ★
Default

Quote:
Originally Posted by bcrobert View Post
My brain hurts. So basically this means we wouldn't have to choose between (A)tediously hex editing every structure or (B)juggling tool after tool, forced to trust that they're all compatible, because with this we could (C)produce and edit safe, easy to understand json docs?

That's oversimplified but is that the gist of it? Cuz it sounds magical.
That is the gist of it, yes. And a large portion of this is already done. I've just gotten ridiculously busy with college stuff this week because I procrastinated on it. But yeah. It's coming soon!
__________________

★ full metal.

#busy
Full Metal ★ is offline   Reply With Quote
Old 2nd February 2014, 05:40 AM   #7
Full Metal ★
Unstable?
Ex-StaffPHO VIP
 
Full Metal ★'s Avatar
 
Join Date: Oct 2012
Posts: 326
Full Metal ★ Full Metal ★
Default

* bump *

Update:

* Currently detects entity files in a directory correctly
* Detects dependencies within an entity for static and dynamic entities.
* Execution flow is set up and mostly complete. Actual patch exporting not yet implemented.
* LZ77 Decompression is available, but not yet used.
* LZ77 Compression is not yet available.

I have some thinking to do about constants / enumerations, etc and how I'm going to store and handle that. But the skeleton to this program is really nearing completion fairly quickly.
__________________

★ full metal.

#busy
Full Metal ★ is offline   Reply With Quote
Old 2nd February 2014, 08:16 AM   #8
bcrobert
n00b
 
Join Date: Nov 2013
Posts: 6
bcrobert
Default

Quote:
Originally Posted by Full Metal ★ View Post
That is the gist of it, yes. And a large portion of this is already done. I've just gotten ridiculously busy with college stuff this week because I procrastinated on it. But yeah. It's coming soon!
Cool, just don't burn yourself out between all that schoolwork and programming. Better to work happily than fast.
__________________
If all data could be organized into tables...that would be Heaven.
bcrobert is offline   Reply With Quote
Old 22nd February 2014, 05:25 AM   #9
Full Metal ★
Unstable?
Ex-StaffPHO VIP
 
Full Metal ★'s Avatar
 
Join Date: Oct 2012
Posts: 326
Full Metal ★ Full Metal ★
Default

update:

I've finally determined a data structure that should encompass all the immediate needs I can think of for an Entity object:

Code:
{
  data:{ // default data. Consider this to be a "bottom layer", see comments below.
    field_name: field_value
  },
  specialization:{ // the "top layer" for data. Overrides any data in the root "data" node.
    game_code:{
      field_name: field_value
    }
  },
  meta:{ // not used by the compiler. simply used by the editor for convenience.
    field_name: field_value
  }
}
Then for data structures, my proposed final structure is as follows:

Code:
{
  fields:{
    name:{
     byte_code: struct.compile_code// see specification of struct's bytecode in the python docs.
     order: where_in_structure_should_be_placed
     description: optional_string
     refers_to: null_or_name_of_structure_type
    }
  },
  tables:{
    table_name:{
      index_name: null_or_name_of_field_in_structure // null means use meta information '$index',
      entry_structure: name_of_structure_for_entry // you can also pass in a structure object here instead of a reference string, but it's recommended to use a reference string.
      location: internal_rom_pointer_to_table
    }
  },
  constants:{
    name: value
  }
  specialization:{
    game_code:{
      fields: {},
      tables: {},
      constants: {}
    }
  }
}
There will also be a file used for maintaining a list of known free-space chunks given a game code. I'm undecided on whether this will be stored in a single file with a constant name, or if it will be spread over other files. It's not needed, but if you don't have this, then you do need a ROM so that you can process it for free-space information. Anyways, the structure for an individual game will be this:
Code:
[
  {
    location: internal_rom_pointer,
    size: how many bytes can be overwritten safely,
    value: 0xFF // optional.0xFF is assumed unless specified otherwise. Anything beyond the size of a byte will be interpreted as "any byte".
  }
]
Developers, do you have any questions or feedback for this proposal?
__________________

★ full metal.

#busy
Full Metal ★ is offline   Reply With Quote
Likes circaoffire liked this post
Reply

Tags
hackbuild, [Editor]

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 10:40 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc. User Alert System provided by Advanced User Tagging (Lite) - vBulletin Mods & Addons Copyright © 2017 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Lite) - vBulletin Mods & Addons Copyright © 2017 DragonByte Technologies Ltd.
Pokémon characters and images belong to Pokémon USA, Inc. and Nintendo.
Pokémon Hackers Online (PHO) is in no way affiliated with or endorsed by Nintendo LLC, Creatures, GAMEFREAK inc,
The Pokémon Company, Pokémon USA, Inc., The Pokémon Company International, or Wizards of the Coast.
All forum/site content (unless noted otherwise) and site designs are © 2006-2013 Pokémon Hackers Online (PHO).
Green Charizard Christos TreeckoLv100

"Black 2" by ARTPOP. Kyurem artwork by XOUS.

no new posts