Changing wild Pokemon's species in memory

[DoomInAJar]

I'm working on a lua script for Pokemon FireRed which changes certain values of wild Pokemon when you meet them (for example, turning a wild pokemon into a Bulbasaur, or changing its moveset). My current issue is with changing the species of the Pokemon.

To do this, I read in the data of the wild Pokemon, decrypt it, find the growth substructure, change the first 2 bytes to the new Pokemon's id, calculate the checksum, re-encrypt the data, and then write it all back to memory. See "Pokémon data substructures in Generation III" on Bulbapedia.

It mostly works fine. I can encounter the modified Pokemon, capture it, and fight with it. The only problem is that when I try to view the Pokemon's summary page, the game usually crashes (I say "usually" because it seems to depend on the species of the Pokemon that was modified: out of the one's I've tried, it consistently crashes for Weedle, Caterpie, Kakuna, and Rattata, and works for Pidgey). Any ideas as to what would cause this? The only thing I can think of is that there's some species specific value that I'm neglecting to change.

For anyone who is inclined, the following is a lua script which reproduces the error. I'm using Pokemon FireRed US V1.1 and the VBA-ReRecording emulator. Save the script as a .lua file, run it in the emulator, and every wild Pokemon you meet (and the first Pokemon in enemy trainers' parties) should become a Bulbasaur.

Spoiler:

G = 1
A = 2
E = 3
M = 4
substructureOrders = {
    [0] = {G,A,E,M},
    [1] = {G,A,M,E},
    [2] = {G,E,A,M},
    [3] = {G,E,M,A},
    [4] = {G,M,A,E},
    [5] = {G,M,E,A},
    [6] = {A,G,E,M},
    [7] = {A,G,M,E},
    [8] = {A,E,G,M},
    [9] = {A,E,M,G},
    [10] = {A,M,G,E},
    [11] = {A,M,E,G},
    [12] = {E,G,A,M},
    [13] = {E,G,M,A},
    [14] = {E,A,G,M},
    [15] = {E,A,M,G},
    [16] = {E,M,G,A},
    [17] = {E,M,A,G},
    [18] = {M,G,A,E},
    [19] = {M,G,E,A},
    [20] = {M,A,G,E},
    [21] = {M,A,E,G},
    [22] = {M,E,G,A},
    [23] = {M,E,A,G}
}
enemyPokemonPointer = 0x0202402C -- pointer to the enemy pokemon's data structure
dataStructureOffset = 32
checksumOffset = 28
substructureSize = 12
pokemonId = 1 -- bulbasaur

function run()
    print('battle started')
    otId = memory.readdword(enemyPokemonPointer+4)
    personality = memory.readdword(enemyPokemonPointer)
    encryptionKey = bit.bxor(otId, personality)
    substructureOrder = substructureOrders[personality%24]

    -- read the data in 12 byte chunks, one chunk for each substructure
    encryptedData = {}
    for i=1,4 do
        encryptedData[i] = memory.readbyterange(enemyPokemonPointer+dataStructureOffset+(i-1)*substructureSize,substructureSize)
    end
    
    -- decrypt the data
    decryptedData = {}
    for i=1,4 do
        decryptedData[i] = xor(encryptedData[i], encryptionKey)
    end
    
    -- organize the data
    growth = decryptedData[indexOf(substructureOrder,G)]
    attacks = decryptedData[indexOf(substructureOrder,A)]
    ev = decryptedData[indexOf(substructureOrder,E)]
    misc = decryptedData[indexOf(substructureOrder,M)]
    
    -- set the pokemon species
    growth[1] = bit.band(pokemonId, 0xFF)
    growth[2] = bit.rshift(pokemonId, 8)
    
    -- reassemble the data
    newUnencryptedData = {}
    newUnencryptedData[indexOf(substructureOrder,G)] = growth
    newUnencryptedData[indexOf(substructureOrder,A)] = attacks
    newUnencryptedData[indexOf(substructureOrder,E)] = ev
    newUnencryptedData[indexOf(substructureOrder,M)] = misc
    
    -- calculate the checksum
    checksum = calculateChecksum(newUnencryptedData)
    
    -- re-encrypt the data
    newEncryptedData = {}
    for i=1,4 do
        newEncryptedData[i] = xor(newUnencryptedData[i], encryptionKey)
    end
    
    -- write the new data
    for i=1,4 do
        for j=1,#newEncryptedData[i] do
            memory.writebyte(enemyPokemonPointer + dataStructureOffset + (i-1)*substructureSize + (j-1), newEncryptedData[i][j])
        end
    end
    memory.writeword(enemyPokemonPointer+checksumOffset, checksum)
    
end

function calculateChecksum(unencryptedData)
    sum = 0
    for i=1,#unencryptedData do
        for j=1,#unencryptedData[i],2 do
            sum = sum +
                unencryptedData[i][j] +
                unencryptedData[i][j+1]*0x100
        end
    end
    return bit.band(sum, 0xFFFF)
end

-- xor a table of bytes with a double word repeatedly
function xor(byteTable, val)
    valTable = {
        bit.band(val,0xFF),
        bit.ror(bit.band(val,bit.rol(0xFF,8)), 8),
        bit.ror(bit.band(val,bit.rol(0xFF,16)), 16),
        bit.ror(bit.band(val,bit.rol(0xFF,24)), 24)
    }
    ret = {}
    for i=1,#byteTable do
        ret[i] = bit.bxor(byteTable[i], bit.bxor(valTable[(i-1)%4+1]))
    end
    return ret
end

-- get the index of val in table. Return 0 if not found
function indexOf(table, val)
    for i=1,#table do
        if table[i] == val then
            return i
        end
    end
    return 0
end

print('script started')
memory.registerexec(0x08010672,run) -- address is executed whenever a battle starts

[DoomInAJar]

I think I have a lead on the problem: I need to be changing experience and/or levels along with the Pokemon species.

The reason for this is that different Pokemon level up at different rates. For example, Pidgey and Bulbasaur level up at a medium-slow rate, whereas Weedle, Caterpie, Kakuna, and Rattata level up at a medium-fast rate; thus, a Rattata with X experience should be at an equal or higher level to a Bulbasaur with X experience. When I convert a Rattata to a Bulbasaur and don't touch the level or experience, the game gets a little bit confused since the Pokemon has a higher level than it should. This fault triggers a failure when trying to calculate the experience required to reach the next level, which is shown in the Summary screen. The game did not crash when converting a Pidgey to a Bulbasaur since they level up at a similar rate.

A possible solution would be to calculate the experience that the new Pokemon should have at its current level, and write that value to the Pokemon.

[DoomInAJar]

I figured I'd follow up for anyone with the same problem as me. The fix described above appears to work, and I have updated the code below.

The important updates are contained between the "NEW CODE" comments. I have also include a csv file used in the script which lists the minimum experience points Pokemon of each level and level up type will have, courtesy of Bulbapedia.

Spoiler:

script.lua:

G = 1
A = 2
E = 3
M = 4
substructureOrders = {
    [0] = {G,A,E,M},
    [1] = {G,A,M,E},
    [2] = {G,E,A,M},
    [3] = {G,E,M,A},
    [4] = {G,M,A,E},
    [5] = {G,M,E,A},
    [6] = {A,G,E,M},
    [7] = {A,G,M,E},
    [8] = {A,E,G,M},
    [9] = {A,E,M,G},
    [10] = {A,M,G,E},
    [11] = {A,M,E,G},
    [12] = {E,G,A,M},
    [13] = {E,G,M,A},
    [14] = {E,A,G,M},
    [15] = {E,A,M,G},
    [16] = {E,M,G,A},
    [17] = {E,M,A,G},
    [18] = {M,G,A,E},
    [19] = {M,G,E,A},
    [20] = {M,A,G,E},
    [21] = {M,A,E,G},
    [22] = {M,E,G,A},
    [23] = {M,E,A,G}
}
enemyPokemonPointer = 0x0202402C -- pointer to the enemy pokemon's data structure
pokemonBaseStatsPointer = 0x08254810
dataStructureOffset = 32
checksumOffset = 28
substructureSize = 12
pokemonId = 1 -- bulbasaur
levelOffset = 84

function run()
    print('battle started')
    otId = memory.readdword(enemyPokemonPointer+4)
    personality = memory.readdword(enemyPokemonPointer)
    encryptionKey = bit.bxor(otId, personality)
    substructureOrder = substructureOrders[personality%24]

    -- read the data in 12 byte chunks, one chunk for each substructure
    encryptedData = {}
    for i=1,4 do
        encryptedData[i] = memory.readbyterange(enemyPokemonPointer+dataStructureOffset+(i-1)*substructureSize,substructureSize)
    end
    
    -- decrypt the data
    decryptedData = {}
    for i=1,4 do
        decryptedData[i] = xor(encryptedData[i], encryptionKey)
    end
    
    -- organize the data
    growth = decryptedData[indexOf(substructureOrder,G)]
    attacks = decryptedData[indexOf(substructureOrder,A)]
    ev = decryptedData[indexOf(substructureOrder,E)]
    misc = decryptedData[indexOf(substructureOrder,M)]
    
    -- set the pokemon species
    growth[1] = bit.band(pokemonId, 0xFF)
    growth[2] = bit.rshift(pokemonId, 8)
    
    ----------------------- NEW CODE --------------------------
    -- read the level of the pokemon
    level = memory.readbyte(enemyPokemonPointer+levelOffset)
    
    -- read the csv file
    levelsToExpCsv = {}
    for line in io.lines('levels.csv') do
        levelsToExpCsv[#levelsToExpCsv+1] = line
    end
    levelsToExp = {}
    for i=2,#levelsToExpCsv do
        levelsToExp[#levelsToExp+1] = ParseCSVLine(levelsToExpCsv[i], ",")
    end
    
    -- read the level up type from memory
    levelUpType = memory.readbyte(pokemonBaseStatsPointer + (pokemonId-1)*28 + 19)
    
    -- look up how much exp the pokemon should have
    exp = levelsToExp[level][levelUpType+2]
    
    -- overwrite the exp.
    growth[5] = bit.band(exp, 0xFF)
    growth[6] = bit.band(bit.rshift(exp,8),0xFF)
    growth[7] = bit.band(bit.rshift(exp,16),0xFF)
    growth[8] = bit.band(bit.rshift(exp,24),0xFF)
    ----------------------- NEW CODE --------------------------
    
    -- reassemble the data
    newUnencryptedData = {}
    newUnencryptedData[indexOf(substructureOrder,G)] = growth
    newUnencryptedData[indexOf(substructureOrder,A)] = attacks
    newUnencryptedData[indexOf(substructureOrder,E)] = ev
    newUnencryptedData[indexOf(substructureOrder,M)] = misc
    
    -- calculate the checksum
    checksum = calculateChecksum(newUnencryptedData)
    
    -- re-encrypt the data
    newEncryptedData = {}
    for i=1,4 do
        newEncryptedData[i] = xor(newUnencryptedData[i], encryptionKey)
    end
    
    -- write the new data
    for i=1,4 do
        for j=1,#newEncryptedData[i] do
            memory.writebyte(enemyPokemonPointer + dataStructureOffset + (i-1)*substructureSize + (j-1), newEncryptedData[i][j])
        end
    end
    memory.writeword(enemyPokemonPointer+checksumOffset, checksum)
    
end

function calculateChecksum(unencryptedData)
    sum = 0
    for i=1,#unencryptedData do
        for j=1,#unencryptedData[i],2 do
            sum = sum +
                unencryptedData[i][j] +
                unencryptedData[i][j+1]*0x100
        end
    end
    return bit.band(sum, 0xFFFF)
end

-- xor a table of bytes with a double word repeatedly
function xor(byteTable, val)
    valTable = {
        bit.band(val,0xFF),
        bit.ror(bit.band(val,bit.rol(0xFF,8)), 8),
        bit.ror(bit.band(val,bit.rol(0xFF,16)), 16),
        bit.ror(bit.band(val,bit.rol(0xFF,24)), 24)
    }
    ret = {}
    for i=1,#byteTable do
        ret[i] = bit.bxor(byteTable[i], bit.bxor(valTable[(i-1)%4+1]))
    end
    return ret
end

-- get the index of val in table. Return 0 if not found
function indexOf(table, val)
    for i=1,#table do
        if table[i] == val then
            return i
        end
    end
    return 0
end

-- via http://lua-users.org/wiki/LuaCsv
function ParseCSVLine (line,sep) 
	local res = {}
	local pos = 1
	sep = sep or ','
	while true do 
		local c = string.sub(line,pos,pos)
		if (c == "") then break end
		if (c == '"') then
			local txt = ""
			repeat
				local startp,endp = string.find(line,'^%b""',pos)
				txt = txt..string.sub(line,startp+1,endp-1)
				pos = endp + 1
				c = string.sub(line,pos,pos) 
				if (c == '"') then txt = txt..'"' end
			until (c ~= '"')
			table.insert(res,txt)
			assert(c == sep or c == "")
			pos = pos + 1
		else
			local startp,endp = string.find(line,sep,pos)
			if (startp) then 
				table.insert(res,string.sub(line,pos,startp-1))
				pos = endp + 1
			else
				table.insert(res,string.sub(line,pos))
				break
			end 
		end
	end
	return res
end

print('script started')
memory.registerexec(0x08010672,run) -- address is executed whenever a battle starts

levels.csv:

Level,Medium Fast,Erratic,Fluctuating,Medium Slow,Fast,Slow
1,0,0,0,0,0,0
2,8,15,4,9,6,10
3,27,52,13,57,21,33
4,64,122,32,96,51,80
5,125,237,65,135,100,156
6,216,406,112,179,172,270
7,343,637,178,236,274,428
8,512,942,276,314,409,640
9,729,1326,393,419,583,911
10,1000,1800,540,560,800,1250
11,1331,2369,745,742,1064,1663
12,1728,3041,967,973,1382,2160
13,2197,3822,1230,1261,1757,2746
14,2744,4719,1591,1612,2195,3430
15,3375,5737,1957,2035,2700,4218
16,4096,6881,2457,2535,3276,5120
17,4913,8155,3046,3120,3930,6141
18,5832,9564,3732,3798,4665,7290
19,6859,11111,4526,4575,5487,8573
20,8000,12800,5440,5460,6400,10000
21,9261,14632,6482,6458,7408,11576
22,10648,16610,7666,7577,8518,13310
23,12167,18737,9003,8825,9733,15208
24,13824,21012,10506,10208,11059,17280
25,15625,23437,12187,11735,12500,19531
26,17576,26012,14060,13411,14060,21970
27,19683,28737,16140,15244,15746,24603
28,21952,31610,18439,17242,17561,27440
29,24389,34632,20974,19411,19511,30486
30,27000,37800,23760,21760,21600,33750
31,29791,41111,26811,24294,23832,37238
32,32768,44564,30146,27021,26214,40960
33,35937,48155,33780,29949,28749,44921
34,39304,51881,37731,33084,31443,49130
35,42875,55737,42017,36435,34300,53593
36,46656,59719,46656,40007,37324,58320
37,50653,63822,50653,43808,40522,63316
38,54872,68041,55969,47846,43897,68590
39,59319,72369,60505,52127,47455,74148
40,64000,76800,66560,56660,51200,80000
41,68921,81326,71677,61450,55136,86151
42,74088,85942,78533,66505,59270,92610
43,79507,90637,84277,71833,63605,99383
44,85184,95406,91998,77440,68147,106480
45,91125,100237,98415,83335,72900,113906
46,97336,105122,107069,89523,77868,121670
47,103823,110052,114205,96012,83058,129778
48,110592,115015,123863,102810,88473,138240
49,117649,120001,131766,109923,94119,147061
50,125000,125000,142500,117360,100000,156250
51,132651,131324,151222,125126,106120,165813
52,140608,137795,163105,133229,112486,175760
53,148877,144410,172697,141677,119101,186096
54,157464,151165,185807,150476,125971,196830
55,166375,158056,196322,159635,133100,207968
56,175616,165079,210739,169159,140492,219520
57,185193,172229,222231,179056,148154,231491
58,195112,179503,238036,189334,156089,243890
59,205379,186894,250562,199999,164303,256723
60,216000,194400,267840,211060,172800,270000
61,226981,202013,281456,222522,181584,283726
62,238328,209728,300293,234393,190662,297910
63,250047,217540,315059,246681,200037,312558
64,262144,225443,335544,259392,209715,327680
65,274625,233431,351520,272535,219700,343281
66,287496,241496,373744,286115,229996,359370
67,300763,249633,390991,300140,240610,375953
68,314432,257834,415050,314618,251545,393040
69,328509,267406,433631,329555,262807,410636
70,343000,276458,459620,344960,274400,428750
71,357911,286328,479600,360838,286328,447388
72,373248,296358,507617,377197,298598,466560
73,389017,305767,529063,394045,311213,486271
74,405224,316074,559209,411388,324179,506530
75,421875,326531,582187,429235,337500,527343
76,438976,336255,614566,447591,351180,548720
77,456533,346965,639146,466464,365226,570666
78,474552,357812,673863,485862,379641,593190
79,493039,367807,700115,505791,394431,616298
80,512000,378880,737280,526260,409600,640000
81,531441,390077,765275,547274,425152,664301
82,551368,400293,804997,568841,441094,689210
83,571787,411686,834809,590969,457429,714733
84,592704,423190,877201,613664,474163,740880
85,614125,433572,908905,636935,491300,767656
86,636056,445239,954084,660787,508844,795070
87,658503,457001,987754,685228,526802,823128
88,681472,467489,1035837,710266,545177,851840
89,704969,479378,1071552,735907,563975,881211
90,729000,491346,1122660,762160,583200,911250
91,753571,501878,1160499,789030,602856,941963
92,778688,513934,1214753,816525,622950,973360
93,804357,526049,1254796,844653,643485,1005446
94,830584,536557,1312322,873420,664467,1038230
95,857375,548720,1354652,902835,685900,1071718
96,884736,560922,1415577,932903,707788,1105920
97,912673,571333,1460276,963632,730138,1140841
98,941192,583539,1524731,995030,752953,1176490
99,970299,591882,1571884,1027103,776239,1212873
100,1000000,600000,1640000,1059860,800000,1250000