~PSP files are definately compressed

[steddy]

I have been looking at some of the none encrypted files on the Ridge Racers UMD and comparing them to the same files in the 1.0 firmware which are encrypted. These are the results:-

libparse_http.prx
UMD = 5,116
Firmware = 3,008

libatrac3plus.prx
UMD = 19,524
Firmware = 10,192

pspnet_inet.prx
UMD = 247,248
Firmware = 130,994

All files show a similar difference in size. I also know that the top two files are actually the same version and the bottom file is a different version.

The reason I know these are the same files is the header of the files contains the uncompressed file size. I don't know if this has already been reported.

At offset 0x28 in the encrypted file you have four bytes which show the uncompressed file size. libparse and libatrac3plus show the correct values and pspnet_inet is slightly different. This tells me that pspnet_inet is actually a different version.

These file ratios tells me we are looking at ZIP type compression ration of almost 2:1. So it could be ZIP, Zlib, etc.

Now the interesting question is are they "comrpessed and signed" or "compressed, signed and encrypted". If they are encrypted, then can we work out form the above if they are decompressed before or after decryption?

I am working on understanding the rest of the headers. I'll post once I have decoded more of them. Apologies if this has already been posted by somebody earlier, but i couldn't find it.

[MelGibson]

Nice :)

I may point you to the PSP File (research thread), too :D

http://forums.ps2dev.org/viewtopic.php? ... c&start=30

[steddy]

Thanks for that link.... useful stuff.

After looking at UMDs, BIN files and the Firmware my decoding of the header so far I beleive to be:-

~PSP File format explained
00-03 : ~PSP Magic Identifier
04 : File Type -

• 00 - BIN
  06 - PRX Library
  07 - Driver / Service

05 : Sub Type (or kernel ring)?? -

• 00 - User Module
  08 - Library Module / System
  10 - Kernel Module

06 : Compression flag - 00 Uncompressed, 01 Compressed
07 : Padding (00)
08-09 : 0x01 0x01 - Part of file name shown in uncompressed file (section 1?)
0A-25 : Filename (26 characters)
26-27 : 0x01 0x02 - Always the same (section 2?)
28-2B : Uncompressed file size
2C-2F : Compressed file size including header
B0-B3 : Compressed file size excluding header

Also, I modified some of the header data in the DATA.PSP file (like the name) and got an error I haven't seen reported before. It was 0xFFFFFED2 (the game could not be started).

I have a feeling the header isn't part of the Hash so can actually be changed. I think the error I am getting is because I have modified the PARAM.SFO or the 1.50 firmware update to be 1.51 and it starts to actually run it, but then realizes its the actually the same version.

BTW I am running this on a firmware 1.50 PSP.


Steddy

[yackom]

if the files are compressed then they wont compress very well under zip or rar. see what kinda ratios you get out of other compression formats

[steddy]

I tried to ZIP both the uncompressed and compressed versions of the same file and they both GREW in size.

This isn't because one is compressed and the other isn't, its because of the chaos the encryption is adding to the file. I now know for sure that both files ARE encrypted.

I have also answered my other question above, files are conmpressed first, then encrypted. There would be no point in doing it the other way round since the encrypted files wouldn't compress, they would expand.

My guess from looking at the file is that it contains an RSA session key or a passphrase used to generate a key which encrypts this file. Finding it should be relatively easy then it can be decompressed.

However, that doesn't mean we can run our own code since the files will also be signed. Since the signature checking will be using the Public keys from the Certs directory and only Sony will have access to the Private key, we won't be able to sign them ourselves.

What I hope to get out of decrypting the files is access to all the raw code to debug and dissassemble. That should lead to clues on exploits.

Steddy

[MelGibson]

I tried to ZIP both the uncompressed and compressed versions of the same file and they both GREW in size.

Really ? Which files where you checking ? I tried boot.bin vs. eboot.bin and
libparse_http.prx (umd) vs. (firmware) and both times the uncompressed(=unencrypted) version compressed really well...

But you may have tried with different files... wonder which ones ??

[steddy]

Thanks for making me question that, it uncovered something interesting.

The files I checked were:-
UMD0:\PSP_GAME\USRDIR\Data\Modules\kmodule\mpegbase.prx
Flash0:\kd\mpegbase.prx

UMD0 - Unzipped = 9.32KB
UMD0 - Zipped = 9.40KB
Flash0 - Unzipped = 4.20KB
Flash0 - Zipped = 4.29KB

The difference between the UMD0 file and the Flash0 file is offset 0x06. UMD0 reports this as 0x00 (uncompressed see above) and Flash0 reports this as 0x01 (Compressed). Neither file reduces in size when I zip it as you can see which I put down to both being encrypted. Both files actually have the ~PSP header so one of them isn't a straight ELF file as I had first thought.

I took a look at the BOOT.BIN and EBOOT.BIN on the Ridge Racers UMD too and found the following:-

BOOT.BIN - Unzipped = 2.45MB
BOOT.BIN - Zipped = 926KB
EBOOT.BIN - Unzipped = 2.45MB (exactly the same size)
EBOOT.BIN - Zipped = 2.45MB (actually 2517KB rather than 2516KB)

BOOT.BIN obviously compresses well because its a RAW executable and compresses how you would expect it too. EBOOT.BIN is encrypted so as expected this means the compression can't do its job. To me this proves that files are compressed (optional) then encrypted.

ELF type files (.PRX or .BIN) compress with a standard ZIP ratio of around 2:1. Encrypted files don't compress at all. It doesn't matter if the file is also compressed, the encryption introduces to much complexity in the file for ZIP to reduce it size.

Interesting stuff.

[MelGibson]

Encrypted files don't compress at all. It doesn't matter if the file is also compressed, the encryption introduces to much complexity in the file for ZIP to reduce it size.

Yes, it's like encryption aims at maximum disorder (= high entropy) thus making encrypted files nearly impossible to compress..

But your findings are very interesting. The thing with the compression bit is really interesting, too....

From your findings i think one could gather this:

28-2B : Size of uncompressed unencrypted ELF File
2C-2F : complete size of actual ~PSP file
B0-B3 : Size of unencrypted File (Note: in case of NO compression this equals 28-2B, in case of compression this is roughly 50% of 28-2B)

[steddy]

I think B0-B3 has a slightly different meaning than you are describing. In compressed files its usually a SMALLER number than the 2C-2F offset so its actually related to the compressed / encrypted file.

I think its the size of the compressed / encrypted file file minus the headers and NOT 16 byte aligned. Ie its the actual encrypted file size containing USEFUL data. The last bits after it are for the padding bytes to 16 byte align the data for the AES algorithm.

As an example, in the mpegbase.prx that is encrypted / compressed the following values can be seen:-

28-2B : 0x000023F8 - unencrypted / uncompressed original file size
2C-2F : 0x000010D0 - encrypted file size including headers
B0-B3 : 0x00000F77 - Above less headers and 16 byte padding

As you can see from the above, 0x10D0 - 0x150 = 0xF80. B0-B3 is saying even though there are 0xF80 data bytes, only 0xF77 of them are needed really.

I have also been doing some more reading on the likely crypto techniques employed by the PSP and come to the following conclusions on the order things are done in.

1. If the data is going to be compressed (flag at 0x06) then compress the data using method X (unknown now but likely zlib or similar)
2. Sign the data and place the signature in the header. This will be either a 16 byte MD5 hash encrypted with a private key or a 20 byte SHA1 (more likely) encrypted with the private key.
3. Encrypt the data using a random AES session key (16 bytes) and store the key in the header
4. Use a random or calculated IV and store that in the header too (8 bytes for AES-128).

The assumptions are based on the following common crypo knowledge:-

1. Compressing the data after crypto wouldn't acheive anything.
2. Public / Private key crypto is too slow for encrypting large files and is vulnerable to clear text attacks. Instead it is usually used for encrypting small amounts of data such as a hash (digital signature)
3. Its safer and common practise to sign data BEFORE encrypting it. There are known crypto attacks against RSA if done the other way around.
4. SHA1 is stronger than MD5 and from the thread on documenting library calls we know the PSP has an implementation of SHA1 in ROM.
5. It wouldn't be sensible for Sony to include the Private key in ROM, so it will only contain the Public key.

The file loader should do all of the following and this may give a clue to those other fields:-

1. Decrypt the data using the Session key and IV value contained in the file (its possible this is encrypted with another key in ROM)
2. Hash the decrypted data with SHA1.
3. Decrypt the digital signature in the header with the Public key in ROM
4. Compare the output from 2 and 3 to ensure the file hasn't been tampered with and was created by Sony
5. Expand the file if its compressed.
6. Maybe write it to a temporary area unencrypted or just place it directly in memory.

I suspect the AES key, IV and signature are all in the block from 0x80 - 0xB0. I am going to have a play with these and see if I can decode the file.

We won't be able to sign our own code with this knowledge unfortunately but we would have access to the unencrypted files that actually implement the crypto functions which may provide an attack vector.

I hope this stuff is useful.

Steddy

[steddy]

Why the heck as this been moved to the general discussion???

This is in know way general and entirely dev related.

Steddy

[MelGibson]

I think B0-B3 has a slightly different meaning than you are describing. In compressed files its usually a SMALLER number than the 2C-2F offset so its actually related to the compressed / encrypted file.

I think its the size of the compressed / encrypted file file minus the headers and NOT 16 byte aligned. Ie its the actual encrypted file size containing USEFUL data. The last bits after it are for the padding bytes to 16 byte align the data for the AES algorithm.

As an example, in the mpegbase.prx that is encrypted / compressed the following values can be seen:-

28-2B : 0x000023F8 - unencrypted / uncompressed original file size
2C-2F : 0x000010D0 - encrypted file size including headers
B0-B3 : 0x00000F77 - Above less headers and 16 byte padding

As you can see from the above, 0x10D0 - 0x150 = 0xF80. B0-B3 is saying even though there are 0xF80 data bytes, only 0xF77 of them are needed really.

I think you got a point there... But wouldn't that mean that in the case of no compression the encrypted filesize always more or less matches the unencrypted one, because in all ~PSP Files with no compression B0-B3 == 28-2B.

For example eboot.bin from Wipe0ut:
28-2B: 0xB54A3900 - unencrypted / uncompressed original file size; matches the ELF File boot.bin
2C-2F: 0x104C3900 - encrypted file size including headers
B0-B3: 0xB54A3900 - Here the same as 28-2b

So here this would mean that the encrypted filesize less header/16 byte padding would be the same as the original filesize.

I have to admit that I am not so much into encryption. So I don't know if this is the reasonable or not... Just wanted to raise the question... ;)

And oh yeah... i think this stuff is usefull :D

Mel

[Guest]

Dunno for sure why it was moved, but it does duplicate earlier threads on compressability of encryption. Tons of threads on this topic.

One mod suggested it did not contribute directly to homebrew, but was more of the vein of "Lets figure out other stuff that may be cool". So in other words, if it doesn't help one write cool games and apps, and there are lots of material on that right now, it goes in general.

[cyod]

Except without understanding how the files are encrypted/compressed/signed then there is no way to run code on the PSP 1.5+ firmware, although I admit that it still seems extremely unlikely that we will be able to sign and encrypt our own programs anytime in the near future.

If no exploitable bug can be found to run some sort of bootloader on a PSP with firmware 1.5, then the only even theoretical way to run homebrew code on a PSP would be through knowledge related to this...

[Guest]

Before it was even possible to run homebrew code, you may have had somewhat of a point. However, the situation changes a bit now that homebrewing is possible.

Attention is now focused away from merely trying to run any code, to exploring and experimenting.

People who are doing the most work in this area are focused on using their 1.0 PSP's, or trying to acquire one.

I suspect some folx are hard at work on 1.5, but there is no evidence that any crypto-analysis has helped in any way to run dev thus far or is likely to do so in the future. Direct analysis of the crypto is still resulting in 100% speculation after all of this time. Meanwhile, other are surging ahead with near 100% results.

Just a matter of where to spend one's energy.

[steddy]

I must admit I don't understand the attitude of 'we have an exploit for 1% of PSP's out there so lets not concentrate our efforts on getting the other 99%'. Surely the most important thing is to write apps that everyone can use.

If you read my posts, I do provide information that has not been posted else where and is directly related to the effort of getting code onto 1.5. This thread is actually just adding more meat to the file research thread for PSP files which didn't go into this level of detail. That thread has been added the the Interesting Threads sticky in development so its obviously considered a development issue for some. I didn't know it was there at the time, but maybe Admin could move the detail sections over too it or I could make a link in that post to this one?

I know we will not be able to sign our own code by decrypting the file, but it will enabled us to have a much better understanding of the attack vectors once we can debug ALL the ROM modules (which are currently encrypted).

In answer to MelGibson thanks again for pointing out a flaw in my logic, I really appreciate the feedback. I think that the value of B0-B3 is used differently between compressed and uncompressed files. This actually suggests to me that it is related to the number of comrpessed bytes and is used by the compression functions, not the encryption functions. If this is the case then Sony would ignore it for uncompressed files and just set it to the file size.

Given my numbered list of points on how the file is reassembled to make the executable, it is most likely the number of compressed bytes to feed into the algorithm (step 5). Since encryption functions require 16 byte alignment, 0x2C-0x2F records the encrypted size for decryption and possibly signing.

Once the data is decrypted though, the decompression function needs to know how many bytes need decompressing. This figure is different since it doesn't include the headers and doesn't include the padding. If however no decompression is required then this information is not required and B0-B3 can contain anything.

I would really like to know if the signature includes all the headers too. Could somebody with a 1.0 firmware take a BIN file that is signed and modify the FILENAME portion of the header and see if it reports a signing problem?

Steddy

[Guest]

I must admit I don't understand the attitude of 'we have an exploit for 1% of PSP's out there so lets not concentrate our efforts on getting the other 99%'. Surely the most important thing is to write apps that everyone can use.

The attitude is actually more like "crypto analysis is 99% wasted effort, look for ways in that don't require PhDs and thousands of CPU years".

But don't let that stop you. Everyone gets their enjoyment from different aspects of poking at a new console. :)

[MDave]

If we're going to avoid the encrytion stuff, then the only other methods I can think of are:

• We find a way to flash 1.5 psp's to 1.0 psp's using a hardware based method like serial port flashing.
• Same as above, but with a modchip that flashes the firmware to 1.0, or routes the psp to load the firmware from the modchip instead of the onboard flashchip.
• A homebrew application that runs on 1.0 that uses the GameShare function to send homebrew applications to a psp 1.5 firmware. However I feel that a ISO of Namco Museum is required to find out the libs used for the gameshare function. Even then the 1.5 might not accept the incoming unsigned, unencrypted app. But did Sony think that far ahead to make checks for stuff like that?.

Some of that stuff borders on the 'bad' side of things though, and thats whats going to stop people poking around in those areas :( with good reason though ...

[Guest]

You forgot buffer overflows, or other hidden things. Basically stuff even so vague would have higher chances of success, and still legit ways to go about it. :)

But hey, you don't need to listen to me if you want to keep focusing on crypto. ;)

[MDave]

I want to avoid cryto :P I think its going to be next to impossible to decrypt it.

About buffer overflows, I'm pretty sure sony have made sure stuff like that is hardly ever going to happen, with that psp shutdown watchdog and other tricks they might have to stop such things. If 1.5 will only read encrypted code, then won't code thats used on a overflow exploit, need to be encrypted too? That 8mb of resereved RAM for the XMB/Media Engine/OS might be a barrier to overcome.

On another note: something I've come across during my experiments:

I've packed a data.psp from the GBEmu eboot.pbp and a data.psar from the 1.51 update into one eboot.pbp. The strange thing is it dosn't behave the exact same as when you try to run 1.00 homebrew eboot.pbp's on 1.50 psp's. 1: It stays on the PSP white screen a longer time then usual, and the MS is accessed a lot. 2: The PSP logo fades to the black screen instead of instantly going to it and showing an error. It shows the same error number after its finished this little seqence though. Thought it was worth mentioning ...

[steddy]

gorim / MDave

I think you may be missing the point of what I am trying to do. I don't think the PSP crypto will be difficult to reverse, we know the PSP can do it because it does it every time you load a file.

Crypto nowadays is never based on ignorance of the alogorithm used, it is based upon known algorithms and ignorance of the key used. In the case of the PSP its security is in the digital signatures, not the encryption of the files.

Even unencrypted files will require a digial signature and without the private key it is IMPOSSIBLE to sign a file that looks like it comes from Sony. This is because the signature is a one-way hash function (likely SHA1) encrypted with the private key. The PSP will contain the public key, but there is no way to derive the private key from it.

However, the encryption key MUST be stored in the file itself and any keys that are required to decode it MUST reside on the PSP. If this wasn't the case then the PSP wouldn't be able to load it. Its a simple case of understanding the file format in order to get the key. Each file will have a different key and each will have to be decoded separately.

Once we have the file format we will be able to unencrypt all the ROM files and look at the code of how the encryption / compression works. This will NOT let us break the signing requirements directly. It will allow us to look for exploits such as buffer overflow attacks that can be ran as gorim as stated will be the main way to run code on it.

Steddy

[pixel]

Heh... ? The AES alone (which is symetric) isn't used to sign anything, that wouldn't make sense. The bios contains RSA certificates, which is an asymetric crypto/signature system, and that does make much more sense...

Now, good luck with RSA.

[steddy]

I didnt say it did Pixel. The AES is the encryption of the file and the digital signatures.

Signatures are always created in a Public/Private key crypto and the PSP will be using RSA. Sorry if my post read differently.

Steddy

[steddy]

Why on the main threads list does it say the last post to this Thread was the 16th May by MelGibson when it clearly isnt? This is when I am looking int he Development discussion.

I never recevied an answer on why this was moved to General when it is clearly Dev.

Steddy

[Guest]

You received somewhat of an answer from me up above, even if you disagree.

But this is clearly "exploit discussion" related, and we now have a forum for that. Off it goes...

[MindWall]

steddy: in the PSP we have hardware AES only, so how do you think the RSA works?

something like...
AES public key off to PSP-os, use RSA key (from hardware?) to decrypt AES public in OS, then send it to the hardware decrypter with encrypted blocks?

how much slower would this be? and don't you think this is very insecure, as this activity is done not in the hardware decrypter (but in the memory) and somehow may be.. "sniffed" and cracked? Isn't this way too much of a security risk?

[steddy]

The PSP uses the RSA BSAFE API which includes implementations of RSA public key crypto algorithms.

I think that all encrypted files are encrypted using AES or even something simpler like a combination of SHA1 and Xor (as used in the encryption of DATA2.BIN mentioned in the following article).

http://forums.ps2dev.org/viewtopic.php?t=1795

If you read my posts higher up, you will see that I think that the RSA will only be used for signing the file, not encrypting it. The file when decrypted will have a digital signature, which is a hash encrypted with a Sony private key. The PSP includes a few RSA public keys which are located in Flash0\data\cert in the main ROM. Checking the signature is trivial since it only requires 20 bytes to be decoded (for SHA1).

The security in the system comes from the fact we can never sign our own code without Sony's private key. If we crack the encryption Sony will not have lost security of the platform, they will have lost the data hiding facilities that encryption provides.

RSA will only be used for executable and modules though, not SAVEDATA. SAVEDATA must be using a system similar to DATA2.BIN since the Private key could never reside on the PSP (thats why Devs need to send code to Sony).

Steddy

[mrbrown]

SHA1 is not an encryption method, it is a hashing algorithm. You are not encrypting anything when you use SHA1 - instead you are generating a unique hash that can be used to determine if the original data has been tampered with.

[pedroleite]

He could be referring to the SHA1CipherStream from the decoded JAR file on the wipeoutpure download site.

This uses XOR to cipher data with SHA1 used on previous data for the XOR counterpart.

[steddy]

Yes Mr Brown, this is what I am referrring to (if this is illegal then I apologize and please remove):

Code: Select all

    public SHA1CipherStream(byte seed[])
    {
        this(seed, INTERNAL_KEY);
    }

    public SHA1CipherStream(byte seed[], byte internalKey[])
    {
        this.seed = seed;
        this.internalKey = internalKey;
        digest = Util.getDigest();
        if(digest == null)
        {
            throw new NullPointerException("SHA-1 not set");
        } else
        {
            digest.update(this.seed);
            digest.update(this.internalKey);
            buffer = digest.digest();
            bufferOffset = 0;
            return;
        }
    }

    public void xor(byte buffer[])
    {
        xor(buffer, 0, buffer.length);
    }

    public void xor(byte buffer[], int offset, int length)
    {
        for&#40;int i = 0; i < length; i++&#41;
            buffer&#91;offset + i&#93; ^= &#40;byte&#41;&#40;read&#40;&#41; & 0xff&#41;;

    &#125;

    public int read&#40;&#41;
    &#123;
        if&#40;bufferOffset >= buffer.length&#41;
        &#123;
            digest.reset&#40;&#41;;
            digest.update&#40;buffer&#41;;
            digest.update&#40;internalKey&#41;;
            buffer = digest.digest&#40;&#41;;
            bufferOffset %= buffer.length;
        &#125;
        int value = buffer&#91;bufferOffset&#93;;
        bufferOffset++;
        return value;
    &#125;

It seeds SHA1 with a code stored in the file, then XOR's all the bytes into the algo. This is how DATA2.BIN is encrypted.

If anyone is interested I will post the Java code to decrypt and write DATA2.BIN in the clear.


Steddy

[oneman]

Very interested ;]