Mysterious problems..

[skeezixcodejedi]

You know, despite having done a lot of embedded systems hacking, working on the bizarre ARM/pseudo-68k Palm OS 5 systems and other strage environments, the PSP is most mystifying sometimes :) Most of the time its very easy going, but a few times I've just lost hours or days screwing with weird stuff I'm sure is obvious to someone ;)

First, a question..

(Assuming all of this isn't caused by stack corruption or buffer overruns, of course; most of this code runs on a half dozen other platforms so I'm relatively certain its kocher.)

When is malloc() supposed to be used, versus sceGuGetMemory()?

For instance, I'm finding a lot of my code won't work with malloc, but works fine with sceGuGetMemory. It would seem sceGuGetMemory should be for RAM needed right on the Gu such as the vertice list for displays or the like.. but I've got linked list code for handling floppy disk emulation that works fine with sceGuGetMemory returns, but not with malloc. (Still, other system work fine on malloc or sceGuGetMemory.) Just using #define to map all malloc calls to sceGuGetMemory seems to work fine, though I'm sure it _can't_ be good :)

Now, some examples of mystifying behaviour..

1) msabuff test

extern unsigned char *msabuff;
extern unsigned char *disc[2];
msabuff = /*sceGuGetMemory*/malloc ( 1050L*1024L );
disc [ 0 ] = /*sceGuGetMemory*/malloc ( 1050L*1024L );
disc [ 1 ] = /*sceGuGetMemory*/malloc ( 1050L*1024L );
if ( ! msabuff || ! disc [ 0 ] || ! disc [ 1 ] ) {
cj_psp_die();
}

//int i;
for (i=0;i<1050L*1024L;i++) msabuff='\0';
//for (i=0;i<1050L*1024L;i++) disc[0]='\0';
//for (i=0;i<1050L*1024L;i++) disc[1]='\0';

Notice that whats really going on here is the for-loop at the end that fills msabuff with '\0'; this is just for debugging sake.

This always crashes, using either malloc or sceGuGetMemory. There is the check for NULL there that is passing, so its gotten _something_ back from either function.. but writing to it fails.

Whats extra weird is that uncommenting one of the other guys is fine -- the disc[0] and disc[1] loops work fine. They're allocated after msabuff even.. so one would think if msabuff is getting back a crap buffer, the others would as well.

They're defined externally:

unsigned char *msabuff = NULL;
unsigned char *disc[2];

2) Blitting too quick?

I'm using the following blit function for debugging; I've got variations of it that use pre-declared vertice lists and other stuff. Works _great_ in the emu with this dumb one, or the slightly more elaborate ones.. this gets called 50-70fps and works fine...

But if I call it twice in a row, the device crashes:
cj_psp_2d_blit ( pixels );
cj_psp_2d_blit ( pixels );
cj_psp_2d_blit ( pixels );

Naturally, I don't need to call it twice in a row.. but I do need to draw a menu, have the user hit a button, and then draw another menu. Sometimes it hangs, sometimes it doesn't.. and I narrowed it down to that blitter. If I comment it out, the menus work fine (invisibly, but I can hear them and see the proper memstick access).

I'm not sure what criterion makes it work.. right now I'm limited to one menu (the disk picker in CaSTaway) since that works.. but if I dismiss the menu and then redraw it, the device hangs. If I let the user operate it (redrawing each button press) it works fine.. but I can't bring it up again too quickly.

Wha sort of criterion need I wait for? Do you need to wait for a vblank or anything before starting a new draw?

Or perhaps memory gotten with sceGuGetMemory goes away on its own after awhile or as needed if the OS or Gu get low?

void cj_psp_2d_blit ( unsigned short int *pixels ) {
unsigned int j;

// begin logging GPU commands to our command buffer
sceGuStart ( GU_DIRECT, gpu_list );

// setup the source buffer as a 512x512 texture, but only copy 480x272
sceGuTexMode ( GU_PSM_4444, 0, 0, 0 );
sceGuTexImage ( 0, 512, 512, 512, pixels );
sceGuTexFunc ( GU_TFX_REPLACE, GU_TCC_RGB );
sceGuTexFilter ( GU_NEAREST, GU_NEAREST );
sceGuTexScale ( 1.0f/512.0f, 1.0f/512.0f ); // scale UVs to 0..1
sceGuTexOffset ( 0.0f, 0.0f );
sceGuAmbientColor ( 0xffffffff );

// do a striped blit (takes the page-cache into account)
unsigned char stripe = 0;
for ( j = 0; j < 480; j = j+SLICE_SIZE ) {
//vertex_t *vertices = g_vertices + ( 2 * stripe );
vertex_t *vertices = (vertex_t*)sceGuGetMemory(2 * sizeof(vertex_t));

vertices[0].u = j; vertices[0].v = 0;
vertices[0].color = 0;
vertices[0].x = j; vertices[0].y = 0; vertices[0].z = 0;
vertices[1].u = j+SLICE_SIZE; vertices[1].v = 272;
vertices[1].color = 0;
vertices[1].x = j+SLICE_SIZE; vertices[1].y = 272; vertices[1].z = 0;

sceGuDrawArray ( GU_SPRITES,
GU_TEXTURE_16BIT | GU_COLOR_4444 | GU_VERTEX_16BIT |
GU_TRANSFORM_2D, 2, 0, vertices );

sceKernelDcacheWritebackInvalidateAll();
//sceKernelDcacheWritebackAll();

stripe++;
} // for

// wrap up the rendering pipe
sceGuFinish();
sceGuSync(0,0);

// sceDisplayWaitVblankStart();
sceGuSwapBuffers();

return;
}


Sorry for the long post, but this sort of thing is driving me completely mental; the other posting I had where I neededd to invalidate the cache to make things happy.. I knew it was something like that, just couldn't put my finger on it. But this stuff is just bizarre.. when malloc returns a value, but the device hangs when writing into it? Gah!

(Sorry, I'm just bitter today after having wasted 4 hours on these :)

Thanks for any suggestions you can provide!

jeff

[holger]

sceGuGetMemory() allocates memory inside the command buffer, by advancing the pointer and inserting a JUMP command to let the GE jump over this embedded area. Useful if you have a few vertices and don't want to care about when to free them after the GE has done it's job. malloc()'ed memory would have to get free()'d here.


(1) may have many reasons, check the memory layout after linking (use e.g. psp-objdump). One reason may be cached/uncached aliasing, be sure to keep a minimum distance of 64 bytes (one cache line) between variables accessed cached and uncached.

(2) Maybe you want to use __pspgl_dlist_dump(), __pspgl_ge_register_dump() (both in pspgl_mish.[hc]) and pspgl/tools/decode_ge_dump to find out what's happening here. Insert the display list dump function before you submit the command buffer to the GE and add a register dump thenafter. You're probably overwriting part of the command buffer, maybe for the same reason as (1), or you are writing behind the end of the command buffer allocation. Please check what variables are alllocated by the linker before and after the command buffer (again using objdump).

Have you ensured that your command buffer is big enough to load all the commands and vertices you are inserting? Maybe you should put sceGuFinish()/Sync inside the for() loop. Or take a dynamic approach like you can see in pspgl_dlist.[ch] - then you don't need to care about these things at all.

[skeezixcodejedi]

So if you're using sceGuGetMemory, you do not need to free the returned buffer? At what point does it become free'd on its own? ie: If you perform a GuFinish(), does it kill everything allocated with sceGuGetMemory? (that would explain much)

It is curious why malloc and sceGuGetMemory 'work' at different times; ie:

If I use a #define to change all the allocations in the app to malloc(), nothing works at all (ie: the app batrely begins. I'll have to debug into it and find out where she dies.) If I change everything to sceGuGetMemory, the app works, but there is some flakiness.

Perhaps I'll start growing it from smaller parts again and refactor it, to try and see where malloc stops working. I'll try the for-loops in a small tiny file and see if they crash.

cache aliasing though? I should't think a cache alias would break writing into a just malloc'd buffer. How can I avoid that? Fairly brutal a platform if you can't trust a block of buffers alloc'd with malloc :)

Perhaps I'll just change them to global arrays and be done-with :P

jeff

[skeezixcodejedi]

Edit: removing the post for now.

Will the compiler and linkage properly support large globals?

Changing the alloc's over to this:

unsigned char msabuff[1050L*1024L];
unsigned char disc[2][1050L*1024L];

Device still hung, but I promptly ran out of power so I'll have to wait a bit to look into that further. Perhaps the large arrays get stripped out at some point, or theres more to it.

Perhaps time to start small and rebuild up again to test more.

jeff

[holger]

Take a look in the implementation of sceGuGetMemory() to see what's happening: http://svn.ps2dev.org/filedetails.php?r ... rev=0&sc=0

Nothing gets allocated, you just get back a chuck of memory inside your command buffer. Replacing all occurences of malloc() by sceGuGetMemory() does not makes much sense.

This function is really not intended for general use.

[skeezixcodejedi]

Take a look in the implementation of sceGuGetMemory() to see what's happening: http://svn.ps2dev.org/filedetails.php?r ... rev=0&sc=0

Nothing gets allocated, you just get back a chuck of memory inside your command buffer. Replacing all occurences of malloc() by sceGuGetMemory() does not makes much sense.

This function is really not intended for general use.

*g* But it works ;) Changing all occurances of sceGuGetMemory to malloc stops the app from working at all. I'll have to see if the malloc() source is there as well and see wat its doing.. perhaps it cannot handle 1MB allocs or the like, though I doubt that could be a problem.

Thanks for the thoughts,

jeff

[holger]

check the number of entries in your command buffer (gpu_list) and see if it's overflowing.

Dump the content of the gu_list->current variable (part of libgu) and see if it's smaller than the size of your gpu_list buffer (measured in 32-bit words).

[skeezixcodejedi]

What about these two questions..

1) When does sceGuGetMemory() allocations go away? ie: As soon as you do a GuFinish, do they get potentially invalidated? Or just "sometime in the future" as garbage collection or the like?

2) Why would sceGuGetMemory() work and malloc() not work?

ie: To try and debuf some of these mysterios problems, I've copied some of the code to a simple test project (essentially the blit sample from the SDK, organized nicer.)

So I copied the UI code that shows a menu; works fine, with its allocations using sceGuGetMemory. Changing them to malloc and it hangs.

This is a pretty small project.. the vertice list is like in blit sample.. 2 vertices per stripe, and only some 10 or 12 stripes to the image.. so nothing that would really hit the GU memory; of course, using GuGetMemory for lots of things is going to hit it.. but converting them to malloc hangs the project. It works pretty well using GuGetMemory.

Bizarre :/

Course, the test project also hangs when I draw the menu twice in a row, but that could be because its using sceGuGetMemory and the memory has been garbage collected by the Gu.... but of course, it doesn't work at all with malloc.

Fun :) :/

jeff

[holger]

What about these two questions..

1) When does sceGuGetMemory() allocations go away? ie: As soon as you do a GuFinish, do they get potentially invalidated? Or just "sometime in the future" as garbage collection or the like?

It is valid as long the pointer you passed in sceGuStart() is valid, and as long you did called sceGuStart() again, there the "gu_list->current" pointer is getting resetted.

2) Why would sceGuGetMemory() work and malloc() not work?

ie: To try and debuf some of these mysterios problems, I've copied some of the code to a simple test project (essentially the blit sample from the SDK, organized nicer.)

So I copied the UI code that shows a menu; works fine, with its allocations using sceGuGetMemory. Changing them to malloc and it hangs.

This is a pretty small project.. the vertice list is like in blit sample.. 2 vertices per stripe, and only some 10 or 12 stripes to the image.. so nothing that would really hit the GU memory; of course, using GuGetMemory for lots of things is going to hit it.. but converting them to malloc hangs the project. It works pretty well using GuGetMemory.

Bizarre :/

Course, the test project also hangs when I draw the menu twice in a row, but that could be because its using sceGuGetMemory and the memory has been garbage collected by the Gu.... but of course, it doesn't work at all with malloc.

Fun :) :/

jeff

What are you trying to debug? The malloc() allocator or your GU blit code? Keep in mind that the memory used by the GU is the array you passed in sceGuStart(), if you overflow this buffer by issuing too many GU commands, you'll overwrite your own data memory right behind this array.

[skeezixcodejedi]

Alrighty, okay, so sceGuGetMemory allocates out of the pointer you passed into GuStart - gotcha. (I should've caught that from the source to sceGuGetMemory.) So I'm definately blowing way past the end of that, which could explain much.

Theres two main problems (as above), that I can summarize..

(1) Sometimes I can malloc() some memory, and get a pointer back.. but when writing into the allocaed buffer, the device hangs. You mentioned cache aliasing and the like, so I'll look into that.. but I cannot imagine that would be it, since no one would code for a platform where malloc cannot be trusted. If you have to guard youre accesses to malloc'd memory, it would kill performance and would make code ports painful, so I also doubt the need of that.. so it simply _must_ be a result of buffer overun due to the above.

(2) Rendering twice in a row is crashing the device; could also be to buffer overrun above, or something else.

Whats annoying me is that I can use sceGuGetMemory all day long and its ifne, but malloc isn't working for me except in small projects. As soon as I carry over some code I want to use, it gets all buggered :) (And the code is all interdependant and means bringing over some 10,000 lines of code across a dozen files, with lots of malloc's.)


Now, if it is true that one cannot always write to buffers alloc'd with malloc, I'll have to just go kill myself :)

jeff

[skeezixcodejedi]

Alright.. narrowing in some more. After adjusting a few things around due to some of this discussion (order of operations), I can get much of the project to work .. excepting the menu code, which depends on malloc (but I nolonger think its a problem with malloc.) I'll trace the size of the GU list needed and some other stuff and narrow in more.

With luck I'll find something very soon; so far getting any 2 of 5 pieces to work is annoying, but now I'm up to 4 out of 5 pieced working.. just need the last piece and this whole last week will have been worth it :)

Thanks for bouncing ideas back and forth holger,

jeff

[skeezixcodejedi]

Hmm.. alright, I think I may have narrowed it down to some endian conversion issue in a TGA (targe) decoder, thats improperly detecting the presence of an alpha layer when there isn't one.. so its over-writing some goodies and botching up the menu rendering somewhere. (If I force it to no alpha layer, everything works _perfect_).

So I think thats it!

(for now ;)

Most of the problems were caused by my lack of understanding of where sceGuGetMemory was getting memory from, and from an obscure bug which seems to only show up on the PSP out of the half dozen platforms this code works on.. and thats something I can easily fix in a bit.

Many thanks holger!

I'll have to blog up my experience, and maybe let out some of the sample apps I've whipped up.. maybe help some other PSP newbies out :P

jeff

[skeezixcodejedi]

aha!

memset.

Some of the platforms I work on use a backwards memset..

MemSet ( buf, size, ptr ) versus
memset ( buf, ptr, size )

Damn thing was using 'memset' instead of 'MemSet', so the size was 0 ('\0') and thus the alpha layer wasn't zero, it was random.

I'll check the other memsets to be safe.

Sonovabitch!

jeff

[skeezixcodejedi]

Ah, one more thing that could help others..

Presumably, things 'worked' because sceGuGetMemory zeros out the memory it returns, or it so happened to be zero'd out. malloc() doesn't normally zero out the buffer, and so I ran into issues.

Goddamn :P

jeff

[chp]

sceGuGetMemory() zeroes nothing. All it does is return a non-cached address.

[Jim]

malloc() doesn't normally zero out the buffer, and so I ran into issues

calloc() does that. I think a quick brush-up on the C library might help you get round most of your bugs.

Jim

[skeezixcodejedi]

I know the C lib like the back of my hand, thanks ;) Up a bit I mentioned the problem was that in swpping between platforms, one lib I was using had the args to memset backwards and so after the malloc it wasn't clearing anything . (Note that on Palm OS, memset has its arguments backwards for some reason.. so the size and character are swapped.. so setting '\0' to a buffer turns into a zero size unless you swap the args.) Normally my Palm OS code has a wrapper for memset so that it adjusts on a per-platform basis, but in this case the wrapper wasn't used and so it led down to the dark side.)

(Note that some platforms do zero out memory returned with malloc, as a security precaution; back in the day folks used to malloc as big a chunk as we could get, then scan throgh the 'garbage' for interesting words. Windows was notorious here, as this alone coudl net you all sorts of goodies, as could opening the swapfile and surfing through it.)

jeff