New PSPGL tree w/ lots of fixes

Discuss the development of new homebrew software, tools and libraries.

Moderators: cheriff, TyRaNiD

Post Reply
jsgf
Posts: 254
Joined: Tue Jul 12, 2005 11:02 am
Contact:

New PSPGL tree w/ lots of fixes

Post by jsgf »

Hi all,

I have a whole pile of bug fixes and improvements to PSPGL. I've been talking these over with Holger, but we can't come to an agreement about how these should be applied to the SVN PSPGL tree.

So, I've got my own tree. I'm tracking Holger's tree, but extending it a lot. Over time, maybe they'll converge; I'm certainly following all the VFPU work with interest...

A quick summary of fixes and improvements:
  • Texturing works, supporting a wide range of formats
  • Compressed and paletted textures
  • Vertex arrays, with vertex format conversion
  • Compiled vertex arrays
  • Vertex array buffers
  • Depth testing works
  • No need to muck about with dcache flushes, etc
  • lots more...
See http://www.goop.org/psp/gl/ for all the details, code downloads, etc.
webjeff
Posts: 66
Joined: Thu May 05, 2005 2:51 am

Post by webjeff »

I have been using Holger's version of pspgl for a long time and always had intermittent problems. Especially when it came to caching issues. If you are really thinking about moving forward with pspgl, I'd really suggest getting jeremy's patches. Using his patches fixes all the problems I had and has been very stable.

Good job Jeremy, keep it up.

Jeff.
Warren
Posts: 175
Joined: Sat Jan 24, 2004 8:26 am
Location: San Diego, CA

Post by Warren »

What problems are there merging this with the pspgl in SVN?
memon
Posts: 63
Joined: Mon Oct 03, 2005 10:51 pm

Post by memon »

There seems to be a problem to download the zip (or tar.gz) files. I got some python key error.
holger
Posts: 204
Joined: Thu Aug 18, 2005 10:57 am

Post by holger »

What problems are there merging this with the pspgl in SVN?
I want to implement vertex and texture format conversion using dynamic code generation with a runtime assembler, that's why I'm concentrating on the VFPU codegen work.

Jeremy's work is inspired by Mesa's and SGI's approach (which is not a bad thing, it just leads to bigger code size and won't ever reach the performance of tight loops generated for this particular pixel/vertex format -- but it's easier to implement and straightforward to debug).

So Jeremy's tree is right now the better choice for all ones who are not familiar with the challenge of direct DMA-access of the GE to user buffers or don't want to convert their texture and vertex formats into native PSP formats.

Applying patches for texture or vertex array conversion right now would lead to conflicts soonish, too many internal structures are touched. So it's better to split into a "experimental" tree and a "conservative" one. Bug fixes are interchanged immediatly, conceptually they will eventually get merged as soon the code generation works reliably.
Last edited by holger on Wed Oct 05, 2005 7:19 am, edited 1 time in total.
jsgf
Posts: 254
Joined: Tue Jul 12, 2005 11:02 am
Contact:

Post by jsgf »

Warren wrote:What problems are there merging this with the pspgl in SVN?
Differences in approach, mainly. I won't speak for Holger, but here's my view:

I'm interested in incrementally improving PSPGL while trying to keep it useful and working at all times. Lots of stuff is missing, but the stuff that's there should work like "real" OpenGL, at least within the parameters of what we want to support. It should be easy to have the same source compile for a desktop system and on the PSP, as a useful development tool.

I'm also not particularly interested in micro-optimisations; I want the code to be high performance, but also readable and maintainable. It should be optimised in response to measuring real programs running. I expect that ultimately there'll be a fair amount of inline assembler, special cases, dynamically generated code and the like, but that's not much use while so much of the basic functionality is missing.

I want the code to be small and efficient, but code size isn't an overriding concern. At present all the .o's total about 55k; I don't know how much more this will grow, but I think 100k will be an absolute limit. Fortunately, a lot of this code will only be pulled in if you use the GL features it implements. The most important part of having small code size is making efficient use of the icache; this means putting all the hot code together and having short hot paths, but unused code doesn't matter so much.

There are also stylistic differences. For example, one of my early patches converts all the literal constants used for the command/register names into symbolic constants. I find this is essential to make the code readable, and it flushes out bugs on a regular basis. Also, my PSPGL is careful to put all its private symbols into a private namespace to prevent collisions with application symbols. Unfortunately this makes tracking the SVN version tricky because there are widespread collisions with my patches.
jsgf
Posts: 254
Joined: Tue Jul 12, 2005 11:02 am
Contact:

Post by jsgf »

memon wrote:There seems to be a problem to download the zip (or tar.gz) files. I got some python key error.
Oops. Fixed.
jsgf
Posts: 254
Joined: Tue Jul 12, 2005 11:02 am
Contact:

Post by jsgf »

holger wrote:I want to implement vertex and texture format conversion using dynamic code generation using a runtime assembler, that's why I'm concentrating on the VFPU codegen work.
This is good stuff. It should be fairly simple to replace the various inner loops where this makes a difference. This kind of dynamic codegen is all through Mesa and SGI's OpenGL, and it makes a good few percent difference.
Jeremy's work is inspired by Mesa's and SGI's approach...
Well, extensive use of dynamic codegen is a very SGI techique... ;)
So Jeremy's tree is right now the better choice for all ones who are not familiar with the challenge of direct DMA-access of the GE to user buffers or don't want to convert their texture and vertex formats into native PSP formats.
The buffer object API I've implemented allows you to have direct access to vertex arrays, while managing syncronization, caches, etc, using a standard API. If you use the API correctly, and format the vertices in hardware layout (which is just a matter of getting the right structure definition), there should be no performance difference from libGU.

Buffer objects can be easily used in an analogous way for textures, and many of the GL texture formats are 1:1 with PSP formats. Buffer objects can also be used to allocate new draw buffers, which can be easily used for render-to-texture without lots of context switching.

Internally I'm moving towards making buffer objects the single data structure for all arrays of data which are touched by hardware. This means that EDRAM management can happen in a unified way for textures, vertex arrays, render buffers, etc.
holger
Posts: 204
Joined: Thu Aug 18, 2005 10:57 am

Post by holger »

jsgf wrote:
holger wrote:I want to implement vertex and texture format conversion using dynamic code generation using a runtime assembler, that's why I'm concentrating on the VFPU codegen work.
This is good stuff. It should be fairly simple to replace the various inner loops where this makes a difference. This kind of dynamic codegen is all through Mesa and SGI's OpenGL, and it makes a good few percent difference.
not really... Keith played with some tcc-based code generation for Mesa, and it was discussed for a while on the mailing list, but it never made it into the main tree -- it would be even hard to incorporate elegantly since everything is designed for the function-pointer based pipeline setup.

The SGI reference implementation does not generates any code at runtime for portability. Both Mesa and OpenGL are hooking "extract_texel" function pointers into the pipeline and call this again and again. Same for vertex extraction...
jsgf
Posts: 254
Joined: Tue Jul 12, 2005 11:02 am
Contact:

Post by jsgf »

holger wrote:not really... Keith played with some tcc-based code generation for Mesa, and it was discussed for a while on the mailing list, but it never made it into the main tree -- it would be even hard to incorporate elegantly since everything is designed for the function-pointer based pipeline setup.
Perhaps you aren't looking at current versions. The current release Mesa (6.3.2) uses dynamic codegen in the tnl path (tnl/t_vertex_sse.c) and in specific drivers for vertex format conversion (r200/r200_vtxfmt_*.c). Keith put all that in place in May or so. More recently there was talk about doing this for vertex programs.
The SGI reference implementation does not generates any code at runtime for portability.
The SI may not, but their real implementations do.
Oobles
Site Admin
Posts: 347
Joined: Sat Jan 17, 2004 9:49 am
Location: Melbourne, Australia
Contact:

Post by Oobles »

One solution is to maintain a branch on SVN until things are ready to merge? However its up to you guys to sort out. Might be worth looking at what patches can be done without effecting Holger's ultimate goals. I assume these patches are already added.

David. aka Oobles.
holger
Posts: 204
Joined: Thu Aug 18, 2005 10:57 am

Post by holger »

Oobles wrote: One solution is to maintain a branch on SVN until things are ready to merge?
I would welcome both trees in SVN, whether branched or in two seperate directories is a matter of personal preference...
Oobles wrote: Might be worth looking at what patches can be done without effecting Holger's ultimate goals. I assume these patches are already added.
yes, there have been quite a lot commits in the past months, but I have to admit that I forgot to mention Jeremy's name in the commit log message in a few cases, but most patches should be easy distinguishable when browsing the SVN logs of the past two months.
Oobles
Site Admin
Posts: 347
Joined: Sat Jan 17, 2004 9:49 am
Location: Melbourne, Australia
Contact:

Post by Oobles »

I've setup SVN write access for Jeremy(jsgf). Will leave it to you guys to work out how best to progress.
holger
Posts: 204
Joined: Thu Aug 18, 2005 10:57 am

Post by holger »

:)
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Re: New PSPGL tree w/ lots of fixes

Post by Shine »

jsgf wrote:I have a whole pile of bug fixes and improvements to PSPGL. I've been talking these over with Holger, but we can't come to an agreement about how these should be applied to the SVN PSPGL tree.
I've tried to compile your version and the "test-glut" example doesn't work with the latest PSPSDK and toolchain:

Code: Select all

$ make && make -C test-glut clean install
make: Nothing to be done for `all'.
make: Entering directory `/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut'
rm -f *.d *.o glut-simple
psp-gcc -g -Wall -O2 -MD -I/usr/local/pspdev/psp/sdk/include -I.. -c simple.c
psp-gcc simple.o -g -Wall -O2 -DMODULE_NAME="test-glut" psp-setup.c -L/usr/local/pspdev/psp/sdk/lib -L.. -lglut -lGLU -lGL -lm -lpspdebug -lpspge -lpspdisplay -
lpspctrl -lpspsdk -lpsplibc -lpspuser -lpspkernel -o glut-simple
simple.o: In function `reshape':
/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut/simple.c:35: undefined reference to `__psp_log'
/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut/simple.c:34: undefined reference to `__psp_log'
/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut/simple.c:33: undefined reference to `__psp_log'
/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut/simple.c:32: undefined reference to `__psp_log'
/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut/simple.c:31: undefined reference to `__psp_log'
simple.o:/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut/simple.c:30: more undefined references to `__psp_log' follow
../libGL.a(pspgl_dlist.o): In function `now':
/home/Frank/pspgl-jsgf-6c71f61d23f0/pspgl_internal.h:226: undefined reference to `sceRtcGetCurrentTick'
/home/Frank/pspgl-jsgf-6c71f61d23f0/pspgl_internal.h:226: undefined reference to `sceRtcGetCurrentTick'
../libGL.a(pspgl_stats.o): In function `__pspgl_ticks_to_us':
/home/Frank/pspgl-jsgf-6c71f61d23f0/pspgl_stats.c:14: undefined reference to `sceRtcGetTickResolution'
../libGL.a(pspgl_vidmem.o): In function `now':
/home/Frank/pspgl-jsgf-6c71f61d23f0/pspgl_internal.h:226: undefined reference to `sceRtcGetCurrentTick'
../libGL.a(pspgl_vidmem.o): In function `__pspgl_vidmem_setup_write_and_display_buffer':
/home/Frank/pspgl-jsgf-6c71f61d23f0/pspgl_internal.h:226: undefined reference to `sceRtcGetCurrentTick'
/home/Frank/pspgl-jsgf-6c71f61d23f0/pspgl_internal.h:226: undefined reference to `sceRtcGetCurrentTick'
collect2: ld returned 1 exit status
make: *** [glut-simple] Error 1
make: Leaving directory `/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut'
test-egl compiles, but doesn't work at all on my 1.00 firmware PSP (looks like the same freeze bug, because you can still exit the game), but it works on a 1.50 firmware PSP.

The latest version from pspgl from SVN doesn't compile both projects. The following error occurs:

Code: Select all

 no MIPS32-ELF binary, not installing...
The reason may be this:

Code: Select all

$ file pspGL-cube
pspGL-cube: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), statically linked, not stripped
but in psp-install this string for testing the file type is included:

Code: Select all

IMGTYPE=`file "$IMAGE" | grep "ELF 32-bit LSB MIPS-II executable, MIPS"`
I know that the image type is ok, so after setting it to 'IMGTYPE="ok"' it compiles and both samples work for my 1.00 firmware version.

And now the good news: The freeze example from http://forums.ps2dev.org/viewtopic.php?t=3399 doesn't freeze any more with the SVN version! Now it's time for me to include it in Lua Player :-)
holger
Posts: 204
Joined: Thu Aug 18, 2005 10:57 am

Post by holger »

mmh... did you updated toolchain or your system's "file" utility? Can you please update your pspgl SVN tree, I made the detection a little more tolerant...
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Post by Shine »

holger wrote:mmh... did you updated toolchain or your system's "file" utility? Can you please update your pspgl SVN tree, I made the detection a little more tolerant...
I've updated both, because I needed the ccache tool for jsgf's version and the cygwin installer updated all packages automaticly. Your new detection works, but I wonder if it is not better just to test for "MIPS" and "ELF", with case ignoring "-i". As far as I understand the test is for avoiding to use the wrong compiler, so this should be sufficient. Of course, this wouldn't catch the case when someone compiles it with the native compiler on SGI, but perhaps the current test doesn't catch this, too.
holger
Posts: 204
Joined: Thu Aug 18, 2005 10:57 am

Post by holger »

the intention was only to install executables, libraries and the like usually should to to the pspsdk directory...

As far I know MIPS/SGI/IRIX is big-endian, is this right?
jsgf
Posts: 254
Joined: Tue Jul 12, 2005 11:02 am
Contact:

Re: New PSPGL tree w/ lots of fixes

Post by jsgf »

Shine wrote:

Code: Select all

$ make && make -C test-glut clean install
make: Nothing to be done for `all'.
make: Entering directory `/home/Frank/pspgl-jsgf-6c71f61d23f0/test-glut'
rm -f *.d *.o glut-simple
psp-gcc -g -Wall -O2 -MD -I/usr/local/pspdev/psp/sdk/include -I.. -c simple.c
psp-gcc simple.o -g -Wall -O2 -DMODULE_NAME="test-glut" psp-setup.c -L/usr/local/pspdev/psp/sdk/lib -L.. -lglut -lGLU -lGL -lm -lpspdebug -lpspge -lpspdisplay -
lpspctrl -lpspsdk -lpsplibc -lpspuser -lpspkernel -o glut-simple
There are two things wrong with this link line: It puts the installed libraries before -L.., and it needs -lpsprtc. I need to go through and fix all the test program Makefiles up.
And now the good news: The freeze example from http://forums.ps2dev.org/viewtopic.php?t=3399 doesn't freeze any more with the SVN version! Now it's time for me to include it in Lua Player :-)
Are you saying that this test program locks up on 1.0 with my PSPGL, but works on 1.0 with the SVN one? Mysterious... I don't think anything has happened in SVN which would make a difference to this.
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Re: New PSPGL tree w/ lots of fixes

Post by Shine »

jsgf wrote: Are you saying that this test program locks up on 1.0 with my PSPGL, but works on 1.0 with the SVN one?
Yes. Would be interesting to find out which difference caused the problem, to avoid it in future and to learn more about the hardware of the PSP.
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Post by Shine »

holger wrote:the intention was only to install executables, libraries and the like usually should to to the pspsdk directory...

As far I know MIPS/SGI/IRIX is big-endian, is this right?
Yes, you are right, so your current version is ok, I'll complain if it doesn't work again :-)
Post Reply