I'm working on Quake again, requesting some advice

PeterM · Post by **PeterM** » Thu Apr 05, 2007 2:01 am

Hey guys,

Sorry for the long post!

It's been a heck of a long time since Chris Swindle and I did a Quake release, but hopefully there are still some people out there enjoying it!

I've been spending a few hours lately on hardware rendering, but before I go down any dead ends, I wanted to check my assumptions with the GU gurus around here.

First off, the current state of Quake hardware rendering is this:

1. Based off GLQuake.
2. All GL calls have been commented out, and it doesn't crash. ;-)
3. Some GL calls replaced with GU equivalents, most things are rendered with random flat-shaded polys.
4. No texturing yet.
5. No clipping yet.

I plan on sticking with GLQuake's non-multitexture method of drawing the scene -- polys are generally sorted by texture and drawn in two passes, one for the texture maps, another for the light maps.

I need to clip the polys, since big ones (like walls, floors and large doors) often get culled by the hardware when they go outside the tiny guard band.

I'm a clipping noob, so any advice would be welcome!

Regarding textures, they are all 8-bit, and will either use a greyscale lightmap palette (unless the PSP has a luminance format?), or the standard Quake palette.

I would welcome any advice people have regarding texture management.

I can fit them all in RAM, but probably not in VRAM without quality loss. Polys are sorted by texture so VRAM thrashing hopefully isn't too much of an issue.

I'm planning on just doing a FIFO or LRU cache, where texture memory is split into chunks of some known size. Maybe 32x32 or 64x64 texels. Big textures would probably take up several blocks, and smaller textures would waste some space.

Swizzling can be done easily enough, but not until I verify that they're loaded and rendering okay.

Thanks for your time guys, sorry again for the long post.

Pete

rapso · Post by **rapso** » Thu Apr 05, 2007 2:59 am

my first advice would be to convert the diffuse textures (which are probably 8bit-palettet) to dxt1, this has 4bit/pixel whick should cut the memoryusage for this to half (and allow you to make better usage of the vram).
the psp has no luminanz-texture support, but if the lightmaps are just luminanz-textures you just need to create one simple palette with grayvalues as colorentries, straight forward.

Maybe you could check first the amount of texturespace needed for diffuse and for luminanz-textures/lightmaps. this could help in giving you advices how to manage them ;)

I think I would put mipmaps into vram, as they're mostly used and just for the higher resolution textures use mainmemory, simple memorymanagement as well :)

PeterM · Post by **PeterM** » Thu Apr 05, 2007 3:14 am

Hi rapso,

Thanks for the advice!

I'd not thought about using DXTC, I'll definitely look into that as it looks very useful indeed, especially given Quake's limited palette.

I agree that for lightmaps, 8 (or even 4-bit) textures may be the best approach. Maybe DXT1 could be a win here too.

You're right - I need to know how much memory they take up before I can manage them effectively. I guess what I'll do first is try to load them all into VRAM and see if I run out.

I'm currently using 32-bit display and draw buffers, but maybe these could be downgraded to 16-bit if needed and the quality difference isn't that big. Probably commercial games often use 16-bit too.

dot_blank · Post by **dot_blank** » Thu Apr 05, 2007 6:13 am

32bit back/draw buffer always is best with
16bit front/display buffer ...this works magic and really
makes video memory effecient

PeterM · Post by **PeterM** » Thu Apr 05, 2007 6:37 am

Wow, I never knew 2 different formats could be combined. How does double buffering work? Presumably the buffer pointers can't be swapped, since one of the buffers won't be big enough. Can the GE do a blit + dither?

Raphael · Post by **Raphael** » Thu Apr 05, 2007 8:29 am

With dithering a 16bit display really isn't a image quality concern. Plus it saves you a lot of VRAM to use for textures, hence gives double speed improvement.

For VRAM texture caching I found LFU to be superior to LRU in certain cases, especially in scan-through cases as the Quake render path might create, since LFU soon degenerates to a static cache that holds the most 'important' textures and therefore saves the upload bandwidth. But I didn't test it with that specific environment, so it might be worse than LRU actually.

Regarding clipping, just be sure to read up on Sutherland-Hodgeman algorithm and it should be easy to get it working.

rapso · Post by **rapso** » Thu Apr 05, 2007 10:16 am

Raphael wrote:With dithering a 16bit display really isn't a image quality concern. Plus it saves you a lot of VRAM to use for textures, hence gives double speed improvement.

Are you sure? I'd say the quality might be quite messy if using multipass rendering like he needs for lightmap+diffuse.

Regarding clipping, just be sure to read up on Sutherland-Hodgeman algorithm and it should be easy to get it working.

another solution might be to tesselate the geometry. actually I think the quake engine generates the triangles on the fly each frame, it could be faster to just split them into smaller ones than cutting them (and for ppl who never did proper clipping it's a way simpler as well ;) )

Raphael · Post by **Raphael** » Thu Apr 05, 2007 6:10 pm

rapso wrote:
Raphael wrote:With dithering a 16bit display really isn't a image quality concern. Plus it saves you a lot of VRAM to use for textures, hence gives double speed improvement.
Are you sure? I'd say the quality might be quite messy if using multipass rendering like he needs for lightmap+diffuse.
Well, Quake only does two texture stages and the result shouldn't look much worse. But that's left to trying actually and it's definitely worth a try. If full 16bit rendering is really too bad quality, dot_blank's idea of a mixed 32bit/16bit rendering could be applied.

Regarding clipping, just be sure to read up on Sutherland-Hodgeman algorithm and it should be easy to get it working.
another solution might be to tesselate the geometry. actually I think the quake engine generates the triangles on the fly each frame, it could be faster to just split them into smaller ones than cutting them (and for ppl who never did proper clipping it's a way simpler as well ;) )

Yeah, that's a viable alternative for sure :)

PeterM · Post by **PeterM** » Thu Apr 05, 2007 6:31 pm

Thanks guys,

I was thinking about tessellation too, but I think McZonk had a problem doing that in Quake 2. IIRC he found that tessellating the polys to even small ones didn't quite solve the clipping issues, and generated a lot more geometry to draw.

I added some basic texture loading and management. I'm using 8-bit textures currently, and still a 32-bit framebuffer.

The frame buffers, depth buffer and textures take about 3MB, probably more for larger levels, but it looks like most of them will fit in VRAM with some tweaking.

Currently my approach allocates from fixed-size VRAM chunks, until there's none left, then allocates from system RAM using memalign(). It seems to work okay. I'll look into a VRAM cache later, but it may not be required if most textures fit in VRAM anyway.

GLQuake doesn't seem to free unused textures anywhere (unless I've missed it), so I'll need to find a good place to do that.

Thanks again for your advice.

P.S. Here are some "before and after" screenshots I made while working on texturing.

Textures seem really damn dark - I'll look into making them brighter.

noxa · Post by **noxa** » Tue Apr 10, 2007 10:39 am

Sweet! What will your release policy be on this? Will you be posting up the code? I'd love to see how you handled all the texturing issues you've been discussing.

J.F. · Post by **J.F.** » Tue Apr 10, 2007 2:54 pm

noxa wrote:Sweet! What will your release policy be on this? Will you be posting up the code? I'd love to see how you handled all the texturing issues you've been discussing.

Uh, there's no asking IF he'll be posting the code... Quake is under the GPL license. Anyone making changes is required to post the code unless they never distribute binaries based on the changes. So you keep it completely to yourself, or you are required to post the code.

PeterM · Post by **PeterM** » Tue Apr 10, 2007 5:39 pm

Indeed. When the binary is released, the source will be available too.

noxa · Post by **noxa** » Tue Apr 10, 2007 7:57 pm

Well, I was looking for the source of the Super Mario War port the other day and couldn't find it - that's GPL. Unfortunately the GPL is not law, and even if it were most hobbyist developers would still ignore it. It's good to know that some people will follow it, but in general it's a pretty sucky situation. Thanks for clarifying your stance!

rapso · Post by **rapso** » Tue Apr 10, 2007 8:08 pm

noxa wrote:Unfortunately the GPL is not law

if you break a licenses that you agreed, you break the law.

PeterM wrote:I was thinking about tessellation too, but I think McZonk had a problem doing that in Quake 2. IIRC he found that tessellating the polys to even small ones didn't quite solve the clipping issues, and generated a lot more geometry to draw.

just tesselating the tris that intersect with the frustum and enabling clipping of the psp solved that problem for me quite well and does not add much polys.

ufoz · Post by **ufoz** » Tue Apr 10, 2007 10:24 pm

J.F. wrote:
noxa wrote:Sweet! What will your release policy be on this? Will you be posting up the code? I'd love to see how you handled all the texturing issues you've been discussing.
Uh, there's no asking IF he'll be posting the code... Quake is under the GPL license. Anyone making changes is required to post the code unless they never distribute binaries based on the changes. So you keep it completely to yourself, or you are required to post the code.

Heh, tell that to the people who were working on porting Quake 2. They even released a bunch of binary-only demos, with only vague promises about the source ("eventually", "when it's done", etc.)
Well, they seem to have disappeared off the internet, guess that tells you something about the character of GPL violators.

PeterM · Post by **PeterM** » Tue Apr 10, 2007 10:33 pm

Well, just like our previous Quake releases, the source code of this one will be available for anybody to download. I've never violated any license and I don't intend to.

Anyway, to help steer this thread back on topic, I'd like to ask about storing vertex data in system RAM. Does anyone know if there's a noticeable performance drop caused by not having vertices stored in VRAM?

Currently I allocate from display list memory using sceGuGetMemory, but would like to clip and store polygons in system memory.

rapso · Post by **rapso** » Wed Apr 11, 2007 12:37 am

PeterM wrote:Anyway, to help steer this thread back on topic, I'd like to ask about storing vertex data in system RAM. Does anyone know if there's a noticeable performance drop caused by not having vertices stored in VRAM?

I think you're creating vertices dynamically anyway, so there shouldn't be any difference of ram/vram. I store all my vertices always in ram, never tried vram tho, but I've up to 15MVertices/s (8bytes per vertex), so that should be fast enough for Q2.

before you start optimizing, you should probably test what limits your performance.

1. dont do any drawing (just return immediatelly from all draw functions) to get the maximum possible framerate after all graphic optimizations
2. render, but set the scissorrect to 1pixel (this shows you how limited you are by the vertextransform and clippig
3. force all textures sizes to 4x4 pixel, should look like mess, but it shows you how good 'nice' your textures are for the psp.

if 2 is limiting, you've obvsiously to optimise the vertexformats
if 3 is limiting, that's really bad, as you'd be just fillrate limited, maybe switch to 16bit, but you cant do much about it, disabling z-compare (as quake should always draw back2front in correct order for the map) could reduce bandwitch
if non of those, you're probably just texturebandwitch limited, I guess u're not using mipmaps then, try to generate mipmaps, this would maybe boost ur framerate ;), additionally try it with swizzling. you dont have to really swizzle, just enable the flag and check how the framerate improves.

Insert_witty_name · Post by **Insert_witty_name** » Wed Apr 11, 2007 4:03 am

There's no performance difference between having vertices in RAM or VRAM.

PeterM · Post by **PeterM** » Wed Apr 11, 2007 6:33 am

Thanks. That saves some wasted time.

cloudhunter · Post by **cloudhunter** » Wed Apr 11, 2007 8:51 am

ufoz wrote:Well, they seem to have disappeared off the internet, guess that tells you something about the character of GPL violators.

... No they haven't, they still post on this very forum. As far as I'm aware, all what happened was that their domain expired...

Having said that, last I heard was that it wasn't going to be released just yet, if ever.

Anyhow PeterM, great work :)

Cloudy

PeterM · Post by **PeterM** » Wed Apr 11, 2007 9:07 am

Thanks.

I've since gotten in touch with Chris Swindle, and this code is now in Subversion. There's still lots to do obviously, but I guess the code's there if people want an early peek!

(Please don't let it appear on DCemu or QJ as an unofficial release or alpha...)

ataxy · Post by **ataxy** » Mon Apr 16, 2007 3:04 am

hi i have been hoping for an update of quake and of quake2 by mczonk but you are the first one that seem to show interest in updating there project, now i wish you good luck in this futur release but i would like to try out what you have done so far but i do not know how to compile file from a cvs, wich i think is what its called, so could anybody point me to a tutorial on how i could do it

PeterM · Post by **PeterM** » Mon Apr 16, 2007 6:05 am

Hi ataxy,

If you're brave, the rough steps you need to follow are:

Download and install cygwin (I assume you're on Windows)
http://www.cygwin.com/

Install the PSP toolchain install script
http://wiki.ps2dev.org/psp:toolchain

Run it
(see above link)

Wait for it to download and build the toolchain

Download the Quake source using svn (Look at our SourceForge SVN page).
https://sourceforge.net/svn/?group_id=158726

Compile the Quake source
cd into the Quake/psp dir and type "make"

Success. Make sure you read the readme.html before posting "bugs".

ataxy · Post by **ataxy** » Mon Apr 16, 2007 8:14 am

ok i got to this point

ATAXY@santa ~
$ cd psp-quake/quake/psp

ATAXY@santa ~/psp-quake/quake/psp
$ make
Makefile:221: warning: overriding commands for target `obj/HARDWARE/EBOOT.PBP'
/usr/local/pspdev/psp/sdk/lib/build.mak:178: warning: ignoring old commands for
target `obj/HARDWARE/EBOOT.PBP'
clipping.cpp
Assembler messages:
FATAL: can't create obj/HARDWARE/psp/clipping.o: No such file or directory
make: *** [obj/HARDWARE/psp/clipping.o] Error 1

ATAXY@santa ~/psp-quake/quake/psp

PeterM · Post by **PeterM** » Mon Apr 16, 2007 10:07 am

mkdir -p obj/HARDWARE/psp

ataxy · Post by **ataxy** » Mon Apr 16, 2007 11:30 am

thx a million it work, ill let you know of any bug i may notice that arnt allready explained in the read me and as you ask will not in any way post this as a release, unofficial, beta or wath ever title it could be given

on second note have you tought about looking into the quake2 port or your interrest are solely for the quake port

edit: ok i have tryed it out so far i must say i am trully impress in relation to the previous release texture are way cleaner and less pixelated but and this is the only one so far as i am not expiriencing any bug or slow down it seems like the texture are a bit to bright but aside from that absolutly no critic so far

Be3f · Post by **Be3f** » Mon Apr 16, 2007 6:43 pm

PeterM - thx for the subversion, I'm going to compile it!

on second note have you tought about looking into the quake2 port or your interrest are solely for the quake port

I've spoken to McZonk about 3 weeks ago and he said, that he is going to reliase the new much improved version of Q2 port someday in future...

PeterM · Post by **PeterM** » Mon Apr 16, 2007 7:00 pm

ataxy wrote:it seems like the texture are a bit to bright but aside from that absolutly no critic so far

Light maps aren't in yet, which is probably what you're seeing.

PeterM · Post by **PeterM** » Mon Apr 16, 2007 7:47 pm

I've not really got time to work on anything else at the moment!

Be3f wrote:I've spoken to McZonk about 3 weeks ago and he said, that he is going to reliase the new much improved version of it someday in future...

That's cool news. Q2 is probably my favourite over Q1, but Q1 is a heck of a lot easier to port to PSP.

Be3f · Post by **Be3f** » Mon Apr 16, 2007 11:03 pm

I'm playing the alpha right now and it is amazing!

Perfect graphics in 480x272 with bilinear texture filtering and proper polygon clipping, perfect underwater polygons rendering and awesome perfomance (slowdowns only in heavy-geometry scenes)...

Only particles dissapoint - they have incorrect (always black) color and slowdown framerate much.

Lightmapping will make the graphics much more eyecandy... :rolleyes:

I've found, that there is no V-Sync and no mipmapping in this build - Pete, try to test Quake with these things! With mipmapping perfomance will be higher, and textures that are far from the camera will be not too sharp (but, maybe, too smooth). I hope, mipmaps for 32x32 (mostly) textures will not eat too much memory...

Huge thx & GL!