Let's say I have a circular mask and a rectangle as shown in the picture above. I'm looking for a blending mode and pixel test mode that would allow me to render both the mask and the rectangle so that only the area of the rectangle covered by the mask is shown. I'm fairly new at this, so I would appreciate any suggestions.
You could render the MASK into the alpha channel then render the rectange using dest alpha test. I am not sure if this is supported within any of the homebrew libs, but it shouldn't been too hard to do.
Draw order becomes an issue when you do this as to get the correct result you should really start with a blank alpha channel, write the MASK in then the rectange. If there is stuff in the alpha channel already you may have to clear it, which in itself may provide its own issues.
