PS3SDK collaboration invitation

mc · Post by mc » Sat Jul 04, 2009 3:17 am

Well, we've all heard the rumors about ongoing PS3SDK development efforts,
but so far nobody has shown any code.

I think the main problem with getting a PS3SDK out the door is that
it's a rather large project for one or two persons to undertake, so eventually
anyone who tries on his own will get stuck and/or fed up.

So what I'd like to do is invite to a collaborative effort, and the correct way to
do this is of course to present all the code I have so far, and allow people
to start expanding it.

What I have currently is a small multithreading kernel (supports use of both
HW threads), integration with newlib, and some simple GPU support.

There are plenty of work packages for interrested parties to get down with,
including (but not limited to)

* Interrupt handling
* A more sophisticated MMU driver
* Networking
* USB (the driver in mcload can't be used as it contains GPL code)
* Storage drivers and filesystems
* SPE support
* VMX support (currently context switching does not preserve vector registers)

To get going, check out the following bzr repository:

http://mc.pp.se/bzr/ps3sdk

Note that you will need to rebuild newlib with the patch contained in
the "misc" directory (simply copy the patch over the old one in the ps3toolchain
distribution).

If you want to start working on something, please let me know, and preferably
publish the URL of your own bzr branch, so that others can follow your progress,
and so that I can merge things back to my main branch.

Let's get hacking, people!

Post by **jbit** » Sat Jul 04, 2009 3:26 am

It would be nice if we can keep the core GPL clean, as in all code is BSD/AFPL/whatever licensed, which means of course no copying code/headers/etc from linux..
GPL causes a lot of headaches at later stages for a lot of people, so it's best to avoid for this kind of thing.

(Other peoples opinions are welcome of course :P)

mc · Post by mc » Sat Jul 04, 2009 3:37 am

@jbit: Yes, that's why I mentioned this issue with the existing USB driver.

I've reused the LICENSE from the PSPSDK.

Currently I have lv1callarg.h, lib1calltab.h and hv.S still copied from Linux for generating
the HV call stubs, so I guess another work package would be to cleanroom that.

Post by **jbit** » Sat Jul 04, 2009 3:43 am

Ah, I missed that sorry... But even so it makes sense to make people aware :)

Also I think we have enough info on the wiki about the lv1 interface to validly make a cleanroom BSD implementation. Should only be a days work or so.

mc · Post by mc » Sat Jul 04, 2009 3:51 am

Yup, then information in the wiki is more than adequate (you actually only need
<name, id, # of inputs, # of outputs> for each call). I've already started creating
a header file with prototypes for the HV calls based on the wiki (src/hypervisor/ps3hv.h).

Hm, one file I'm a little unsure about is the linkfile, which is based on the linker
script generated by binutils. It does not contain any license information,
so I don't know if it is GPL, LGPL, or something different.

J.F. · Post by **J.F.** » Sat Jul 04, 2009 7:21 am

It would also be helpful if the different parts (like usb or net) were separate and independent as much as possible. For example, let's say you made a PC or Mac emulator and just wanted to pass the hardware for the USB on to the emulator. If you FORCE people to load a USB driver, then you also FORCE people to emulate the hardware using the driver in some cases, like the one just mentioned. Now clearly an input driver would require a USB driver, but if you don't use anything that needs it, don't force people to load/use it anyway.

mc · Post by mc » Sat Jul 04, 2009 7:35 am

The approach so far is pretty modular. I don't even FORCE anyone to turn on the MMU,
although of course most drivers won't work without it... ;-)

J.F. · Post by **J.F.** » Sat Jul 04, 2009 9:32 am

mc wrote:The approach so far is pretty modular. I don't even FORCE anyone to turn on the MMU,
although of course most drivers won't work without it... ;-)

Good. I've done a checkout and will look it over to see what I can contribute.

ps2devman · Post by **ps2devman** » Mon Jul 06, 2009 4:04 pm

Strong advise : implement right from the beginning a strong 'DPC' mechanism for all interrupts. If you get strange and impossible to understand issues think about interrupt that fires while you are already handling the previous one. The direct interrupt handling routing must allow reentrance and be as fast as possible. Don't treat interrupt there, just enqueue what you need to treat it later. Then have something that dequeues sequentially events and call clean DPC's (deferred procedure calls) to treat them. The DPC's are normal routines and can be interrupted anytime like any other code part.

(can someone write the url to some page that explain how bzr work?
for me page is empty...)

mc · Post by mc » Mon Jul 06, 2009 9:07 pm

Yes, I was also thinking about this kind of solution for interrupt processing. I have experience from work with commercial microkernels which use this method, and it really makes stuff easier for the clients. For it to work properly, I think preemption needs to be added to the scheduler though (it doesn't have it yet).

For a quick bzr tutorial, see http://doc.bazaar-vcs.org/latest/en/min ... index.html. Basically, you just do "bzr branch htpp://mc.pp.se/bzr/ps3dev" to get your own branch that you can commit locally to.

nbauernf · Post by **nbauernf** » Tue Jul 07, 2009 4:36 am

Hmm I'm having trouble checking the branch out.

$ bzr branch http://mc.pp.se/bzr/ps3sdk
bzr: ERROR: Invalid http response for http://mc.pp.se/bzr/ps3sdk/: Unable to handle http code 501: Not Implemented

Any clues?

nbauernf · Post by **nbauernf** » Tue Jul 07, 2009 5:02 am

Turns out adding a '--no-plugins' took care of that problem!

gkr · Post by **gkr** » Thu Jul 09, 2009 6:47 am

Hi,
I get error message when trying to check out

c:\Python24\Scripts>bzr branch http://mc.pp.se/bzr/ps3sdk
bzr: ERROR: exceptions.NotImplementedError: We must have one of fcntl, pywin32,
or ctypes available to support OS locking.

mc · Post by mc » Thu Jul 09, 2009 7:23 am

@gkr: Did you follow the instructions here?

Or maybe you are using MS-DOS?

ps2devman · Post by **ps2devman** » Thu Jul 09, 2009 5:21 pm

If you install the windows version of bazaar with minimum installation
(no plugin) it definitely works without a fuss (at least on xp)

bazaar in 5 minutes :
http://doc.bazaar-vcs.org/bzr.dev/en/mi ... index.html

You don't even need bazaar in the path if you specify a path for bzr in a .bat file. Create a .bat file in a c:\bzr_projects directory with this inside :

c:\progra~1\bazaar\bzr whoami "myname <myemail>"
c:\progra~1\bazaar\bzr branch http://mc.pp.se/bzr/ps3sdk

and launch it

rapso · Post by **rapso** » Mon Jul 13, 2009 10:20 pm

A homebrew PS3SDK would be definitely incredibily cool :),
but from the text here it rather seems to be another OS. is there something I miss? (that's no offense. I just don't really see the goal of this project as the topic seems to promise something else than the content, imho).

rapso

J.F. · Post by **J.F.** » Tue Jul 14, 2009 12:57 am

OTHEROS means the program is running on bare metal with only the hypervisor below it. There are no libraries or anything else to help do stuff, so yes, it should seem like another OS when discussing the SDK - it needs to do everything the OS would do.

rapso · Post by **rapso** » Tue Jul 14, 2009 7:07 am

so it's mcload with BSD license?

for me, it wouldn't need to do all the stuff to get into development. flat memory (instead of an sophisticated MMU driver), no context switching (just a call into main(..) and main2(...) with two HW threads), video out and pad would be all needed from an OS to get into it, similar to PS2 dev.

I don't say this is bad, I even praise you guys for doing such a community work, I just want to say I (and maybe other guys) would be happy about far less, as long as it's simple to get running and with access to basic hardware to get (homebrew)games running.

Post by **emoon** » Tue Jul 14, 2009 7:48 pm

I will release the code I have worked on to mc.

It has a semi working Ethernet driver that can actually receive and start an elf (the code is very crude and needs lots of work though)

I don't have much time to work on it but hopefully it will give some base to work from at least.

Post by **emoon** » Tue Jul 14, 2009 8:34 pm

Ok, I actually forgot stuff I had implemented :)

* tcp/ip (semi working) and is using LWIP
* More cleaned up code
* echo server over tcp/ip (telenet style that echos back what you send it)
* lv1 calls renamed to lv1NetControl(...) instead of lv1_net_control(...) to try to remove the 'GPLisnes' of it and also some other changes to try to make it different from the linux code
* Totally custom written smb support (to be used to load files remote) It has been implemented by Chip and is way more lean than samba and focus on whats needed (never tested on ps3 afaik but code is clean and documented)
* Includes docs on the coding standard we intended to use
* Includes a early version of ps3link (its able to kick elfs, but not very tested and needs to be fixed up to be really useable)
* Also a early version of ps3client is included.

Hopefully you will find the work that I and Chip have done useful

mc · Post by mc » Wed Jul 15, 2009 6:46 am

emoon wrote:* lv1 calls renamed to lv1NetControl(...) instead of lv1_net_control(...) to try to remove the 'GPLisnes' of it

Actually, I'm not sure this is productive. You can't copyright an API, and I'm pretty sure these names come from the IBM/Sony HV code anyway, so I don't see any "GPLisnes" to remove as far as the actual names are concerned.

But if others have a different opinion, feel free to motivate. :)

mc · Post by mc » Wed Jul 15, 2009 6:58 am

@rapso:

If you don't want to use the MMU driver, that's fine. Just don't call ps3MmuInit () and ps3MmuOn(), and you'll get your flat memory model. Note however that due to the way hardware virtualization works in the Cell, the flat memory model contains 128MB of RAM _only_. No video memory, no USB register file, no nothing. So for things like the console driver and any upcoming USB driver, the fancy schmancy MMU stuff is _required_. It isn't there just for kicks.

Same with the scheduler; if you just create one new thread from main (your main2), and never call any functions which can deschedule (sleep, mutex operations etc), you will never get any context switches. But some divers may require them because it is impractical to make them otherwise,

rapso · Post by **rapso** » Wed Jul 15, 2009 7:52 pm

thx for clarifying that MC, I wasn't aware of all those implications that makes all those tasks u listed a _must_ (I really miss the console feeling tho)

J.F. · Post by **J.F.** » Thu Jul 16, 2009 1:40 am

rapso wrote:(I really miss the console feeling tho)

The problem is that modern "consoles" aren't consoles anymore - they're compact computers. Just as you wouldn't try to write a PC ethernet driver that ignored memory mapping, PS3 drivers shouldn't ignore memory mapping as well.

maaku · Post by **maaku** » Mon Aug 31, 2009 6:06 pm

mc, this is an interesting project. I would like to help you out, although the realities of my other commitments are such that I cannot make any promises.

Might I suggest though that instead of writing an operating system from scratch (as much fun as that might be) we should instead focus on porting an appropriately small, but functional embedded OS like Contiki? That way we can focus on providing PS3 API's rather than repetitive low-level tasks...

maaku · Post by **maaku** » Mon Aug 31, 2009 6:37 pm

Or MINIX, actually. That might be a better route to take than Contiki.

rapso · Post by **rapso** » Mon Aug 31, 2009 7:27 pm

Why porting another OS if you can use linux?
I think the point was to have something lightweight from scratch dedicated for this one and only purpose: homebrew gamedev on ps3.

maaku · Post by **maaku** » Tue Sep 01, 2009 2:33 am

Because when you look at what people really want for a homebrew development, Linux would not satisfy their needs. For homebrew you want an embedded operating system (if anything at all). Linux is a server OS. The two extremes couldn't be more different.

It would not be a very large undertaking. MINIX is already ported to the PowerPC, and the kernel itself is only 4000 lines of code.

EDIT: Another option (just to throw it out there) would be L4Ka::Pistachio.

rapso · Post by **rapso** » Tue Sep 01, 2009 9:39 am

maaku wrote:Because when you look at what people really want for a homebrew development, Linux would not satisfy their needs.

Thats simply not true. the linux kernel is very generic and exists in tons of versions,
all packages are optional, there are kinda rip-offs here at the forum that are already some basic game development os, the only issue ppl saw was the licensing via gpl, afaik.

For homebrew you want an embedded operating system (if anything at all). Linux is a server OS. The two extremes couldn't be more different.

there are plently of embeded linux versions, from realtime to pure virtual machine aimed like android.

It would not be a very large undertaking. MINIX is already ported to the PowerPC, and the kernel itself is only 4000 lines of code.

as far as i know, minix is just a micro kernel, so its probably not doing more than MC's current version of the gamedev os.

EDIT: Another option (just to throw it out there) would be L4Ka::Pistachio.

i think, as long as the license is compatible with the bsd alike license this new game dev os is made for, you can rip of any other os and "port" it to this one. but lets see what MC says:)

maaku · Post by **maaku** » Tue Sep 01, 2009 12:35 pm

rapso,

I'm sorry I don't have the time to engage in a full discussion on this, but let me offer at least a few points (from over a decade of experience in embedded software, linux kernel customization, and game development):

* Linux can be streamlined to an embedded profile, but at the cost of many man hours initially and continually in maintenance. Easy for a corporation to do, not so much for a loose-nit homebrew community.

* The existing kernels are all desktop/server derived variants of Linux. There is not, to my knowledge, any true real-time Linux kernel for the PS3/Cell. Real-time linux is always highly-coupled with the architecture that it is designed for, and developing such a kernel for PS3/Cell will take a non-trivial amount of time and resources to do properly. It's not just a matter of hitting some checkboxes and compiling.

* The ps3sdk has very small, but very specific needs that do not match up very closely with Linux (even with the patches Sony and those on this forum have made).

* Don't fall into the "Linux will be easier because it's Linux" fallacy. Linux is around ~3 million lines of code. MINIX is just 4000 lines of kernel code. L4 is of similar size, and already ported to 64-bit PowerPC. That's three orders of magnitude difference (yes, I know we are comparing micro- to macro-kernels, but for reasons of software engineering complexity it is relevant). What's more, these kernels already have reliable implementations of feature sets needed for real-time embedded programming and are simple enough to not require the extensive tweaking in the port that Linux RTOS options do.

Most people don't know, or underestimate even if they do the amount of work that must go into creating and maintaining a custom linux kernel for a real-time application. It is my professional opinion that the ps3sdk could be finished faster, and with better satisfaction by adopting an existing, smaller, non-Linux solution.

EDIT: But perhaps we should agree to disagree, if you're still not convinced. My free time is very limited, and any part of it I spend here would otherwise be spent on ps3sdk. And I think we all agree that flawed approach or not, any ps3sdk is better than no ps3sdk.

MC, are you still here? Unless I hear from you, I'm going to begin porting either MINIX or L4 to Cell, but I could use some of your expert knowledge... (or anyone else here that is familiar with kernel mode PPE).

forums.ps2dev.org

PS3SDK collaboration invitation

PS3SDK collaboration invitation

Contiki OS?