Core Project: External libraries

[nevyn]

As was discussed in the 'pre-implemented FFT routine' thread (how the heck do you "pre-implement something? :P (rhetorical question)), dynamically loadable libraries (as in compiled, C libraries, not lua libraries) would be real nice. Adding functionality without bloating the LuaPlayer codebase. We've already heard about the advantages (divx decoding, fft, webkit? you name it.). How'd one go about doing it though? there's

Code: Select all

loadlib (libname, funcname)

, but that's just for individual c functions. hm, but that could be wrapped with a lua file...

It does seem entirely possible. I'm throwing this thread in here anyway, to gather thoughts in a specific place.

(Hm, I'm taking a course in compiler technologies and specifically runtime linking right now, perhaps I can convince my prof that psp hacking is really school work ;) )

(please reserve the thread for technical discussion only, and not "yeah that'd be cool do that!" or "with that, we could do x and y and neatIdea!" (unless it's a *very* neat idea :P))

[Shine]

there's

Code: Select all

loadlib (libname, funcname)

, but that's just for individual c functions. hm, but that could be wrapped with a lua file...

This function could be standard entry point like our other functions for luagraphics, luawlan etc. Since it is a Lua function, the function is called with a lua_State pointer and it can register the rest. I think the best would be to load all those libraries like this, perhaps with the boot.lua at startup for the standard libs. Then Lua Player itself is a module loader, only and all the functionality is provided by the loaded modules (and can be replaces independently or not loaded, if not needed).

The module concept itself should use PSP PRX files (something like shared libraries). I think they have the same address space and the link between the other modules could be the lua_State pointer or some predefined functions (e.g. getLuaPlayerVersion() etc.).

The advanced feature would be to implement something like in the Eclipse IDE: Every module has its own version and has a dependency list, which other modules with which version are needed. This could be used e.g. for the lua3d lib, because it needs a bit of the luagraphics lib. But calling a function of another library from within a library could be interesting :-)

[JoshDB]

I'm pretty sure I posted a topic a while ago asking what the "Library" folder was for. I'm not a C coder, but I've made the Hello World program... Is there a way to drag/drop LUA libraries from LUA.org into there?

[Oobles]

I've just been thinking about porting a library to use in LuaPlayer, and I'd say having a loadable module would be the best method.

As has already been mentioned, I agree that the best way to do this would be to use PRX files. I've been looking at the examples that show the C API layer at:

http://lua-users.org/wiki/SampleCode

All the examples have a function which will bind the additional library methods, classes, etc into Lua. If LuaPlayer called a specific export to perform this function then the library could basically install itself. So call a function like:

int registerLuaLibrary( lua_State * L );

That function then does everything it needs to bind the loaded library into Lua. The only question I have is how does the library access the required Lua C functions. Do these need to be exported from the elf, or does Lua itself need to be put into another library? Can anyone solve that one?

David. aka Oobles.
www.livemedia.com.au/Blog

[Oobles]

After doing some more research and looking at the pspsdk examples of loading a PRX I've got some additional thoughts. There are two main functions that need to be called:

sceKernalLoadModule
sceKernelStartModule

It would be easier to just pass in lua_State into the sceKernelStartModule than to use special export. It would nice to be able to identify Lua specific PRX's, however, I suspect that a naming convention is probably easiest for Lua modules.

I still need to do some more looking at if lua functions need to be exported. I suspect that if the lua library doesn't keep any static data then the required functions could be duplicated in the PRX and just modify the lua_State passed in?

David. aka Oobles.
www.livemedia.com.au/Blog

[Oobles]

Another quick update. I compiled a very simple PRX which linked in lua and lualib. It created a PRX that was 182kb. Seems like a high price to pay for statically linking in Lua.

Code looked something like this...

Code: Select all

PSP_MODULE_INFO("SIMPLELRX", 0x1000, 1, 1);

#define WELCOME_MESSAGE "Simple LRX loaded\n"

static int lua_doSomething(lua_State *L)
{
        if (lua_gettop(L) != 0) return luaL_error(L, "wrong number of arguments"
);
        lua_pushstring(L, WELCOME_MESSAGE );
        return 0;
}

static const luaL_reg Simple_functions[] = {
  {"doSomething",       lua_doSomething},
  {0,0}
};

int main(int argc, char **argv)
{
        lua_State * L;

        if ( argc != 1 )
        {
                return 1;
        }

        L = (lua_State *) argv[0];
        luaL_openlib(L, "Simple", Simple_functions, 0);

        return 0;
}

[LuMo]

we were talking about external "plugins" for a while...
if you plan to integrate prx files...
the post with GL OR GU is solved, cause you can have both in seperate PRX files then....

@the size of the prx files...
i think noone actually cares about the size of the data
main prob might be the memory usage (RAM not MS)
i guess noone uses 32mb sticks (i use it for dev. only)

but one question came to my mind...
will it be possible to eg. extend functions of a finished class?
you have eg. image:blit in your code
so is it possible to implement another fkt to image?

greets
lumo

[Oobles]

I've completed some more tests. This time I added a new System.loadModule function to see if I could load the PRX and install a function.

Code: Select all

static int lua_loadModule(lua_State *L)
{
        u32 loadResult;
        u32 startResult;
        int status;
        char *argv[2];

        const char *path = luaL_checkstring(L, 1);
        if (!path) return luaL_error(L, "Argument error: System.loadModule(name)
 takes a module name as string as argument.");

        loadResult = sceKernelLoadModule(path,0, NULL);
        if (loadResult & 0x80000000)
        {
                return luaL_error(L, "Argument error: Failed to load module");
        }

        argv[0] = (char *) L;
        argv[1] = 0;

        startResult = sceKernelStartModule( loadResult, 1, argv, &status, NULL );
        if ( loadResult != startResult )
        {
                return luaL_error(L, "Module error: Failed to load module");
        }

        return 0;
}

This all works, however, when the lua_openLib function is called inside the PRX main, the PSP crashes requiring reboot. This either means that you can't pass pointers via main argv, or calling lua functions with statically linked in libraries is not going to work. Either way, I think it will require that lua itself is pushed into a PRX.

If anyone can see errors in the code, or suggest another method that doesn't involve putting lua into a PRX.. let me know.

Edit: Just found on this page to confirm that Lua will need to be placed into a PRX. http://lua-users.org/wiki/CreatingBinar ... ionModules

"If those functions exist within host application ( like lua.exe), we cannot link our extension module. We must move the Lua API into a DLL, so that the host application and the extension DLL can share the same copy of the Lua API code."

David. aka Oobles.
www.livemedia.com.au/Blog

[nevyn]

Nice work on the research there. I concur on the filesize not being a major problem. However, if the lualib is put externally in a prx, it's not a problem anyway. (This is not a difficult task, is it? I haven't done much dynamic library work)

Might have some term gap time next week. If so, I'll check it out.

[Oobles]

I've created a lua.prx which contain just the lua libraries. I've exported all the required functions and got LuaPlayer to link with these instead of the static libraries. Everything is compiled, however, I'm having trouble loading the lua.prx from within LuaPlayer.

If you like I can commit these changes so you can have a look. I'll keep trying to get this to work though. Just don't want us duplicating work.

[nevyn]

I've created a lua.prx which contain just the lua libraries. I've exported all the required functions and got LuaPlayer to link with these instead of the static libraries. Everything is compiled, however, I'm having trouble loading the lua.prx from within LuaPlayer.

If you like I can commit these changes so you can have a look. I'll keep trying to get this to work though. Just don't want us duplicating work.

You're very welcome to commit, I'm pretty curious :) You go ahead with your code, I'll fail another course if I get into hack mode now anyways...

[Oobles]

Ok, I've committed the lua.prx in src/luaprx with its own makefile. I haven't modified any of the actual luaPlayer code. I've added the following to main.cpp in user_main

Code: Select all

        // change directory.
        getcwd(path, 256);
        chdir(path);

        // load the Lua PRX
        strcat(path, "/lua.prx" );
        int retVal = sceKernelLoadModule( path, 0, NULL );
        if &#40;retVal < 0&#41;
        &#123;
                debugOutput&#40;"Error&#58; failed sceKernelLoadModule load lua.prx\n",
31&#41;;
                sceKernelSleepThread&#40;&#41;;
        &#125;

        debugOutput&#40; "LoadModule\n", 12 &#41;;

        int fd;
        retVal = sceKernelStartModule&#40; retVal, 0, NULL, &fd, NULL &#41;;
        if &#40; retVal < 0 &#41;
        &#123;
                debugOutput&#40;"Error&#58; failed sceKernelStartModule lua.prx\n", 44 &#41;
;
        &#125;

You will also need at the top of main.cpp the following to ensure that enough memory is available to load the prx.

PSP_HEAP_SIZE_KB(10000);

You also need to modify the makefile to include LuaLib.o in the objects. To create LuaLib.c you run in the luaprx directory:

psp-build-exports --build-stubs exports.exp

You also remove the lua libraries from the Makefile.

After all that is done, it still doesn't work. :) I'm not exactly sure why though. Anyone got any ideas?

[edit: more info] As far as I can tell it now loads the lua.prx correctly, but after that, the screen goes black and nothing happens. I'm using a very basic lua script to check if it works. I'm wondering if anything else needs to be done to link in the PRX exports with the imports of LuaPlayer?

David. aka Oobles.
www.livemedia.com.au/Blog

[nevyn]

[edit: more info] As far as I can tell it now loads the lua.prx correctly, but after that, the screen goes black and nothing happens. I'm using a very basic lua script to check if it works. I'm wondering if anything else needs to be done to link in the PRX exports with the imports of LuaPlayer?

David. aka Oobles.
www.livemedia.com.au/Blog

Great work :)

Can you printf from c (not lua) or is the black screen a complete crash/infinite loop or something?

[Oobles]

I've had another go at it, but still not working. I've run it under PSPLINK and now know that there is an "Address load/inst fetch" exception happenning in LuaPlayer. I haven't had time to investigate it further than that yet.. will leave it for another day. :)

David. aka Oobles.
www.livemedia.com.au/Blog

[Oobles]

Three months later and I'm still working on this problem. I finally loaded a module yesterday and got things working. However, there might be some issues with keeping full compatibility with the last version of LuaPlayer.

The current solution is to put luaplayer into a user mode PRX file and use a small bootstrap code in kernel mode to load it. The luaplayer exports all the Lua functions. A module is loaded and in its main function it registers the module with luaplayer.

I still need to check if there are any kernel mode functions that can no longer be called? Can anyone tell me what parts of luaplayer requrie kernel mode?

Thanks,
David. aka Oobles.
http://www.livemedia.com.au/Blog

[Shine]

Can anyone tell me what parts of luaplayer requrie kernel mode?

The USB support, SIO and stdout/stderr handler requires kernel mode functions, but Lua Player itself works without these functions.

[Oobles]

Well, I did it. I committed version v0.18alpha to Subversion. It is a bit of a re-arrangement of LuaPlayer, so it needs some testing. The big change is that luaplayer is now in a user mode PRX which is loaded from the kernel mode bootstrap. The net effect of all this is that luaplayer can now load other PRX modules.

I've added two example modules. The first named simple is mostly just a template for how to extend LuaPlayer. The second named sound is extracting the sound module out from the main LuaPlayer. I haven't actually removed sound from the main luaplayer, however, it could be done using this method. The only problem is that there would be issues with backward compatibility and require developers to update their code to load the sound or whatever other module.

The changelog has a few other bits and pieces that were changed a long the way. The main one that comes to mind is the debug message output.

Please someone test this and let me know how they go compiling and running code. There may still be some issues.

I would like to modify the module loading a little and make it more inline with the Lua loadlib function. I hope to get time to do this soon.

Regards,
David. aka Oobles.
http://www.livemedia.com.au/Blog

[LuMo]

skip v0.17?
:D
great work, again ;)

greets
lumo

[nevyn]

This kills post-1.5 version PSP support due to the kernel requirement, I s'pose? Personally I don't mind (since i have 1.5 myself :P), I love to see this progress.

[Oobles]

LuMo, I skipped v0.17 because it was already in the ChangeLog.

Nevyn, yes, module loading does require kernel support. However, having said that, it should be easier to build LuaPlayer for post-1.5 because all the kernel code has been split out into the bootstrap.

I did some more testing before and found a major bug. I didn't put the exports file into the luaplayer makefile. This stops module loading from working. Oops. I will commit the fixes in a few hours during my lunch break. I don't have access to commit code right now.

Regards,
David. aka Oobles.

[Oobles]

As promised, I've committed the fixes. At some stage I'll add an example for module loading. If you want to give it a go you just need.

Code: Select all

System.loadModule( "simple" )

txt = Simple.doSomething()
-- then print text somewhere on screen to receive the secret message :)

The current behaviour for load module is that it will load modules from the application boot path and add .lrx on the end of the name.

David. aka Oobles.

[daurnimator]

love it :D

coding with it now; but that bug is extremly annoying, just a hack to get around it for now would help...

[daurnimator]

btw, for others that want it, i've compiled,ziped and uploaded it (with a lua file that uses Oobles's lrx.

Binaries: http://s35.yousendit.com/d.aspx?id=02A0 ... KU4TL3UVG8
src: http://s25.yousendit.com/d.aspx?id=3G1T ... 8NMFR66623
(you need a recent toolchain)'

[Oobles]

I've just committed an update to the module loading to Subversion. I've replaced the System.loadModule with the more traditional loadlib function. The way you write modules and load them has changed a little. Check the modules in subversion to see how. It now works like this in lua.

Code: Select all

simple = loadlib( "simple", "init" )
simple()

The module name is passed in as the first parameter and the function to call to initialise the module as the second. It does not automatically call the init routine but instead returns it. You must call it yourself.

The PRX module name must be the same as the first parameter.

Let me know if you have any problems.

David. aka Oobles.

[DiabloTerrorGF]

Layman's terms please?

[romero126]

You can make your own elf files. and run them in lua player ontop of your LUA script (So it can be used for display purposes).

You can load modules by useing the command

Code: Select all

simple = loadlib( "simple", "init" ) 
simple() 

[be2003]

Would it make more sense to not add .lrx to the end?

Why not let the user(me) define it? (for more flexebility)

Code: Select all

test = loadlib("test.lrx","init") -- or test.prx?
test()

Also, can lua load prx, dll, so?

The current behaviour for load module is that it will load modules from the application boot path and add .lrx on the end of the name.

David. aka Oobles.

[be2003]

an lrx is just a renamed prx, right

[nevyn]

Would it make more sense to not add .lrx to the end?

Why not let the user(me) define it? (for more flexebility)

That'd decrease portability; a Lua Player app would then require different versions for PSP / Windows / Linux / Mac versions (as windows uses .dll, mac .dylink, linux .so)

[be2003]

but cant a prx be a library too, in one makefile it executes cp ./effect.prx ,/effect.lrx. so in that sense an lrx = prx

never mind, i am an idiot...