out of core data storage

[anmabagima]

Hi there,

I'm not sure whether this post has the right subject or not. However, I've seen articles on the web discussing "out of core data storage" which some PC OS/applications are capable of. This mean that in the case the application need to process huge ammount of data it is not loaded into the RAM (core) but left in the file. The file access in this scenario is like accessing memory using a pointer to the data once the file was opened.

As the PSP has no harddisk but only SD card I question myself whether the PSP is capable of supporting this approach or not. And now I would like to ask you.... As on a common PC the access speed would be very much slow due to limmitations of the hardware but the SD card access on PSP should work like accessing RAM from a performence point of view, shouldn't it ?

The sceIO operations just gives file handles.You need to use them with READ methods to get data from the file into a buffer. I would like to have something like a pointer to the raw data of the file and beeing able to directly access this data I'm interested in, as I would do using them within a buffer in RAM. Is that possible ?

Any thoughts and Ideas are welcome and much appriciated.

Regards
AnMaBaGiMa

[a_noob]

I am not to sure about via the sceIo, I think it would need to use some reversing because it just returns a handle no actual data, but if you where to use stdio maybe, unless it was just hacked to store the handle and then accessed, but it it is properly used/coded then here is the FILE datatype wich has the info you need.

Code: Select all

typedef struct  {
       int             level;      /* fill/empty level of buffer */
       unsigned        flags;      /* File status flags          */
       char            fd;         /* File descriptor            */
       unsigned char   hold;       /* Ungetc char if no buffer   */
       int             bsize;      /* Buffer size                */
  unsigned char   *buffer;    /* Data transfer buffer       */
  unsigned char   *curp;      /* Current active pointer     */
       unsigned        istemp;     /* Temporary file indicator   */
       short           token;      /* Used for validity checking */
}       FILE;

[Raphael]

What you talk about is not possible on the PSP, because it does not have an MMU, especially not one that is capable of memmapping files.
So there is no direct way of pointer->file mapping, unless you write your own software layer, which pipes an smart pointer abstraction through sceIo function calls.
Unless you use C++ and operator overloading and templates, it wouldn't change anything about the feel of having to use functions to write data to files and even then, you still won't be able to write directly to constant addresses as with memory: *((int*)0xabcdef) = somevalue;

[anmabagima]

Hi,

thanks. I will try this. But "my" FILE structure looks pretty much different:

Code: Select all

typedef __FILE FILE;

struct __sFILE {
  unsigned char *_p;	/* current position in (some) buffer */
  int	_r;		/* read space left for getc() */
  int	_w;		/* write space left for putc() */
  short	_flags;		/* flags, below; this FILE is free if 0 */
  short	_file;		/* fileno, if Unix descriptor, else -1 */
  struct __sbuf _bf;	/* the buffer (at least 1 byte, if !NULL) */
  int	_lbfsize;	/* 0 or -_bf._size, for inline putc */

and some more stuff ... The _bf is defined as

Code: Select all

struct __sbuf {
	unsigned char *_base;
	int	_size;
};

I would assume the pointer to _base will do the job as well. I'll give it a try.

[m0skit0]

but the SD card access on PSP should work like accessing RAM from a performence point of view, shouldn't it ?

Abolutley NOT. DDR2 SDRAM offers 3200 MB/s, when best SD card you could find can only perform at 45 MB/s. Also SD Cards (as flash memory) have a latency measured in miliseconds, while main memory latency is usually measured in nanoseconds. So no, no way you can use SD as virtual RAM and expect close performance. RAM wins high hand.

I would like to have something like a pointer to the raw data of the file and beeing able to directly access this data I'm interested in, as I would do using them within a buffer in RAM. Is that possible ?

It is possible, but as I stated before, you would have a big drop in performance, not to speak about the software layer you'd have to set up as commented by last answer.

[anmabagima]

Hi,

thanks to all for the valuable statements.

Regards
AnMaBaGiMa