host transfer speed

radad · Post by **radad** » Tue Aug 30, 2005 9:44 am

When I try to use the host filesytem I get a transfer rate of about 40 Kb/s. Is anybody getting a faster rate? Would this just be due to the protocol?

muckle · Post by **muckle** » Sat Sep 10, 2005 2:14 am

depends on what you're doing. Since EVERY read/write/lseek command is remoted-- it's pretty awful. The underlying ethernet driver is much faster-- I can get 6-7Mbps raw packet throuhput. So the short answer -- "it's the protocol".

dlanor · Post by **dlanor** » Sat Sep 10, 2005 3:41 am

muckle wrote:depends on what you're doing. Since EVERY read/write/lseek command is remoted-- it's pretty awful. The underlying ethernet driver is much faster-- I can get 6-7Mbps raw packet throuhput. So the short answer -- "it's the protocol".

That's both true and false, actually...

It's true that the host: protocol doesn't allow the full ethernet speed. But no real-life file transfer protocol can do that. There are always some protocol overhead costs.

It's also true that the remote command handling of host: causes response delays that lower the throughput far below what could be achieved with a protocol designed for network operations. (The operations of host: are all oriented around normal file I/O, originally designed for local storage media.)

But it's not at all true that those limitations would cause a throughput of 40 Kbps (40 Kbps = 5 KB/s = horribly slow) or even 40 KB/s (still very slow).

Using uLaunchELF v3.41k I usually get speeds of around 500 KB/s when copying from host: to hdd0:. Similar speed is also achieved by the newer versions of the AVI player SMS, to which I recently added host: support. (Though in SMS that is the cache refill burst speed, as total throughput is synced to the total AVI bitrate.)

A speed of 500 KB/s is equivalent to appx 4Mbps, which is not too shabby, all things considered.

Best regards: dlanor

muckle · Post by **muckle** » Sat Sep 10, 2005 3:49 am

Again, I'll say it depends on what you're doing. For instance a sequence of

lseek (somewhere [like the next byte position in the file if you really want to see inefficient/low results))
read (10 bytes or some small [annoying] number

put this into a loop and you'll see that the protocol overhead dominates everything. The key to this performance is to spend more cpu cycles doing the actual drudgery of moving bytes and less on the mechanics of protocols, handshakes, decoding message headers and the like.

In actual practice on the PS2, I've built video player applications which can transfer 3-4MBs into the EE-- these rates were achieved by optimizing the block sizes while minimizing the "extra crap" (including extra copies) that goes along with it.

dlanor · Post by **dlanor** » Sat Sep 10, 2005 8:51 am

muckle wrote:Again, I'll say it depends on what you're doing.

I quite agree. I never argued against that.

For instance a sequence of lseek (somewhere [like the next byte position in the file if you really want to see inefficient/low results))
read (10 bytes or some small [annoying] number

put this into a loop and you'll see that the protocol overhead dominates everything. The key to this performance is to spend more cpu cycles doing the actual drudgery of moving bytes and less on the mechanics of protocols, handshakes, decoding message headers and the like.

Again I agree, though I have to add that this is true of any transfer operations, regardless of the media. A large number of operations handling small data blocks will always have higher overhead costs than a small number of operations handling large data blocks. The one exception to that is if you can expect errors causing the need for retries. (Which is why low-level IP uses fairly small packets.)

In actual practice on the PS2, I've built video player applications which can transfer 3-4MBs into the EE-- these rates were achieved by optimizing the block sizes while minimizing the "extra crap" (including extra copies) that goes along with it.

Again I agree completely. That is how to do it.

Perhaps you misunderstood my earlier post. That was not intended to criticize what you said, but rather to clarify the situation to 'radad', who clearly had an abnormally low throughput. (At least if we assume that he was doing normal file transfers.)

Best regards: dlanor

muckle · Post by **muckle** » Sat Sep 10, 2005 10:02 am

I think we're in fine agreement here.

Having monkeyed with supercomputers when they still occupied buildings and the cpu boards were immersed in liquid nitrogen (for speed, NOT for cooling), I can unequivocally say [at least] two things if you're a "plumber":

1. never touch the data. Don't copy it, look at it, count it.
2. spend even less time "managing it" (a fine lesson for all business people!).

In general this means crank up the block sizes till they hurt. During the analysis of expected failure rates and frequencies, you get to make the tradeoff about "doing as little over again (think retransmission)" and "assume it all works-- run as fast as you can and pick up the pieces only when it breaks". The latter was especially amusing with vector supercomputers where the cost of firing up the vector unit was great (and only slightly less expensive than picking up the pieces if an exception occurred during a long vector operation "ok dang, anyone know what got done here???".

IP's block sizes, TCP's windows etc. have all come under great examination-- if you host it on a perfect transmission medium-- it's overkill. TCP/IP is even worse-- which is why things like RTSP are around for moving large amounts of isochronous data where losing bits here and there "may" be "ok".

[end of ramble]

radad · Post by **radad** » Mon Sep 12, 2005 9:11 am

That was 40 KB/s I am getting using LaunchELF or the SMS player. How are you acheiving 500 KB/s? What is wrong/different in my setup?

dlanor · Post by **dlanor** » Tue Sep 13, 2005 3:50 am

radad wrote:That was 40 KB/s I am getting using LaunchELF or the SMS player. How are you acheiving 500 KB/s? What is wrong/different in my setup?

That is very hard to say without knowing more about your setup, but something must be very wrong indeed, as most people do get the higher rates I mentioned.

Some things to consider:

Have you doublechecked your IP configuration at PS2 VS the PC ?
Are you using a router as gateway, or direct (crossed) cable ?
In case of a router, have you checked its LAN config VS PC and PS2 ?
I realize that you have communication, but some cases of misrouting can lead to inefficient connection (like you apparently have)

Are you using any firewall or netlimit software on the PC ?
Have you used the same interface of your PC in tests with other PCs/Internet, to verify that all its drivers are OK, and allow higher speeds ?

I hope you can find and fix the problem, whatever it really is.
Best regards: dlanor

radad · Post by **radad** » Tue Sep 13, 2005 5:08 pm

I have a crossover cable, no router. Only the XP firewall software.

I shall have a closer look at my setup. I thought had a normal transfer rate.

radad · Post by **radad** » Wed Sep 14, 2005 3:16 pm

Checked out everything I could last night. Confirmed all the ip config settings are correct. Tried turning off the firewall. Tried taking out links in the cabling (my normal setup has it in a different room). Ping doesn't works fine, no packet loss. Traceroute shows only one hop.

The only thing left I can think of is the software. I am using LaunchELF 3.41i and the latest SMS. Not sure about the ps2client end. Does anybody have a new build of ps2client?

dlanor · Post by **dlanor** » Wed Sep 14, 2005 7:48 pm

radad wrote:Checked out everything I could last night. Confirmed all the ip config settings are correct. Tried turning off the firewall. Tried taking out links in the cabling (my normal setup has it in a different room). Ping doesn't works fine, no packet loss. Traceroute shows only one hop.

Sounds all normal then, except for what you said about 'ping'.
How can it have "no packet loss" if it "doesn't works fine" ? I don't get that part...
I believe you meant "does" rather than "doesn't". Right ?

The only thing left I can think of is the software. I am using LaunchELF 3.41i and the latest SMS.

There have been no changes to the LaunchELF host: code since v3.41i (basically the same as in SMS), but there have been other bugfixes, so I recommend that you upgrade to the current version (v3.41k). It won't help the networking speed, but just fix some other problems.

Not sure about the ps2client end. Does anybody have a new build of ps2client?

It could be a problem with the exe file you use of course. The latest one should be available right here. Or elsewhere on this site rather, not right here in the forums.

Just go to the main ps2dev.org page and check out the "Loaders" link in the left navigational bar. That link will pring you to a page with various 'loaders', such as PS2Link, and their PC-side clients, such as ps2client. (I'd better test that version myself too.)

----- some downloads later -----
OOOPS !!!
Someone has made a very bad mistake in preparing those downloads.
There is NO version of the ps2client binary available, but only source code.
I'm afraid the same goes for SVN as well, with just source and no binary...

I'm sure it can be found somewhere else, but I don't have time to search for it right now.

Anyway, if you send me a PM with your email address, I'll send you a copy of the ps2client binary I use.
Best regards: dlanor

gawd · Post by **gawd** » Thu Sep 15, 2005 4:30 am

I started investigating this as well. As I either convert my stuff with bin2c, or just load all datafiles from host0:.

I get ~120K when loading the ELF (xp->router->ps2.)
And then ~760K when reading from host0 without any seeks.

I will further check how much time is wasted at ps2client side, hopefully that's all I will have to debug. I will keep you posted.

-- ga wd ?

Post by **ooPo** » Thu Sep 15, 2005 7:31 am

http://oopo.net/consoledev/files/ps2client.exe - ps2client v2.0

You should also have the latest version of ps2client fresh from svn if you've built the toolchain using the autobuild script.

radad · Post by **radad** » Thu Sep 15, 2005 9:44 am

dlanor wrote:
radad wrote:Checked out everything I could last night. Confirmed all the ip config settings are correct. Tried turning off the firewall. Tried taking out links in the cabling (my normal setup has it in a different room). Ping doesn't works fine, no packet loss. Traceroute shows only one hop.
Sounds all normal then, except for what you said about 'ping'.
How can it have "no packet loss" if it "doesn't works fine" ? I don't get that part...
I believe you meant "does" rather than "doesn't". Right ?

I started writing 'Ping doesn't lose any packets', then changed my mind. Oops.

I shall try the ps2client.exe v2 tonight. I shall also try building my own from the latest sources. Thats how I got the one I am using now.

dlanor · Post by **dlanor** » Thu Sep 15, 2005 1:39 pm

ooPo wrote:http://oopo.net/consoledev/files/ps2client.exe - ps2client v2.0

I can confirm that this version transfers data at good rates. I just copied a 32MB avi in almost exactly 60 seconds, so that's over 500KB/s.

Best regards: dlanor

Post by **ooPo** » Thu Sep 15, 2005 10:19 pm

How about the version in svn? It has quite a few changes and should be tagged v3.0 any time now. :)

Post by **Oobles** » Thu Sep 15, 2005 11:10 pm

Did someone say *crossover cable*? These can be bad! They are usually fine when you do small aounts of data like ping, etc. However, as soon as you start cranking them up you will get packet loss.

You can pick up a very cheap 5port switch these days, so no excuse not to have one. Infact $120 AUD gets you a nice linksys wireless router with a few wired ports.

Anyway, try and test it with a switch and then the cross over and let us know the different in transfer rates.

David. aka Oobles.

radad · Post by **radad** » Sat Sep 17, 2005 11:11 am

Ok, I tried the ps2client v2.0 uLaunchELF 3.41k combination and got no differences, still 40 KB/s.

I tried my crossover cable setup conncted to a old pc I have and got 600 KB/s using normal windows network filesystem. That would indicate to me that it is not a hardware problem.

I haven't tried the router option as I don't have one yet.

I was wondering about the other thread talking about some timings in the ps2 ip side. Could this be the reason for the problems I am seeing? Maybe these timings don't work with the crossover setup I have?

forums.ps2dev.org

host transfer speed

host transfer speed

re: throughput question...

Re: re: throughput question...