[zarzu]

i am currently coding a rest server for the n95, the server let's you access sensor data (gps, wlan, bluetooth etc) stored in a e32db, which is all running smooth so far, well except for the serializing. the service returns xml by default, kml, json and rss on demand, i have templates for each of them, they consist of up to 3 files for each format, i load them in variables and then build the outgoing file by concatenating them after inserting the values, so the files may look something like this:

template1:
<?xml>
<root sensor="%(sensor)s">
%(entries)s
</root>

template2:
 <entry name="%(name)s" %(otherattributes)s>
  %(sometext)s
 </entry>

and they are processed as follows:

tmpl1 = load template1
tmpl2 = load template2

entries = ""
for entry in dbrows:
 otherattributes = "date=\"somedate\""
 name = entry["name"]
 sometext = entry["description"]
 
 entries += tmpl2 % {"name": name, "otherattributes": otherattributes, "sometext": sometext}

output = tmpl1 % {"sensor": sensor, "entries": entries}
return output

this seems to be the straight forward thing to do for me, but the problem is for a 30kb file this is taking up to 25 seconds, which is insanely long, so there must be something that is slowing this down terribly, i have no prior experience with python and i didn't find anything related to my problem.
i'd appreciate any help here. thanks in advance.

[marcelobarrosalmeida]

Hum, nearly 1 package per second ... so, could you show your transmission routine ?

Marcelo

[zarzu]

the transmission is handled by the mobile web server and i am not talking about how long the whole request takes to come back, the 25 seconds come from taking the time before and after serializing, it's sent as an attribute of the root element of the xml file.

[JOM]

Your system is very complicated, so "naturally" it requires a lot of time. Templates with 3 files each: file system access is slow. GPS requires several seconds to activate even under the most optimal circumstances, several minutes usually, even 15 mins without Assistance (A-GPS). Even importing socket requires seconds.

If you're not happy with performance, maybe you should measure where all the time is spend? Then you can consider whether you want to simplify your system in some places or use lazy import.

Here's a lazy profiler:

# Lazy import, after user looking at screen:
#import time
#t0 = time.clock()
import socket
import urllib
#print "Time: %f" % (time.clock() - t0)
# 5 seconds in emulator DEBUG mode
# 3.8 seconds in emulator NON_DEBUG mode

Cheers,

--jouni

[zarzu]

i have to correct myself a little, it seems to take 25 seconds for a 200kb file, someone else checked the size and obviously it wasn't right, so this doesn't feel that horrible anymore, but still i feel like it's taking too long, on the other hand i don't really have any comparisons, but still.

i have to reiterate though, the time i measured is only dictionary access, string interpolation and string concatenation, imports are done at the beginning, this has no effect, file loading takes 0.02 seconds, sensors (like gps) have no impact here, i read them with a separate program that writes them to a db.

i can give some measurements of the other parts of this program, for a 500kb output, it takes 6s to read the database and write all rows in a list, 2s to convert all of those rows into dictionaries and then 70s to write these rows into xml. and that jump just seems out of whack to me.

[jethro.fn]

Originally Posted by zarzu

entries = ""
for entry in dbrows:
  ...
  entries += ...

This is always a bad idea in Python or any other language with immutable strings (Java, .NET). What is happening here is that for each dbrows entry, a new entries string is created and the old one destroyed. The new entries string and the old one are bigger and bigger each round, slowing down the execution due to unnecessary pressure on memory allocator and garbage collector of Python.

Here's how you usually do it in Python:

entries = []
for entry in dbrows:
  ...
  entries.append(...)

entries_joined = "".join(entries)

[zarzu]

i really hoped that was it, tried it, but it actually increased the time needed for processing by 10%

correction: i was a bit over enthusiastic with using the join it seems, i only used it on the really large concatenations now and it's running 10% faster than the original version.
thanks for the tipp!