Quote:
Originally Posted by ZenMasterJG
IMHO, you're optimizing prematurely. DaWei is right. Find out how your algorithm does, and *then* optimize. If your RAM can't handle the number of objects your creating, trying to think of an alternative solution is probably a better idea then trying to re-write Python's memory management.
|
Normally, I'd agree, but if the number of objects in memory plainly exceeds the amount of RAM available, then this sort of optimization is necessary. However, RAM tends to be fairly large these days, so one might want to consider whether it really is necessary; but in principle, at least, this might not fall under the category of premature optimization, but more a matter of necessity.
Quote:
Originally Posted by Sane
And the memory went waaay up. Shouldn't it only go up a little bit, since shelve won't keep it in the memory?
|
Whilst Python discards memory instantly, your OS usually does not. If a chunk of memory is freed by a program, and you have plenty of free RAM around, then the OS will often keep the chunk of memory around until it's needed.
In order to properly check that the application works, you need to operate on many different objects. For instance:
x = "-" * 100 * 1024
del x # x deleted, but OS doesn't bother to reassign 100M straight away
# The following code will likely result in an out of memory error:
l = []
for i in range(100):
l.append("-" * 100 * 1024)
# The following code will not, since the OS will reassign the freed bits of
# memory when it deems it necessary:
for i in range(100):
x = "-" * 100 * 1024 Try your class with a large number of objects that would normally result in an out-of-memory error if they were all loaded in at once.