DaWei

True, but so is the converse. If you, or another application, need more memory than is available, the OS will swap a page of memory to disk until that memory is required again. The memory swapped out will be chosen using an algorithm such as "least recently used" or "least frequently used". While the approach isn't perfect (thrashing can occur under certain conditions), it is likely to be more effective than off-the-cuff steps.

__________________
Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code.
Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers

Arevos

This is true, but it seems to me that the efficiency of this approach depends on the amount of data being swapped from disk to memory and vice versa, as a specialist solution would be able to more intelligently optimise the data. If we're dealing with hundreds of GB worth of files, and relatively small amounts of data being held in memory, I can't see the memory paging of OS being that effective. Also, some operating systems use a swap partition, rather than a swap file, so even with virtual memory, the amount of data that can be stored in memory is limited to a few gigabytes on your average modern PC.

I agree that it may be premature optimisation, but to my mind it all rather depends on how large the data being processed actually is.

DaWei

This is also true. I'm not trying to be recalcitrant here, I'm trying to point out some facts. It is unlikely that the average user is a specialist in either demand paging or cache control.

The system, as a whole, will be trying to optimize the operation of the system, as a whole, and not a particular application. This may result in a solution which is not optimal for the application that the user perceives to be the most important.

On the other hand, if the application writer takes things upon himself and attempts to control the operation by forcing commits to the disk, he or she may very well degrade the performance of the application, compared to what it would be, if left alone.

If one observes the gross effects on memory usage, such observations may be entirely out of sync with what one expects to be happening.

There are things one may tweak that will shift emphasis around and there are third-party packages that will promise to improve memory usage. I find that these are typically effective for certain types of usage and are not a panacea or universally applicable.

At the very least I would suggest that an experimenter use a professional performance test suite to gauge the results of tinkering, or become versed in the subject to the point of professionalism in the arena. I would also suggest that tinkering with less then the projected actual load will mostly be wasted time.

__________________
Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code.
Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers

Arevos

True. I do suppose it wouldn't hurt to test to see how well the application performs normally under load.

Ooble

Just remember the first rule of code optimisation: don't. This isn't intended to mean that you shouldn't optimise, but to remind you that it's a lot more complex than you would think at first, and to do a lot of research into optimization techniques and the underlying infrastructure surrounding your program before you even start. Personally, I don't even pretend to think I can optimise applications, past improving on already-written code.

The second rule? Finish, then optimise.

__________________
Me :: You :: Them

Sane

That sounds like a load of shit.


...

But it truly is nice to see you again Ooble!

__________________
Waterloo's Canadian Computing Competition (CCC) - Stage 2 Problems, Solutions, and Test Data

Ooble

I don't mean to say you should give up all hope of optimisation - just that if you plan on playing about with memory and caches and things, you may want to put a bit of research into finding out the best way of optimising your code first. While your idea sounds good, in practice, due to the way the OS works and other aspects beyond your control, it may not be the best idea. DaWei said much the same thing a few posts up.

__________________
Me :: You :: Them

DaWei

It may sound like a load of shit, but it's correct. If you chop a half-second off a one-second operation that's used once a month, how much time can you afford to spend doing it? If you can chop a mere sliver off a short operation used thousands of times a second, then you're on the right track.

Some of those are easy to spot, particularly when it's all your code. In an application that's coupled to an OS for services, and the coder knows nothing about the operation of those services except end results, the job's definitely tougher.

Any sensible engineer will tell you that you need to profile your code before you even think about optimizing it. That's based on statistics: they encounter very few gods or miracle workers among their coders.

__________________
Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code.
Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers

Sane

Well, I'll explain where I'm coming from in my (potentially skewed) perspective.

By nature of the list being an indefenite size, its size should be considered infinite. If I had an ideal amount of 1,000,000 entries, 1,000,000 multiplied by an average class instance of 2kb, is 4 GB. My computer can't handle 4 GB of RAM. Even if the OS were to help minimize RAM by %97.5, my computer can't even handle a process of 100 MB cleanly.

Therefore, I know that this shouldn't even be labelled "optimization"... it's a necessity for the application to even function properly. It's a requirement.

It could be compared to MySQL and it's caching. One could consider that an optimization, but I'm almost sure that the MySQL developers considered it a requirement. I am guessing that the first versions of MySQL were designed with caching in mind. I speculate that they were coding up the methods for caching frequently accessed tables, before it was even first released.

Pertaining to the "do a lot of research into optimization techniques", like I said before. I don't consider this optimization. I consider this an obvious necessity to design a single function that will dump old RAM into a file when it hasn't been accessed in a while. This doesn't seem like something that one would need to research, it seems like there could be only one solution: dump the list indexes that haven't been needed.

That said, I've accomplished this in one function that does not slow down the program, and meets the requirement for RAM reduction. The variable parts however that DO need to be changed are the dump frequencies and variable ages. These will be changed depending on certain statistics from bench marking the performance. This is the the optimization that will happen at the end. Later, I will be able to analyze what kind of intervals and frequencies are required, and modify them accordingly to optimize. For now, the essential part of reducing RAM is more than adequate, and can be optimized later.

For now, I'm done, and it works perfectly. I've actually got an ingenius solution for randomizing which indexes are checked, that will mathematically guarantee RAM reduction while doing mere constant time checks.

All in all, this is just another part of the program. Not optimization. Optimizing small sections of the code, to increase the performance by small factors, will come later. I'm done with this portion of the program for now, so thanks for the help Arevos!

__________________
Waterloo's Canadian Computing Competition (CCC) - Stage 2 Problems, Solutions, and Test Data

DaWei

I'm not saying you can't do your own caching. Critical applications obviously do. I'm saying that you need to make sure you don't buck or duplicate what the OS is doing. Good database apps use low-level IO/file system operations to avoid it.

The OS does a couple of things. One is cache and one is virtual memory. They aren't precisely the same thing. If you work directly with the file system, you will avoid OS caching. If you use dynamic memory, you will kick in the virtual memory operations. I don't know precisely what Python offers, even though it's based on C/C++. In C/C++, aside from dynamic memory, you can memory map a file. The OS takes care of keeping that synced up with the disk.

If the frequency of access of various objects is random, there's very little you can do. Objects which are used more often than others need to bubble to the portion of the collection which is currently mapped into memory or is able to be accessed with minimal latency/access time on the drive. I think that's what you're trying to do, which is fine. I'm just saying you CAN degrade the operation if you don't make sure you and the OS aren't handling the same stuff twice when once (and quickly) is what you want.

__________________
Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code.
Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers