Programming Forums > Application Development > C++

slyadams · Mar 28th, 2006, 7:26 AM

Chaps,

I have s system where I am pushing objects into a vector inside a function. At the moment I 'new' the object and push the pointer and it works fine.

I was thinking that I might be able to save some memory space by pushing the object itself and not the reference. i.e.

currently:

void func1() {
vector.push_back(new ClassName(param1,param2));
}

possible:

void func1() {
vector.push_back(ClassName(param1,param2));
}

The problem is that the destuctor for ClassName gets called everytime that func1 returns in the 2nd example. The problem is that the object in the 2nd example is static and its scope is limited to the function func1.

So my question is: is there anyway to push objects dynamically into a vector without using pointers. The only reason I don't want to use pointers is that these vectors are at the bottom level of a tree structure and in a standard instance of the process will, in total, store about 30million objects. Thus, storing the pointer, as well as the data 'wastes' about 100meg. We are moving the process onto 64bit platforms for clients, so this overhead could, potentially, double!

Thanks in advance for any help.

P.S. :banana:

grumpy · Mar 28th, 2006, 7:40 AM

I suggest you read the sticky post on "How to ask a question". It will give you tips on how to ask questions to increase your chances of getting a useful answer.

I'm assuming the vectors in your two versions of func are of different types (eg the first is a std::vector<ClassName *> and the second is a std::vector<ClassName>).

The push_back() function of std::vector appends a COPY of the value it receives to the end of the vector. It doesn't matter whether the value it receives is a pointer (as in your first example) or an instance of a class (as in your second example), as long as the vector is of an appropriate type. All that is required to work with an instance of a class is that your class has a copy constructor and associated assignment operator that work as expected.

If you use a std::vector<ClassName *> and push_back new'd objects, you need to delete all objects in the vector before allowing the vector to be destroyed or pass out of scope. For example, the following are functionally equivalent (except the first works with dynamically allocated objects being pushed, and the second doesn't;

(Toggle Plain Text)

#include <vector>

std::vector<ClassName *> myvec;

void func()
{
    myvec.push_back(new ClassName(param1, param2));
}


int main()
{
     func();
     //  delete the object pushed by func() before allowing the vector to be destroyed
     delete myvec[0];
     // myvec will pass out of scope and be released
}

vs

(Toggle Plain Text)

#include <vector>

std::vector<ClassName> myvec;

void func()
{
    myvec.push_back(ClassName(param1, param2));
}


int main()
{
     func();
     // myvec will pass out of scope and be released, as will its elements
}

PS Children will insist on playing with bananas

slyadams · Mar 28th, 2006, 8:19 AM

OK, but in the first example there is an increased memory overhead isn't there? Won't the Object be created in memory somewhere and the pointer pushed into the vector? Thus there is a memory overhead increase of sizeof(void *)?

Polyphemus_ · Mar 28th, 2006, 8:30 AM

Quote:

Originally Posted by slyadams

OK, but in the first example there is an increased memory overhead isn't there? Won't the Object be created in memory somewhere and the pointer pushed into the vector? Thus there is a memory overhead increase of sizeof(void *)?

True, but you will notice the 4 bytes of overhead (at least on most 32 bit systems) is worth it.

slyadams · Mar 28th, 2006, 8:38 AM

The only problem is that when I've got my tree structure fully populated we're talking in the order of 30million nodes and so around 100meg of extra overhead.

But pushing the objects in directly I get 30 million calls to the class destructor. Which actually might not be too bad. Increased processing overhead, but all the destructor does it modify some variables that monitor the internal size/state of the process.

Polyphemus_ · Mar 28th, 2006, 9:40 AM

Well, when pushing 30 million of objects 100 mB isn't that much in comparison (well, at least I guess so, unless one object is 2 bytes big). If you don't need pointers in this situation, you don't have to, remember that. When using tree structures, you often use inheritance etc. though (I don't know if you are using it) and you will need to use pointers.

I'm trying to make clear that the use of pointers and dynamic memory allocation depends on the situation.

grumpy · Mar 29th, 2006, 12:42 AM

OK; technically, if you are dynamically allocating the objects, there is an overhead of storing the pointer to it somewhere. As a std::vector typically allocates more memory than it needs (so, for example, it doesn't have to reallocate every time you add an object to it) the difference is usually insignificant. As The Dark said, you can use reserve() to work around that. In your example, if the vector reserves one element more than your required size, that's 160 bytes or the equivalent (on a typical 32 bit system) of 40 pointers.

Given that your collection of 30 million objects will take (roughly) 5 GB, a difference of 120MB in memory usage is insignificant ---whether you save it or not.

Yes, the vector<ClassName> invokes the ClassName destructor when you destroy the vector object. It also invokes copy constructor (or assignment operator depending on context) and the destructor when resizing. That doesn't happen with the vector<ClassName *> as a pointer has neither a destructor nor a constructor, BUT this approach guarantees you a memory leak unless you explicitly delete every instance.

I would describe what you're doing as a case of premature optimisation. It is better to implement your application as simply as possible, and be confident of it running correctly. In practice, for your application, I would actually prefer the vector<ClassName> approach. But NOT because it avoids the memory overhead of the vector<ClassName *> approach --- I would prefer it because it works correctly and you don't have to jump through hoops to avoid memory leaks. And I would accept the cost of constructor/destructor calls on resizing the array as a cost of having more chance of getting it to work correctly the first time.

Practically, I'd worry more about reducing the number of objects you need to keep in memory at any one time than I would about techniques to minimise the memory overhead of keeping 30 million objects in memory.

slyadams · Mar 29th, 2006, 2:32 AM

Thanks for your help guys!

Sorry, realised a made a blunder in one of my posts. 160bits is the size of the objects, not 160bytes. :o

I thought about ways to reduce the number of objects, but the problem is the data is storing rows from a database. This data is basically a 4 element data point (long,int,int,int) and the process needs to store loads of them to make them available for front end applications to query. Basically the process is a data cache and provides significantly quicker access than querying the database directly.

The process does work and has been in production for about a year now and I'm pretty sure doesn't have any leaks because the process is usually up for months on end, constantly reading new rows from the database and dropping old ones. We've just started an implementation for a new client who wants a 64bit platform. During this implementation I realised that the memory overhead of all these pointers will double.

I will write a copy constructor today and trace through the allocation and see what happens.

I will try and get around the vector resizing issue by looking at the impact of moving to a list. Although, the maximum length of a vector at the bottom of my tree is going to be 1440. With a object size of 160bits, this comes out at just under 30kB. Not that much at all, so maybe I don't need to worry about vector resizes.

grumpy · Mar 30th, 2006, 2:33 AM

Quote:

Originally Posted by slyadams

I thought about ways to reduce the number of objects, but the problem is the data is storing rows from a database. This data is basically a 4 element data point (long,int,int,int) and the process needs to store loads of them to make them available for front end applications to query. Basically the process is a data cache and provides significantly quicker access than querying the database directly.

That is normally the purpose of a cache. That doesn't mean the cache has to hold EVERYTHING though. For example, are there particular objects that are accessed frequently? Is there some other pattern to access (eg based on object X being accessed, is there a higher likelihood of some other object Y being accessed shortly afterwards)?

Quote:

Originally Posted by slyadams

The process does work and has been in production for about a year now and I'm pretty sure doesn't have any leaks because the process is usually up for months on end, constantly reading new rows from the database and dropping old ones. We've just started an implementation for a new client who wants a 64bit platform. During this implementation I realised that the memory overhead of all these pointers will double.

If the client has a 64 bit platform, odds are they won't be skimping on RAM anyway. One of the purposes of going 64 bit is that more RAM can be addressed. It might help if you asked the client how much RAM on the target machine can be assumed as a minimim. Then you can decide if you need to worry about memory usage or not.

Quote:

Originally Posted by slyadams

I will write a copy constructor today and trace through the allocation and see what happens.

All you really need to check is that the number of constructor calls matches the number of destructor calls. Keep in mind you need to track ALL constructors though.

Quote:

Originally Posted by slyadams

I will try and get around the vector resizing issue by looking at the impact of moving to a list. Although, the maximum length of a vector at the bottom of my tree is going to be 1440. With a object size of 160bits, this comes out at just under 30kB. Not that much at all, so maybe I don't need to worry about vector resizes.

A typical implementation of a list is as a linked list. If that's true, it means each node of the list contains your object, plus a pointer to another node in the list.

nnxion · Mar 29th, 2006, 2:59 AM

So what do you do if you want to push in some pointers to objects?
Say the object is 1 KB, you don't want to copy that 30 million times right?
What if you make a pointer to each of your objects and put that in your vector?

I think that's what the OP means.

Mar 28th, 2006, 7:26 AM	#1
slyadams Newbie Join Date: Mar 2006 Posts: 12 Rep Power: 0	Pushin an object into a vector Chaps, I have s system where I am pushing objects into a vector inside a function. At the moment I 'new' the object and push the pointer and it works fine. I was thinking that I might be able to save some memory space by pushing the object itself and not the reference. i.e. currently: void func1() { vector.push_back(new ClassName(param1,param2)); } possible: void func1() { vector.push_back(ClassName(param1,param2)); } The problem is that the destuctor for ClassName gets called everytime that func1 returns in the 2nd example. The problem is that the object in the 2nd example is static and its scope is limited to the function func1. So my question is: is there anyway to push objects dynamically into a vector without using pointers. The only reason I don't want to use pointers is that these vectors are at the bottom level of a tree structure and in a standard instance of the process will, in total, store about 30million objects. Thus, storing the pointer, as well as the data 'wastes' about 100meg. We are moving the process onto 64bit platforms for clients, so this overhead could, potentially, double! Thanks in advance for any help. P.S. :banana:

Mar 28th, 2006, 7:40 AM	#2
grumpy Programming Guru Join Date: Jun 2005 Location: Adelaide, South Australia Posts: 1,221 Rep Power: 5	I suggest you read the sticky post on "How to ask a question". It will give you tips on how to ask questions to increase your chances of getting a useful answer. I'm assuming the vectors in your two versions of func are of different types (eg the first is a std::vector<ClassName > and the second is a std::vector<ClassName>). The push_back() function of std::vector appends a COPY of the value it receives to the end of the vector. It doesn't matter whether the value it receives is a pointer (as in your first example) or an instance of a class (as in your second example), as long as the vector is of an appropriate type. All that is required to work with an instance of a class is that your class has a copy constructor and associated assignment operator that work as expected. If you use a std::vector<ClassName > and push_back new'd objects, you need to delete all objects in the vector before allowing the vector to be destroyed or pass out of scope. For example, the following are functionally equivalent (except the first works with dynamically allocated objects being pushed, and the second doesn't; (Toggle Plain Text) #include <vector> std::vector<ClassName > myvec; void func() { myvec.push_back(new ClassName(param1, param2)); } int main() { func(); // delete the object pushed by func() before allowing the vector to be destroyed delete myvec[0]; // myvec will pass out of scope and be released } #include <vector> std::vector<ClassName > myvec; void func() { myvec.push_back(new ClassName(param1, param2)); } int main() { func(); // delete the object pushed by func() before allowing the vector to be destroyed delete myvec[0]; // myvec will pass out of scope and be released } vs (Toggle Plain Text) #include <vector> std::vector<ClassName> myvec; void func() { myvec.push_back(ClassName(param1, param2)); } int main() { func(); // myvec will pass out of scope and be released, as will its elements } #include <vector> std::vector<ClassName> myvec; void func() { myvec.push_back(ClassName(param1, param2)); } int main() { func(); // myvec will pass out of scope and be released, as will its elements } PS Children will insist on playing with bananas

Mar 29th, 2006, 2:59 AM	#10
nnxion Programming Guru Join Date: Jun 2005 Location: elemental plane Posts: 1,429 Rep Power: 5	So what do you do if you want to push in some pointers to objects? Say the object is 1 KB, you don't want to copy that 30 million times right? What if you make a pointer to each of your objects and put that in your vector? I think that's what the OP means. __________________ "Employ your time in improving yourself by other men's writings, so that you shall gain easily what others have labored hard for." -- Socrates

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode

Mar 28th, 2006, 8:19 AM	#3
slyadams Newbie Join Date: Mar 2006 Posts: 12 Rep Power: 0	OK, but in the first example there is an increased memory overhead isn't there? Won't the Object be created in memory somewhere and the pointer pushed into the vector? Thus there is a memory overhead increase of sizeof(void *)?

Mar 28th, 2006, 8:38 AM	#5
slyadams Newbie Join Date: Mar 2006 Posts: 12 Rep Power: 0	The only problem is that when I've got my tree structure fully populated we're talking in the order of 30million nodes and so around 100meg of extra overhead. But pushing the objects in directly I get 30 million calls to the class destructor. Which actually might not be too bad. Increased processing overhead, but all the destructor does it modify some variables that monitor the internal size/state of the process.

Mar 28th, 2006, 9:40 AM	#6
Polyphemus_ Expert Programmer Join Date: Aug 2005 Location: Rotterdam, the Netherlands Posts: 942 Rep Power: 4	Well, when pushing 30 million of objects 100 mB isn't that much in comparison (well, at least I guess so, unless one object is 2 bytes big). If you don't need pointers in this situation, you don't have to, remember that. When using tree structures, you often use inheritance etc. though (I don't know if you are using it) and you will need to use pointers. I'm trying to make clear that the use of pointers and dynamic memory allocation depends on the situation.

Mar 29th, 2006, 12:42 AM	#7
grumpy Programming Guru Join Date: Jun 2005 Location: Adelaide, South Australia Posts: 1,221 Rep Power: 5	OK; technically, if you are dynamically allocating the objects, there is an overhead of storing the pointer to it somewhere. As a std::vector typically allocates more memory than it needs (so, for example, it doesn't have to reallocate every time you add an object to it) the difference is usually insignificant. As The Dark said, you can use reserve() to work around that. In your example, if the vector reserves one element more than your required size, that's 160 bytes or the equivalent (on a typical 32 bit system) of 40 pointers. Given that your collection of 30 million objects will take (roughly) 5 GB, a difference of 120MB in memory usage is insignificant ---whether you save it or not. Yes, the vector<ClassName> invokes the ClassName destructor when you destroy the vector object. It also invokes copy constructor (or assignment operator depending on context) and the destructor when resizing. That doesn't happen with the vector<ClassName > as a pointer has neither a destructor nor a constructor, BUT this approach guarantees you a memory leak unless you explicitly delete every instance. I would describe what you're doing as a case of premature optimisation. It is better to implement your application as simply as possible, and be confident of it running correctly. In practice, for your application, I would actually prefer the vector<ClassName> approach. But NOT because it avoids the memory overhead of the vector<ClassName > approach --- I would prefer it because it works correctly and you don't have to jump through hoops to avoid memory leaks. And I would accept the cost of constructor/destructor calls on resizing the array as a cost of having more chance of getting it to work correctly the first time. Practically, I'd worry more about reducing the number of objects you need to keep in memory at any one time than I would about techniques to minimise the memory overhead of keeping 30 million objects in memory.

Mar 29th, 2006, 2:32 AM	#8
slyadams Newbie Join Date: Mar 2006 Posts: 12 Rep Power: 0	Thanks for your help guys! Sorry, realised a made a blunder in one of my posts. 160bits is the size of the objects, not 160bytes. :o I thought about ways to reduce the number of objects, but the problem is the data is storing rows from a database. This data is basically a 4 element data point (long,int,int,int) and the process needs to store loads of them to make them available for front end applications to query. Basically the process is a data cache and provides significantly quicker access than querying the database directly. The process does work and has been in production for about a year now and I'm pretty sure doesn't have any leaks because the process is usually up for months on end, constantly reading new rows from the database and dropping old ones. We've just started an implementation for a new client who wants a 64bit platform. During this implementation I realised that the memory overhead of all these pointers will double. I will write a copy constructor today and trace through the allocation and see what happens. I will try and get around the vector resizing issue by looking at the impact of moving to a list. Although, the maximum length of a vector at the bottom of my tree is going to be 1440. With a object size of 160bits, this comes out at just under 30kB. Not that much at all, so maybe I don't need to worry about vector resizes.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Programming Forums > Application Development > C++

Pushin an object into a vector