Login | Register
My pages Projects Community openCollabNet

Discussions > dev > Re: [propel-dev] PropelResultset Iterator

propel
Discussion topic

Back to topic list

Re: [propel-dev] PropelResultset Iterator

Reply

Author Alan Pinstein <apinstein at mac dot com>
Full name Alan Pinstein <apinstein at mac dot com>
Date 2006-10-04 08:13:23 PDT
Message > 1) I just realized how much overhead we have when fetching results:
>
> while($row = $stmt->fetch(PDO​::FETCH_NUM)) {
> $obj = new $cls();
> $obj->hydrate($row);
> $results[] = $obj;
> }
>
> We can use PDO::FETCH_CLASS here. PDO will then create a new "Book"
> instance and set the fields. We can even pass constructor
> arguments, very schweet. I think I'll test this later. This is a
> non-breaking change for 1.3 that should make things a bit faster.

Hydrating is definitely slow, esp. b/c of creole's incessant type
checking and the array_key_exists issue (which I fixed for PGSQL, but
would still be very slow for the other drivers).

Certainly a C implementation of this would be a great improvement.
Seems that you can also specify the class to be created. My only
concern would be making sure that the constructor is properly called,
but I'd assume it would be.

This looks great.

> 2) Regarding this iterator thing. I guess the best idea would be to
> use PDO to populate the results, basically as above. I don't think
> we need FETCH_INTO, though. Consider the following situation:
>
> foreach (BookPeer::doSelect($c) as $book) {
> echo $book->getTitle();
> }
>
> With an iterator, each book objekt would be created and returned on
> the fly. I realize that there's no way to force the GC to kick in,
> but after some rows, it will discard the "old" objects still in
> memory (i.e. those who'se titles have been echoed already).

Actually, I think that GC in php is not an out-of-band process...
from my testing, as soon as the last reference to an object goes out
of scope, it gets GC'd. PHP has no threads, so without an "inline"
GC, GC would basically be pointless. It works very differently from
Java GC.

All the iterator would do is simply defer hydration until the row is
accessed. Also, in this case we wouldn't keep an internal array of
objects, o/w the individual rows wouldn't be GC'd due to the internal
reference.

The use case should be:

1. If you really want an array of all results, use doSelect() and get
the completely populated array (fine for known small data sets)
2. If you are fetching a *huge* data set and want to be thrifty with
memory, use a new doSelectPropelRS, and use the iterator interface,
which lazy-hydrates. I suppose we could add ArrayAccess interface,
too, if the underlying RS supports seeking. Otherwise you'd have to
iterate it only. You would know that PropelRS do not keep an internal
list of the results, thus being memory-thrify (or maybe we make it an
option to the PropelRS constructor?).

Something along those lines... we'd need to get more use cases from
users before deciding for sure.

If you are coding efficiently in general, these are basically the 2
camps that all DB access falls into; unbounded, and bounded. Pretty
much all web pages will be bounded (using some kind of pagination
system) and thus work fine as-is (although faster hydration due to
PDO will be nice). Usually it's scripts that operate in unbounded
mode, and usually in this case you are only working with 1 row at a
time anyway. Any other need is definitely application-specific, and
in this case you can code your own or make sure you have enough
memory to do what you need to.

> I don't really see a need for FETCH_INTO there!?

Me neither!

Alan

>
>
> David
>
>
>
> Am 04.10.2006 um 11:56 schrieb Sven Tietje:
>
>> David Zulke wrote:
>>> 1) memory consumption is a lot lower, e.g. when iterating over 1000
>>> rows to display them, a row is fetched, the data is displayed, and
>>> the GC can then go ahead and destroy the object.
>>
>> Think, that should be an option. Sometimes, i want do iterator over a
>> resultset and to write baseobjects with special information into
>> an array or
>> something like that. PDO handles it the same way - Fetchmode gives
>> you
>> special options for fetching as object:
>>
>> http://de.php.net/ma​nual/de/function.pdo​statement-setfetchmo​de.php
>> Using PDO::FETCH_CLASS will result into an object-instance for
>> each row: 200
>> rows in your resultset will give you 200 object-instances.
>>
>> PDO::FETCH_INTO will give you the same object-instance for each
>> row: 200
>> rows, but one object-instance.
>>
>> Having the same feature / option for propel would be nice:
>>
>> 1. One Instance for all rows
>> $resultset = TablePeer::doSelect(new Criteria(), PROPEL::ONE_OBJECT);
>>
>> foreach ($resultset as $row) {
>> echo $row //will give you the same object instance
>> }
>>
>> I would use PROPEL::ONE_OBJECT to publish big resultsets. I don`t
>> manipulate
>> the object. With $myrow = clone $row it is possible to get my own
>> instance
>> of the row.
>>
>> 2. An Instance for each row
>> $resultset = TablePeer::doSelect(new Criteria(),
>> PROPEL::EACH_OBJECT);
>> foreach ($resultset as $row) {
>> echo $row //will result in Object#1, Object#2 ..... Object#200
>> }
>>
>> Internally Propel uses clone() to create a single instance of each
>> row.
>>
>>> 2) it opens the door for a potential
>>> CategoryPeer::doSele​ctJoinProducts() support, where fetching 1:n
>>> relations with a join would become possible.
>>
>> We should discuss to use
>> PDOStatement::setFetchMode ( int PDO::FETCH_CLASS, string
>> classname ) or
>> PDOStatement::setFetchMode ( int PDO::FETCH_INTO, object object )
>> instead of hydrate();
>>
>> I wrote a little script to test it:
>>
>> $stmt = $pdo->prepare('select name as `customer.name` from
>> customer');
>> $stmt->execute(array());
>>
>> class baseobject /*implements ArrayAccess*/ {
>> private $data = array(
>> 'customer.name' => null,
>> );
>>
>> // for hydrate
>> public function __set($key, $value) {
>> $this->data[$key] = $value;
>> }
>>
>> public function setName($value) {
>> $this->data['customer.name'] = $value;
>> }
>>
>> public function getName() {
>> if (isset($this->da​ta['customer.name'])​) {
>> return $this->data['cus​tomer.name'];
>> }
>>
>> }
>> }
>>
>> $blub = new baseobject;
>> $stmt->setFetchM​ode(PDO::FETCH_INTO,​ $blub);
>>
>> $arg = array();
>> while ($b = $stmt->fetch()) {
>> // $b->setName('blah');
>> var_dump($b->getName());
>> }
>>
>> Implement ArrayAccess into an BaseObject could be a possibility,
>> too. We
>> should discuss it.
>>
>> Your opinion? I think to use the power of pdo would be nice.
>>
>> --------------------​--------------------​--------------------​---------
>> To unsubscribe, e-mail: dev-unsubscribe@prop​el.tigris.org
>> For additional commands, e-mail: dev-help at propel dot tigris dot org
>>
>>
>
> --------------------​--------------------​--------------------​---------
> To unsubscribe, e-mail: dev-unsubscribe@prop​el.tigris.org
> For additional commands, e-mail: dev-help at propel dot tigris dot org
>

« Previous message in topic | 26 of 26 | Next message in topic »

Messages

Show all messages in topic

PropelResultset Iterator Sven Tietje <tietje at topconcepts dot de> Sven Tietje <tietje at topconcepts dot de> 2006-10-02 01:43:05 PDT
     Re: [propel-dev] PropelResultset Iterator =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> 2006-10-02 02:04:45 PDT
     Re: [propel-dev] PropelResultset Iterator =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> 2006-10-03 05:13:38 PDT
         Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-03 06:07:14 PDT
         RE: [propel-dev] PropelResultset Iterator Sven Tietje <tietje at topconcepts dot de> Sven Tietje <tietje at topconcepts dot de> 2006-10-04 02:56:24 PDT
             Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-04 07:15:25 PDT
                 RE: [propel-dev] PropelResultset Iterator Sven Tietje <tietje at topconcepts dot de> Sven Tietje <tietje at topconcepts dot de> 2006-10-04 07:48:00 PDT
                     Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-04 07:52:18 PDT
                         Re: [propel-dev] PropelResultset Iterator hlellelid Hans Lellelid 2006-10-04 07:56:14 PDT
                             Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-04 07:58:28 PDT
                             Re: [propel-dev] PropelResultset Iterator =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> 2006-10-04 08:00:10 PDT
                                 Re: [propel-dev] PropelResultset Iterator hlellelid Hans Lellelid 2006-10-04 08:05:23 PDT
                                     Re: [propel-dev] PropelResultset Iterator hlellelid Hans Lellelid 2006-10-04 08:09:56 PDT
                                         Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-04 08:15:54 PDT
                                             Re: [propel-dev] PropelResultset Iterator hlellelid Hans Lellelid 2006-10-04 08:19:39 PDT
                             RE: [propel-dev] PropelResultset Iterator Sven Tietje <tietje at topconcepts dot de> Sven Tietje <tietje at topconcepts dot de> 2006-10-04 08:06:19 PDT
                                 Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-04 08:25:56 PDT
                                     Re: [propel-dev] PropelResultset Iterator =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> 2006-10-04 08:31:18 PDT
                                     Re: [propel-dev] PropelResultset Iterator Sven Tietje <mail at sven-tietje dot de> Sven Tietje <mail at sven-tietje dot de> 2006-10-04 11:11:24 PDT
                                         Re: [propel-dev] PropelResultset Iterator =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> 2006-10-04 11:20:35 PDT
                                             Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-04 11:29:47 PDT
                                                 Re: [propel-dev] PropelResultset Iterator Sven Tietje <mail at sven-tietje dot de> Sven Tietje <mail at sven-tietje dot de> 2006-10-04 12:26:37 PDT
                                                     Re: [propel-dev] PropelResultset Iterator Alan Pinstein <apinstein at mac dot com> Alan Pinstein <apinstein at mac dot com> 2006-10-04 12:47:23 PDT
             Re: [propel-dev] PropelResultset Iterator =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> =?ISO-8859-1?Q?David_Z=FClke?= <dz at bitxtender dot com> 2006-10-04 07:57:22 PDT
                 RE: [propel-dev] PropelResultset Iterator Sven Tietje <tietje at topconcepts dot de> Sven Tietje <tietje at topconcepts dot de> 2006-10-04 08:11:30 PDT
Page: of 2 « Previous | Next »
Messages per page: