[Announce] libzdb 2.1

[Jan-Henrik Haukeland]

On 3. mars. 2008, at 09.44, Paul J Stevens wrote:

> Jan-Henrik Haukeland wrote:
>> On 2. mars. 2008, at 22.09, Paul J Stevens wrote:
>>
>>> I've been thinking about this. This implies that you can't use
>>> multiple
>>> statements within a single transaction (at least with sqlite). Is  
>>> that
>>> correct? Because that would make code like below invalid, right?
>>
>> Not necessarily, though a fair chance that it will not work. The
>> indicator is this SQLite error message "SQL statements in progress".
>
> I am aware of that caveat in the docs. Is that a limitation of  
> sqlite per se? Or
> rather how libzdb talks to sqlite?

I believe it is a combination. Using PreparedStatement_executeQuery()  
the prepared statement is reused in the ResultSet and not closed. One  
place where this is a problem is in an explicit transaction. Upon a  
commit or rollback, at least SQLite check for open statements and  
issue a "SQL statements in progress" error, unless Connection_clear()  
is called to explicit close the statement.

For Connection_execute() and Connection_executeQuery() which also use  
a statement internally, the statement is closed upon the next call to  
any of these methods and therefor does not have this problem. But for  
PreparedStatement_executeQuery() the same internal statement is  
reused, and since a prepared statement is longer lived [1] than a  
ResultSet it may be a problem. A better option than to reuse, is  
probably to create a new statement for the ResultSet when  
PreparedStatement_executeQuery() is called, though this need more  
research which I haven't had time to prioritize. Until then, a  
workaround is available by calling Connection_clear() at appropriate  
places to close prepared statements explicit.

> The reason I'm asking is because I'm still trying to get a feel for  
> libzdb's
> behaviour and implicit expectations. A few things I've noticed:
>
> - I can't use Connection_PreparedStatement on a postgresql  
> connection until I've
> issued a Connection_executeQuery. Doing so segfaults in libzdb. I've  
> reported
> this one already, and I can easily work around this.

Its on my TODO list and I hope to have it moved over to Martin's TODO  
list :)

> - I'm getting stack corruption and segfaults in certain hotspots in  
> my code that
> use glib's GObject interface throught the GMime code. Now is GObject  
> a known
> not-threadsafe interface, and putting mutexes around my calls solves  
> the
> segfaults. But thing is: my code is (still) single-threaded (apart  
> from the
> reaper thread). So apparently thread semantics apply even to single- 
> threaded
> apps when using libzdb. If that is true, I need to start studying on  
> threads
> pronto because I havent a clue as to how thread(un)safe my code is  
> at the moment.

Linking with pthread makes your application threaded, per def, *but*  
as long as you only use one thread, i.e. the application main-thread  
you should not need to consider yourself with tread-safeness nor  
synchronizing shared resources. This is also the case if you run  
libzdb with a reaper thread since it does not affect any of your  
application data structures.

Thread-safe is one thing, reentrant is another and since I know you  
are using libevent, you may also want to check your event handlers.  
Even a single threaded application may have reentrancy problems -  
think signal handlers, and using libevent may introduce reentrancy  
problems in your code which may look like a thread bug. Maybe linking  
with pthread/libzdb exposed these bugs, but I don't think its the root  
cause.


> - Valgrind is generating massive amounts of 'invalid read' warnings  
> in the
> ResultSet_getXXX code. Whether this results in segfaults is  
> dependent on the
> backend. The mysql driver appears to segfault where the postgresql  
> driver simply
> returns null values. I havent yet tested sqlite. I'm left wondering  
> if this too
> is thread related.

Well, this is unexpected. Not that valgrind issues lots of warnings,  
but that you get SIGSEGV from using ResultSet_getXXX. Sounds like a  
Heisenbug (http://en.wikipedia.org/wiki/Heisenbug) someplace.  
Hopefully not in libzdb. One thing to check is  
PreparedStatement_setXXX. String or blob values set in a  
PreparedStatement_setXXX method are set by *reference* and if those  
references has disappeared somehow before doing a  
PreparedStatement_execute.. there will be Heisenbugs.

> However, since I only just finished the code changes that de- 
> globalize the
> result and connection pointers (leaving only the pool pointer as a  
> global) I
> havent had enough time yet to do some thorough backtracking.
>
> keep you posted.


Appreciated!

[1] A prepared statement is closed when its parent Connection is  
returned to the pool. Though for efficiency it should live longer.  
libzdb should/may in the future internalize and cache them. I have not  
verified this, but I hope the database backends already does this.