From vincent at nexedi.com  Tue Aug 28 11:17:47 2012
From: vincent at nexedi.com (Vincent Pelletier)
Date: Tue, 28 Aug 2012 11:17:47 +0200
Subject: [Neo-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and
 redundant storage for ZODB
In-Reply-To: <20120827143737.2bc1a837@vincent-tkpad>
References: <20120827143737.2bc1a837@vincent-tkpad>
Message-ID: <20120828111747.1934fc60@vincent-tkpad>

On Mon, 27 Aug 2012 14:37:37 +0200,
Vincent Pelletier <vincent at nexedi.com> wrote :
> Under the hood, it relies on simple features of SQL databases

To make things maybe a bit clearer, from the feedback I get:
You can forget about SQL presence. NEO usage of SQL is as a relational
as a handful of python dicts is. Except there is no way to load only
part of a pickled dict, or do range searches (ZODB's BTrees are much
better in this regard), or writable to disk atomically without having to
implement this level of atomicity ourselves.

Ideally, NEO would use something like libhail, or maybe even simpler
like kyotocabinet (except that we need composed keys, and kyotocabinet
b-trees have AFAIK no such notion).
SQL as a data definition language was simply too convenient during
development (need a new column ? easy, even if you have a 40GB table),
and it stuck - and we have yet to find a significant drawback to
implement a new storage backend.

As a side effect, SQL allows gathering some statistics over the data
contained in a database very efficiently. Number of current objects,
number of revisions per object, number of transactions, when
transactions occured in base history, average object size, largest
object, you name it.

-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises


From mj at zopatista.com  Tue Aug 28 16:31:20 2012
From: mj at zopatista.com (Martijn Pieters)
Date: Tue, 28 Aug 2012 16:31:20 +0200
Subject: [Neo-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and
 redundant storage for ZODB
In-Reply-To: <20120827143737.2bc1a837@vincent-tkpad>
References: <20120827143737.2bc1a837@vincent-tkpad>
Message-ID: <CAJkXj+af7PJCSYpfZVg7brH5ZCSyDLKGXoDOn+6=zB-bzq2UUQ@mail.gmail.com>

On Mon, Aug 27, 2012 at 2:37 PM, Vincent Pelletier <vincent at nexedi.com> wrote:
> NEO aims at being a replacement for use-cases where ZEO is used, but
> with better scalability (by allowing data of a single database to be
> distributed over several machines, and by removing database-level
> locking), with failure resilience (by mirroring database content among
> machines). Under the hood, it relies on simple features of SQL
> databases (safe on-disk data structure, efficient memory usage,
> efficient indexes).

How does NEO compare to RelStorage? NEO appears to implement the
storage roughly in the same way; store pickles in tables in a SQL
database.

Some differences that I can see from reading your email:

* NEO takes care of replication itself; RelStorage pushes that
responsibility to the database used.
* NEO supports MySQL and sqlite, RelStorage MySQL, PostgreSQL and Oracle.
* RelStorage can act as a BlobStorage, NEO can not.

Anything else different? Did you make any performance comparisons
between RelStorage and NEO?

-- 
Martijn Pieters


From vincent at nexedi.com  Tue Aug 28 18:31:05 2012
From: vincent at nexedi.com (Vincent Pelletier)
Date: Tue, 28 Aug 2012 18:31:05 +0200
Subject: [Neo-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and
 redundant storage for ZODB
In-Reply-To: <CAJkXj+af7PJCSYpfZVg7brH5ZCSyDLKGXoDOn+6=zB-bzq2UUQ@mail.gmail.com>
References: <20120827143737.2bc1a837@vincent-tkpad>
	<CAJkXj+af7PJCSYpfZVg7brH5ZCSyDLKGXoDOn+6=zB-bzq2UUQ@mail.gmail.com>
Message-ID: <20120828183105.20330a53@vincent-tkpad>

On Tue, 28 Aug 2012 16:31:20 +0200,
Martijn Pieters <mj at zopatista.com> wrote :
> Anything else different? Did you make any performance comparisons
> between RelStorage and NEO?

I believe the main difference compared to all other ZODB Storage
implementation is the finer-grained locking scheme: in all storage
implementations I know, there is a database-level lock during the
entire second phase of 2PC, whereas in NEO transactions are serialised
only when they alter a common set of objects.
This removes an argument in favour of splitting databases (ie, using
mountpoints): evading the tpc_vote..tpc_finish database-level locking.

Also, NEO distributes objects over several servers (aka, some or all
servers might not contain the whole database), for load balancing/
parallelism purposes. This is not possible if one relies on relational
database replication alone.

I forgot in the original mail to mention that NEO does all conflict
resolutions on client side rather than server side. The same happens in
relStorage, but this is different from ZEO. Packing on client side
makes it easier to get the setup right: with ZEO you will get more
conflicts than normal if it cannot load some class which implements
conflict resolution, and this might go unnoticed until someone worries
about a performance drop or so. With client-side resolution, if you
don't see Broken Objects, conflict resolution for those classes works.

Some comments on some points you mentioned:
> * NEO supports MySQL and sqlite, RelStorage MySQL, PostgreSQL and
> Oracle.

It should be rather easy to adapt to more back-ends.
We (Nexedi) are not interested in proprietary software, so we will
probably not implement Oracle support ourselves. For PostgreSQL, it's
just that we do not have a setup at hand and the experience to
implement a client properly. I expect that it would not take more than a
week to get PostgreSQL implemented by someone used to it and knowing
python, but new to NEO.

Just to demonstrate that NEO really does not rely on fancy features of
SQL servers, you may dig in older revisions in NEO's git repository. You
can find a "btree.py"[1] test storage, which is based on ZODB.BTree
class. It was just a toy, without persistence support (I initially
intended to provide it, but never finished it) and hence limited by
the available amount of RAM. But it was otherwise a fully functional NEO
storage backend. I think it took me a week-end to put it together,
while discovering ZODB.Btree API and adapting NEO's storage backend
API along the way (this was the first non-MySQL backend ever
implemented, so API was a bit too ad-hoc at that time).

sqlite was chosen as a way to get rid of the need to setup a
stand-alone SQL server in addition to NEO storage process. We are not
sure yet of how well our database schema holds when there are several
(10+) GB of data in each storage node.

> * RelStorage can act as a BlobStorage, NEO can not.

I would like to stress that this has nothing to do with design, rather
it's just not implemented. We do not wish to rely on filesystem-level
sharing, so we consider something along the lines of providing a
FUSE-based to share blob storage, which then can abstract the blobs
being distributed over several servers. This is just the general idea,
we don't have much experience with blob handling ourselves (which is
why we preferred to leave it asides rather than providing an
unrealistic - and hence unusable - implementation).

[1]http://git.erp5.org/gitweb/neoppod.git/blob/75d83690bd4a34cfe5ed83c949e4a32c7dec7c82:/neo/storage/database/btree.py

Regards,
-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises