[Erp5-dev] testing with transactions and activities

[jp at nexedi.com]

Hi,

There are at least two kinds of tests:
  1- those related to testing a functionality
  2- those related to testing what happens with many users

When we test a functionality, we want to execute tests in a well defined order, with explicit transactions. Then we sometimes reorder that order of inidvidiaul tests to cover differen scenarii, including those you mention. Long ago, we had the notion of "sequence" object to try to generate sequences of steps in somehow random order. But we found by experience that it was not so useful and that the current way could cover all issues we had.

When we test what happens with many users, we use a different framework (mechanize) to see what happens when 300 concurrent users are using the system.

We only do the second kind of test after the first kind of test is made. It is important not to mix them.

The "mechanize" tests are usually not published because they cover customer specific trade secrets. But it would be good to add some of the second type tests to the core of ERP5, at least for sharing knowledge on how we make them. I think we can plan that in our roadmap for the next 6 months since we got a budget recently to implement a kind of system of this type in another field.

Regards,

JPS.


> I recently found to serious issues which seem to have slipped through
> unnoticed, despite a massive test suite running every night, testing
> every corner of ERP5. I'm beginning to have an impression that there is
> a fundamental shortcoming in the whole testing system. Every test I've
> seen (disclaimer: there are plenty of tests which I have not seen, so I
> may be completely wrong here) explicitly commits a transaction and then
> executes self.tic() after anything it did. So there is no way it could
> possibly detect things like the two I reported recently. Maybe apart
> from testing like:
> 
> self.doSomething()
> get_transaction().commit()
> self.tic()
> ...check...
> 
> we should also test like:
> 
> self.doSomething()
> self.tic()
> get_transaction().commit()
> self.tic()
> ...check...
> 
> and
> self.doSomething()
> get_transaction().commit()
> self.doAnotherThing()
> get_transaction().commit()
> self.tic()
> ...check...
> 
> This would bring the test scenarios closer to a real-life, multi-user,
> heavily used environment.
> 
> Bartek
> _______________________________________________
> Erp5-dev mailing list
> Erp5-dev at erp5.org
> http://mail.nexedi.com/mailman/listinfo/erp5-dev
>