[Erp5-dev] ERP5 Release System - proposal

[Yoshinori Okuji]

Hello Łukasz,

On Thursday 27 September 2007 12:21, Łukasz Nowak wrote:
> We present our idea how to organise release management for ERP5. We
> would like to know your opinion about this idea. Any constructive
> comments as well as criticism are welcome. We are making this proposal
> in good faith - to provide the best possible solution in implementing
> and development of ERP5 system, to give as much as possible possibility
> to arise community of skilled developers around such great project as
> ERP5 is itself.

First of all, I would like to thank you for writing this proposal. I highly 
appreciate that you have made significant effort towards a better maintenance 
process.

Let me describe how Nexedi has been maintaining the code, and succeeding in 
implementing customers' projects so far:

- All the code base, including Business Templates, unit tests, functional 
tests, and many other types of tools, required for generic ERP5, has been 
always published on the subversion repository. No permission is required to 
access the source code at any time. Everything is immediately open to anybody 
in the world freely. This way, more people can take part in the reviewing 
process, and facilitating contribution. Also, developers recognize more 
responsibility for the quality, because of the potential risk.

- All the test results are sent to the public mailing list erp5-report every 
day. People are free to check for the current status in the cutting-edge 
development, thus it is easy to determine the latest stable revision from the 
history of reports. If a result is good, but the system is still unstable, 
this only means that we lack tests. Such tests should be complemented as soon 
as possible, and contribution is very welcome.

- Chanages are, in principle, peer-reviewed. Only trivial changes, such as 
fixing a typo, do not require such a review. Some people are not allowed to 
directly check in changes, when we still don't know the quality level of code 
produced by them. Once a person proves that she has a good skill, we give 
more permissions to her, so that we will not prevent her work. However, even 
experienced developers, such as me and Jean-Paul, need a peer-review, before 
committing a significant change. This way, although it may not be perfect, 
the quality of software is kept acceptally good all the time. Note that I 
have been using the same policy in another Free Software project for many 
years successfully.

- Backward compatibility is respected all the time. We break backward 
compatibility, only if it is inevitable. Incompatibility is an evil, because 
it produces too much work at a time when upgrading a system. We have been 
publishing the information on incompatibility we made in the public wiki at 
www.erp5.org, so that others will not get troubled. Still, we are always 
trying to reduce incompatibility, and only deprecate old usage.

- Code base used for a customer's project is taken out of the public 
subversion repository, under the control by a project manager. A decision is 
made mostly based on the status of unit tests and functional tests, but also 
with reviwing recent changes of the code, because some kind of problems may 
not be very visible only with automatic testing. Then, the code base is 
upgraded from time to time (the frequency depends on the constraints in a 
project) so that bug fixes and feature enchancement are incorporated. Again, 
the decision of which revision should be taken is based on the same criteria 
as above, as well as project conditions in some cases. This way, we can 
implement a stable system without much trouble.

As the technical leader of ERP5, I have been thinking of how to make releases 
for some years, in a way that the maintenance cost would be negligible, and 
woouldn't disturb development and contribution. The important questions are: 
When and How. I consider that there is no question about "Why". As you 
pointed out, releases make things more apparent and visible, so they are 
definitely nice to have.

Here are some of my ideas:

- When

In my understanding, there are three different ways to manage the timings of 
making releases.

One of them is feature-basis. When something significant is made, a new 
release is published. This is often used in proprietary software products for 
marketing purpose. Then, bug fixes are provided only in the form of patches. 
In Open Source development, this does not make sense very much, since patches 
can be included from the beginning. Instead, feature enhancement increments a 
major version. For now, I am not talking about how to name versions, so I 
don't take this way into consideration any more.

Another is regularity-basis. If I understand correctly, this is the same as 
your suggestion. In this scheme, releases are made in the same frequency, no 
matter whatever is happening. The advantage is that it is guaranteed that the 
realtime characteristics of releases against the head is quite good, that is, 
the distribution of delays behind the head is minimized. However, this way 
has a critical drawback that it does not reflect the reality of development. 
Sometimes a significant amount of development is performed in a short term, 
and sometimes not. As the nature of development, changes, in particular, 
which address new problems, tend to make software less stable. I will discuss 
this issue more below.

The other is quality-basis. A new release is made when the head has good 
quality of code. This is rather the reverse of the regularity-basis, so the 
pros/cons are also reversed. This proceeds with the reality of development, 
thus a delay can be potentially huge, but the quality of a release is well 
maintained.

Of course, in most projects, these ways are more or less mixed. Even if the 
quality-basis approach is taken, it does not make sense to make a new release 
every day, only because the quality is high. Even with the regularity-basis 
one, if the quality cannot meet criteria on time, a delay must be inserted to 
prevent catastrophy. Many factors should be taken into consideration to keep 
good release management.

As you might know, I have +10-year experience in many other Open Source/Free 
Software projects. And, I have tried both the regularity-basis and 
quality-basis ways. Overall, I have found that the quality-basis works far 
better, because, for example, when a problem can be resolved appropriately by 
a certain few people, and if they are not available for some reason for a 
while, the regularity gets broken so easily. Enforcing a release quickly, 
simply because of the policy, only produces unusable software.

Also, I have found that release management is far from trivial. Of course, as 
you suggested, we use the result of tests for making estimate on stability, 
but tests may not cover everything. It is sometimes because not all tests are 
implemented yet. It is sometimes because managers are not capable of 
determining whether software is stable enough or not. Especially in the 
preparation of a new release, bug fixes must be encouraged, but contributors 
have their own time constraints, and may not always follow a release cycle. 
In this case, managers must be able to define what bug fixes can be postponed 
with the idea of possible danger, and address problems by themselves, 
whenever necessary.

Another perspective is how to facilitate people to contribute. Usually, 
excellent developers want good feedback from managers. If they feel that 
managers fully understand their changes, and evaluate the contribution 
appropriately, they tend to help managers. Otherwise, they stop. This is not 
related to whether they are paid or not. For instance, although workers at 
Nexedi are paid, they are not always paid to contribute publicly. This part 
really depends on their feeling and will. If they think that managers would 
not understand their effort, they will just put changes locally and hide 
them. This has been always the same as in volunteer-based projects, so I am 
rather sure. Therefore, I believe that highly skilled persons must be 
selected as release managers for ERP5. Otherwise, release management will 
corrupt.

- How

There are two matters to consider: how many branches should be maintained or 
developed, and how long a branch should be maintained.

The simplest approach is to use a single trunk, and no other branch. This is 
the current way. The benefit is that all the effort heads towards one place, 
so the efficiency of development is very high. We do not need to worry about 
forking. The drawback is that the quality can be random along the time, no 
matter however much effort is put into stabilization.

The semi-simple approach is, as you proposed, to use a main trunk, and one 
branch to keep stability. And, to abandone older branches, once a new branch 
is created. This is somehow easy and realistic. The benefit is that, without 
having a lot of maintenance cost, stability is more or less ensured on a 
branch. The drawback is that this way additionally requires porting changes 
between a trunk and a branch, thus the development energy is scattered, and 
extra effort and time are required for the maintenance.

For now, I compare only these two. Again, I have experience with both 
approaches in the past. In short, I met difficulty in the latter approach, 
because the cost of porting patches was quite higher than I expected. After 
all, I gave it up, and I chose a single trunk method.

The way I did was the following: when I started to feel that I wanted to make 
a new version, for example, because we had a good set of new functionality, I 
forced people to stop writing a new feature, but encouraged people to test 
the head, and fix it. In the meantime, all patches sent to me were waited, 
and kept pending. Once I was satisified with the quality of code, I put a tag 
and release it. Then, I restarted the normal cycle of development, and 
integrating pending contribution.

Actually, this was so effective. All contributors understood that this was 
required, and accepted that patches were in a pending state. Besides stopping 
people to modify the head brutally, no extra cost was raised.

Surely, we must analyze if this way is effective for ERP5 as well, but I feel 
that this is more appropriate.

The reason is really the cost. Unfortuantely, nobody is a superman. I don't 
think it is a bad thing to have a stable branch, but at the same time, I 
doubt if it is feasible. If we use branching, another team for a branch would 
be necessary to be formed, just like the security team in Debian, because the 
backporting is surprisingly hard and time-consuming.

If anybody wants to perform the backporting job, I would appreciate very much. 
But be prepared that it is not trivial at all. One must be able to say what 
is a pure fix, and must be able to segregate safe parts of patches from a 
trunk, and adapt API usage, and so on. For now, I don't think this is 
feasible with the current size of our community, but I would be very happy if 
I am wrong.

So, at the moment, my suggestion is to use a single trunk. We have proven that 
writing tests for a good coverage makes projects successful, because tests 
always notify us of the current status, and what must be fixed, without any 
branching. Release managers should promote people to write more tests for 
missing parts, and negotiate with developers when next release should be 
made, so that others can collaborate.

Now, this mail is getting too long, but let me describe some more additional 
notes.

- It is a very good idea to describe API changes. We already do upgrade notes, 
but this is a nice complement.

- Upgrade tests must be automated very well, otherwise it is too costly. Not 
only unit tests, but also functional tests must be performed. Currently, the 
number of functional tests is not good enough, in particular about 
configuration-related stuff. Contribution is very welcome.

- Release management may not be independent of development management. Both 
are very tightly coupled, because releasing depends on the status of 
development. Only experts can perform this kind of job, so that other 
developes will listen to their advise.

- I myself prefer co-maintenance, that is, having multiple maintainers. This 
has a significant advantage in that even if one is not available (sick, busy, 
or whatever), the project will not be stalled. I was proposed to be a 
co-maintainer in a project, and it worked very well, so I'm proposing others 
to do the same with me these days. I like it. For ERP5, 2-3 persons will be 
good. Having too many is no good.

I hope my message would be useful for further discussion. Please let me know 
what you think.

Regards,
YO
-- 
Yoshinori Okuji, Nexedi CTO
Nexedi: Consulting and Development of Free / Open Source Software
http://www.nexedi.com
ERP5: Full Featured High End Open Source ERP
http://www.erp5.com
ERP5 Wiki: Developer Zone for ERP5 Community
http://www.erp5.org