[Erp5-dev] Proposal for CSV import/export enhancements

[Łukasz Nowak]

Hello Jerome,

On 2007-10-17, 12:02:29
Jérome Perrin <jerome at nexedi.com> wrote:

> Łukasz Nowak a écrit :
> > Hello,
> > 
> > While playing with production we are able to import BOMs and many
> > other useful information from various CAD/CAM/etc programs using
> > CSV.
> > 
> > As CSV is quite...hm...undefined, there are many problems. Many
> > programs use different implementation of CSV - some of them use ";"
> > as separator some ",". Some use '"' as quote mark, some "'". Some
> > use UTF-8 encoding, some other ASCII, and so on. So, IMHO,
> > importing of CSV might be two-phase system. First one upload file,
> > second one choose some attributes and provide user special listbox,
> > which will "simulate" how will system understand provided CSV.
> > Something similar to OpenOffice CSV import/export dialog.
> 
> Hi,
> 
> As you pointed out, despite looking simple, csv is in reality
> extremly complex (you can look at python csv module implementation),
> that's one more reason why we concentrate on importing from
> OpenOffice. It still needs improvement, but here's what we have today:
> 
> - Import of categories from a spreadsheet (Import/Export action from 
> portal_categories). This uses oood to convert this input file, if for 
> instance you upload a .xls file. I haven't try with CSV, I don't know
> if oood can convert from CSV.
> 
>   - There is an interesting prototype on person module: "Import
> Persons from OpenOffice Calc".
> This tool propose to map openoffice column to document properties. As 
> far as I know it doesn't work for categories, and probably have other 
> problems, but it's IMHO a good start.
> 
> As far as I know this is the current status of OOo import in ERP5. If
> we want better csv support, one way might be to add more csv to ods /
> ods to csv support directly in oood, by "normalizing" csv in python
> using csv module. This is just an idea, I haven't tried it at all,
> and I'm not sure that information like "this column is numeric" can
> be kept with csv only anyway ...

OK. Understood. I'd very like to help with improving another way of
importing data into ERP5 from external systems. But I do not have
starting point, working example, or anything for now. I _think_ I
*feel* general idea, but I might be wrong. I'll _try_ to reference to
unit tests, but any nice working example what is working right now
would speed up my learning of those aspect and my future contributions.

As for now I've prepared some "intelligence" (or stupidity ;) ) to CSV
related imports, those are attached as .tar. As going to have any user
interaction is quite hard (I cannot do it as I thought, and I'll wait
until work on using ODS will be ready) and it is possible to figure out
what is quote and separator when you have CSV file I done it. Please
treat is as proof of concept, but it will be used (and roughly tested)
in our environment. Data used for tests are real data with some
scramblification applied.

Regards,
Luke

-- 
Łukasz Nowak R&D Ventis http://www.ventis.com.pl/
tel: +48 32 768 16 85 fax: +48 32 392 10 61
``Use the Source, Luke...''
I am only craftsman. Maybe good one, but only craftsman.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CSV-magic.tar.gz
Type: application/x-gzip
Size: 3525 bytes
Desc: not available
URL: <http://mail.tiolive.com/pipermail/erp5-dev/attachments/20071017/4da2e3f1/attachment.bin>