[Erp5-dev] Proposal for CSV import/export enhancements

[Brice LEROY]

Hello,
  To continue about OOorg import, I just finish to import with this tool more 
than 20000 part (Sale Order, Person...). It's work's find under 5000 
treatment, after, xml parser eat all RAM (4Gb) and stop working ! I know that 
this problem is not due to ERP5 special stuff and I don't thing I use OOorg 
import in the right way (kool to import frequently small document, but not 
made to import mass document in 1 time :) ). For Information I used a script 
to transforme an CSV to dict format and launch file with an external method 
(Ext method has been destroy since imports are done).

  About the use of Category, I haven't find a way to use it correctly : I used 
an OOorg find method to correct field before import :( ... The most important 
think for me is the relation between object (ie. source,destination in 
sale_order). It could be interresting to indicate a given portal_type list 
research for a given property (I do it but hardcoded), build a dictionnary of 
possible response at the beginning of import like :

  corresponding_dict['destination'] = { title: relativeUrl,... }

and find correct object with : 

if corresponding_dict.has_key(property):
  value = corresponding_dict[property].get(value, value)


  
On Wednesday 17 October 2007 12:02:29 Jérome Perrin wrote:
> Łukasz Nowak a écrit :
> > Hello,
> >
> > While playing with production we are able to import BOMs and many
> > other useful information from various CAD/CAM/etc programs using CSV.
> >
> > As CSV is quite...hm...undefined, there are many problems. Many
> > programs use different implementation of CSV - some of them use ";" as
> > separator some ",". Some use '"' as quote mark, some "'". Some use
> > UTF-8 encoding, some other ASCII, and so on. So, IMHO, importing of
> > CSV might be two-phase system. First one upload file, second one
> > choose some attributes and provide user special listbox, which will
> > "simulate" how will system understand provided CSV. Something similar
> > to OpenOffice CSV import/export dialog.
>
> Hi,
>
> As you pointed out, despite looking simple, csv is in reality extremly
> complex (you can look at python csv module implementation), that's one
> more reason why we concentrate on importing from OpenOffice. It still
> needs improvement, but here's what we have today:
>
> - Import of categories from a spreadsheet (Import/Export action from
> portal_categories). This uses oood to convert this input file, if for
> instance you upload a .xls file. I haven't try with CSV, I don't know if
> oood can convert from CSV.
>
>   - There is an interesting prototype on person module: "Import Persons
> from OpenOffice Calc".
> This tool propose to map openoffice column to document properties. As
> far as I know it doesn't work for categories, and probably have other
> problems, but it's IMHO a good start.
>
> As far as I know this is the current status of OOo import in ERP5. If we
> want better csv support, one way might be to add more csv to ods / ods
> to csv support directly in oood, by "normalizing" csv in python using
> csv module. This is just an idea, I haven't tried it at all, and I'm not
> sure that information like "this column is numeric" can be kept with csv
> only anyway ...
>
> Jérome
> _______________________________________________
> Erp5-dev mailing list
> Erp5-dev at erp5.org
> http://mail.nexedi.com/mailman/listinfo/erp5-dev


-- 
Brice LEROY
CWD Sellier / Sellerie de Nontron
Service Informatique
tel : 05 53 60 72 70
fax : 05 53 60 72 79