iCal Import

[jefigo]

Hello
is there a limitation (number of iCal events) when importing iCal files.
I exported several Calendars from my Mac to .ical and tried to import them into OX.
The biggest file (600kb) did not finish the import process.
No messages in the gui or anything in a server log after more than 5 minutes.

regards
Jens

[Martin Heiland]

Hi Jens,

i'm not aware of any limitation. Could you increase the log level to DEBUG and try again? Does the groupware process consumes more cpu ressources than normally after starting the import?

Greetings

[jefigo]

Hi Martin,
thank you for your reply.
I will try it again tomorrow. So far i was able to import the calendar using the oxtender for mac. This worked well.
Normally i would like to use the oxtender for mac but due to the fact that i am using mobileme too, it doesn't work for me very well. If i use both sync services i get every event multiple times. I think that there is no way to select, which calendar should be synched with oxtender for mac. In my case i only want the calendar called office to be synched with oxtender. But this is another problem. As mentioned before, i'll try to import the large .ical file again with increased debugging.

Greetings

[jefigo]

debugging

i hope i increased the debug level right.
I tried to import the .ical file with a test user.
This is what i found in open-xchange.log.0

Apr 18, 2011 5:44:19 PM com.openexchange.groupware.importexport.importers.ICalImporter importAppointment
SEVERE: APP-0100 Category=1 Message=Can not insert appointment with uid M2CD-3-1-D50BCC3B-794C-E045-8AD7-AE5A8E6A7492. Uid already exists. exceptionID=-1943014261-16244
APP-0100 Category=1 Message=Can not insert appointment with uid M2CD-3-1-D50BCC3B-794C-E045-8AD7-AE5A8E6A7492. Uid already exists. exceptionID=-1943014261-16244
	at com.openexchange.calendar.CalendarMySQL.handleUid(CalendarMySQL.java:1548)
	at com.openexchange.calendar.CalendarMySQL.insertAppointment0(CalendarMySQL.java:1408)
	at com.openexchange.calendar.CalendarMySQL.insertAppointment(CalendarMySQL.java:1395)
	at com.openexchange.calendar.CalendarSql.insertAppointmentObject(CalendarSql.java:447)
	at com.openexchange.calendar.CalendarSql.insertAppointmentObject(CalendarSql.java:106)
	at com.openexchange.groupware.importexport.importers.ICalImporter.importAppointment(ICalImporter.java:417)
	at com.openexchange.groupware.importexport.importers.ICalImporter.importData(ICalImporter.java:250)
	at com.openexchange.groupware.importexport.ImporterExporter.importData(ImporterExporter.java:126)
	at com.openexchange.ajax.ImportServlet.doPost(ImportServlet.java:136)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:616)
	at com.openexchange.ajax.AJAXServlet.service(AJAXServlet.java:413)
	at com.openexchange.ajax.SessionServlet.service(SessionServlet.java:183)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
	at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doServletService(AJPv13RequestHandlerImpl.java:523)
	at com.openexchange.ajp13.AJPv13Request.response(AJPv13Request.java:134)
	at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.createResponse(AJPv13RequestHandlerImpl.java:335)
	at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.createResponse(AJPv13ConnectionImpl.java:227)
	at com.openexchange.ajp13.najp.AJPv13Task.call(AJPv13Task.java:352)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
	at java.util.concurrent.FutureTask.run(FutureTask.java:123)
	at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:750)
	at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:776)
	at java.lang.Thread.run(Thread.java:595)
Apr 18, 2011 5:44:19 PM com.openexchange.groupware.importexport.importers.ICalImporter importAppointment
SEVERE: APP-0096 Category=1 Message=Unnecessary recurrence information for recurrence type "no recurrence". exceptionID=-1943014261-16245
APP-0096 Category=1 Message=Unnecessary recurrence information for recurrence type "no recurrence". exceptionID=-1943014261-16245
	at com.openexchange.calendar.RecurrenceChecker.checkNo(RecurrenceChecker.java:96)
	at com.openexchange.calendar.RecurrenceChecker.check(RecurrenceChecker.java:71)
	at com.openexchange.calendar.CalendarSql.insertAppointmentObject(CalendarSql.java:423)
	at com.openexchange.calendar.CalendarSql.insertAppointmentObject(CalendarSql.java:106)
	at com.openexchange.groupware.importexport.importers.ICalImporter.importAppointment(ICalImporter.java:417)
	at com.openexchange.groupware.importexport.importers.ICalImporter.importData(ICalImporter.java:250)
	at com.openexchange.groupware.importexport.ImporterExporter.importData(ImporterExporter.java:126)
	at com.openexchange.ajax.ImportServlet.doPost(ImportServlet.java:136)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:616)
	at com.openexchange.ajax.AJAXServlet.service(AJAXServlet.java:413)
	at com.openexchange.ajax.SessionServlet.service(SessionServlet.java:183)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
	at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.doServletService(AJPv13RequestHandlerImpl.java:523)
	at com.openexchange.ajp13.AJPv13Request.response(AJPv13Request.java:134)
	at com.openexchange.ajp13.najp.AJPv13RequestHandlerImpl.createResponse(AJPv13RequestHandlerImpl.java:335)
	at com.openexchange.ajp13.najp.AJPv13ConnectionImpl.createResponse(AJPv13ConnectionImpl.java:227)
	at com.openexchange.ajp13.najp.AJPv13Task.call(AJPv13Task.java:352)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
	at java.util.concurrent.FutureTask.run(FutureTask.java:123)
	at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:750)
	at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:776)
	at java.lang.Thread.run(Thread.java:595)

[Tobias Prinz]

Ah, one of my favourite bugs... took quite a while and was finally fixed at 25th Mar 2011

The problem is that older versions of the OX decided that two appointments cannot have the same UUID, because it was, well, supposedly unique. So the second appointment is not imported.

Of course, series with change exceptions use UUIDs to mark the exception as belonging to the original series.

For a long time, of did not support change- (and delete-) exceptions to series. You are running one of those. That explains the first log message, "Can not insert appointment with uid".

The second one, "Unnecessary recurrence information for recurrence type "no recurrence", is probably related: I guess the exception does contain some series information that would make sense if the OX could see that it actually was an exception, but without support for exceptions, it just looks like a broken series.

This leaves you with two options:
1) Remove this one entry from your ical file - it is going to be easy to find, since it has a UUID ;-)
2) Get your hands on a new version of OX where this is fixed. I guess that is going to be 6.20

Why this fails without proper user feedback, I don't know, though. That sucks. Sorry.

[jefigo]

Hi Tobias,
this sounds very confusing for me.
1) we have installed 6.20
2) there is no such uuid in the ical file

So i think the uuid must be generated during the import process or something else.

[Tobias Prinz]

Hmmm. It sounded exactly like the bug report that pointed me to the UUID problem. Anyway, that should not happen with 6.20. Can you do me a favour and create a bug report at https://bugs.open-xchange.com and attach that ICAL file? I'm pretty sure with the data at hand, I can figure it out fast and if it is some broken importer behaviour, I can fix it in the nearest PTF.

Thanks in advance,
Tobias Prinz

[jefigo]

Sorry, i have to correct my last post a little bit.
The uid is in the .ical - but only once. So the message about the already existing uid must depend on previous import trials.

[Tobias Prinz]

Yes. The UUID is supposed to be unique context-wide. To reproduce the problem you'll have to delete the data from previous attempts.

So that's not the original problem. Good. I'll sleep better now.^^

Back to the original problem: First, let's look at the hardware: 600kb of ical data is a lot. Since you are testing, I guess it is something virtualized. What amount of RAM do you have assigned to the OX system?

[jefigo]

the OX runs on a VM with debian 5.08 64 bit. It has 4 CPUs and 4GB of RAM.

[Martin Heiland]

Do you have modified the groupware/ox-scriptconf.sh file to match the memory settings of the JVM and the available memory (-Xmx2048 for example)? We identified some performance/memory issues with the current iCal implementation, however the first step would be configuring the system in an optimal way.

[jefigo]

Hi Martin,
no i haven't done any tuning yet. I am not quite sure how to use the script for our needs.
What is your proposal?

[Martin Heiland]

Well, the crucial thing is configuring the maximum amount of memory the JVM can allocate. Otherwise you don't really have any advantage in adding physical memory. This is configured by the "-Xmx" parameter which is 512MB per default. Note that JVM will allocate most of the memory it is allowed to to use so you should check how many memory is typically unused at the system and add the difference to this parameter. If your machine has 4GB of memory and only consumes 512mb in total without OX running it's a safe bet to use 2048MB. Just configure and check the results, the perfect configuration depends on the system and services running on it.