[IPT] text file upload (larger record numbers)
"Markus Döring (GBIF)"
mdoering at gbif.org
Fri Feb 27 21:03:21 CET 2009
you can upload zipped file archives now with the latest subversion
trunk version. it automatically detects if your file is a ZIP and then
inserts all its files as IPT sources.
Soon to be on ipt.gbif.org.
Markus
On Feb 27, 2009, at 5:24 PM, Tim Robertson wrote:
> Agree - need to support GZip and Zips...
>
> and then limit to 100M I reckon
>
>
> On 27 Feb 2009, at 17:17, Markus Döring (GBIF) wrote:
>
>> Andrea,
>> the upload of files is limited to 100MB at the GBIF installation.
>> We might want to increase it, but it might be better to allow
>> upload of compressed files.
>> what do you think?
>>
>> markus
>>
>>
>> On Feb 27, 2009, at 4:57 PM, Andrea Hahn (GBIF) wrote:
>>
>>> Hi,
>>>
>>> quick note: I have been playing a bit with text file uploads for
>>> occurrence records, using a tabular export from the GBIF index on
>>> a 500,000+ record dataset. Using different subsets of the same
>>> recordset, I have not been able to get beyond the 310,000 records
>>> cutdown of the test set so far - this equals a file size of
>>> 99,974kB in the given example. The next larger set (320,000
>>> records, 100MB) and all larger ones fail with
>>> Yikes!
>>>
>>> javax.servlet.ServletException: Expression resource is undefined
>>> on line 87, column 30 in WEB-INF/pages/manage/sources.ftl.
>>> at
>>> net
>>> .sf.ehcache.constructs.web.filter.Filter.logThrowable(Filter.java:
>>> 152)
>>> at
>>> net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:97)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core
>>> .ApplicationFilterChain
>>> .internalDoFilter(ApplicationFilterChain.java:215)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
>>> 188)
>>> at
>>> org
>>> .appfuse
>>> .webapp.filter.LocaleFilter.doFilterInternal(LocaleFilter.java:74)
>>> at org.springframework.web.filter.OncePerRequestFilter.doFilter
>>> (OncePerRequestFilter.java:76)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core
>>> .ApplicationFilterChain
>>> .internalDoFilter(ApplicationFilterChain.java:215)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
>>> 188)
>>> at
>>> org
>>> .springframework
>>> .orm
>>> .hibernate3
>>> .support
>>> .OpenSessionInViewFilter
>>> .doFilterInternal(OpenSessionInViewFilter.java:198)
>>> at org.springframework.web.filter.OncePerRequestFilter.doFilter
>>> (OncePerRequestFilter.java:76)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core
>>> .ApplicationFilterChain
>>> .internalDoFilter(ApplicationFilterChain.java:215)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
>>> 188)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:359)
>>> at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke
>>> (FilterSecurityInterceptor.java:109)
>>> at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter
>>> (FilterSecurityInterceptor.java:83)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui
>>> .ExceptionTranslationFilter
>>> .doFilterHttp(ExceptionTranslationFilter.java:101)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .providers
>>> .anonymous
>>> .AnonymousProcessingFilter
>>> .doFilterHttp(AnonymousProcessingFilter.java:105)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui
>>> .rememberme
>>> .RememberMeProcessingFilter
>>> .doFilterHttp(RememberMeProcessingFilter.java:116)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .wrapper
>>> .SecurityContextHolderAwareRequestFilter
>>> .doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui
>>> .basicauth
>>> .BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:172)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui
>>> .AbstractProcessingFilter
>>> .doFilterHttp(AbstractProcessingFilter.java:268)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:87)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui
>>> .SessionFixationProtectionFilter
>>> .doFilterHttp(SessionFixationProtectionFilter.java:61)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .context
>>> .HttpSessionContextIntegrationFilter
>>> .doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
>>> at
>>> org
>>> .springframework
>>> .security
>>> .ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>> at org.springframework.security.util.FilterChainProxy
>>> $VirtualFilterChain.doFilter(FilterChainProxy.java:371)
>>> at
>>> org
>>> .springframework
>>> .security.util.FilterChainProxy.doFilter(FilterChainProxy.java:174)
>>> at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate
>>> (DelegatingFilterProxy.java:236)
>>> at org.springframework.web.filter.DelegatingFilterProxy.doFilter
>>> (DelegatingFilterProxy.java:167)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core
>>> .ApplicationFilterChain
>>> .internalDoFilter(ApplicationFilterChain.java:215)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
>>> 188)
>>> at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal
>>> (CharacterEncodingFilter.java:96)
>>> at org.springframework.web.filter.OncePerRequestFilter.doFilter
>>> (OncePerRequestFilter.java:76)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core
>>> .ApplicationFilterChain
>>> .internalDoFilter(ApplicationFilterChain.java:215)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
>>> 188)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
>>> at
>>> org
>>> .apache
>>> .catalina
>>> .core.StandardContextValve.invoke(StandardContextValve.java:174)
>>> at
>>> org
>>> .apache
>>> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>> at
>>> org
>>> .apache
>>> .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>>> at
>>> org
>>> .apache
>>> .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
>>> 108)
>>> at
>>> org
>>> .apache
>>> .catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>>> at
>>> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:
>>> 200)
>>> at
>>> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
>>> at
>>> org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
>>> at
>>> org
>>> .apache
>>> .jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
>>> at org.apache.jk.common.ChannelSocket
>>> $SocketConnection.runIt(ChannelSocket.java:895)
>>> at org.apache.tomcat.util.threads.ThreadPool
>>> $ControlRunnable.run(ThreadPool.java:685)
>>> at java.lang.Thread.run(Thread.java:595)
>>>
>>>
>>> ---------------------------------------------------------------
>>> Andrea Hahn (ahahn at gbif.org)
>>> Data Portal Manager
>>> Global Biodiversity Information Facility Secretariat
>>> Universitetsparken 15, DK-2100 Copenhagen, Denmark
>>> Tel: +45-35321491 Fax: +45-35321480
>>> URL: http://www.gbif.org
>>> ---------------------------------------------------------------
>>>
>>> _______________________________________________
>>> IPT mailing list
>>> IPT at lists.gbif.org
>>> http://lists.gbif.org/mailman/listinfo/ipt
>>
>> _______________________________________________
>> IPT mailing list
>> IPT at lists.gbif.org
>> http://lists.gbif.org/mailman/listinfo/ipt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.gbif.org/pipermail/ipt/attachments/20090227/bb0a2712/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Picture 4.png
Type: image/png
Size: 66354 bytes
Desc: not available
Url : http://lists.gbif.org/pipermail/ipt/attachments/20090227/bb0a2712/attachment-0001.png
More information about the IPT
mailing list