"maximum weight must not be negative" when requesting image

[thopp-tudelft]

after installing from the docker file from the current version of the master branch (b06be44) on a different system we encounter this error whenever we request an image with eg /stack//z//jpeg-image: maximum weight must not be negative

we use this docker compose file, it builds the image using the normal docker file in the render repo: https://github.com/hoogenboom-group/em-infrastructure/blob/3a83f76b0cbff31a77f72a75932e26ea09689464/compose-files/render/docker-compose.yml

the same setup has been running on our server for multiple years but causes this error when installed on the new system. the same small dataset does not cause any issues on the old system.

log:

Details

``` 2026-01-29 15:42:10,814 [qtp690521419-65] ERROR [org.janelia.render.service.util.RenderServiceUtil] service failure java.lang.IllegalArgumentException: maximum weight must not be negative at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) at com.google.common.cache.CacheBuilder.maximumWeight(CacheBuilder.java:452) at org.janelia.alignment.util.ImageProcessorCache.(ImageProcessorCache.java:110) at org.janelia.render.service.util.SharedImageProcessorCache.setSharedCache(SharedImageProcessorCache.java:39) at org.janelia.render.service.util.SharedImageProcessorCache.getInstance(SharedImageProcessorCache.java:18) at org.janelia.render.service.util.RenderServiceUtil.validateParametersAndRenderImage(RenderServiceUtil.java:252) at org.janelia.render.service.util.RenderServiceUtil.renderImageStream(RenderServiceUtil.java:179) at org.janelia.render.service.util.RenderServiceUtil.renderImageStream(RenderServiceUtil.java:142) at org.janelia.render.service.util.RenderServiceUtil.renderJpegImage(RenderServiceUtil.java:74) at org.janelia.render.service.RenderImageService.renderJpegImageForZ(RenderImageService.java:127) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130) at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524) at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:476) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:434) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:408) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:69) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364) at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:60) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:313) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:267) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:170) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:192) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at java.base/java.lang.Thread.run(Unknown Source) ```

this error is shown every time an image is requested. we can provide more logs if needed.

[trautmane]

Hi @thopp-tudelft ,

On the master branch if you have defined a WEB_SERVICE_MAX_IMAGE_PROCESSOR_GB property, the ImageProcessorCache is setup here. Your exception above indicates that the weight value is negative. I'm not sure why/how that happened on your new system.

Can you run a test with the configured property removed to see if that works?

I think you just need to remove this line from your docker-compose.yml file .

--Eric

[thopp-tudelft]

I will try that, thanks

[thopp-tudelft]

removing WEB_SERVICE_MAX_IMAGE_PROCESSOR_GB does indeed fix the issue, thank you so much for your help!

do you have a suggestion on how we can increase the cache size? even with this parameter we experience issues where render does not use all memory allocated to it.

[trautmane]

Do you know how much memory is available on the new machine?
It would help to know how much is available when nothing else is running on the machine and how much is "normally" available if you have other things running.

[thopp-tudelft]

we have about 128 gb, we'd like to allocate at least 10 of it to render.

[trautmane]

I think the following fix which was made on the newsolver branch a couple of years ago will resolve the problem:
avoid potential overflow problems from int multiplication

The master branch is confusingly very old and has been left as is (since 2019-ish) to reduce impacts on the Allen Institute as we introduced changes at Janelia.

Can you try running with the newsolver branch or pulling the small change above into your code to confirm that it resolves the problem?

If that is not easy for you to do, let me know and we can figure out another option.