[jira] Created: (COCOON-2146) Using EventAware cache implementation breaks persistent cache restore on restart

[Ellis Pritchard (JIRA)]

Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------

Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard


In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0

However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...

Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.

This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)

Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Grzegorz Kossakowski (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grzegorz Kossakowski updated COCOON-2146:
-----------------------------------------

Priority: Minor (was: Major)

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Priority: Minor
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[josepascual]

Hello,
we are trying to deploy a cocoon web application in a Sun One 6.1 Server. We
have some problems, one of them is that when the application is starting to
run, fail with this error:

ERROR [main] (DiskStore.java:668) - cocoon-ehcache-1Cache: Failed to write
element to disk 'EVENTREGWRAPPER'. Initial cause was
org.apache.commons.collections.map.MultiValueMap
java.io.NotSerializableException:
org.apache.commons.collections.map.MultiValueMap

Following these post, I have seen that the solution is upgrade to 2.1.11
version of cocoon, we had 2.1.10 version, but the problems are still
happening. We have copied in our war deploying file and install the war in
our Sun One.

Do you have any idea what is happening?

Regards.
Jose Pascual




--
View this message in context: http://cocoon.10839.n7.nabble.com/jira-Created-COCOON-2146-Using-EventAware-cache-implementation-breaks-persistent-cache-restore-on-ret-tp45160p58527.html
Sent from the Cocoon - Dev mailing list archive at Nabble.com.

[Francesco Chicchiriccò]

Post by josepascual
Hello,
we are trying to deploy a cocoon web application in a Sun One 6.1 Server. We
have some problems, one of them is that when the application is starting to
ERROR [main] (DiskStore.java:668) - cocoon-ehcache-1Cache: Failed to write
element to disk 'EVENTREGWRAPPER'. Initial cause was
org.apache.commons.collections.map.MultiValueMap
org.apache.commons.collections.map.MultiValueMap
Following these post, I have seen that the solution is upgrade to 2.1.11
version of cocoon, we had 2.1.10 version, but the problems are still
happening. We have copied in our war deploying file and install the war in
our Sun One.
Do you have any idea what is happening?

Hi,
this mail subject refers to [1] which was fixed in 2007 (!), with
partial split to [2] which is still unresolved.
I guess there are no many people around that can help you, neither do I,
unfortunately.

Having been working with such technology for a long while, I need to
ask: really there is some SunOne WebServer 6.1 still around?!?

Regards.

[1] https://issues.apache.org/jira/browse/COCOON-2146
[2] https://issues.apache.org/jira/browse/COCOON-2152

[josepascual]

Hi francesco,
thanks for your answer ...

in our case, the problem is that we try to deploy the web application in our
local environment under our pc windows 7 system.

We can deploy the app under client environment without any problem, the
customer can use it correctly. We received the maintenance of this
application several years ago, and when we have to modify something we work
in our computers and prove over client systems, in Integration environment.

As you will guess this is very difficult manner to work, because all
developers share the same environment to prove.

So we decided to try install in our local environment the Sun One over our
windows and install the app to work. I'm trying to do this for two weeks and
unfortunately it doesn't work.

thanks for advance.
Regards
Jose Pascual



--
View this message in context: http://cocoon.10839.n7.nabble.com/jira-Created-COCOON-2146-Using-EventAware-cache-implementation-breaks-persistent-cache-restore-on-ret-tp45160p58529.html
Sent from the Cocoon - Dev mailing list archive at Nabble.com.

[Francesco Chicchiriccò]

Post by josepascual
Hi francesco,
thanks for your answer ...
in our case, the problem is that we try to deploy the web application in our
local environment under our pc windows 7 system.
We can deploy the app under client environment without any problem, the
customer can use it correctly. We received the maintenance of this
application several years ago, and when we have to modify something we work
in our computers and prove over client systems, in Integration environment.
As you will guess this is very difficult manner to work, because all
developers share the same environment to prove.
So we decided to try install in our local environment the Sun One over our
windows and install the app to work. I'm trying to do this for two weeks and
unfortunately it doesn't work.

Why not trying a VM with Linux or (Open)Solaris, then?

[Peter Hunsberger]

It's likely going to be some difference in the class paths used in the two
systems and what libraries are sitting in which parts of the class path.
To check, work backwards from the error; what library is the class in,
where do those libraries sit on your class path, is the 1st one found the
right version? If it is, are there dependencies it has on other
libraries? If yes. rep[eat the check for those libraries. Usually you'll
find the system has set a class path that includes a version of some
duplicate library that will be found earlier than the version Cocoon
wants. If you can change the class path order, or move libraries around
you should be able to fix the problem.

Peter Hunsberger

On Fri, Oct 23, 2015 at 2:07 AM, josepascual <

Post by josepascual
Hi francesco,
thanks for your answer ...
in our case, the problem is that we try to deploy the web application in our
local environment under our pc windows 7 system.
We can deploy the app under client environment without any problem, the
customer can use it correctly. We received the maintenance of this
application several years ago, and when we have to modify something we work
in our computers and prove over client systems, in Integration environment.
As you will guess this is very difficult manner to work, because all
developers share the same environment to prove.
So we decided to try install in our local environment the Sun One over our
windows and install the app to work. I'm trying to do this for two weeks and
unfortunately it doesn't work.
thanks for advance.
Regards
Jose Pascual
--
http://cocoon.10839.n7.nabble.com/jira-Created-COCOON-2146-Using-EventAware-cache-implementation-breaks-persistent-cache-restore-on-ret-tp45160p58529.html
Sent from the Cocoon - Dev mailing list archive at Nabble.com.

[Ellis Pritchard (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ellis Pritchard updated COCOON-2146:
------------------------------------

Attachment: patch.txt

Patch file for EventRegistryDataWrapper.java and AbstractDoubleMapEventRegistry.java to restore cache serializability and thus recovery.

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Priority: Minor
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Grzegorz Kossakowski (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grzegorz Kossakowski reassigned COCOON-2146:
--------------------------------------------

Assignee: Grzegorz Kossakowski

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Assignee: Grzegorz Kossakowski
Priority: Minor
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ellis Pritchard (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ellis Pritchard updated COCOON-2146:
------------------------------------

Fix Version/s: 2.1.11-dev (Current SVN)
Other Info: [Patch available]

Flagged fix and patch!



NB issue does not effect Cocoon 2.2, since that version was never updated and is still using the MultiHashMap...

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Assignee: Grzegorz Kossakowski
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ard Schrijvers (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545506 ]

Ard Schrijvers commented on COCOON-2146:
----------------------------------------

FYI : IMO, the AbstractDoubleMapEventRegistry is a very bad implementation, and OOM sensitive. I was facing sites needing a restart every few days (very high traffic sites, > 100.000 pages ) because of the double map event registry in combination with ehcache, It has been more than a year ago, so I might be off at some places, but :

The AbstractDoubleMapEventRegistry used for event caching was build a long time ago (I wasn't around) and was based on an internal cocoon cache, which in turn was managed by the StoreJanitor. This internal cache has been replaced by ehcache, or jcscache. These caches handle their own cache (TTL, LRU, ETERNAL, etc etc). This means, that when this cache decides to remove a cached entry, this remove was not initialized by the StoreJanitor, hence not propagated to the event registry. This ends up in an ever growing event registry.

Also, I totally did not like the AbstractDoubleMapEventRegistry. Keeping double mapped maps in sync....it kind of is stupid. And, this is exactly the thing you use WeakReferences for. So I rebuild the registry for our projects to use WeakReferences. Only problem I faced, was that for a reason I have never been able to find or reproduce outside cocoon, it didn't play well with ehcache because references seemed to change. Therefor I did implement it in combination with JCSCache (which by the way performed better). While I was busy I changed the registry to enable multiple caches because I wanted filesystem caches for binary repository data, a seperate cache for repository xml files, and a seperate one for pipelines.

I am not sure if anyone is interested... :-) Because of the ehcache reference problems and the fact that I had no easy way (without reading the entire cache at startup) to have a persistent cache with a registry (WeakReferences cannot be persisted ofcourse) I never considered the code suitable for Cocoon. OTOH, I left EHCache without problems, and IMO, needing a persistent cache does seem to me that you should have implemented your application differently. I have always been concerned performance, and I just know that an uncached pipepline with external repository sources, with webdav calls en searches, can easily finish within 50-100 ms. If you must rely on your cache that havily that it should survive a restart, I think you are misusing cocoon's cache anyway. Just my 2 cents

Anyway, if anybody is interested, the code is over here:

https://svn.hippocms.org/repos/hippo/hippo-cocoon-extensions/trunk/eventcache/src/java/org/apache/cocoon/caching/impl/

and then mainly AbstractWeakRefMapEventRegistry.java.

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Assignee: Grzegorz Kossakowski
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ellis Pritchard (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545589 ]

Ellis Pritchard commented on COCOON-2146:
-----------------------------------------

Thanks for your comment, Ard.

Your analysis may well explain some performance problems we were seeing on our system; unfortunately I was not around at the time to analyze them, so I can't really be sure what caused them. The action they took at the time was simply to cache less, i.e. only the expensive stuff, rather than, well, basically everything!

Presumably though, if site-usage is fairly uniform, stuff that expires and is still referenced from AbstractDoubleMapEventRegistry will get regenerated again at some fairly near point, and replace the old entry with the new one, allowing it to be freed by the JVM? Only if you're generating lots of unique pages or short-lived pages and caching them for a long time may it become a 'real' problem?

I think that ehcache 1.4 (now in beta) may provide at least one solution to this problem, with its new listeners functionality, which can get notification of the keys of entries removed (for any reason) from the cache.

However, maybe there's a 'redesign the whole thing' kind of solution too?

As for whether we need it or not: according to "the powers that be" we do need a persistent cache, so that when we bring the site up under load, on n instances (each with their own cache), it doesn't hammer our database. The fact that was running without one for 8 months due to this bug is not enough to convince them otherwise! It's either that, or having to pre-warm the cache during site-down time (and making site down time longer), or even worse, running the whole site behind mod_cache with an expiry time of about an hour or so, to cover any lag in getting it up again. A persistent EventCache just looks so attractive compared to those options...

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Assignee: Grzegorz Kossakowski
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ellis Pritchard (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545589 ]

ellispritchard edited comment on COCOON-2146 at 11/26/07 12:27 PM:
--------------------------------------------------------------------

Thanks for your comment, Ard.

Your analysis may well explain some performance problems we were seeing on our system; unfortunately I was not around at the time to analyze them, so I can't really be sure what caused them. The action they took at the time was simply to cache less, i.e. only the expensive stuff, rather than, well, basically everything!

Presumably though, if site-usage is fairly uniform, stuff that expires and is still referenced from AbstractDoubleMapEventRegistry will get regenerated again at some fairly near point, and replace the old entry with the new one, allowing it to be freed by the JVM? Only if you're generating lots of unique pages or short-lived pages and caching them for a long time may it become a 'real' problem?

I think that ehcache 1.4 (now in beta) may provide at least one solution to this problem, with its new listeners functionality, which can get notification of the keys of entries removed (for any reason) from the cache.

However, maybe there's a 'redesign the whole thing' kind of solution too?

As for whether we need it or not: according to "the powers that be" we do need a persistent cache, so that when we bring the site up under load, on n instances (each with their own cache), it doesn't hammer our database. The fact that the site was running without one for 8 months due to this bug is not enough to convince them otherwise! It's either that, or having to pre-warm the cache during site-down time (and making site down time longer), or even worse, running the whole site behind mod_cache with an expiry time of about an hour or so, to cover any lag in getting it up again. A persistent EventCache just looks so attractive compared to those options...



was (Author: ellispritchard):
Thanks for your comment, Ard.

Your analysis may well explain some performance problems we were seeing on our system; unfortunately I was not around at the time to analyze them, so I can't really be sure what caused them. The action they took at the time was simply to cache less, i.e. only the expensive stuff, rather than, well, basically everything!

Presumably though, if site-usage is fairly uniform, stuff that expires and is still referenced from AbstractDoubleMapEventRegistry will get regenerated again at some fairly near point, and replace the old entry with the new one, allowing it to be freed by the JVM? Only if you're generating lots of unique pages or short-lived pages and caching them for a long time may it become a 'real' problem?

I think that ehcache 1.4 (now in beta) may provide at least one solution to this problem, with its new listeners functionality, which can get notification of the keys of entries removed (for any reason) from the cache.

However, maybe there's a 'redesign the whole thing' kind of solution too?

As for whether we need it or not: according to "the powers that be" we do need a persistent cache, so that when we bring the site up under load, on n instances (each with their own cache), it doesn't hammer our database. The fact that was running without one for 8 months due to this bug is not enough to convince them otherwise! It's either that, or having to pre-warm the cache during site-down time (and making site down time longer), or even worse, running the whole site behind mod_cache with an expiry time of about an hour or so, to cover any lag in getting it up again. A persistent EventCache just looks so attractive compared to those options...

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Assignee: Grzegorz Kossakowski
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Jörg Heinicke (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jörg Heinicke closed COCOON-2146.
---------------------------------

Resolution: Fixed
Assignee: Jörg Heinicke (was: Grzegorz Kossakowski)

I just switched back to the original MultiHashMap as suggested by Ellis. I'm going to create a new issue/enhancement request where I'll put your information. I hope you are fine with it.

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10, 2.1.11-dev (Current SVN)
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Joerg Heinicke]

Post by Jörg Heinicke (JIRA)
Jörg Heinicke closed COCOON-2146.
---------------------------------
Resolution: Fixed
Assignee: Jörg Heinicke (was: Grzegorz Kossakowski)

Post by Ellis Pritchard (JIRA)
URL: https://issues.apache.org/jira/browse/COCOON-2146

Sorry Grek, that I supplanted you. I only saw on closing the issue that
you actually assigned it to you. I hope you don't mind :-)

Joerg

[Grzegorz Kossakowski]

Post by Joerg Heinicke

Post by Jörg Heinicke (JIRA)
Jörg Heinicke closed COCOON-2146.
---------------------------------
Resolution: Fixed
Assignee: Jörg Heinicke (was: Grzegorz Kossakowski)

Post by Ellis Pritchard (JIRA)
URL: https://issues.apache.org/jira/browse/COCOON-2146

Sorry Grek, that I supplanted you. I only saw on closing the issue that
you actually assigned it to you. I hope you don't mind :-)

No problem, Jörg! I assigned it to myself because I thought that the issue was trivial but recent
comments confused me greatly. Thanks for taking care of applying a patch.

[Ard Schrijvers]

Post by Grzegorz Kossakowski

Post by Joerg Heinicke
Sorry Grek, that I supplanted you. I only saw on closing the issue
that you actually assigned it to you. I hope you don't mind :-)

No problem, Jörg! I assigned it to myself because I thought
that the issue was trivial but recent comments confused me
greatly.

Sry about that :-)

-Ard

Post by Grzegorz Kossakowski
Thanks for taking care of applying a patch.
--
Grzegorz Kossakowski
Committer and PMC Member of Apache Cocoon
http://reflectingonthevicissitudes.wordpress.com/

[Grzegorz Kossakowski]

Post by Ard Schrijvers

Post by Grzegorz Kossakowski
No problem, Jörg! I assigned it to myself because I thought
that the issue was trivial but recent comments confused me
greatly.

Sry about that :-)

Don't be sorry, Ard :-)
It's just me who has currently no time to dig into cache-related issues deeply. I'm really looking
forward to finish all of my math exams because I have so many nice, Cocoon-related things to show
laying on my disk...

[Jörg Heinicke (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jörg Heinicke updated COCOON-2146:
----------------------------------

Affects Version/s: (was: 2.1.11-dev (Current SVN))

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ard Schrijvers (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545761 ]

Ard Schrijvers commented on COCOON-2146:
----------------------------------------

"Presumably though, if site-usage is fairly uniform, stuff that expires and is still referenced from AbstractDoubleMapEventRegistry will get regenerated again at some fairly near point, and replace the old entry with the new one, allowing it to be freed by the JVM? Only if you're generating lots of unique pages or short-lived pages and caching them for a long time may it become a 'real' problem? "

Yes, you would think you absolutely had a point here.....but, unfortunately I found out, that the MultiHashMap doen *not* act that way !! (note, it has been a year, and i am not in a position now to dive in the code again but from the top of my head) :

If you have a MultiHashMap and you put something like

m_keyMMap.put("key1", "value1"); multiple times, your map grows! you'll have 5 times "value1" in it (obviously from a performance POV this is easy to understand)

So, exactly what you describe above, will result in OOM. Even if you have only a few possible links, resulting in limited cachekeys, you'll run into OOM in the end....I am sorry :-)

Anyway, the WeakReference solution is capable of being persisted, but needs some extra stuff to be done at restart. OTOH, notification from ehcache with some listeners might be a solution as well, though

1) it has to be independant of ehcache
2) I still think a DoubleMap is really not the way it should be solved. I would always use WeakReferences for these kind of problems, because the overhead of keeping the maps in sync is not needed.

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ellis Pritchard (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ellis Pritchard reopened COCOON-2146:
-------------------------------------


Not sure whether to reopen thisbug, or treat this as a 'sub-optimal' implementation issue...(!)

When using the DefaultEventRegistryImpl the functionality now works as expected (events are persisted and restored) after the patch applied in COCOON-2146.

However, there's still a problem with StoreEventRegistryImpl.

The behaviour is that it doesn't seem to actually write/restore any event entries: the maps in the EventRegistryDataWrapper are empty (but not null) after restart, even though the actual cache entry (key EVENTREGWRAPPER) was found in the Store, and the entries were present when persist() was called.

The effect of this is to correctly restore the cached entries, but discard all the events, which means that event-flushes don't work any more, which is not a good thing.

I've tracked this down to the fact that AbstractDoubleMapEventRepository#dispose() which performs the persist(), then immediately clear()s the maps, WHICH HAVEN'T YET BEEN WRITTEN TO DISK BY EHCACHE SHUTDOWN!

This code has probably never worked :)

Patches to follow; I propose modifying dispose() to null the map fields, but not perform clear() on them.

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ard Schrijvers (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545962 ]

Ard Schrijvers commented on COCOON-2146:
----------------------------------------

Hmmmm, I am really confident that persistent caches with the event registry used to work (we have used it for years until i disabled it...but the last time i used it was with 2.1.8, so not sure if something changed :-) ). A restart did use to work. You are really sure that you have a restart persistent cache configuration?

"I've tracked this down to the fact that AbstractDoubleMapEventRepository#dispose() which performs the persist(), then immediately clear()s the maps, WHICH HAVEN'T YET BEEN WRITTEN TO DISK BY EHCACHE SHUTDOWN! "

This shouldn't matter. When the registry is persisted, you can clear the double maps. The cache takes care of persisting itself. Can you comment your <store> configuration of your ehcache?

Can you confirm, that on disk, in your work dir after shutdown, you have serialized ehcache files? Can you also confirm you have (i think in the WEB-INF dir somewhere) serialized .ser registry files. Open them with notepad to see that they contain some data if you want....

At startup, I know that for every entry in the registry, the cachekey is checked wether it is still present in the cache. When not, the registry of this cachekey is discarded. I think you have a problem somewhere in this (though depends on wether your .ser files are empty or not...if empty, the problem is ofcourse in the shutdown)

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Jörg Heinicke (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546023 ]

Jörg Heinicke commented on COCOON-2146:
---------------------------------------

Can you please make it a new issue though? This one should really be only about the serializability.

Thanks.

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Jörg Heinicke (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546023 ]

***@gmx.de edited comment on COCOON-2146 at 11/27/07 2:13 PM:
-----------------------------------------------------------------

Can you please make it a new issue though? This one should really be only about the serializability. Thanks.

And btw., they made COLLECTIONS-240 a blocker after I added a comment yesterday, so it will at least be fixed in Commons Collections 3.3.

was (Author: ***@gmx.de):
Can you please make it a new issue though? This one should really be only about the serializability.

Thanks.

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.

[Ellis Pritchard (JIRA)]

[ https://issues.apache.org/jira/browse/COCOON-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ellis Pritchard closed COCOON-2146.
-----------------------------------

Resolution: Fixed

Moving new StoreEventRegistryImpl problem to new bug COCOON-2152.

Ard, can you please copy over your last comment (on this aspect) to the new bug?

Post by Ellis Pritchard (JIRA)
Using EventAware cache implementation breaks persistent cache restore on restart
--------------------------------------------------------------------------------
Key: COCOON-2146
URL: https://issues.apache.org/jira/browse/COCOON-2146
Project: Cocoon
Issue Type: Bug
Components: Blocks: Event Cache
Affects Versions: 2.1.10
Reporter: Ellis Pritchard
Assignee: Jörg Heinicke
Priority: Minor
Fix For: 2.1.11-dev (Current SVN)
Attachments: patch.txt
In revision 412307 (Cocoon 2.1.10), AbstractDoubleMapEventRegistry and EventRegistryDataWrapper were changed (without an informative SVN comment!) to use the commons MultiValueMap instead of the MultiHashMap; I presume this was done in good faith because the latter map is deprecated and will be removed from Apache commons-collections 4.0
However, as a result, the persistent cache cannot be restored if the EventAware cache implementation is used, since MultiValueMap is not Serializable! The old MultiHashMap was...
Depending on whether StoreEventRegistryImpl or DefaultEventRegistryImpl is used, either the event cache index is never written (ehcache doesn't store non-serializable objects on disk), or a java.io.NotSerializableException is thrown (and caught, causing a full cache-clear) when attempting to restore the event cache index.
This is Major for us, since we use Event-based caching alot, and this is causing the *entire* cache to no-longer persist across restarts (it's been like that for 8 months, since I upgraded Cocoon to 2.1.10 in the last week I was working here, and now I'm back, they've actually noticed!!)
Work-around at the moment is to down-grade AbstractDoubleMapEventRegistry and EventRegistryDataWrapper to the 2.1.9 versions (pre-412307), which works so long as Apache-commons 3.x is still in use.