Java Heaps and Virtual Memory - Part 2

The story so far... in my earlier post I described a memory stress test that demonstrated the soundness of the advice to keep your Java heap size smaller than your physical memory.

I also wanted to check on the very limited explanations that I'd found and get a better understanding of what was going on. In particular, I wanted to know how Java (i.e. the Sun JVM) allocates heap memory and whether it adopts any strategies to avoid swap thrashing.

I did some further digging using the various things under the /proc file system and the JDK source code to find out.

The first surprise was in /proc/meminfo - the only counter that was going up significantly during the test was 'Mapped' - i.e. memory mapped files. I was expected this approach to be used for reading in .jar files and native libraries (and it was), but I wasn't expecting this for the heap. Digging into the source code explains why - The JDK uses the mmap system call to request more heap memory from the O/S.

I also took several snapshots of /proc/PID/smaps to see exactly what memory regions were being used in the processes address space. What this showed was:-

1. There was a memory region (in my case starting from 0x51840000) that was clearly growing as the app allocated more and more heap. During the early part of the app's execution this would show up with a resident size 7Mb less than its overall size and with all of the resident pages showing up as being dirty.
   
2. Once memory started to become scarce, many of the other memory regions start to show a reduction in their resident sizes and their shared sizes.
3. Once swap thrashing was happening, the memory region which had been growing still had a 6-7Mb difference between the resident size and the allocated size. The big difference, however was that 15Mb of the space was now showing up as 'Private_Clean'.

So what does it all mean? Here's what I think is happening...

1. During the early stages of execution the app is getting as much memory as it asks for but Linux delays giving it real physical memory until the specific pages are really accessed. This explains why the resident size is less than the allocated size - Java has probably extended the heap but hasn't yet accessed all of the allocated space.
2. Memory is getting scarce, so Linux starts to reclaim pages that have the smallest impact. In the first instance it is hunting around for less critical pages (e.g. pages of jar files or libraries that haven't been used recently) that it can reclaim.
3. This behaviour surprised me a little - I was expecting the resident size of the heap to have reduced, but this doesn't seem to have happened. What we can see is that part of the heap is now 'clean' - this tells us that Linux has indeed flushed part of the heap out to the swap file. The fact that the resident size has not reduced significantly tells us that we aren't getting much benefit - basically I think that the swapper is trying to swap pages out but the garbage collector is pulling them all back in again.
   

Finally I went back to the JDK sources again to see whether these would help me to understand what was going on. What I really wanted to understand was where the per-object data used by the garbage collector resides. The answer appears to be that it resides at the beginning of the memory block allocated to the object itself. The implication of this is that with 'normal' sized objects, the garbage collector run will need to access a few fields at the start of every single object on the heap, thus generating read and write accesses to practically every page contained in the heap.

So it would seem to me that the design choices in terms of the in-memory layout of objects and their garbage collector data mean that the garbage collectors really do conflict with the swapper once memory becomes tight. Based on the simple test that I did earlier, this happens both suddenly and with a severe impact.

In real life situations there may be several other Java and non-Java apps running on the same machine. I think this has a couple of implications:-

1. The requirements of other apps may mean that memory becomes scarce much sooner - i.e. well before your Java heap size reaches the amount of physical memory.
   
2. The swapper is not redundant - there may be plenty of 'low risk' pages belonging to other apps (or JAR mappings used only at startup time) that can be swapped out before the system gets to the point of swap thrashing.

Java Heaps and Virtual Memory - Part 1

Virtual memory has been around for a long time - Wikipedia reckons that it was first introduced in the early 1960s and it's still with us today. When we start using Java for large scale applications, however, it seems that virtual memory is perhaps not such a good thing. Several sources around the Internet recommend sizing the Java heap so that it fits within physical memory. The reason given is that the Java garbage collector likes to visit every page, so if some pages have been swapped out the GC will take a long time to run.

This question has cropped up on several occasions in my current project. While I have no reason to disagree with the advice on heap sizing, I was a little uncomfortable that I hadn't seen much real evidence to back it up or indication of how bad things would be once the limit was reached, so I decided to find out for myself.

The first thing I tried was creating a simple Java class to stress the heap. This class will progressively populate an ArrayList with a large number of Java objects each owning five 80 byte random strings. It can also be asked to 'churn' the objects by selecting and replacing groups of them, thus making the old ones eligible for garbage collection. I ran this on a small Linux box and watched what happened using 'top' and 'vmstat' ...

What I found was this...

1. While there was plenty of free memory, the resident size of the process grew.
2. Once free memory became short, the shared size started to shrink
3. Very soon after that, the swap file usage started to grow
4. If the 'churn' feature of the stress test was enabled, the system quickly got into heavy swap thrashing and the stress test ground to a halt.
5. With no churn (probably not realistic for most real apps), the app could get a little further, but not much and would still get into swap thrashing.

My original intention was to capture some numbers and draw a graph or two to illustrate what happens. In practice what I found was that the results were rather variable, even on the same machine. In every case though there was a point soon after swapping started where the test tipped dramatically into swap thrashing and was unable to make any further progress.

I drew two conclusions from my simple test:-

1. The advice to keep the Java heap smaller than physical memory is very sound.
2. The degradation in performance if you let your Java heap grow bigger than physical memory is both sudden and severe.

I also wanted to check on the very limited explanations that I'd found and get a better understanding of what was going on. In particular, I wanted to know how Java (i.e. the Sun JVM) allocates heap memory and whether it adopts any strategies to avoid swap thrashing. I'll save this for a later post.

An Accident Waiting to Happen

My client is in the process of rolling out a multi-tier J2EE application which has been in development for a couple of years. A couple of weeks ago, it started to run into mysterious 'hanging' states in production. Many crisis meetings took place, DBAs and app server experts were engaged, actions happened and everybody had a generally stressful time without quite getting to the bottom of the issue.

In a quieter moment, going over emails from the DBAs, thread dumps and source code, I spotted a problem. A Java class was querying Oracle for the next value of a sequence, putting it in an instance variable and then using it later to do an insert or an update. Nothing too scary on the face of it... until you realise that an instance of this class was being held in an instance variable in another class, and that class was a WebLogic web service skeleton.

Now the WebLogic manual is very clear that you must write thread-safe code for your skeleton classes because the server will use a single instance of the skeleton to service all client requests. End result: a single instance of our class is running in multiple threads and the database keys are getting mixed up between threads, resulting in multiple threads trying to lock the same database row and causing the app to hang.

So should we blame the developers of this class?, well maybe, but bear in mind that the class in question is not itself a web service skeleton - it just happens to be used by one. Maybe we should blame the developer of the skeleton?, well that may be closer to the mark, but I have another option - maybe we should blame the developer of the web service framework!

My question is: is this rule reasonable? The expectation nowadays is that developers will quickly be up to speed with new technologies and that businesses will want to take advantage of them quickly. We cant expect development shops to be peopled entirely by seasoned professionals, so surely the burden should fall on the developers of frameworks to implement default behaviours which are safe.

Having hit this issue, I cross-checked what Axis does in the same situation. At first, I couldn't find any definitive statement in the manual on the subject, so I ran a service in the debugger and found that it created lots of instances of my skeleton as calls arrived. I eventually found that the bahaviour is configurable via the 'scope' property in the web service deployment descriptor (WSDD). The default seems to be 'session' - i.e. an instance of the skeleton is created for each client that connects. Not 100% safe, not 100% optimal performance-wise, but probably safe enough for most situations.

My client now needs to either check all web-service related code for thread safety or switch to another web service framework that at least makes this behaviour configurable... and then roll out the change into production and retro-fit the two releases which are still in the pipeline. This is not going to be a painless process.

I will just be thankful that I wasn't around when the framework was chosen or when the code in question was written - would I have spotted this accident before it happened?

Big Boxes

I like to work in an Agile project environment. Recently I have found myself in the happy position of being able to work for a client who is really buying into the whole Agile movement. The client is a very big telco, so its systems need to be... well... very big too.

This presents a few problems. One which is on my mind right now is how to size and budget for buying suitable hardware to host our new system. At the moment, we have tackled some of the lower volume products but in later cycles we will be doing some really big stuff against some really imperative deadlines.

We dont yet have detailed requirements for the big stuff (it's Agile, remember?), but we need to put a box in place which is going to have to cope with them. How to do it? Here are a couple of options...

• Rely on commodity boxes and horizontal scalability. That way, we can roll out incrementally and we dont need to know up front how many boxes we will eventually need. This works well for the app server tier, but is of less help for the database tier.


• Get the best picture we can of upcoming volumetric requirements (I dont think we can bet on 'You Arent Going To Need It') and pick a hardware range with enough memory, processor etc expansion capacity to support the eventual need.

To further complicate matters, we need to budget for floor space in the data centre - if we get it wrong, we may find that our boxes are hundreds of miles apart, which isnt going to do much for latency.

Challenges to Agile Development

It must be at least 12 years now since I attended the excellent two day course entitled 'Object Oriented Project Management'. Frankly it wasn't much about Object Oriented development and would perhaps have been better entitled 'Enlightened Project Management'. It introduced me to concepts like Boehm's spiral lifecycle, Tom Gilb's evolutionary delivery ideas and books of wisdom like 'The Mythical Man Month'. I had zero project management training prior to this but the wisdom on this course rather ruined me as an audience for the purveyors of more traditional project management techniques who have crossed my path since then.

As a result of this and my experience since then, I have become an advocate of agile development approaches (although it wasnt called 'agile' 12 years ago!). I have also encountered a lot of situations which pose challenges for agile practices. The main ones are:-

• Management of non-functional requirements - agile user stories tend to focus on obvious functional value. The fit of agile techniques to cross-cutting (but critical) concerns like security, performance or availability is less clear. Some have proposed non-functional user stories, but at the moment I think the jury is out.


• Outsourcing - having a customer/supplier relationship introduces barriers in communication, potentially conflicting motivation and reward systems and, in its usual form requires some form of big specification up front saying what is being paid for. All of these are driving us in the oppposite direction from the Agile Manifesto.


• Offshoring - an extension of outsourcing which introduces the additional challenges of physical distance, poor communications infrastructure, timezones, language and cultural differences - all of which conspire to hinder the free flow of communication which is the life blood of an agile project.

I dont claim to have all the answers to these issues. Some would say that a 'high ceremony' methodology is the only answer, but I'm not so sure. I do find myself facinng them on a regular basis, so at least I have some material for future posts!