Groovy for database business logic

My first impressions of Groovy as a way to solve my database-oriented business logic problem are very good.

I started out by re-implementing the stored procedure logic that I'd prototyped as a Groovy class. I got it working and passing the unit test in a couple of hours. My first implementation was fairly horrid, but over the next few hours I figured out some more of the Groovy idioms and removed some Java style code which made the whole thing more expressive and easier to follow.

One thing which took some figuring out was date handling and date arithmetic. I started out using the java Calendar class, which was somewhat verbose. I then found http://groovy.codehaus.org/JN0545-Dates which explained some useful stuff like the Groovy Duration classes. These made my code much more concise, although I'm not too comfortable about the fact that you can do 'duration + java date', but not the other way round. I can see exactly why it is so (because java.util.Date doesnt have a 'plus' method), but it's rather counter-intuitive.

While it was in the state of being a 'like for like' copy, I also took the opportunity to do a couple of comparisons:-

• lines of code (including blanks and comments) - 66 for Groovy, 87 for the stored procedure - the savings were mainly down to the elimination of variable declarations at the start of the procedure and the much simpler looping syntax with Groovy.
• performance - I was only able to test on my local Windows machine, which is not ideal, but with a sensible amount of data, the Groovy version was taking about 7% longer than the stored procedure version.

Since my code has an inner loop with a complex SQL insert/select statement in it, I thought it worth looking at what Groovy was doing with this. Using the (Eclipse) debugger, I found that it was creating a new PreparedStatement for each iteration of the loop, which I thought may be costing some performance. I tried expanding this operation out to explicitly create a PreparedStatement outside the loop and execute it inside the loop. The result was zero improvement in performance, but some pretty ugly code so I undid this change.

Overall, Groovy looks like it is meeting my requirements as a way to implement complex database business logic, so I'm now about to start on turning my prototype logic into something which can meet the rather more complex requirements of the real thing.

Time to be more Groovy

Following a recommendation from a colleague, I've had good intentions to learn and use more Groovy for quite a while now, but somehow havent quite had a problem that really presents itself as the ideal starting point.

A few months back I took the initial plunge and used Groovy to do a real job. The job probably wasnt the best choice as a first Groovy project - it was basically pulling data from a multi-gigabyte binary file (actually WebLogic JMS storage files), but it needed to be done and I got the job done with Groovy. In the end I probably wished I hadn't done it this way because I couldn't then use the resulting script anywhere else without getting Groovy set up there first and the alternative was shifting the huge files to the machine where the script was working. I don't think I really learned much idiomatic Groovy either.

Today I have a new problem to solve - basically I have some complex pricing calculations to do based on data in a relational database. The calculations need to be done both in scheduled batch mode and 'on the fly' within a J2EE app for 'what-if' style illustrations. They also need to be very pluggable so that entirely different pricing structures can be supported.

Thinking about this problem, what I need is:-

• Must be runnable from a J2EE app
• Must have good DB integration
• Must have automated unit tests
• Must have concise, expressive logic and preferably polymorphism
• Must be easy to debug

The initial prototype (with lots of simplifying assumptions) was done as a stored procedure with DBUnit test cases, which nicely meets the first three requirements, but runs out of steam on the third. I think that continuing with this approach is going to result in something pretty cumbersome as I gradually remove the simplifying assumptions.

Native Java didn't seem too attractive in terms of really tight DB integration - both JDBC and Hibernate fall a long way short of the simplicity with which SQL can be used in a stored procedure.

So I took another look at Groovy - it seems to fit the bill pretty well. I suspect that GSQL will be rather less cumbersome than either raw JDBC or Hibernate, although probably not quite as closely integrated as running SQL inside a stored proc. I was also worried (based on past experience of Jython and Javascript) about debugging, but it seems that this angle is also covered with both JSwat and Eclipse being supported. Hopefully it will also get me into more idiomatic Groovy in the process...

We shall see!

An Accident Waiting to Happen

My client is in the process of rolling out a multi-tier J2EE application which has been in development for a couple of years. A couple of weeks ago, it started to run into mysterious 'hanging' states in production. Many crisis meetings took place, DBAs and app server experts were engaged, actions happened and everybody had a generally stressful time without quite getting to the bottom of the issue.

In a quieter moment, going over emails from the DBAs, thread dumps and source code, I spotted a problem. A Java class was querying Oracle for the next value of a sequence, putting it in an instance variable and then using it later to do an insert or an update. Nothing too scary on the face of it... until you realise that an instance of this class was being held in an instance variable in another class, and that class was a WebLogic web service skeleton.

Now the WebLogic manual is very clear that you must write thread-safe code for your skeleton classes because the server will use a single instance of the skeleton to service all client requests. End result: a single instance of our class is running in multiple threads and the database keys are getting mixed up between threads, resulting in multiple threads trying to lock the same database row and causing the app to hang.

So should we blame the developers of this class?, well maybe, but bear in mind that the class in question is not itself a web service skeleton - it just happens to be used by one. Maybe we should blame the developer of the skeleton?, well that may be closer to the mark, but I have another option - maybe we should blame the developer of the web service framework!

My question is: is this rule reasonable? The expectation nowadays is that developers will quickly be up to speed with new technologies and that businesses will want to take advantage of them quickly. We cant expect development shops to be peopled entirely by seasoned professionals, so surely the burden should fall on the developers of frameworks to implement default behaviours which are safe.

Having hit this issue, I cross-checked what Axis does in the same situation. At first, I couldn't find any definitive statement in the manual on the subject, so I ran a service in the debugger and found that it created lots of instances of my skeleton as calls arrived. I eventually found that the bahaviour is configurable via the 'scope' property in the web service deployment descriptor (WSDD). The default seems to be 'session' - i.e. an instance of the skeleton is created for each client that connects. Not 100% safe, not 100% optimal performance-wise, but probably safe enough for most situations.

My client now needs to either check all web-service related code for thread safety or switch to another web service framework that at least makes this behaviour configurable... and then roll out the change into production and retro-fit the two releases which are still in the pipeline. This is not going to be a painless process.

I will just be thankful that I wasn't around when the framework was chosen or when the code in question was written - would I have spotted this accident before it happened?

Contracts for Agile Outsourcing

The question of how to frame contracts and customer/supplier relationships for an agile project is a tough one. To use some patterns terminology, the fundamental 'forces' are:-

1. The customer wants to be sure that they will get value for money


2. The supplier wants to know what they need to do to get paid


3. Agile tells us that we cant specify systems well, so we need to 'embrace change' and minimise the cost of making changes mid-course.

Force 3 is not a big issue for the customer, but will make suppliers used to 'Big Design Up Front' (BDUF) pretty uncomfortable since it is making the commitment rather open-ended and conflicting with force 2.

One possible way to resolve the forces is with a time and materials (T&M) contract. This resolves forces 2 & 3 from the supplier's perspective nicely, but at a first glance, is not a good answer for force 1. In practice, it is not so bad as it seems, - if iteration cycles are kept short and real value is delivered during each iteration then the customer is protected by a rapid feedback mechanism to ensure that things are on a productive track (even if it is not the same track which was envisioned at the start!).

Life isnt always like that though and sometimes customer purchasing policy may insist on fixed price contracts. So what do we do then?

One variation on the fixed price contract is the fixed effort contract in which the supplier commits to providing a specified number of man days to the project. This may be thought of as T&M in fixed price clothing, so its resolution of the forces is similar to those of T&M, with the additional benefit of keeping the purchasing department happy.

Then there are customers who insist on fixed price AND fixed scope. This is a tricky situation to reconcile with agile, because 'fixed scope' sounds a lot like BDUF, which is exactly what Agile tells us we don't know how to do well. On the face of it then, the customer's expectations of Agile and fixed scope are contradictory, so how does a supplier cope? Well, there are several possibilities:-

• Accept the business and the risk of unscoped changes. This may be acceptable where the customer and supplier have a track record with each other and are comfortable that the risk can be assessed.


• Say 'no thanks' and walk away. This doesnt happen very often in my experience. Maybe it should!


• Adopt a two-tiered approach to requirements and agility in which 'broad brush' high level requirements (epic user stories?) are used to fix scope but at a detailed user story level, agile practices are followed.


• Allocating a fixed size change budget which the customer can spend during the period of the project, although this does require impact assessment or detailed tracking of activity for each change, both of which may be counter to the spirit of 'embrace change'

And finally, when you really know your stuff, there are customers who want a fixed price and business benefit-related payments. This needs the supplier to at least be able to size the work up front without any commitent to a scope (correct or not) from the customer. The commitment here from the supplier's side is so open-ended that the he must be really sure that he knows both the business and the customer. My feeling on this is that Agile is actually very well suited to such a scenario - in this case the rapid feedback mechanism which I described earlier will help the supplier to pick up problems early, although his options for resolving them without affecting the bottom line may be limited. Clearly it is better if the benefit measurement can be broken down into the same short iterations that the delivery is broken into, so that tracking is clear and course corrections can be done quickly. Tom and Kai Gilb's article on Incremental value delivery and measurement includes some very relevant advice for suppliers and customers who are using value-based payments.

Big Boxes

I like to work in an Agile project environment. Recently I have found myself in the happy position of being able to work for a client who is really buying into the whole Agile movement. The client is a very big telco, so its systems need to be... well... very big too.

This presents a few problems. One which is on my mind right now is how to size and budget for buying suitable hardware to host our new system. At the moment, we have tackled some of the lower volume products but in later cycles we will be doing some really big stuff against some really imperative deadlines.

We dont yet have detailed requirements for the big stuff (it's Agile, remember?), but we need to put a box in place which is going to have to cope with them. How to do it? Here are a couple of options...

• Rely on commodity boxes and horizontal scalability. That way, we can roll out incrementally and we dont need to know up front how many boxes we will eventually need. This works well for the app server tier, but is of less help for the database tier.


• Get the best picture we can of upcoming volumetric requirements (I dont think we can bet on 'You Arent Going To Need It') and pick a hardware range with enough memory, processor etc expansion capacity to support the eventual need.

To further complicate matters, we need to budget for floor space in the data centre - if we get it wrong, we may find that our boxes are hundreds of miles apart, which isnt going to do much for latency.

Challenges to Agile Development

It must be at least 12 years now since I attended the excellent two day course entitled 'Object Oriented Project Management'. Frankly it wasn't much about Object Oriented development and would perhaps have been better entitled 'Enlightened Project Management'. It introduced me to concepts like Boehm's spiral lifecycle, Tom Gilb's evolutionary delivery ideas and books of wisdom like 'The Mythical Man Month'. I had zero project management training prior to this but the wisdom on this course rather ruined me as an audience for the purveyors of more traditional project management techniques who have crossed my path since then.

As a result of this and my experience since then, I have become an advocate of agile development approaches (although it wasnt called 'agile' 12 years ago!). I have also encountered a lot of situations which pose challenges for agile practices. The main ones are:-

• Management of non-functional requirements - agile user stories tend to focus on obvious functional value. The fit of agile techniques to cross-cutting (but critical) concerns like security, performance or availability is less clear. Some have proposed non-functional user stories, but at the moment I think the jury is out.


• Outsourcing - having a customer/supplier relationship introduces barriers in communication, potentially conflicting motivation and reward systems and, in its usual form requires some form of big specification up front saying what is being paid for. All of these are driving us in the oppposite direction from the Agile Manifesto.


• Offshoring - an extension of outsourcing which introduces the additional challenges of physical distance, poor communications infrastructure, timezones, language and cultural differences - all of which conspire to hinder the free flow of communication which is the life blood of an agile project.

I dont claim to have all the answers to these issues. Some would say that a 'high ceremony' methodology is the only answer, but I'm not so sure. I do find myself facinng them on a regular basis, so at least I have some material for future posts!