Arquillian initially overpromises and frustrates, but delivers real benefits

Arquillian is changing fairly rapidly, and the Arquillian folks are paying a lot of attention to feedback. This post discusses Arquillian 1.0.0.Final, and more importantly the ShinkWrap Resolver and ShrinkWrap Dependency extensions. A lot has already improved since I wrote this, though most of it hasn't hit -Final versions yet.

---

After seeing a lot of talk, hype and excitement about Arquillian on twitter for several months, I finally got around to introducing it into a new project to give it a try. I'm told it'll make testing massively easier and save me tons of time, so using it is a no-brainer.

Tweet to @craigdevel After my recent experience I recommend that you start using it too - but you'll need to be prepared for some rough edges and the need for workarounds until a few point releases have gone by.

To jump straight to the summary of it all, click here, or read on for the whole experience.

Easy and magical

Like everything new in the Java EE world, people were talking about Arquillian as if it made it trivial to deploy real-world tests to the container so you could run realistic, proper integration tests of your container-based applications.

Like everything in the Java EE world, the reality fell greatly short of the hype, leading to lots of swearing, disappointment, and frustration. Again, like everything in the Java EE world, it was also released in a state where it looked superficially finished, but didn't feel finished once I tried to run more than trivial examples.

A great initial experience

I was really excited by Arquillian; it promised to finally make it practical to test more than tiny isolated fragments of my app in useful and meaningful ways. Being able to do things like test changes to the entity definitions and data access facade independently from the app front-end was very exciting. So was being able to test all my CDI-driven container-hosted code that just wasn't really testable in standalone JUnit.

Initial results were good - add the bom to my pom.xml, add the dependencies, add the container adapter, write and deploy a simple unit test and watch it run happily on the container.

Great! Now lets test part of the application by bundling the entities and persistence unit up with a couple of test EJBs to exercise them.

At this point, things started to go south^*.

How do I add dependencies I need for testing?

Arquillian uses ShrinkWrap, which produces test archives that only include the assets you explicitly specify. Great idea, but how do I tell it to include apache commons lang3? I'm using Maven, so I don't exactly want to have to go specifying a path to that jar archive in the "lib/" dir I don't have, but that's what all the examples seem to do - those that don't avoid the issue by having no external dependencies at all. This seems really weird for a project that's so heavily Maven based.

I eventually found out that the ShrinkWrap Maven Dependency Resolver 2.0.0-alpha-1 extension was what I needed, but getting there took some doing.

Some digging in the docs and Google revealed the maven dependency resolver plugin for ShrinkWrap, which can be enabled by specifying it as a test dependency in your pom.xml . This would've been a lot easier to find if it'd been referenced from the Arquillian guides, reference guide or getting started guide, but none of those examples seem to use any external dependencies. (Or maybe the examples are there and I'm just blind.)

In any case, once I found out about the maven dependency resolver I added it to my pom.xml and tested it out - only to discover that it could load commons lang 3 if I specified an explicit version, but no matter what I tried it wouldn't add the seam 3 security module I needed. Discussion on the bug report revealed that there was a newer version, 2.0.0-alpha-1, than what was referenced by the Arquillian 1.0.0.Final BOM, and that this greatly improved the resolver. I updated to the newer resolver version and all was well. I could add all dependencies from my pom automatically, add them selectively without specifying versions, etc. Great!

Well, great once I worked out how to override the version specified in the Arquillian bom by putting the shrinkwrap resolver bom first in dependencymanagement, since just adding the new version to the dependencies list produced a broken deployment. Small things.

Time to add my persistence.xml and beans.xml.

Descriptor pain

My next adventure was adding the extremely-commonly-used beans.xml and persistence.xml descriptors to my project.

The only examples that use beans.xml just specify an empty beans.xml. I needed to add an alternative so I could swap a production class out for one that ensured the test environment was completely unable to access the production database. Easy, right?

Well, it's easy enough if you don't mind maintaining multiple copies of your descriptors by hand, so you can have a test-beans.xml and test-persistence.xml etc that you add as ShrinkWrap assets during archive creation. I thought that rather defeated the purpose of testing what I was then going to deploy in production as closely as possible, though - ie integration testing - so I wanted to load my production descriptors and make the minimum changes to make them safe for testing.

That was harder than expected. I found the ShrinkWrap descriptors module and added a dependency for it to my pom.xml. I had a few issues with it but got there once I updated to 2.0.0-alpha-2 (now standard in Arquillian 1.0.0.Final's bom) and found workarounds for some of the problems. I still can't actually load my beans.xml and replace one alternative with another because of SHRINKDESC-115 ... but I'm closer than I was.

Most of the ShrinkWrap descriptors issues I found were usability issues (ie: I did stupid things, but it let me do them and didn't make it obvious), but a few were bugs. Here are the reports I filed on Descriptors alone:

• (bug) Descriptors 2.0 module creates duplicate entries (test case attached)
• (usability) Confusing exception when filename string passed to DescriptorImporter.from(...)
• (feature req): Automatic paths for descriptor import from project and packaging into archives
• (usability)Descriptors 2.0 module BeansDescriptor API: common operations difficult, API confusing

Still, I got there. I now had a test case that created the required descriptors, put them in the right place in the archive (by full explicit path), added the required maven dependencies, wrapped the test and application classes up, produced an archive, deployed it to the server, and ...

It still didn't work!

This one was my fault, but it sure was frustrating. It took me quite a while to figure out that the apparently inexplicable ClassNotFoundException I was getting where the test class its self was the subject of the exception was due to a one-character mistake in a string. I'd forgotten to change the archive name from ".jar" to ".war" when I changed from using JavaArchive to WarArchive.

At this point I'd spent a day and a half trying to get a simple unit test working, and was well past beginning to wonder if it was worth throwing the towel in and giving up on this "easy" testing framework.

I'm glad I didn't.

I finally started finding bugs in my code

Once I got the final issues with my test setup worked out, I started writing and running tests, quickly finding several issues with my entity definitions and database access code. It would've been much harder to track these down and debug them by hand, having to redeploy and then re-test via a browser, than it was with Arquillian. The edit, "mvn test", edit cycle is a lot faster, and more reproducible as well.

After a few hours fixing issues, I ran into one that seemed downright inexplicable.

Arquillian identifies a container bug that's a data-loss/corruption risk

AS7-4552 was a surprise.

I had a group of entities that had to be committed together because they had circular dependencies. Entity A had to have a minimum of one each of entity B and C referencing it. This required that entity A be INSERTed before entities B and C could be added to reference it, but entity A couldn't be committed until entities B and C had been INSERTed. This is easy to implement using a DEFERRABLE INITIALLY DEFERRED constraint in PostgreSQL, and worked fine in my SQL tests. When I tested insertion of the group of entities via JPA, though, I was getting constraint violation errors despite inserting them all in one transaction.

Examination of the database logs showed a conspicuous lack of any BEGIN or COMMIT statements, but I thought they might be being done at the protocol level not the SQL level.

My method was an EJB business method annotated with @TransactionAttribute(TransactionAttributeType.REQUIRED) and used container managed transactions with a JTA enabled EntityManager and persistence unit, so it had to be running within a transaction. Right?

Exhausting all other possibilities, I wrote a quick test that performed an operation PostgreSQL only allows when an explicit transaction is open (ie: autocommit is off).

It failed.

I'd found a container bug where, within transactional EJB business methods there was no transaction open. Statements were committed one by one as they ran, with no atomicity and no possibility of a rollback if the method failed. This could lead to extremely wrong results if atomic commit was required for correctness, not least because it effectively meant that every transaction was running with a DIRTY_READ isolation level, seeing "uncommitted" data from other transactions.

Nasty.

More testing showed that this wasn't confined to PostgreSQL; it affected the embedded H2 database and presumably others too. Something as simple as creating a savepoint then rolling back to it would fail because the creation and rollback happened in different transactions.

Other people had been bitten by this before, but it never seemed to get tracked down to the cause: JBoss AS 7 (and 5, and 4, and probably 6) have two different ways to define data sources, and only one of them works correctly (AS7-4552). Define your DS via a -ds.xml file or jboss-cli where JBoss wraps a JDBC driver's java.sql.Driver and everything is fine. Define your DS via web.xml / jboss-web.xml or via @DataSourceDefinition annotations, where JBoss AS 7 uses a 3rd-party provided javax.sql.DataSource, and you get busted transactional behaviour, leaked connections, and more.

Arquillian allowed me to write a simple, repeatable unit test that conclusively isolated the issue.

 Overall, Arquillian:

• Cost me a couple of days of productivity to get working properly for a simple real-world test using a .Final released version;
• Nearly made me give up in frustration several times;
• Relies on unfinished and un- or under-documented add-ons for important and commonplace testing tasks;
• Does a fantastic job once you finally get it working;
• Helped me quickly find several significant bugs in my app;
• Is increasing my confidence in the reliability and robustness of the code I add to my app; and
• Helped me find a critical data loss/corruption bug in one of the most used Java application servers before I lost anything.

Was it worth it? Absolutely, but it was a miserable journey, and there are lessons to be learned here. There's something important to remember first, though:

Above all else, I really appreciate that the Arquillian and ShrinkWrap teams have chosen to release the software they created as open source, for free, so anybody can use it and benefit from it. Even if I thought it were complete crap (and I certainly don't, it's been really useful and does an amazing job) I'd have no right to complain, because I'm benefiting from their work, for free. Too many people forget that when they're dealing with open source projects, and as a contributor to a few OSS projects I've been on the receiving end of enough demanding users with attitudes of entitlement to know how annoying it is. It's kind of funny when they threaten to stop using your software if you don't do what they want, though - as if their using it does you any good. In open source, if you don't like it, don't use it, help fix it, or at least keep your comments constructive and focused on helping.

Of course, I slip on that sometimes myself - frustration can get the best of all of us occasionally, and the commercial-open-source nature of much of the Java world blurs the lines a lot. If I've done so at all in this post, my apologies, it can be a hard line to find sometimes.

Suggestions

• Balance hype with realistic discussion of flaws and incomplete functionality.
• Real-world examples, easily found.
• Don't depend on alpha components in a final release if possible

Lessons

First: If something is 1.0.0.Final, but it depends on alpha code specified in the bom, then it's still alpha, not Final, unless those alpha dependencies are very clearly marked as incomplete and unsupported extensions - in which case they shouldn't be in the bom. The Arq docs don't point to them, but they're right there in the bom, and plenty of the Arq and ShrinkWrap blogs use them in unit tests without mentioning their limitations and defects, so it's easy to think they're ready for real-world use.

Second: If you claim it makes something easy, make sure you test this with realistic and real-world values of "something". I'm not just talking about Arquillian here, though I do think releasing it as Final without fairly solid support for adding dependencies to tests was a bit premature. I'm talking about everything I've ever used in the Java EE world: Java EE 6 its self, with its exciting warts and pitfalls; Glassfish 3.0.0 with CDI/Weld so broken I just got tired of reporting bugs; RichFaces 4.0.0; Mojarra/JSF2; you name it. I was moderately pleasantly surprised by JBoss AS 7.0.0.Final, which was released pretty broken and incomplete but unlike most Java EE software got fixed fast and rapidly improved to a very usable state.

I want to stress at this point that I know I'm not paying for any of this software, except in terms of the time I spend reporting issues, writing the odd patch, writing docs when I can, helping answer others' questions occasionally, etc. I haven't helped build it, and I'm not complaining that it's not good enough. What I'm saying is that the real quality level of a release needs to be more explicitly declared, that the hype and excitement about how incredibly wonderfully amazing and life-changing something is leads to a major letdown if it turns out to be as flawed as every other piece of software you've ever seen.

The Java world does hype more than pretty much anyone except Apple.

On a wider level, perhaps less hype would be a good thing for the Java community to cultivate? Almost every great amazing new technology I've tried in Java has barely deserved to be called an alpha release, with important functionality missing or broken and exciting bugs whenever you go to use even fairly obvious features.

Being constructive

Rather than just complain, I'm trying to do a bit to help:

• FAQ: How do I add maven artifacts to my ShrinkWrap archives?
• When I run my Arquillian test, I get a ClassNotFoundException for the unit test class its self! How is that possible?
  
• FAQ: When I run an Arquillian test that uses ShrinkWrap descriptors, I get a nonsensical SAXParseException "content is not allowed in prolog"
  
• FAQ: How do I create or copy and modify persistence.xml, beans.xml, etc?
• ... and a bunch of bug reports (even one trivial patch)

---

^* I live in Australia, so no offense ;-)