Thursday, April 19, 2012

Arquillian initially overpromises and frustrates, but delivers real benefits

Arquillian is changing fairly rapidly, and the Arquillian folks are paying a lot of attention to feedback. This post discusses Arquillian 1.0.0.Final, and more importantly the ShinkWrap Resolver and ShrinkWrap Dependency extensions. A lot has already improved since I wrote this, though most of it hasn't hit -Final versions yet.


After seeing a lot of talk, hype and excitement about Arquillian on twitter for several months, I finally got around to introducing it into a new project to give it a try. I'm told it'll make testing massively easier and save me tons of time, so using it is a no-brainer.

After my recent experience I recommend that you start using it too - but you'll need to be prepared for some rough edges and the need for workarounds until a few point releases have gone by.

To jump straight to the summary of it all, click here, or read on for the whole experience.

Easy and magical

Like everything new in the Java EE world, people were talking about Arquillian as if it made it trivial to deploy real-world tests to the container so you could run realistic, proper integration tests of your container-based applications.

Like everything in the Java EE world, the reality fell greatly short of the hype, leading to lots of swearing, disappointment, and frustration. Again, like everything in the Java EE world, it was also released in a state where it looked superficially finished, but didn't feel finished once I tried to run more than trivial examples.

A great initial experience

I was really excited by Arquillian; it promised to finally make it practical to test more than tiny isolated fragments of my app in useful and meaningful ways. Being able to do things like test changes to the entity definitions and data access facade independently from the app front-end was very exciting. So was being able to test all my CDI-driven container-hosted code that just wasn't really testable in standalone JUnit.

Initial results were good - add the bom to my pom.xml, add the dependencies, add the container adapter, write and deploy a simple unit test and watch it run happily on the container.

Great! Now lets test part of the application by bundling the entities and persistence unit up with a couple of test EJBs to exercise them.

At this point, things started to go south*.

How do I add dependencies I need for testing?

Arquillian uses ShrinkWrap, which produces test archives that only include the assets you explicitly specify. Great idea, but how do I tell it to include apache commons lang3? I'm using Maven, so I don't exactly want to have to go specifying a path to that jar archive in the "lib/" dir I don't have, but that's what all the examples seem to do - those that don't avoid the issue by having no external dependencies at all. This seems really weird for a project that's so heavily Maven based.

I eventually found out that the ShrinkWrap Maven Dependency Resolver 2.0.0-alpha-1 extension was what I needed, but getting there took some doing.

Some digging in the docs and Google revealed the maven dependency resolver plugin for ShrinkWrap, which can be enabled by specifying it as a test dependency in your pom.xml . This would've been a lot easier to find if it'd been referenced from the Arquillian guides, reference guide or getting started guide, but none of those examples seem to use any external dependencies. (Or maybe the examples are there and I'm just blind.)

In any case, once I found out about the maven dependency resolver I added it to my pom.xml and tested it out - only to discover that it could load commons lang 3 if I specified an explicit version, but no matter what I tried it wouldn't add the seam 3 security module I needed. Discussion on the bug report revealed that there was a newer version, 2.0.0-alpha-1, than what was referenced by the Arquillian 1.0.0.Final BOM, and that this greatly improved the resolver. I updated to the newer resolver version and all was well. I could add all dependencies from my pom automatically, add them selectively without specifying versions, etc. Great!

Well, great once I worked out how to override the version specified in the Arquillian bom by putting the shrinkwrap resolver bom first in dependencymanagement, since just adding the new version to the dependencies list produced a broken deployment. Small things.

Time to add my persistence.xml and beans.xml.

Descriptor pain

My next adventure was adding the extremely-commonly-used beans.xml and persistence.xml descriptors to my project.

The only examples that use beans.xml just specify an empty beans.xml. I needed to add an alternative so I could swap a production class out for one that ensured the test environment was completely unable to access the production database. Easy, right?

Well, it's easy enough if you don't mind maintaining multiple copies of your descriptors by hand, so you can have a test-beans.xml and test-persistence.xml etc that you add as ShrinkWrap assets during archive creation. I thought that rather defeated the purpose of testing what I was then going to deploy in production as closely as possible, though - ie integration testing - so I wanted to load my production descriptors and make the minimum changes to make them safe for testing.

That was harder than expected. I found the ShrinkWrap descriptors module and added a dependency for it to my pom.xml. I had a few issues with it but got there once I updated to 2.0.0-alpha-2 (now standard in Arquillian 1.0.0.Final's bom) and found workarounds for some of the problems. I still can't actually load my beans.xml and replace one alternative with another because of SHRINKDESC-115 ... but I'm closer than I was.

Most of the ShrinkWrap descriptors issues I found were usability issues (ie: I did stupid things, but it let me do them and didn't make it obvious), but a few were bugs. Here are the reports I filed on Descriptors alone:

Still, I got there. I now had a test case that created the required descriptors, put them in the right place in the archive (by full explicit path), added the required maven dependencies, wrapped the test and application classes up, produced an archive, deployed it to the server, and ...

It still didn't work!

This one was my fault, but it sure was frustrating. It took me quite a while to figure out that the apparently inexplicable ClassNotFoundException I was getting where the test class its self was the subject of the exception was due to a one-character mistake in a string. I'd forgotten to change the archive name from ".jar" to ".war" when I changed from using JavaArchive to WarArchive.

At this point I'd spent a day and a half trying to get a simple unit test working, and was well past beginning to wonder if it was worth throwing the towel in and giving up on this "easy" testing framework.

I'm glad I didn't.

I finally started finding bugs in my code

Once I got the final issues with my test setup worked out, I started writing and running tests, quickly finding several issues with my entity definitions and database access code. It would've been much harder to track these down and debug them by hand, having to redeploy and then re-test via a browser, than it was with Arquillian. The edit, "mvn test", edit cycle is a lot faster, and more reproducible as well.

After a few hours fixing issues, I ran into one that seemed downright inexplicable.

Arquillian identifies a container bug that's a data-loss/corruption risk

AS7-4552 was a surprise.

I had a group of entities that had to be committed together because they had circular dependencies. Entity A had to have a minimum of one each of entity B and C referencing it. This required that entity A be INSERTed before entities B and C could be added to reference it, but entity A couldn't be committed until entities B and C had been INSERTed. This is easy to implement using a DEFERRABLE INITIALLY DEFERRED constraint in PostgreSQL, and worked fine in my SQL tests. When I tested insertion of the group of entities via JPA, though, I was getting constraint violation errors despite inserting them all in one transaction.

Examination of the database logs showed a conspicuous lack of any BEGIN or COMMIT statements, but I thought they might be being done at the protocol level not the SQL level.

My method was an EJB business method annotated with @TransactionAttribute(TransactionAttributeType.REQUIRED) and used container managed transactions with a JTA enabled EntityManager and persistence unit, so it had to be running within a transaction. Right?

Exhausting all other possibilities, I wrote a quick test that performed an operation PostgreSQL only allows when an explicit transaction is open (ie: autocommit is off).

It failed.

I'd found a container bug where, within transactional EJB business methods there was no transaction open. Statements were committed one by one as they ran, with no atomicity and no possibility of a rollback if the method failed. This could lead to extremely wrong results if atomic commit was required for correctness, not least because it effectively meant that every transaction was running with a DIRTY_READ isolation level, seeing "uncommitted" data from other transactions.

Nasty.

More testing showed that this wasn't confined to PostgreSQL; it affected the embedded H2 database and presumably others too. Something as simple as creating a savepoint then rolling back to it would fail because the creation and rollback happened in different transactions.

Other people had been bitten by this before, but it never seemed to get tracked down to the cause: JBoss AS 7 (and 5, and 4, and probably 6) have two different ways to define data sources, and only one of them works correctly (AS7-4552). Define your DS via a -ds.xml file or jboss-cli where JBoss wraps a JDBC driver's java.sql.Driver and everything is fine. Define your DS via web.xml / jboss-web.xml or via @DataSourceDefinition annotations, where JBoss AS 7 uses a 3rd-party provided javax.sql.DataSource, and you get busted transactional behaviour, leaked connections, and more.

Arquillian allowed me to write a simple, repeatable unit test that conclusively isolated the issue.

 Overall, Arquillian:

  • Cost me a couple of days of productivity to get working properly for a simple real-world test using a .Final released version;
  • Nearly made me give up in frustration several times;
  • Relies on unfinished and un- or under-documented add-ons for important and commonplace testing tasks;
  • Does a fantastic job once you finally get it working;
  • Helped me quickly find several significant bugs in my app;
  • Is increasing my confidence in the reliability and robustness of the code I add to my app; and
  • Helped me find a critical data loss/corruption bug in one of the most used Java application servers before I lost anything.

Was it worth it? Absolutely, but it was a miserable journey, and there are lessons to be learned here. There's something important to remember first, though:

Above all else, I really appreciate that the Arquillian and ShrinkWrap teams have chosen to release the software they created as open source, for free, so anybody can use it and benefit from it. Even if I thought it were complete crap (and I certainly don't, it's been really useful and does an amazing job) I'd have no right to complain, because I'm benefiting from their work, for free. Too many people forget that when they're dealing with open source projects, and as a contributor to a few OSS projects I've been on the receiving end of enough demanding users with attitudes of entitlement to know how annoying it is. It's kind of funny when they threaten to stop using your software if you don't do what they want, though - as if their using it does you any good. In open source, if you don't like it, don't use it, help fix it, or at least keep your comments constructive and focused on helping.

Of course, I slip on that sometimes myself - frustration can get the best of all of us occasionally, and the commercial-open-source nature of much of the Java world blurs the lines a lot. If I've done so at all in this post, my apologies, it can be a hard line to find sometimes.

Suggestions

  • Balance hype with realistic discussion of flaws and incomplete functionality.
  • Real-world examples, easily found.
  • Don't depend on alpha components in a final release if possible

Lessons

First: If something is 1.0.0.Final, but it depends on alpha code specified in the bom, then it's still alpha, not Final, unless those alpha dependencies are very clearly marked as incomplete and unsupported extensions - in which case they shouldn't be in the bom. The Arq docs don't point to them, but they're right there in the bom, and plenty of the Arq and ShrinkWrap blogs use them in unit tests without mentioning their limitations and defects, so it's easy to think they're ready for real-world use.

Second: If you claim it makes something easy, make sure you test this with realistic and real-world values of "something". I'm not just talking about Arquillian here, though I do think releasing it as Final without fairly solid support for adding dependencies to tests was a bit premature. I'm talking about everything I've ever used in the Java EE world: Java EE 6 its self, with its exciting warts and pitfalls; Glassfish 3.0.0 with CDI/Weld so broken I just got tired of reporting bugs; RichFaces 4.0.0; Mojarra/JSF2; you name it. I was moderately pleasantly surprised by JBoss AS 7.0.0.Final, which was released pretty broken and incomplete but unlike most Java EE software got fixed fast and rapidly improved to a very usable state.

I want to stress at this point that I know I'm not paying for any of this software, except in terms of the time I spend reporting issues, writing the odd patch, writing docs when I can, helping answer others' questions occasionally, etc. I haven't helped build it, and I'm not complaining that it's not good enough. What I'm saying is that the real quality level of a release needs to be more explicitly declared, that the hype and excitement about how incredibly wonderfully amazing and life-changing something is leads to a major letdown if it turns out to be as flawed as every other piece of software you've ever seen.

The Java world does hype more than pretty much anyone except Apple.

On a wider level, perhaps less hype would be a good thing for the Java community to cultivate? Almost every great amazing new technology I've tried in Java has barely deserved to be called an alpha release, with important functionality missing or broken and exciting bugs whenever you go to use even fairly obvious features.


Being constructive

Rather than just complain, I'm trying to do a bit to help:


* I live in Australia, so no offense ;-)

7 comments:

  1. It seems that that is pattern with Jboss guys lately. They seem to be focused on hype and advertisement much more coding and bug fixing. That produces the situation, that even when they announce something as final, it is not really so. More often than not, 'final' means it is just ready for the BETA stage. They wait for prople to start doing the testing, and than they will start producing versions x.0.1,x.0.2 etc. And 5-6 subversions later product begins to be useful in a non-trivial application.
    I learned this the hard way by introudced seam 3 to my clients when they announced it final.

    ReplyDelete
    Replies
    1. While there has been a lot of hype from JBoss about recent releases like AS7, Seam 3, and Arquillian, it is certainly not at the expense of coding. A truly incredible amount of work has gone into each of those projects, and continues to go into them daily. Perfection isn't achieved in a day, and focusing *too* much on the flaws can lead to overlooking the other 90% that's excellent. Understandable when the flaws are preventing you from using the good stuff, though.

      There is also a difficult balance to strike with how you communicate about a project. Too much caution and reserve can mean nobody tries it out. That means less testing, fewer bug reports, fewer potential new contributors, and potentially a much slower path to *actually* finished. Too much hype has problems, but so does not enough.

      I concur that Seam 3 wasn't ready, though. Most of it was broken on Glassfish, requiring additional undocumented configuration and additional dependencies not listed in the pom to work. Some of it was broken on AS7 too. Maybe it was release quality for AS5 as a target container? It didn't help that the different modules were at greatly varying stages of "ready" but were all released together and all included in the Seam 3 bom, including the ones that were still alpha. (Sound familiar?).

      Of course, I rely on Seam 3 solder and Seam 3 security continually now. I think they., and probably seam 3 persistence, should form part of a "Java EE 6.1" along with JSF2.2, JPA2.1, a JAX-RS with VDI support and its own injection mechanism deprecated, deprecation of @ManagedBean (both of them), mandatory warnings if @Inject is found with no beans.XML, etc. My point? Seam 3 had a rough start but has become a viral tool. I suspect Arquillan will be the same - but with a significantly less rough start.

      Delete
    2. For what it's worth, I also agree that the Seam 3 release was premature. I think there was some great work being done in the individual modules, but the process of mashing them together into a stack release was a bit of a train wreck.

      That said, it has turned a corner since. Like you, I agree that the value is still in the individual modules.

      You'll be glad to know that much of what you identify for fixing is either headed for Java EE 7 (to the degree the EGs will be responsive to the ideas) or DeltaSpike (http://tinyurl.com/deltaspike-wiki). The reason DeltaSpike is important is because it finally addresses in a vendor-neutral way an unavoidable reality: we are always going to want more out of Java EE than what the specs will give us. In other words, gaps are inevitable. Though, DeltaSpike could prove to be an important change agent to reduce the size of that gap.

      Delete
  2. I assure you that with Arquillian that is not the case. We are dedicated to coding, bug fixing and evangelism. That's not to say we are perfect. We will make mistakes. A final release in the community means that we have reached a stable API. Naturally bugs will be found, and we will write tests, fix those bugs and get out a point release. But we are aiming to be as polished as possible, as the effort in both Arquillian as a cross-cutting testing platform and arquillian.org hopefully demonstrates.

    On the other hand, this is open source and is not expected to be an enterprise product. We do put value on innovation and pushing the boundaries to advance software development. The more eyes we have on it, the more we can stay on the leading edge and have something that's very polished. As I said, we are aiming for high standards.

    I will read and value every line in this blog entry. We take this stuff very seriously.

    ReplyDelete
  3. After posting, I realized that my point about Arquillian 1.0.0.Final reaching a stable API does not address the dependencies on non-final ShrinkWrap APIs (in one case an alpha). Allow me to revise.

    The API for the Arquillian core platform is stable. That includes a stable version of ShrinkWrap archive (the main ShrinkWrap library). The other ShrinkWrap dependencies are add-ons and therefore don't "taint" the final status of Arquillian, per se.

    Of course, as you mention, they do appear in the Arquillian BOM, so are they or aren't they part of Arquillian.

    The reality that we have to face is that while we see a distinction between Arquillian and ShrinkWrap, from the user's standpoint, there isn't one. To them, it's all just Arquillian. Since resolvers and descriptors is needed to write tests with more than basic complexity, they are "forced" to depend on unstable APIs.

    At the moment, that's the best we've got. But we couldn't hold back the Arquillian core platform from a 1.0.0 release any longer. To move forward, we had to draw a line in the sand. We are confident it will prove to be the right decision for the evolution of the platform.

    ReplyDelete
  4. I enjoyed reading this because now I don't feel so alone! I have experienced some of your pain over the last few weeks, being new to both Maven and Arquillian. I am not quite there yet but it is encouraging that you found it worthwhile in the end. I think I will as well.

    ReplyDelete
  5. I enjoyed reading this because now I don't feel so alone! I have experienced some of your pain over the last few weeks, being new to both Maven and Arquillian.

    and glassfish


    +1

    ReplyDelete