full fathom five

At my enterprisey day job I’m finding the JRuby platform to be the perfect glue for bringing together a (sadly necessary) slew of Java libraries and middleware (3rd-party Ant tasks, Jakarta VFS, ServiceMix, Maven, Weblogic) to solve a complex problem crucial to our enterprise. Stellar support for exception handling is obviously important when integrating so many libraries and middleware so how well does JRuby help with this? Is it just a Java fair-weather friend?

  require 'java'

  begin
    # some bad Java mojo
  rescue RuntimeError => e
    puts "Java or Ruby exception: #{e}"
    raise
  end

  require 'java'

  begin
    # some bad Java mojo
  rescue NativeException => e
    puts "Native exception's cause: #{e.cause}"
  end

  begin
    # some bad Java mojo
  rescue NativeException => e

    #creates a Ruby proxy for the Java exception
    proxied_e = JavaUtilities.wrap(e.cause)

    proxied_e.print_stack_trace
  end

  require 'java'
  include_class 'java.lang.IOException'

  begin
    # bad IOException-inducing mojo
  rescue IOException => e
    puts "Java IO exception: #{e.message}"
  end

I recently needed to debug a library which seemed to be misbehaving when used within Ant. After a bit of research here’s the best way I found to debug within Ant that doesn’t require sprinkling your bin/ant or bin\ant.bat with temporary JPDA pixy dust.

Ant doesn’t directly accept the -X arguments needed to set up the JPDA settings so the question is: how do you pass these from a shell or command line? In this, the ANT_OPTS environment variable is your friend. Its contents will be directly passed to the Java process Ant calls.

So here’s how you do it:

Step 1: Set ANT_OPTS to the debug options appropriate for your JVM:

for Java 1.4:

>> export ANT_OPTS=-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8453

for Java 1.5:

>> export ANT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8453

Step 2: Run ant like normal:

  
>> ant

Step 3: Set your breakpoints.

Step 4: Now attach your debugger.

There might be better ways to do this but this was the least invasive route for me.

I've been developing in Java for pretty much since Java 1.0. While working on the ANTLR Java 5 grammar I was suprised to rediscover some features of the language I'd just plain forgot.

First up, while working on annotation array initialisers, I was pleasantly reminded that the final element of a regular array can have a trailing comma:

    String[] foo =
        new String[]
        {
            "qwerty1",
            "qwerty2",
            "qwerty3",
        };

This is a nice little convenience for when you need to expand or contract the array elements in the initialiser - it means you don't have to fiddle with the final element to make sure there is no trailing comma.

Later I was reminded that interfaces may be package protected. For some reason i'd deluded myself into thinking interfaces could only be public:

    package foo;
    interface Bar {}
    ...
    package foo2;
    class Bar2
    {
       foo.Bar b;
           //compilation error: foo.Bar can't
           //be accessed from outside its package
    }

Duh! How could I have lived without that one? I often complained to co-workers that I couldn't do this - by their response they obviously didn't know you could either.

Finally, I was recently reminded that you can use the .class 'field' on primitive types:

    Class c = int.class;

I've long been aware of this 'field' on regular classes and interfaces (e.g. String.class) but when faced with the need to get the class of a primitve type I'd use the helper fields (e.g. Integer.TYPE). This forgotten feature is nicely consistent.

I use quotations around the word field here because it's not really a field, just a few grains of syntactic sugar (decompile a class that uses the .class 'field' to see what I mean).

What's your forgotten language feature?

After years of toying with Antlr but never actually doing anything useful with it; and after months of merely skimming articles on new Java 5 language features I thought I’d deepen my understanding of both and update the Antlr Java 1.4 grammar to recognize the new language features.

Suprisingly it only took about a week to polish off, and most of that was spent just cleaning up the grammar and deciding between different ways to structure the output AST (rather than strugling with Antlr tool itself).

For anyone interested, my grammar can be found on my site and it’s also hosted at antlr.org

As for how well tested it is: I’ve used it to successfully recognise the entire Java 5 source base, jdigraph’s source base (an early adopter of generics) and some custom tests I developed. I found the accompanying AST recogniser (as opposed to the token stream recogniser) is a good tool for confirming your tree is structured as you expected it to be. I also visually inspected the AST to ensure its structure was what I expected. An alternative to visual inspection would be great to automatically ensure the grammar stays correct over a lifetime of modifications. I’d love to hear from anyone who’s worked on such a tool. A testing framework that used XQuery or XPath querying/assertions on DOM-adapted AST (similar to PMD’s AST interface) would be my first approach. Comments or experience anyone?

More comments on Antlr and Java 5’s new language features later.

One of my favourite things at the moment is the HBO series Sopranos. I’m probably not alone in saying one of my least favourite things is dealing with IT recruitment consultants. Over the past few days, dealings with a recruiter has allowed me to reflect on the strange similarities between the two.

One thing that struck me about the Sopranos series is that everyone’s existences are predicated on lies. Every conversation is one where nothing is delivered as it’s meant, even to those most close to the speaker: Tony and his crew lie to each other about their observance of honour and family; Tony’s wife, Carmella, lies to herself about the goodness of her husband; Tony and Carmella choose euphamisms to describe the ‘business’ to their children and how they might ‘brush up against organised crime’.

Dealing with recruitment consultants can be a lot like that (minus the organised crime). Whenever I deal with thim I find my slightly cynical inner voice whispering to me what I really think they’re saying. Here’s a kind of transcript from my most recent encounter with one of them.

Mr Headhunter: It’s Mr Headhunter from Bum-fluff IT. What’s your availability and status?
Mr Developer: Currently available, looking for permanent roles.

Mr Headhunter: Would you consider a short term contract (4-6 weeks)?
Mr Developer: Sure, depending on the opportunity.

Mr Headhunter: [Clueless guff about the role and the company followed by…] For some reason when this role came up I thought of your CV.
Translation: This client bills itself as a “New Media” company; your first role was for a company with “New Media” in its title (if not in its projects); the CV database returned must have returned your CV first from a list of thousands through some tenuous Bayesian relevance.

Mr Headhunter: So you do have extensive New Media experience then?
Mr Developer: Well, I’d be lying if I said the majority of my experience was in that area but I did work on one highly interactive content management-driven international portal for an Australian NGO so I’m very comfortable in a New Media environment.
Translation: Do people actually still use the term New Media?
Mr Headhunter: You sound like you have the right skills for the position, I’d like to send your resume accross but I’ll need you to tailor it for my client’s requirements. I’ll send over an email with some points I think you should address.
Translation: Do you know how to spell ‘spin’?

So, into my inbox arrived this little gem of an email, complete with sentiment as sincere as one of those feel-good bank commercials:

Hi Michael,

> I reckon the XYZ Company role could be the direction you should be taking 
with your current situation. 
> 
> Please send me an updated CV highlighting all your new media experience...
> 
> Please make sure the following is evident in your CV:
> 
> extensive new media experience
> java , j2ee, web development, html, EJB, weblogic, struts, unix system admin, 
php,
...
... 

Obviously when Mr Headhunter cut and pasted this from his email to somebody else he either didn’t have the patience to remove the ’>’ quote markers or he used some email client like Outlook where quote markers aren’t obvious but are translated on my end by Yahoo mail. It pleases me to no end that he’s only thinking of my best interests.

Notice too how he asks me to tailor my CV so that “extensive new media experience” is evident even though I indicated, in the most self-promoting of ways, that New Media wasn’t where most of my experience lay? Obviously a man who eats spin for breakfast.

Mr Headhunter: So what did you think of… [yadda yadda]
Mr Developer: Well I very much liked… [yadda yadda]
Mr Headhunter: That’s great. So what were some of the questions he asked you, specifics would be good?
Translation: So that I can sell you down the river and maximise the probability of placing someone, I need to know all the interview questions he asked so that I can supply them to the next candidate I put forward.
Mr Developer: Oh, just general questions on what I’ve worked on in the past.
Translation: You think I’m that stupid?

Mr Headhunter: The client really liked you and thought you’d be perfect for the job and would like to take you on. He indicated, however, that he is having some internal budgetary problems and will have to sort those out first, I’ll call you on Friday afternoon to update you.
Translation: I’ll call you if there’s still a contract, if not I will ignore you and your phone calls unless you hound me.
Mr Developer: Sure, I look forward to it.
Translation: I have your number on auto-dial.

Needless to say there were budgetary problems, the contract fell through and I had to hound the recruiter to find all this out.

Mr Headhunter: Yeah, sorry I didn’t get back to you.
Translation: Who are you again? Oh, you, you didn’t make me any money did you?
Mr Headhunter: Yeah, that contract did fall through but don’t worry, since you interviewed well I’ve put this down in my records and this will be the start of a great working relationship.
Translation: Until the next cold call, you, whatever your name is.
Mr Developer: Yes, Please keep me in mind for any other opportunities that may come up.
Translation: Didn’t you say the same things to me last year?

It would be unfair of me to claim that all recruitment consultants are as bad as this buffoon. I’ve been fortunate enough to come accross a number of excellent recruiters such as Paul at Cautela-IT in Colchester and Carol McCormack from CCS Index back in Australia. I’m sure there are many others out there, I just hope I happen to stumble across them when looking for my next role.

One of the first problems I faced when I deployed this blog was I'd wake up in the morning to check that it was still running from the previous night and find all its database connections hosed and the web app in dire need of a restart. If you're finding this too, here's how I solved it.

Essentially the problem is that MySQL will, after a fairly long period of inactivity (8hrs?), drop database connections. If your application is holding a bunch of these and it is inactive for that period of time then all your connections will be invalid. Unfortunately there is no way to asynchronously detect that you've been disconnected (and thus rectify the problem) so you'll only find out once you try to use the connection.

This in itself is nothing new, all the major web application servers I've used provide some strategy for handling this situation. It's just that Hibernate is such a powerful tool that alleviates so much of the Object-Relational headache that I expected it would have taken care of this for me too. Aparently I'm not alone: http://www.hibernate.org/117.html#A13. This is no criticism of Hibernate, more a compliment, in fact.

The FAQ entry is quite clear - Hibernate's default database connection pool is not well suited to production use - it is not error tolerant so failures like the above will not be recovered from. Luckily, it's no great pain to configure Hibernate so that it uses a decent connection pool that detects and recovers from these problems before your code has to choke on them.

Firstly, you need to decide on a database connection pool - DBCP or C3PO seem to be the two major ones supported directly. Since I've only used DBCP, here's what I did to get going with it.

Your Hibernate configuration probably looks something like this:

<hibernate-configuration>
    <session-factory>
        <!-- properties -->
        <property
         name="connection.driver_class"
           >com.mysql.jdbc.Driver</property>
        <property
         name="connection.url"
           >jdbc:mysql:/localhost/mydb</property>
        <property
         name="connection.username"
           >foo</property>
        <property name="connection.password"
           >bar</property>
        ...
        ...
    </session-factory>
</hibernate-configuration>

You need to amend this to select DBCP as the connection pool to use and provide it with properties to correctly configure it. Here are the parameters I use:

        <property name="dbcp.maxActive">15</property>
        <property name="dbcp.maxIdle">5</property>
        <property name="dbcp.maxWait">120000</property>
        <property name="dbcp.whenExhaustedAction">1</property>
        <property name="dbcp.testOnBorrow">true</property>
        <property name="dbcp.testOnReturn">true</property>
        <property name="dbcp.validationQuery">
            select count(*) from some-table
        </property>
        <property name="dbcp.ps.maxActive">0</property>
        <property name="dbcp.ps.maxIdle">0</property>
        <property name="dbcp.ps.maxWait">-1</property>
        <property name="dbcp.ps.whenExhaustedAction">2</property>

Briefly, the significance of each is as follows:

• maxActive - the maximum number of connections in the pool in use at any one time. The pool will not grow beyond this size
• maxIdle - the maximum number of idle connections in the pool at one time. The pool will shrink in size if there are more than this number of idle connections.
• maxWait - the maximum milliseconds to wait when blocked on a connection becoming free
• whenExhaustedAction - what to do when the pool has run out of available connections. 0 - fail, 1 - block, 2 - grow
• testOnBorrow - should the connection pool test each connection for liveliness before giving to clients. I set this to true.
• testOnReturn - should the connection pool test each connection for liveliness after the client has given it back. I set this to false.
• validationQuery - the SQL query to run to determine if the connection is alive - this must return at least one row. I recommend this is a simple select of count(*) on some table with a small number of rows so as to minimise network transfers between the database and the business tier.
• The last three properties (ps.* for PreparedStatement pooling) are there because Hibernate bombs out if they aren't present. The values I've used cause no PreparedStatement pooling. I leave their explanation as an exercise for the reader.

In case you're wondering why there was no property that selected DBCP as the connection pool implementation, it's because Hibernate infers this from the presence of the associated properties (dbcp.* for DBCP). I'm not sure I like this as it introduces uncertainty as to what will happen when more than one set of properites are present (will the first or last one win, will an exception be thrown?). Still it's only a minor quible.

One final problem you might encounter if deploying to a shared Tomcat instance (for example if you deploy your app through my excellent ISP Metawerx) is DBCP raising a ClassNotFoundException for your mysql driver even though it can be found by Hibernate and at other points in your code.

The context to this problem was that my ISP adds no jar files to his Tomcat server/lib nor to his shared/lib apart from those that ship with Tomcat. Additionally of note is that Tomcat ships with DBCP. With this knowledge in hand I initially tried putting a mysql jar file in my WEB-INF/lib and expected it to work. Sadly this wasn't the case - Hibernate could happily instantiate the driver but DBCP/JDBC could not when passed the same class name. The problem was related to classloading issues (DBCP was higher up in the classloader tree than MySQL and most classloaders only ask for a class definition up the tree, not down the tree) but was easily solved by placing a copy of DBCP in my web app's WEB-INF/lib directory.

So with your configuration updated and any classloading issues behind you, you should be ready to deploy your application without fear of waking up to screaming collection of broken connections.

The JSP expression language (EL) has made writing JSPs so much more pleasurable but like standard JSP expressions you need to take care they don't expose your site to Cross Site Scripting (XSS) attacks. JSP 2.0, however, gives us a way to ensure our JSPs are both XSS-safe and compact.

Matt, over at Madbean, points out that whenever you have an EL outside a <c:out value="${someExpression}"> JSTL action tag you may be in danger of opening up yourself to a XSS attack. Naturally the solution is to ensure all output that could possibly have injected code (parameters etc), is wrapped in this tag when being output. In practice this can be cumbersome.

Where you might ordinarily have:

   <fmt:message key="someResourceKey">
       <fmt:param value="${param.someRequestParameter}"/>
   </fmt:message>

You should actually have:

   <fmt:message key="someResourceKey">
       <fmt:param>
           <c:out value="${param.someRequestParameter}" escapeXml="true"/>
       </fmt:param>
       <%-- Note, the escapeXml attribute defaults to 
               true but is here for emphasis --%>
   </fmt:message>

Not so painful, I hear you say. But what happens when you have a tag that doesn't allow you to set the value by tag body rather than by attribute?

Well, you have to do something like this:

    <c:set var="myEscapedVar" scope="page">
        <c:out value="${param.someRequestParameter}" escapeXml="true"/>
    </c:set>
    <some:tag value="${pageScope.myEscapedVar}"/>

Have to use that tag a few times in a page on a few different values? It can get quite nasty.

JSP 2 functions to the rescue

To recap, JSP 2 functions are simple Java functions that are made available to JSP EL expressions through the same mechanism as custom tags (taglibs). JSTL 1.1 (part of JSP 2.0) comes with some long overdue functions such as length() which lets you find the length of a collection, array and so on.

They're simple to write but I'll leave a detailed explanation of how to others.

What if you were to write a very simple JSP function that escaped the XML and printed the string for you instead of relying on the c:out / c:set solution above?

Your function definition would look something like this:

   public static String escapeXml(String value)
   {
       ...
       ...
       return escapedValue;
   }

And the two examples above would be rewritten like this:

   <% taglib prefix="xss" uri="http://www.whatever.com/jsp/xss">
   ...
   ...
   <fmt:message key="someResourceKey">
       <fmt:param value="${xss:escapeXml(param.someRequestParameter)}"/>
   </fmt:message>

   <% taglib prefix="xss" uri="http://www.whatever.com/jsp/xss">
   ...
   ...
    <some:tag value="${xss:escapeXml(param.someRequestParameter)}"/>

This makes for more concise XSS-safe JSPs, me thinks.

Final words: XSS-safe JSP expressions

DeveloperWorks explored this issue back in 2002 in an interesting article found here. Of course, this predates EL and so only addresses XSS with respect to JSP expressions.
e.g.

<%= request.getParameter("FOO")%>

Their solution was to implement a tag similar in effect (if not intent) to the c:out tag where XML-sensitive characters of the value attribute are escaped.

Their solution presents the same problem I've discussed today - overly verbose JSPs. With a bit of JSP jiggery-pokery a similarly brief solution can be made to work for those still wading in JSP expression goo. I imagine it would involve: a JSP base class declaring a method similar to escapeXml(); JSPs that derive from this base class; and XSS-risky JSP expressions enclosed in escapeXml() calls
e.g.

<%= escapeXml(request.getParameter("FOO"))%>

And another thing

I hope there aren't any XSS holes in my site but if there are, please let me know