Probably my favorite example was an expansion on a previous blog post about using JRuby and HtmlUnit.  In the presentation, I created a simple Domain Specific Language on top of the great HtmlUnit library to allow for simple parsing of pages with content in JavaScript.  This was a great example of how JRuby can be used to take advantage of both Java and Ruby

The presentation is available at http://www.innovationontherun.com/presentation-files/Introduction%20to%20JRuby-10-23-2007.pdf and the associated source code can be found at http://www.innovationontherun.com/presentation-files/JRuby-JUG.zip.  If you are looking at the source code, start with the file ABOUT.txt.  It describes the prerequisites needed to get the code to run and the order of the listed files follows the order that they were used in the presentation.

If you have questions or comments, feel free to send me an email.

Bookmark to:

Hide Sites

The idea that a programming language will make your team more productive is the Holy Grail of software development.  Many languages have been promoted as a cure all for productivity (C++ in the 80s, Java in the 90s, Ruby now, Lisp perpetually), and each turns out to have their benefits and weaknesses.  But there is no doubt that certain features in languages can lead to leaps in improvements in productivity.  For example, having built in garbage collection is a sine qua non for modern development languages.  No one can argue (at least no one can argue well) that garbage collection does not improve developer productivity for 99.9% of development efforts.

What is Mixin?

Since many readers may not be aware of what a mixin is, I will try to describe it briefly.  Ruby is an object-oriented language.  With all object-oriented languages, the designers had to decide whether or not to support multiple inheritance.  Perl and C++ designers allow for multiple inheritance, Java does not allow for multiple inheritance but does allow for polymorphism through the use of interfaces.  Ruby’s designers opted to not allow multiple inheritance, but to allow for code from one module to be able to be included in another class.

So consider the following code:

module BarModule
  def hello_world
    puts "Hello World"
  end
end

class BaseClass
  def class_method
    puts "In class method"
  end
end

class Foo < BaseClass
  include BarModule
end

f = Foo.new
f.class_method
f.hello_world

 In this example, we are creating a module named BarModule with a hello_world method, a class named BaseClass, and then another class Foo that extends BaseClass and includes BarModule. The class Foo will then have both the methods from BarModule and BaseClass, but it will only BE an instance of BaseClass as that is its only parent. Somewhat different than what a Java developer would do, but it makes sense. Running this file will result in the following output:

$ ruby foo.rb
In class method
Hello World
$

So that is a basic example of what a mixin is.

Adding send() into the equation

Another interesting and powerful thing about Ruby is that all method calls are actually message passing calls.  So for example, we could rewrite:

f = Foo.new
f.class_method
f.hello_world

to

f = Foo.send(:new)
f.send(:class_method)
f.send(:hello_world)

and the results would be exactly the same! Again a little weird if new to Ruby, but it this language feature can lead to some very interesting and powerful results.

Combining include and send

Now let’s combine these two methods.  What if I wanted to take my BarModule and apply it to a class that is not sign, say the ruby base class String.

String.send(:include, BarModule)
s = "Arbitrary String"
s.hello_world

Running the above code would produce:

$ ruby include-bar-module-on-string.rb
Hello World
$

That’s right, at runtime, I was able to add an arbitrary method onto the base String class. That method is now available to any Strings that I instantiate within my application.

The Implication of this Feature on Rails

Because of this mixin feature, a developer can add arbitrary methods and modify behavior of core classes at runtime.  This is amazingly powerful if you are trying to write plugins and extensions to the framework.  Because you can add functionality to existing objects, users can install your plugin and start taking advantage of new functionality without having to make changes to the objects that they are instantiating in their application.  In frameworks written in other languages, such as Java, plugging in new functionality means that you need to change how your objects are instantiated.  This will require code changes and/or potentially configuration changes (if you are using a dependency injection framework like Spring).  Hard to develop, hard to maintain, and a pain for plugin developers to support.  But because of the mixin feature, Rails plugin developers can customize the base objects and the users of the plugins do not have to change any of their code or configuration logic.  A good example of this can be seen in the Row Version Rails plugin.  This particular plugin puts a created_at, UPDATED_AT and a row_version on every row inserted into the database.  It requires ZERO code change to make this happen.  You just install the plugin and go.  It works by adding hooks into the ActiveRecord::Base (the base persistence class in Rails) so that when records are saved, the correct information is put in to those fields.  A very easy and powerful plugin to install and use.

Conclusions

The point to take away from this is not that Ruby on Rails rocks and Java sucks.  Far from it.  But choosing a framework with lots of extensions that can take care of many of the mundane tasks allows your developers to spend more of their time focused on the problems of the user and not on common problems.  The mixin feature of Ruby allows for the development of easy to use but powerful plugins that will be hard for any non-Ruby based framework to compete with.

Bookmark to:

Hide Sites

For the unaware, Embedded JBoss allows a developer to embed much of the functionality of the JBoss application server (EJB, Messaging, JNDI, Security, Transactions) in another application.  A great use for the technology is in the creation of unit tests that can actually test Session, Entity, and Message Driven Beans.  At Health Market Science, we heavily used the Embedded EJB container (a predecessor to Embedded JBoss) connecting to a in-memory Hypersonic DB for improving out unit tests.  It made developing and debugging significantly easier and all of the developers loved doing it.  I highly encourage any developer that is developing using EJB3, and especially anyone using JBoss, to use the combination of Embedded JBoss.

Full Presentation can be found at http://www.innovationontherun.com/presentation-files/Intro%20to%20Embedded%20JBoss.pdf

Bookmark to:

Hide Sites

With the 1.12 release of HtmlUnit, this headless web browser can now support parsing and executing JavaScript.  This allows a scraper to access this dynamic content as simply as the scraper accesses static content, and without having to fire up a heavy execution engine like Gecko.

JRuby is a great technology for easily construct a script that calls into the HtmlUnit functionality without having to deal with all the syntactic sugar that Java requires.

Step 0: About The Example Code

The tar ball, scraper.tgz, contains:

• scraper.rb – the JRuby script we will be executing.  All code discussed in this example comes from there
• lib/*.jar – all of the JAR files needed to run the example
• run.sh – a simple shell script that points JRuby at the lib directory and silences some warning messages

Step 1: Enabling JRuby to Use the Java JAR files

# Require Java so we can use the Java libraries
require 'java';

# Get HTML Unit and all of its required libraries
require 'htmlunit-1.13.jar';
require 'commons-httpclient-3.1.jar';
require 'commons-io-1.3.1.jar';
require 'commons-logging-1.1.jar';
require 'commons-lang-2.3.jar'
require 'commons-codec-1.3.jar'
require 'xercesImpl-2.6.2.jar'
require 'xmlParserAPIs-2.6.2.jar'
require 'jaxen-1.1.1.jar'
require 'commons-collections-3.2.jar'
require 'js-1.6R7'
require 'nekohtml-0.9.5.jar'

# Include the Web Client class
include_class 'com.gargoylesoftware.htmlunit.WebClient';

In this block, we first use the are telling JRuby to use the JAR files required by HtmlUnit.  Some notes:

• You have to specify every JAR file that HtmlUnit depends upon, even if you are not calling the method directly
• All JAR files must be in the LOAD_PATH for JRuby.  This is done by -I<DIR_NAME> arguments passed in to JRuby from the command line.
• The include_class is similar to an import statement in Java and puts the WebClient object in scope.

At this point, we can now instantiate and use the WebClient class

Step 2: Parsing a Basic HTML Page

Before we get into parsing a dynamic page, let’s take a look at how to parse a simple page.  In this example, I am going to parse out information from the Maven 2 Archive for HtmlUnit found at http://repo1.maven.org/maven2/htmlunit/htmlunit.

# Function for getting a list of all directories
def get_htmlunit_maven_pages
  wc = WebClient.new;

  page = wc.getPage("http://repo1.maven.org/maven2/htmlunit/htmlunit");

  # List the directories...
  page.getByXPath('//img[@alt="[DIR]"]').each do |img|
    a = img.getNextSibling.getNextSibling
    puts 'DIR: ' + a.getHrefAttribute
  end

  # List the files...
  page.getByXPath('//img[@alt="[TXT]"]').each do |img|
    a = img.getNextSibling.getNextSibling
    puts 'FILE: ' + a.getHrefAttribute
  end
end

The first step in the method is instantiating a new instance by calling WebClient.new and then download the page using wc.getPage.

When requesting a page with a content type of text/html, the getPage call will return an instance of HtmlPage, and we can now use XPath expressions and DOM calls to get the URLs for the directories and for files.  Very simple to get at the appropriate data.  HtmlUnit has a bunch of other methods that you can use to navigate the page, check out the source documentation for the HtmlPage object.

Step 3: Parsing Data Written By JavaScript Functions

The code behind parsing a HTML page that uses JavaScript to dynamically create content is actually no harder than the previous example.  HtmlUnit will detect the script tags in the page you are downloading and execute the appropriate script in line.  For an example, I will use my blog home page and its inclusion of a JavaScript widget from MyBlogLog.  This script makes a call to MyBlogLog and finds out who the most recent registered users to visit my site have been.  In our example, we will parse out these users name and URLs.

# Function for seeing who the most recent my blog log users were

def search_iotr
  wc = WebClient.new;

  page = wc.getPage("http://www.innovationontherun.com");
  my_blog_log_info = page.getHtmlElementById("MBL_COMM")
  my_blog_log_info.getByXPath('//td[@class="mbl_mem"]').each do |td|
    td.getByXPath('//a').each do |a|
      puts a.asText + ":" + a.getHrefAttribute
    end
  end
end

If you look at the source for this page, you will see a script tag that downloads a JavaScript file from MyBlogLog.com.  The downloaded JavaScript will make calls to document.write that will insert an HTML table into the page.  The id of the table is MBL_COMM, so our first step is to find that HTML element.  Once we have the element, it is a couple of simple XPath expressions to find the anchor tag that contains the recent visitors name and URL.  All of the implementation of downloading the data and putting into the HTML page is hidden from us by HtmlUnit so we can easily use DOM to get at the information we are interested in.

Other Situations Where HtmlUnit Rocks!

Anytime JavaScript is being used to either enable navigation or modify the HTML document, HtmlUnit can be a great asset in your parsing.  This includes:

• Content from AJAX requests
• Situations where JavaScript events are being used to impact behavior.  An example would be a page using an onChange handler on a select list to modify form values and/or submit the form.  HtmlUnit is very handy for simplifying this interaction.

A word of caution, the JavaScript implementation is not fully featured in HtmlUnit, so some sites still may not work.  However, the HtmlUnit team is validating the browser against a fair number of popular libraries, so hopefully in future HtmlUnit releases, this will be less of an issue.

Appendix

Prerequisite Information To Run the Example

Make sure that you have Java installed.  I am using Java 1.6, but HtmlUnit and JRuby should support older versions.

Download JRuby from http://dist.codehaus.org/jruby/ and put the jruby executable (found in the bin directory of the downloaded file) in your path.

To verify that Java and jruby are set up correctly, just run jruby from the command line and ask for the version:

  > jruby -version
ruby 1.8.5 (2007-08-23 rev 4201) [x86-jruby1.0.1]

My Environment Details

• JRuby version 1.0.1
• HtmlUnit 1.13
• Java version 1.6.0_02-b05
• Ubuntu 7.0.4

Reference

• JRuby Home Page
• HtmlUnit Home Page
• HtmlUnit Java Doc
• http://www.innovationontherun.com/jruby-scraper/scraper.tgz - The source code for this example

Bookmark to:

Hide Sites

Flash has been around for a long time and is installed and active on almost every browser, it seems most of the major video sites (e.g. YouTube) are currently using Flash as their RIA platform and the Flash 9 player has some new video/audio codec supports as well as support for ActionScript 3 that meets the ECMAScript specification (basically JavaScript).  Flash/Flex are mature, have solid tools, great documentation, a vibrant community and a huge installation base.  Flash/Flex has first mover advantage in the space.

Silverlight has just released their 1.0 and has gotten a ton of press.  From the development side, the combination of JavaScript and XML to create applications feels a lot like building applications with Flex.  Silverlight has also been designed to hook into .NET languages including IronRuby and IronPython and is really focusing on high quality audio and video experience.  For some cool examples of the technology check out a visual search engine, tafiti.com or the live streaming television broadcasting application LiveStation.  There is cross-browser and cross-platform support for the toolkit.  Not a huge community or install base just yet, but the technology is easy to install (about 4 MB download), easy to develop with, produces a fast and clean user experience, and the easy hook in to .NET languages plus its inevitable preloading into base Windows installs means that these communities will build up fast.

JavaFX is basically a nicer way to write applets.  There is no focus on offline applications, no focus on media, and a small effort on reducing download sizes.  Really nothing very interesting or new from what Java developers have had before.

It seems no one at Sun has ever read The 22 Immutable Laws of Marketing[1].  Let’s look at how many of the marketing laws Sun is violating.

• Law of Leadership – Being the first to market.  Sun actually had an opportunity 10 years ago when applets were starting to be built.  But while they may have been first to market for embedded applications in a browser, they let Adobe Flash be the first into the easy to install browser plugin for media applications.
• Law of Mind and Law of Perception – Java GUI applications are thought of as being big, slow, and ugly.  By including the term Java in the JavaFX name, people will clearly associate JavaFX applications with those terms as well.  Sun should rebrand.
• Law of the Ladder/Law of Duality - It is very hard to change market position.  And if you are not one of the top two players in a market, you need to find a new market.  JavaFX will not be in the top two, they need a new market.
• Law of Line Extension – Another problem with Java, the brand is so generic and means so many things that you don’t know what it is.  Java ME, Java SE, Java EE, JavaFX.  They all leverage the Java brand, but are all wildly different products.  (Sidenote, Sun’s changing their NASDAQ stock ticker symbol to JAVA continues these violations)
• Law of Resources – Adobe is going around with their Adobe AIR tour.  Microsoft is partnering with a bunch of big media players (e.g. check out http://silverlight.net/Showcase/ for samples for WWE, CBS, MLB) to work with Silverlight.  Sun spent some money on a press release.

For JavaFX to be a success, for their to be a community, I would recommend the following:

• Figure out what market segment you are solving problems for.  Do you want to be in the consumer RIA space or should the focus be on RIA applications developed within corporate intranets?  Where Adobe and Microsoft are focusing on the consumer space (explaining the drivers of media and online/offline play), perhaps Sun could focus on a different problem space with different needs.
• Stop with every name starting with the Java brand!  Get some differentiations with product names so people know what they are talking about.
• Come up with a real roadmap for the technology and let the development community know it is a core part of Sun’s development path, not a one off project.

[1] EVERYONE should own this book.  It’s $10, will take an hour to read, and will change your view of how you look at your company and business.

Bookmark to:

Hide Sites

This really irritates the hell out of me.  How can there not be an easy to find example and demonstration of this functionality?  Maybe I am just blind, but I could not find anything from Ethan’s blog pages or by searching Google where I could see examples and understand how the implementation works.  Will there be tight coupling with JavaFX?  How significantly will it reduce start up times.  Is this really such a minor feature it should be released in a JDK update rather than JDK 7?  This announcement seems to create more questions than it answers.

Some other discussion pages:

• http://forums.java.net/jive/thread.jspa?forumID=25&threadID=22378
• http://www.ddj.com/blog/javablog/archives/2007/05/the_java_kernel.html

Bookmark to:

Hide Sites

The hardest part of the development has been trying to debug why the hell all of the scripts have been failing.  Most of the problems seem to fall into one or more of a few categories:

• Hacks in the YUI libraries to deal with the fact that IE and Firefox both operate differently and NEITHER is fully compliant with the DOM specification.  That has made for lots of fun times.  I have found a certain irony that I am cursing the Yahoo developers for the hacks that they wrote while probably cursing
• Bugs with the Rhino JavaScript libraries.  For example, I could not even start the YUI tests until I upgraded to 1.6R6.  Also, I think I found a subtle bug that is causing one of the unit tests to fail related to the for…in syntax, but it has been a bitch to reproduce.
• No integrated debugging of the JavaScript code.  The only way I can diagnose what is really breaking is by liberal use of the alert() method and then a review of the output.  Hopefully this will change in Rhino 1.7.  Huge problem.
• Unhelpful error messages.  For example, I was getting a NullPointerException being thrown from SOMEWHERE in a script, but the stack trace was lost when it was transformed to a ScriptException.  The only way I was able to debug the problem was to put a breakpoint on the constructor for NullPointerException and then look at the runtime stack when it was thrown, then grep through the Yahoo JavaScript to try to figure out where the problem could possibly be.

So if you ever having to work with the implementation

• Make sure you are using a good IDE with a debugger.  I cannot stress this enough.  I have had to put breakpoints all over the damn place.  And integrate as much of the source code around as you can and integrate it with the debugger.
• If you are using Java 6, I would STRONGLY RECOMMEND NOT USING THE DEFAULT SCRIPTING IMPLEMENTATIONS!!!  I have now worked with both the JRuby and JavaScript libraries.  It is way easier to upgrade versions of your libraries to work around bugs if you are using the standalone implementations rather than the versions built into the JVMs.  By the way, how much fun will it be when a later version of Java comes out with new scripting libraries that I am sure will not be 100% backwards compatible.
• Take Mozilla, MSDN, and especially W3C reference documents with a grain of salt.  They may not fully describe the behavior that you are seeing in the browser.

Bookmark to:

Hide Sites

To support JavaScript, I wanted to just use the ScriptEngineManager found in Java 6 to execute JavaScript.  Well, one problem that I found is that it is a real pain to get at objects that have been created in an executed script.  Consider the following JavaScript example:

sample.js

var foo = ["a","b"];

To execute that in Java 6, you would do the following:

ScriptEngineManager sm = new ScriptEngineManager();
ScriptEngine e = sm.getEngineByExtension(“js”);

String script = readFromFile(“sample.js”);

e.eval(script);

OK, so far so good.  Now let’s say you want to retrieve the value of foo from the ScriptEngine environment and iterate over the values.  The ScriptEngine class has a get method, however, this returns an Object of type sun.org.mozilla.javascript.internal.NativeArray, not a Java Array.  So to access values, you will need to cast the Object to a NativeArray.  Then, as you iterate the array, you might have to cast each value to a NativeObject as the get method on NativeArray only returns an object.

NativeArray a = (sun.org.mozilla.javascript.internal.NativeArray)e.get(“foo”);

for (int i = 0; i < a.getLength(); i++) {

  System.out.println(a.get(i, a));

}

Now your code is relying on a Sun internal class and not a base JDK class.  The appropriate classes exist in the $JAVA_HOME/lib/rt.jar file.  Unfortunately, this jar file is NOT used by default when running javac.  So to get my code to compile, I had to use the rt.jar in my bootclasspath.  Since I am using Maven 2.0 as my build system, I had to add the following to the pom.xml where I am defining my compiler

<plugin>
  <artifactId>maven-compiler-plugin</artifactId>
  <configuration>
        <source>1.6</source>
        <target>1.6</target>
        <compilerArguments>
          <verbose />
          <bootclasspath>${java.home}/lib/rt.jar</bootclasspath>
        </compilerArguments>
  </configuration>
</plugin>

This put the rt.jar into the bootclasspath when building and allowed my code to compile.  What I had hoped would be a simple task turned out to be a real ordeal.

Using the sun.org.mozilla.javascript.internal classes is very, very ugly and has the added distraction that it is hard to build and will be hard to maintain.  I do not see a way around the problem and I have found other people having similar problems working with the scripting API.  If anyone has a better approach, I am all ears.

Bookmark to:

Hide Sites