Active Record’s default_scope should be renamed to something like always_prepend_scope to better describe what the method does.

The Inspiration

The other day, "default_scope" is an anti-pattern came across my twitter feed.  Having been burned by this issue I get the sentiment.

The Problem with default_scope

Let’s say I have a simple class like the following (stolen from the default_scope documentation)

Now let’s look at some sample ruby calls and the SQL they generate

When I run Article.all the SQL executed is SELECT "articles".* FROM "articles" WHERE "articles"."published" = ‘t’.  Great, the default scope has been used and I only see published articles.

But let’s say now I want to find all articles with the title ‘a’ REGARDLESS of their published status.  My initial thought was that Article.where(title: "a") would work because my where would override the default where.  But as we can see, the SQL is now SELECT "articles".* FROM "articles" WHERE "articles"."published" = ‘t’ AND "articles"."title" = ‘a’.  The where statement that I added does not override the where of the default scope, it appends it.  OK, I find that to be unexpected, but to be fair, the documentation says:

    ‘Use this macro in your model to set a default scope for all operations on the model’

So punish me with 50 lashes with a wet noodle for not reading documentation.  But noodle flagellation (aside: I hope this blog post eventually ranks high in searching for that phrase) doesn’t write code.  So how do I get around the default_scope.  This is the real problem is that it is REALLY HARD AND UGLY to now form an arel statement that removes the default_scope.  The best way I have found to do it is to call the protected, thinly documented, method with_exclusive_scope from an instance_eval block on an ActiveRecord::Base object (to get around the protected part).  BLAH!!!

An Anti-Pattern?

In my mind, this fits right into the Anti-Pattern criteria.  At first blush, the functionality seems useful but as time unfolds, it leads to ugliness and bad consequences.

But the real problem is not with the functionality, rather that default_scope is deceptively named. 

The Webster’s definition for default:

So in my mind, when I set the default scope it is the scope that should be used if I do not make an explicit choice.  But once I make an explicit choice, like when I say where(title: "a") then the default should NO LONGER be used.

Because of the inappropriate naming, the developer, when he first uses default_scope, does not realize what he is setting himself up for. Perhaps renaming the method to something like always_prepend_scope would make developers realize what is happening and make their code easier to manage.

UPDATED: Discussion

Chap Ambrose brought up using unscoped.  I’ve tried that in the past, you need to remember to use it in a block Article.unscoped{Article.where(title: ‘a’)} rather than just straight up Article.unscoped, which I would always screw up.  Secondly, I did not see unscoped out on http://api.rubyonrails.org so I wasn’t sure if that was still appropriate.  Finally, I still think it holds up as an anti-pattern as you initially think the default_scope will make your life easier, but it instead leads to having to put your arel in these unscoped blocks.  Still ugly.

For our purposes, let’s say I had the following data set:

If a user was to search using the phrase "Boone Moving", we would expect all of these records to match because the two terms in the phrase "Boone" and "Moving" should match a term in either the Name or Location fields.

My application is built in Rails and I am using the Sunspot gem for doing searches. Sunspot gives us some nice functionality including a nice DSL for indexing and searching for ActiveRecord model objects.  I was using the default schema.xml from Sunspot, so my initial work looked something like:

So my first question, why did we only get 2 results instead of all four. Well, the default Solr parser uses the Lucene parser which uses Levenshtein for fuzzy searches.   However, as I mentioned, I am using Sunspot, which uses the dismax parser and dismax does not support fuzzy searching like this.  So I changed my code to manually set the Solr parameters to sort on

Let’s dig into this code a little bit.  In the custom_search method, I am using Sunspot’s search method, but instead of setting up keywords, I am manually setting up the Solr parameters.  I break up a search query into tokens and then require that each token be found in at least one field.  The "~" appended to each token tells Lucene to allow for fuzzy searching on this term.  The result is that this matches all of our records.  At this point, it is sort of ugly, I’m shortcutting some Sunspot functionality, but not too bad.

Now on to the next problem, sorting.  The client requirements for the searching was that we want to give priority to those businesses whose names start with the query terms.  So if I searched for "Boone Moving", I would want the businesses named "Boone Moving" and "Boone Moving Services" to show up higher than "Jimmy’s Movers" located in Boone.

In the default schema.xml, the name_text field uses the StandardTokenizerFactory which breaks up our names into individual tokens based on word boundary rules.  But in order to search on the whole term, we do not want to have terms broken up.  So, the first thing that I needed was to create a field that is not tokenized at all.  To do this, we want to use a KeywordTokenizerFactory and copy our business name into that field.

With this new schema.xml, we need to restart Solr and reindex all data to get the field indexed.  Now we can tweak our custom_search method to use a highly weighted prefix query to boost the score for businesses with an exact match of the name.

Success!   The key change that we have added an OR clause (_query_:"{!prefix f=name_untokenized v=$qq}")^10 and an additional parameter into our hash :qq=>query.downcase.  This new clause tells Solr to give a big boost to any results where the term matches the untokenized version of our name.  This is somewhat similar to the boost query option in the dismax parser.  Because we are using this as an OR, records that do not match this prefix query are not excluded, but those that do match get a big boost.  This results in a successful ordering of the records.

- John A. Hartford

From The Great A&P and the Struggle for Small Business Success

Very interesting to hear what people are working on.

Some of the topics (that I remember) from today’s Tech Breakfast

• Multi-Tenant in Rails
• Using Capybara-Webkit
• Using Mechanical Turk instead of Selenium/SauceLabs/BrowserMob for Q/A of a website
• Using Maven with the Play Web Framework
• Using Fourier Transforms and Kalman Filters in image processing

• Unicorn – Rack HTTP server
• Nassim Nicholas Taleb and his books Fooled by Randomness and The Black Swan as
  • Related discussed Benoit Mandelbrot
• A bunch of chat about Google Wave and how it would have been received if it were built by a small startup rather than Google
  • Led to discussion of Christensen’s The Innovator’s Dillema as well Moore’s Crossing the Chasm
• Comparison of work/life in NYC vs. Philly
• Professional poker playing

I’m sure there were more that I am forgetting now (shouldn’t have waited 48 hours to do it).  Very worth getting up early

The Default Maven Install Run Configuration Fails

This issue can be seen when running a project that has multiple modules in it. In my case we have an ejb module and then an ear module that is dependent on the ejb module.  When we try to run the default maven install task for the project (see image to the right), we get an error:

The following mojo encountered an error while executing:

Group-Id: org.apache.maven.plugins

Artifact-Id: maven-ear-plugin

Version: 2.3.1

Mojo: ear

brought in via: packaging: ear

While building project:

Group-Id: edu.psu.gv

Artifact-Id: classtwo-ear

Version: 0.0.1-SNAPSHOT

From file: /Users/rdimarco/Documents/workspace/classtwo/ear/pom.xml

Reason: Cannot copy a directory: /Users/rdimarco/Documents/workspace/classtwo/ejb/target/classes; Did you package/install active project artifact:

artifact = edu.psu.gv:classtwo-ejb:ejb:0.0.1-SNAPSHOT:compile;

project: MavenProject: edu.psu.gv:classtwo-ejb:0.0.1-SNAPSHOT @ /Users/rdimarco/Documents/workspace/classtwo/ejb/pom.xml?

The problem is that there is a bug when the "Resolve Workspace Artifacts" button is checked.  To resolve the problem, we need to create a custom run configuration, set the goals to install, and make sure the "Resolve Workspace Artifacts" is unchecked.

The build will now proceed correctly

Unable to Locate the Javac Compiler

Many of my students saw the following error: 

Reactor Summary:

[INFO] ------------------------------------------------------------------------

[INFO] Class Two ............................................. SUCCESS [1.483s]

[INFO] ClassTwo EJBs ......................................... FAILED [2.388s]

[INFO] ClassTwo EAR .......................................... NOT BUILT

[INFO] ------------------------------------------------------------------------

[ERROR]

 
Mojo: org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
 
FAILED for project: edu.psu.gv:classtwo-ejb:ejb:0.0.1-SNAPSHOT

Reason:Unable to locate the Javac Compiler in:  C:\Program Files\Java\jre6\..\lib\tools.jarPlease ensure you are using JDK 1.4 or above and not a JRE (the com.sun.tools.javac.Main class is required).In most cases you can change the location of your Javainstallation by setting the JAVA_HOME environment variable.

As the error message suggests, this issue occurs when you are trying to run Maven and your the runtime JRE used to run the Maven executable is from a JRE and not a JDK.  To fix this, you need to do the following things:

1. Install a valid JDK on to your computer.  You will need to download the JDK from Sun and then install it on your computer.
2. Tell Eclipse about the JDK.  To do this, you will
   1. Go to the Preferences. 
      • On Windows, choose the top menu "Window" and then select sub-menu "Preferences"
      • On a Mac, choose the top menu "Eclipse" and then select the sub-menu "Preferences"
   2. Click on >Java and then on > Installed JREs
   3. You will see a box listing one or more JREs.  Click on the button to the right of this box "Add"
   4. You will be given a list of types of JREs to use.  Choose Standard VM
   5. You will then get a screen asking for the JDK directory.  Click on the Directory button and navigate to the newly installed JDK directory
      • On Windows, this will be something like C:\Program Files\Java\jdk1.6_xx
      • On Mac, this will be something like /System/Library/Frameworks/JavaVM.framework/Versions/1.6.xx/Home
   6. After choosing your directory, Eclipse will think for a bit and then fill in some additional fields on the form.  You do not need to make any changes, just hit Finish.
3. Now we will need to tell Maven to use this JDK when it runs
   1. Right click on your project and choose Run As -> Run Configurations (as you normally would to build)
   2. In the Run Configuration dialog, click on the tab JRE.
   3. Click on the radio button for Alternate JRE
   4. From the drop down list, choose your JDK

You will now be able to run Maven without the error.

• Investment funding
• Finding customers
• Distribution channels
• Capital investment decisions

I’ve seen a lot of my startup clients make some of the same assumptions and mistakes that the Flying Fish people have.  It is always interesting to me how some problems domains.

#1 Development Process Risk Mitigation

Spending $3-5k for travel will not kill a project, but bad communication can…  Going to Israel, our plan was to have a "mini-project", ship something in the week I was there.  Day 1 involved project planning with a goal of shipping Day 5.  Going through this process showed me what holes we had in communicating with each other.  By communicating, I don’t just mean language issues, I mean effectively explaining what needed to be done to plan, develop, and ship the project.  If we have problems when we are in the same room, we are definitely going to have problems when we are separated distance and time zones.  In my week there, we did not ship on time.  I learned that we had a problem being overly optimistic in some of our estimating techniques.  To me, it showed that some of our slippages in the past were not as much due to the outsourcing model, rather to poor project planning techniques.  At the end of the week, it was my conclusion that either I (or my other team lead) that we really needed to change certain parts of the process to be better at planning and execution of our development interactions.

#2 More People Should Visit

The country was beautiful.  I was turned off a little by the architecture in Tel Aviv at the beginning of my trip, but by the end, I found it to be quite beautiful, especially the older parts of Tel Aviv.  Also, it was awesome to be on the beach in shorts in January, reminded me a little of California.

#3 There Will Never Be Peace

One thing that is missing in most American news analysis, the conflicts around Israel are not as simple as Israel vs. Palestine.  There are so many conflicts, between the secular and the religious, between Israeli Jews and Israeli Arabs, between people of Eastern European/Russian lineage and other groups, between recent immigrants and those that have been there longer, between Fatah and Hamas, the list goes on.  There is no where near a unified front on how the ending should look and people are militant about it on all sides.  The conflict is so much deeper than just us vs. them; I don’t see how the people (especially in Jerusalem) will ever agree on zoning laws, let alone bigger issues.

#4 An International Midge of Mystery

Christina (my wife, aka Midge) came with me on the trip and got to be a tourist while I worked.  Being out of the Philadelphia area really agreed with us.  Philly is a great town, but so insular and stifling.  I think we need to move, just not sure where yet.  If I could find a 3-6 month stint overseas, we would leap at it.

But they suck.  And they suck because they convey a level of certainty that is completely unfounded.  It’s pretty rare that we put a plan together where it turns out we actually understood:

• What needed to be done
• How long each task would take
• What other distractions (vacation, crucial issues, etc.) would distract resources from full concentration on the project.

My hatred of Gantt charts stems from the way they make the schedule seem black and white.  So what I want is a way to create a project plan that shows the levels of gray in the timing.  So a task would not take 3 days, it would perhaps have the highest probability of being complete in 3 days, but there are non-zero probabilities that it would take 4, 5, 6 days to complete.  I’d also say that the more tasks in the project and the more resources involved the higher the likelihood of missing tasks, pushing out the schedule even more.

The closest I have found to this is FogBugz‘s Evidence Based Scheduling that shows a probability of a release shipping.  I like that it is based on data from developers, but I want to find a generalized way to represent this schedule outside of entering every task into Fogbugz. 

Maybe there’s a product development idea in here somewhere…