Aug 22, 2011

Update: Lion sleep woes solved

In my previous post I was complaining how my MacBook would no longer sleep. The sleep function worked well after installing Lion but then stopped working unexpectedly.
It turns out that this is not an uncommon problem with many people complaining of the same issue. Advice on how to fix it varies wildly with solutions including (but not limited to): SMC reset, PRAM reset, turn off preference X, delete folder Y - there are even 3rd party application to force your Mac to sleep.

However the advice that worked for me was actually fairly logical:

1. Use pmset to view the power management settings:
localhost:work teabot$ pmset -g
    ...
    disksleep	10
    sleep		0 (imposed by 46)
    hibernatemode	3
    ...

The interesting thing in the output is the line that states 'sleep 0' - presumably indicating that sleep mode is disabled. Further along this line we find 'imposed by 46'. This is actually telling us the id of the process (PID) that is blocking sleep mode.

2. List the process with the matching PID:
localhost:work teabot$ ps -ef | grep 46
    0  46  1 0 9:05pm ??  0:00.16 /usr/sbin/cupsd -l
    0 107 46 0 9:05pm ??  0:00.59 ipp://nano.local.:631/printers/HP_Photosmart_C4100_series 7...
    ...

So we can see that a queued print job is inhibiting sleep mode and in this case I resolved the issue by clearing my print queue at which point my MacBook was happy to sleep once more.

It's pretty bizarre that a queued print job should disable sleep mode. The action of me closing the MacBook lid should make clear my intention of wanting the computer to sleep. If the print job is so important - perhaps a warning dialog should be issued so that I might have some idea why sleep mode will not be activated?

I assume that other types of process may inhibit sleep mode. Thankfully the pmset command includes some unexpectedly useful output to help us figure out which process is responsible. I can't imagine how long it would've taken me to find the errant print job without it.

More OSX Lion woes

I having more small yet significant problems with OSX Lion. I'm running the latest version - 10.7.1 - on both my MacPro and MacBook.

My MacPro:
I find that I frequently lose the ability to resolve DNS names. This is only resolved by restarting the machine. Some Googling suggests that this issue may be due to a failing mDNSResponser process. I use this machine as a media server, so in practice I get up in the morning and try to tune some Internet radio only to find that I need to reboot my media server before I can do so. I'm going to try killing the process so that I don't have to restart the machine each time.

My MacBook:
Sometimes my MacBook just won't sleep. Closing the lid, Cmd-Alt-Eject, and the Apple menu option have no effect. It's really frustrating. Some people report that this can happen if Internet Sharing is enabled - it's not in my case. Apple's 'sleep' implementation used to be its crowning jewel. Coupled with OSX's stability I could: go months without rebooting my laptop, have a working system within seconds of opening the lid, happily sling my MacBook into a bag just moments after shutting the lid. I can't anymore...

When I wake this machine from sleep - when I can actually get it to sleep that is, I often find that the wireless card is in an unknown state. This makes it impossible to connect to a wireless network or switch the wireless off. The machine seems to sort itself out after a few minutes but it's quite annoying - especially if I'm trying to look something up in a hurry.

Summary: With this extremely poor OSX Lion release Apple has reminded us what it was like to run Windows XP on a cheap laptop in 2002.

Aug 10, 2011

Inheriting Maven profiles

I'm in the process of moving a number of projects from Ant/Ivy to Maven. Many of our projects use Thrift and we use a Maven Thrift plugin to conveniently compile our Thrift IDL. Given that all of our Thrift projects have the same setup, the plugin is declared in a our parent POM.
The plugin is declared in a profile that is only activated when a thrift source directory is present:
  <profile>
    <id>uses-thrift</id>
    <activation>
      <file>
        <exists>${basedir}/src/main/thrift</exists>
      </file>
    </activation>
    <build>
      <plugins>
        <plugin>
          <groupId>org.apache.thrift.tools</groupId>
          <artifactId>maven-thrift-plugin</artifactId>
          ...

I actually have a number of plugins defined in such profiles so that they become active only when relevant source artefacts are present in the sub-project.

This all looks good, but in practice I found that the profiles were never activated in sub-modules that were being built from a parent project. It was as if the activation state was determined at the parent level, not on a per module basis.

It seems that profile inheritance does function as I expected in Maven 2. It does however work perfectly with Maven 3.

Jul 22, 2011

AFP failing between OSX Lion machines

I recently upgraded both my MacPro and MacBook to Lion. The upgrades went fine, but I now find that I cannot open a non-guest AFP connection between the two machines. On the client I see 'Connection Failed...' and in the server console I find:

22/07/2011 08:39:20.350 AppleFileServer: _Assert: /SourceCache/afpserver/afpserver-581/afpserver/AFPRequest.cpp, 2005
22/07/2011 08:39:20.354 AppleFileServer: _Assert: /SourceCache/afpserver/afpserver-581/afpserver/AFPRequest.cpp, 2005
22/07/2011 08:39:20.400 AppleFileServer: Calling ReadLock on lock < 0x2a833128 > while holding WriteLock
22/07/2011 08:39:20.524 ReportCrash: DebugSymbols was unable to start a spotlight query: spotlight is not responding or disabled.
22/07/2011 08:39:20.965 com.apple.launchd: (com.apple.AppleFileServer[424]) Job appears to have crashed: Illegal instruction: 4
22/07/2011 08:39:21.062 ReportCrash: Saved crash report for AppleFileServer[424] version ??? (???) to /Library/Logs/DiagnosticReports/AppleFileServer_2011-07-22-083921_localhost.crash
22/07/2011 08:39:22.389 com.apple.launchd: (com.apple.AppleFileServer) Throttling respawn: Will start in 8 seconds

I also have a crash report.

This is fairly annoying as my MacPro acts as the Time Machine backup for my MacBook. Has anyone experienced anything similar and are there any workarounds?

Update: I found a work around: I removed all folders from 'System Preferences/Sharing/File Sharing/Shared folders' and I can now connect. The unfortunate side effect of this workaround is that now the whole file system on the server is shared - subject to the authenticated user's privileges.

May 24, 2011

An ordered builder pattern

I was recently implementing the builder pattern for some immutable objects and was taking the approach described by Joshua Bloch in his excellent book 'Effective Java'. I particularly like the readability that the pattern provides when constructing complex objects and I think this arises because each property value is contextualized by the preceding method name:

  Person.Builder builder = new Person.Builder();
  Person person = builder.id(10).name("Jim").age(21)
    .pets("cat", "dog").build();

I also like working with builders in an IDE because the auto-complete process can nicely lead you through the object's construction with a few taps of Ctrl+Space. However, there are a few things that could be improved with the pattern:

Handling of mandatory state
There are two obvious ways we can enforce mandatory properties with the pattern. Firstly we could pass mandatory property values in the builder constructor. This gives us a compile-time enforcement of mandatory properties. However, for many mandatory properties it can suffer from the problem that the pattern was meant to avoid - telescoping constructors. Imagine that the id and name are mandatory:

  Person.Builder builder = new Person.Builder(1, "Jim");
  Person person = builder.age(21).pets("cat", "dog").build();

The second way we can enforce mandatory properties is by checking the builder state in the build() method. This preserves the readablility API but relegates the mandatory property checking to run-time - quite a penalty.

Redundant invocation paths
Normally a property setter method on a builder just returns the builder instance:

  public Builder id(int id) {
    this.id = id;
    return this;
  }

This means that on the returned value we can call any other builder method, including the one that we just invoked. We could - if we wanted to - chain many redundant calls that overwrite the builder state:

  Person person = builder.id(10).id(11).id(12).id(13).build();

While this isn't a likely to occur, it shows that the path we take through the building of the object isn't incredibly directed - it would be nice if we couldn't set the id more than once. We could do this at run-time but it would be far better if we were prevented from even writing such code! This lack of order also degrades our IDE auto-complete experience a little - imagine a builder with 20 properties - every time we see the auto-complete suggestions we have to mentally parse a list of 20 property setters - including those we've already set.

Adding order to the builder pattern
I had a go at addressing these (minor) short comings of the pattern and with some simple additions was able to create a builder that can lead you through the building process and enforce mandatory properties at compile-time in a nice contextualized way. The trick is to create a number of subordinate builder classes.

Consider an immutable value class:

  public class Person {
    private final int id; // MANDATORY
    private final String name; // MANDATORY
    private final int age; // OPTIONAL
    private final Set pets; // OPTIONAL
    ...

Now let's deal with the mandatory fields. It makes sense that you must first set the id property before doing anything else. Therefore the only possible method invocation from our builder class will be id(int id) - but what should this return? The next step in our builder sequence is to set the mandatory name, therefore we should return a builder class on which you can only invoke the name(String name) method, so we defined a subordinate builder class NameRequiredFieldBuilder, that only allows the name to be set and return this from the invocation of id(int id):

  public static class Builder {
    ...
    public NameRequiredFieldBuilder id(int id) {
      this.id = id;
      return requiredName;
    }
    ...

    public static class NameRequiredFieldBuilder {
      private final Builder builder;

      private NameRequiredFieldBuilder(Builder builder) {
        this.builder = builder;
      }

      public ? name(String name) {
        builder.name = name;
        return ?
      }

So we can now change the builder invocations like so: builder.id(10).name("Jim") - in fact we no option but to call these methods, in this order. Let's deal with providing the builder with the the optional fields. At this point we'd like to be able to call age(int age), pets(String... pets), or build(). and we'd like to do this using the object returned by name(String name). To achieve this we define a subordinate builder class for the optional fields and the build() method:

  public static class OptionalFieldsBuilder {
    private final Builder builder;

    private OptionalFieldsBuilder(Builder builder) {
      this.builder = builder;
    }

    public OptionalFieldsBuilder age(int age) {
      builder.age = age;
      return builder.optionalFields;
    }

    public OptionalFieldsBuilder pets(String... pets) {
      for (String pet : pets) {
        builder.pets.add(pet);
      }
      return builder.optionalFields;
    }

    public Person build() {
      return new Person(builder);
    }

  }

An instance of OptionalFieldsBuilder is then returned by NameRequiredFieldBuilder.name(String name):

  public OptionalFieldsBuilder name(String name) {
    builder.name = name;
    return optionalFields;
  }

So now we can call the builder property setters for the optional fields and then finally call the method to build an object instance. Full source code for this example.

Some observations
Breaking the builder into a number of subordinate builder classes allows us to introduce order into the build process. We can lead users of our API through the construction process, having them specify mandatory fields values first, only once, and potentially in order of importance. The modified pattern ensures that all mandatory fields are supplied at compile-time as it's not possible to invoke build() otherwise. The modified pattern does away with constructor supplied parameters and potential telescoping issues. Furthermore the subordinate builder classes limit method choices in our IDE's auto-complete mechanism to only the pertinent properties.

We could introduce additional structure to the invocations of the setters of the optional properties, however this might lead to a proliferation of subordinate builder classes. I feel that there is no need to dictate the order in which optional properties are set, and indeed how many times they are set.

From an implementation perspective, the overhead of creating subordinate builder classes for a few mandatory properties is small. A more scalable approach would be to use an annotation processor to generate the builder classes at compile time as had been done with Jan-Kees van Andel's make-builder tool.

Change of tack

It's been a while since I've developed anything for IOS and I'm not back working with Java again at Last.fm. I have been dabbling with C/C++ on some Arduino projects at the London Hackspace - but other than that I'm pretty much a full-time Java developer. However, I really enjoyed writing about my programming experiences and have missed jotting my thoughts down on a this blog. Rather than start a new blog for general programming posts I've decided to repurpose this one given that it already contains some possibly useful programming content.

Hopefully my (3) followers won't be too put out by this :-) I have updated the strap-line and description of this blog to reflect this new direction.

For the record, the most successful Swish Movement post was that relating to private methods in Objective-C - I still get comments on that post!

Sep 9, 2009

Accessing SSHFS within finder

I usually need to access remote systems using SSHFS and SFTP from my Mac. Unfortunately OSX Finder does not support this by default. To make my life a little easier I used MacFUSE and created a shell script to mount the relevant file system:

#!/bin/sh
sshfs user@myhost.com:/aPath/ ~/Desktop/myhost.com.mount-point \
-oauto_cache,reconnect,volname=myhost

However, this can be a little frustrating so I was very happy when a friend introduced me to MacFusion - a GUI based client for mounting SSHFS volumes. Try it!



Jun 22, 2009

Dependency Injection with the iPhone SDK

Like many Enterpise Java developers - my first exposure to the Inversion of Control principle (IOC) was when applying the then radical Spring Framework some years ago. I believe that there was an IOC/DI micro container project being incubated at Apache Jakarta prior to Spring but it certainly didn't enjoy the same adoption rate.

We now find that we rely heavily on the Dependency Injection provided by our containers and frameworks. They take the responsibility for the monotinous, repetative, and tedious creational and configuration concerns of our classes leaving with much cleaner code. With some small amount of configuration they create our objects, configure them with appropriate values, and wire then up - injecting references into dependent objects. As developers this makes out lives much easier - we're no longer writing lots of boiler plate code - knowing that the frameworks will take care of it for us. Indeed we find the IOC principal in the latest releases of the EJB specification and in popular frameworks such as Spring, Seam and Guice.

But what does this have todo with iPhone development? Since starting work on a large iPhone project I have mostly concentrated my efforts on improving the data access and business logic layers of our application. However, over time I began to move towards the view tier - refactoring some of our UIViewContollers. More often than not the existing view controllers where monolithic classes servicing quite complex views. The views themselves could be broken down into logical units so I worked hard to move the various responsibilities of the view controllers into seperate classes that then became composite members of a parent controller - for example:
  • MyViewController
Became:
  • MyParentViewController
  • MyListViewController
  • MyToolbarController
  • MyDetailViewController
While this neatly divided and encapsualted responsibility - it created many dependencies that needed to be satisfied. For example, the MyViewController would have to construct it's sub-controllers and pass in references to UI components such as UIButtons etc. so that they could then be managed by the sub-controller object. To do of this I would have to write a lot of boring, error-prone code - passing view components in to object via init method arguments or properties. As well as being a little tedious this also prevents the code being as clean as it could be.

Fortunately I was missing an iPhone SDK Inversion of Control trick. The Interface Builder application is commonly used to create views using a hierarchical view component model - a concept that I assume most are familiar with - you might draw a rectangular view, add a few buttons and arrange them etc. The magic begins when you specify which UIViewController class should act as controller; Once the class has be set you can begin drawing connecting lines between the actual view components and the member variable names in that class - you describe you object dependencies using a visual metaphor - very cool!


This object dependency configuration is stored in the view's XIB (nib) file and is used by the iPhone frameworks to instantiate and wire up your objects at runtime.

While this is fairly intuitive for situations where we have views mapping to single monlithic controllers, it takes a little more work when we have composite controllers. In this scenario subsets of our view components might be managed by a class that is not and does not need to be a fully fledged UIViewController. However, this sub-contoller class will be a member of a parent composite UIViewController that pulls together multiple sub-controllers - each with their own distinct responsibilities for various view components within the larger view. Clearly, we would like to use Interface Builder to direct the instantiation of these sub-controllers and satisfy the referential dependencies between them, the parent controller, and the view components.

We can do this in Interface Builder by using the 'Object' component from the component library. We create a new object for the NIB by dragging the Object component from the library. We can then specify the class of the object we wish to instantiate using the Inspector.


However, this new object - created when the NIB is loaded - will be orphaned as it's not currently associated with any other objects. Lets assume that the parent controller may need to communicate with the sub-controller. In this event we need to create an IBOutlet in the MyParentViewController class:

@interface MyParentViewController : UIViewController {
@private
IBOutlet MySubController* subController;
...

In addition to this lets assume that the responsibility of the MySubController includes something button related and hence must have a reference to a UIButton in the view. In the MySubController class we need to create another IBOutlet - this time for the button:

@interface MySubController : NSObject {
@private
IBOutlet UIButton* button;
...

Notice that MySubController does not extend UIViewController - it doesn't need to as it's responsibilities are small and may be focused purely on a subset of view components. We can now wire these objects up in Interface Builder:


Now when the NIB is loaded the sub-controller object will be instantiated and a reference injected into the parent controller - all without any code or additional effort. With this approach we can begin to break-up those monolithic view controllers into classes that adhere to the Single Responsibility Principle so that our view controllers aren't doing 'too much'. In addition to this we can keep our classes nice and clean as we don't have to litter them with extra properties and init method arguments to pass references to dependent objects. We can rely on the platform to satisfy these dependencies by injection according to the relationships we defined previously in Interface Builder.

Clearly the above example does not live in the real world. However, I have found that for complex views an controllers the approach works extremely well.

May 28, 2009

SQLite optimization on the iPhone

Our current iPhone application makes quite heavy use of SQLite. We persist the model in the database using a Data Access Object pattern built on top of my own JDBC like SQLite layer and it is all still quite lightweight. CoreData whould have been prefereable but we have to be compatible with iPhone SDK versions prior to 3.0. We also use the database to rank trends using indexed and weighted keywords and some additional temporal factors. Thankfully the text data we search through on the device is pre-indexed so we don't have to worry too much about getting suitable data into the tables. However we do need the database to run queries against this data and we spotted early on that optimization would be beneficial here.

I had some ideas of where we could improve and with a little Googling I found a sparse but useful SQLite optimization FAQ compiled by Jim Lyon. I quickly put together a hit list of optimizations that I'd try.

Use a between construct instead of LIKE
The FAQ explains that a LIKE cannot use an index and thus where possible it should be replaced with the > and < operators. For example:

    word LIKE 'dog%'

Can be replaced with the more efficient:

    word > 'dog' AND word < 'doh'

It's not a trivial replacement. Firstly the strings you are comparing must be of one case only. I also ran into trouble with international characters which of course don't fit nicely into the byte ranges. Furthermore you must write a little logic to generate the last character(s) of the 'upper limit term' (in this case 'dog'). Incrementing the last character is all well and good - but what about when you want to perform:

    LIKE 'oz%'

I simply appended a low value character to the string to obtain the 'upper limit term' and ended up with something like: 'oz!' Thankfully I could work within these limitations for our use-cases and performance was much improved.

Move members of IN clauses into temporary tables
Many of our queries used variable length IN clauses. This made it unfeasible to prepare and cache the resultant statements and they would be prepared fresh each time. This sucked up and incredible amount of time - we might spend a second just preparing the statement. A typical clause is shown here:

    AND id IN (32, 45, 67, 68, 80)

I decided that if I moved these values into a temporary table I could use a sub-select within the IN clause and hence end up with a static statement that I could prepare once and cache:

    AND id IN (SELECT id FROM temporary_ids)

In addition to this I hoped to use a PRAGMA directive described in the FAQ to move the temporary tables off of the flash disk and into memory:

    PRAGMA temp_store MEMORY

However, this setting does not seem to take effect on the iPhone version of SQLite which was somewhat disappointing. That said, the restructuring of my IN clauses did provided yet another significant performance improvement. I wouldn't be surprised if the row inserts into the temp table actually take longer than the execution of a given IN clause. But in this instance I am avoiding the costly preparation of a statement on each call.

Order sub-queries so that smaller results are returned first
The FAQ suggests in section 5.3 that sub-queries or criteria should appear in an order such that the criteria that will exclude the most rows should appear first. I take this to mean that this (poor) example query:

    SELECT o.id
    FROM owner o, pet p
    WHERE o.age > 12 AND p.name = 'nathan' AND p.id = o.pet_id

Should be rewritten as:

    SELECT o.id
    FROM owner o, pet p
    WHERE p.name = 'nathan' AND o.age > 12 AND p.id = o.pet_id

Because the p.name criteria is far more selective than that using o.age. Okay, so it's not a great example query. However, in our queries it was quite clear which criteria would be the most selective.

Other practices
Prior to implementing these optimizations we were using many of the best practices recommended in the FAQ including:
  • Batching commands into transactions
  • Using indexes where appropriate and justifying the indexes with the EXPLAIN command
  • VACUUMing the database file before making release builds - this had a noticeable effect on the database file size but I couldn't say that it improved the performance.

Never forget

Often when we are developing applications for the iPhone we want to enable extra 'tools' in the execution environment so we can get a better grasp of what our software is actually doing. Typically we'll use some kind of logging, and after reading Adhamh Findlay's post on the use of NSZombie I'll often have NSZombieEnabled turned on.

Both of these tools will have an impact on the behaviour of our application if enabled. Logging will certainly slow the application down and NSZombie can make the app run out of memory. Therefore, we need to be conscious of these potential hazards so that we don't try to start fixing problems that are actually just artifacts of the tools. More importantly we must be certain that none of these useful developer settings make their way into released code. It's not hard to imagine that one could let logging slip into a build - at a minimum this will cost you another cycle of building a release and potentially much, much much more.

We found ourselves making this mistake a couple of times with internal releases so thankfully it was only QA who complained and not customers. But having made this mistake we resolved to make it very difficult to repeat. So here's the first thing that our developers see when they start-up the application in a development mode:


Sure, dismissing the alert is a little annoying but it reminds us of the particulars of the current execution environment and saves us from making a costly mistake. We can then release our software with greater confidence. From a code perspective it's lightweight - in our viewDidLoad method of our root UIViewController we call:

    [ForgetfulDevelopersAlert showDeveloperWarningIfRequired];

In the ForgetfulDevelopersAlert implementation we check for logging like so:

    BOOL logging = NO;
    #if LOG_LEVEL != NONE
        logging = YES;
    #endif

And we detect for NSZombie usage with:

    + (BOOL) __isNSZombieEnabled {
        return getenv("NSZombieEnabled")
            || getenv("NSAutoreleaseFreedObjectCheckEnabled");
    }

With these flags available it is straight forward to construct a message and show the UIAlert:

    if (logging || zombies) {
        UIAlertView* alert = [[ForgetfulDevelopersAlert alloc]
            __initAlertWithLogging:logging andZombies:zombies];
        [alert show];
        [alert release];
    }

This can of course be extend to check for other development settings. For example I know of one particular case where an app was submitted to the AppStore and it was not noticed until it had finally been approved that only 'test' ad banners had been enabled. Potentially this lost some revenue and required another approval cycle with Apple - but it is a very easy mistake to make that perhaps could have been avoided with something like:

    [ForgetfulDevelopersAlert showWarningIfTestAdsEnabled];

:-)