unblock-us and DynDNS

My new ISP (Vodafone) provided me a new router (a Huawei HG658c) with their own firmware on it.  They also don't guarantee me a static IP address.  Unblockus says you can handle this with a DynDNS account.

 However, my router does not seem to be able to correctly update DynDNS, and even if it did I'm not entirely convinced that these magical "IP update" packets would get picked up by the good folks at Unblockus.  I've solved the DynDNS issue by installing and configuring ddclient, but Unblockus explicitly don't update your IP address when done by software DynDNS clients.

With a little bit of googling around and finding some decent partial solutions, I've devised what seems to be a pretty straight-forward shell-scripting mechanism that will keep track of your public IP address (it does assume you have only one public IP address) and if it changes will log you back in at unblock-us.com.  This script is also cron-able, should you so desire.  It should work on any Unix (Linux, OSX) with wget installed.

Also, this can entirely avoid a DynDNS account if all the more you want it for is to update Unblockus.  If you don't have a Linux machine, let me suggest getting a RaspberryPi and sticking that onto your network!

#!/bin/bash

usage() {

  echo "usage: $(basename $0) -u unblock-us_userid -p unblock-us_password"

  exit 1

}

unblock_user=""

unblock_pass=""

while getopts "u:p:" name; do

 case $name in

      u) unblock_user="$OPTARG" ;;

      p) unblock_pass="$OPTARG" ;;

 esac

done

if [ -z "$unblock_user" -o -z "$unblock_pass" ]; then

  usage

fi

lastIP_file=lastip.public

log_file=updater.log

ipechoURL="http://ipecho.net/plain"

unblockURL="https://api.unblock-us.com/login?${unblock_user}:${unblock_pass}"

cd $(dirname $0)

echo "Updating at $(date)" >> $log_file

if [ ! -f $lastIP_file ]; then

  touch $lastIP_file

fi

lastIP=$(cat $lastIP_file)

if [ -z "$lastIP" ]; then

  lastIP=0.0.0.0

fi

currentIP=$(wget -qO- $ipechoURL)

if [ ! -z "$currentIP" ]; then

  ping -q -c 1 $currentIP >/dev/null 2>&1

  if [ $? -ne 0 ]; then

    echo "ping of currentIP=[${currentIP}] obtained from $ipechoURL was unsuccessful" >> $log_file

    exit 1

  fi

else

  echo "Empty IP address from $ipechoURL" >> $log_file

  exit 1

fi

if [ "$lastIP" != "$currentIP" ]; then

  echo "IP has changed from $lastIP to $currentIP" >> $log_file

  echo $currentIP > $lastIP_file

  rm -f unblock.out

  wget -qO- $unblockURL > unblock.out

  if [ ! -f unblock.out -o "$(cat unblock.out)" != "active" ]; then

    echo "Output from $unblockURL is not as expected, may not be logged in. Content follows:" >> $log_file

    cat unblock.out >> $log_file

    exit 1

  fi 

else

  echo "IP unchanged" >> $log_file

fi

Being not-so-Genius

So I came across a post on Facebook from Be Genius which I'll reproduce here:

This is why the Japanese are better at math. Once you figure this out it'll blow your mind...

The lines and numbers above may not make any sense at first, but according to William C. at Themetapicture.com:

“The lines over the circles are color coded. Notice the single red line and 3 blue lines representing "13" group together while the single green and 2 black lines take their own group. [Simply] draw your first group of lines in one direction then your second group of lines going over the first, count the groups of intersections and there's your answer.”

Well, this has really gotten up my nose for some reason.  This tool is not "why the Japanese are better at math," and it doesn't even really explain "why the Japanese are better at standardized math tests" (I suspect the reasons for that are an emphasis on rote memorization).

Really, this is a manual tool for doing two-digit multiplication...and it isn't even a very good one at that.  If curious, this blog has a pretty good explanation for how this technique works (it is indirectly FOILing).  For me, what this technique most critically lacks is facilitating any sort of insights as to "what is going on in the maths" - the fact that it takes a blog post to describe is somewhat telling.

Try the technique with 18 x 19 (= 342):

  

You're sure as snot not going to count in the lower right...so hopefully you've memorized your single-digit multiplication tables (8 x 9 = 72), there's 8 + 9 = 17 on the diagonal, and 1 in the upper-left.  It's not as simple as just concatenating together these numbers (as the simple =156 example might lead you to think).  The upper left represents 100s, so that's really 1 x 100 = 100, the diagonal 17 is 10s, so that's really 10 x 17 = 170, and then the lower right is 1s, so that's 1 x 72 = 72.  Combine these by addition: 100 + 170 + 72 = 342.

(That's a lot of lines to draw!)

Now, try to see if you can figure out how to use this tool:

Same answer (and a lot fewer lines to draw!).  Crucially to me, you could probably figure out how this works on your own, without resorting to a blog post...and that is the power of the tool - it is closer to how the math works, not just some sort of accelerative technique that gets you an answer more quickly than the "multiply and carry" technique so many of us (of a certain age, at least) were taught in primary school.

Where this tool gets even more powerful is when Algebra gets introduced...remember the FOIL method?

                     First     Outside   Inside    Last 

(A + B) x (C + D) = (A x C) + (A x D) + (B x C) + (B x D)

Doesn't the solving the problem 18 x 19 with this square look suspiciously like (10 + 8) x (10 + 9)?   That's because that is exactly what you're doing.  So, you can teach the FOIL method by recalling something the students are already adept at, simply substituting different values for the polynomial terms, before getting into more complexities like squared numbers.

The Japanese technique  (assuming that's what it is) doesn't really help you understand that you're FOILing...you're just using a handy accelerator and might as well be using a calculator for all the math it is teaching you...

Get thee to sbt

I was struggling to work out all of what the maven prototype gave me, and then struggling with figuring out maven itself.  My gripes against maven aside, I came across this which walked through a "from the ground up" Hello, World application.  It's not the most complete "walk through", but it uses sbt (simple build tool).

New Skeleton Application

After following the instructions, my files look like:

project/build.properites is:

#Project properties
#Mon Sep 19 22:11:42 IST 2011
project.organization=philgra.org
project.name=contactdb
sbt.version=0.7.7
project.version=1.0
build.scala.versions=2.9.1
project.initialize=false

project/build/LiftProject.scala is:
import sbt._
class LiftProject(info: ProjectInfo) extends DefaultWebProject(info) {
val liftVersion = "2.4-M4"
override def libraryDependencies = Set(
"net.liftweb" %% "lift-webkit" % liftVersion % "compile->default",
"net.liftweb" %% "lift-common" % liftVersion % "compile->default",
"net.liftweb" %% "lift-mapper" % liftVersion % "compile->default",
"org.mortbay.jetty" % "jetty" % "6.1.22" % "test->default"
) ++ super.libraryDependencies
}

src/main/scala/bootstrap/liftweb/Boot.scala is:

package bootstrap.liftweb
import net.liftweb.http.LiftRules
import net.liftweb.sitemap.{SiteMap, Menu}
class Boot {
def boot {
LiftRules.addToPackages("org.philgra.contactdb")
LiftRules.setSiteMap(SiteMap(Menu("Home") / "index"))
}
}

src/main/org/philgra/contactdb/snippet/HelloWorld.scala is:

package org.philgra.contactdb.snippet
class HelloWorld {
def howdy = <span>World!! The current date is: {new java.util.Date}</span>
}

src/main/webapp/templates-hidden/default.html is:

<html xmlns="http://www.w3.org/1999/xhtml" xmlns:lift='http://liftweb.net'>
<head>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
    <meta name="description" content="" />
    <meta name="keywords" content="" />
    <title>Hello World</title>
</head>
<body>
    <lift:bind name="content" />
    <lift:Menu.builder />    
</body>
</html>

and src/main/webapp/index.html is:

<lift:surround with="default" at="content">
<h1>Hello World Snippet</h1>
Hello <lift:HelloWorld.howdy />
</lift:surround>

I won't go through the explanation given by Sanj, you can follow the link and get his explanation which seemed reasonable enough to me :)

From the sbt console, running the "update" and "jetty-run" commands gave me a nice and happily working "Hello, World" application which I can wrap my head around.  I'm sure that as I move into more complete things my LiftProject.scala will become more complicated, but as there are ample examples of folks using sbt to build their Scala projects I'll have good company.

And I won't have to contend with the farce that is XML-as-a-build-language.

Post Script: this didn't quite work.  I just realized that I ended up with the correct "index.html" but the HelloWorld.howdy piece ("World!! The current time is blah") and the Menu piece (basically the lift:surround piece) don't seem to have worked.  I will update this post when I get to the bottom of it...

Diagnosing the Problem
Looking at the HTML source coming through to the browser, we can sort of see what has gone wrong:

<lift:surround with="default" at="content"> <h1>Hello World Snippet</h1> Hello <lift:HelloWorld.howdy /> </lift:surround>

Basically, the Lift code results aren't getting injected into the HTML.  Why not?

Exploring the Skeleton Application

In Starting the Project, I discovered that the skeleton application provided by the basic maven archetype "magically" gave me a working application.  So...what's lurking in here?

When I get down to it, the first thing I'll want to do is write some tests.  First, though, let me see what I can figure out.  The page has some text "Welcome to your project!".  Where is that?

src/main/webapp/index.html



<div class="lift:surround?with=default;at=content">
  <h2>Welcome to your project!</h2>
  <p>
    <div class="lift:helloWorld.howdy">
      Welcome to contactdb at <span id="time">The current time</span>
    </div>
  </p>
</div>


So this is Lift at work.  I'm no UI or Web developer, but this is clearly HTML using div tags (rather than tables) to control layout.  Both of these specify a class.

The first specifies lift:surround which ultimately is going to reference the content of main/webapp/templates-hidden/default.html .  More on that file later, but for now it should be sufficient to accept that this is a way to give a common look-and-feel to all pages within the application (including the menu).

The second references lift:helloWorld.howdy which is talking about the class (and method) specified at ./main/scala/org/philgra/snippet/HelloWorld.scala :


package org.philgra {
package snippet {


import _root_.scala.xml.{NodeSeq, Text}
import _root_.net.liftweb.util._
import _root_.net.liftweb.common._
import _root_.java.util.Date
import org.philgra.lib._
import Helpers._


class HelloWorld {
  lazy val date: Box[Date] = DependencyFactory.inject[Date] // inject the date


  // bind the date into the element with id "time"
  def howdy = "#time *" #> date.map(_.toString)


}
}
}


I'm a bit surprised to find so many imports, particularly boiler-plate looking things.  Without (yet) getting into the nuances of Scala we can sort of see how the HTML <span id="time"> links into the method howdy and a binding.  I have to be honest, this doesn't make a huge amount of sense to me at this point: it seems that the developer of this is trying to be too clever by half.  That or Lift is goofy :)

So, that's about a wrap for now...I'll next look into the tests.  There's also User and DependencyFactory classes, but at least we have the jist of how the HTML is being built.

Determining Value in Agile

The Problem
One of the central tenets of Agile is that "value" should be delivered as soon as possible.  That sounds great on paper, but how can one value software code?


For functional user stories, this can be easy.  If you believe that some sort of enhancement will allow you to increase the profitability of your product, the value calculation is straight-forward: the "value" of the enhancement is the increased profitability.  But, it can also be hard: a new report might give you a new insight on a new (more profitable) way to operate...but you won't really know until you see the data in the new report.  In this case, the user story has the potential for increasing value but you won't really be able to put an accurate figure on it until the deed is done.


For non-functional user stories, the question of "value" starts to become impossible to determine.  An example of a non-functional story might be enhancing your logging architecture to be able to better troubleshoot problems.  (Of course, you're writing perfect software with no problems in the first place, so this is a silly example...but humour me please!) 


You would only pursue this a story if the current logging architecture makes it difficult to troubleshoot problems.  But how much does the "difficulty" cost?  And if you're writing this software for customers you don't yet have, how much does it cost?  This user story falls into what I would call "avoiding the loss of value"...it's defensive in nature.  Scalability, performance, operability...all of these (and many more) are important non-functional capabilities and attributes of good software, but how do they get built if you can't gauge their value?


I suspect this is a problem that a lot of Agile ISVs (Independent Software Vendors, or people who write and sell software for use outside of their own company) must struggle with.  The most vocal stakeholders (the sales team and sales-oriented product managers) demand functional enhancements: they're the sexy bits that sell software.  But there are a whole bunch of stakeholders on the operations and support side who aren't as vocal...and these are the folks who would derive great value on stories that avoided the loss of value.


One Idea
So, an idea I've had floating about in my head is this...what if the stakeholders in your product had to put their money where their mouth is?  I think the executives of ISVs (at least those that I would want as an employer) "get" the importance of non-functional capabilities.  Customers need a good ownership experience, but this is made difficult if they're always "losing value" because of poor quality, poor performance, or difficulty in its operation.

I might be so bold as to suggest that they could probably give a pretty decent guess as to how much they would like to have invested in functional versus non-functional enhancements.

So, let's assume that I give the executive team 1,000,000 gold dubloons.  (Avast!  Thar be maths a'comin'!), and let's say in the next release they want to see invest 70% of these invested in functional enhancements and 30% in non-functional enhancements.

The game is this...and this is really just a brain dump so it might not read that well:

• Stakeholders (who represent the Agile customer interests) have authored Stories (which include both enhancements and defects).  It is possible to be a Stakeholder without having authored a story, more on that later.
• Each Story must be well-articulated in the sense that it outlines what the story should do to be considered "complete", and is well-groomed in the sense that the story can be earned (written, accepted, and tested) within a single iteration.  
• Each Story also has a "value pitch" which outlines how the story adds (or avoids the loss of) value, using qualitative and/or quantitative language.
• It is up to the product management team to ensure these stories are "good".
• All good stories are placed on the product backlog...REGARDLESS of what the product managers believe the value of these stories to be.  
• Assuming total executive dubloon allocation is D, the total number of Stories on the backlog (up for consideration) is N, and the total number of Stakeholders is S.  
• Each story author is automatically entitled to allocate 20% x D / N dubloons.  
• Story owners are therefore automatically Stakeholders.
• This leaves 80% x D dubloons to spread across all Stakeholders.
• One does not need to be a story owner to be a Stakeholder.  Anybody with dubloons is considered a Stakeholder.
• Because the executives desired a 70/30 split between functional and non-functional value-add, the remaining 80% of dubloons are allocated along these lines.  The primary functional product manager is therefore given 56% x D dubloons, and the primary non-functional product manager is given 24% x D dubloons.
• Any individual Stakeholder can hold at most 2 x D / S dubloons.
• The product managers who have been given such a generous pile of dubloons will therefore need to seek out others who they feel can represent their respective interests.  These could be other product managers, architects, sales team members, support team members...whoever they think would be able to make a value determination.
• As more Stakeholders are added, the quantity 2 x D / S decreases, meaning that some stakeholders may need to give up some dubloons.
• Next, Stakeholders allocate dubloons to the stories on the backlog.
• Stakeholders should be independent; there should be no coercion or lobbying for one story over another.  The "value pitch" is the story authors sole means of trying to attract dubloons.
• A Stakeholder can allocate their dubloons however they want.  They could put all their dubloons onto a single story, indicating that they believe ALL of the value of the backlog is embodied in this single story.  
• A Stakeholder can "spend" the dubloons in their hand, but no more.  They must spend all their dubloons or give them to another person (who might suddenly find themselves as a stakeholder).
• Upon completion of a Story, the development and test teams get to keep all of the dubloons allocated to the story.  
• They will therefore be highly motivated to complete the stories that have attracted the most dubloons, and will not be highly motivated to complete the stories with very few dubloons.
• There should be some sort of real-life compensation for holding a dubloon.  Maybe this is money, some sort of time-in-lieu, gift certificate, stock...something.  This is to give the development and test teams a real incentive to complete stories.  There ends up being a real-world budget for this effort...maybe it's $0.01 per dubloon ($10k for the release)...it doesn't need to be much.  Perhaps it is some number of dollars per developer+tester.
• The ratio of dubloons between developers and testers should vary by completed story.  Basically, developers should be incented to get it right the first time...if they do they get a disproportionate fraction of the dubloons (maybe 120% of what they would if it were divided evenly).  BUT, if testers reject a story because of a legitimate problem, the balance shifts...maybe the first retest developers get 100% of an even division, and if there is a second retest it falls to 70%.  Basically, this is to incent the testers to find problems.  
• The stakes here aren't huge, and the development manager would be the person to police abuse of the system.

So...what does this complex game achieve?  Quite a few things actually...

1. The executives "created" dubloons because they wanted to unlock additional value in the product.
2. They had an idea about the sort of value they wanted to see added...70% functional and 30% non-functional.  They were able to achieve this by giving 80% of the dubloons to the two individuals who represented functional and non-functional interests.
3. Story authors (who are the proxy for customers) get a say too...they have 20% of the dubloons to allocate as they see fit.
4. Because of the "wealth cap" a wider set of individual value judgements are made.  Basically, this "crowd-sources" value determination of stories.
5. Priority rankings now become completely trivial: development will now do whatever they can to earn as many dubloons as possible.  Because dubloons are a proxy for value...value is delivered as quickly as possible.

So...am I nuts?  What other ways are there to solve this problem?

Starting the Project

Setting up Version Control

First things first, we need a home:

$ mkdir agilephil
$ cd agilephil
$ git init
Initialized empty Git repository in /Users/phil/agilephil/.git/

Okay, one repository, created!  No immediate plans to put onto github, though that's really more to do with the fact that I have nobody asking for it.  Next, a branching model.  The one proposed by Vincent Driessen at nvie.com is nicely explained, so I'll be aiming to follow this one.

$ cat <<EOF > README
This is a CRM-type of application I've created to learn Scala and Lift.  It is super-basic and won't win any awards.  Probably.


Follow along with me at agilephil.blogger.com.
EOF
$ git add README
$ git commit -m 'initial project version'


$ git checkout -b develop master
Switched to a new branch 'develop'
$ git checkout -b features/MultiUser1 develop
Switched to a new branch 'features/MultiUser1'


Creating the Project

So now, we're ready to rock and roll.  We have a features branch called MultiUser1 into which we'll complete our first user story.  The only trick: I need a project structure that Lift will like.  To do this, there are a number of maven archetypes available as described on liftweb's wiki page Using Maven.  I think I'll start out with "basic":


$ mvn archetype:generate \
 -DarchetypeGroupId=net.liftweb \
 -DarchetypeArtifactId=lift-archetype-basic_2.9.1 \
 -DarchetypeVersion=2.4-M4 \
 -DarchetypeRepository=http://scala-tools.org/repo-releases \
 -DremoteRepositories=http://scala-tools.org/repo-releases \
 -DgroupId=org.philgra \
 -DartifactId=contactdb \
 -Dversion=0.1

$ git add contactdb
$ git commit -m 'creating initial project structure from basic archetype'


The Using Maven page also describes a pom.xml file (the maven build metadata), but I'm not entirely sure why, as the archetype included one of these.  Needless to say, lesson learned: ignore that incomplete-looking pom.xml and stick with the one that came with the archetype.


Starting the Project

A quick test that it works, as allegedly we should have a working application:

$ cd contactdb
$ mvn jetty:run 

And browse to http://localhost:8080 ... presto!  Well, not so "presto"... maven resolved a TON of dependencies and spent the better part of 5 minutes downloading bits (my 3 Mbps "broadband" connection is stellar).

Curiously, I note a number of links on the menu, including "Login" and "Sign up".  Perhaps my story has already been written for me!  

Stopping jetty, and then checking out a few new additions to my project folder:

$ git status
# On branch features/MultiUser1
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       lift_proto.db.h2.db
#       lift_proto.db.trace.db
#       target/

So, "target" is the compiled code and it seems strange to track anything there.  Furthermore, I don't think I care to keep these .db files in my repository (they are presumably the underlying database...I'll figure out how to get that onto a MySQL server later).  

A .gitignore file will come in handy:

$ cat <<EOF > .gitignore

/target/

/*.db

EOF

$ git add .gitignore

$ git commit -m 'ignore target and database files'

Well, that's me done then for the night.  The next task will be to work out how, exactly, test-driven development works in Lift/Scala.  No friends, we do not write the application first...we write our tests first, and then we fix our broken tests.

Getting Started with Scala / Lift

My first User Story involves allowing multiple users to log into the application at the same time.


The Environment

But, to accomplish this...I first need a working development environment!  I'm not going to use an IDE: this will force me to do things the "hard" way which means I'll probably learn more.  Plus...there was no IDE in the early days of JDK 1.0.2, so we'll keep it old school.

Well, almost old school.  First things first, I'll need a version control system.  I've recently picked up on git, having been a user of various tools like Subversion, VCSS, RCS, and PVCS.  I like git a lot: branching for individual features is virtually "free" and it's easy to isolate commits to a subset of team members.

Next, a build tool...well...where even to begin.  I've used make, ant, maven, and gradle.  Were I honest with you, I'd say that I've been woefully unimpressed with the state of the art of build tools.  Maybe this is just because the underlying software in Java has become so fragmented as to make dependency mapping an impossible task, I don't know.  What I do know is that the Lift/Scala community seem to be using maven...so since I have no strong opinion either way, maven it will be.

So:


$ uname -a
Darwin mieslep-personal-mbp15.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 i386

$ java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)

$ mvn --version
Apache Maven 3.0.2 (r1056850; 2011-01-09 00:58:10+0000)
Java version: 1.6.0_24, vendor: Apple Inc.
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: "mac os x", version: "10.6.8", arch: "x86_64", family: "mac"

$ git --version
git version 1.7.5.4

$ scala -version
Scala code runner version 2.9.1.final -- Copyright 2002-2011, LAMP/EPFL


There you go: I'm currently on OS X 10.6 "Snow Leopard" and have what I assume to be a recent-ish set of utilities.


"Hello, World" ala Scala

Using the example at scala-lang.org:


$ cat <<EOF > HelloWorld.scala
> object HelloWorld {
>     def main(args: Array[String]) {
>       println("Hello, world!")
>     }
>   }
> EOF
$ scalac HelloWorld.scala 
$ scala HelloWorld
Hello, world!

Things are working Scala-wise anyhow.


"Hello, World" ala Lift

Using the example at liftweb.net:

mvn archetype:generate -U \
  -DarchetypeGroupId=net.liftweb \
  -DarchetypeArtifactId=lift-archetype-blank \
  -DarchetypeVersion=2.0 \
  -DarchetypeRepository=http://scala-tools.org/repo-releases \
  -DgroupId=demo.helloworld \
  -DartifactId=helloworld \
  -Dversion=1.0-SNAPSHOT

(taking the defaults when prompted)

cd helloworld
mvn jetty:run

And then, http://localhost:8080 returns the nice and basic example as given.  It's just that easy!  Wow, I'll be done in no time!  Though...one might suspect that there's a lot going on under the covers that I'll soon be pulling back :)