Boy Maas (m4nic)

CoffeeScript screencasts

2015-10-31T00:00:00Z

Client side CoffeeScript hacking screencasts

Some other fast-forwards of me hacking a proof-of-concept videoslot machine on top of a home-grown rendering engine. Notice the old school switch window, refresh, "re-massage the application in the desired state" development style. Something clojurescript's figwheel and other technologies have fixed for me. Thanks bhauman!!!

Part I

Part II

After these proof of concepts a complete new way of approaching client side development arose. But still nostalgic to see how I worked in 2014.

Clojure screencasts

2015-08-05T00:00:00Z

A while ago, I started the habit of recording hacking sessions, and replaying them with a factor of 10. This way I can review my own actions and learn from the way I work, how I refactor, and the way I solve problems.

Monadic bitfinex trading bot

Part of me developing a trading bot, on top of datomic to allow multi-variable regression testing based on actual orderbooks configurations gathered while running the bot on bitfinex.

Part I

Part II

An alternative view on the foundations of our society

2012-07-03T00:00:00Z

A few years back I accidently stumbled on Adam Curtis The Century of Self and All watched over by Machines of Loving Grace).

I have found them to be an interesting view on the foundations of our modern society. I really liked the two mentioned above and embedded them below for your viewing pleasure. I recommend watching at least the first 20 minutes and then decide wether to continue. I have found them worthy of my time.

All watched over by Machines of Loving grace

All Watched Over by Machines of Loving Grace is a 2011 BBC documentary series by filmmaker Adam Curtis. The series argues that computers have failed to liberate humanity and instead have "distorted and simplified our view of the world around us".

All watched over by machines of loving grace from mayo11 on Vimeo.

The century of Self

This series is about how those in power have used Freud's theories to try and control the dangerous crowd in an age of mass democracy." Adam Curtis

The Century of the self [1/4] Happiness Machines from zerzoura on Vimeo.

Ruby exception handling benchmark

2012-06-13T00:00:00Z

At work a discussion arised on how fast a ruby exception is. And if it is usable in Rails projects. The discussion started after the conclusion that performance dramatically degraded in a project using the Rails framework.

The question arose how fast ruby handles exceptions. After doing a quick google I could not find an isolated benchmark so I whipped up one myself.

Below the result of the benchmark.

ruby 1.8.7 (2012-02-08 patchlevel 358) [i686-darwin11.2.0]

  Rehearsal ----------------------------------------------------
  exception          0.770000   0.030000   0.800000 (  0.809420)
  exception with e   0.780000   0.040000   0.820000 (  0.814608)
  multi exception    0.850000   0.040000   0.890000 (  0.885843)
  symbol             0.040000   0.000000   0.040000 (  0.035782)
  ------------------------------------------- total: 2.550000sec

                         user     system      total        real
  exception          0.770000   0.030000   0.800000 (  0.808482)
  exception with e   0.780000   0.040000   0.820000 (  0.817195)
  multi exception    0.850000   0.040000   0.890000 (  0.889478)
  symbol             0.040000   0.000000   0.040000 (  0.036175)

ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.2.0]

  Rehearsal ----------------------------------------------------
  exception          0.890000   0.040000   0.930000 (  0.929162)
  exception with e   0.890000   0.030000   0.920000 (  0.934193)
  multi exception    0.970000   0.040000   1.010000 (  0.999004)
  symbol             0.020000   0.000000   0.020000 (  0.017207)
  ------------------------------------------- total: 2.880000sec

                         user     system      total        real
  exception          0.830000   0.040000   0.870000 (  0.867102)
  exception with e   0.830000   0.030000   0.860000 (  0.873007)
  multi exception    0.910000   0.040000   0.950000 (  0.938906)
  symbol             0.020000   0.000000   0.020000 (  0.017294)

jruby 1.6.7.2 (ruby-1.8.7-p357) (2012-05-01 26e08ba) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_29) [darwin-x86_64-java]

  Rehearsal ----------------------------------------------------
  exception          8.044000   0.000000   8.044000 (  8.006000)
  exception with e   7.704000   0.000000   7.704000 (  7.704000)
  multi exception    7.791000   0.000000   7.791000 (  7.791000)
  symbol             0.038000   0.000000   0.038000 (  0.038000)
  ------------------------------------------ total: 23.577000sec

                         user     system      total        real
  exception          7.603000   0.000000   7.603000 (  7.603000)
  exception with e   7.605000   0.000000   7.605000 (  7.605000)
  multi exception    7.529000   0.000000   7.529000 (  7.529000)
  symbol             0.009000   0.000000   0.009000 (  0.009000)

The benchmark code

  
  require 'benchmark'

  class E1 < StandardError; end
  class E2 < StandardError; end
  class E3 < StandardError; end

  def outer_conditional
    if inner_returns_symbol == :argument_error
      :agument_error
    end
  end

  def outer_multi_exception
    inner_multi_exception
  rescue E1
    :E1
  rescue E2
    :E2
  rescue E3
    :E3
  end

  def inner_multi_exception
    raise [E1, E2, E3][rand(3)]
  end

  def outer_exception
    inner_raises_exception
  rescue ArgumentError
    :argument_error
  end

  def outer_exception_with_e
    inner_raises_exception
  rescue ArgumentError => e
    :argument_error
  end

  def inner_raises_exception
    raise ArgumentError
  end

  def inner_returns_symbol
    :argument_error
  end

  COUNT = 100_000

  Benchmark.bmbm { |x|
    x.report("exception") do
      COUNT.times { outer_exception }
    end
    x.report("exception with e") do
      COUNT.times { outer_exception_with_e }
    end
    x.report("multi exception") do
      COUNT.times { outer_multi_exception }
    end
    x.report("symbol") do
      COUNT.times { outer_conditional } 
    end
  }

Ruby org-mode parser development log

2012-02-10T00:00:00Z

Usin Vim for a long time, and having used e-macs for a while I got hooked on org-mode. The flexible plain-tekst planning mode.

Using org-mode was an eye-opener, brilliant idea, using it almost a reason to stick to emacs in combination with evil=mode. But it turns out I am more of a VIM person.

Looking on the web I found the vim-orgmode bundle. Which provides highlighting and other basic functionality. But I need more. An agenda view, A quick overview of open tasks per project, anything I would like to extract in the future.

To do this I need a nice abstracted setup of parsing org-files and re-writing/re-presenting the information in a nice and clean way.

Goal

Making lightweight/fast ruby org-mode executable to parse/reformat/analyse and report on my org-files or the one in my buffer. And following the Unix philosopy making a seperate process for it.

What I want

flexible parsers
flexible writers
potentially database storage
maybe an org-mode website to upload / store org-mode files
well tested ruby code
integration with vim
flexible reporting
remember mode
git integration

Code

Follow my progress at: github

Main cmdline pipeline

An org-mode script does the following:

org-file(s) => parse => perform action => present/write result

Design

After some experimenting I came down with the following architecture for this commandline utility.

Domain objects

The domain objects of an org-mode file is pretty simple

OrgMode
  FileCollection
  File
  Node

Where file represents an org-mode file. It contains all necessary data to generate a nicely formatted org-mode file.

Node is an item in an org-mode file, always starting with a title following some content.

* TODO I am a OrgMode node
Content of orgmode node

You can see the source here: org_mode.rb

Parsers

A parser is responsible for building up the domain objects out of a org-mode formatted file.

# parser -> returning domain objects
OrgMode::FileParser -> OrgMode::File
OrgMode::NodeParser -> OrgMode::Node

I start with a simple regexp based parser. Which ofcourse is well tested. parser spec

Reporters

Reporters take the domain model and extract certain information out of the domain model. It essentially constructs datastructures which can be easily presented by the presenters.

OrgMode::Reporters
  Agenda

Presenters

Presenters present the data contructed by the reporters in some kind of format. This can be anything. I will start with simple commandline interaction so they will be strings.

OrgMode::Presenters::String
  Agenda

OrgMode::Presenters::Html

Commands

Commands handle the command line interaction cmdline interaction essentially parses the supplied org-mode files or recursively scanned directory and displays the information use one of the available presenters

OrgMode::Commands
  Agenda

Software architecture

2012-01-29T00:00:00Z

I was working with some legacy code lately where commercial goals where given precedence over technical quality. All rules on writing solid, maintainable software where broken. The situation was terrible, and with no future sign of improvement. Architecture just wasn't part of the culture and had no real value in the perception of the technical lead and the commercial department. I was working with this code for over 2 weeks, and just could not continue working with it.

I had to give the advise to either start over from scratch and do it right, or continue with a cycle of bugfixes and "hoping" everything would keep on running without me. The quote below adequately describes the state of their software.

A Big Ball of Mud is a haphazardly structured, sprawling, sloppy, duct-tape-and-baling-wire, spaghetti-code jungle. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.

It resulted in me and another healthy motivated programmer leaving the team.

We belong to the later category: "Programmers with a shred of architectural sensibility shun these quagmires."

Software design is like a game of chess

Software architecture/design is like a game of chess. You have many options but some just give your position more options than other positions. You could argue that the adaptability of you position is an indicator of quality. Higher adaptability is higher quality. Since the probability you can adapt to the attacks of your opponent increases.

Good, solid and robust software also share these characteristics. In this article I would like to address cohesion and coupling, and show you they (should) underly many architectural choices.

Cohesion is often contrasted with coupling, a different concept. Nonetheless high cohesion often correlates with loose coupling, and vice versa. The software quality metrics of coupling and cohesion were invented by Larry Constantine based on characteristics of "good" programming practices that reduced maintenance and modification costs.

So we could state that any choice that increases cohesion and decreases coupling in a certain domain would be a good architectural choice.

I believe this is correct from a purely technical point of view. There is also the human (cognitive) point of view. Good programs tend to have their naming down. Reading this software is easy, the whole structure adapts naturally to the domain and to the vectors of change.

There are only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton

The vectors of change is an important element, some vectors of change are easy to spot and others are difficult to spot. It's a predicting game, and is heavily dependent on the perception of the architect and the domain expert at the time the design is constructed.

Too much anticipation on change may lead to Inner Platform anti-pattern. Writing a language a language.

Extract knowledge from the domain expert

The knowledge of the domain-expert is captured in the language he/she uses to express this knowledge. As a good architect, one of the more important skills is knowing how to listen, and extract the right knowledge and patterns so you can model the domain correctly in software.

If it's modelled correctly the probability the system can adapt correctly to anticipated change increases and the software communicates the knowledge of the domain. The domain expert should be able to read the domain level code and verify it's correctness.

What is the "best" approach

Following these assumptions you can conclude there is no such thing as "one superior design methodology", good software design is partly science and partly art.

Lets look at some popular architectural choices made these days.

Rails architecture

Rails is a great platform for quickly realising a prototype. You can create models, attach validation to these models, and render forms using the models meta-data and more. This is all great for initial speed, but the whole idea that form validation (view) information comes directly out of the model (persistence) ties the view to the model. Something I would rather avoid.

The design of Rails rewards this behaviour (speed). Behaviour which finally will end the creation of yet another big ball of mud, especially when applications grow >50k lines of code.

Using Rails as an architectural choice for all domains, is essentially saying MVC is a solution for all problems.

DCI could be a solution in some domains to keep everything clean and simple within a MVC structure.

Data, Context and Interaction to supplement MVC

The paradigm separates the domain model (data) from use cases (context) and roles that objects play (interaction). DCI is complementary to model–view–controller (MVC). MVC as a pattern language is still used to separate the data and its processing from presentation.

Why would we want this?

To improve the readability of object-oriented code by giving system behavior first-class status
To cleanly separate code for rapidly changing system behavior (what the system does) from code for slowly changing domain knowledge (what the system is), instead of combining both in one class interface
To help software developers reason about system-level state and behavior instead of only object state and behavior
To support an object style of thinking that is close to peoples' mental models, rather than the class style of thinking that overshadowed object thinking early in the history of object-oriented programming languages

So what is it ...

Context makes sure the data stays decoupled by providing an extra layer on top of the interaction. Introducing the role on the data and directing the interaction of all the components

Essentially it boils down to this:

1) Data(objects) are simple, they only contain functionality regarding persistence. 2) Roles describe a certain behaviour of data object. 3) The Context loads the different DataObjects, injects certain roles on them and directs the interaction between them.

So how does this tie into the cohesion and coupling paradigm.

Data are kept slim, and no behaviour is specified on them. Since behaviour is specified in Roles it is decoupled from the Context.

Context
initialize
   # instantizes all dataobjects and makes sure they can perform
   # the requested roles

call/execute
   # perform the sequential logic using the dataobjects

So by this we could say a Context is coupled to (one or more) Roles. And a Role is coupled to a DataObject.

Context --> *Roles --> *DataObject

For more information read: Data context and interaction

Ruby community heuristics (and the relation to coupling and cohesion)

Spending a lot of time in the Ruby community lately. I hear the following a lot from experienced Ruby programmers: Tell don't ask, Law of Demeter, Single Repsonsibility Principle

It's my belief that all these elements can be traced back to the root ideas of cohesion and coupling.

Tell don't ask

Reduces data coupling, especially 2 way coupling. If object (A) request data form (B) both object get tightly coupled. There is 2 way communication. Tying both A -> B, and B -> A.

Law of Demeter

Reduces data coupling. If (A) accesses a specific data-structure of (B) one cannot freely change the internals of B since A will have to be changed as well. Keeping knowledge internal.

Single Responsibility Principle

Increases cohesion, always a good thing.

Cohesion and Coupling

So cohesion and coupling should be a dominant factor on which you should make architectural choices.

For an overview of the different forms of cohesion I have grabbed their definitions from Wikipedia. Maybe an interesting read.

Cohesion

Modules with high cohesion tend to be preferable because high cohesion is associated with several desirable traits of software including robustness, reliability, re-usability, and understandability whereas low cohesion is associated with undesirable traits such as being difficult to maintain, difficult to test, difficult to reuse, and even difficult to understand.

Coincidental cohesion (worst)

Coincidental cohesion is when parts of a module are grouped arbitrarily; the only relationship between the parts is that they have been grouped together (e.g. a "Utilities" class).

Logical cohesion

Logical cohesion is when parts of a module are grouped because they logically are categorized to do the same thing, even if they are different by nature (e.g. grouping all mouse and keyboard input handling routines).

Temporal cohesion

Temporal cohesion is when parts of a module are grouped by when they are processed - the parts are processed at a particular time in program execution (e.g. a function which is called after catching an exception which closes open files, creates an error log, and notifies the user).

Procedural cohesion

Procedural cohesion is when parts of a module are grouped because they always follow a certain sequence of execution (e.g. a function which checks file permissions and then opens the file).

Communicational cohesion

Communicational cohesion is when parts of a module are grouped because they operate on the same data (e.g. a module which operates on the same record of information).

Sequential cohesion

Sequential cohesion is when parts of a module are grouped because the output from one part is the input to another part like an assembly line (e.g. a function which reads data from a file and processes the data).

Functional cohesion (best)

Functional cohesion is when parts of a module are grouped because they all contribute to a single well-defined task of the module (e.g. tokenizing a string of XML).

Coupling

Coupling can be "low" (also "loose" and "weak") or "high" (also "tight" and "strong"). Some types of coupling, in order of highest to lowest coupling, are as follows:

Content coupling (high)

Content coupling (also known as Pathological coupling) is when one module modifies or relies on the internal workings of another module (e.g., accessing local data of another module). Therefore changing the way the second module produces data (location, type, timing) will lead to changing the dependent module.

Common coupling

Common coupling (also known as Global coupling) is when two modules share the same global data (e.g., a global variable). Changing the shared resource implies changing all the modules using it.

External coupling

External coupling occurs when two modules share an externally imposed data format, communication protocol, or device interface.This is basically related to the communication to external tools and devices.

Control coupling

Control coupling is one module controlling the flow of another, by passing it information on what to do (e.g., passing a what-to-do flag).

Stamp coupling (Data-structured coupling)

Stamp coupling is when modules share a composite data structure and use only a part of it, possibly a different part (e.g., passing a whole record to a function that only needs one field of it). This may lead to changing the way a module reads a record because a field that the module doesn't need has been modified.

Data coupling

Data coupling is when modules share data through, for example, parameters. Each datum is an elementary piece, and these are the only data shared (e.g., passing an integer to a function that computes a square root).

Message coupling (low)

This is the loosest type of coupling. It can be achieved by state decentralization (as in objects) and component communication is done via parameters or message passing (see Message passing).

No coupling

Modules do not communicate at all with one another.

Object-oriented programming

Subclass Coupling

Describes the relationship between a child and its parent. The child is connected to its parent, but the parent isn't connected to the child.

Temporal coupling

When two actions are bundled together into one module just because they happen to occur at the same time.

Kinetic Typography

2011-09-07T00:00:00Z

How movement conveys emotion

Kinetic typography refers to the art and technique of expression with animated text. Similar to the study of traditional typography of designing static typographic forms, kinetic typography focuses on understanding the effect time has on the expression of text. Kinetic typography has demonstrated the ability to add significant emotive content and appeal to expressive text, allowing some of the qualities normally found in film and the spoken word to be added to static text. Kinetic type has been widely and successfully used in film as well as in television and computer-based advertising. Perceptual psychology research on attention, reading performance, and comprehension has indicated that time-based presentation of text can be used effectively to capture and manipulate a viewer's attention and in some cases improve overall reading performance.

Below a few examples of material I found interesting ...

Ira Glass on Storytelling from David Shiyang Liu on Vimeo.

Terence McKenna - Reclaim your mind from Olivier Ferland on Vimeo.

Terence McKenna - Community of mind Kinetic Typography from Olivier Ferland on Vimeo.

Bill Hicks - It's just a ride from Olivier Ferland on Vimeo.

Terence McKenna - Nobody is smarter than you are from Olivier Ferland on Vimeo.

Terence McKenna - Being Cool from Olivier Ferland on Vimeo.

Ira Glass on Storytelling from David Shiyang Liu on Vimeo.

Rails experiences

2011-02-20T00:00:00Z

Revisited rails and the ruby after a few years to implement Keygames Network, and found it extremely productive. Why?

High level view of a software project

Doing a new software project incorporates doing the following tasks:

Idea\ Think about and spec the idea, provide the rationale behind the idea and prove design is valid.
Selecting open source components\ Select and look into the different components to base the implementation on. Look into the code-base, read the forums to get an idea on the quality of those components.
Implementation\ start incremental implementation of idea based on components. Every subcomponent will have a test-suite.
Dependency management\ put all components into a dependency manager, like the Bundler and RVM combo, to make sure all component versions can be included and are documented. This will speed up further development with other developers since they will be able to get up and running quickly. And this process will ease deployment.
Deployment\ automate the deployment procedure. Spend some time on this since this will be a big time saver in the future. (Capistrano)

Rails, and the extremely productive ruby community provide excellent tools to handle all these phases of the development process.

Convention to ease the burden on human memory

It's a well known fact that most humans, ranging in intelligence can hold only 7 to 10 items simultaneous in memory. This limitation in cognition explains why we as humans structure everything in layers, often every layer consists of those 7 or 10 elements. Think of how a business is organised, how a text-book is divided into chapters, or how the law of a country is divided.

When building software, one of the challenges is handling complexity in a way you and the other people working on a project can manage this complexity. We have different techniques to do this. Modularisation, object oriented designs, functional designs, and convention. Rails uses convention extensively, it is one of its key features and part of the marketing behind it: "Convention over configuration".

The big plus of using convention is that one only has to keep the logical relations into memory. All derived elements can be generated from the naming of the defined root elements in combination with the logical relations.

Programmers, which (in my view) essentially means, organisers of information or complexity, develop a special skill for remembering methods. The "how" does one arrive to a specific goal. Facts are of less importance, structures are.

In rails a simple application reflects the naming and relations of the model throughout the system. This makes naming very consistent and predictable.

And as we all know consistency and predictability are measures of quality in software design, how this is implemented will follow.

One abstract view of Rails application

Most dynamic web-applications provide an interface to manipulate a model and provide a certain view to the data stored in the model. These views are controlled by a controller. This is the standard M(odel) V(iew) C(ontroller) paradigm. Rails is a certain implementation of this paradigm.

One could state that a Model (orders) is a resource which is manipulated by a view/controller combination. Rails provide a standard way of managing a resource using the following actions:

list
create
edit
delete
show

Most controllers will implement one or more of these five and some more complicated actions or specified actions to change state of the specified member or collection of the resource. This is the called CRUD.

The end-user will navigate the dynamic web-app using links. Which will point to screens to either View, List, or Manipulate state.

How is it implemented

Rails makes it easy to create web applications which more or less conform to the abstract view presented above. The normal method of working goes like this.

One defines a model, one continues to define controllers and routes. From within these definitions the rails system generates url functions which is a specific combination of the model names.

A very simple model example:


  class Shop < ActiveRecord::Base
    has_many :orders
  end

  class Order < ActiveRecord::Base
    belongs_to :shop
  end

A simple resource_full route definition:

resources :shops do
  resources :orders
end

Will generate:

    shop_orders GET    /shops/:shop_id/orders(.:format)          {:action=>"index", :controller=>"orders"}
                POST   /shops/:shop_id/orders(.:format)          {:action=>"create", :controller=>"orders"}
 new_shop_order GET    /shops/:shop_id/orders/new(.:format)      {:action=>"new", :controller=>"orders"}
edit_shop_order GET    /shops/:shop_id/orders/:id/edit(.:format) {:action=>"edit", :controller=>"orders"}
     shop_order GET    /shops/:shop_id/orders/:id(.:format)      {:action=>"show", :controller=>"orders"}
                PUT    /shops/:shop_id/orders/:id(.:format)      {:action=>"update", :controller=>"orders"}
                DELETE /shops/:shop_id/orders/:id(.:format)      {:action=>"destroy", :controller=>"orders"}
          shops GET    /shops(.:format)                          {:action=>"index", :controller=>"shops"}
                POST   /shops(.:format)                          {:action=>"create", :controller=>"shops"}
       new_shop GET    /shops/new(.:format)                      {:action=>"new", :controller=>"shops"}
      edit_shop GET    /shops/:id/edit(.:format)                 {:action=>"edit", :controller=>"shops"}
           shop GET    /shops/:id(.:format)                      {:action=>"show", :controller=>"shops"}
                PUT    /shops/:id(.:format)                      {:action=>"update", :controller=>"shops"}

As one can see in this simple example the model is reflected in the routes and in the generated url functions using linguistic functions. This was a new technique invented by Rails.

Actions

As you can see above every resource route generates routes to 4 standardised actions inside the targeted controller. Conforming CRUD. Which one could split up in 2 categories. Actions manipulating the collection of the resource, and actions which manipulate a member of the collection.

* collection ->      shop\_orders
  * index
  * create
  * new     -> new\_shop\_order
* member
  * edit    -> edit\_shop\_order
  * show    ->       shop\_order
  * update
  * destroy

Notice the pluralization and singularization of the different pars and how the relations as defined in the data-model is reflected in the routes.

More to come

I will specify more functionality in the future. Models and relations, and more on controllers.

Clocks webdsl postmortem

2010-06-15T00:00:00Z

In this post I will summarize the lessons learned while implementing clocks. An opensourced test project to learn the in an outs of macros in clojure.

You can see the code at: http://www.github.com/boymaas/clocks

On macros

if you don't like something in the language, just adapt it.

This is a core design choice of lisps. It is called homoiconic. When code is data, data is changable by all the existing routines.

In practice this means when calling a macro you get the unevaluated code. You can than do all kinds of operations to this code, return it and the returned form will get evaluated.

This gives you the oppertunity to generate/transform/compile code which will get evaluated after the macroexpantion is done.

First expand then evalutate

The macro expander first expands all code recursively and then executes it. So try to keep this in mind when creating more complex macro's. Sometimes the two worlds get blurred. Such as I experienced while first generating functions in a macro and then storing references to these generated functions in expantion time when they are not yet evaluated.

It helps to think in two layers, macro-expantion time and evalutation time.

Pitfall quasiquote symbol resolving

Scheme has hygienic macros, which work well for me, Clojure resolves all symbols first in the namespace the macro is defined. (when quasiquote is used)

outermacro defined in ns outerns
   body resolved in ns outns
   innermacro defined in ns innerns
     body gets resolved in innerns

This is a good thing in most cases since it avoids variable capture. But can be annoying when you just want to work with symbols.

When you want unresolved symbols, a problem with quasiquote

When defining a dsl using symbols not defined in the namespace this can pose a problem since quasiquote will try to resolve every symbol. There is a way to unresolve symbols. The name function:

(name resolved-symbol)

will give back the original symbol name as a string

(symbol (name resolved-symbol))

will convert the returned string back to the original symbol

(prewalk (fn [exp] (if (symbol? exp) (symbol (name exp)))) sexp)

We can combine this with a walker to unresolve all resolved symbols in a form. Combine this with a lookup on a symbol map and you will leave only specified symbols unresolved.

Prewalk? tools for transforming code, The walker

clojure.walk provides prewalk and postwalk, the walkers take a function and replace the walked node by the return value of this function.

Sequence conversion

While walking a piece of code you wish to convert, clojure has the tendency to convert the walked forms to sequences. When walking code you want vectors to stay vectors, etc. These have to be converted back.

As can be seen in the code the walk function as defined in clojure.walk does this for you.

(defn walk
  "Traverses form, an arbitrary data structure.  inner and outer are
  functions.  Applies inner to each element of form, building up a
  data structure of the same type, then applies outer to the result.
  Recognizes all Clojure data structures except sorted-map-by.
  Consumes seqs as with doall."
  {:added "1.1"}
  [inner outer form]
  (cond
   (list? form) (outer (apply list (map inner form)))
   (seq? form) (outer (doall (map inner form)))
   (vector? form) (outer (vec (map inner form)))
   (map? form) (outer (into (if (sorted? form) (sorted-map) {})
                            (map inner form)))
   (set? form) (outer (into (if (sorted? form) (sorted-set) #{})
                            (map inner form)))
   :else (outer form)))

Evaluation

What exactly happens when calling a macro with the namespaces and the variables.

(ns nsdefined)
(def rv 100)
(defmacro nstest [name]
         (let [ns *ns* 
              rv rv
              rvr (resolve rv)]
         `(def name rv))

(ns nscalling)
(def rv 1000)
(nsdefined/nstest blah) # ns = nscalling, rv = 100, rvr = #'nscalling/rv

Conclusion: macro gets executed inside namespace its defined in. The ns is set to the calling ns, so resolve which uses ns to resolve will resolve to nscalling/rv.

(defmacro nstest [name]
         (let [ns *ns* 
              rv rv
              rvr (resolve rv)]
         `(def name ~'rv))

(nsdefined/nstest blah) # ns = nscalling, rv = 100, rvr = #'nscalling/rv

In this case ~'rv will not be resolved by quasiquore and the symbol of nscalling will be used (symbol capture). Thus be bound to 1000.

Summarizing ns is the namespace of the caller, quasiquote will try to resolve symbols inside the namespace the macro was defined. Unresolved symbols can lead to variable capture in the calling namespace.

Conclusion

It was an interesting journey into the world of macro's inside clojure. A very different approach to the hygienic macro's in scheme. Who are more template based. Resolving the symbols is a very good feature for the simpler macro's but have to be compensated when designing a symbol dsl, It would be nice to have a special quasiquote which does not resolve.

Hack, Loli, the rake/webpy for Haskell?

2010-05-24T00:00:00Z

I am investing alternative ways of developing web apps. I've looked at PLT-Scheme which was a nice framework to develop in, but missing a nice database abstraction layer like fe. SQLAlchemy in python. And in retrospect my conclusion on Scheme is that I find it a little verbose, expecially compared to Haskell. I like succintness, it gives a lot of power as descibed by Paul Graham in Succintness is Power.

The quantity of meaning compressed into a small space by algebraic signs, is another circumstance that facilitates the reasonings we are accustomed to carry on by their aid.
Charles Babbage, quoted in Iverson's Turing Award Lecture

Wat is hack?

Hack is inspired from ruby's Rack, WSGI Python, Ring in Clojure. In abstract terms Env -> Reponse. Hack is the Rack protocol implemented in Haskell. I stumbled upon hack while examining Loli developed by the same author Nfjinjing.

MPS notiation

Nfjinjing likes to turn the normal Haskell syntax around and uses his own util library MPS (Message passing style) which turns the order of normal Haskell syntax around making it look like a OO language (to relieve his already punished OO brain).

See MPS for the syntax implmeneted. I have collected the important code snippets from MPS:

  {-# INLINE (.) #-}
  (.) :: a -> (a -> b) -> b
  a . f = f a
  infixl 9 .

  (>) :: (Category cat) => cat a b -> cat b c -> cat a c
  (>) = (>>>)
  infixl 8 >

  (<) :: (Category cat) => cat b c -> cat a b -> cat a c
  (<) = (<<<)
  infixr 8 <

  (^) :: (Functor f) => f a -> (a -> b) -> f b
  (^) = flip fmap
  infixl 8 ^

Nothing really complicated but makes the code more succint. Which can be a good thing ;)

How does hack work?

type Application = Env -> IO Response

type Middleware = Application -> Application

In essence it is a mimimal representation of what every webserver does. It gets a Request which build an environment to which a Repsonse gets formed.

To see the typedefinitions for Env and response look below. It comes back in Loli. It's a nice and minimal implementation consisting of one file Hacks.hs. Very minimal. Nice..

Hack-contrib

Contains functions to interact with the hack specification. The hack datatypes:

Constants.hs constants for headers and other stuff
Mime.hs mime-type map
Request.hs which is in essence the Env, or the AppReader type in loli, explained later.
Response.hs which is also Response in hack, and the AppState in loli, explained later
Utils.hs handling hack namespaces ... what are they?

Hack-contrib middleware, or how to extend Hack

Just as in Clojures Ring, and Ruby's rack, middleware consists of a function which wraps another function which does something with either dispatching or the environment or the response. It is a very clean seperation of concerns which enables composition of components in a very flexible decoupled way.

Take as example the regexp router, which is an example of a middleware compononent:

  type RoutePath = (String, Application)

  type Middleware = Application -> Application

  type Application = Env -> IO Response

  regexp_router :: [RoutePath] -> Middleware
  regexp_router h app = \env ->
    let path = env.path_info
    in
    case h.find (fst > flip match path > isJust) of
      Nothing -> app env
      Just (_, found_app) -> found_app env

Returns a lambda which receives environment. Acts upon this evironment and runs the found_app if a route matches or dispatches to then next in middleware pipeline, ending utlimately in returning an IO Repsonse at the en of the chain.

Below another example of a middleware, this is static file serving.

  file :: Maybe String -> Middleware
  file root _ = \env -> do
    let path = env.path_info .unescape_uri
    if ".." `isInfixOf` path
      then forbidden
      else path.serve root

Hack-contrib provides at this moment (Tue May 25 ) the following Middlewares:

BounceFavicon.hs
Censor.hs
Config.hs
ContentLength.hs
ContentType.hs
Debug.hs
ETag.hs
File.hs
Head.hs
Hub.hs
IOConfig.hs
Inspect.hs
Lambda.hs
Lucky.hs
NotFound.hs
RegexpRouter.hs
ShowExceptions.hs
SimpleAccessLogger.hs
Static.hs
URLMap.hs
UTF8Body.hs
UserMime.hs

Loli

What is loli?

Minimal Web DSL written on top of hack. It's a very succint dsl language to define web apps. As 'webpy' in python.

Example app

main ::  IO ()
main = run 3000 . loli - do
     get "/bench" - do
        name <- ask ^ params ^ lookup "name" ^ fromMaybe "nobody"
        html ("<h1>" ++ name ++ "</h1>")

     get "/" (text "loli power")

Lets dive in ..

Looks pretty efficient, what exactly happens here? To understand let's look at the used types.

run ::  Hack.Handler.SimpleServer.Port -> Hack.Application -> IO ()

loli ::  Network.Loli.Type.Unit -> Hack.Application

-- Network.Loli.Type.Unit    
type UnitT a = State Loli a
type Unit    = UnitT ()

data Loli = Loli
  {
    current_router  :: Router
  , routes          :: [RouteConfig]
  , middlewares     :: [Middleware]
  , mimes           :: Assoc
  }

So in abstracto Loli dsl functions like get. Define an Appunit.

get :: String -> Network.Loli.Type.AppUnit -> Network.Loli.Type.Unit

Appunit is defined as:

type AppReader    = Env
type AppState     = Response
type AppUnitT     = ReaderT AppReader (StateT AppState IO)
type AppUnit      = AppUnitT ()

And could be seen as the AppUnit which will be run on a Hack Application

AppUnitT is a double MonadTransformer

State over Response (modifyable)
Read over Env (the hack-environment)

Env and Response are defined inside the Hack libraries and defined below.

data Env = Env 
  {  requestMethod :: RequestMethod            -- ^ HTTP request method ....
  ,  scriptName :: String                      -- ^ The initial portion o....
  ,  pathInfo :: String                        -- ^ The remainder of the ....
  ,  queryString :: String                     -- ^ The portion of the re....
  ,  serverName :: String                      -- ^ When combined with SC....
  ,  serverPort :: Int                         -- ^ preference to SERVER_....
  ,  http    :: [(String, String)]              -- ^ All http header varia....
  ,  hackVersion :: [Int]                      -- ^ The Array [0,1], repr....
  ,  hackUrlScheme :: Hack_UrlScheme          -- ^ HTTP or HTTPS, depend....
  ,  hackInput :: B.ByteString              -- ^ body of the request
  ,  hackErrors :: HackErrors                  -- ^ error stream
  ,  hackHeaders    :: [(String, String)]      -- ^ custom headers, inten....
  }
  deriving (Show) -- careful with showing this, it now causes an infinite
                  -- loop with certain handlers due to the use of a
                  -- lazy bytestring

Response defined by

data Response = Response
  {  status :: Int                              -- ^ must be greater than ....
  ,  headers :: [(String, String)]              -- ^ The header must not c....
  ,  body :: B.ByteString                       -- ^ body of the response
  }
  deriving (Show)

Let's dive in some more

Let's take for example the first line of code in our example:

name <- ask ^ params ^ lookup "name" ^ fromMaybe "nobody"

What happens here:

ask gets State from ReaderT countaining AppReader which is Hacks Env
grabs query_string from Hacks Env and decodes the params

params :: Env -> [(String, String)] params env = if env.query_string.empty then [] else env.query_string.formDecode
which returns a map of the params on whick lookup is called which returns a Maybe.
fromMaybe gives a default of nobody on a Nothing.

Templating using `Network.Loly.Template`

Uses StateMonads update function to update Response setting content-type and body.

html :: String -> AppUnit
html x = do
  update - set_content_type _TextHtml
  update - set_body (x.fromString)
  render_layout

Hacks middleware on Loli level

Network.Loli.Engine

add_middleware :: Middleware -> Loli -> Loli
add_middleware x s = 
  let xs = s.middlewares in s {middlewares = xs.insert_last x}

And ofcourse defined in Network.Loli.DSL on unit level.

middleware :: Middleware -> Unit
middleware = add_middleware > update

Loli Engine

Contains code

loli :: Unit -> Application
loli unit = run unit not_found_app
  where
    not_found_app = not_found dummy_app
    run_route x = (x.router) loli_captures run_app (x.route_path)

    run :: Unit -> Middleware
    run unit' = 
      let loli_state    = execState unit' def
          route_configs = loli_state.routes
          route         = route_configs.map run_route .use
          mime_filter   = user_mime (loli_state.mimes)
          stack         = loli_state.middlewares.use
          pre           = pre_installed_middlewares.use
      in
      use [pre, mime_filter, stack, route]

Where use is defined as:

-- | usage: app.use [content_type, cache]
use :: [Middleware] -> Middleware
use = reduce (<<<)

essentially right to left composition of all middlewares, thus wrapping them in one another:

pre-installed middlewares
the mime filter
the subwrapped stack of user defined middlewares (loli-state)
route config (loli-state)

Routing

data RouteConfig = RouteConfig
  {
    route_path :: RoutePath
  , router     :: Router
  }

data Loli = Loli
  {
    current_router  :: Router
  , routes          :: [RouteConfig]
  , middlewares     :: [Middleware]
  , mimes           :: Assoc
  }

  instance Default Loli where
    def = Loli loli_router def [dummy_middleware] def

Zipping up ..

This is just a quick glance at how Hack and Loli where implemented. Shedding some light on the structure of this approach to web apps inside haskell.

Currying in scheme

2010-05-20T00:00:00Z

Coming from haskell i was looking around for a way to apply currying inside scheme. I came on the following macro via: this paste.

(define-syntax curried
(syntax-rules ()
    ((curried () body ...) (lambda () body ...))
    ((curried (arg) body ...) (lambda (arg) body ...))
    ((curried (arg args ...) body ...)
     (lambda (arg . rest)
         (let ((next (curried (args ...) body ...)))
             (if (null? rest)
                     next
                     (apply next rest)))))))

(define-syntax define-curried
    (syntax-rules ()
        ((define-curried (name args ...) body ...)
         (define name (curried (args ...) body ...)))))

Notice how define-curried uses the pattern match on name recursively to recursivly apply the curried macro to every argument.

(define-curried (mad a b c)
    (+ (* a b) c))

    (mad 2 3 4)
    ((mad 2) 3 4)
    ((mad 2 3) 4)
    (((mad 2) 3) 4)

Web-apps with PLT-Scheme, snoozing while dispatching?

2010-05-15T00:00:00Z

Doing the tutorial on writing a blog using PLT-Scheme opened my eyes to some of the possibilities. Working lately primary on webapps using web.py, sqlalchemy I wanted to see what could be done using scheme. In this case browsing PLT's available libraries. After scanning PLT's PLaneT I have found some interesting libs opensourced by a company called untyped.

The libs I found interesting where:

untyped/dispatch
untyped/snooze
untyped/mirrors

I decided to take a closer look at them and how untyped implemented its magic.

Dispatch, declare what you want

Dispatch is essentially syntax to declare routes and controllers. I wanted to know more so I analysed an example given by the documentation.

Define your site

#lang scheme/base

;; automaticly grab packages from PLaneT
(require (planet untyped/dispatch))

(define-site blog
             ([(url "/") index]
              [(url "/posts/" (string-arg)) review-post]
              [(url "/archive/" (integer-arg) "/" (integer-arg))
               review-archive]))

Essentially stating that /archive/10 will call review-archive with one parameter 10.

Controllers

Controllers for the rules above may be defined with:

(define-controller (controller request args ...)
    exp ... )

An example being:

; request string -> html-response
(define-controller (review-post request slug)
  `(html (head (title ,slug))
         (body (h1 "You are viewing " ,(format "~s" slug))
               (p "And now for some content..."))))

Implementation

To find out what happens behind the syntax it helps to expand the syntax (or macro). The declarative statements above expand to:

   (require (planet untyped/dispatch))
   (begin
     (define-values (blog index review-post review-archive)
       (let-values (((blog controllers)
                     (make-site 'blog '(index review-post review-archive))))
         (let-values (((index review-post review-archive)
                       (apply values controllers)))
           (set-site-rules!
             blog
             (list
               (make-rule (make-pattern "/") index)
               (make-rule (make-pattern "/posts/" (string-arg)) review-post)
               (make-rule
                 (make-pattern "/archive/" (integer-arg) "/" (integer-arg))
                 review-archive)))
           (values blog index review-post review-archive)))))

Effectively defining blog and the controllers index review-post review-archive on module level by the function make-site. This procedure also assigns an undefined-controller to the controllers index review-post review-archive to catch the undefined ones. These syntax-extentions are defined in syntax.ss.

One can use the export-helper in a provide as such (provide (site-out blog)) to export the defined site and the configured controllers.

Rule based routing

There is a function defined (controller-url controller args ..) which can be used to translate back from controller to path.

Rules

Rules like [(url "/posts/" (string-arg)) review-post] are declared when configuring the site. This rule effectively states that /posts/([^/]+) maps to controller review-post. This rule will be used to dispatch this request, but will also be used to translate (controller-url 'review-post 10) to the following link /posts/10.

Arguments

How is this implemented. In the files pattern.ss and arg.ss and struct-private.ss contain statements which describes elements to build reqexp matches and encode and decode arguments between the url and scheme domain. The args and eventually custom args are defined in the following struct. (i removed the display routines for clarity)

(define-struct arg
  (id      ;; symbol to be used declaration 
   pattern ;; regexp pattern
   decoder ;; how to decode from captured arg to lisp domain
   encoder ;; how to encode from lisp domain to captured arg
  #:transparent)

A working example is string-arg:

(define (string-arg)
  (make-arg 
   'string                      ;; id
   "[^/]+"                      ;; regexp
   (lambda (raw)                ;; decode
     (uri-decode raw))
   (lambda (arg)                ;; encode
     (if (string? arg)
         (uri-encode arg)
         (raise-exn exn:fail:dispatch
           (format "Expected string, given: ~s" arg))))))

Adding arguments

Adding cutom arguments is made trivial by this setup since one can just build a structure with encode and decode routines and can use it.

Snooze, ORM for Postgresql or Sqlite

Will come later ..

Mirrors, xhtml/xml generation or javascript. Whatever you want ..

Will come later ..

Ying Yang call/cc or the call-with-current-continuation puzzle

2010-05-14T00:00:00Z

One of the more interesting paradigms arising from Scheme is the call-with-current-continuation. Let me try to explain this in abstract terms, a call/cc is passed a procedure p. This procedure takes one argument k which represents the continuation. The continuation k can be applied with a value. This value will become the value of the application of call/cc. Easy heh?

Ok? Let's play a little with them

To give an example, this first example does call/cc which does not apply the continuation k:

(call/cc
 (lambda (k)
  (* 5 4))) =>  20

The lambda is called with our current-continuation which is ignored as it just returns the result of the calculation.

In this next example the continuation is applied with an integer argument of 4, the return value of the call/cc lambda becomes the value passed to the continuation k and thus prints 4.

(call/cc
 (lambda (k)
  (* 5 (k 4)))) => 4

We could argue that the continuation acts as a return statement. This facilitates returning from a recursion as described below.

(define product
  (lambda (ls)
    (call/cc
      (lambda (break)
        (let f ((ls ls))
          (cond
            ((null? ls) 1)
            ((= (car ls) 0) (break 0))
            (else (* (car ls) (f (cdr ls))))))))))

This is simple, let's try to find something more interesting

Try this piece of code which I found in The Scheme Programming Language.

(let ((x (call/cc (lambda (k) k))))
  (x (lambda (ignore) "hi"))) => "hi"

What happens here?

x gets bound to the continuation k
- since call/cc calls lambda k: which returns itself
x, which is the continuation k, is applied with the (lambda (ignore) "h1")
- this has the effect of rebinding x to (lambda (ignore) "hi")
control continues and comes again to the statement (x (lambda (ignore) .. )
- and thus applies the lambda to the rebond x returning "hi"

Now for something really interesting, the call/cc ying yang puzzle.

Reading the wikipedia article I stumbled upon this interesting puzzle. I was immediatly fascinated as it seemed difficult to explain why the below described call/cc combinations deliver the output as described.

(let* ((yin
         ((lambda (cc) (display #\@) cc) (call-with-current-continuation (lambda (c) c))))
       (yang
         ((lambda (cc) (display #\*) cc) (call-with-current-continuation (lambda (c) c)))))
    (yin yang))

Outputs an infinitive sequence, I have extracted the first output of this function below.

@*@**@***@****@*****@******@*******@********@*********@**********@*******
****@************@*************@**************@***************@**********
******@*****************@******************@*******************@*********
***********@*********************@**********************@****************
*******@************************@*************************@**************
************@

Keep in mind that (let* (b1 b2) e1 e2 ..) translates to (let (b1) (let (b2) e1 e2 .. ). Try to explain this one .. good luck, heheh

Live coding OpenGl in PLT-Scheme

2010-05-11T00:00:00Z

Live coding (fluxus) captured my interest. Evolutionary programming a game system using scheme will be a nice pet project to experiment with.

PLT-Scheme

Checking out different versions of scheme i stumbled upon gambit and ypsilon and plt-scheme. After some deliberation i chose plt-scheme. Nicely organised, good documentation and standard libraries and big community. Very active irc channel on irc.freenode.org #scheme which answered my questions within 5 minutes.

Wikipedia describes PLT scheme in a nice fashion:

From the beginning, the PLT project was also an experiment in evaluating the suitability of Scheme as a programming language for large projects. In a sense, the language failed and yet it also proved to be an ideal platform. Scheme per se failed because the standard language, as defined in the reports, is too small for a team of 20-30 developers, distributed across three continents. If taken as an ideal kernel, however, Scheme succeeded beyond the team's expectations.

We need a window to run our app in

PLT Scheme has a gui toolkit which can be loaded with.

(require scheme/gui/base)

This library needs primitive graphics support at runtime an cannot be executed with mzscheme. For these programs we have mred.

MzScheme, MrEd, PLT's graphical user toolkit

As I understand it, PLT-Scheme has mzscheme as an ideal lightweight scripting environment. And MrEd for GUI based apps.

 mred -iz -l readline -l scheme/enter

We load 2 libraries not standard included in my version of mred:

readline, up/down, tab-completion etc
scheme/enter, to be able to reload modules and change evaluation contexts, provides enter! procedure.

Windowing time

The gui/base consist of an hierarchy of classes, interesting for us are:

canvas%, a general-purpose window for drawing and handling events.
frame%, a top-level container window. It has a title bar (which displays the frame's label), an optional menu bar, and an optional status line.

Every painter needs a canvas

For our app to at least draw anything we need an opengl ready canvas. The window should also be able to recognize size changes and a method to actually draw something on our canvas.

the functions resize and paint will be defined later.

(define gl-canvas%
  (class* canvas% ()
          (inherit with-gl-context swap-gl-buffers)

          (define/override (on-paint)
                           (with-gl-context
                             (lambda ()
                               (paint)
                               (swap-gl-buffers))))

          (define/override (on-size width height)
                           (with-gl-context
                             (lambda ()
                               (resize width height))))

          (super-instantiate () (style '(gl)))))

Without the frame we cannot hang the painting

Our canvas needs to framed for the os to display it.

(define window 
  (new frame% (label "Minimal") (min-width 200) (min-height 200)))

Rendering time

Now we have setup a minimal frame with a canvas which is opengl ready. Now we can execute OpenGl calls to actually start drawing. resize is called on a resize and since this is minimal does nothing. paint is called when the canvas is exposed or resized so that the image in the canvas can be repainted.

;; gl-canvas functions resise and main draw
(define (resize w h) #t)

(define (paint)
  (glClearColor 0.0 0.0 0.0 0.0)
  (glClear GL_COLOR_BUFFER_BIT)
  (glColor3d 1.0 1.0 1.0)

  (glMatrixMode GL_PROJECTION)
  (glLoadIdentity)
  (glOrtho 0.0 1.0 0.0 1.0 -1.0 1.0)
  (glMatrixMode GL_MODELVIEW)
  (glLoadIdentity)

  (glBegin GL_QUADS)
  (glVertex3d 0.25 0.25 0.0)
  (glVertex3d 0.75 0.25 0.0)
  (glVertex3d 0.75 0.75 0.0)
  (glVertex3d 0.25 0.75 0.0)
  (glEnd)
  )

Putting it all together

To actually create the frame canvas, and fire the on-paint and on-size triggers send the window instance the show message. This should create all.

;; show window
(send window show #t)

And now for live coding!

The essential element here is the live coding. This is accomplished by changing the paint function and sending it to your repl. First we need to change the evaluation context. This is done by:

(enter! "minimal.ss")

and redefining paint:

(define (paint)
  (glClearColor 0.0 0.0 0.0 0.0)
  (glClear GL_COLOR_BUFFER_BIT)
  (glColor3d 0.0 1.0 1.0)

  (glMatrixMode GL_PROJECTION)
  (glLoadIdentity)
  (glOrtho 0.0 1.0 0.0 1.0 -1.0 1.0)
  (glMatrixMode GL_MODELVIEW)
  (glLoadIdentity)

  (glBegin GL_QUADS)
  (glVertex3d 0.25 0.25 0.0)
  (glVertex3d 0.75 0.25 0.0)
  (glVertex3d 0.75 0.75 0.0)
  (glVertex3d 0.25 0.75 0.0)
  (glEnd)
  )

now change size of the window to trigger a on-paint and the colour of the square changes. The basic elements of live coding are accomplished.

Playing around with the HTML5 Canvas element

2010-05-04T00:00:00Z

After all the fuzz cause by Steve Jobs objections against Flash and indirect promotion of HTML5 I wanted to see whats possible using HTML5 and the canvas element. After searching a bit online I stumbled upon some interesting articles. One implementation of Quake2 using HTML5, and the implementation of an SWF player using HTML5. This seeded my interest a little more and I wanted to see how easy it is to get something on the screen.

Simple outline of a tree based renderer

As an example project I wanted to create a simple tree graph renderer inspired by the Flash MovieClip hierarchy. The final outcome should become something like this.

// add root shapes
var canvas = new Canvas(c.canvas, c.ctx);
canvas.addShape(new Symbol(50, 25));
canvas.addShape(new Symbol(160, 25));
canvas.addShape(new Symbol(160, 135));
canvas.addShape(new Symbol(50, 135));

The Canvas element should take care of the rendering process and render the symbols. Rendering the complete tree at 20 frames a second will be done using the following code.

canvas.init();
canvas.render();
canvas.run(20);

Shapes

At a minimum our renderer should be able to place shapes onto the screen. The minimal information I want to update through the Tree is the x and y position. They should be able to update their state and be capable of rendering themselves on a context.

update(timeInMilliseconds), function to update properties of the shape.
draw(context), actual rendering on the canvas

The Box as you can see it moving above has been defined as:

function Box(x, y, width, height) {
    // simple element, just renders a strokeRect
    Shape.call(this, x, y);

    this.width = width;
    this.height = height;

    this.draw = function (ctx) {
        ctx.strokeRect(this._x, this._y, this.width, this.height);
    }
}

This setup enables composition of complex shapes using simpler shapes as can be seen below.

function Symbol(x, y) {
    // initialise shape
    Shape.call(this, x, y);

    // compose this shape out of 2 boxes
    this.box = this.addChild( new Box(0,0,100,100) );
    this.innerbox = this.box.addChild(new Box(10,10,80,80));
}

Behaviours

To create animations we have multiple options. One can always supply a function to update x,y,width,heigh or rotation. But many behaviours are standard, and especially with complex behaviours we would like to compose them. So I added the possibility to add behaviours to specific shapes.

function Behaviour() {
    // Basic methods for a behaviour
    // stores shape in shape and original coords
    this.init = function (s) {
        this.shape = s;
        this.original_x = s.x;
        this.original_y = s.y;
    }

    // apply applies the behaviour .. should
    // be implemented ina specific behaviour
    this.apply = function (time) {
    }
}

To test this behaviour I implemented the Wobble as can be seen above:

function BehWobble(duration,amplitude) {
    // BehWobble is a behaviour
    Behaviour.call(this);

    this.wobble = 0;
    this.duration = 1000*duration;
    this.amplitude = amplitude;

    this.apply = function (time) {
        this.shape.x -= this.wobble;
        this.wobble = this.amplitude * Math.sin(toRad(360*((time%this.duration)/this.duration))); 
        this.shape.x += this.wobble;
    }
}

Putting it together

I update the symbol function to add the behaviour to the boxes. And voila the result can be seen above:

function Symbol(x, y) {
    Shape.call(this, x, y);
    this.box = this.addChild( new Box(0,0,100,100) );
    this.innerbox = this.box.addChild(new Box(10,10,80,80));

    // let the innerbox wobble
    this.box.addBehaviour(new BehWobble(4,10));
    this.innerbox.addBehaviour(new BehWobble(2,10));
    this.innerbox.addBehaviour(new BehWobble(1,-10));
}

Conclusion

Just a little abstraction to rendering stuff in the HTML5 canvas element. So far I am pretty happy with it but started looking for some existing frameworks. Landed on processing.js which looks promising.

Code can be read here, Simple engine code, and the wobbling squares are defined here.

Procedural Generation, a way to go?

2010-02-17T00:00:00Z

The light flows, the smell of electricity; implicit order, outward manifestation ..
=> Boy Maas (2005) <=

Working as a small development studio, often in small teams with limited resources, you start looking at alternative ways to do things. Large development studios can spend millions of dollars on top-notch designers to get state of the art graphics. But as we all know, on a small salary working as a freelance programmer I can't just go the bank and state: I have this brilliant concept, please front me a few million because the graphics have to be brilliant.

Working a day job even time is a limited resource, designing all graphics yourself or even when you find artists willing to put in the time, is almost not feasible. But when you have the stamina and determinism you would probably have to rewrite your code and graphics for the next quantum computer. Since that will be the standard when you finish your product.

There are more roads to Rome

I often find myself conflicted while brainstorming on projects, on the one hand you have the conceptual world in which everything is possible. On the other side you have the real world, and with this real world comes various challenges.

Once you find a way to overcome, or work around the various challenges, you have a shot at realising your concept. One tool to overcome the practical challenges of content is procedural generation.

Let's dive in with .kkrieger

Before studying the process of procedural generation in some more detail let's look at what is done at this moment. A brilliant example of procedural generation arose from the demo-scene. .kkrieger A proof that almost all content is generated is the size of the game. All content is packed in 96Kb! The way the developers achieved this is is by basing all the elements on basic solids, such as boxes and cylinders with a function to deform them. Textures are stored using creation history thus only having to store the pixmap and the functions to manipulate them to get the end result. This way they could pack 300Mb worth of textures and data in a sheer 97,280 bytes of disk space.

Was this done before?

A few of the first games ever developed on BBC Mirco (16Kb ram) where confronted with severe space constraints. This poses some serious real world challenges on what is possible. One has to be very inventive to be able to produce an interesting game. The earliest computers where severely limited by memory constraints. This made it impossible to store everything, level information, artwork, there was only room for a few elements.

Elite, the open ended space trader game

An interesting example of a game I really loved is Elite, the creators designed a formula to store 2⁴⁸ (282 trillion) galaxies with 256 solar systems each. The solar systems had their own political systems, the trading prices, even the names of the planets where generated. They limited the final game to only 8 universes. The are many more examples of "old-school" games which used this technique. Often based upon a seeded Pseudo Random number generator used to feed the generation functions.

Exile, the cavern explorer game

Another Magnum Opus of programming which conquered serious technical challenge is Exile), I found an interesting read giving some interesting quotes from the developers:

Technical issues aside, and against all modern business rules, the Exile game itself was allowed to evolve from little more than having a spaceman exploring a cavern system. We didn't know where it was going in the early stages and really just thought about new experiences for the hero and new technical features (like line of sight vision for inhabitants) as we went. Personally I believe that the best creations occur if allowed to develop at least partly in this evolutionary way rather than completely designed from scratch at the outset. It takes much longer of course to produce something this way and you are in danger of never finishing if you lose sight of the goal, but is worth it in the end.

Notice the evolutionary process. Planning in detail ahead is a killer for any form of creativity.

The amount of innovating techniques and content creation used in programming the gameplay of Exile is stunning. Think only about the map in Exile, another quote from the article:

The huge map of Exile was generated by an algorithm which from an input of the xy map coordinates created tunnels, solid rock, or one of the few special blocks that could have manually defined objects in them, like creatures, doors, guns, etc. It took a long time to fiddle with this algorithm to produce a tunnel system that was all linked up and interesting, but the storage required for it was actually tiny.

To see the world which they generated procedurally: Exile's complete map

The ideal world

Imagine a completely generated game world, where even the creatures you interact with are generated. Their intelligence adapts to the way you play the game. Keeping the difficulty at your level. The AI is finding your weak spots and is trying to exploit them. You have to adapt to a world that is adapting to you and challenging you. All you see is generated on the fly. No element of the game is designed by an artist. Any ideas?

Hakyll blog setup

2010-02-09T00:00:00Z

Wondering for a while on how to setup a blog, one of my favourite languages is haskell. One of my favourite editors is vim. And one of my favourite principles is to automate as much as possible. It's nice to live life according to principle which could be defined as:

A basic truth, law, or assumption: the principles of democracy.
A rule or standard, especially of good behavior: a man of principle.
The collectivity of moral or ethical standards or judgments: a decision based on principle rather than expediency.
A fixed or predetermined policy or mode of action.

Another favourite principles, next to minimalism, is doing stuff as efficient as possible. What is more efficient as serving static pages? Hence the hakyll setup.

Maintaining my blog

I really like to write in vim in combination with other linux system tools to automate away all the routine tasks. For this end I have included nemesis in my toolchain. For 2 reasons:

keep everything haskell
blog will have dependencies I do NOT want to track mentally
put deploying the website including dependencies which need to be resolved inside Nemesis

What is hakyll?

Like the homepage of hakyll states:

Hakyll is a Haskell library for generating static sites. It is written in a very configurable way and uses an xmonad-like DSL for configuration.

Why static site generators?

Static sites are extremely fast. You dont have to worry about security issues.

Based upon

I based this site upon the blog of the creator of hakyll. But it will adapt to a more custom layout in the future.