Macro in Racket

Syntax Object

• Definition

  A syntax object combines a simpler Racket value, such as a symbol or pair, with a scope set (lexical information) at each phase level, source-location information, syntax properties, and tamper status (to control the way that unexported and protected module bindings are used).

• Usage

  ;; normal usage
  (define s1 (syntax (+ x y)))
  ;; shorthand for the above expression, like quote
  (define s2 #'(+ x y))
  (define-values (x y) (values 1 2))
  ;; like quasiquote, there is the quasisyntax, the use of them are very similar
  (define s3 #`(+ #,x #,y))
  (define lst (list 1 2 3))
  (define s4 #`(list #,@lst))
  (define s5 #`(+ #'x #'y))
  ;; rs1, rs2 and rs3 have same value, 3
  ;; but s3 has different syntax->datum from s1 and s2
  (define-values (rs1 rs2 rs3) (values (eval s1) (eval s2) (eval s3)))
  (syntax->datum s1)
  (syntax->datum s2)
  (syntax->datum s3)
  
  ;; #`#,x => <syntax 1>
  #`#,x
  ;; #`#'x => <sntax:21:9 (syntax x)>
  #`#'x
  ;; #`#,#'x => #<syntax:23:13 x>
  #`#,#'x
  

Hygiene

• Definition

  The organizing policy of Racket's macro system. Hygiene could be seen as the way of managing lexical context (lexical scope). There is the unhygiene, which cloud be seen as the way of managing dynamic context (dynamic scope). That says that Racket adopts lexical-binding instead of dynamic-binding.

  I am not going to explain what lexical-binding and dynamic-binding are. There are many outstanding sources on Internet.

  And don't forget read Macro-Introduced Bindings.

Phase Level

• Phase

  It is a way to separate computations in a pipeline of processes where one produces code that is used by the next. The pipeline is not fixed (I guess, it seems that Racket may have many phase levels than I think).

  For example:

  pipeline: run-time (+1)-> compile-time (+1)-> meta-compile-time (-1)-> compile-time (-1)-> run-time

• Some phase levels and key words related to them
  1. phase level 0: top level, run-time, normal code, run-time phase, base environment
  2. phase level 1: compile-time, compile-time phase, macro, transformer environment
  3. phase level 2: meta-compile phase level (how about phase level > 2?).
  4. phase level -1: template phase level, template environment. If a macro uses a helper function that is imported with for-syntax, and if the helper function returns syntax-object constants generated by syntax, then identifiers in the syntax will need bindings at phase level -1 (how about the level <= -1?).
  5. label phase level: does not correspond to any execution time; used to track bindings without implying an execution dependency

• Shift phase level

  1. begin-for-syntax: shift phase level increased by one
  2. syntax-shift-phase-level: returns a syntax object with all of its top-level and module bindings shifted by shift phase levels
  3. for-syntax: shift phase level +1, used in require or provide form
  4. for-template: shift phase level by -1, used in require or provide form
  5. for-meta: shift phase level by n you specify, used in require or provide form

  Here is the example that shows how to use begin-for-syntax to shift phase level,

  ;; define "var" at phase level 0
  (define var 1)
  ;; define "var" at pahse level 1
  (begin-for-syntax (define var 2))
  ;; eval at phase level 0, we will get 1 printed
  (eval #'(displayln var))
  ;; eval at pahse level 1, we will get 2 printed
  (eval #'(begin-for-syntax (displayln var)))
  

• Mismatch phase level

  ;; two modules, this is a bad example to illustrate the mismatch
  ;; this will raise: var: unbound identifier in module in: var
  (module a racket
    ;; define "var" at phase level 0
    (define var "Variable")
    (provide var))
  (module b racket
    (require (for-syntax 'a))
    ;; imported "var" will be at phase level 1 due to for-syntax
    (define-syntax (m stx)
      #'(displayln var))
    (m))
  

  1. define-syntax defines a macro m (or syntax transformer), whose identifier is in phase level 0, and its body will be executed at phase level 1, we say, its body is at level 1. Something else you need to know is that m is called at phase level 0 and returns a syntax object #'(displayln var) at phase level 0, which will be evaluated then, we called this process "expanding".
  2. Here is the problem, the imported var is in phase level 1, but the "expanding" result is in phase level 0 in module b, phase level 0 and phase level 1, obviously, are not the same phase level. That is a mismatch.

  3. To fix the mismatch is to shift the mismatch objects to same phase level. Here, the m is defined at phase level 0, we can not just write (begin-for-syntax (m)). Let's think about this, after the "expanding", (m) will be replaced by (displayln var) and the mismatch is between that var at phase level 1 and that m is being called at phase level 0, so, to fix it, we should rewrite the require form in module b as (require 'a) or the body of m as

     ;;
     (define-syntax (m stx)
      #'(begin-for-syntax
          (displayln var)))
     

  4. Now, you have seen an interesting truth that the identifier and body (definition) of a same binding might be at different phase levels, the body executes at one phase higher than the context of its definition. To check out if match or not, just to list the bindings in the places where they are defined and where they are evaluated.

• More examples about mismatch phase level and how to fix them (explained in comments)

  1. Function called in the body of macro

     ;; this will be very easy to be fixed
     (define (greet) (displayln "hello"))
     (define-syntax (m stx) (greet))
     (m)
     ;; 'greet' defined at level 0, evaluated at phase level 1 when evaluating (m) at phase level 0
     ;; there two ways (or more) to fix it,
     ;; 1. define greet at phase level 1
     ;; (begin-for-syntax (define (greet) (displayln "hello")))
     ;; 2. the simplest way, defer the call to greet, have greet called at level 0
     ;; (define-syntax (m stx) #'(greet))
     

  2. That same module has different instantiations causes different bindings

     ;; raise "eval:1:0: var: unbound identifier"
     (module a racket
       (define var 0)
       (define var-syntax #'var)
       (provide var-syntax))
     (module b racket
       (require 'a (for-syntax 'a))
       ;; module b has four bindings:
       ;; for (require 'a):
       ;; 'var' bound at phase level 0
       ;; 'var-syntax' bound at phase level 0 refers to 'var' bound at phase level 0
       ;; for (require (for-syntax 'a))
       ;; 'var' bound at phase level 1
       ;; 'var-syntax' bound at phase level 1 refers to 'var' bound at phase level 1
       (define-syntax (m stx) var-syntax)
       ;; here is the problem, after "expanding", (m) will be #'var, which is imported
       ;; at phase level 1, but used at phase level 0.
       ;; why the 'var-syntax' in m is at phase level 1 instead of phase level 0? don't
       ;; forget the body of macro is at phase level 1 if macro is defined at phase
       ;; level 0.
       ;; back to the problem, to fix it is to shift #'var bound to 'var-syntax' at
       ;; phase level 1 to phase level 0. rewrite body of 'var-syntax'.
       ;; (define var-syntax (syntax-shift-phase-level #'var -1))
       (m))
     (require 'b)
     ;; but, don't forget other modules which imports module a if they exsit. it's a
     ;; bad idea to rewrite the body which are imported/used by the other, easy
     ;; to get errors.
     

  3. Function which returns a syntax object referring to a already defined object

     ;;raise "eval:1:0: var: unbound identifier"
     (module a racket
       (define var 0)
       (define (var-f) #'var)
       ;; var-f bound at phase level 0 refers to 'var' bound at phase level 0
       (provide var-f))
     (module b racket
       (require 'a (for-syntax 'a))
       (define-syntax (m stx) (var-f))
       ;; when evaluating "(m)" at phase level 0, "(var-f)" is evaluated at phase level 1 and returns
       ;; a syntax object "#'var" which is bound at phase level 1, in short, evaluating "(m)" at phase
       ;; level 0 then getting "var" bound at phase level 1 there. mismatch appeared.
       ;; to fix this, the simplest way is to rewrite the body of 'var-f' like
       ;; (define (var-f) (syntax-shift-phase-level #'var -1))
       (m))
     (require 'b)
     

  4. Template, helper function to be used in macro and identifiers used in helper function

     ;; it will raise "let: unbound identifier"
     (module helper racket/base
       (provide helper-f)
       (define (helper-f x y)                ; helper-f is bound at phase level 0
         #`(let ([tmp #,x])
             (set! #,x #,y)
             (set! #,y tmp))))
     ;; the problem is same to one in example 3 in this section. we need to take care
     ;; of the bindings of "let" and "set!"
     
     ;; to fix it, just add (require (for-template racket/base)) to 'module helper',
     ;; we need to have the identifiers 'let' and 'set!' bound at phase level -1.
     ;;
     (require (for-syntax 'helper))
     (define-syntax (swap stx)
       (syntax-case stx ()
         [(_ a b)
         #`(begin
             #,(helper-f #'a #'b)
             (list a b))]))
     (define x 3)
     (define y 4)
     (swap x y)                   ; should return '(4 3) if no error raised
     

     ;; This a working example by rewriting the above one.
     (module madd-mod racket/base
       (provide madd)
       (define (madd x y) (+ x y)))
     
     (module helper racket/base
       (require (for-template 'madd-mod)
       (for-template racket/base))
       (provide helper-f)
       (define (helper-f x y)
         #`(madd #,x #,y)))
     
       (require (for-syntax 'helper))
       (define-syntax (msum stx)
         (syntax-case stx ()
           [(_ a b)
             #`(begin
               #,(helper-f #'a #'b))]))
       (define x 3)
       (define y 4)
       (msum x y)