#| $Id: footnotes.lsp,v 1.5 2003/05/05 18:26:59 traverso Exp $
(C) 2003 Carlo Traverso, traverso@dm.unipi.it
    http://www.dm.unipi.it/~traverso

freely available and redistributable under GNU GPL, see 
http://www.gnu.org/licenses/gpl.txt


Utilities to exctract the footnotes from a DP proofed file, and to
merge them back (inline, at the end of the book/chapter/paragraph)


a) request to proofers:

* mark the footnote position with a number in brackets [1]; in the
  case of a double series of footnotes, mark the second series with
  an additional #, [#1]. It is possible that the same footnote has
  more than one marker. 

* mark footnote text in the following way:

[1 text of the footnote]

i.e.:
  * start the footnote at the first character of a line
  * insert the footnote number (or letter) without space after the bracket
  * leave at least one space between the number and the text of the
    footnote
  * end the footnote with a bracket at the end of a line (paragraphs,
    poems, etc can be in the body of the footnote

* if a footnote is a continuation of a previous page, number it with 0:

[0 nuation of a footnote in a previous page]

* trailing spaces are discarded and multiple spaces are equivalent to
  one space in the recognition of footnotes (so a proofer does not
  have to care of spaces) (this is mainly a remark for programmers)


b) special exceptions

Automatic note-moving software will consider a footnote marker
anything being a number or a single letter in brackets; and a footnote
anything starting at the beginning of a line with a bracket followed
by a number or a single letter. Moreover a footnote text ends at the
first ] that ends a line. To avoid misinterpretation, add a *. Hence:


* If a bracket followed by a number (or a standalone letter) followed
  by another bracket is a part of ordinary text, insert a * inside (
  [*27] instead of [27], [*a] instead of [a])

* if a bracket followed by a number starts a
  line, insert a * after the bracket 

[12 in brackets at the beginning of a paragraph]

should be

[*12 in brackets at the beginning of a paragraph]


* If in the text of a footnote a ] appears at end of line, add a *

[1 this is a text of a footnote: look at this poem
/*
here's a bucket
full of brackets ]*
any ace
gets a brace }   
*/
and tell me what you think]

(this is to avoid misunderstanding the end of the footnote)


c) post-processing software

Action of the footnote-moving software; three functions:

1) check for possible inconsistencies in the markup

2) collect all the footnotes, renumbering them consecutively for the
   whole book. When a footnote is a continuation it does not get a new
   number, it is marked [NIL] and the join has to be performed
   manually. The output consists two files, one with the markers
   numbered consecutively for the whole book, the other with the
   footnotes

3) when both files have been post-processed, the two files will be
   joined. This is done in four modes:

  a) leave the footnotes at the end of the book: just join the two files

  b) put the footnotes at specified places; there are two predefined
     operations modes:
     b1) put the footnotes at every paragraph break (i.e. at every blank 
         line); the footnotes will be preceded and followed by a blank line.
     b2) mark explicitly where the footnotes will be dumped, inserting a line
         containing   @@@   (the line will be erased)
     One can choose if the renumbering of the footnotes restarts whenever 
     the footnotes are dumped, or continues in the book.
  d) inline the footnotes, removing the markers. It is the responsability 
     of the post-processor to rewrap the text after the footnote insertion.


The software in modes 1 and 2 requires the page separators, otherwise
confusion will arise with footnotes with the same mark in different
pages. The case of multiple markers for the same footnote is handled.
Hence the splitting action has to be made before removing the
separators.  Page separators are inserted in the footnote file too, to
allow better manual cross-checking.

The software also allows to de-inline the inlined footnotes, but has
problems when an inlined footnote text contains a "]" that is
always interpreted as end of the footnote; moreover when inlined
footnotes and end-of-page footnotes appear in the same page some
manual action will be needed.


d) post-processing actions


The post-processor has to check first that the footnotes have been
correctly marked, possibly with the help of the checking function, and
correct them; then split into two files; process them, then join them.

There might be different styles for the footnote insertion; currently
I only support footnotes indented as block quote, with the marker
indented 2 spaces less; and a blank line between the footnotes.

  [1] bla bls blabla bls blabla bls blabla bls blabla bls blabla bls
    blabla bls blabla bls blabla bls blabla bls blabla bls blabla bls
    blabla bls bla

  [2] bla bls blabla bls blabla bls blabla bls blabla bls blabla bls
    blabla bls blabla bls blabla bls blabla bls blabla bls blabla bls
    blabla bls bla

Other styles are easy to add.



e) softrware features

Problems addressed:

- paragraphs and poems inside footnotes are handled correctly

- footnotes inside footnotes (has to be solved at the project level -
  a [23] inside a footnote is NOT considered a footnote marker)

[17 this is a footnote[1] that contains a footnote.
  
  [1] this subfootnote has a third-level[*] footnote

    [*] Wow!!! ]

(this is transformed into

  [17] this is a footnote[1] that contains a footnote.

    [1] this subfootnote has a third-level[*] footnote

      [*] Wow!!!

- the first version of the software only handles one type of
  footnotes; in future many types of footnotes will be possible.

|#



(setq *print-pretty* nil)



(defvar *page-markup*  "-----File:" "string characterizing a page separator")

(defvar *fn-indent* "    " "Indentation for footnote body")

(defvar *inline-footnote* "[Footnote:")

(defun split-footnotes(file1 file2 file3
			     &aux inline (fncounter 0) fnmalist fntlist infn
			     (page 0) fnms embfn)
  (with-open-file
   (ori file1 :direction :input)
   (with-open-file
    (txt file2 :direction :output :if-exists :rename)
    (with-open-file
     (fn file3 :direction :output :if-exists :rename)
     (loop
      (setq inline (read-line ori nil nil))
      (unless inline (return))
      (cond
       ((search *page-markup* inline)
	;;;  at page break: re-initialize internal page data
	(write-line inline txt)
	(terpri txt)
	(write-line inline fn)
	(terpri fn)
	(when infn (warn "Some footnote is not terminated in p. ~A" page))
	(match-fn fnmalist fntlist page)
	(setq fnmalist nil fntlist nil page (get-pn inline) rest t))
       (embfn
	;;; inside an embedded footnote
	(setq brak (search "]" inline))
	(write-indented-line (subseq inline 0 brak) fn)
	(when brak 
	  (setq embfn nil)
	  (princ (subseq inline (1+ brak)) txt)
	  (terpri fn)
	  (terpri txt)))
       (infn
	;;; inside an end-of-page footnote
	(write-indented-line (strip-footnoteend inline) fn)
	(when (ends-footnote inline)
	  (terpri fn)
	  (setq infn nil)))
       ((setq ifm (search *inline-footnote* inline :test #'equalp))
	;;; the line contains the beginning of an inlined footnote
	(princ (subseq inline 0 ifm) txt)
	;;; insert mark in both files
	(princ
	 (princ (footnote(incf  fncounter)) txt)
	 fn)
	(princ (princ #\space txt) fn)
	(setq embfn t)
	(setq ifm (position #\space inline :start ifm))
	(when ifm
	  (setq brak (search "]" inline :start2 ifm))
	  (princ (subseq inline ifm brak) fn)
	  (terpri fn)
	  (when brak
	    (terpri fn)
	    (princ (subseq inline (1+ brak)) txt) 
	    (terpri txt)
	    (setq embfn nil))) )
       ((starts-footnote inline)
	;;; start of an end-of-page footnote
	(setq fnm (extract-footnotemark inline))
	(if (assoc fnm fnmalist :test #'equal)
	    (pushnew fnm fntlist :test #'equal)
	  (warn "Footnote ~A is not called in pg. ~A" fnm page))
	(princ (footnote (cdr(assoc fnm fnmalist :test #'equal))) fn)
	(princ #\space fn)
	(write-line (extract-footnotetext(strip-footnoteend inline)) fn)
	(if (ends-footnote inline) 
	    (progn
	      (terpri fn)
	      (terpri txt))
	  (setq infn t)))
       ((equal (string-trim " " inline) "")
	;;; blank line
	(write-line inline txt))
       (T
	(when (and fntlist rest)
	  (warn "Text following the footnotes in p. ~A~%~A" page inline)
	  (setq rest nil))
	(setq fnml (footnotemarks-in inline))
	(dolist (fnm fnml)
	  (unless (assoc fnm fnmalist :test #'equal)
	    (setq fnmalist (acons fnm (incf fncounter) fnmalist))))
	(write-line 
	 (if fnml (change-fnm inline fnml fnmalist) inline) txt))))))))


	
	  
	

(defun footnotemarks-in (line &aux res (fnb 0) fne fnm)
  (loop
   (setq fnb (position #\[ line :start fnb)
         fne (if fnb (position #\] line :start (incf fnb))))
   (unless fne (return (reverse res)))
   (setq fnm (subseq line fnb fne))
   (if (good-fnm fnm) (push fnm res))))


(defun good-fnm(str)
  (not (find-if-not #'digit-char-p str)))
	 



(defun get-pn (str)
  "get the pagenumber from a separating line"
  (subseq str 29 32))

(defun match-fn (fnmalist fntlist page)
  (if (= (length fnmalist)(length fntlist))
      T
    (warn "Some footnote text is missing, page ~A ~%Markers: ~A Footnotes: ~A" 
	  page (mapcar #'car fnmalist) fntlist )))

(defun starts-footnote(line)
  (and (plusp (length line)) 
       (equal #\[ (elt line 0))
       (good-fnm (subseq line 1 (position #\space line)))))




(defun extract-footnotemark(line &aux (spos (position #\space line)))
  (if spos
      (subseq line 1 (position #\space line))
    (subseq line 1)))
  
(defun extract-footnotetext(line &aux (spos (position #\space line)))
  (if spos
      (subseq line (position #\space line))
    ""))

(defun write-indented-line (line str)
  (princ *fn-indent* str)
  (write-line line str))

(defun footnote (fnm)
  (concatenate 'string "[" (princ-to-string fnm) "]")) 

(defun change-fnm (str fnm fnmalist &aux (pos 0) (printed 0))
  (with-output-to-string 
    (*standard-output*)
    (dolist (mark fnm)
      (setq pos (search (footnote mark) str :start2 printed :test #'equal))
      (princ (subseq str printed pos))
      (princ (footnote (cdr (assoc mark fnmalist :test #'equal))))
      (setq printed (+ pos (length (footnote mark)))))
    (princ (subseq str printed)) ))


(defun strip-footnoteend(line)
  (if (ends-footnote line)
      (subseq line 0 (1- (length(string-right-trim " " line))))
    line))


(defun ends-footnote(line)
  (and (< 1 (length line))
       (eq #\] (elt line (1-(length(string-right-trim " " line)))))))



  

(defun footnotes-from (file &optional (pre-string "")
			      &aux fnalist footnote mark body)
  (with-open-file
   (str file :direction :input)
   (loop 
    (setq pk (peek-char #\[ str nil 'eof))
    (when (eq pk 'eof)(return fnalist))
    (setq line (read-line str))
    (setq mark (car (footnotemarks-in line))
	  body (with-output-to-string
		 (fns)
		 (princ pre-string fns)
		 (write-line (subseq line (position #\space line)) fns)
		 (loop
		  (setq line (read-line str))
		  (setq pk (peek-char nil str nil 'eof))
		  (cond
		   ((eq pk 'eof) (return))
		   ((eq pk #\[)(return))
		   ((search *page-markup* line))
		   (T (write-line line fns))))))
    (setq fnalist (acons mark(string-right-trim '(#\newline) body)fnalist)) )))



(defvar *footnote-place-mark* "@@@"
  "mark to use for the spots to insert footnotes")

(defun merge-footnotes-inline(txtfile fnfile mergefile &aux fnmal)
  (setq fnmal (footnotes-from fnfile "Footnote:"))
  (with-open-file
   (txt txtfile :direction :input)
   (with-open-file
    (ms mergefile :direction :output :if-exists :rename)
    (loop
     (setq line (read-line txt nil nil))
     (unless line (return))
     (setq line-marks (footnotemarks-in line))
     (write-line
      (if line-marks
	  (change-fnm line line-marks fnmal) 
	line) ms)))))



(defun merge-footnotes(txtfile fnfile mergefile 
			       &optional (footnoteplace *footnote-place-mark*)
			                 (restart-footnotes t)
			       &aux fnal fnmal (fncounter 0))
  (setq fnbody (footnotes-from fnfile ""))
  (with-open-file
   (txt txtfile :direction :input)
   (with-open-file
    (ms mergefile :direction :output :if-exists :rename)
    (loop
     (setq line (read-line txt nil nil))
     (cond
      ((not line) 
       (when (> fncounter 0)
	 (warn "~A footnotes remain unprocessed" fncounter))
       (return))
      ((equal (string-trim " " line) footnoteplace)
       (when (equal  "" footnoteplace) (terpri ms))
       
       (dolist (item (nreverse fnmal))
	 (setq fnm (cdr item)
	       fnt (cdr(assoc(princ-to-string(car item))fnbody :test #'equal)))
	 (princ (concatenate 'string "  " (footnote fnm) fnt) ms)
	 (terpri ms)   ;;; this should be governed by a flag....
	 (terpri ms))
       (setq fnmal nil fncounter (if restart-footnotes 0 fncounter)))
      (T
       (setq line-marks (footnotemarks-in line))
       (dolist (fnm line-marks)
	 (unless (assoc fnm fnmal :test #'equal)
	   (setq fnmal (acons fnm (incf fncounter) fnmal))))
;       (when line-marks (princ line-marks)(princ fnmal)(terpri))
       (write-line 
	(if line-marks (change-fnm line line-marks fnmal) line) ms)))))))




;;; Known problems:
;;;
;;; 1 - if an embedded footnote ends on a line, and another strats
;;;   there, the second is not recognized.
;;;
;;; 2 - continuation footnotes require reformatting as offline footnotes
;;;   with 0 mark;
;;;
;;; 3 - slightly non-standard markup makes a footnote unrecognized.
;;;
;;; Most problems are solved looking for Footnote in the output, and
;;; correcting the source accordingly. 2 risks being unrecognized.
;;;