Hcfyk

In answer to exercise 11.5 there is this comment:

def\{ifspacenext % assume that next is unexpandable

What does this comment mean? If next may be expandable,
must we use the following instead?

def\{expandafterifxspacenext %

Can anybody think about an example which illustrates the difference?

Also, the result does not change if we change endlist to just end as follows:

-defdodolist{ifxnextendlist letnextrelax

+defdodolist{ifxnextend letnextrelax

-defendlist{endlist}

+%defend{endlist}

-defdemobox#1{setbox0=hbox{dolist#1endlist}%

+defdemobox#1{setbox0=hbox{dolist#1end}%

What is the point of using endlist?

EDIT

I think defining a control sequence, which need not be defined, the way
endlist is defined here, gives the user a way to tell whether a particular name
is in use, when he chooses a name for a new control sequence.

This is the full example:

defdolist{afterassignmentdodolistletnext= }

defdodolist{ifxnextendlist letnextrelax

  else \letnextdolist fi

  next}

defendlist{endlist}

defhidehrule#1#2{kern-#1%

  hrule height#1 depth#2 kern-#2 }

defhidevrule#1#2{kern-#1{dimen0=#1

    advancedimen0 by#2vrule widthdimen0}kern-#2 }

defmakeblankbox#1#2{hbox{lowerdp0vbox{hidehrule{#1}{#2}%

    kern-#1 % overlap the rules at the corners

    hbox to wd0{hidevrule{#1}{#2}%

      raiseht0vbox to #1{}% set the vrule height

      lowerdp0vtop to #1{}% set the vrule depth

      hfilhidevrule{#2}{#1}}%

    kern-#1hidehrule{#2}{#1}}}}

defmaketypebox{makeblankbox{0pt}{1pt}}

defmakelightbox{makeblankbox{.2pt}{.2pt}}

def\{ifspacenext % assume that next is unexpandable

 else setbox0=hbox{next}maketypeboxfi}

defdemobox#1{setbox0=hbox{dolist#1endlist}%

  leavevmodecopy0kern-wd0makelightbox}

demobox{Tough exercise.}

bye

For readers who don't find the line:

def\{ifspacenext % assume that next is unexpandable

in their TeXbook, it is in the errata published by Donald E. Knuth (“page A311”).

Suppose you do:

deffoobar{cat}



noindent

demobox{The foobar in the hat}

After next has grabbed the foobar token, \ will expand to something equivalent to:

ifspacenext %

else setbox0=hbox{next}maketypeboxfi

with next being let-equal to foobar. According to the documentation of if (TeXbook p. 209), TeX is going to expand tokens following the if until it finds two non-expandable ones. space expands to an explicit space token in one step, so TeX goes on with next (which has the same meaning as foobar at this point), because it needs one more non-expandable token. After next has been expanded, the input is equivalent to:

if〈space token〉cat %

else setbox0=hbox{next}maketypeboxfi

where 〈space token〉 represents an explicit space token (one could define a control sequence that is let-equal to an explicit space token and use it instead of 〈space token〉, see footnote 1 below). Now, TeX has two non-expandable tokens following the if: a space token and a c character token (of category 11 under the normal catcode regime). So, the outcome of the if can be decided: it is false because the character codes of a 〈space token〉 and of c differ, so TeX will skip to the else clause.

There is no big problem so far, though we are going to box the whole cat at once instead of each character separately (c, a, and t); but let's back up a little bit. Had we used:

deffoobar{ cat}

the input would have been equivalent to:

if〈space token〉〈space token〉cat %

else setbox0=hbox{next}maketypeboxfi

The test would have been true and TeX would have left cat in the input stream, which is plain wrong, because we were supposed to test what we just grabbed in next, not to insert new text!

So, the comment “assume that next is unexpandable” could be rephrased more generally, in my humble opinion, as “assume that next ultimately expands to either (1) exactly one character token or (2) exactly one chardef token or (3) a control sequence token that is let-equal to (1) or (2)”² (the character token in (1) is necessarily non-active, because of the “ultimately”). Indeed, you can test that demobox works perfectly when next recursively expands to a single character token, as in:

defmyspacei{myspace}

defmyspace{space}



noindent

demobox{Abc defmyspacei pU gHi}

Using myspacei here gives the same result as using an explicit space token, because it recursively expands to such a token.

Here is another example that additionally uses a control sequence that recursively expands to a non-space character token:

defmyspacei{myspace}

defmyspace{space}



defmyxii{myxi}

defmyxi{myx}

defmyx{X}



noindent

demobox{Abc defmyspacei pUmyxii gHi}

Your proposal:

def\{expandafterifxspacenext %

...

would also work, as long as next has been let-equal to a space token (explicit or implicit). But it wouldn't work with input containing spaces in the form of macros like space or our myspacei macro defined above. Indeed, ifx distinguishes between character tokens and macros (see specification of ifx p. 210 of the TeXbook).

Finally, although it would work, your replacement of endlist with end does not sound like the best coding style to me, because end is an existing TeX primitive; Knuth chose something more “unique” to mark the end of the text to be worked on. Besides, the name endlist was visibly chosen to match dolist: it is a matter of consistency. See in particular:

defdemobox#1{setbox0=hbox{dolist#1endlist}%

...

Footnotes

1. 
   

   You can define a control sequence stoken that is let-equal to an explicit space token like this:

   
   
   

   {def\{globalletstoken= }\ }% now, stoken is an implicit space token
   
   

   
   
   

   (adapted from the TeXbook p. 376). Two other ways are given in the TeXbook p. 336 (exercise 24.6):

   
   
   

   def\{letstoken= }\ %
   
   

   
   
   

   and

   
   
   

   def\#1\{}futureletstoken\ \%
   
   

   
   



2. This is in particular the case when next has been let-equal to a non-active character token—which is, I think, the case Knuth had in mind when he used the word “unexpandable” (indeed, a non-active character token, or a control sequence that has been let-equal to such a token, never expands). In other words, the condition “next is unexpandable” from the comment you quoted is a sufficient condition for ensuring that the macros behave sanely, and is only a particular case of the more general condition I gave. :-)

It means what it says: only unexpandable tokens are allowed in the argument to demobox. More precisely, character tokens or unexpandable control sequences that correspond (via chardef or let) to printable characters (including spaces).

If you try

demobox{abc def}



defexpandable{expandable token}



demobox{expandable}

you get

which is probably not what you were expecting. On the other hand, a definition like

defexpandable{ expandable token}

would yield something very far from the expectations.

So the demobox macro can be used only with its argument consisting of unexpandable tokens or macros expanding to a single unexpandable token.

Also chardef tokens are allowed, as well as implicit character tokens. However bgroup would also be problematic: compare demobox{A bgroup ABegroup} with demobox{A AB}, to see the issue.

You might want to extend demobox in various ways, but that's not the object of the exercise.

About your suggestion to redefine dolist to use end instead of endlist: you can do it, if you prefer. Knuth doesn't prefer it; instead he uses a control sequence that's specific to dolist processing. Note that the definition of endlist will produce an infinite loop whenever endlist is expanded (probably by a mistake in the macros using dolist), whereas end wouldn't.