music

OSdata.com

Forth-like routines for UNIX/Linux shell

summary

    This subchapter looks at a set of Forth-like routines for use in the BASH UNIX (Linux, Mac OS X) shell.

    You can obtain the latest copy of these functions at http://www.osdata.com/programming/shell/forthfunctions place the file in an appropriate directory.

working forth interpreter

    The search engines are diverting searches for my Forth interpreter to this webpage. The free open soruce Forth interpreter is available at http://www.osdata.com/milo.html. It works directly from your web browser.

free book on UNIX/Linux System Administration Teach Yourself UNIX/Linux System Administration and Shell Programming free computer programming text book project table of contents If you like the idea of this project, then please donate some money. more information on donating

Forth-like routines for UNIX/Linux shell

    This subchapter looks at a set of Forth-like routines for use in the BASH UNIX (Linux, Mac OS X) shell.

    You can obtain the latest copy of these functions at http://www.osdata.com/programming/shell/forthfunctions place the file in an appropriate directory.

warning

    At the time this paragraph was written the www.osdata.com was hosted on HostGator and at the time this paragraph was written HostGator tech support used the last four digits of a credit card as their security method for identifying the account holder. This is a well-known security flaw. The security hole was made famous when a hacker used this hole at Apple and Amazon to get Mat Honan’s account information and wipe out his hard drive. Both Apple and Amazon quickly changed their security procedures, but for some reason HostGator (at the time this was written) hasn’t bothered to change their security procedures. See http://www.wired.com/gadgetlab/2012/08/apple-amazon-mat-honan-hacking/ for details.

    For this reason, there is the distinct possibility that the forthfunctions file might have been hacked. Please take the time to read through it and make sure it looks correct and doesn’t have any suspicious or malicious code inserted. I have a distinctive programming style, so something written by someone else might stand out even if you don’t understand what I have written. Of course, it is better if you understand the code and can spot things that look wrong.

    I apologize for this security hole, but my website is hosted for free for me and I have no control over the hosting company chosen. The person providing me with the free hosting doesn’t believe me that this is a security flaw. This is one of the many hassles of poverty.

basic approach

    This supplementary material describes a method for creating Forth-like routines for use in the BASH shell.

    The data and return stacks will be stored using a pair of shell variables, one of which is an array and the other is an ordinary scalar variable.

    The Forth-like routines will be created as shell functions.

    Everything will be stored in a shell function file that can be loaded manually or loaded as part of the normal shell start-up.

    Eventually I intend to write two sets of C source code, one that people can compile into their own copy of the official BASH source code and one that is part of an entirely new open source shell project that doesn’t have the GNU infecting license. This alternative shell will be designed so that it can run on bare metal, allowing it to be used to deploy embedded systems.

stacks

    Forth uses two stacks: the main data stack and the return stack.

    For the purposes of the BASH shell, we will implement both stacks with a combination of a top of stack pointer variable and an array for the stack. The top of stack pointer will hold the array index of the top of the stack. The stack will build up from the array indexed as one. An empty stack is indicated by having the top of stack pointer variable point to the zero index of the stack array variable. The bottom of the stack will actually be the one index of the stack array variable.

    $ export DataStackPointer=0
    $ declare -a DataStack; export DataStack
    $ export ReturnStackPointer=0
    $ declare -a ReturnStack; export ReturnStack
    $

    That’s it for the required variables! And, yes, those could all be placed on one line, but I spread them over four lines for additional clarity.

    One very important difference between the real Forth stacks and the Forth-like function stacks is that the real Forth stacks deal with binary data and the Forth-like function stacks deal with character strings. Each item on a real Forth stack is a binary number (or binary bit string). Even characters are stored by their binary representation. Every item on a Forth-like function stack is a character string. Numbers are stored by their character string representation. And an individual item on a Forth-like stack can be a character string of arbitrary length, even an entire book.

direct random access

    While Forth insists on the stack always being accessed as a stack, there is nothing that prevents direct random access to Forth-like stack elements.

    A shell script mixing BASH and the Forth-like functions can directly read or write any individual element in the data or return stacks. For that matter, a mixed shell script can do any manipulations to the stacks. It is the programmer’s responsibility to maintain stack integrity.

    One particular gotcha is that I am not currently emptying out stack locations that are no longer used. This means that when the stack shrinks, there will be entries in the array that are no longer valid. Don’t be fooled by their existence. Modify my scripts to delete this information if it presents a security risk of any kind.

functions

    These are the functions for a Forth-like working environment. Not all of these functions exist in a real Forth. Most of these functions are have modifications from standard Forth. Many standard Forth functions are missing.

    In particular, this approach treats the stack as a stack of objects. No matter how much memory space an object takes up, it only takes up a single stack slot. In a real Forth, data is stored on the stack as raw binary strings, characters, or numbers and the programmer is responsble for keeping track of how much memory every item actually uses.

stack functions

    The double number versions of Forth stack functions are purposely left out. Our stack elements are variable size objects and can be any size number, as well as strings or characters.

ShowDataStack

    The ShowDataStack function shows the current contents of the data stack in the order of top of stack to bottom of stack. This function is built for debugging purposes, but I am making it available for anyone to use.

    $ ShowDataStack ()
    >{
    >    echo "stack size of " $DataStackPointer;
    >    if [ "$DataStackPointer" -gt "0" ] ; then
    >        loopcounter=$DataStackPointer
    >        while [ $loopcounter -gt 0 ] ;
    >        do
    >            echo ${DataStack["$loopcounter"]}
    >            let "loopcounter=$loopcounter - 1"
    >        done
    >    else
    >        echo "the stack is empty"
    >    fi
    >    unset loopcounter #clear out variable
    >} # end ShowDataStack
    $

push

    The push function pushes a single object onto the data stack. Forth doesn’t have this word because simply typing a number is sufficient to push the number onto the stack. This function will be used by other functions to manipulate the data stack.

    NOTE: This function does not yet have error checking. Need to check for exactly one item to push.

    $ push ()
    >{
    >    let "DataStackPointer=$DataStackPointer + 1"; #preincrement
    >    DataStack["$DataStackPointer"]="$1"; #push data item on top of stack
    >} # end push
    $

    It is very important to notice that the item to be pushed onto the stack comes after the Forth-like function push, which is very different than the normal order for Forth. This variation appears again in other Forth-like functions and is done to work within the limits of the expectations of the BASH shell.

    Note that push will fail inside a series of piped commands or any other situation with a subshell.

pop

    The pop function pops a single object from the data stack. Forth has a different function for this functionality, but this function is intended for internal use by other functions to manipulate the data stack.

    NOTE: This function does not yet have error checking. Need to check for empty stack.

    $ pop ()
    >{
    >    let "DataStackPointer=$DataStackPointer - 1" #postdecrement
    >} # end pop
    $

popout

    The popout function pops a single object from the data stack and prints it out for use in a pipe. Forth has a different function for this functionality, but this function is intended for internal use by other functions to manipulate the data stack.

    Note that the value is printed without a new line character, so it might seem to “disappear” into the beginning of the prompt.

    Use the DOT function to pop and print with a new line.

    NOTE: This function does not yet have error checking. Need to check for empty stack.

    $ popout ()
    >{
    >    printf ${DataStack["$DataStackPointer"]} #pop data item from top of stack and print
    >    let "DataStackPointer=$DataStackPointer - 1" #postdecrement
    >} # end popout
    $

the power of Forth-like functions

    For those who don’t already know the power of Forth, let me use the shell commands pushd, popd, and dirs as an example. For those who already know the power of Forth, you can either read along and nod about how great Forth is, or skip ahead to actual implementation of Forth-like functions for the BASH shell.

directory stack

    You may have noticed that the C shell and BASH include the pushd, popd, and dirs command to manipulate a stack of directories.

    In case you never noticed these commands or have forgotten what they do, a quick summary:

    pushd allows the user or a script to push the current working directory onto a stack of directories and change to a new named directory. The command acts slightly differently if you leave out a directory name. We won’t discuss that case because it takes us on a tangent from our current discussion.

    You can simulate the normal use of the shell command pushd (named directory) with the Forth-like function push and shell command cd.

    $ push `pwd`; cd testdir
    $

    You have to explicitly place the current working directory onto the stack (using pwd and backticks) and you have to issue a second command to actually change the directory. Don’t worry about the extra steps, because you will continue to use pushd for its intended purpose. We are showing the value of Forth through analogy to something you already use to save time and effort.

    popd allows the user or a script to pop the directory stack and change to that directory.

    You can simulate the normal use of the shell command popd with the Forth-like function pop and shell command cd.

    $ cd `pop`
    $

    Notice that we used the results of the pop function as the parameter for the shell command cd.

    dirs allows the user or a script to see all of the directories on the directory stack.

    You can simulate the normal use of the shell command dirs with the Forth-like function ShowDataStack.

    $ ShowDataStack
    stack size of  1
    /Users/admin
    $

taking it to the next level

    At this point, there is no advantage to using our Forth-like functions rather than using the already existing shell commands. It is a good idea to continue to use the shell commands for their intended use. Less typing.

    You could also use named variables for saving directory names. By using short variable names, you can save a lot of typing over typing out the full path names. And you don’t have to memorize a bunch of long path names. Of course, you have to memorize all of your named variables.

    The shell commands pushd, popd, and dirs were developed because there are a lot of advantages of a stack over a bunch of named variables.

    With a stack, you don’t have to memorize any alternate short variable names for directory paths. And you can use the same directory stack for work that moves from an old set of directories to a new set of directories.

    There is a lot of power in having a stack and three simple commands for manipulating the stack.

more sophisticated directory stack use

    The shell commands pushd and popd only work with the single directory that is at the top of the stack. Yes, you can see the entire stack with the dirs command and use one of several alternate techniques for picking one directory out of the middle of the stack.

    What if you could easily swap the directory that was on the top of the stack with the directory that was one down? You could then use the popd command to switch to the directory that was formerly one down from the top of the stack and the former top of the stack would still be in the stack, ready for use.

    As it turns out, Forth has a command called SWAP that does exactly that operation. Forth has a rich set of stack operation words, that can do things like move (or pick) an aribitrary item from deep in the stack to the top of the stack, make a duplicate (or dup) of the top of the stack, delete (or drop) the item on the top of the stack, and other cool things.

    The Forth-like functions bring variations of these stack operation words to the shell.

arithmetic computations

    Both Forth and the Forth-like functions work on much more than just directories.

    A common use of Forth is computations.

    Many of the scientific and engineering calculators (which used to be stand-alone devices, but are now commonly apps) use a system called Reverse Polish Notation or RPN

    An RPN calculator is a very powerful and efficient way of performing calculations.

    For ordinary calculations, such as adding a series of numbers, RPN only saves a single keystroke. That’s not enough to justify learning an entirely different method of calculating.

    But when the calculations start to become complicated, RPN really shines.

    With standard algebraic notation, you suddenly have to start typing a whole bunch of open and close parenthesis and figure out all of your grouping and keep track of how many layers of parenthesis are currently still open so that at the end of your calculation all of the open and close parenthesis balance out. Anyone who has done any C programming knows about this hassle.

    With RPN, there are no parenthesis. Once you learn to think in RPN, any arbitrarily complex algebraic computation has a straight-forward and simple RPN expression!

    You dramatically save typing, you avoid the headache of balancing nested parenthesis, and you have confidence in the correctness of your straight-forward expression.

    Well, both Forth and the Forth-like functions use the RPN system for computing.

    You bring the power of scientific and engineering calculators to the BASH shell. Well, actually, you will need to write a few additional functions of your own to have all of the features of your favorite scientific or engineering calculator, but you have all of the tools available to finish out that task.

    This means that you can use RPN calculations in all of your shell scripts.

strings

    Both Forth and the Forth-like functions can also do manipulations of characters and character strings.

    The Forth-like functions store all information as character strings, even numbers. This is different than real Forth, but has its own advantages and disadvantages.

    BASH and all of the other shells work with streams of characters. Storing everything (even numbers) on a special string-oriented stack allows our Forth-like functions to play nice with the BASH shell.

    This allows us to combine all of the power of the BASH shell with much of the power of Forth.

the power of Forth

    The power of Forth lies in three characteristics:

    (1) Forth has powerful stack-based processing. The Forth-like functions bring this power to the BASH shell in a manner compatible with BASH’s expectations.

    (2) Forth is extensible. You can build new Forth-like functions by threading together other Forth-like functions. You won’t have quite the flexibility of real Forth and you won’t have the Forth advantage of all new Forth words being treated the same as all built-in threaded words. But with modern processors, you will be pretty close to the power of real Forth.

    (3) Forth can run on its own without an operating system. This one can’t be done with the BASH shell. I do intend to eventually write the code to combine a stripped down shell and Forth as native code for use in embedded systems. The Forth-like functions are intended as a demo to prove that adding Forth to the shell is a dramatic improvement in both power and flexiibilty.

    Once you learn how to use the Forth-like functions in your shell scripts (and even at the interactive command line), you will gain a huge amount of added power and possibilities for your shell scripts.

    The one big hassle is that you can’t simply import existing Forth programs, which is unfortunate given the huge library of very useful open source and public domain Forth programs.

    I do intend to write a translator, which will convert existing Forth programs into their BASH shell Forth-like equivalent.

    With that introduction to the power of Forth and Forth-like functions, let’s take a look at the actual Forth-like functions.

and now, the functions

dot

    The dot function pops a single object from the top of the data stack and echos the result. Forth uses the period ( . ) for this word, but the period is already used for a different BASH command, creating a namespace conflict. Hence, the name dot.

    NOTE: This function does not yet have error checking. Need to check for empty stack.

    $ dot ()
    >{
    >    echo ${DataStack["$DataStackPointer"]} #pop data item from top of stack and print
    >    let "DataStackPointer=$DataStackPointer - 1" #postdecrement
    >} # end dot
    $

DUP

    The DUP function is nearly the same functionality as the Forth word DUP (the difference is that this function works with character strings). The top item on the stack is duplicated, increasing the size of the stack by one.

    NOTE: This function does not yet have error checking. Need to check for empty stack.

    $ DUP ()
    >{
    >    temporary=${DataStack["$DataStackPointer"]} #make temporary copy of data item from top of stack
    >    let "DataStackPointer=$DataStackPointer + 1" #preincrement
    >    DataStack["$DataStackPointer"]=$temporary #store duplicate
    >    unset temporary #clear out variable
    >} # end DUP
    $

qDUP

    The qDUP function is nearly the same functionality as the Forth word ?DUP (the difference is that this function works with character strings). The top item on the stack is duplicated only if it is non-zero, increasing the size of the stack by one if the item on the top of the stack is non-zero.

    NOTE: This function does not yet have error checking. Need to check for empty stack and non-number object on top of stack.

    $ qDUP ()
    >{
    >    if [ ${DataStack["$DataStackPointer"]} -eq 0 ] ; then #test for zero string on top of stack
    >        return 0 #if zero, take no action, return OK
    >    fi
    >    temporary=${DataStack["$DataStackPointer"]} #make temporary copy of data item from top of stack
    >    let "DataStackPointer=$DataStackPointer + 1" #preincrement
    >    DataStack["$DataStackPointer"]=$temporary #store duplicate
    >    unset temporary #clear out variable
    >} # end qDUP
    $

OVER

    The OVER function is nearly the same functionality as the Forth word OVER (the difference is that this function works with character strings). The second item down from the top of the stack is duplicated, increasing the size of the stack by one.

    This function uses all caps, OVER, to prevent conflict with the already existing over UNIX tool for pretty printing and scrolling source code on a terminal.

    NOTE: This function does not yet have error checking. Need to check for empty stack or stack with only one item in it.

    $ OVER ()
    >{
    >    let "TemporaryPointer=$DataStackPointer - 1" #create pointer down one from top of stack
    >    temporary=${DataStack["$TemporaryPointer"]} #make temporary copy of data item one down in stack
    >    let "DataStackPointer=$DataStackPointer + 1" #preincrement
    >    DataStack["$DataStackPointer"]=$temporary #store duplicate
    >    unset temporary #clear out variable
    >    unset TemporaryPointer #clear out variable
    >} # end OVER
    $

PICK

    The PICK function is nearly the same functionality as the Forth word PICK (the difference is that this function works with character strings). The n^th item down from the top of the stack is duplicated, increasing the size of the stack by one.

    NOTE: This function does not yet have error checking. Need to check that a number was entered. Need to check for stack that is deep enough.

    $ PICK ()
    >{
    >    let "TemporaryPointer=$DataStackPointer - $1" #create pointer down one from top of stack
    >    temporary=${DataStack["$TemporaryPointer"]} #make temporary copy of data item one down in stack
    >    let "DataStackPointer=$DataStackPointer + 1" #preincrement
    >    DataStack["$DataStackPointer"]=$temporary #store duplicate
    >    unset temporary #clear out variable
    >    unset TemporaryPointer #clear out variable
    >} # end PICK
    $

    The PICK function and the Forth word PICK should rarely be used. If you find yourself using this word often, then you should refactor your code to be more simple.

DROP

    The DROP function is nearly the same functionality as the Forth word DROP (the difference is that this function works with character strings). Pops a single object from the top of the data stack.

    NOTE: This function does not yet have error checking. Need to check for empty stack.

    $ DROP ()
    >{
    >    let "DataStackPointer=$DataStackPointer - 1" #postdecrement
    >} # end DROP
    $

SWAP

    The SWAP function is nearly the same functionality as the Forth word SWAP (the difference is that this function works with character strings). The item on the top of the stack is swapped with the item one down from the top of the stack.

    NOTE: This function does not yet have error checking. Need to check that the stack has at least two items.

    $ SWAP ()
    >{
    >    let "TemporaryPointer=$DataStackPointer - 1" #create pointer down one from top of stack
    >    temporary=${DataStack["$TemporaryPointer"]} #make temporary copy of data item one down in stack
    >    DataStack["$TemporaryPointer"]=${DataStack["$DataStackPointer"]} #move the top item to 2nd location down
    >    DataStack["$DataStackPointer"]=$temporary #put former 2nd down on top of stack
    >    unset temporary #clear out variable
    >    unset TemporaryPointer #clear out variable
    >} # end SWAP
    $

ROT

    The ROT function is nearly the same functionality as the Forth word ROT (the difference is that this function works with character strings). Rotate third item to top of stack.

    NOTE: This function does not yet have error checking. Need to check that the stack has at least three items.

    $ ROT ()
    >{
    >    let "TemporaryPointer=$DataStackPointer - 2" #create pointer down two from top of stack
    >    let "SecondPointer=$DataStackPointer - 1" #create pointer down one from top of stack
    >    temporary=${DataStack["$TemporaryPointer"]} #make temporary copy of data item one down in stack
    >    DataStack["$TemporaryPointer"]=${DataStack["$SecondPointer"]} #move the 2nd down item to 3rd location down
    >    DataStack["$SecondPointer"]=${DataStack["$DataStackPointer"]} #move the top item to 2nd location down
    >    DataStack["$DataStackPointer"]=$temporary #put former 3rd down on top of stack
    >    unset temporary #clear out variable
    >    unset TemporaryPointer #clear out variable
    >    unset SecondPointer #clear out variable
    >} # end ROT
    $

ROLL

    The ROLL function is nearly the same functionality as the Forth word ROLL (the difference is that this function works with character strings). Rotates the nth object to the top of the stack.

    NOTE: This function does not yet have error checking. Need to check for no user input.

    $ ROLL ()
    >{
    >    if [ $DataStackPointer -gt "$1" ] ; then #check to make sure enough items on stack
    >        let "DestinationPointer=$DataStackPointer - $1"
    >        SavedItem=${DataStack["$DestinationPointer"]} #save old item
    >        let "SourcePointer =$DestinationPointer + 1"
    >        LoopCounter=$1
    >        while [ $LoopCounter -gt 0 ] ; #move data loop
    >        do
    >            DataStack["$DestinationPointer"]=${DataStack["$SourcePointer"]} #move data
    >            let "DestinationPointer=$DestinationPointer + 1" #post increment
    >            let "SourcePointer=$SourcePointer + 1" #post increment
    >            let "LoopCounter=$LoopCounter - 1"
    >        done
    >        DataStack["$DataStackPointer"]=$SavedItem
    >        unset LoopCounter #clear out variable
    >        unset SavedItem #clear out variable
    >        unset SourcePointer #clear out variable
    >        unset DestinationPointer #clear out variable
    >    fi # end if good input
    >} # end ROLL
    $

    The ROLL function and the Forth word ROLL should rarely be used. If you find yourself using this word often, then you should refactor your code to be more simple.

DEPTH

    The DEPTH function is nearly the same functionality as the Forth word DEPTH (the difference is that this function works with character strings). Find existing size of stack and then put that number on top of the stack.

    NOTE: This function does not yet have error checking.

    $ DEPTH ()
    >{
    >    temporary=$DataStackPointer #save the existing depth of the stack
    >    let "DataStackPointer=$DataStackPointer + 1" #preincrement
    >    DataStack["$DataStackPointer"]=$temporary #push depth on top of stack
    >} # end DEPTH
    $

archive copy

    You can obtain the latest copy of these functions at http://www.osdata.com/programming/shell/forthfunctions place the file in an appropriate directory.

---