Functions in shared libraries

Introduction

VFX Forth supports calling external API calls in dynamic link libraries (DLLs) for Windows and shared libraries in MacOS, Linux and other Unix-derived operating systems. Various API libraries export functions in a variety of methods mostly transparent to programmers in languages such as C, Pascal and Fortran.

Floating point data is converted to use SSE2 instructions on the operating system side. On the VFX Forth side FPSYSTEM identifies the type of floating point package in use.

Before a library function can be used, the library itself must be declared, e.g.


LIBRARY: /usr/lib/libSystem.B.dylib

The default calling convention is nearly always applicable. The following example shows that definitions can occupy more than one line. It also indicates that some token separation may be necessary for pointers:


Library: libc.so.6

Extern: int execve(
  const char * path,
  char * const argv[],
  char * const envp[]
);

This produces a Forth word execve.

  execve       \ path argv envp -- int

The parser used to separate the tokens is not ideal. If you have problems with a definition, make sure that * tokens are white-space separated. Formal parameter names, e.g. argv above are ignored. Array indicators, [] above, are also ignored when part of the names.

The input types may be followed by a dummy name which is discarded. Everything on the source line after the closing ')' is discarded.

The default for the Linux\ and OS X versions is "C" using the SystemV ABI. The default is always used unless overridden in the declaration.

Format


EXTERN: <return> [ <callconv> ] <name> '(' <arglist> ')' ';'

<return>    := { <type> [ '*' ] |  void }
<arg>       := { <type> [ '*' ] [ <name> ] }
<args>      := { [ <arg>, ]* <arg> }
<arglist>   := { <args> | void }        Note: "void, void" etc. is illegal.
<callconv>  := { PASCAL | WINAPI | STDCALL | "PASCAL" | "C" }
<name>      := <any Forth acceptable namestring>
<type>      := ... (see below, "void" is a valid type)

Note that during searches <name> is passed to the operating system exactly as it is written, i.e. case sensitive. The Forth name is case-insensitive.

As a standard Forth's string length for dictionary names is only guaranteed up to 31 characters for portable source code, very long API names can cause problems. Therefore the word AliasedExtern: allows separate specification of API and Forth names (see below). AliasedExtern: also solves problems when API functions only differ in case or their names conflict with existing Forth word names.

Calling Conventions

In the discussion caller refers to the Forth system (below the application layer and callee refers to a a function in a DLL or shared library. The EXTERN: mechanism supports three calling conventions.

Unless otherwise specified, the Forth system's default convention is used. Under Windows this is WINAPI and under Linux and other Unices it is "C".

Floating point data is converted to and from SSE2 instructions for use in the operating system.

Promotion and Demotion

The system generates code to either promote or demote non-CELL sized arguments and return results which can be either signed or unsigned. Although Forth is an un-typed language it must deal with libraries which do have typed calling conventions. In general the use of non-CELL arguments should be avoided but return results should be declared in Forth with the same size as the C or PASCAL convention documented.

Argument Reversal

The default calling convention for the host operating system is used. The right-most argument/parameter in the C-style prototype is on the top of the Forth data stack. When calling an external function the parameters are reordered as required by the operating system; this is to enable the argument list to read left to right in Forth source as well as in the C-style operating system documentation.

Under certain conditions, the order can be reversed. See the words "C" and "PASCAL" which define the order for the operating system. See L>R and R>L which define the Forth stack order with respect to the arguments in the prototype.

C comments in declarations

Very rudimentary support for C comments in declarations is provided, but it is good enough for the vast majority of declarations.

The example below is taken from a SQLite interface.


Extern: "C" int sqlite3_open16(
  const void * filename,  /* Database filename [UTF-16] */
  sqlite3 ** ppDb         /* OUT: SQLite db handle */
);

Controlling external references

1 value ExternWarnings? \ -- n
Set this true to get warning messages when an external reference is redefined.

0 value ExternRedefs?   \ -- n
If non-zero, redefinitions of existing imports are permitted. Zero is the default for VFX Forth so that redefinitions of existing imports are silently ignored.

1 value LibRedefs?      \ -- n
If non-zero, redefinitions of existing libraries are permitted. Non-zero is the default for VFX Forth so that redefinitions of existing libraries and OS X frameworks are permitted. When set to zero, redefinitions are silently ignored.

1 value InExternals?    \ -- n
Set this true if following import definitions are to be in the EXTERNALS vocabulary, false if they are to go into the wordlist specified in CURRENT. Non-Zero is the default for VFX Forth.

: InExternals   \ --
External imports are created in the EXTERNALS vocabulary.

: InCurrent     \ --
External imports are created in the wordlist specified by CURRENT.

Library Imports

In VFX Forth, libraries are held in the EXTERNALS vocabulary, which is part of the minimum search order. Other Forth systems may use FORTH vocabulary or the CURRENT wordlist.

For turnkey applications, initialisation, release and reload of required libraries is handled at start up.

variable lib-link       \ -- addr
Anchors the chain of dynamic/shared libraries.

variable lib-mask       \ -- addr
If non-zero, this value is used as the mode for dlopen() calls in Linux and OS X.

struct /libstr  \ -- size
The structure used by a Library: definition.

  int >liblink          \ link to previous library
  int >libaddr          \ library Id/handle/address, depends on O/S
  int >libmask          \ mask for dlopen()
  0 field >libname      \ zero terminated string of library name
end-struct

struct /funcstr \ -- size
The structure used by an imported function.

: init-lib      \ libstr --
Given the address of a library structure, load the library.

: clear-lib     \ libstr --
Unload the given library and zero its load address.

: clear-libs    \ --
Clear all library addresses.

: init-libs     \ --
Release and reload the required libraries.

: find-libfunction      \ z-addr -- address|0
Given a zero terminated function name, attempt to find the function somewhere within the already active libraries.

: .Libs         \ --
Display the list of declared libraries.

: #BadLibs      \ -- u
Return the number of declared libraries that have not yet been loaded.

: .BadLibs         \ --
Display a list of declared libraries that have not yet been loaded.

: Library:      \ "<name>" -- ; -- loadaddr|0
Register a new library by name. If LibRedefs? is set to zero, redefinitions are silently ignored. Use in the form:

  LIBRARY: <name>

Executing <name> later will return its load address. This is useful when checking for libraries that may not be present. After definition, the library is the first one searched by import declarations.

: topLib        \ libstr --
Make the library structure the top/first in the library search order.

: firstLib      \ "<name>" --
Make the library first in the library search order. Use during interpretation in the form:

  FirstLib <name>

to make the library first in the search order. This is useful when you know that there may be several functions of the same name in different libraries.

: [firstLib]    \ "<name>" --
Make the library first in the library search order. Use during compilation in the form:

  [firstLib] <name>

to make the library first in the search order. This is useful when you know that there may be several functions of the same name in different libraries.

Mac OS X extensions

The phrase Framework <name.framework> creates two Forth words, one for the library access, the other to make that library top in the search order. For example:

  framework Cocoa.framework

produces two words

  Cocoa.framework/Cocoa
  Cocoa.framework

The first word is the library definition itself, which behaves in the normal VFX Forth way, returning its load address or zero if not loaded. The second word forces the library to be top/first in the library search order. Thanks to Roelf Toxopeus.

As of OSX 10.7, FRAMEWORK (actually dlopen()) will search for frameworks in all the default Frameworks directories:

: framework     \ --
Build the two framework words. See above for more details. If LibRedefs? is set to zero, redefinitions are silently ignored.

Function Imports

Function declarations in shared libraries are compiled into the EXTERNALS vocabulary. They form a single linked list. When a new function is declared, the list of previously declared libraries is scanned to find the function. If the function has already been declared, the new definition is ignored if ExternRedefs? is set to zero. Otherwise, the new definition overrides the old one as is usual in Forth.

In VFX Forth, ExternRedefs? is zero by default.

variable import-func-link       \ -- addr
Anchors the chain of imported functions in shared libraries.

: ExternLinked  \ c-addr u -- address|0
Given a string, attempt to find the named function in the already active libraries. Returns zero when the function is not found.

: init-imports  \ --
Initialise Import libraries. INIT-IMPORTS is called by the system cold chain.

: +DebugExterns \ --
Causes EXTERN: and friends to issue debugging information.

: -DebugExterns \ --
Stops EXTERN: and friends issuing debugging information.

The parameter passing depends on the installed floating point package.

0 value FpSystem
The value FPSYSTEM defines which floating point pack is installed and active. Each floating point pack defines its own type as follows:

: InExternals   \ --
External imports are created in the EXTERNALS vocabulary.

: InCurrent     \ --
External imports are created in the wordlist specified by CURRENT.

: Extern:       \ "text" --
Declare an external API reference. See the syntax above. The Forth word has the same name as the function in the library, but the Forth word name is not case-sensitive. The length of the function's name may not be longer than a Forth word name. For example:

  Extern: DWORD Pascal GetLastError( void );

: AliasedExtern:        \ "forthname" "text" --
Like EXTERN: but the declared external API reference is called by the explicitly specified forthname. The Forth word name follows and then the API name. Used to avoid name conflicts, e.g.

AliasedExtern: saccept int accept( HANDLE, void *, unsigned int *);

which references the Winsock accept function but gives it the Forth name SACCEPT. Note that here we use the fact that formal parameter names are optional.

: LocalExtern:  \ "forthname" "text" --
As AliasedExtern:, but the import is always built into the CURRENT wordlist.

: extern        \ "text" --
An alias for EXTERN:.

: ExternVar     \ "<name>" -- ; ExternVar <name>
Used in the form

  ExternVar <name>

to find a variable in a DLL or shared library. When executed, <name> returns its address.

: AliasedExternVar      \ "<forthname>" "<dllname>" --
Used in the form

  AliasedExternnVar <forthname> <varname>

to find a variable in a DLL or shared library. When executed, <forthname> returns its address.

: .Externs      \ -- ; display EXTERNs
Display a list of the external API calls.

: #BadExterns   \ -- u
Silently return the number of unresolved external API calls.

: .BadExterns   \ --
Display a list of any external API calls that have not been resolved.

: func-pointer  \ xt -- addr
Given the XT of a word defined by EXTERN: or friends, returns the address that contains the run-time address.

: func-loaded?  \ xt -- addr|0
Given the XT of a word defined by EXTERN: or friends, returns the address of the DLL function in the DLL, or 0 if the function has not been loaded/imported yet.

Pre-Defined parameter types

The types known by the system are all found in the vocabulary TYPES. You can add new ones at will. Each TYPE definition modifies one or more of the following VALUEs. )

argSIZE

Size in bytes of data type.

argDEFSIGN

Default sign of data type if no override is supplied.

argREQSIGN

Sign OverRide. This and the previous use 0 = unsigned and 1 = signed.

argISPOINTER

1 if type is a pointer, 0 otherwise

Each TYPES definition can either set these flags directly or can be made up of existing types.

Note that you should explicitly specify a calling convention for every function defined.

Calling conventions

: "C"           \ --
Set Calling convention to "C" standard. Arguments are reversed, and the caller cleans up the stack.

: "PASCAL"      \ --
Set the calling convention to the "PASCAL" standard as used by Pascal compilers. Arguments are not reversed, and the called routine cleans up the stack.

: WinApi        \ --
A synonym for PASCAL.

: R>L   \ --
By default, arguments are assumed to be on the Forth stack with the top item matching the rightmost argument in the declaration so that the Forth parameter order matches that in the C-style declaration. R>L reverses this.

: L>R   \ --
By default, arguments are assumed to be on the Forth stack with the top item matching the rightmost argument in the declaration so that the Forth parameter order matches that in the C-style declaration. L>R confirms this.

Basic Types

: unsigned      \ --
Request current parameter as being unsigned.

: signed        \ --
Request current parameter as being signed.

: int           \ --
Declare parameter as integer. This is a signed 32 bit quantity unless preceeded by unsigned. Note that the int used inside an EXTERN: declaration is not the same as an int in a Forth structure definition.

: char          \ --
Declare parameter as character. This is a signed 8 bit quantity unless preceeded by unsigned.

: void          \ --
Declare parameter as void. A VOID parameter has no size. It is used to declare an empty parameter list, a null return type or is combined with * to indicate a generic pointer.

: *             \ --
Mark current parameter as a pointer.

: **            \ --
Mark current parameter as a pointer.

: ***           \ --
Mark current parameter as a pointer.

: const ;       \ --
Marks next item as constant in C terminology. Ignored by VFX Forth.

: int32         \ --
A 32bit signed quantity.

: int16         \ --
A 16 bit signed quantity.

: int8          \ --
An 8 bit signed quantity.

: uint32        \ --
32bit unsigned quantity.

: uint16        \ --
16bit unsigned quantity.

: uint8         \ --
8bit unsigned quantity.

: Long          \ --
A 64 bit signed or unsigned integer. At run-time, the argument is taken from the Forth data stack as a normal Forth single

: LongLong      Long  ;
A 64 bit signed quantity. See Long above.

: SHORT         \ --
For most compilers a short is a 16 bit signed item, unless preceded by unsigned.

: BYTE          \ --
An 8 bit unsigned quantity.

: float         \ --
32 bit float.

: double        \ --
64 bit float.

: point_double  \ --
MPEism for points using double floats. FOR OUTPUTS ONLY.

: complex_double        \ --
MPEism for complex numbers using double floats. FOR OUTPUTS ONLY.

: bool1         \ --
One byte boolean.

: bool4         \ --
Four byte boolean.

: bool8         \ --
Eight byte boolean.

: ...           \ --
The parameter list is of unknown size. This is an indicator for a C varargs call. Run-time support for this varies between operating system implementations of VFX Forth. Test, test, test.

Linux Types

: OSCALL        "C"  ;
Used for portable code to avoid three sets of declarations. For Windows, this is a synonym for PASCAL and under Linux this is a synonym for "C".

: FILE          uint32  ;
Always use as FILE * stream.

: DIR           uint32  ;
Always use as DIR * stream.

: size_t        Long  ;
Linux type for unsigned INT.

: off_t         Long  ;
Linux type for unsigned INT.

: int32_t       int32  ;
Synonym for int32.

: int16_t       int16  ;
Synonym for int16.

: int8_t        int8  ;
Synonym for int8.

: uint32_t      uint32  ;
Synonym for uint32.

: uint16_t      uint16  ;
Synonym for uint16.

: uint8_t       uint8  ;
Synonym for uint8.

: time_t        Long  ;
Number of seconds since midnight UTC of January 1, 1970.

: clock_t       Long  ;
Processor time in terms of CLOCKS_PER_SEC.

: pid_t         unsigned int  ;
Process ID.

: uid_t         unsigned int  ;
User ID.

: mode_t        unsigned short  ;
File mode.

: pthread_t     void *  ;
Thread.

Mac OS X Types

: OSCALL        "C"  ;
Used for portable code to avoid three sets of declarations. For Windows, this is a synonym for PASCAL and under OS X this is a synonym for "C".

: FILE          uint32  ;
Always use as FILE * stream.

: DIR           uint32  ;
Always use as DIR * stream.

: size_t        Long  ;
OS/X type for file size.

: off_t         Long  ;
OS/X type for file position.

: int32_t       int32  ;
Synonym for int32.

: int16_t       int16  ;
Synonym for int16.

: int8_t        int8  ;
Synonym for int8.

: uint32_t      uint32  ;
Synonym for uint32.

: uint16_t      uint16  ;
Synonym for uint16.

: uint8_t       uint8  ;
Synonym for uint8.

: time_t        Long  ;
Number of seconds since midnight UTC of January 1, 1970.

: clock_t       Long  ;
Processor time in terms of CLOCKS_PER_SEC.

: pid_t         unsigned int  ;
Process ID.

: uid_t         unsigned int  ;
User ID.

: mode_t        unsigned short  ;
File mode.

: pthread_t     void *  ;
Thread.

Compatibility words

These words are mainly for users converting code from other Forth systems. They provide a more "Forth-like" interface that does not permit copy/paste from the operating system header files or documentation.

Compatibility layers using two stacks, e.g.

  function: foop ( inta intb intc -- ) ( doublea doubleb doublec -- )

will provide a correct call for calls with six or fewer integer arguments and eight or fewer float arguments (admittedly the majority of cases).

The following section provides shared library imports in the form:

  function: foo  ( a b c d -- x ) ( F: r1 r2 -- rx )

where the brackets must be space delimited. The first argument list represents the arguments seen by Forth as integers. The second arguments list is optional, and is used when F.P. doubles are involved. Imports use the default calling convention for the operating system.

: FUNCTION:     \ "<name>" "<parameter list>" --
Generate a reference to an external function. The Forth name is the same as the name of the external function. The first parameter list is a Forth integers list plus an optional second Forth doubles list. There should always be an integer list, the first list, even when not used. Use in the form:

 function: foo1 ( a b c d -- )
 function: foo2 ( a b c d -- e )
 function: foo3 ( a b c d -- e ) ( r1 r2 r3 -- )
 function: foo4 ( a b c d -- ) ( r1 r2 r3 -- r4 )
 function: foo5 ( a b c d -- ) ( r1 r2 r3 -- r4 r5 )
 function: foo6 ( -- ) ( r1 r2 r3 -- r4 )
 function: foo7 ( -- ) ( r1 r2 r3 -- r4 r5 )
 function: foo7 ( -- a ) ( r1 r2 r3 -- )

The inclusion of Forth stack indicators like S: F: N: R: is optional.

They will be ignored while parsing and are only informative on which

Forth stack you must place the parameters.

The returned value may be 0, 1 integer or 0, 1, 2 doubles corresponding to void, int/long and 64 bit double

: ASCALL:       \ "<synonym-name>" "<name>" "<parameter list>" --
Generate a reference to an external function. The Forth name is not the same as the name of the external function. Use in the form:

 ascall: forthname funcname ( a b c d -- e )

: GLOBAL:       \ "<name>" --
Generate a reference to an external variable. Use in the form:

 global: varname