The Billboard on the Nullarbor OxIDE

Welcome

Welcome to The Billboard on the Nullarbor. This site is for developers with existing code written in the xbase family of languages, specifically Xbase++, Clipper and (x)Harbour, who might wish to translate that code to other languages and platforms.

The OxIDE Transpiler

OxIDE is a tool to assist developers to easily and incrementally migrate their Xbase++ and/or Clipper applications into the .NET v2 environment, by translating Xbase++/Clipper code into IronPython, C# or JavaScript. OxIDE is free and open-source software.

OxIDE contains a preprocessor and parser for the Xbase++/Clipper language which converts input (.prg) files to XML representations of the parse result. It also contains an integrated XSLT processor (a wrapper around the excellent NXSLT2 product, for transforming the XML parse result to other languages, documentation extraction etc. Included are XSLT scripts for converting to IronPython, JavaScript, Csharp and Gsharp.

The product is also provided as a library so that translated scripts are able to interpret (parse and translate) Xbase++ codeblocks at runtime (they may be stored inside data tables, or XML files for example).

The main goals of the project, in order of priority, are.

• That the result of the translation(s) be rigorously correct in producing application behaviour as near as possible to identical that of the original source.
• That the result of the translation should as far as possible be readable, usable, modifiable code, and not overly obfuscated code that has obviously been "shoe-horned" into the target language.
• That an absolute minimum of modifications need be made to the Xbase++/Clipper source in order for the translation to be successful, and those few modifications that are required should also improve the overall quality of the source code.

It should be apparent to the attentive reader that the above goals are somewhat antithetical to each other - particularly the first two. How successful I have been in meeting and reconciling the above goals I leave for others to judge.

Downloads

Full source code can be downloaded from here. It has been developed using Visual C# Express 2005, and the project files are included. Compiled executables are included, as are all the XSLT scripts and the runtime support IronPython scripts.

Alternatively, you can download just the executables, the XSLT scripts and the runtime support from here.

Please visit the IronPython website to download your copy of IronPython.

In order to run any of the above you will need to have the .NET v2.0 Runtime installed.

Licensing

The OxIDE product is free and open-source software. No registration is required for download.

All files, including C#, XSLT and runtime support programs/scripts, are released under by a standard BSD style license. Note that this license will have absolutely no effect on any generated (translated) code produced by the tool.

Support

Ad-hoc support is available for the price of $AU 120.00 per incident.

One year's pre-paid support, is available for the price of $AU 500.00, with no limit on the number of incidents.

Support services can be paid-for here.

Differences between OxIDE and Xbase++

In order to be able to generate valid useful IronPython code, the input language the Transpiler accepts has some differences to Xbase++/Clipper. "OxIDE" is the name I give to the actual language that the Transpiler accepts.

The following differences arise from the need to only accept constructs directly supportable in .NET, or at least those which could be effectively rendered into the .NET world without production of unreadable gobbledygook that mortal programmers could not understand.

OxIDE:

• has no implicit main procedure option (/n)
• has no SET PROCEDURE TO support (/m)
• requires all variables to be declared
• does not support PRIVATE variables (publics are ok though)
• requires pass-by-reference parameters to be explicitly declared in the routine definition
  • callers still get to choose whether to call by reference or not
  • In OxIDE source, this is achieved by the prepending of a "@" character to the formal parameter name. To do this to Xbase++/Clipper code would render it invalid, so the declaration is made by modifying the relevant project.xml element to reflect the possible call-by-reference usage.
• is implemented as a translator to IronPython (with C# and JavaScript planned to follow)
  • translation is in two stages
    • first stage is achieved by having the "OxIDE Parser" product translate an Xbase++/Clipper program to an XML representation of the parse result (.xst file)
    • The "OxIDE Parser" product is open-source software, under the BSD license and makes use of the "Jay" parser generator
    • includes a full implementation (in C#) of the Xbase++ preprocessor
    • does not involve the usage (or ownership) of any external Xbase++ or Clipper compiler or preprocessor products.
    • requires .NET v2 (or better) runtime
    • comments in source files (including copyright notices etc.) are preserved
  • second stage is to translate the XML parse tree (.xst) into a particular target language
    • these products are referred to as "OxIDE Translator"s
    • they (currently) use XSLT scripts (and the free NXSLT2 product) to perform the translations
    • comments (including copyright notices etc.) are preserved
    • current Translators are focused on .NET languages (but if someone wants to pay me to write one for Java ... ;-)
    • Translators are free and open-source under a BSD style license, and can be easily and freely modified to suit individual requirements or even used as starting points for implementing your own Translators for other target languages/environments.
    • currently three Translators are included - for OxIDE, Xbase++ and IronPython. Of these the only one that does anything remotely useful is the IronPython one.

Targets

At the moment, XSLT transforms are provided for four different target languages:

• IronPython
• JavaScript
• Csharp
• Gsharp

Some of the above also come with runtime support libraries. The specifics of each output target are addressed below

IronPython

The IronPython Translator is fully functional in the sense that it always generates valid IronPython source code, but the runtime support to run said code is still incomplete. At the moment, the general runtime library is well advanced, but the database layer stuff, which attempts to map ADO.NET to the Xbase++/Clipper runtime database interface is somewhat rudimentary, and the user-interface layer is virtually non-existent. Hopefully, both these situations will be remedied in the near future.

The translator generates generic stubs for routines it is unable to resolve, and issues warnings that it is doing so. In order to make a working system, the user will have to resolve all missing symbols in one way or another. (See below for facilities OxIDE supports to achieve this aim.)

For anyone interested in just perusing the results of the translation process, all the Xbase++ source code archives available on this site also now include their equivalent IronPython translations, and the (incomplete!) re-implementation of the Xbase++ runtime in IronPython can be obtained from here.

At the moment, the IronPython support code is incomplete and covers only fairly basic functionality. Rudimentary database layer functions ( dbUseArea(), dbAppend(), eof(), etc. ) are included, but there remains much to be done before these could be used for anything much more than exposition and experimentation. Same goes for the user-interface functionality - either console or gui mode (other than the ubiquitous alert boxes). These database and gui assemblies are still very much under development, but whether, and how soon, I am able to get round to implementing them properly only time will tell. It may well be that I am soon forced into getting a job, which will at the very least slow things down enormously, if not completely stop them. If you would like to assist me in developing the project further, please consider taking out a support contract, or even making a donation.

Differences between IronPython and Xbase++

Following are the salient differences requiring explicit support from the Translator. Simple syntactic differences or features of IronPython not supported by OxIDE or Xbase++ are not mentioned.

IronPython:

• has no pass-by-reference parameters (although tuple-unpacking is probably an altogether better way!)
  • When a routine passes a variable by reference to another routine, all usages of that variable are converted to usages of an "oxValue" object - which is what is actually passed to the called routine.
  • The receiving routine has been explicitly annotated such that the generated code always assumes the use of an oxValue object Passed parameters are silently checked on the way in, and are automatically converted to oxValue objects when calls are actually made by-value and not by-reference.
• does not permit "forward-references" due to it's interpreted nature
  • Classes are generated in the correct order to prevent forward references
  • Unfortunately, it is still possible that the generated code can contain cycles of class references in multiple files that prevent the generation of working code. In such cases, either the source or the object code will need to be modified to break the vicious cycle. This is generally most easily achieved by combining files that all the offending classes are contained in a single module (file).
  • The order of import statements is calculated to minimise occurences of circular imports, but such can still occur, and will require some restructuring of the source code to eliminate. In general, this is a good thing though, as it leads to clearer and more manageable interdependencies between modules (files).
• assignments are not expressions (hence do not return a value - not even None!!)
  • When, and only when, the results of assignments are captured by the source code, the generated code is transformed to contains special calls allowing the assignment to be used as an expression.
• functions (and methods) are first-class
  • In Xbase++ ivars and methods live in separate (distinct) namespaces, but in Python referring to the method without the call parens returns the method object itself, whilst in Xbase++ it refers to the ivar of that name. This is impossible to resolve consistently without whole-of-program analysis (which I ain't doing .... yet ;-). Therefore:
    • When a method/ivar nameclass is detected, the ivar name has an underscore character prepended to it. Internal references to the ivar are automatically renamed, but external references are not - the assumption being that external access will be through the methods.
    • To fix this properly requires the Xbase++ code to be modified - preferably by prepending the ivar name with an underscore (which is a convention in Python indicating privacy).
    • This makes Access/Assign method modifiers more difficult to support, and support for these has not yet been implemented.
• functions and classes can be nested
• local variables are not declared, but created by assignment
  • conversely an assignment is _always_ to a local variable unless "global" has been explicitly declared
• array and string indexes start from 1 in OxIDE and 0 in Python
• variables captured in closures are read-only
  • make it more difficult to support "detached locals" in codeblocks.
• lambda forms accept only a single expression - no support for expression lists - ( .. , .. , .. ) form
• constructor syntax is a simple call on the class object
• multiple inheritance model is very different (no metavariables or casting)
• supports modularisation via import statements
  • Xbase++ has no similar mechanism. It implicitly and globally exports all routines and classes not declared as "static"
  • Support for this is mainly provided by the maintenance of the "project.xml" file in each source directory, which is automatically populated with sufficient information to allow the Translator to resolve external references, and generate appropriate import statements and explicit module references as necessary.
  • The "import <module>" form is used exclusively, as the "from <module> import <name, ...> form creates insoluble problems with circular imports.
  • This has numerous consequences. Among them, the names of local variables can clash with the names of imported modules. In this case, the Translator detects the clash, and prepends the local/static/parameter variable names (along with their usages) with an underscore.
• has none of the old dBASE concepts regarding implicit workareas, field variables, etc. etc.

Notes on IronPython Translator

• conditionals (if functions) containing expression lists eg."if( .. ,( .., .. , .. ), .. )" are implemented by generating nested functions and inserting calls to them (ugly!).
• imports are calculated and inserted into the target modules (.py files)
  • the Parser generates and maintains a project file for each directory (called 'project.xml" containing all the relevant metadata required to allow automated calculation of imports
  • code is generated with explicit module references, and the imports are of the "import <module>" form, rather than the "from <module> import <name1>,<name2>, ..." form. This was necessary to prevent problems with circular imports, and adheres to the pythonic "explicit is better than implicit" maxim.
  • libraries are supported by inserting <lib ..> elements into the project.xml project files with appropriately configured path attributes
  • files of the same name in different libraries (directories) may cause problems.

JavaScript

I find JavaScript extremely interesting, and have high hopes of eventually being able to make this target at least as good as the IronPython one. At the moment only the barest beginning has been made on the runtime support, and the translation script is also fairly rudimentary.

C Sharp

The C# translation is still very rudimentary with much work remaining to be done. Whilst the translation script works reasonably well, I am yet to make any inroads in implementing the runtime library support. I also remain unclear on some of the thornier technical details e.g. around runtime compilation of codeblocks. Development on this has stalled somewhat due to me having to make a living.

G Sharp

Gsharp is the name is use for the language that the OxIDE parser actually recognises. (and I readily admit the fact that arranging for the extension for G# files to be "gs" was an act of the purest hubris). G# is very nearly a superset of Xbase++, with only exception being that G# provides no support for dynamically scoped variables (aka PRIVATE variables). So any Xbase++ module containing no PRIVATE variables is also a valid G# module.

Unlike Xbase++ however, G# is designed to be (at least partially) an interpreted language, strongly influenced by Python. Just as with Python, G# requires that external references be made explicit using using syntax inspirted by Python import statements. For example:

FROM arraytools REQUEST abinscan,adrop,aminus

would be used to make the four routines "abinscan", "adrop" and "aminus" defined in the arraytools module available throughout the remainder of the current module.

G# also contains syntactic support for the static declaration of types. The only manifestation in the real world of G# is that the OxIDE transpiler can generate Gsharp code using the included xst2gs.xslt script. Conversely, G# programs can be compiled by Xbase++ compilers by inclusion of the supplied "gsharp.ch" header file.

It is planned to convervatively extend the language to include support for Generic Lists and Dictionaries (as provided by the .NET runtime) and TupleTypes which are a bit of a research topic of mine for implementing advanced database binding into the language.

So, the OxIDE tool can be used to process Xbase++/Clipper programs, and output G# programs containing import statements for runtime linking to external modules, and all preprocessing already performed. If the "/infertypes" command line option has been used, then the G# programs will also contain a whole lot of type information not present in the original.

It is my intention to continue developing Gsharp to the point of it being a fully fledged .NET language, with the technical details of how the code is generated, in what language etc. being implementation details not needing to be known or understood by average users. I am still a long way from making this a reality however, and the need to generate some income may well derail me from ever achieving this goal at all. In the meantime, I am hopeful that it might still be useful to people in it's current form.