close
The Wayback Machine - https://web.archive.org/web/20050220100913/http://pyre.third-bit.com:80/pipermail/extprog/2005-January/000019.html

[Extprog] MetaL: an extensible programming compiler engine implementation

Manuel Lemos mlemos at acm.org
Thu Jan 27 23:36:13 EST 2005


Hello,

I was invited to join this list by Miles Thibault due to my work in the 
MetaL project.

First I would like to mention that English is not my mother language. 
So, I hope you can excuse me in advance if I do not express myself in 
correct English.

Last year, somebody posted a mention in Slashdot regarding Extensible 
programming article by Gregory Wilson.

http://developers.slashdot.org/article.pl?sid=04/05/26/2231214

To which I posted this comment regarding MetaL .

http://slashdot.org/comments.pl?sid=109013&cid=9273795

I have mailed Gregory Wilson mentioning that his article basically 
describes what can be done with the MetaL engine. Gregory replied 
mentioning that it does not have a WYSIWYG editor and a mechanism to add 
foreign data. I did not had the time to follow up as I am usually very 
busy but I will try summarize the scope of the project.

- The MetaL engine was developed with the main intention solve the 
problem: "What if my current language of choice will no longer in use in 
the future?" .

In the future I may have to use different languages due to project 
requirements that are not under my control. In that case, any libraries 
and components developed in other languages will be rendered useless and 
I would not be able to reuse them.

That was an hard learned lesson in 1993 when Commodore, the companny 
that owned the Amiga computers, has gone bankrupt. Basically that was 
the end of the Amiga computers as a viable platform for most of us that 
used to developed software for it.

I used to develop in C then. Most of my code in C became useless as it 
was so attached to the Amiga platform that could not be reused in other 
platforms.

In 1997 I started developing Web based applications in PHP. In 1998, I 
was invited to develop a sort of Web based ERP application. My question 
then was: will PHP always be the right language for this project?

In 1999 I have started concieving and implementing MetaL in direct 
response to my doubt about the right language for the present and the 
future.

- MetaL was concieved as a language of redefineable commands. I have 
chosen XML to create all types of files used by MetaL, as XML became a 
stardard for describing platform independent data for many purposes, 
specially for hierachical data structures like most types of programming 
source code.

XML is annoying for the fact that it requires an excessive number of 
characters to be typed in order to write well-formed documents. However, 
the wide availability of tools and libraries that make it simple to 
parse and generate XML, convinced me that it would be a good choice of 
the format of all files used by MetaL.

- MetaL is mainly a code generation language. It is not meant to replace 
any language, but rather generate code in one of many possible target 
languages. So, MetaL should be understood as a tool to generate code at 
application compile time.

- MetaL is not just a programming language. The MetaL engine is not just 
a compiler that generates code in a target language from some XML files.

The MetaL engine is a totally modular engine. It consists of some engine 
core classes and many modules that implement aspects of programming 
languages like execution flow control, expression manipulation, OOP, etc...

There are also modules for generating code for interfacing with existing 
libraries that are used in different types of applications like: 
database access, Web forms generation and validation, text template 
processing, retrieve text and locale data internationalized 
applications, etc..

Although most of the MetaL engine modules are meant for code generation, 
  any module may just execute actions that do not have to do anything 
with code generation, but may be useful for other task releated with the 
software development process .

For instance, there are modules for things like documentation extraction 
and generation, build process managment similar to the make tool, 
generating XML documents, generate application package distribution 
archives, etc.. In sum, you can automate the whole software development 
process.

- The MetaL language is defined in such way that you can define any 
commands with whatever names you want for the purposes you want.

For this you need to define some definition files usually named with the 
.input file name extension. These files determine which modules of the 
MetaL engine will be called to generate the corresponding code in the 
target language.

This means that you can define a language as you want. However, there 
are some basic .input files that define a set of commands and associate 
them with names that anybody that knows english can understand.

If you do not like the command definition files that come with the 
standard distribution of the MetaL engine, fine, create your own.

.input files are like the include files that exist for many languages 
that contain only definitions and no code in them.

The actual programming code files are named .output . These files 
contain XML tags that trigger the execution of commands defined in the 
.input modules when the MetaL compiler engine processes them.

To make it simple for people to understand this file structure, usually 
I start telling that programming is like cooking: you have the list of 
ingredients in the .input files, and then you have the actual actions of 
the recipe in the .output files.

The existing MetaL engine modules are totally independent. They are not 
even aware of the existence of each other. They just pipe information to 
each other, but it is up to the compiler core class to pass the 
execution context data and the command results between them.

The association between commands, compiler modules and supported target 
languages is described in .bindings files. These define which module 
classes should be loaded to implement the commands provided by a module 
to generate code in a given target language.

If you do not like one or more of the existing modules for whatever 
reasons, fine, write your own modules for your purposes.

There is only one detail, everything was written in PHP, about 43,000 
lines of PHP code so far. This means that every module is written as a 
class that implements a common interface.

If this is a problem to anybody that would like to write new modules, 
there are PHP extensions for integrating with code written in Java, 
Python, Perl and probably some other languages that I am not aware.

- The MetaL engine is Open Source. Currently, the license is BSD similar 
to Apache or PHP project licenses.

This means that you can use for whatever purposes you may want without 
even saying "Please" or "Thank you". You can also distribute closed 
source versions and even sell the MetaL engine, if anybody is willing to 
pay your for it . Obviously, this does not mean that you can omit 
copyrights or claim MetaL engine copyrights for yourself.

I choose a BSD like license with the intention to not prevent people 
from adopting MetaL due to licensing restrictions, as it happens with 
software that uses the GPL license.

The way the MetaL engine was made to be modular, making its modules 
totally unaware of each other, was also meant to avoid any licensing 
restrictions usually imposed by tight coupling.

- My most inportant project that is being actively developed and 
throughly documented is Metastorage.

It is the first real domain language tool implemented with MetaL. It is 
a project to generate data object classes that store and retrieve 
objects in persistent containers such as SQL databases. I use this all 
the time in my applications with great productivity gains.

You may find more about Metastorage it here:

http://www.meta-language.net/metastorage.html

It uses a module named persistence that implements what I call 
meta-metaprogramming, or just level 2 metaprogramming.

Level 1 metaprogramming is implemented by modules that translate MetaL 
commands directly into a target language.

In level 2 metaprogramming there is a first stage of translating an high 
level definition into MetaL (XML) source code that just uses level 1 
metaprogramming code.

As you may imagine now, in theory you could concieve any other high 
level language that could be translated to lower level (XML) source code.

You could even concieve even higher level (level 3) that would translate 
to level 2 metaprogramming code and so on. In theory the sky is the limit.

As long as there are real world applications that justify the 
development of new higher level languages, higher level abstraction 
languages can be concieved and implemented with the MetaL engine.

This also serves to demonstrate the benefit of using for everything XML. 
The fact is that the classes of the core MetaL compiler and other 
modules already provide means to parse and generate intermediate XML 
code without a great effort.

- There would be a lot more to say about MetaL. I leave any issues that 
I did not mention or were not clearly explained, for any follow up 
messages. Meanwhile you may find some documentation, presentation 
slides, a FAQ, etc. here:

http://www.meta-language.net/documentation.html

http://www.meta-language.net/presentations.html

http://www.meta-language.net/faq.html

- I also wanted to say is that I have developed MetaL to address my real 
software development needs. This project is not experimental. It has 
been developed for almost 6 years. I am not associated to any academic 
institution or business company.

I developed MetaL to assist me in my projects. In most of them the use 
of MetaL is not even noticed by the users because MetaL is mostly used 
to generate code or documentation that looks pretty much as if it was 
hand written.

If other people see MetaL as something that can be used for their 
purposes, fine. Third party adoption is not my main goal, but I will 
gladly assist anybody as my time permits.

My experience of developer of many Open Source projects tells me that 
when people adopt my software they often provide valuable feedback in 
the form of bug reports and improvement suggestions.

You just have to be a little patient because the existing documentation 
is scarce.

If somebody that could use MetaL prefers some other solution, that is 
fine for me too, as it is none of my business. I just wanted to say that 
I am not here to compete with anybody that provides similar solutions.

I live by the rule that "cooperating is better than competing". So, I am 
just glad with the opportunity to discuss about a subject of my interest 
in the sense that it can help me to improve my own solutions.

-- 

Regards,
Manuel Lemos

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

PHP Reviews - Reviews of PHP books and other products
http://www.phpclasses.org/reviews/

Metastorage - Data object relational mapping layer generator
http://www.meta-language.net/metastorage.html



More information about the Extprog mailing list