Dynamic scanner

Scripts > Class elements and c++ instructions > Parser class methods > Plugin methods > Dynamic scanner

Because of dynamic scanners the TextTransformer is capable of learning.

Parts of text (literals), which are recognized by a general regular expression, can be assigned to a placeholder token. If the same part of text appears again in the input, the dynamic scanner can recognize it.

Since the next token always is already recognized, a newly added dynamic token can be recognized only if the next token was consumed. If necessary, xState.SetPosition(xState.Position()) can be invoked after AddToken. So, a new recognition is forced at the current position.

Since TextTransformer 1.3.4 AddToken can be executed as a transitional action. The token is then already available before the determination of the next token.

Independently of the project settings placeholder tokens always are case sensitive.

bool AddToken( const str& xsText,

                        const str& xsDynTokenName)

AddToken adds the string xsText as an alternative to the placeholder token xsDynTokenName, which must have been defined in advance on the token-page. If then the parser tests for an occurrence of the token xsDynTokenName in the text, the test will match, if xsText or another string added by AddToken is found.

The function returns true at success. If the placeholder-token isn't defined, false is returned.

bool AddToken( const str& xsText,

                        const str& xsDynTokenName,

                        const str& xsScope)

The relatively complex AddToken method with three parameters has been designed especially to parse programming languages.

If this method is called, xsText will be added as an alternative like in the case of a call of AddToken with only two parameters.  But this alternative is recognized in the text only, if the actual text scope is xsScope or a scope subordinated to xsScope, that means added after xsScope.

If for example the declaration of a class is parsed, the names of the class variables can be assigned to the class - that means to the name of the class. In the definition part the name of the class can be set again and the dynamic scanner will recognize the variable names. Outside of the class scope the same names can have a different meaning.

Example: " Eval; Eval; Eval "

ID

{{

AddToken(xState.str(), "USER_FUNCTION", "FUNCTION_SCOPE");

PushScope("FUNCTION_SCOPE");

PushScope("BLOCK_SCOPE");

}}

// the next token already is recognized, therefore Eval can be recognized the next but one as USER_FUNCTION.

";"

USER_FUNCTION

// Eval will be recognized as a USER_FUNCTION, because the actual scope BLOCK_SCOPE is subordinated to FUNCTION_SCOPE.

{{

PopScope();

PopScope();

}}

";"

USER_FUNCTION

// Eval will not be recognized, because actually there isn't any scope set; especially FUNCTION_SCOPE and none of its subordinated scopes aren't set.

Example:

In some grammars of other parser-generators the alternatives of a syntax rule are listed  in such a manner, that the name of the production and the definition symbol "::=" is put in front of each of them. E.g.:

Rule1 ::= A B

Rule1 ::= C Rule1

Rule2 ::= D

Rule2 ::= Rule1 E

If such rules shall be imported into the TextTransformer, then dynamical tokens can be used, to combine the alternatives into a single rule:

{{ str sScope; }}

ID

{{

sScope = xState.str();

AddToken( xState.str(), "SAME", sScope );

}} 

"::="

Expression   // GrammarExpression

{{ PushScope(sScope); }}

(

SAME

{{ PopScope(); }}

"::="

Expression   // GrammarExpression

{{ PushScope(sScope); }}

)*

void ClearTokens(const str& xsScope)

With this function all dynamic tokens which were defined for the text area xsScope are deleted. If xsScope is an empty string, all tokens for all areas are removed.

If tokens were assigned to certain text areas, ClearTokens should be invoked with an empty string before finishing a program to avoid memory leaks.

This page belongs to the TextTransformer Documentation