To perform the tokenization of input program, we need to implement lexical analyzer. According to the reference site, https://www.geeksforgeeks.org/compiler-design/introduction-of-lexical-analysis/, DFA, or, Deterministic Finite Automaton seems to be the machine to be used. It takes me back to the college days, I think, the subject was ‘Principles of Compiler Design’ with a book of reference being the one written by Aho, Ullman. DFA seems to be the machine for it. The word ‘deterministic’ indicates that there are no ambiguities in the operation of the machine. There is a symbolic representation of the machine, available if you search. One characteristic of the machine seems to be ‘no memory’, indicating that, no memory is needed for the operation of the machine, like a stack or a list, input comes in and is converted to output, using rules, I think, simple regex rules suffice for the programming languages.

It is interesting to note that, ‘Regular Expression’ is itself classified as DFA.

(𝑄,Σ,𝛿,𝑞0,𝐹) is the 5-tuple that represents this machine. The description can be found on Google. Implementation of a lexical analyzer will fit this machine description.

Q: Set of states, initial state, intermediate states, final state of token table
Σ: alphabet, input character set, a-z, 0-9, other symbols
𝛿: set of rules, regexes
𝑞0: initial state, empty token table
F: final state, output token table

Sample program:

int main:
  v = 10
  for i: 1 to v
    write “hello”

The first step in the stabdard practice of implementing interpreters seems to be to tokenize input. This seems easy. Here are the tokens:

According to the information that I am referring to at, https://www.geeksforgeeks.org/compiler-design/introduction-of-lexical-analysis/, there 2 columns needed, first one is lexemes and second one is tokens. How these tokens are to be used seems to be the next step.

I think, I’d better visit some kind of tutorial to get the knowledge about interpreter implementation. I searched the Internet and found one. Link: https://craftinginterpreters.com/introduction.html.

This one is actually an online book. Skimming through the first few parts takes me back to the college. I searched ‘recursive descent parser’ on Google, to take a quick look back. Back to the book, it brings back many terms from the college, as well as introducing some more ideas. There are 2 phases here, lexical analysis, which would give us tokens, with unnecessary stuff cleaned out. The next phase is parser generation, which would give us the parser to parse the language. There are many things. to visit, like, symbol table, etc. The book has a complete design for 2 interpreters, one done in Java and one done in C.

I wanted to continue on the code for my Lazy (language) interpreter. I needed to pull what I committed from the browser from ChromeBook, I needed to pull it on my MacBook. A simple ‘pull’ did not work. I needed to ‘push’ a first commit before I did it, I came to this guess from the thought of the ‘-u’ switch needed on the first ‘push’. First ‘push’ needed access to the repo, I added the ssh key to GitHub after generating it. I did not perform the ‘ssh-agent’ step. Adding the key to GitHub now takes a verification. The verification mail did not come through. I am stopping now.

I have harboured the idea of making compiler for some time now. Today, I searched a little on Google and looking at some code in a tutorial, I got the idea. That, I can get started fairly fast by writing a simple lexical parser, which parses input code, using a hierarchy of if-else statements. No need to have stages for lexical analysis and parser generation. I’ll need regex support though, to detect numbers.

I think, a stack will be needed, to execute loops, etc.

Here is some very basic code, just to get started, it reads in a file.