Programming fromtoModeling…and back againMarkus VoelterIndependent/itemisvoelter@acm.org
A Journey…
„Houston,we have aProblem!“
Programming LanguagesErlangC# C++PythonJavaRubyGroovyFortranC
Programming Languagesare notexpressiveenough.ErlangC# C++PythonJavaRubyGroovyFortranC
Programming Languagesare nothigh-levelenough.ErlangC# C++PythonJavaRubyGroovyFortranC
Programming Languagesare notabstractenough.ErlangC# C++PythonJavaRubyGroovyFortranC
Programming Languagesare notdomain-specificenough.ErlangC# C++PythonJavaRubyGroovyFortranC
Programming Languagesare notenough.ErlangC# C++PythonJavaRubyGroovyFortranC
Programming LanguagesFormatsErlangC# C++PythonJavaRubyGroovyFortranjsonCHTMLCSSyamlXML
Programming LanguagesFormatsFrameworksErlangC# C++PythonJavaDjangoRubyGroovyWPFFortranjsonCJEEJMSHTMLCSSRailsyamlXMLlift
Programming LanguagesFormatsFrameworksenough.are notErlangC# C++PythonJavaDjangoRubyGroovyWPFFortranjsonCJEEJMSHTMLCSSRailsyamlXMLlift
not enough?for what?
not enough?for what?
not enough?for what?
So?
Modeling… Higher Level… Domain SpecificConcepts & Notations… Code GenerationInterpretation
Modeling… Higher Level… Domain SpecificConcepts & Notations… Code GenerationInterpretationSolvestheProblem!
But:
Different WorldsProgramming Tools!=Modeling Tools
Different WorldsModeling Tool!=Modeling Tool
Different WorldsMix Models and Programs
Different WorldsMix Models and ProgramsAST Navigation & Query
Different WorldsMix Models and ProgramsAST Navigation & QueryIntegration of 3GL code
Different WorldsMix Models and ProgramsAST Navigation & QueryIntegration of 3GL codeCode Constraints
pareCom
pareCom
Domain SpecificNotationsandAbstractionsFlexible!Limited!FrameworksLibraries(Fluent) APIs
GraphicalTextualFormsTablesLimited!Flexible!TextualTrees
CustomizeGeneratororInterpreterLimited!Flexible!ReflectionMeta ProgramsOpen Compilers
Define customQueryorNavigateorTransformLimited!Flexible!AST APIsStatic AnalysisRegex
CustomValidationorError ChecksLimited!Flexible!IDE pluginsStatic AnalysisOpen Compilers
DifferentRepresentationsandProjectionsLimited!Flexible!Text is TextCode FoldingTree ViewsVisualizations
MixingandComposingLanguagesLimited!Flexible?Python-to-C-likeInternal DSLsEmbed-As-StringSpecific: LINQ
ScalableUsableIDE SupportMature!Brittle!Modeling Tools…!?
DebuggingRefactoringTestingMature!Brittle!?
VersioningDiff, MergeBranchingMature!Brittle!for some tools…
Gets theGets someJob Done!Jobs done.some people doubt that…everybody agrees…
Whythe difference?
History?
ModelingProgramming
ModelingProgramming… (Mostly) Textual     Notations… Concrete Syntax     Storage… (Fancy) ASCII Editors… Read-Only     Visualizations
ModelingProgramming… (Mostly) Graphical    Notations… Abstract Syntax     Storage… Projecting Editors… Different editable    views for model… (Mostly) Textual     Notations… Concrete Syntax     Storage… (Fancy) ASCII Editors… Read-Only     Visualizations
Whythe difference?
It is time for …
… a Different Perspective
Programmingthe waywe doModeling?Modelingthe waywe doProgramming?
ModelingProgramming====ProgrammingModeling
We don‘t want tomodel,we want toprogram!
We don‘t want tomodel,we want toprogram!… at different levels of abstaction… from different viewpoints… integrated!
We don‘t want tomodel,we want toprogram!… with different degrees of domain-specificity… with suitable notations… with suitable expressiveness
We don‘t want tomodel,we want toprogram!And always:precise and tool processable
Where do we go from here?
A vision forprogramming
A vision forprogrammingEnablingTechnologies
A vision forprogrammingEnablingTechnologiesAvailableTooling
A vision forProgramming
Programming Languagesare notMODULARenough.
Programming Languagesare notCOMPOSABLEenough.
Programming Languagesare notCONFIGURABLEenough.
Programming Languagesare notADAPTABLEenough.
Programming Language Syntaxis notFLEXIBLEenough.
Big Language?withmany first class concepts!
Small Language?with a few, orthogonaland poweful concepts
abcdefghijklModular Languagewith many optional,composable  concepts
Modular LanguageLikeframeworksandlibraries,
Modular LanguageLikeframeworksandlibraries, but withsyntaxand IDE support
Not a new idea…
Growing A Language(Guy L Steele)
LanguageWorkbench(Martin Fowler)
LanguageWorkbench(Martin Fowler)Freelydefinelanguages andintegratethem
LanguageWorkbench(Martin Fowler)use?persistentabstractrepresentation
LanguageWorkbench(Martin Fowler)language ::=schemaeditors+generators+
LanguageWorkbench(Martin Fowler)?projectionalediting
LanguageWorkbench(Martin Fowler)persistincompleteorcontradictoryinformation
LanguageWorkbench(Martin Fowler)powerful+editingtestingrefactoringdebugginggroupwarelanguage definitionimpliesIDE definition
LanguageWorkbench(Martin Fowler)+support for„classical“programmingand„classical“modeling
Syntaxprimarilytextual
Syntaxprimarilytextualwithmoresymbolsthink: mathematics
Syntaxprimarilytextualsometimesbox&line style
Syntaxprimarilytextualsophisticatedvisualizations
Viewpoints
Viewpointssuitableabstractionsandnotationsforeach
ViewpointsIntegratedviasymbolicreferencesandseamlesstransitions
ViewpointsBusinesscustompurpose-builtcreate/include
ViewpointsBusinessCustomNotationsrealbusinessexpert integration
ViewpointsBusinessCustomNotationsrealbusinessexpert integrationBut that‘s another talk…
ViewpointsTechnicalpredefinedlibraryconfigure
Example Languages
(Seemingly)Simple ExampleAddingmatricesto C in an embeddedenvironment.
Currently:
Currently:Matricesnot supportedin XML format and generator
Currently:Tool teamwould have todo the extension… a lot of work… busy… one central tool
Currently:No real compiler supportin the resulting C code… type checks… operator overloading… generics (matrix<int>)… matrix syntax?
Better Solution
Better Solutiongeneric matrix and vector types
Better Solutionreal matrix and vector literals
Better Solutionsyntaxhighlights for vectors and matrices
Better Solution operatoroverloading
Better Solution operatoroverloadingstaticoptimization… symmetrical  matrices… identity matrix… diagonal matrices
Better Solutionmathnotation
Better Solutionaseparate language moduleused only bythose whoreally need it
In addition: PLE Variabilityannotatingvariabilityexpressionsto arbitrary regions
In addition: PLE Variabilityannotatingvariabilityexpressionsto arbitrary regions statically  checked
In addition: PLE Variabilityannotatingvariabilityexpressionsto arbitrary regions project variant
more later…(tool examples)
EnablingTechnologies
EnablingTechnologiesAdvancedParserGenerators
Modeling as Programmig… (Mostly) Textual     Notations… Concrete Syntax     Storage… (Fancy) ASCII Editors… Read-Only     Visualizations
Custom SyntaxGraphicalTextualSymbolic++
IDESupportTeamworkDebuggingCustom Editors
CompleteSymbolicIntegrationGoto DefFind RefsRefactoring
Xtext-like Toolsprovide editor supportCustom EditorsTeamworkGoto DefFind RefsRefactoring
Limited toUnicodehow to handlenon-character symbols
Graphics !=Texttwo worlds…separate editors… per syntax/viewpoint… models can still be ref integrated
What if!combine languagessame seq<Char> asdifferent tokens?
Ambiguous!Not Parsable.Bad Luck.
Traditionally:Language ::=      Token* | Rule*;
Traditionally:Language ::=      Token* | Rule*;Token ::=      Character*;
Traditionally:Language ::=      Token* | Rule*;Token ::=      Character*;Rule ::=      Token* | Rule;
Traditionally:Language ::=      Token* | Rule*;2 Phases:1. Global Tokenization(aka Scanning)2. Grammar Recognition(aka Parsing)
Modern:Language ::=             Rule*;Rule ::=             Char* | Rule*;1 Phase:Grammar Recognitiondirectly on char*
ScannerlessParsingNo Ambiguitywhen combining languagesAlways Parsable.(well, almost. There are some corner cases…)
EnablingTechnologiesProjectionalEditing
text… to tree… to textParser-based
Projectionaltree… to text-lookalike (editor)… to other trees … [*]… to text
Programming as Modeling… (Mostly) Graphical    Notations… Abstract Syntax     Storage… Projecting Editors… Different editable    views for model
Programming as Modeling… (Mostly) Graphical Any kind of    Notations… Abstract Syntax     Storage… Projecting Editors… Different editable    views for model
LanguageCompositionThere‘sno parsing.Unique Language Element Identity.Unlimted language composition.
FlexibleNotationsTextuallike ASCIItreated the same}Graphicalcan be mixedbox & lineSemi-Graphicalmathematical
AutomaticIDE Extensiontool support is inherentfor languages build withprojectional toolslanguage definitionimpliesIDE definition
MultipleNotations… for the same conceptse.g. in different contextsor for different tasks
PartialProjections… different views… for different roles/people… only a particular variant
Storage!= Schema… store arbitraty meta datachange logconflicting informationvariability annotations… independent of language schema!… „aspects“, overlay
LiveProgramsthink: spreadsheeta change to one part of programcan lead to (dependent) changesin other parts
Tree Editing… is different from editing text… try to make it feel like text… takes some gettingused tobut: for more flexible notations a more general editing paradigm is needed
InfrastructureIntegration… storage is not text… diff/merge must be in tool… existing text tools don‘t work
ProprietaryTools… no standards… no interop
AvailableTooling
AvailableToolingEclipseXtext
http://eclipse.org/modeling
http://eclipse.org/xtext
Eclipse XtextBuilding Textual Editors
Eclipse XtextBuilding Textual Editors
Xtext: Specify Grammar
Xtext: Gen. Meta Model
Xtext: Constraints
Xtext: Generated Editor
Xtext: Generated EditorCode Completion
Xtext: Generated EditorSyntax ColoringCustom KeywordColoring
Xtext: Generated EditorRealtimeConstraintValidation
Xtext: Generated EditorCustomizableOutlines
Xtext: Generated EditorCode Folding
Xtext: Generated EditorGoto Definition Find ReferencesCross-File ReferencesModel as EMF
Xtext: Generated Editor
PlatformArchitectureDSL
As you understandanddevelopyourPlatformArchitecture…
Develop a language to express it!
Language resembles architectural concepts…
Express the applications with the language.
Another Story…Architecture As Language
DEMO IBuilding DSLs with Eclipse Xtext
AvailableTooling
AvailableToolingJetbrains‘Meta ProgrammingSystem
also do…IntelliJ IDEAResharper
released inQ3 2009currently 1.1 RC1
licensed underApache 2.0
Build new standalone DSLs
Build new standalone DSLsBuild DSLs that reuse partsof other languages
Build new standalone DSLsBuild DSLs that reuse partsof other languagesJava++(MPS comes with BaseLanguage)extend base language
Build new standalone DSLsBuild DSLs that reuse partsof other languagesJava++(MPS comes with BaseLanguage)extend base languagebuild DSLs that reuse partsof BaseLanguage
Language Extension ExampleJava + Extension
Language Extension ExampleOldJavaJava + ExtensionReadWriteLock l = …l.readLock().lock();try {  //code} finally {  l.readLock().unlock();}
Language Extension ExampleOldNewJavaJava + ExtensionReadWriteLock l = …l.readLock().lock();try {  //code} finally {  l.readLock().unlock();}ReadWriteLock l = …lock (l) {  //code}
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Structure  Editor   Typesystem  Generator
Language Extension ExampleResult behaves like a nativebase language construct
Language Extension ExampleResult behaves like a nativebase language construct
Language Extension ExampleTranslated to regular Java codebased on the generatorpackage jaxdemo.sandbox.sandbox;import java.util.concurrent.locks.Lock;public class DemoClass {  private Lock lock;  public DemoClass() {    try {      this.getLock().lock();      SharedResouce.instance().doSomething();    } finally {      this.getLock().unlock();    }  }  private Lock getLock() {   return this.lock;  }}
Example LanguagesUI Language
Example LanguagesHTML Templates
Example LanguagesPersistent Classes
DEMO IBuilding DSLs with JetBrains MPS
AvailableTooling
AvailableToolingIntentional SoftwareDomainWorkbench
Version 1.0 released inDec 15, 2008currently at 1.4
CommercialProduct
Eval availableupon request
Statemachine ExampleNative Projection
Statemachine ExampleTabular Projection
Statemachine ExampleTextual DSL Projection
Statemachine ExampleJava Projection
Statemachine ExampleRuby Projection
Statemachine ExampleTwo projections side/side
Statemachine ExampleTwo projections side/side
Pension Workbench ExampleText Editing Domain
Pension Workbench ExampleInsurance Mathematics Domain
Pension Workbench ExamplePension Contract Rules Domain
Pension Workbench ExampleAll in one Document
Pension Workbench ExampleSymbolically integrated
A Journey…Arewethere?
A Journey….coordinatesweb email skypexinglinkedinwww.voelter.devoelter@acm.orgschoggladhttp://www.xing.com/profile/Markus_Voelterhttp://www.linkedin.com/pub/0/377/a31
From Programming to Modeling And Back Again
From Programming to Modeling And Back Again
From Programming to Modeling And Back Again

From Programming to Modeling And Back Again