The key challenges in designing a programming language

I have been involved with Python’s development process as an official core developer of the language for a decade over Python’s 24 years in existence and we are still trying to get it right. I have learned a bunch of programming languages to varying levels of expertise to learn from them both as a programmer and a contributor to Python: C, C++, Clojure, Dart, Forth, Go, Haskell, Icon, Java, JavaScript, Lua, MIPS assembly, OCaml, PHP, Ruby, Scala, Scheme, Standard ML, and Tcl. None of these languages are perfect.

Why is it so bloody hard to design programming language? I think it boils down to two overarching topics and the difficulty of finding the balance between them which will never please everyone all the time.

Maintainability

You are either writing code, or you are fixing it. That’s (roughly) the life of a programmer.

Now this encompasses several bits. One is simply the ease of writing new code. If your language provides an easy way to write good, solid code the first time then you hopefully minimize the amount of time spent maintaining it. This is typically a balance of verbosity; type systems requiring more code and effort for the tradeoff of maintainability of having the compiler do some low-level sanity checks for you.

After that comes ease of debugging. This typically relates to what your compiler doesn’t check for you statically. How easy is it to work backwards from an exception or crash? How does the language facilitate you verifying what things are correct? How hard is it to write tests (which I’m assuming you write for all your code)?

And speaking of ease of writing tests, what the language facilitates by providing a standard library also comes into play, at least initially. If you have to write a map-like data structure from scratch or an HTTP header parser in order to make HTTP requests, your productivity is going to be low. This won’t necessarily last, however, as the community around a programming language matures and third-party projects being to flourish and evolve faster than the programming language. But the activeness of the community along with the standard library of the language to help get you going play a role in maintainability (both getting your code written and not having to maintain that code which you use).

Performance

But no matter how easy a programming language makes it to write new code and later debug it, none of it is worth anything if the language is too slow. This is when flexibility is sacrificed, which can often mean sacrificing maintainability in terms of how much extra work is required to accomplish the same thing in a slower language. You can statically declare everything in your language as much as you want, but if all you can do is integer math and array usage (e.g. a Turing machine), you are not exactly going to be productive.

Balance

How you balance maintenance against performance decides what shape your programming language takes. And since you can never make the perfect programming language it will influence what programmers are attracted to your programing language.

Take C as an example. Tight, well-written C code is hard to beat short of writing assembly when it comes to performance. But I think any programmer would find it hard to argue that C code is easy to maintain compared to other options. Getting your memory management right can be tricky. Segfaults are not the easiest things to debug. The list goes on. C exemplifies maintainability sacrificed for speed.

Towards the other end of the spectrum is Python. Writing code in Python is fast and highly productive. The language is a good balance of high-level constructs along with a small core language so that an experienced programmer can keep the whole language (sans black magic) in their head. And the black magic is there to help when needed (e.g. frame access for debugging). You shouldn’t be able to segfault your program from straight Python code that doesn’t do anything segfault-y (e.g. ctypes). And the debug cycle is extremely fast thanks to the lack of a expensive compilation step. But Python’s performance is not that of C. While you can get great performance with PyPy, Python’s “good enough” speed is not always good enough for everyone in all situations.
Then you start looking at languages that try to strike a balance between high maintainability and great speed; do you pay for your program with man-hours or CPU cycles? Go seems to be the newest favourite amongst people trying to find more of a middle ground. While typed, it heavily relies on type inference and uses interface typing over inheritance typing. It has fast compilation time to have a faster iteration cycle. Memory management is handled for you. And it’s fast. But it does have the issue (for some) of very explicit error handling (which if you remember Go’s systems programming pedigree makes it not surprising). Lack of generics and access to the low-level guts of the language can lead to more work to accomplish something than it would in Python.

Language development is hard

So what’s the point of this rant? Mainly to point out that programming languages are hard and thus there is no perfect language, now or ever. When you choose a language for a project you need to weigh performance against maintainability of what you will be producing; I would say you should figure what kind of performance needs you have and what computing resources are at your disposal and then find the most productive language that meets those performance needs (e.g. why the Python community has always said to prototype in Python before worrying if you actually need to go faster than what Python + C or PyPy can provide). Also be willing to learn a new language to see if it fits your style of coding better than your current favourite language since people will never stop trying to strive for that impossible goal of the perfect programming language.