First, let's look at some of the current popular scripting languages available today.
Lua's popularity in the gaming industry can be attested to a few primary features. Perhaps first and foremost, Lua is down right easy to embed. It doesn't pretend to be a Real Language like Python does; it may have all of the language features necessary for general application development, but it isn't packages with a huge standard library and its stand-alone interpreter is nothing more than a few simple wrapper around the Lua library. It's small and has no dependencies. This makes it excellent for embedding in any larger program. It also was once at the forefront of interpreted language efficiency, practically pioneering the register-based interpreter design that is now popular with many high-performance interpreters. Lua is also highly portable, needing nothing more than an ANSI C compliant compiler and environment. Unfortunately, Lua's dedication to remaining ANSI C compatible is also its biggest downfall, as the requirement has forced it to lag far behind the state of the art interpreters available for other languages. Last but perhaps not least, Lua is very liberally licensed, removing any potential legal headaches when using it in a commercial game.
Python has also seen some popularity in the game industry, though almost exclusively at the lower end of the scale. In most cases, the engines and games making use of Python are using it as the primary development language around a smaller C/C++ core, where as Lua is used mainly as an extension and logic language embedded in a large and full-featured C/C++ core. Python is in many respects a beautiful and powerful applications programming language, though it is not without its faults. Python makes it quite easy to put together casual and hobbyist games and through that has seen some small uptake in smaller scale commercial games, such as those based on Panda3D. Python has two large issues, however. The first is that Python is... well, large. It is a complete application programming environment with a massive standard library, the vast majority of which is totally undesired in the embedded use case. Secondly, Python is heinously slow in a way that can't easily be fixed. Some of its language semantics such as automatic promotion of integers to BigNum types on overflow actually make some of the new mega-fast interpreter techniques unusable, or at the very least significantly less effective.
A few other languages are around and used in games. C# through Mono has seem use in a few games, and Thief made use of a Java interpreter for its scripting. Languages like GameMonkey Script, AngelScript, and Squirrel have been created to directly serve the embedded/games industry but have not picked up much steam. There has also been a fair share of use of languages like LISP, Scheme, Forth, and so on over the years, though their popularity has waned. In many cases, engines created their own scripting languages and runtimes. These custom script languages can run from simple modifications of established languages, such as UnityScript, to completely unique languages like the venerable QuakeC or UnrealScript.
With so many options to choose from, it can be a bit daunting to select an option for a new game engine. Simple going with Lua is not at all a bad choice to make, as it has proven itself good enough time and again in the professional world. Your needs or simply your preferences may drive you towards one language or another. The most important thing to do is of course to identify what you will be using a scripting engine for and what you need the scripting engine to do. You may well have a requirement or three that drives you entirely towards a particular existing solution, or perhaps even necessitates a custom solution.
- Portability - Game developers deal with multiple platforms on a far more frequent basis than any other group of developers in today's computing industry. Three popular home consoles, two popular handheld gaming systems, a wide variety a incompatible mobile phone platforms, Web/Flash games, and the three popular variants of desktop/notebook operating systems. Common architectures include ARM, PPC, and x86, with x86-64 seeing a slow rise in use, along with a plethora of lesser-known architectures. These platforms range from big to little endian, from 16-bit to 64-bit instruction sets, with one core to an ever increasing many, and with a variety of unique quirks, limitations, and power features. Your target platforms will clearly limit your options. The iPhone does not allow dynamic script execution, for example, so you will be required to use a script language that can be compiled to native machine code, e.g. using Mono as a runtime. The XBox360 does not allow JIT-compiled code, so you may find yourself needing the fastest pure-interpreter language. Other embedded platforms are just very limited on CPU and memory and so need extremely small and compact languages.
- Special Features - Sometimes you just need a specific set of special features that lies outside of the least common denominator amongst all the popular options. Perhaps you really need or just really want a form of static typing. Maybe you need really, really fast vector math. Perhaps you just absolutely need first-class closure objects. Could be you just have to have UTF-16 string support. Whatever the case may be, if you know you need something out of your script language that only one or two options support, you may have to give up on a personal preference and go with what gets the job done. This seems fairly obvious and perhaps even redundant as advice, but I have seen a few teams pick a language that simply lacked a necessary feature because they didn't evaluate what they were going to use scripting for.
Developing a Custom Language
In the end, you may decide that you have a strong case for developing a custom script language for your engine. If you're a student or hobbyist, you may simply decide to write your own for the sake of learning how such things work at an intimate level, which is something I recommend everyone do at least once given the opportunity. A custom language can fill the exact niche you need and meet all of your requirements, assuming you're willing to sink in the (likely large) amount of time that will be necessary.
That aside, developing a custom engine gives you the opportunity to solve a few problems that no existing language currently does. My instructor and I were contemplating a purely interpreted language with a small embeddable core with support for efficient vector math and type hinting, for example. We need a purely interpreted language for XBox360 compatibility, we prefer small embeddable languages for maintenance and ease-of-development reasons, and fast vector math makes it far more feasible to do various animations and particle effects in scripts. Targeted use of type hinting just saves a lot of time and effort in the long run after years of experience on both our parts, and it also allows the engine to potentially optimize the code better. No existing freely available language meets all three of those requirements. On top of those needs, we also have a few personal preference bullet points, such as having a comfortable C++-like syntax familiar to student developers, a strong optimization pass before generating the bytecode, and non-obnoxious license that lets us use the languages however we want without legal questions.
Given my interest and experience in the field of language design and implementation, I may well take on a project to develop such a language over the summer, just for kicks. I will be clear though that even replicating something as small and simple as Lua would take a few weeks of full-time work. Implementing something more efficient than Lua, using direct threading or context threading, will easily take a few weeks more. Tacking on a JIT engine for use on PCs is at least relatively easy given the availability of several existing high-quality JIT engines from Mozilla, Apple, and others. Adding in a good set of high-end optimizations and a solid debugging and profiling system turns it into a several month task at best. That is again assuming I could work on it full time, which I clearly can't. I'm no stranger to this area, either.
If you are considering a custom scripting engine, realize that it is truly a large task that will take considerable development time to accomplish a fully usable and complete implementation of any non-trivial design. If that's what you need to do to nail your project's requirements, then that's what you've got to do, but I can't recommend strongly enough that you evaluate the existing solutions and use one of those if your project is commercial or otherwise has development deadlines to meet.