Cleaner Object API
My approach for DSA would go beyond a simple addition of functions and instead would revolve the addition of some new GL types and accessors. Get rid of the GLuint names for objects and instead use opaque types.
typedef struct glContext glContext;
typedef struct glTexture glTexture;
typedef struct glBuffer glBuffer;
/* etc. */
Rename functions as appropriate and provide a parallel API that focuses purely on these objects.
glContext* ctx = current_context;
/* create a new texture from existing texture data */
glTexture* tex = glTextireCreate2D(ctx, usage, format, width, height, levels);
glByte* map = glTextureMap(tex, 0, -1);
memcpy(map, image_data, width * height * bytes_per_texel);
Direct State Access
Use simple and concise accessor functions. The map example above is a good indicator of what I mean, but here's a few more examples.
/* framebuffer creation and configuration */
glFrameBuffer* fb = glFrameBufferCreate(ctx);
glFrameBufferAttachTexture(fb, GL_COLOR_BUFFER_0, my_color_buffer_texture);
glFrameBufferAttachTexture(fb, GL_DEPTH_STENCIL_BUFFER, glTextureCreate2D(ctx, GL_USAGE_RENDER_BUFFER, GL_FORMAT_D24S8, width, height));
/* shader program creation */
glShader* fragment_shader = glShaderCreateFromSource(ctx, GL_FRAGMENT_SHADER, source_string, file_name, 1); /* last two parameters allow setting debugging source information for error messages generated by the GL */
Shader Semantics, Layouts, Bindings
The bind-by-name approach used by GLSL normally is cumbersome and slow. Bind-by-semantic simply makes more sense from both a performance a usability perspective.
I'd take things a step further and provide a layout configuration for drawing as well. Layouts are a continuation of the FrameBuffer Object support, and in fact can probably just supplant FBO's entirely.
glLayout* layout = glCreateLayout(ctx);
glLayoutAddAttributeByName(layout, "POSITION", my_buffer, GL_TYPE_VEC3_32F, pos_offset, buffer_stride);
glLayoutAddAttributeByName(layout, "TEXCOORD0", my_buffer, GL_TYPE_VEC2_32F, uv_offset, buffer_stride);
glLayoutAddAttributeByIndex(layout, bumpmap_semantic_slot, my_buffer, GL_TYPE_VEC2_32F, bumpmap_offset, buffer_stride);
glLayoutAddOutputByTarget(layout, GL_COLOR_BUFFER_0, my_output_texture);
Separate Contexts from OS Windows
The platform GL context bindings such as WGL, AGL, and GLX all currently assume a single distinct context per window, which is similar to how Direct3D 9 and earlier worked as well. Unfortunately, this is pretty inflexible. It leads to a lot of pain on some platforms where simply resizing a window (or more likely, switching between a windowed display and a fullscreen display) requires destroying the platform window and hence destroying the GL context.
It also means that resources can't easily be shared between different windows/displays without yet more platform-specific functionality and extensions.
GL platform bindings should all be updated to offer two distinct sets of functions that parallel the DXGI functions for creating "devices" and "swap chains." One set of functions creates an OpenGL context that deals with resource management and access. A second set of functions allows binding to an OS window.
These could then contain a new set of objects, similar to how DXGI has separate SwapChain objects.
Alternatively, a context hierarchy could be used, where sub-contexts can be created from the main context with additional semantics, similar to how API's like Cairo implement their state objects.
As examples of both approaches, see the following:
/* distinct objects */
glContext* ctx = glWin32CreateContext(HAPP);
glDisplay* display = glWin32CreateDisplay(ctx, HWND, GL_BIT_DOUBLE_BUFFER|GL_BIT_FULLSCREEN);
glFrameBuffer* fbo = glDisplayGetFrameBuffer(display);
/* hierarchial contexts */
glContext* ctx = glWin32ContextCreate(NULL);
glContext* display = glWind32ContextCreateForWindow(ctx, HWND, GL_BIT_DOUBLE_BUFFER|GL_BIT_FULLSCREEN);
glFrameBuffer* fbo = glContextGetFrameBuffer(display);
/* this would just return NULL */
fbo = glContextGetFrameBuffer(ctx);
Shader Assembly Revisited
The assembly-language shader support has been deprecated in favor of forcing developers to always use GLSL. This is troublesome as it means that the development of alternative shading languages is much more difficult and limited to the features added to the core GLSL language. A simpler assembly language format alleviates this problem.
One of the problems with assembly languages is that they usually lack a lot of semantic information necessary to properly optimize code. I propose that a "high level" assembly language be implemented. Utilize simple three-address-form operators in place of C-like expressions, allow direct indexing into buffers and input sets, but still provide high level flow control and semantic hints for inputs and output.
Something like the TGSI format used by Mesa 3D would be a good start, though it's not a complete solution.
This would allow tools like Cg to more easily target GL and optimize shader inputs and outputs properly. It would also open the doors to getting a high quality port of HLSL or even the development of an entirely new high-level shading language. Which is actually the next point.
GLSL was one of the first high-level shading languages and was very revolutionary for its time, but it many of the assumptions and design considerations of GLSL simply haven't panned out all that well. The bind-by-name behavior is still one of the better examples. The lack of a high level effect framework, the inability to combine multiple shaders into a single source file, and the lack of overloading by version/profile are all additional negative points.
I'll admit that I'm largely in favor of simply adopting Cg as an official shading language at this pointer, as it solves most of the above problems and already has a large body of documentation, examples, tools, and experience built up behind it. I'm not at all opposed to an entirely new language though so long as it does fix of all the above problems without introducing new ones.
I'm certainly in favor of retaining column-major, right-handed math. That row-major left-handed stuff from D3D is just... odd.
Simply expanding GLSL wouldn't be too terribly complex. Shader semantics need to become first class syntax, rather than the nameless and chunky layout qualifiers introduced in GLSL 4.10. The idea of a "main" function needs to be replaced with user-named entry points that are qualified with their pipeline stage. Finally, add profile/version overloading to functions, so shaders can be written that fallback easily to older hardware profiles as the GPU scene continues to evolve.
Improved Utility Library
Give us at least a few convenience library functions that most developers need. Some basic functions for loading textures and shaders from files, for instance. Even if you can only load DDS, TGA, and PNG images, that will save a lot of developers a lot of time and headaches. Almost everyone has to write their own, or use some non-standard add-on library. Almost nobody even implements these correctly; the number of load_shader_from_file() functions on the Net that break all kinds of rules (fseek, ftell, fseek, fread patterns, for instance) is simply huge, for instance.
One wouldn't be opposed to a simple TTF font rendering library, or an improved GLUT that at the very least offered timers so animations and simple game loops could be implemented without resorting to platform-specific API calls would be nice.
The interesting thing I think about the above proposals is that they're all rather trivially implementable on existing drivers. They mostly just come down to developing a new set of API entry points over the existing functionality found in the drivers today. As both the NVIDIA and AMD drivers already expose EXT_direct_state_access, simply updating the entry points to use the new opaque types and naming is mostly just busywork rather than serious development. The vendors' shader compilers are mostly all about the low-level optimization and code generation, and adding new frontends is relatively easy compared to the effort of adding a new hardware profile, for instance.
Such an API could almost be implemented as a wrapper around OpenGL 4.1, EXT_direct_state_access, EXT_Cg_shader. It can almost be implemented as a wrapper around Direct3D 10/11.