[Open-graphics] The state of Linux Graphics...

[Richard Cooper]

Sorry for this not being a reply to the actual message it is a reply to,  
but I just signed up to this list two minutes ago.  I was ranting  
elsewhere and someone pointed me to it, so I had to sign up and reply:

> I'd like to put out a call for 4 or 5 driver writers to help with the
> following pieces of software for OGD1, OGC1, etc.:
>
> - VGA BIOS
> - Linux kernel driver
> - Linux fbconsole driver
> - Linux DRI driver
> - X11 DDX module
> - Mesa driver (later than the others)
>
> Those that volunteer will be lent OGD1 cards.  Most likely, those who
> check in working code will get their loans made permanent.
>Takers?

So what have we got here?  We don't seem to actually have OGD1 cards.  We  
don't seem to have a clear understanding of what we're even going to do.   
We seem to be assuming that graphics in Linux suck because of lack of  
hardware vendor support, and that if we simply create our own video card,  
then we'll have support and everything will magically become better.  So  
we're planning ahead, and asking for people to write the code now and then  
we'll get around to creating the actual video card when we have time.   
Basically, I guess we're looking to do as much work as possible before  
figuring out that are base assumptions were wrong.  The problem isn't lack  
of vendor support, the problem is that everyone on the open-source side of  
things is to stupid to see what they're doing wrong.  Linux graphics could  
be fixed right now, without a new video card, but no one cares to do it.

Now I realize that what I'm saying isn't what anyone wants to hear, so  
rest assured that I'll ignore most people's replies because I've delt with  
Linux graphics as much as I care to.  I've given up.  However, I do know  
what needs to be done, and so I figure the least I can do is explain it so  
that, assuming everyone here is more interested in improving Linux  
graphics than they are in the happy idea of creating an open graphics  
card, they'll at least know what needs to be done.  Just don't expect any  
programming help from me.  I've programmed this nonsense for the last  
eight years and I've had all I can take.

So anyway, if you want to fix graphics in Linux, here's what you need to  
do:

1.  Remove the VT-100 emulator from the kernel, as well as the code which  
does console switching, and even the VESA framebuffer code.  Remove every  
lame excuse the kernel has for a graphics driver as it's all simply done  
the wrong way and any attempt to build a new system based on the old  
system will result in a new system that sucks just like the old system.

2.  Create an easy, standardized, and well documented kernel video driver  
interface that allows card manufacturers to easily create a video driver  
for Linux.  The manufacturers don't have to create the drivers now or even  
ever, but the interface needs to exist.  This would be a simple interface  
that allows someone to copy a file from their video card's CD-ROM, tell  
Linux to load it, and then it's their video driver.  It would not require  
the driver to be open source, it would not require it to link into the  
kernel, it would have it's own driver API where the kernel simply tells it  
things like "what modes are available" and it says "these modes are  
available" and the kernel says "I would like to switch to this mode now"  
and the driver code simply does the necessary hardware access and then  
reports back "that mode is now set" and then the kernel asks "so where is  
the video memory located at" and the driver says "it's right over there..."

This interface has to be easy because the cost of a video card  
manufacturer creating a driver has to be less than the revenue they will  
recieve from increased sales, and a complicated interface isn't going to  
make that possible.  It has to be standardized because tech support is  
also a large expense for manufacturers, and they can't claim "this card  
supports Linux" on the box and then refuse to accept tech support calls  
about it.  If the interface isn't stable, if it is going to change with  
every new release of the kernel, then that would be a huge expense to  
anyone who cares to create a driver.  It also has to be well documented  
because video card manufacturers don't want to support Linux for  
idealogical reasons and so while the average Linux hacker has no problem  
with spending weeks digging up information on an undocumented or poorly  
documented interface, a video card manufacturer who is only in it for the  
money will want the information to be easy to find, easy to understand and  
comprehensive or else they simply won't bother because it will cost them  
more to create the driver than the additional profits they anticipate it  
will create.  There shouldn't be any problem with the information being  
up-to-date because, like I said, the interface has to be standard meaning  
that it doesn't change over time.

This is one of the biggest failings of Linux graphics.  In the days of  
DOS, some applications such as AutoCAD had a driver included with every  
video card, but less important applications such as Microsoft Windows  
weren't so lucky and had to either write their own drivers or use the  
BIOS.  Linux seems to think it is simply going to jump in the game, make  
no effort to make creating a video driver easy, and all of the hardware  
vendors are simply supposed to bend over backwards to support it.  What no  
one wants to acknowledge about Windows is that it got to where it is by  
making things easier for people.  Windows wanted video drivers, but I  
don't think they demanded that anyone create them because at one time  
Windows wasn't big enough to demand such things.  Instead, Windows just  
made things easy.  DOS developers switched to writing Windows games  
because Windows supplied easy memory access and easy graphics access.   
Office application developers switched to Windows because Windows supplied  
easy printer access and easy network access.  Linux on the other hand  
doesn't want to make anything easy, it simply wants to scream "I'm an  
operating system too!" and expect businesses to give it what it wants.

Hardware manufacturers aren't stupid.  Businesses become big by being  
smart.  They create a formula to calcuate wether or not it is a good idea  
to support Linux, which involves the potential increase in sales revenue,  
and the costs of driver development, driver maintainence, and technical  
support.  Linux simply doesn't offer much for increased sales revenue, its  
lack of a standard API for video drivers to utilize means that driver  
development costs are high, and the general poor design of the entire  
system means that making the driver work for everyone is no easy task,  
which means it will fail for many people and the technical support costs  
will be high.  If Linux wants video card drivers, it needs to first of all  
provide an API for video card drivers and then make things such that the  
formula solves to a positive dollar amount.

In particular, an open video card isn't going to fix everything.  It would  
allow open source developers to create video drivers, but look at the  
quality of our current VGA driver and our current VESA driver?  What's  
that?  You don't know what drivers I'm talking about?  Yes, I don't know  
what I'm talking about either.

The kernel has no video drivers, instead it has about 1/4 of a VGA driver  
and 1/17 of a VESA driver.  SVGAlib is a little more complete, it has 2/3  
of a VESA driver, but no VGA driver for reasons that are a complete  
mystery to me.  X11 has a VGA driver, at least as far as graphics modes  
go, but it supplies a similar 2/3 of a VESA driver and the occasional  
support for an odd video card here and there.

What I'm saying is that whole mess shouldn't exist.  X11 and SVGAlib  
shouldn't have drivers at all, they should use the same graphics API  
everything else has available, which ultimately ends in the kernel having  
a video driver API.  There would be a default VGA driver and a VESA  
driver, and anyone who has specs on a video card can write a driver, and  
in particular the easy API would make it easy for hardware manufacturers  
to supply their own drivers.

What I am ususally told is that graphics do not belong in the kernel  
because graphics are unstable, but this simply isn't true.  What is  
unstable is the way that the kernel handles, or rather doesn't handle,  
graphics.  The reason graphics are unstable is that they currently involve  
direct hardware access from userspace, which everyone agrees is a big  
no-no, but for some reason everyone also seems to agree is the way that  
graphics should be done.  I have no idea why people think this way, but I  
think it's obvious that they are wrong and I have no idea how to explain  
it to anyone who does not agree with me.

3.  Create a VESA BIOS driver that uses this interface, and maybe even a  
"Basic VGA" driver.  Direct access is preferred for cards for which  
programming information is known, but VESA is an excellent thing for cards  
for which drivers do not yet exist, and so it is the driver that should be  
created first.  Sure, the VESA BIOS runs in real mode and Linux runs in  
protected mode, which is the biggest lame excuse for why the VESA BIOS  
isn't used, but that's what vm86 mode is for.  I've used the VESA BIOS  
 from user space in Linux before, and SVGAlib and X11 use it as well with  
their VESA drivers, so if anyone cares to tell me that it can't work,  
don't bother because you are wrong.  It's absolutely possible, it's just  
that everyone makes up lame excuses that it isn't possible because they  
can't concieve of any other possible reason for why it hasn't been done  
yet.  It is so stupid that it hasn't been done yet that it forces people  
to assume that it simply isn't possible.

4.  In user space...  Create a "console multiplexer" which does much as  
the virtual console code did in the kernel, allowing users to switch  
between "virtual consoles" with an Alt-Fn sequence.  Ideally, this should  
use the kernel's API for graphics support and provide for other  
applications what appears to be a dozen different video devices.  Each  
pseudo video device should work just like the real thing:  It should allow  
applications to select whatever video mode they want and it should not  
require applications to be aware of wether or not their virtual console is  
the one which is currently active.  It makes much more sense that someone  
writing this one application deal with that issue and not require anyone  
else to.

The kernel currently does the opposite.  If any application wishes to use  
graphics, not only do they have to use direct access to set the graphics  
mode, and they damn well better restore it before exit because the kernel  
has no idea how to do so, but they also have to cooperate with a flaky  
kernel API to work with the virtual console system to ensure that they do  
not access the video device when their console isn't active, and the fine  
details of this interface seem to change often between kernel releases,  
and in particular the interface is designed solely for X11, making other  
graphics solutions such as SVGAlib less reliable as any effect that  
changes might have on something other than X11 seems not to be considered  
when changes are made.  In particular, I don't like the instability of  
SVGAlib, so I created my own graphics solution for Linux which, among  
other things, doesn't require applications using it to be suid root, and  
if those applications crash, it is impossible for the video mode to not be  
correctly restored to text mode.  So I had it working perfectly, but I  
found it broken every time I downloaded a new kernel, and so I simply gave  
it up.  It was a nice system, it simply used standard VGA and so it could  
get 640x480x4 and 320x240x8 on every system no questions asked, but then  
came framebuffer mode and I had to modify it to detect framebuffer mode  
and spit out a message that read "your card is not in a VGA mode and I  
only know VGA so I cannot change your video mode" and then the next kernel  
release after that broke the way I was detecting framebuffer mode by  
allowing /dev/fb0 to be opened even when the kernel wasn't in framebuffer  
mode.  It also did something to the specifics of cooperating with the  
kernel which causes the text on the screen to be corrupted on program exit  
which I still haven't managed to figure out what has changed.  I can't  
simply look it up because it isn't documented anywhere as, like I said,  
the system was designed soley for X11 and so it is assumed that only the  
X11 developers need to know how it works.  I spent years figuring out how  
to make it work in the first place, and so I gave up the project because I  
simply don't care to figure thigns out anymore.  The kernel is simply too  
much of a bitch.

This is why I understand why hardware manufacturers don't supply Linux  
drivers, whereas seemingly no one else does.  Everyone just uses SVGAlib  
or X11 or SDL which uses SVGAlib or X11, and so they have no idea what is  
actually involved.  I've actually done graphics in Linux, and so when I  
say it's a bitch, it's a bitch, and if you haven't created your own  
complete graphical system which runs under Linux, don't even begin to tell  
me I'm wrong because you simply do not know what you are talking about.

So, back to what I was saying...  The kernel's video code doesn't need to  
slack off in the way that seems to be popular with the kernel people.   
It's like the VESA framebuffer driver.  It can't change modes, it can't do  
console multiplexing, it can't even allow applications to say "I'm going  
to give you 24-bit RGB data" but instead requires them to look and see  
that today it is accepting 16-bit BGR data with 3 bits for blue, 9 for  
green and 4 for red, and also with the most signifigant bit in the least  
signifigant bit position, and then it is the task of the application to  
translate its video data into that fucked up format.  The idea of a driver  
is to make things easy for users, not to provide the bare minimum of  
support.  On one extreme we have a driver that does everything, and on the  
other extreme we have a driver that doesn't even exist, which is right  
next to a driver that does nothing it doesn't absolutely have to, which is  
the kinds of drivers Linux typically has.  This is why Windows wins.  In  
Windows, you tell Windows what format you want to supply your graphics  
data in and if it isn't what they video card is using, Windows translates  
it so that it is.  Linux needs to be as easy as Windows.  In particular,  
there is no reason why thousands of application developers should have to  
deal with translating video data formats when just one video driver  
developer can deal with it on behalf of everyone.

If Linux wants to beat Windows, it needs to beat Windows where it counts,  
and that isn't in the GUI interface.  Windows wins because it has the  
applications, and it has the applications because it's easy to develop  
software for.  Make it easy to create games in Linux, and Linux will have  
games.  Make it easy to develop office applications, and Linux will have  
office applications.  It's not about shiny buttons and wizards to do  
everything, it's about being able to request a video mode and get it no  
questions asked and especially no additional software or libraries  
downloaded and no reconfiguration of anything ever, just like you get in  
Windows.  If you write a game for Windows, your game can request a  
640x480x256 graphics mode and it gets it, not just on your Windows system  
but on every Windows system on the planet.  Linux absolutely does not have  
that, and that is why Linux does not have the games.  It's the same thing  
with other software.  The only software Linux has is the software that is  
actually not too difficult to get to work in Linux.  Linux has the Opera  
web browser, origionally a Windows-only application, because as long as  
X11 is working, getting any GUI software to work isn't too difficult, and  
in particular, TCP/IP is one thing that is really easy in Linux, so  
porting a web browser to Linux isn't a big deal.  We also have the Adobe  
PDF reader, because again, it doesn't have to do anything that isn't easy  
in Linux.  There's a Yahoo Messenger for Linux, again it's a GIU and  
network application, both of which are easy in Linux.  We have the  
Macromedia / Adobe Flash Player, but it uses audio in Linux, which isn't  
easy, and what do you know?...  The audio doesn't work for a lot of  
people, and when it does it's typically out-of-sync with the video.  They  
also haven't updated it since version 7, in all likelyhood because making  
it work is such a bitch that they can't convince themselves to care.

So why doesn't Linux have games?  Well, simply, it's because Linux sucks.   
It doesn't have to be that way, and I hope someone will fix it, but I'm  
not going to because the level of resitance to progress is too intense for  
me.  Despite my good arguments even this email will cause many responses  
telling me that I'm wrong.  Who wants to fix something when everyone is  
simply going to tell you that you're fixing something that isn't broken?   
I can't change the kernel without help from the kernel developers, and if  
they won't like what I'm doing, then why try?  The same applies to the X11  
developers, as my new system would be incompatibile with how things  
currently work, requiring X11 to be updated to work with the new system,  
but if they don't think that what I'm doing needs to be done, then I won't  
get any help from them either.  Not to mention, it's simply a project that  
is too large for a single person.  So I'm not even going to try.  Either  
someone else can do it, or it can simply continue to suck, I really don't  
care anymore.  I've given up on ever seeing a cool game in Linux or ever  
having as much fun writing software for Linux as I had when I used DOS.

Anyway, back to the list of stuff that needs to be done...

5.  Take that VT-100 emulator code from the kernel and make it into an  
ordinary application which runs in user space.  Run a copy on six or eight  
of the pseudo video devices created in step 4.  This will create six or  
eight pseudo TTY interfaces, on which the usual crappy login programs can  
be run, allowing people to log in and run bash and whatever other terminal  
applications they care to use.

Note:  There are two distinct ideas here, the "virtual console" and the  
"terminal emulator."  The virtual console is the result of software which  
connects to the kernel to access the real video device, and creates a  
dozen fake video devices for other applications to access.  These other  
applications are currently "terminal emulators" which emulate the current  
Linux VT-100, but it should be possible to use other applications as well,  
either a different terminal emulator, or perhaps simply X11 connected  
directly to the virtual console without the terminal emulator in the  
middle.

6.  Add to the terminal emulator a means for an application which is  
running under one of the virtual terminals to access the graphics  
interface of the virtual console it is running on.  There is no reason why  
running a graphical application should require a new virtual console,  
instead, simply provide a means for it to take control of the current  
virtual console for the time it is running.  The VT-100 emulator should  
detect when this application exits and restore the graphics back to the  
state it was in before the application changed the video mode, in case the  
application should crash before restoring the text mode, but more  
importantly because it shouldn't have to restore the text mode.  Ideally  
this interface should be identical to that of the virtual console so that  
applications can be run under the terminal emulator or directly connected  
to the virtual console without modification.  This can be accomplished  
simply by having an environment variable which tells applications where  
they need to go to gain graphics access.

7.  Remove all video code from X11.  It never should have been in there to  
begin with.  It should be using the same graphics API that every other  
application now has available.  Unfortunately I guess when X11 was first  
written for Linux, Linux had no kernel video driver, so X11 had to become  
the video driver.  That's just one more reason the state of Linux graphics  
sucks.  So remove the video driver from X11, and have it use the identical  
virtual console and terminal emulator APIs just like every other  
application.  Some other benefits of this would be that X11 would no  
longer have to be run as root, if X11 crashed then system would still be  
completely useable, and most importantly, writing a driver for the kernel  
provides access for X11.

8.  Create a virtual console application which runs under X11.  This would  
create exactly the same API which is available from the virtual console  
application that connects directly to the kernel, except that this one  
connects directly to X11.  With this, everything that runs outside of X11  
works inside of X11 without modification.  The same terminal emulator  
which runs outside of X11 would run inside of X11 as well, simply  
connecting to this virtual console application instead of the one outside  
of X11.  Programs which use graphics outside of X11 could run inside of  
X11 as well.  X11 would essentially have a "video emulator" which would  
allow people to never leave X11 if they did not want to because X11 could  
do everything that could be done outside of X11.

However, people will still want to leave X11.  A game running in 640x480  
is much faster than a game running in 1024x768, and a full screen 640x480  
is much better than a 640x480 window on a 1024x768 desktop.  Similarly, if  
a game only needs 256 colors, it's a waste of CPU power to remap that to  
24-bit color, and because games are largely limited by the speed of video  
RAM, using three times as many bits per pixel just makes the game run  
three times as slow.

That there is the biggest failing of Linux graphics.  Applications simply  
run better when they can choose their graphics mode, but that is extremely  
flaky under Linux because of the way that graphics are implemented in  
Linux.  The only reliable thing is to start Linux in a 1024x768x24  
framebuffer, which guarantees that X11 will be able to start up, but also  
requires everything to run in 1024x768x24 because X11 cannot switch video  
modes when in framebuffer mode.

The result is that game developers have every reason in the world to stay  
away from Linux.  Their games will be able to choose an appropriate video  
mode on only a minority of systems, and on the rest their performance will  
simply suck, assuming they work at all.  Using graphics in Linux should be  
easy and reliable, and it could be if it were simply designed well, as  
opposed to the current hack we have now where the kernel was initally  
written with no expectation of doing anything besides a VT-100 emulator,  
then X11 decided to do direct video access instead of modifying the  
kernel, and SVGAlib decided to go the same route, both eventually realized  
that VESA calls were the only reliable way to do it, and then some clever  
person half-ass implemented framebuffer acces in the kernel, but even  
screwed that up.  Linux graphics is currently a nightmare for anyone who  
wants to do anything besides spend the majority of their time getting  
graphics to work correctly for their graphics intensive application in a  
way that works on every Linux system.  I've looked into this myself and  
the only reliable way to do it is to do it all over again yourself.  So  
what we need to do is just pick up the current system, crumple it into a  
ball, and toss it in the trash can.

Yes, Linux's graphics support sucks as the open graphics people have  
apparently figured out, but it has nothing to do with the graphics cards  
and everything to do with the worst system design imaginable.  As such  
it's a bit pointless to worry about an open video card right now.  Fix all  
of Linux's other graphics problems, then create a new video card if you  
still want to.  At that point, writing the driver for it will be a piece  
of cake because you'll be able to use the same easy and simple interface  
that all of the hardware manufacturers have available to them and you  
won't have to ask anyone to volunteer because writing that video driver  
will be such an easy task that you can do it all yourself.