I do my development under Windows using Microsoft Visual C++. If you use VC++ to create a new project called "Hello", you get a directory structure like this:
/Hello Hello.cpp Hello.data Hello.sln Hello.sdf Hello.suo Hello.vcxproj Hello.vcxproj.filters Hello.vcxproj.user /Debug Hello.exe ... many compilation intermediate files /Release Hello.exe ... many compilation intermediate files
The green files are the ones I would release as part of the executable demo zip file. The blue files are part of the source. All the other files are temporary files I don't want to include with either the source or demo.
I personally don't like seeing all these VC++ project files mixed in with my source code files. One project file would be OK, but as you can see, VC++ creates six of them. Plus a seventh that appears when the project is open in the IDE. I also don't like mixing source files with data files (texture images, option files, etc.)
If I were just working on Windows, this would only be a matter of taste. Since I'm also compiling under Linux and Mac, it's more of a nuisance. I don't really want the Linux project files or the Mac XCode project files mixed into the Windows project files. So I changed the directory structure by moving my files up into a parent directory. That looks like this:
/Hello /Source Hello.cpp /docs Hello.data /BuildWin Hello.sln Hello.sdf Hello.suo Hello.vcxproj Hello.vcxproj.filters Hello.vcxproj.user /Debug Hello.exe ... many compilation intermediate files /Release Hello.exe ... many compilation intermediate files /BuildLinux makefile /Objects Hello (executable under Linux) ... many compilation intermediate files /BuildOSX XCode project files ... too many files to list!
This is the structure you'll see if you download any of the source zips on this project. It has worked well enough until now.
A couple of weeks ago, I got Shamus Young to try my GUI code in his project. That led to an unpleasant discovery -- my VC++ project files didn't work for him! There were two reasons.
In the Release build, I discovered some time ago that Windows XP will not run my demos because they expected "MSVCR100.dll". It's odd that all the other Windows DLLs referenced by OpenGL or DirectX are there, but not the C runtime. However, you can eliminate this requirement by changing some compiler options, to make the project link to the C runtime statically, instead of via the DLL.
This is important, since otherwise I think I'd have to write a Windows installer to put my version of the C runtime into the System directory.
When I use a static C runtime, everything works fine for me and I can produce .exe files that run on all versions of Windows. Unfortunately, my libraries no longer link with programs compiled with the default flags. In order for Shamus to use my libraries, I had to go back to the default settings.
In the Debug build, I use my own versions of the C++ new and delete operators, in order to track maximum memory use and check for memory leaks. This also affects the libraries and interfered with anyone else using them. So I had to remove this feature as well.
Since I really wanted both of these features, I decided to add them a different way. VC++ lets me define new configurations, so I added DebugMemory and Distrib builds that worked the way I wanted. This lets other programmers get the standard builds that work for them, and lets me continue working the way I like. However, now there are four targets and even more junk files in the build.
The Release Process
At the beginning of this project, there was a single VC++ project file which had everything -- both the application code and my framework code. I had even integrated a version of the Jpeg image library into my project, so there was only one thing to build.
Later, I had multiple projects (Crafty, McrView and SeaOfMemes), so I had to split the framework away as a separate library. This is a nuisance when debugging, but I didn't want multiple copies of that code in each project.
Sometime later, when I was bringing in more open source libraries like Ogg, Vorbis, ZLib and FreeType, I split it all into multiple VC++ projects. In addition to the open source libraries, I had two libraries of my own -- mgUtil and mgFramework
In Part 55 I decided to split the framework into pieces, to increase the chances that people would use it. I also added the GUI code. So now there are many projects.
The games: SeaOfMemes, Crafty, McrView, DontHitMe
That's a total of 25 projects that have to be built to compile all of the source code. And there are versions of each on Windows, Linux and Mac.
My release process goes like this:
If I discover that something doesn't work on Linux or Mac, I fix the code or the project file, copy them back to Windows, and start over on this whole process (because the source has changed.)
So a release with the current setup requires compiling projects 73 times, copying files from source to demo directories 36 times, and creating 20 zip files. This whole process takes at least a couple of hours and I'm more and more reluctant to do it. Which is why there hasn't been an update to GitHub in three months.
Improving the Process
There is unfortunately no shortcut for testing this on all the platforms and going through some kind of build process on each machine. However, it could be simpler if I'm not quite so fussy.
Working from the end result backwards, I should do something about the 20 zip files. I broke it into that many files because I assumed 1) people don't want to download everything (most don't want source code), and 2) a big download is slow for users, and 3) uses more bandwidth on the web site.
Right now, if I put everything into a single zip per platform, it would be about 15 megabytes, with another megabyte at least added for each new demo. I suspect this is tolerable for most people with high speed connections. Since I don't have thousands of people downloading the demos, it probably isn't a problem for the site (I currently only pay $11 a month to Hosting Matters, since the usage is so low.)
To make each zip easier to build, I need to coerce VC++ and XCode into building something like the final structure. That means putting all the temporary files and databases outside the project tree. I've been playing with VC++ today and think I can get what I want. I know I can get makefiles to do what I want under Linux. I have no idea if I can get XCode to move all the crap files it produces out of the project directories.
To make a zip with both source and executable more understandable to naive users, I need to rearrange it a bit. I want something like this structure:
/Top /SeaOfMemes SeaOfMemes.exe options.xml /docs /Code /Source /BuildWin /BuildLinux /BuildOSX /Crafty ... a directory per demo.
Finally, I need to do something about compiling all these projects. Under Linux, I assume I can put together some makefile that calls all the project makefiles. Under VC++ and XCode, I have no idea if I can automate things. VC++ supports nested projects, but I'm not sure it will do what I want. I don't have any idea about XCode.
I could switch everything to a tool like CMake and perhaps have a big makefile of makefiles that works on all platforms. I'm reluctant to dive into the documentation and figure it out on all three platforms, but if it saves me hours each time I want to do a release, it would be worth it. Any suggestions?
I haven't decided what to do about this mess. I'm still doing GUI work with Shamus and work on DontHitMe. I still have to get the new refactored framework debugged on Linux and Mac. Last week was another lousy health week, but I'll have something for you soon.
UpdateA lot of the advice in the comments (and since the beginning of the project) boils down to "Ditch the Visual C++ build process and do it with tool X." I have had reasons for not doing this:
So I have resisted doing this conversion. However, the build process was getting so tedious that I figured it was worth it.
Under Linux, I have existing makefiles for each build. I just need to write a larger "publish" makefile that builds all the pieces and then creates the zip files. It's much simpler than doing it by hand, and I really should have done that months ago.
One of the comments (from Joe_W) pointed out that nested makefiles are a bad idea in general, but that doesn't apply here. I'm just building all my libraries in the correct order, then building all the applications. It's exactly what I would do by hand, and there are no weird dependencies.
I can't really say why I didn't do this sooner. The project just sort of grew, from one to two to ten makefiles. I never use the Linux machine except when publishing a new part, so it just wasn't much of problem. At 23 makefiles, I hit my limit!
My real problem was Windows and Mac. I knew there were command line interfaces to the compiler and linker, but as I said, it meant learning all the options and having to mess with a new tool. I had it in my mind that there was no way to build the VC++ or XCode project directly from the command line, only from within the IDE.
Well as they say, if you want to find something, it sure helps to look. Googling "Visual Studio Command Line Interface" took me to MSBuild, which does exactly what I need. Give it the project file and configuration, and it builds the project, with all the options I've set interactively. No need to maintain two versions or regenerate the project file from CMake source. The same thing is true on the Mac, where the tool is called xcodebuild.
So now I can create my "publish" makefile on all three platforms without changing any of the existing project files. I've finished the one for Windows, and it sure beats building all the projects and creating all the zips interactively.
Of course, now I feel like an idiot for assuming there was no way to do this and wasting time all these months.
Let us never speak of this again...
blog comments powered by Disqus