Thursday, 24 April 2014

One-file C++ Utilties

Summary:

Here I show off self-contained utilties I have accumulated for C++ over time. Included are small functions for:
  1. Directory manipulation and searching
  2. Creating a timer
  3. Simple performance stats
  4. Printing the type of an object.
  5. printf-like string manipulation
The name is slightly dishonest, but few-file C++ utilities didn't have the same ring.

Overview:


While working on lanarts I frequently had to write code that was simply an #ifdef between UNIX land and Windows land. I made sure to minimize the amount of #ifdef's inside the actual game engine, pushing it into small C++ classes or one-file modules

Time has passed and they have remained useful to me; I publish them here for anyone to use. I figure stand-alone C++ utilities, where you can simply paste one or two small files into your code are worth releasing. You can incorporate them into namespaces if the need is felt; I usually go global for convenience.

Note 1: If you find yourself hungering endlessly for classes like these, I heartily recommend Boost. Personally though, I like to make due if I can with smaller dependencies.

Note 2: Most of these are from C++03 usage, so if there's a better way to do it in C++11 I may have missed it.

1. Simple directory access


directory.h
directory.cpp

It uses tinydir.h:

tinydir.h

This works on Linux/Mac/UNIX-like systems and Windows.

Don't expect much options here, but it is quite complete. 'search_directory' provides recursive directory searching, retrieving a filename list. The name pattern takes wildcards.

'ensure_directory' can create a directory if one does not already exist, returning false on failure.

See the header file for details.

2. Simple timer utility

Timer.cpp
Timer.h

Ever messed about trying to have a simple timing class in C++?
This one relies on using gettimeofday on Linux/Mac/UNIX-like systems, and emulating it on Windows. The Windows code was from Stack Overflow.

Provides these member functions:

Timer
():
   Sets a timer,
get_microseconds() == 0 at this point.
void start
():
   Resets a timer, makes
get_microseconds() == 0 again
unsigned long get
_microseconds():
  
Time in microseconds, multiply by 10e-6 to get seconds. May want to change signature to 'long long' by editing the source.

Example usage:


Timer timer;
... stuff occurs ...
cout << "Seconds have passed: " << timer.get_microseconds() * 1e-6;


3. Simple performance stats

PerfTimer.h
PerfTimer.cpp
perf_timer.h

Simply, PerfTimer uses the above Timer class to create timers suitable for use with long-running methods (a map to the type name is used for convenience, but isn't super-fast).

Example usage:


Simply include perf_timer.h, and have:
PERF_TIMER();


at the start of your function. It will record simple statistics such as average time, max, standard deviation, etc.

Then, when you want to see the results, simply run:
perf_print_results();

And you will get fairly nicely formatted results. The advantage is this approach is enough to see obvious bottlenecks (ie, this function is taking 75% of the time) and does not require the use of external tools. 

Example Output:


func bool analyzer_follow_entity(AnalysisState&, int, double):
        >> total 3797.9561ms
        calls 2801
        average 1.3559ms
        max 23.2120ms
        std.dev +-1.8125ms, +-133.67%

4. Simple typename printing utility

typename.h
typename.cpp

Useful for templated functions, when you want to pretty-print an object's type. Note that proper use of a debugger may or may not be more appropriate; littering print statements still has its place, though.

Example usage:


template <typename T>
void my_template(const T& obj) {
       cout << "got object of type: " << cpp_type_name_no_namespace(obj);
}

5. 'sprintf' into an std::string


strformat.h
strformat.cpp

This provides a global function called 'format' (again, edit to your needs!) that takes printf-style arguments and returns an std::string. To me, this clears a pain-point where I prefer printf-style syntax for output, but cannot use it to make temporary strings in output functions.

Note that the code is not designed for efficiency for strings > 512 bytes, but it does guarantee correctness.

Example usage:

std::string my_str = format("(%d, %d)", 1, 2); // -> Goes to "(1, 2)"


What else?

I have a Lua wrapping library (C++03 compatible) very similar to luabind but with no dependencies, and quite small size. It may be worth talking about in a follow-up.

Monday, 16 December 2013

A Parameterized Attack on Programming


I will be frank; I don't like the programming language L. If I could stop people from using it directly I would, forcing them to first formulate their problem in a better language.

Practitioners are subtly but profoundly swayed to take shortcuts with correctness because of a lack of simple or idiomatic expression in L -- after which, of course, they are encouraged to compromise performance for some measure of the correctness gained back. Double-of-course -- correctness is everything!

Truthfully, there is little wrong with L, but more the culture surrounding it. By refusing to use anything but L, a mono-culture forms, and L is forced to handle more and more concept mappings that bring L to the very limits of (and, sometimes, off the charts of) its intended cost/benefit trade-off.

The economic viability of programming has given it a very dense and awe-inspiring history -- but correct language is essential -- all industries are doing what will inevitably be called 'early programming'.

Consider the culture of programming as a whole:

Money drives programming hours in a series of chaotic directions. Man-millenia become spent on redundant interfaces to obsolete versions of redundant programs by obsolete and redundant companies. Those observing these practices identify similar but frustratingly hard to generalize flaws over and over. It is discouragingly hard to incrementally improve something for which your system has no intelligence.

And of course, why bother with thinking about a gradual path of improvement when you face costly ad-hoc reformulation, creeping obsolescence, changing requirements, no market demand, et al staring at you. You can almost feel the boulder rolling back as you reach to push it up the hill. Individuals in these situations sometimes find solace in the delusion that since they are avoiding changes they are avoiding the problem of programming -- only to be pulled back in for one last 'hit' by the winds of change. And this describes our dense landscape of opportunity, where history is not only doomed to repeat itself but to synchronize at many points as industries struggle to solve problems in excitingly identical ways.

And that's precisely what's wrong with the programming language L, and it has almost nothing to do with the programming language L.

When our most fundamental set of tools are thought of, therefore spoken of, therefore implemented as constraints not strictly the minimum of our problem domain, we create (at least slightly) divergent paths of progress. As long as we continue to rely on competing ad-hoc imperative logic models to handle conceptually non-imperative tasks we have literally no hope for normalization (viz Halting Problem).

Only when we leave 'early programming' can the very confused non-software-oriented industries return to business as usual. Producing correct software is a minefield compared to ensuring correct manual labour. Once the designer for even a non-technical discipline can manipulate a set of sufficiently expressive symbols that appeal to their intuition without fear of being forced to reason about arbitrarily-imperative behaviour or any other ad-hoc idiosyncrasies, then we can say we are observing modern programming.

Saturday, 21 September 2013

Lanarts, Devblog for Release 13Alpha

I keep wanting to make grand long blogs describing Lanarts development in-depth. That doesn't seem to be working; I'll try shorter blogs.

Lately, there's some exciting stuff happening with Lanarts, and it'd be a shame not to share -- especially because I don't know how long it will take to stabilize all the new features!

Randomly Generated Overworld:
This happened much sooner than I thought! For those of you following, Release 12 introduced an overworld -- however, it was not randomly generated.

I really didn't know how I'd approach the overworld -- connectivity is tricky with non-grid layouts. The approach taken was fairly simple. The algorithm is basically just a bunch of random polygons, and lines connect nearby polygons. I used libxmi to draw the polygons efficiently, and baked it into my map generation library. From there, Lua code controls the map generation, and it it is quite well-performing, even for large maps.

New Stat System:
From a gameplay perspective, this is huge (and criminally overdue). This replaces the previous simplistic stat system (Melee/magic damage types, and resistance/effectiveness/damage/defence for each). Now possible are elemental damage types, bonuses against specific monsters, easily-definable custom status-types, equipment that can grant virtually any ability, etc.

I could probably write a whole blog about the nuances of the stat system, but essentially it was rewritten in clean Lua with many interesting possibilities. The stat system comprises of base stats, and components of these base stats, eg the inventory, result in bonuses to the derived stats. This way eg you can have a spell that is granted only when wielding an item, since it can be added to the derived copy of the stats. This derived copy is computed whenever needed.

Roadmap:Finally, the goals I laid out for Lanarts Release 1.0 have started to take shape. The two issues being tackled by this release plagued me for a long time, and I'm happy to have found elegant solutions to both of them (thanks to the expressiveness of Lua!)

Another, relatively unseen, issue that I have gone a long way to fulfilling is the ad-hoc nature of the Lua API. Recently, a lot of effort has been taken to clean-up and modularize the Lua part of the code. The Lua API continues to become more lower-level and general to take full advantage of Lua's expressiveness.

While development for release 13 is well underway, there is still lots to do. Unfortunately, there is a lot of balancing work ahead with the new stat system, but I have some promising ideas for automating this task.

Feedback:

Please direct any bugs, comments, or inquiries either as a github issue, a comment on this blog, or through IRC or email:

IRC Channel: #lanarts at freenode
Contact Email: domuradical@gmail.com

Feedback very welcome !

Thursday, 22 August 2013

Lanarts release 12, now with overworld!

New release, here is the download for Windows!
Source-code for this release: Source

This one was pretty delayed. It's about time I released a version with some of the large recent changes.

READ BEFORE PLAYING:
The game does not provide much instruction (the README is woefully outdated), but your mission currently is to clear the dungeon that has warning skulls near it. You begin near the first of two dungeons.

It is recommended you move with WASD, melee with H, fire spells with YUIOP, mouse controls for everything else. Mouse controls for spells are fairly suboptimal at the moment.

Additionally, 1 to 9 for inventory slots 1 through 9.
Right click is needed to drop items, however.

Screenshots and Changes


Major changes:
  • There is now an overworld with multiple dungeons!
  • As a result, almost all the areas in the game have been majorly revamped. The red dragon is now on the overworld, a different boss resides in the first dungeon.
  • There is now a new logo, courtesy of Kevin Siapno.
  • Two new spells - Minor Missile, a mana-conserving Mage spell, and Expedite, an archer spell that causes them to move much faster for a short time. 
  • Poison Cloud has been moved from the Mage to the Archer. 
  • Magic Blast is now a 3rd-level Mage spell.
  • Minimap is much more visible. You can click to adjust zoom level, right click to look around.
  • [Code-only] The Lua code has been refactored to be much more modular, and has grown significantly. The area generation code has been rewritten (and improved, in the process) in Lua.
Too many minor changes to list. If curious, you can always check the commit logs.

Special thanks to Matt Simon, Steven Trumble, putterson, and the 2013 Toronto Red Hat interns, for their help testing the game. Thanks as well to Kevin Siapno for the original logo.

Downloads and Discussion

Windows:
Download!

Linux:
Compiling on Linux is very easy!
Download the source here: Source
-or- if you want the very latest updates:
git clone http://github.com/ludamad/lanarts --depth 1

Then from the folder you extracted or cloned to:
./run.sh

This will build and then run the game. If you are missing dependencies on Linux (probably the case) please run either fedora-deps.sh for Fedora, or debian-deps.sh for Debian or Ubuntu.

I do not yet provide linux packages, please do kick me if they'd be useful for you.

Other systems:
The game has been written to be very cross-platform; however it has not yet been tried on Mac or other systems. Anyone willing to help try Lanarts on a different platform is asked graciously to email domuradical@gmail.com.

Feedback:

Please direct any bugs, comments, or inquiries either as a github issue, a comment on this blog, or through IRC or email:

IRC Channel: #lanarts at freenode
Contact Email: domuradical@gmail.com

Feedback very welcome !

Tuesday, 14 May 2013

A 'fork & exec' surprise


See my gist for a self-contained C++ example of the problem code:
https://gist.github.com/ludamad/5579602

So recently during my work at Red Hat I was looking into an issue with Java on Linux (with IcedTea7+ and Oracle's JDK).

The straight-forward seeming code


FileWriter writer = new FileWriter(someScriptName);
writer.write( ... some contents ...);
writer.close();
giveFileExecPrivileges(someScriptName); // eg, using UnixFilePermissions
Runtime.getRuntime().exec(someScriptName);
This code snippet creates a new text file, sets executable permissions, and then tries to execute it.

This code runs just fine when executed in a single thread. Introduce multiple threads (also running this spawning code) and every-so-often you will get an IOException with the message 'error=26 Text file busy'. This IOException means that Linux is refusing to execute the file because it is open for writing.

However, this is clearly the only place we ever open the file - and we promptly close it. So what's going on ?

An aside about executable text files


On Linux (and originally, UNIX), we can execute a text file by looking up the 'shebang' line, eg at the start of the file:

    #!/bin/bash

This tells the kernel which program to use to execute the file. You get 'text file busy' basically if you try to execute a file that is being written to. Normally this would be a text file (you can get this error with binary files as well, though).

Digging deeper


Unfortunately, you can stare at the Java code endlessly; it won't tell you anything. Luckily, we have OpenJDK. A quick foray into UNIXProcess_md.c shows the native implementation of Java's Runtime#exec method. It is implemented using the system calls 'fork' and 'exec'. (Actually, it uses 'vfork', in OpenJDK7, but it's not important.)

Fork, if you're not familiar, essentially creates a new copy of the process, with enough copy-on-write magic to do it efficiently. Worth noting, this child process gets a copy of all the file descriptors opened by the program.

To spawn the program we actually want, we then use 'exec'. This replaces the current process with the specified executable. However, we continue to inherit whatever file descriptors were already open in the new process.

Don't copy my file descriptors, please


Normal programs will take care that the file descriptors copies are closed upon 'exec'. This prevents unwanted information from leaking. Indeed, this is the case in Java-land, where all open file descriptors are marked with FD_CLOEXEC.

This solves a very important problem -- that file descriptors can leak into the child process. However, this does not prevent the copy from occurring in the first place!

Back to the code


Is this knowledge enough to pose an explanation of what's going on ? In fact, it is.
FileWriter writer = new FileWriter(someScriptName);
writer.write( ... some contents ...);
// Say another thread executes 'Runtime.getRuntime().exec(someOtherScriptName);'
// this forked process will have a copy of the file descriptor we are about to close!

writer.close();
giveFileExecPrivileges(someScriptName); // eg, using UnixFilePermissions
// It is very possible that the forked process has not released the copy here!

Runtime.getRuntime().exec(someScriptName);

Unfortunately, the fork from another thread can copy the file-descriptor, suddenly be set aside to let other threads execute, and the file will remain open!

See my gist for a self contained example: https://gist.github.com/ludamad/5579602

Avoiding problems


So what's the takeaway ? If you're going to fork & exec in a multi-threaded program, you cannot do any operations that require file-descriptors to be closed predictably. This is true even if only one of the threads does the process spawning!

icedtea-web 1.4 Released!


icedtea-web 1.4 is here! Lots of fixes and enhancements in this release. This release also fixes known regressions that occurred from 1.2->1.3.

Release announcement by Jiri Vanek:

Hi all!

After long and furious development, I'm finally proud to announce release of icedtea-web 1.4.

http://icedtea.wildebeest.org/download/source/icedtea-web-1.4.tar.gz
97c1d5f02f9a4ab19812a216f39e401a  icedtea-web-1.4.tar.gz

How we were dealing and what we plan to do, can be  checked on wiki:

http://icedtea.classpath.org/wiki/IcedTea-Web#IcedTea-Web_1.4
http://icedtea.classpath.org/wiki/IcedTea-Web#IcedTea-Web_1.5

New in IcedTea-Web 1.4

* Numerous improvements and enhancements in core and system of classloaders
* Added cs localization
* Added de localization
* Added pl localization
* Splash screen for javaws and plugin
* Better error reporting for plugin via Error-splash-screen
* All IcedTea-Web dialogues are centered to middle of active screen
* Download indicator made compact for more then one jar
* User can select its own JVM via itw-settings and deploy.properties.
* Added extended applets security settings and dialogue
* Security updates
  - CVE-2013-1926, RH916774: Class-loader incorrectly shared for applets with same relative-path.
  - CVE-2013-1927, RH884705: fixed gifar vulnerabilit
  - CVE-2012-3422, RH840592: Potential read from an uninitialized memory location
  - CVE-2012-3423, RH841345: Incorrect handling of not 0-terminated strings
* NetX
  - PR1027: DownloadService is not supported by IcedTea-Web
  - PR725: JNLP applications will prompt for creating desktop shortcuts every time they are run
  - PR1292: Javaws does not resolve versioned jar names with periods correctly
* Plugin
  - PR1106: Buffer overflow in plugin table-
  - PR1166: Embedded JNLP File is not supported in applet tag
  - PR1217: Add command line arguments for plugins
  - PR1189: Icedtea-plugin requires code attribute when using jnlp_href
  - PR1198: JSObject is not passed to javascript correctly
  - PR1260: IcedTea-Web should not rely on GTK
  - PR1157: Applets can hang browser after fatal exception
  - PR580: http://www.horaoficial.cl/ loads improperly
* Common
  - PR1049: Extension jnlp's signed jar with the content of only META-INF/* is considered
  - PR955: regression: SweetHome3D fails to run
  - PR1145: IcedTea-Web can cause ClassCircularityError
  - PR1161: X509VariableTrustManager does not work correctly with OpenJDK7
  - PR822: Applets fail to load if jars have different signers
  - PR1186: System.getProperty("deployment.user.security.trusted.cacerts") is null
  - PR909: The Java applet at http://de.gosupermodel.com/games/wardrobegame.jsp fails
  - PR1299: WebStart doesn't read socket proxy settings from firefox correctly



People who helped with this release (If I forgot somebody, please let me know!):


Deepak Bhole <dbhole@redhat.com>
Danesh Dadachanji <ddadacha@redhat.com>
Adam Domurad <adomurad@redhat.com>
Jana Fabrikova <jfabriko@redhat.co>
Peter Hatina <phatina@redhat.com>
Andrew John Hughes <ahughes@redhat.com>
Matthias Klose <doko@ubuntu.com>
Alexandr Kolouch <skolnag@gmail.com>
Jan Kmetko <jan.kmetko.ml@gmail.com>
Omair Majid <omajid@redhat.com>
Thomas Meyer <thomas@m3y3r.de>
Saad Mohammad <smohammad@redhat.com>
Martin Olsson  <martin@minimum.se>
Pavel Tisnovsky <ptisnov@redhat.com>
Jiri Vanek <jvanek@redhat.com>
Jacob Wisor  <gitne@excite.co.jp>


Special thanks to:

 * Adam Domurad - for deep investigations and  fixes in core, and in numerous classloaders or otherwise complicated bugs
 * Jan Kmetko - for initial design of splashscreen
 * Deepak Bhole and Omair Majid - for ever keeping an watchful eye on patches
 * to community - namely:
   -  Jacob Wisor and Alexandr Kolouch  - who voluntary offered and delivered Pl+De and Cz translation

And not finally to all who tested the pre and final versions for regressions


Best regards
  J.

Happy hacking,
-Adam

Wednesday, 17 April 2013

icedtea-web 1.2.3, 1.3.2 Released!



Newest incremental releases in 1.2.x & 1.3.x series for icedtea-web were released! Stay tuned for the much-improved icedtea-web 1.4, in about a month.

From the release announcement by Jiri Vanek (http://mail.openjdk.java.net/pipermail/distro-pkg-dev/2013-April/022790.html)
The changes well... are as they are. Not much, but at least something - as development was strongly 
focused to HEAD.
 From this point of view, I strongly encourage every user to move to 1.4 as smoothly and quickly as 
possible (will be based on future head anyway).
Also please note that 1.2 is getting close to end of its life.

http://icedtea.wildebeest.org/download/source/icedtea-web-1.3.2.tar.gz
http://icedtea.wildebeest.org/download/source/icedtea-web-1.2.3.tar.gz

md5sums:
ab56d94251975a9bb5359ea85136ea79  icedtea-web-1.2.3.tar.gz
94ce02c42c1e4d1411357fb3c1014f67  icedtea-web-1.3.2.tar.gz

New in release 1.3.2 (2013-04-17):
* Security Updates
   - CVE-2013-1927, RH884705: fixed gifar vulnerability
   - CVE-2013-1926, RH916774: Class-loader incorrectly shared for applets with same relative-path.
* Common
   - Added new option in itw-settings which allows users to set JVM arguments when plugin is 
initialized.
* NetX
   - PR580: http://www.horaoficial.cl/ loads improperly
* Plugin
    PR1260: IcedTea-Web should not rely on GTK
    PR1157: Applets can hang browser after fatal exception

For details you can see changesets:
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/2d76719a5e4d
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/88fb945c9397
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/25dd7c7ac39c
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/25dd7c7ac39c
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/19f5282f53e8
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/ca8b00cca4c3
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/f63bdf513e9d
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.3/rev/b619cda99360

New in release 1.2.3 (2013-04-17):
* Security Updates
   - CVE-2013-1927, RH884705: fixed gifar vulnerability
   - CVE-2013-1926, RH916774: Class-loader incorrectly shared for applets with same relative-path.
* Common
   - PR1161: X509VariableTrustManager does not work correctly with OpenJDK7
* Plugin
   - PR1157: Applets can hang browser after fatal exception

For details you can see changesets:
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.2/rev/34b6f60ae586
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.2/rev/cb58b31c450e
   * http://icedtea.classpath.org/hg/release/icedtea-web-1.2/rev/cd4a9f25808e

Please consider for next few hours HEAD as frozen (until you see similar commits;)

Thanks to everyone who helped with this releases:
   Deepak Bhole
   Adam Domurad
   Jana Fabrikova
   Tomas Hoger
   Omair Majid
   Jiri Vanek
Direct all bug reports to:
  http://icedtea.classpath.org/bugzilla/
Direct all inquiries/comments/patches to the distro-pkg-dev mailing list:
  http://mail.openjdk.java.net/mailman/listinfo/distro-pkg-dev

Happy hacking.