This made me giggle like a school girl. Finally, my tax dollars got put to good use. NASA just released several projects open source. I’ve already been digging through their code and am absorbing it like a sponge. Very cool stuff: http://code.nasa.gov/
One of the biggest hurdles I’ve had to overcome in applications that do file logging is the the ability to write to a log file from one process or thread and be *read* from another process or thread in near real time. Its all about timing. If the timing is a little off, a failure is going to occur due to file locking problems. Even when both processes or threads are accessing the file with as little locking as possible.
Sooner or later you’re going to get an exception (usually System.IO.IOexception). I solved this using a transactional method that incorporates the use of a ReaderWriterLock and a specialized type of collection called a Queue. A ReaderWriterLock (System.Threading.ReaderWriterLock) is a special type of lock that supports single writers and multiple readers. What this means is: You can acquire a lock for single thread doing writes, but still allow multiple concurrent threads to read.
A Queue (System.Collections.Generic.Queue<T>) is a type-safe collection of First In, Last Out objects. Yes, this is essentially a FIFO buffer. For our purposes, we’ll use this to queue a message string which we will retrieve when we are ready to actually write the message to a file. The idea is to use a class that starts a queue processor, of sorts, that will take messages out of the queue in a first come, first serve manner and write each message to the file immediately. What this means is the write buffers created by System.IO.StreamWriter and System.IO.FileStream will be immediately flushed to disk instead of hanging out in the buffer until a later time. The only buffering we will be using is the Queue itself. In doing so, we help to ensure that messages are stored before some condition such as a catastrophic error or unexpected shutdown occurs. So with the queue running on a separate thread, we can submit messages to the queue and then the queue processor will process them as they come in. In this way, we avoid collisions because only one thread is writing to the file at a time.
This scheme would allow for a thread in a Windows Service or daemon (for example) to write to the file while allowing another program using a sentinel (such as a FileSystemWatcher) to monitor changes to the file and then load the file changes into a viewer of sorts. You could have multiple viewers reading the file while the service is writing to it. This is particularly handy if you have some kind of monitoring or management application or maybe even an active log viewer monitoring the file being used on multiple computers across a network keeping an eye on that log.
Now because the queue processor is going to run on its own thread but will be given data by another thread, the best way to handle errors in the processor thread is to raise errors in the form of an event. So when our queue processor encounters a corrupted state or a fatal error, we will catch the exceptions and then pass the exception object via an event that fires. Then in the application that implements this logger can handle the events and react accordingly. The logger also has a flag to indicate whether or not it is initialized.
The processor will do the following:
1.) Acquire a ReaderWriterLock.
2.) Open the log file for read/write access.
3.) Acquire a thread lock.
4.) Dequeue the message.
5.) Release the thread lock.
6.) Write the message to file and flush buffers.
7.) Release ReaderWriterLock
8.) Repeat for next message in queue.
This essentially makes up a transaction. You submit the message to queue using a method call, then a subroutine running on another thread (message processor) processes all the messages currently in the queue and starts over. I’m not going to go into all the details of a how ReaderWriterLock or Queue works here….. you can read the MSDN articles that I linked above. I’m going to demonstrate my implementation of it. Lets start with the processor routine (some code omitted for brevity):
// The namespaces we need to import: using System; using System.Collections.Generic; using System.IO; using System.Security; using System.Threading; //Here's some variables that we'll need private static String _logFilePath = String.Empty; private static Boolean _initialized = false; private static Boolean _enableTimestamp = false; private static Thread _writerThread = null; private static Int32 _writes = 0; private static Int32 _writerTimeouts = 0; private static ReaderWriterLock _rwl = new ReaderWriterLock(); private static Queue _msgQueue = new Queue(); private static Mutex _logMutex = new Mutex(); private static readonly Object _writeLock = new Object(); // We raise this event if an error occurs. public static event LoggerEventHandler Failed; // ...and here's the queue processor method. private static void LogLoop() { FileInfo logFile = new FileInfo(_logFilePath); // .... code to create log file if it does not exist goes here. // The outer loop. We continue until we are no longer initialized. Boolean succeeded = false; String message = String.Empty; while (_initialized) { try { // Acquire the lock (timeout in 100ms) _rwl.AcquireReaderWriterLock(100); try { // Process the messages in the queue until empty. while(_msgQueue.Count > 0) { // Create a FileStream in append mode. Append mode can only be used with FileAccess.Write access. // We use FileShare.ReadWrite so other processes or threads can read from it even if we are currently // writing to it. using (FileStream saveStream = new FileStream(logFile.FullName, FileMode.Append, FileAccess.Write, FileShare.ReadWrite) { // Make sure we can write to the stream. if (saveStream.CanWrite) { // Create the actual writer, dequeue the message, then write // it to the file immediately. using (StreamWriter writer = new StreamWriter(saveStream)) { // Automatically flush the buffer when done. writer.AutoFlush = true; // Clear the message string of any previous contents message = String.Empy; // Acquire the thread lock. lock (_writeLock) { try { // Double-check both inner and outter loop conditions to make sure // they haven't changed since the last time we checked, then // dequeue the next message. if ((_initialized) && (_msgQueue.Count > 0)) { message = _msgQueue.Dequeue(); } } catch (InvalidOperationException) { // The queue has been emptied since we last checked _msgQueue.Count. // This is a rare but possible condition. This could, for example, happen if // Stop() was called since we last checked the _initialized flag (outter loop), // since the Stop() method clears the message queue. Since we double-check the // loop conditions right before dequeue, this should almost never happen. } } // If the message is not null or empty, write it to disk. if (!String.IsNullOrEmpty(message)) { writer.WriteLine(message); saveStream.Flush(); succeeded = true; } } } } // Getting here, we had a successful write operation. if (succeeded) { Interlocked.Increment(ref _writes); } } Thread.Sleep(20); } catch (Exception ex) { // We actually want catch as many possible exceptions, but we'll use 'Exception' as an example. Failed(null, new LoggerEventArgs(ex)); } finally { // Release the writer lock. _rwl.ReleaseWriterLock(); } } catch (ApplicationException) { // The lock request timed out. Interlocked.Increment(ref _writerTimeouts); } } }
As you can see, we created our queue to hold the messages with the statement:
private static Queue _msgQueue = new Queue();
Now since the queue processor runs in its own thread, we’ll need a method for initializing it before we can actually submit any messages to it. The initializer’s primary goal will be to perform any necessary checks prior to lift off, and will then create the thread and start it. Like so:
public static void Start() { // If we've already been initialized once before, then // we do nothing here. This also eliminates the need to // catch a ThreadStateException when calling the Thread's // Start() method because the _initialized flag should // only get altered by the Start() and Stop() methods // of this class. If _intialized is false, then the // the state of IsAlive on the thread should also be // false and/or the Thread object should be null. if (_initialized) { return; } // Create the logger thread and start it. _writerThread = new Thread(new ThreadStart(LogLoop)); _writerThread.IsBackground = true; _writerThread.Name = "msgLogger"; try { _writerThread.Start(); _initialized = true; } catch (OutOfMemoryException) { // We could not allocate memory for the new thread. // Just re-throw the exception, preserving the stacktrace. throw; } }
Since we have a start method, we will also need a stop method. The stop method should empty the message queue and then abort the thread we create in the start method:
public static void Stop() { // Do nothing if we are not currently initialized. if (!_initialized) { return; } // Try to stop gracefully first. We use a mutex to help // ensure that we can alter the state flag and empty the // queue. In doing so, the queue processor should abort // quietly on its own. // Note: The handler for the 'Failed' event should call this // this method so that it does not continue on in a // corrupted state. _logMutex.WaitOne(); _initialized = false; Thread.Sleep(100); _msgQueue.Clear(); _logMutex.ReleaseMutex(); // If we're still initialized, then force the thread to abort. if (_initialized) { if (_writerThread != null) { try { if (_writerThread.IsAlive) { _writerThread.Abort(); } } catch (ThreadAbortException) { // By virtue of calling the Thread.Abort() method, // a ThreadAbortException will occur. Since we // expect this to happen and do not intend to // reset the thread, we just swallow this and // and move on. } finallly { // Get to the choppa! _writerThread = null; } } } // If we were still initialized before, then we aren't now. _initialized = false; }
Now that we have all that in place, we’ll need a method for getting messages into the queue so that they can be processed.
public static void WriteEntry(String message) { // Stop right there! If we haven't been initialized yet, then we are // in a bad state to be trying to write to the log. if (!_initialized) { throw new InvalidOperationException("Logger not initialized. You must start the logger before attempting to write."); } // .... code for validating log file path here. // Convert null strings to empty strings. if (message == null) { message = String.Empty; } // If timestamping is enabled, the we prepend the // message with a YMDHMS timestamp. if (_enableTimestamp) { message = String.Format("[{0}] - {1}", DateTime.Now.ToString("yyyy:MM:dd:hh:mm:ss"), message); } // Queue the message for writing. // We ensure the commit occurs by locking the thread until // we are done. lock(_writeLock) { _msgQueue.Enqueue(message); } }
That’s it! So to use this thing, you call the Start() method first, then call the WriteEntry() method and when you’re done logging or if the Failed event fires, you call Stop(). You’ll obviously want to put the above code into your own static class (it is very important that this class be static) and include some public accessors for reading the _initialized flag and setting/getting the _enableTimestamp and _logFilePath values. You probably noticed that we passed a custom event arguments object to the Failed event. The LoggerEventArgs class is just used to pass the exception and/or message containing the details of the failure. The following is the code for the LoggerEventArgs class:
using System; public class LoggerEventArgs : EventArgs { private String _message = String.Empty; private Exception _exception = null; // Constructors. public LoggerEventArgs() : base() { } public LoggerEventArgs(Exception ex) : base() { this._exception = ex; } public LoggerEventArgs(String message, Exception ex) : base() { this._message = message; this._exception = ex; } // Properties. public String Message { return this._message; } public Exception EventException { return this._exception; } }
We can’t fire the event without the delegate to marshal it to the handler, so here is the necessary delegate:
public delegate void LoggerEventHandler(Object sender, LoggerEventArgs e);
Sweet, right? ”But Cyrus, how do we actually use this thing?” Well, assuming you named your class something like “MessageLogger”, then you would do the following:
void Main(String[] args) { MessageLogger.EnableTimeStamp = true; MessageLogger.LogFilePath = "c:\test.log"; MessageLogger.Failed += new LoggerEventHandler(MessageLogger_Failed); MessageLogger.Start(); MessageLogger.WriteEntry("Hello World!"); MessageLogger.Stop(); } void MessageLogger_Failed(Object sender, LoggerEventArgs e) { Console.WriteLine("Oh no! The logger crashed!"); Console.WriteLine(String.Format("Error message: {0}", e.EventException.Message)); MessageLogger.Stop(); }
Tada! Good stuff. I am sure this can be improved, but it works great for every application I’ve implemented it in so far. It has good performance and has been quite reliable. Even if something bad happens I can restart the logger (I use a timer for this) by calling Stop() and then Start() again. There is a good possibility that the condition that caused the failure in the first place can be caught in the next call to Start(), assuming you’ve put all the necessary checking in there. Happy code-slingin’!
Yep…. been a while. I’ve been working on some pretty crazy projects lately, most of which I’m not allowed to talk about (signed the ‘ole NDA), but needless to say pretty neat.
Anywho… Thought I’d get on here and have a look-see only to find out that the forums have been totally drowned in spam. Good times. Looks like there are only a handful of legitimate posts too. So basically no-one is legitimately using it.
At all.
*sigh* So I set out to ban and then delete all the user accounts responsible for the spam and then I realized, why bother? Of course I realized this *after* I went through all that trouble. So there will be some changes to the ‘ole CyrusBuilt coming soon. More than likely the forums will disappear entirely. I’ll also be installing some upgrades while I’m at it.
This is an absolute MUST READ for everyone… not just geeks/I.T. professionals/computer technicians. What? Don’t just sit there reading this… go read THAT.
My hats off to you, Steve Cassidy. You’ve done a wonderful thing by writing that article, but I fear that while all of us in field fully understand and agree with your plight, those who are not in the field will continue on completely oblivious. *sigh*
After receiving some feedback, I’ve fixed a couple small bugs, and a couple features, and done some code cleanup. Version 1.0.0.4 is now available for download on the downloads page.
Changelog for v1.0.0.4:
- [Added] “Now” buttons next to both date fields on the “Details” tab, which set the fields to the current system date.
- [Added] Double-click handlers for both ListView controls in the “References” tab (as requested by “LurchMan”), but are not yet working (see code comments for details).
- [Fixed] Proper indentation for description lines in header comment in newly created project scripts.
- [Added] This changelog.
- [Fixed] autoit3dir path in au3.properties file for 32bit installations. (Thanks to “Fantastic” for reporting this).
Made some minor improvements and bugfixes. New version is available in the downloads page. Additionally, I released my dark theme for SciTE (just called “Black”), which is also available in the downloads page. It’s a SciTEConfig file, so just copy it to your SciTEConfig directory and then apply using the SciTEConfig tool.
See my previous post for details/screenshots.
I am proud to announce that I have released AutoItProjectCreator v1.0.0.2 to the public and is now available for download. AutoItProjectCreator is a tool written entirely in AutoItV3 that is designed to be an addon tool for SciTE that facilitates creating and editing AutoIt “projects”. Currently, AutoIt does not have a full-blown IDE like Microsoft Visual Studio, but it has many tools that provide much of the functionality that you find in an IDE such as a code editor, form designer, compiler, etc. Most of these tools can be found either included with AutoIt or included in the SciTE4AutoIt3 package.
However, as an AutoIt programmer who uses the language to develop both commercial and non-commercial software, whenever I start working on a new program, I create a “project”. I do this with any language I code in. I define a project as being a folder structure containing source files and documentation related to the program being developed. So when I start a new AutoIt “project” I typically end up manually creating the same folder structure as every other AutoIt project, then the script which I tend to start out the same as every other AutoIt script I write (commonly used #region names, header comment block format, as well as certain constants and functions), and then possibly also create a configuration file and/or installer script if I’m building a full-blown app.
As a result of this redundancy, I set out looking for a tool worked kind of like Visual Studio in the sense that it creates a folder structure and source files, associates any references, and also stores project specifics in a special file call a “project file”. In VS, you typically have a “solution” with one or more “projects” in the “solution”. I did not see a need to go as far as implementing a “solution” scheme, but the concept of “projects” seemed valuable to me. AutoItProjectCreator is designed to be used as a SciTE4AutoIt3 addon but can also be run standalone. The installer will also automatically associate the *.au3proj file extension with AutoItProjectCreator so that you can open a project by simply double-clicking on your project’s project file.
This is the first release….. I do not expect it to be perfect….. but so far it suits my needs just fine. If you think you could benefit from such a tool, then I encourage you to download it. Both 32bit and 64bit versions are available as well as the source code, all of which is released under GPL v2.
Enjoy!
So…. now that I regularly use Linux, Snow Leopard, and Windows 7 on a (almost) daily basis… I decided to try and tackle cross-platform development again. Thanks to metasploit, I’ve been doing a whole lot of digging around in the Ruby source code that it is comprised of, as well as writing some of my own hacks in an attempt to better understand the Ruby language. I’ve also been pouring through the source code of the Diaspora project.
So I decided to dive in a lil deeper and look into writing GUI applications on in Ruby. I looked at Shoes and I have to say…. its pretty darn nice. But I wanted something a little more rich than shoes and I wanted a nice GUI designer to go with with it. Enter Qt4. I gotta tell ya….. the Qt4 SDK is DAMN amazing. I love love love it! Now how do I make Qt4 and Ruby work together you might say? Well that is what Qt4Ruby project is for. The way this is supposed to work is:
I only have one problem with this….. I can’t get the damn thing to work! The problem isn’t with Ruby or Qt4. The problem seems to lie in the Qt4Ruby package. I’ve installed Ruby 1.9.x and Qt4 on Linux, WinXP, Win 7, and Mac OS X (Snow Leopard) hosts. I also installed the latest available Qt4Ruby package on all of them as well (at the time of this writing was: qt4-qtruby-2.1.0). On the Windows hosts I tried just using the latest Qt4Ruby GEM, which never issues any errors during installation and the ‘rbuic’ works flawlessly. But…. if I “require ‘Qt’ ” or “require ‘Qt4′ ” from IRB or from a script, I get an epic fail indicating that there is no file to load. Same problem on Linux. I even tried this using the built-in JRuby runtime in NetBeans IDE (on all 3 platforms) to no avail. On Snow Leopard, the problem is even worse…..
On Snow Leopard, there does not seem to be any kind of binary package of Qt4Ruby, so you have to download the source and compile and install it yourself. I’ve tried this every which way I can think of. I even Googled all over the place, read a ton of blogs, forums, and several articles, tried every CMAKE argument I came across and no matter what, I get a failure due to a ‘missing’ AUTOMOC4 dependency, which is bullshit because AUTOMOC4 exists on my system and I can see it sitting on my filesystem.
“But dude…. AUTOMOC4 is part of the KDE Support…. you have to install the KDE Support package”. First of all….. why is this required? It wasn’t required on Windows or Linux when I tried building from source on those hosts! Secondly, just to prove a point, I made sure I installed all the KDE4 packages along with the latest CMAKE packages using Fink. I can successfully run KDE4 in X11 on my Mac…. but I STILL can’t build Qt4Ruby on Mac. I’m not the only one either. I read several unanswered forum posts all over the place from people in the exact same boat as me. So here’s the thing: Is this simply not possible?
I like Ruby…. and I like Qt…. but can I not use them together? I’ve seen a multitude of examples using Qt3 and/or Ruby 1.87. But I want to use the new version of Ruby and the new version of Qt and thus also the new version of Qt4Ruby. Can they not all be used together?? I sort of expected some difficulty on at least one of those platforms but not all three. What am I missing?
Anybody out there got any ideas? I’m effin stumped and I’ve done about all the Googling I think I’m willing to do. I guess I may just have to walk away from this thing all-together. It’s a damn shame really. I had such promise.
*sigh*
I know, I know, I know…… I know what yer thinking…. “Why????”. Well, as much as I am against it, I didn’t have a choice. A client demanded we deploy it at one of their facilities despite all the sensible arguments we had against it. All that aside, I figured if I had to install it on 40 machines, I was at least going to do a mass deployment from remote and not go around installing it one at a time. Screw that.
So I went to work figuring out a way to install it quietly and without user intervention. This turned out to be far more difficult than I had planned. But, in the end, I hashed out what I believe to be a fairly straight-forward solution. Before I go into this tutorial, you’re going to need some tools….
Optionally, another nice thing to have is Notepad ++, which is more like a code editor than Windows Notepad. It isn’t a full-blown IDE or anything, but it is capable of syntax hi-lighting and has support for a multitude of languages. I find it handy for editing config files, etc.
Now that you have the tools, install everything -except- AIM 7. Once you’ve done that, kick off the AIM 7 installer and then STOP.
Stop when you see this screen
Now look for “setup.exe” in your Task Manager and kill the process. When the setup launches, it properly unpacks the contents of the installation package to a subdirectory in your local temp directory. On my Windows XP box, this path is “C:\Documents and Settings\<user name>\Local Settings\Temp\AIM_7.3.14.1″.
At this point, I created a project folder to hold the contents of the package. You can create it anywhere (not inside the unpacked AIM folder). I called mine “AIM” and then created a subdirectory called “source” and another called “build”. Under the “source” directory, I created another called “package”, which I then copied the contents of the AIM7 folder in my temp directory to. Now if you take a look at the “setup.ini” file in your “package” directory, you’ll notice 3 sections at the end of the file called “SEARCH”, “HOMEPAGE”, and “IETOOLBAR”. These are optional components that the AIM7 asks you to install. Simply comment these entire sections out. Like so:
Note the commented sections.
Once you’ve made the changes, save the file. This will give the installer nothing to prompt you for at the beginning of setup AND prevent the installation of AIM’s extra crap. Now launch XN Resource Editor and open “gui.dll” in the “package” folder. Locate dialog # 107 as shown below:
You need to completely remove this dialog. Simply right-click on the dialog in the tree and then click “Delete Resource”. Save the changes. By removing this dialog, we get rid of the prompt at the end of the setup basically just asking us to click “OK”. At this point, we’re done hacking the installer, now we can move on to the fun stuff. First, we need to re-package the contents as a self-extracting archive. Launch PeaZip, then click File–>Create Archive. Choose “self-extracting” as the type, then drag the entire “package” directory onto the PeaZip window, then click the “create .sfx” button to create the package. When asks where to save it, choose your “build” directory we created earlier. You should now have a file called “package.exe” in your “build” folder.
Now we’re going to need a script to kill any existing AIM processes at runtime, then unpack the “package.exe” file and run the “setup.exe” program. Enter AutoIt. I love this scripting language. Its free, lightweight, very fast, extremely powerful and can be compiled to a standalone executable. You can download my script here. Feel free to write your own that accomplishes the same task, or use mine directly, or use mine as an example.
Essentially what the script does is kill any previous AIM process versions using a helper function, then creates a temp directory and moves the “package.exe” file to it. Once there, it unpackages the package into the temp directory (quietly). This will create a “package” subdirectory inside the temp directory containing the package contents we modified earlier. The script then runs the “setup.exe” file inside the “package” directory and waits for it terminate. Once the process terminates, it removes the “package” directory and the “package.exe” file.
Once you have your script, compile it as “aim_silent_setup.exe” in your “build” folder. You can do this by clicking Tools–>Compile from the AutoIt code editor (SciTE4AutoIt3). You should now only have 2 files in your build folder, “aim_silent_setup.exe” and “package.exe”. Now you’re in the homestretch.
Launch SFX Maker and click the “Directory” tab. Choose your “build” folder for the “Directory Path” option, then choose “aim_silent_setup.exe” for the “File to Run” option, then choose where you want to save the SFX (silent self-extracting installer), and then choose the “aim.ico” file from your source “package” folder to use as your icon. See below:
Ready to build the package.
Now all you have to do is click the “Create” button and it will compile the final package. At runtime, this package will unpack our 2 exe’s and then run the “aim_silent_setup.exe” which does the rest. Once “aim_silent_setup.exe” is finished, the SFX will cleanup our 2 exe’s from the temp directory it unpacks them to.
And there you go! A self-contained installer that won’t prompt the user and runs (mostly) silent. I tested running this without an interactive session with SYSTEM privileges and it seemed to work just fine. I had no trouble deploying to the 40 workstations I had to get it installed on via a DeepFreeze maintenance window.
Caveat:
If an error occurs during the AIM setup, it is possible the installer will stop and attempt to prompt the user with an error message. I have found no good way to counter this other than to add some code to the AutoIt script to basically monitor the process and force-terminate if it is still running after a certain length of time. The AIM setup itself does not support silent mode switches or anything like that, so the only good way to handle this is use our boot-strapper script to react based on theoretical scenarios. Yes, you could possibly wait for a window title to appear and simulate a button click or something, but since this is designed to be able to run from a non-interactive session, that may not be an option. Just something to think about.
