December 12, 2007

AIP 1.0.0 Bypassed

Somebody from websecurity.com.ua has contacted me regarding a vulnerability of Auto-Input Protection (AIP).  I was able to confirm that reusing an older web form to bypass the AIP web control, even if the CAPTCHA image is no longer valid, is in fact a vulnerability of AIP 1.0.0.

See the original article here:  http://www.websecurity.com.ua/1568/

Although I believe that Security Through Obscurity [1] is not really security at all, one of the purposes of this control is to prevent unwanted SPAM such as automated comments in blogs, and in my experience it works as intended even though this vulnerability has always existed.  In most cases I think the potential risk of the control being bypassed is outweighed by the fact that its use is not related to security.  I think it's probably used mostly to prevent the nuisance of SPAM in blogs (that's how I use it) and forums, and to prevent unwanted user accounts from being generated, which may still be an appropriate use even with its current vulnerability as long as it works for now.

However, I certainly don't recommend AIP 1.0.0 to be used for the authorization of banking transactions or the arming of a nuclear weapon :)

The Attacks

Other than the obvious uses, AIP should be able to counter the following threats to itself (probably not an exhaustive list):

  1. Attempts to sequester many auto-generated images for pattern analysis by the bot's owner.
  2. Attempts to validate the same challenge repeatedly to learn, in an automated way, about the particular CAPTCHA algorithm being used.
  3. Attempts to repeatedly use the same challenge to bypass security.

AIP 1.0.0 does not provide protection for all of the points above.  If you can think of anything else that should make this list please let me know.

The Problem

In an attempt to remedy the situation I realized that it doesn't seem possible to have a 100% secure image-based CAPTCHA implementation because, at the very least, an implementation must be able to verify the submitted text and therefore must keep track of it either on the client or the server:

Client: Raw Text
When included in a response as raw text, such as in a query string or POST data, there is an obvious security flaw since bots can just find the "answer" in the response itself.  (AIP doesn't do this!)

Client: Encrypted Text
When encrypted in POST data, such as part of ASP.NET Control State (AIP does this in 1.0.0), there is an added level of security however it's still fundamentally flawed without taking extra precautions because the same page can be cached by a client and used repeatedly to bypass CAPTCHA protection, even using the same bitmap text and even when the CAPTCHA image has already been invalidated.

Server: Raw Text With Client Identity
When stored on the server, AIP must be able to identify the user agent posting the request as the same agent that requested the CAPTCHA challenge to begin with.  Regardless of whether the text is stored in-proc or out-of-proc, the easiest way to identify the user agent in ASP.NET is to use session state; however, this adds an additional requirement for a client: it must now respond with a session ID each time that it makes a request.  Commonly, a session ID is stored in a cookie on the client's machine; alternatively, it can be part of the request's query string instead.  But in both cases this can make the CAPTCHA susceptible to a Cross-site Request Forgery (CSRF) attack [2].  Though overall I think this can be the most secure solution with the help of absolute expirations.

The Solution

So here's what I've come up with to make AIP more secure.  The improved implementation will be available soon as part of the 2.0.0 Beta release.  Please let me know if you have any thoughts of your own about how security can be beefed up and I'll definitely consider it.

The main security points are:

  1. Configurable timeouts for requesting the image and responding with an answer.
  2. The same "user" that makes the original request for a page that contains the AIP web control must also be the one to respond with an answer.
  3. Only one CAPTCHA test can be active at any given time, per "user".
  4. Once an answer is given, the test is invalidated and cannot be used again (i.e., the current in-memory test cannot be used again, regardless of whether the answer was correct or not, but the actual text on the bitmap can still be used again if it happens to be generated, randomly, for a different request altogether).

Here's an outline of the implementation details, with the securest settings available all being enabled:

  1. Store the CAPTCHA text on the server instead of in the response and associate it with a value that uniquely identifies the user agent that made the request.
  2. Replace the last challenge, if one exists, so that only one at a time may be active.
  3. Invalidate the CAPTCHA image once it's requested for the first time after being generated (may not be appropriate in all situations).
  4. Minimize the amount of time in which a client has to request the image from the cache after it's been generated.
  5. Minimize the amount of time in which a client has to respond to a CAPTCHA challenge (may not be appropriate in all situations).
  6. Remove the CAPTCHA text from the server when an answer is submitted, regardless of whether it was correct or not.

1. Store the CAPTCHA text on the server and identify it by the user agent's ID.
The purpose is to ensure that only the same user that requested the CAPTCHA challenge can subsequently validate the CAPTCHA text.  However, this is not a RESTful [3] solution and it may not be 100% secure either since a CSRF attack [2] may be still possible due to the way Session state is implemented in ASP.NET (not to imply that there's a better alternative ;).

2. Replace existing CAPTCHA text.
The purpose of replacing the last challenge is to ensure that each user can only have one active challenge at a time, minimizing the chances that a bot will be able to scoop up multiple images to evaluate them asynchronously, without at least responding with an answer first.  It also cleans up server memory, with no reason to keep track of old tests.

3. Invalidate the CAPTCHA image after it's been requested once.
AIP 1.0.0 uses a sliding expiration for the image.  For added security, the expiration should not be sliding though - once the image is requested for the first time it should be removed from the cache.  But instead of assuming that this is the best approach for every case I've added a boolean property named, BitmapCacheKeepAlive that specifies whether the bitmap should be kept alive or expire immediately upon the first request.  The default value is false, meaning that once the first request is made the image is invalidated.  Requesting the image will result in the same message that appears when the sliding timeout has expired.  When the value is true, the sliding expiration will be used.  Note that if the value is false and an image is never requested then it will be removed from the cache automatically in the timeout specified by the BitmapCacheTimeoutSeconds property.

4. Minimize challenge request time.
This is already implemented in AIP 1.0.0 with the use of a property named, BitmapCacheTimeoutSeconds on the AIP web control. 

When a page with the AIP web control is requested an image is generated on the server and the timeout is started.  When the browser reads the page it will discover an HTML image tag and attempt to download it, but it must request the image from the server within the configured timeout period or else the image's key will be invalid.

This feature can be verified by using the same image URL twice in a row, after waiting for the sliding timeout to expire (the default is 15 seconds).  The first time a page is requested that contains the web control a URL will be generated and the image will be visible in the browser.  Copy the URL from the source and browse to it directly after 15 seconds; the result is an image containing the message: * AIP image not found in cache *.  If you're frequently receiving this message or if users report that your web pages take a while to load then increase the value of the control's BitmapCacheTimeoutSeconds property to give browsers more time to request the image.

5. Minimize challenge response time.
This has been implemented in AIP 2.0.0 as a timeout just like the bitmap request timeout, configurable via a property on the AIP web control named, BitmapTextTimeoutSeconds.  The default value is 30 seconds.  Another property on the AIP web control named, BitmapTextKeepAlive enables or disables the timeout.  The default value is false, indicating that the timeout will be used.  When true the challenge doesn't expire until an attempt is made.

Minimizing the response time may be appropriate for sign-in screens and pages with little information to read, but it's inappropriate in long documents and large data-input forms.  If you're using the control on a page that has a lot of reading material, such as a blog, or with many data entry fields, such as a long user membership form, then you should probably set BitmapTextKeepAlive to true; otherwise, increase the value of BitmapTextTimeoutSeconds to something that is more realistic than 30 seconds so that users will have time to read or enter data before submitting the form.

Alternatively, for a more secure solution, you can put the control on a separate web page by itself and show it after the form is submitted, but before the posted data is saved to your database.  This way you can keep the timeout short; maybe even shorter than 30 seconds, which should help to protect your site against the Human Solvers [4] attack and services such as the one provided by www.captchakiller.com.

6. One shot only.
Regardless of whether the first answer submitted is correct or not, the challenge is invalidated and cannot be used again.  If the value is wrong then a new challenge must be requested.  This feature, along with the new server-side storage, should fix the vulnerability in AIP 1.0.0 discovered by websecurity.com.ua.

How To Exploit AIP 2.0.0 Beta

I don't believe that the solution outlined above is 100% secure.  However, I don't think that any solution really is, but this new implementation should be enough to prevent hackers and spammers from bypassing the control's security when it's configured properly, depending upon the context in which it's being used.

To exploit any vulnerabilities in AIP 2.0.0 a malicious client can't just cache the page anymore with the new security measures in place since the session ID that was used to make the request is required in the response as well, in the form of a cookie or as a query string (depending upon how session state is configured in the site).  But even if the cookie value is obtained, somehow, it will eventually expire due to the session timing out.  If it's used before the timeout, only one challenge may be active at any time, per user, so it's possible that the test is no longer valid anyway.  And once the first answer is provided the challenge is invalidated automatically, regardless of whether the answer is correct or not.

In order for the AIP 2.0.0 web control to be breached, assuming that there aren't more vulnerabilities of which I'm unaware, a malicious client would have to:

  1. request the CAPTCHA image within the configured timeout period after the initial request is made for the page that contains the AIP web control.
  2. get the answer to the challenge.
  3. obtain a valid session cookie (or query string if the website is configured to use cookieless session state).
    1. The session ID must belong to the owner of the session that made the initial request for the page with the AIP web control (not the request for the image).  The owner could be the bot itself or it could possibly use a spoofed or hijacked session.
  4. submit the session information with POST data containing the correct answer to the challenge:
    1. within the configured challenge timeout period.
    2. before the user that actually owns the session submits their answer (assuming that the bot is using a spoofed or hijacked session).
    3. and before the user that actually owns the session requests a new challenge (again, assuming misrepresentation).

And finally, once an answer has been submitted the test cannot be used again.

Conclusion

This is still a work in progress.  I'll provide more information, probably in the form of a WIKI on CodePlex, around the time of deployment.  Also, I'd appreciate feedback from anybody that runs tests against the AIP web control to verify its security.  If you find a flaw, please let me know and I'll try to correct it.

References

[1] Security through obscurity. (2007, November 29). In Wikipedia, The Free Encyclopedia. Retrieved 20:40, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=Security_through_obscurity&oldid=174670068

[2] Cross-site request forgery. (2007, December 11). In Wikipedia, The Free Encyclopedia. Retrieved 19:33, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=Cross-site_request_forgery&oldid=177107398

[3] Representational State Transfer. (2007, December 12). In Wikipedia, The Free Encyclopedia. Retrieved 19:34, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=Representational_State_Transfer&oldid=177439704

[4] CAPTCHA. (2007, December 11). In Wikipedia, The Free Encyclopedia. Retrieved 19:33, December 12, 2007, from http://en.wikipedia.org/w/index.php?title=CAPTCHA&oldid=177256271#Human_solvers

November 20, 2007

DocProject 2008 Beta 2 Released

DocProject 2008 Beta 2 has been deployed to CodePlex.

  • Compatible with Visual Studio 2008 RTM (tested against Team Suite Trial).

  • Compatible with Visual C# 2008 Express and Visual Basic 2008 Express.

  • Supports side-by-side installation with the DocProject 1.9.0 RC for Visual Studio 2005.

  • Built using the same codebase as the 1.9.0 RC, with updates to support Visual Studio 2008, a few subtle changes that will also be included in the 1.10.0 RC, and some bug fixes.

Bug Fixes
  1. Topic Management Error: Value cannot be null
  2. Deleting a comment section does not mark the content item as dirty, so if it's the only change that's made it's not committed when the dialog is closed.
  3. The Cancel and Back buttons cause control validation on the Wizard Form, but they shouldn't.
  4. DocProjectEnvironment.InstallVSExtensions adds the same commands and tool windows to the commands and tool window collections even if they're already present.
  5. DocProject Properties window may be added to the wrong Project command bar, depending upon the environment.
Known Issue

The Build Component Stack configuration options display Load Error messages. This issue is related to a problem in Visual Studio 2008 that has yet to be resolved; however, if you manually add Sandcastle's assemblies to the GAC then you will not experience this issue.

See my blog post about this issue for more information.

Side-By-Side Installation

DocProject 2008 Beta 2 will work side-by-side with DocProject 1.9.0 RC, although you must update new and existing 1.9.0 DocProjects and DocSites so that they use a specific version of the DaveSexton.DocProject assembly.

To ensure that your 1.9.0 RC projects continue to work, follow these steps after installing DocProject 2008 Beta 2:

  1. Open an existing 1.9.0 RC DocProject or DocSite in Visual Studio 2005.
    1. If you need to create a new project instead then continue to the next step after completing DocProject's New Project Wizard.
  2. Expand the References folder in Solution Explorer.
  3. Right-mouse click the DaveSexton.DocProject assembly and select Properties.
  4. Set Specific Version to True. The version number, which is read-only, should automatically change from 2.0.0.0 to 1.9.0.0.
  5. Save the project.
If you do not follow these instructions then you will get an error when you attempt to build the project that states, The specified type is not a valid BuildProcessComponent implementation.
November 16, 2007

DocProject 2008 Beta 2 - AssemblyResolve Showstopper

April 27, 2008 Update: The problem is with Type.GetType forcing a complete JIT when executing in VS 2008 (it is used by InstantiateProviders).  I've submitted a new bug report that provides a much cleaner solution to reproduce the problem: https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=340664

Nov. 20, 2007 Update: I've narrowed the problem down to a call to System.Web.Configuration.ProvidersHelper.InstantiateProviders, which tries to resolve all dependencies at the time of invocation even if they are not being referenced in any executing code.  If assembly resolution must be deferred (as is the case in DocProject) then this presents a problem because there will not be an AppDomain.AssemblyResolve event handler registered to help the framework resolve the assembly.  However, this behavior only occurs when VS 2008 is hosting the executing code.  It does not occur in VS 2005 or in a console application built on the .NET 3.5 framework, for example.

I've submitted feedback to the Sandcastle team to have them GAC the required Sandcastle assemblies to prevent the issue in this blog post from occurring.  For now though, you can manually GAC the BuildComponents.dll, BuildAssemblerLibrary.dll and CommandLine.dll assemblies using: gacutil /i assembly_name.dll

I will be releasing DocProject 2008 Beta 2 shortly.

I've finished working on DocProject 2008 Beta 2, however, there is an issue with the new build component stack properties that has prevented me from releasing it: When the stack properties are expanded, all of their components are replaced with load error messages.

If you feel that this issue should not be preventing me from releasing the product then please let me know and I'll consider removing the expandable stack properties for now in 2008 Beta 2.

More Details

The problem is that an assembly cannot be delay-loaded in certain situations while a program is being hosted by VS 2008 since the AppDomain.AssemblyResolve event is not being raised.  The same code works in VS 2005 and using .NET 3.5 in a console app, but it just doesn't work when executed by a VS 2008 add-in.

It appears to be the same bug in VS 2008 that I reported back in August, but at the time I had discovered a work-around that was successful because the assembly was being loaded in another AppDomain.  Unfortunately, it won't work this time since the components must be loaded in the default AppDomain.

I've submitted new comments (starting at 11/16/07) in the feedback with a project that reproduces the bug attached, so I'm hoping that someone will post a workaround (I assume that it's way too late to get fixed in VS now).  Download the attachment from my website if you think you can help.  The instructions are in my feedback post :)

Here's the output that the example produces in the debug window.  Notice how the console version of the program appropriately raises the AppDomain.AssemblyResolve event, where as the VS add-in version does not.  The exception is actually not the problem - it's to be expected since I didn't add any code to actually resolve the assembly.  (The point is that VS 2008 doesn't even give DocProject a chance to resolve the assembly).  And BTW, both hosting scenarios in my example call into the exact same code.

The example was developed on the latest release of the Windows Server 2003 VPC for Visual Studio 2008 Team System Beta 2.

CONSOLE OUTPUT:

'ConsoleApp.vshost.exe' (Managed): Loaded 'C:\Documents and Settings\Administrator\My Documents\Visual Studio 2008\Projects\TestAddIn\ThirdPartyLibrary\bin\Debug\ThirdPartyLibrary.dll', Symbols loaded.
AssemblyLoad: ThirdPartyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8fd3dad2bd735269
*********** Loading type...
AssemblyResolve: ThirdPartyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8fd3dad2bd735269
A first chance exception of type 'System.IO.FileNotFoundException' occurred in PlugInLibrary.dll
The thread 0xbfc has exited with code 0 (0x0).
The thread 0xd08 has exited with code 0 (0x0).
The program '[1180] ConsoleApp.vshost.exe: Managed' has exited with code 0 (0x0).

VISUAL STUDIO OUTPUT:

'devenv.exe' (Managed): Loaded 'C:\Documents and Settings\Administrator\My Documents\Visual Studio 2008\Projects\TestAddIn\ThirdPartyLibrary\bin\Debug\ThirdPartyLibrary.dll', Symbols loaded.
AssemblyLoad: ThirdPartyLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8fd3dad2bd735269
*********** Loading type...
A first chance exception of type 'System.IO.FileNotFoundException' occurred in PlugInLibrary.dll
'devenv.exe' (Managed): Loaded 'C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.TeamFoundation.Build.dll'
AssemblyLoad: Microsoft.VisualStudio.TeamFoundation.Build, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
The program '[3292] devenv.exe: Managed' has exited with code 0 (0x0).

November 09, 2007

DocProject Tips and Tricks I

As you probably already know, DocProject is kind of weak in the documentation department (that is, in its own API documentation :)  I plan to address this issue for the first RTW, after I include first-class support for generating conceptual content and after Sandcastle's first RTW.  If you are new to DocProject, the articles on CodePlex should provide more than enough information to get you started.

In a series of blog posts, with this post as the leader, I'm going to provide some supplementary information that you may find useful.  Tips and tricks that will add value to DocProject and allow you to make the most of it.  Code examples can be found at the end of posts and will have names like, Example #.

Community Tricksters

As the author, I probably know more about DocProject than anyone.  But as an end-user, I haven't taken advantage of the full range of its capabilities yet - including things like using DocProject with Team Build, building AJAX documentation or customizing Help 2.x output.  Therefore, I'd be happy to accept tips and tricks from the community that I can include in this series.  If you have ideas please submit them as blog comments or email them to me and I'll consider including them in my posts.  Thanks!

General-Purpose Tips

Here are some tips for DocProject that apply to all types of templates and build engines:

  1. If you do not want to include build output as project items, and you don't want to be prompted after you build a project for the first time, then set Include Project Output Dialog | Don't ask me this again to True in the DocProject Properties window before you build.  You can leave the default values for the two related options, which should both be False.  (Note that the Apply to all default option is not used when Don't ask me this again is True.)
  2. You can insert dynamic build steps using your project's Build Process Component (see Example 1 below).
  3. Use an external or third-party Build Process Component:
    1. Open the DocProject Properties window (right-mouse click your project in Solution Explorer and select, DocProject Properties).  Visual Studio Express users must use the DocProject External UI.
    2. Locate the Extensibility | Process component type name option.
    3. Enter an assembly-qualified type name (the assembly must be located in the GAC or your project's output folder).
    4. Alternatively, you can enter a full type name and an absolute or relative assembly path, separated by a vertical bar; e.g., MyNamespace.MyType|..\anotherproj\bin\Debug\anotherproj.dll
DocSites Tips

These tips apply to the DocSite Templates only:

  1. To change the DocSite logo simply replace the Images\Logo.jpg file with a file of your own.  To use a different file name, change the value of the ImageUrl attribute on the <asp:Image ... id="logoImage" ... /> element in the Controls\DocSiteHeader.ascx file.
  2. To show rank information in DocSite search results, for the purpose of fine-tuning the search index, open the App_Themes\BasicBlue\DocSiteSearch.skin file and set Visible="True" on <asp:Label ... SkinId="searchResultRank" ... />.  When performing a search the rank value will appear before the title of each matching topic.  If you already have search results in your browser just refresh the page.
  3. To change the DocSite title, company name and copyright, set the value of the AssemblyTitle, AssemblyCompany and AssemblyCopyright attributes, respectively, in the Properties\AssemblyInfo.(cs|vb) file and then restart the web application.  (The DocSite caches these values in memory upon the first request to the site.)
Sandcastle Tips

This section provides some useful tips for projects that use the Sandcastle or Sandcastle/Deployment build engine providers.

  1. Set the Help 1.x Configuration File Name and Help 2.x Configuration File Name options to the same values and DocProject will only run Build Assembler once to build both Help 1.x and Help 2.x output.  The cost of doing this is that reference link types in one of the help output formats might not be valid (this was the main reason for splitting the Sandcastle configuration file into two distinct files in the first place.)
  2. Use XML documentation files generated in the API Topic Management dialog for IntelliSense by adding a short code snippet in your DocProject's Build Process Component (see Example 2 below).
  3. In your Build Process Component, you can get access to all of the Sandcastle-specific project options and APIs.
    1. Add a reference in your DocProject or DocSite to the DaveSexton.DocProject.Sandcastle.dll assembly, commonly found at C:\Program Files\Dave Sexton\DocProject\bin\.
    2. Check out the code in Example 3 below, which illustrates how to access the custom Sandcastle APIs that are provided by DocProject.
  4. Insert images into your documentation using the XML documentation editor in the API Topic Management dialog:
    1. In design mode, place the text cursor where you want to insert an image.
    2. Click the InsertPictureHSbutton on the editor's tool strip or right-mouse click and select Insert > Picture.
    3. In the Picture dialog enter the full path and name of an existing image or click the Browse... button to locate one.
    4. You can enter a local file path or even a web address such as, http://example.com/someimage.jpg.  Click OK and the image is inserted.
    5. After you are done authoring your documentation, click OK on the API Topic Management dialog and DocProject will automatically import the image into your project's Help\Art folder.  It will also rebase the reference in the XML documentation to ..\Art\{image name}.
      1. Note: If you enter a web address then DocProject will copy the image from your web browser's cache; otherwise, the file is copied from the location that you specify.
    6. Note: The next time that you view the XML documentation in the API Topic Management dialog, the path will be rebased so that it's rooted, for display purposes only.  The path in your actual XML documentation on disc will remain relative, however.
  5. Filter out <Module> topics when the Documentation scope option is set to Complete by adding a dynamic regular expression filter:
    1. Open the API Topic Management dialog.
    2. Select the Topic Filters tab.
    3. Select the Regular Expression tab.
    4. Enter the following into the Pattern text box: \<Module\>
    5. Click the Save button.  The Saved Filters tab will be shown automatically and your filter will appear in the list.
    6. For the new filter, ensure that the Build column is checked and that the Include column is unchecked.
    7. Enter a short description of this filter into the Memo field (optional).
    8. Click OK on the API Topic Management dialog to commit your changes.
C# Code Examples

The following examples illustrate points made in the information above.  They are referenced by number.

Example 1:

The following example shows how to inject a dynamic build step immediately before the Help 1.x compiler step using your project's Build Process Component.  The step will be executed in full and partial builds and will run on a background thread.  It outputs some text in the build trace window:

Step # of ##: Hello

World!

To use this example, replace the BuildStarting method in your project's BuildProcess.(cs|vb) file.  (Keep the existing code though).

   1: protected override void BuildStarting(BuildContext context)
   2: {
   3:   // existing code goes here 
   4:  
   5:   base.InsertBeforeBuildStep("Compile Help 1.x", "Hello", true, false, true,
   6:     delegate(BuildContext stepContext)
   7:     {
   8:       stepContext.TraceLine("World!");
   9:     });
  10: }

Example 2:

The following example will overwrite the source XML documentation file (that contains your code comments) with the merged XML documentation file (that contains both your external documentation and the source code comments) after each help build completes.

To use this example, replace the BuildCompleted method in your project's BuildProcess.(cs|vb) file.  (The existing code should be placed at the end).  You must also add a using System.IO declaration at the top of the file.

   1: protected override void BuildCompleted(BuildContext context)
   2: {
   3:   TraceLine();
   4:   TraceLine("Overwriting XML documentation files in source projects..."); 
   5:  
   6:   IDocProject project = context.Engine.Project;
   7:   BuildSettings settings = context.Engine.Settings; 
   8:  
   9:   foreach (ISourceProject source in project.Sources)
  10:   {
  11:     string target = source.Output.XmlDocumentationFile; 
  12:  
  13:     FileInfo autoGenFile = new FileInfo(Path.Combine(
  14:       settings.WorkingXmlDocumentationPath, 
  15:       Path.GetFileName(target))); 
  16:  
  17:     if (autoGenFile.Exists)
  18:       autoGenFile.CopyTo(target, true);  // overwrite
  19:   } 
  20:  
  21:   TraceLine("Done."); 
  22:  
  23:   // existing code goes here
  24: }

Example 3:

You can access Sandcastle-specific project options and APIs in your Build Process Components by adding a reference to the DaveSexton.DocProject.Sandcastle.dll assembly.  The following example shows how to obtain the SandcastleBuildEngine instance, and from that retrieve the SandcastleSettings and SandcastleProjectOptions:

   1: SandcastleBuildEngine engine = (SandcastleBuildEngine) context.Engine;
   2: SandcastleSettings settings = engine.Settings;
   3: SandcastleProjectOptions options = engine.Options;

The context is an argument that is passed to all of the four main Build Process Component overrides: BuildStarting, BeforeExecuteStep, AfterExecuteStep and BuildCompleted.

November 07, 2007

DocProject 1.9.0 RC Deployed

The DocProject 1.9.0 Release Candidate is now available on CodePlex.  I'm still working on updating the wiki documentation though.

Thanks for your patience with this one.  I invested more time than I probably should have in additional features that weren't immediately necessary, but hey, this project is fun for me and some times I can't seem to come to a logical stop :)

New Features in DocProject 1.9.0 RC

  • Compatibility with the Sandcastle October 2007 CTP.
  • Preliminary support for conceptual content using Sandcastle's conceptual configuration file. (The conceptual build process has been integrated although it hasn't been tested and DocProject does not automatically generate a conceptual TOC yet.)
  • Sandcastle's build component stacks are now available as properties in the DocProject Properties window and the DocProject External UI, with support for hosting custom build component editors that can either show a modal dialog, a drop down editor or subproperties.
  • Framework and project reference link types are configurable as subproperties of the build component stacks and can be set by simply choosing one of the following values from a drop-down list: MSDN, Index, Local or None.
  • DocSite Templates include four new buttons under the Contents tab: Save, Bookmark, Email and Print. The latter two are supported in IE6, IE7, Firefox and Opera, but the former two are only visible in IE. Note that Save does not currently download supporting files such as JavaScript or style sheets though.
  • DocSite performance generating the full-text search index has been greatly improved.
  • Version Management Dialog: Create documentation that includes version information by referencing external assemblies and reflection files, and then simply enter custom version names (DocProject uses Sandcastle's VersionBuilder program in the background.)
  • ChmBuilder is used exclusively to build the supporting Help 1.x files (WARNING: The .HHC file format has been changed. Beware if you were modifying it to include custom topics.)
  • Both the Topic Designer and API Topic Management dialog now remember their last saved position, size, and the positions of splitters.
  • Edit all content item documents using the Topic Designer instead of just the shared_content.xml file. All of the available documents are listed in the Create Shared Content step of the New Project Wizard as well.
  • Several new API Topic Management dialog features:
    • Improved performance when applying filters.
    • Topic tree nodes now use icons that reflect access type, as do the tooltips.
    • Filter topics by API accessibility or in combination with API types; e.g., protected fields, private interfaces, public or protected methods and delegates, etc.
    • Save filters and load them later or have them applied automatically during builds (dynamic filters).
    • Author external XML documentation in the appearance of the selected presentation style, with special styles for XML documentation tags such as see, c, code, paramref, etc.
    • New XmlTreeView project and control, which supports binding to an IXPathNavigable data source and allows drag/drop internally and externally (although internal drag/drop support is not currently enabled. This feature is planned for conceptual content in a subsequent release).
    • Drag and drop API elements from the tree into the XML documentation editor to generate see links automatically (works in both source and design mode).
    • The appropriate XML documentation sections for each specific API type (.e.g, summary, remarks, return, value, etc.) are listed automatically in the XML documentation editor, including individual param and typeparam tags for methods and delegates.
    • Add additional documentation sections that support multiple instances (e.g., exception, permission) by selecting them from a drop-down list. The list only shows those sections that are appropriate for the selected topic.
    • Quickly create custom XML documentation sections in the editor by entering a name and pressing enter, or register them as built-in sections in DocProject's configuration file. See this blog post for information on how to use custom tags in Sandcastle's XSL transformations.
    • The XML documentation editor automatically cleans up empty sections from the external documentation when it's saved.
  • Build Process Components can now take advantage of more of DocProject's API and now have the ability to inject dynamic build steps, pass delegates to DocProject's API and register event handlers.
  • Documentation now includes a root Namespaces topic to which the XML documentation's project summary is applied.
October 27, 2007

DocProject 1.9.0 RC Preview

The DocProject 1.9.0 Release Candidate will contain several bug fixes and some exciting new features (well, exciting to me anyway :). In this blog post I'm going to describe some of the new features and show a few screen shots, like usual.

Miscellaneous Features and Bug Fixes

For the sake of time and keeping this blog post short (well, shorter than I could make it), I'm going to first list a few of the important features that I feel deserve to be mentioned, even if I'm not going to elaborate:

  1. The time it takes to generate the DocSite's full-text search index has been greatly reduced - to the point where it should actually be useable now.  In my testing, no more than 10 minutes for over 10 000 topics on a regular desktop computer.  I still plan to improve performance even more though and will develop that SqlSearchProvider very soon.
  2. The DaveSexton.DocProject.DocSites library no longer requires full trust.
  3. DocSites now have Save, Print, Email and Bookmark buttons in the Contents bar to act on the selected topic.  The Print and Email buttons work in IE6, IE7, Firefox and Opera.  The Save and Bookmark buttons only work (and are only visible) in IE6 and IE7.
  4. Build Process Components can now use more of DocProject's API, including the use of cross-AppDomain delegates and event handlers and they are even able to insert dynamic build steps that are actually build steps.   I'd like to blog about this separately though.
  5. The Sandcastle plug-in now uses the ChmBuilder program to generate the supporting Help 1.x files.  Localization has not actually been implemented yet, though I plan to address that soon.
  6. The Topic Designer (shared content) dialog now displays all of the files that have Content/<item> semantics, and they are listed as Content Documents (this includes documents in Shared\Content\ as well).  Selecting a document will automatically list all of its items underneath, and from their the dialog works the same as usual when an item is selected.  By default, shared_content.xml and header are selected, like usual.
  7. Bug fixes galore.  (Though, it was more bugs due to major refactoring in this release than existing bugs from previously releases :)

And now for some elaboration...

API Topic Management Dialog

Lots of new stuff and refactored code here.  I'm not really sure where to begin, so I guess I'll just start off by saying that the API Topic Management dialog now saves its last location and size, and the position of the splitters (and so does the Topic Designer).  Now, let's continue on with some of the more obvious and important changes.

image

Figure 1: API Topic Management Dialog - Tree View and Regular Expression Topic Filter

Tree View

As you can see from the image above, the tree now shows general topics that are defined by the presentation style's doc model (e.g., All Members, Methods, Fields, Properties, etc.).  Theses types of topics cannot be filtered with the checkbox; however, uncheck them and all child nodes will be unchecked automatically.

You may also notice in the image that the icons are now synchronized to the API access levels of each topic.  For example, in the image above, AI.Cells.Enzyme.acids is a private field, which has an icon with a lock.  The tool tip will also indicate that the topic is a "Private Field".

The +/- button above the tree will, when clicked, collapse all nodes if any one is currently expanded, and will expand all nodes if they are all collapsed.  There are other buttons that are currently hidden because they have to do with conceptual topics.  More on that later...

It's probably also worth noting that the performance of the tree when loading data and finding/applying topic filters has been improved.

Topic Filters

Figure 1 also depicts the improved Topic Filters tab and some of its new features.  But let's just start with the basics...

Regular Expression Filter

The options for the Regex filter are now displayed as checkboxes, simply for convenience since there is now more room in the GUI due to the filter types having their own individual tabs.  Also, the pattern can now be entered as multi-line, although since new lines will not match any topic names, entering multiple lines (to improve readability of your patterns) will cause the Ignore Pattern White Space option to be checked automatically.

Categories Filter

The categorical filter has also been improved, for performance and functionality.  It now supports filtering by API access levels, in addition to API category types.  For example, you can now filter by protected fields, or private interfaces and structures.

image

Figure 2: API Topic Management Dialog - Categories Filter

If nothing is checked under Accessibility then it is simply not taken into account when matching.  In other words, any access type is a match - the same as if all access types are checked.

You can also choose an access level without choosing an API type and it will match all APIs with the selected access level.  Choosing more than one access level is an OR operation, just like with API types.

To match protected internal members you must check protected and internal since it's treated as a distinct access type.  In other words, DocProject does not treat protected internal members as just being protected, or internal.  One caveat is that searching for protected internal members will also explicitly match protected-only and internal-only members as well.

Saved Filters

Saved filters are a new addition in 1.9.0.  It allows you to save Regex and Category filters so they can be loaded and applied quickly in the future or, more importantly, applied automatically during each build.

image

Figure 3: API Topic Management Dialog - Saved Filters

You can get filters into the list by clicking the Save button that appears on each of the filter tabs (see Figures 1 and 2).  Alternatively, you can directly edit the XML file in which they are saved, located at Help\dynamicFilters.xml.  The file does not exist until you add a filter using the dialog and then click OK, but you can create the file yourself if you want (see Figure 4 below).

You can also import filters from a Dynamic Filters XML file using the Import... button (see Figure 4 below).  Imported filters are appended to the list and do not replace any existing filters.

Memo is a place to comment on filters.  You can edit the memo directly in the grid after you save a filter and it will be persisted along with the filter list when you save the changes to the API Topic Management dialog.

Clicking a Load button will simply load the corresponding filter into the appropriate filter tab (i.e., Regular Expression or Categories) and will then show the tab and apply the value of the Include column to the Include matching topics Search Option.  The filter will not be applied, however.

The Apply button will immediately apply the filter to the tree using the current Search Options and the value in the Include column, but will not load the filter first.

You can delete the selected filter by pressing Delete on the keyboard or by clicking the delete button (with the red X).

The arrows are used to order the filters, which is important because all filters that have a check mark in the Build column are applied in order, top-down, during a help build.  The Apply all filters during builds option will either check or uncheck the Build column for all filters so that they have the same value as the option and each other.  The Apply All button will apply all filters to the tree view immediately, top-down, but without regard to the Build column.

The Include column indicates whether matching topics are included or excluded, just like the Include matching topics Search Option.

These features together provide a powerful way to filter topics.  For example, in Figure 3 above the 3rd filter excludes all private fields, but then the 4th filter ensures that one private field in particular, acids is included again since the Include column is checked.  The 4th filter will be applied after the 3rd filter, since it appears after the 3rd filter in the list, so the acids field will end up being included after all filters have been processed on that topic.

The schema of the XML file that stores the filters is simple, so I'll just give an example file that was created from the filters in Figure 3:

<?xml version="1.0" encoding="utf-8"?>
<filters>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.RegexTopicFilter">
    <include>False</include>
    <pattern>Carbon$</pattern>
    <memo>Get rid of any topic that ends with "Carbon".</memo>
  </filter>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.RegexTopicFilter">
    <include>False</include>
    <options>IgnoreCase</options>
    <pattern>^Startup$</pattern>
    <memo>Get rid of the Startup classes in each assembly.</memo>
  </filter>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.CategoryTopicFilter">
    <include>False</include>
    <memo>No private fields.</memo>
    <types>Field</types>
    <access>Private</access>
  </filter>
  <filter type="DaveSexton.DocProject.Sandcastle.TopicManagement.RegexTopicFilter">
    <include>True</include>
    <pattern>^acids$</pattern>
    <memo>Reinclude the "acids" field, even though it's private.</memo>
  </filter>
</filters>

Figure 4: Example Help\dynamicFilters.xml file

Note that the type attribute will accept assembly-qualified type names so that you can define your own custom filters; however, I've not actually tested custom filter types yet so they may not actually work, but I plan to support them in a subsequent release.

The element names for a filter's options are defined by the filter itself, although memo, include and build (which is not shown in the example because it defaults to True) are built-in to the base type because they have special meaning to DocProject.

XML Documentation

In 1.9.0 there are some new features to help make authoring documentation easier.  But before I begin on that, I'd like to lead in with some info about where DocProject currently stands on handling conceptual content since it has had a profound effect on 1.9.0.

Conceptual Content

The next release of Sandcastle might be available on Monday (the 29th of October), and you can probably expect DocProject's release to follow within a few days. One thing that I'm really excited about is that the Sandcastle team will be shipping their internal conceptual configuration file for the Build Assembler program, which should allow me to enable the long awaited first-class support for conceptual content in DocProject.  If I can include support in 1.9.0 without too much delay then I certainly will, but if that's not the case then expect it in 1.10.0.

The Tree View and XML Documentation

In preparation for conceptual content, I've refactored the code some more and made some assumptions about how the process will be integrated.  Also, I've included my new XmlTreeView control with DocProject 1.9.0 RC (it's being used in the API Topic Management dialog now).  It supports drag/drop to reorder nodes (to be used for conceptual content only, once it's supported) and to drag nodes onto the XML documentation editor, which automatically creates a <see> link (that is supported in 1.9.0).  Also, the XML documentation editor now applies the style sheets of the selected presentation and a special designer style sheet that you can edit yourself: Help\Styles\TopicDesigner.css.  The designer sheet formats XML documentation markup such as <c> and <see> tags.  All of the style sheets are used in both the Topic Management and Topic Designer (shared content) dialogs.

image

Figure 5: API Topic Management Dialog - Xml Comments

The list of XML documentation sections is now generated automatically from DocProject's configuration file based on the type of topic that is selected.  Then the list is also merged with the actual elements found in the external XML documentation files, which allows users to add custom elements to external XML documentation files outside of the API Topic Management dialog and then use the dialog later to edit them.  There are other ways to manage custom elements, which I'll discuss later.

image

Figure 6: API Topic Management - Xml Comments Sections

The parameter that is selected in the image above was added by DocProject based on the reflection information that was provided by Sandcastle.  The method in the example only has one parameter, but if it had multiple parameters then there would be one param name="value" item in the list for each.

BTW, I just noticed a regression: the GenericClass should not show the back tick in its name - I'll fix that :)

You cannot delete delegate/method parameters or generic type parameters from the list.  You also cannot delete sections that may only appear once, such as summary or remarks.  But you can delete custom sections and sections that may appear multiple times, such as exception and permission, by right-mouse clicking and selecting Delete from the context menu.

To add an instance of a built-in section that may appear multiple times, you can select one from the drop-down list at the bottom.  The list displays only those items that are related to the API type of the selected topic.  You can also add custom sections by simply entering the name into the combo box and pressing Enter on the keyboard.

When a custom item is selected the attribute grid, which is displayed above the XML documentation editor, is completely editable, except for the Type column.  (Although I may change that - I haven't decided yet.)

You can create more built-in sections easily by registering them in DocProject's configuration file (commonly found at, C:\Program Files\Dave Sexton\DocProject\DaveSexton.DocProject.dll.config) under the sections element:

image

Figure 7: documentationTags element in the DocProject configuration file

XML documentation tag usage data has been retrieved from the following sources:

  1. Recommended Tags for Documentation Comments (C# Programming Guide)
    http://msdn2.microsoft.com/en-us/library/5ast78ax(VS.80).aspx
  2. The "Tag Matrix" that is present in Dynicity LLC's XML Documentation Comments Guide:
    http://www.dynicity.com/Products/XMLDocComments.aspx
    (Note: There are differences between the config data and the published matrix due to bug fixes and/or differences in versioning and tag/aspect applicability.)

Conclusion

There's much more, but I have to go catch a train now...

October 21, 2007

C# Multi-line XML Documentation Comments

I was just checking out the documentation on MSDN about XML comments and I discovered that C# supports multi-line comments for XML documentation as well:

/**
 * <summary>Hello World!</summary>
 */
public void HelloWorld() { }

Interesting, but I wouldn't recommend using multi-line comments for a few reasons:

  • They do not support Smart Comment Editing, where by Visual Studio inserts multiple comment lines and an empty summary tag automatically after starting the comment, which in this case is done by typing /** instead of ///.
  • Intellisense is limited:
    • top-level tags are listed but context is ignored (e.g., the para tag is never listed, even if the cursor is within a summary element).
    • Entering a tag name and pressing enter will not produce a closing tag automatically.
  • The rules for processing markup may be problematic since the compiler will look for a prefix pattern consisting of white-space, an asterisk and more white-space; however, a slight change in formatting in the prefix of one or more lines (e.g., less leading white-space) may cause the compiler to include one or more asterisks as part of the comments.
  • They are certainly not as common as triple-slash comments and will probably be confusing to most people (I know if I were to have seen this format used, and I already may have, I would have doubted whether it was even valid.)

I'm curious to know if anyone has found this feature to be more useful than triple-slash comments, and why :)

October 08, 2007

The Help! Project on CodePlex

I've just published the Help! project on CodePlex, which is intended to provide a fully managed compiled help system to replace Help 1.x (.chm).  Please contribute to this project by using the discussions area to post comments and ideas or join the development team.  If you want to join the team then please see the Welcome! thread for information.

For now I'll be using my blog to post news and information about this project.

October 08, 2007

DocProject 2008 Beta 2 Is Delayed

I've decided not to release DocProject 2008 Beta 2 with the 1.8.0 feature set.  Instead, I plan to reduce the number of conversion requirements and release Beta 2 with the next VS2005-compatible release, 1.9.0 RC.  Sorry for any inconvenience this may have caused.

October 04, 2007

DocProject 1.8.0 Release Candidate

DocProject 1.8.0 RC is now available for download on CodePlex.  It contains bug fixes as well as several new features:

  • Compatibility with the Sandcastle September CTP.
  • DocProject Properties context menu item for DocProjects and DocSites (on the project node in Solution Explorer) quickly opens a property page in Visual Studio Standard and higher.

  • DocSite Templates now support MSIE 6.0 (as well as IE 7.0, Firefox and Opera) and are themed, localizable, configurable and componentized.
  • DocSite Templates have a full-text search that supports complex query operations and uses an in-memory search index provider (based on .NET 2.0 Providers) that can be configured on the server, even after deployment.
  • DocSite Templates have a full-text index browse page that makes finding topics very easy.
  • The selected project Configuration and Platform work in MSBuild, Team Build and the DocProject External UI, which now displays drop-down lists for changing these values before building (like in Visual Studio).
  • The DaveSexton.DocProject.targets file has been refactored to make it more flexible.
  • Build output and Sandcastle project items are now located in a root folder named, Help.
  • After a build, compiled help is automatically copied to the project's output directory (for example, Debug\bin); however, for the DocSite templates the target is Help\bin. The DocSite TOC and index files are now generated in the App_Data folder.
  • DocProject External UI now supports the Include Project Output Dialog, which is automatically displayed after a build unless it's disabled by the user.
  • DocProject External UI provides a tools options page for configuring build engine providers just like the Engines tools options page that's available in Visual Studio Standard and higher.
  • Start Menu shortcuts to the DocProject External UI and various resources, including the Source folder in DocProject's installation directory, are now installed.
  • User configuration data is now stored in the system's isolated storage so that one user's changes do not overwrite another's; e.g., the default build engine provider and Sandcastle presentation style.

You can read more about the DocSite features in my previous blog post: DocSites 1.8.0 Preview