September 13, 2007

DocSites 1.8.0 Preview

In the upcoming DocProject 1.8.0 release candidate I've made some changes and added several substantial features to the DocSite templates that I'd like to share with you in this blog post. Changes to the DocSite templates include:

DocSite in Solution Explorer
  • ASP.NET Theme presentation:
    • Ships with one default theme named, "BasicBlue", which has the same appearance as the current DocSite templates.
    • Subtle improvements to appearance, mostly for Firefox and Opera.
    • Retained div-only + CSS layout with a sticky footer; i.e., no tables.
    • Includes some useful .SKIN files.
    • Full support for IE6 as well as continued support for IE7, Firefox and Opera.
  • Componentized Master page, with user controls for the header, footer, breadcrumbs, sidebar, TOC and index.
  • ASP.NET Localization with resource-based text in all pages and controls.
  • Customizable and highly extensible full-text search built from the ground up:
    • Designed for general-purpose indexing but customized specifically for indexing HTML help topics.
    • Has a default in-memory search index provider (I'll also build an SQL Server search provider for a subsequent release).
    • Complex query support with operators such as AND, OR, - (not) and grouping with parenthesis.
    • Customizable, factor-based weight and rank calculations.
    • URL query string supports bookmarking searches.
  • Keyword browse page for live browsing of the search index (also supports a URL query string for bookmarking).
  • Administration page for configuring DocSite options, search factors and for viewing simple search statistics.
  • New Project Wizard – Create DocSite Credentials step allows a user to set the administrative DocSite credentials when a DocSite is first created.
  • The DocSite templates reference a new class library that encapsulates common DocSite functions.

Breakdown of Features

The rest of this article will provide more information on some of the features listed above. If you have any questions or comments please let me know!

New Project Wizard – Create DocSite Credentials

The first change that you'll notice to the DocSite templates is a new wizard step:

New Project Wizard - Create DocSite Credentials
Figure 1: New Project Wizard - Create DocSite Credentials

This step allows you to enter an administrative user name and password for accessing the DocSite administration page (more on that later). The credentials are stored in the web.config file in the standard ASP.NET forms authentication section.

For custom DocSite templates authors can easily enable this feature by adding a single user to the authentication section with "$admin$" as the name (the password will be ignored). The mode of encryption that is chosen for the authentication section (e.g., Clear, SHA1, MD5) is reflected in the dialog's help paragraph automatically, and will also be applied to the user's password before it's stored in the web.config file.

Themes and Localization

The DocSite templates now use a default theme named, BasicBlue. Its appearance is almost identical to the appearance of the current DocSite templates, but now it has full support for IE6 as well as a sticky footer for all .aspx pages that do not use an IFRAME to display page content.

Supporting IE6 without a table layout was difficult, but I succeeded with the help of CSS expressions, which are specific to IE (but that means they're supported by IE7 too). The hard part was trying to support IE7's more standardized rendering engine without breaking IE6, and then supporting IE6 without IE7 falling back to things like CSS expressions. But after some flipping out and loss of hair, I finally got it working :)

Customizing the appearance of DocSites should be much easier now that it's fully themed. And with Sandcastle's high level of flexibility, anyone with some previous experience working with XSL and CSS should be able to come up with a custom Sandcastle presentation style and DocSite theme that work together for a completely customized appearance without affecting the DocSite's functionality, or even better, to improve on it. BTW, I'm certainly interested in hearing new ideas for themes if you have any.

Text that appears in all of the DocSite pages and user controls have been extracted into ASP.NET global and local resource files for easy localization.

Indexed Search and Browse

Finding help content in DocSites will be much easier now with the addition of the new search and browse features.

Note: The screenshots depict the Admin link as a different color than other hyperlinks, but that's simply because I clicked it.  What you see is the "followed" hyperlink color.  Before Admin is clicked it's the same color as the other links in the header: blue.

Search Page

DocSite Search Page
Figure 2: DocSite Search Page

The abstract, locale, namespace and library information are all taken from each topic's MSHelp data found in an XML data island generated by Sandcastle for Help 2.x. Although, due to a possible limitation in the next Sandcastle release, I might have to remove this feature temporarily until I can create a custom solution for a subsequent release.

DocSite search supports complex queries with operators that are defined by the search provider itself. By default, your DocSite will support the Boolean AND and OR operators, - (the negation operator) and parenthesized grouping:

DocSite Search Help
Figure 3: DocSite Search Help

The search help page is accessible by simply leaving the search box empty and clicking the search button (the one with the large magnifying glass).

NOTE: In the future I plan on adding an advanced search page that contains HTML topic-specific search functions, such as searching by the culture in which a topic was written, the topic's title, etc.

Browse Index Page

The button with the tiny magnifying glass, next to the search button, is the browse button. Clicking this button will open the browse page using the keywords entered into the search box, although all operators and grouping will be ignored. Clicking the button with an empty search box simply opens the browse page without any selected keywords.

Browse Index Page
Figure 4: Browse Index Page

The browse page allows you to drill down through a query by removing or adding one keyword at a time until you find a result set that contains the topic in which you're interested. The browse page is also useful for administrative purposes as it displays some statistical information about individual keywords as well.

The letter bar at the top of the page provides a list of keywords that start with the selected letter. Clicking a letter will display the keyword list. Then, clicking a keyword will add it to the filter and the new filtered results will be shown. You can continue to browse the index by the first-letter of keywords or view and modify the active filter at any time. The letter bar itself is encapsulated in a user control and is completely customizable using the administration page (discussed later).

Toggle between Search and Browse

The button next to the search query on the search results page and the button next to the keywords on the browse page allow you to easily toggle between these two distinct modes of locating help topics so that you can choose the one that is better suited for your particular query.

The Index

Both the search and browse features work with a full-text index of the HTML help topics generated by Sandcastle. The index is stored in-memory and generated on-the-fly by the website itself, not DocProject, which means that you don't have to rebuild the project to rebuild the index. It also means that you can tweak factors for weight and rank calculations and then rebuild the index on-demand through the web interface, allowing fine-tuning of the search results for common keywords that you might expect your end-users to search against. You can manage the search index and factors through the administration page, which I'll discuss later.

Your DocSite will automatically rebuild the in-memory index after the application has started upon the first request for the search page or a particular keyword on the browse page. In the future I plan on adding an SQL Server search provider that you can use instead of the default in-memory provider so that the index does not have to be rebuilt each time the ASP.NET worker process is recycled or when changes to the application are published that will cause it to restart (like certain modifications to the web.config file).

Site Administration Page

The DocSite administration page provides a set of configurable options with application-scope.

DocSite Administration Page
Figure 5: DocSite Administration Page

To reach this page simply click the Admin link in the top-right corner of your DocSite. If you haven't logged in then you will be brought to the login page first. Enter the credentials that you configured when the project was first created and then click the Login button to continue to the DocSite Administration page.

Settings

Settings are stored in the DocSite.config file found at the root of the project. Changes made directly to this file will not take effect until the application is restarted; therefore, I highly recommend using the web interface to configure settings whenever possible so that you do not have to restart the application (and lose the entire in-memory search index).

There are two main categories in the administration page: General and Search. Changes to any options in either category are applied immediately.

The Create Index link allows you to regenerate the index on-demand. This is useful for ensuring that the index is generated immediately after the DocSite has been published and for updating the index after modifying keyword, weight or rank settings.

Client Settings

Letter bar

Configure the characters that will appear in the letter bar at the top of the browse index page. Add characters from your native language or remove existing characters without having to modify any code.

Sidebar size persisted

Controls whether a cookie will be used on the client to remember the last position of the sidebar's handle. This settings is enabled by default.

The Search category has a few subcategories for viewing statistics and managing the DocSite Search and Browse Index pages.

Index Statistics

This category provides read-only information:

Provider name

Name of the current search index provider. DocSiteMemorySearchProvider is the only implementation that will ship in DocProject 1.8.0; however, expect SqlSearchProvider in a subsequent release.

Last creation date

The last date and time that the index was created.

# keywords

The total number of distinct keywords found among all documents that were indexed.

# documents

The total number of HTML help topic files that were indexed.

Settings

Provides general index-related settings:

Root search path

Virtual path of the root directory in which the provider will begin the search, recursively. I highly recommend that you do not change this setting.

Public search enabled

Indicates whether unauthenticated users can use the DocSite search feature. Note that if you're logged in after disabling this setting you will still see the search box since it's always available to authenticated users. This setting is enabled by default.

Public browse enabled

Same as the option above but for the DocSite browse index page instead of the search page. This setting is enabled by default.

Keyword Settings

Settings related to indexed keywords and keywords in search queries.

Minimum keyword length

Defines the minimum number of characters that a word must contain in order to be included in the index. This setting is also used to ignore words in queries that have fewer characters than the specified value. The default value is 2.

Excluded keywords

Comma-separated list of keywords that must be excluded from the index and queries. You can add or remove keywords as you see fit.

Hot title keywords

Comma-separated list of keywords that have special semantics in titles of HTML help topics and are used to add additional value to matches in query results through the Title Rank Factors values (below).

You can customize this list if you want to improve search results with queries that contain full or partial matches to these particular words when found in titles.

Weight Factors

Weight factors are integers that affect the weight of indexed keywords. Note that weight is calculated while the index is being created and rank is calculated when a keyword matches a search query. The values in this section do not affect rank directly; however, the weight of a keyword does add additional rank to matches.

Early position keyword

Factor used to calculate the weight of a keyword depending upon its location in the document being indexed. Typically, the earlier the position the higher the weight.

The formula used to calculate weight is:

keyword position / source length * -{value} + {value}

Where {value} is the value of this factor (10 by default).

Title keyword

Factor used to calculate additional weight for keywords found in the titles of indexed documents. The base weight of title keywords is the value of the Early factor (above), but calculated relative to the title and not to the entire document. The value of this factor is then multiplied by the result. The default value is 1.2, which gives 20% more weight to keywords found in titles.

Title Rank Factors

Additional rank values for keyword matches that are found in titles.

Exact title hot keyword match

Additional rank for exact keyword matches found in titles and the Hot title keywords list (above). The default value is 250.

Partial title hot keyword match

Additional rank for partial keyword matches found in titles and exact matches found in the Hot title keywords list (above). The default value is 100.

Exact title keyword match

Additional rank for exact keyword matches found in titles. The default value is 50.

Partial title keyword match

Additional rank for partial keyword matches found in titles. The default value is 20.

Conclusion

In this blog post I described new features to come for the DocSite templates in DocProject 1.8.0 RC. These new features will surely be appreciated by your end-users since they make it much easier than before to locate and browse help topics. As a site administrator you now have a much easier way to configure some of the DocSite features and will have an easier time modifying the appearance and localizing text to meet your particular needs and the needs of your clients.

August 14, 2007

DocSites Online Part II

A couple more public DocSites have come to my attention and I thought I'd share them with my readership. You can read my original DocSites Online post here.

If you haven't been paying attention and you're not sure what a DocSite is, it's an AJAX-enabled ASP.NET Web Application project in Visual Studio 2005/2008 that provides a web interface for API documentation. The documentation can be built using Microsoft's Sandcastle utilities from managed assemblies and optional XML documentation files that contain your triple-slash code comments. DocSite templates come in two flavors: C# and VB.NET (although either template can build help for projects written in other languages, including managed C++).

DocSite templates are available as part of DocProject.

MbUnit 2.4

Ben Hall provided some great feedback on DocProject and has done an excellent job creating his DocSite at http://docs.mbunit.com/, which includes a custom TOC, topics and appearance.

According to the docs…

MbUnit is an advanced, extensible unit testing framework originally developed by Jonathon 'Peli' de Halleux and Jamie Cansdale.

Ben posted about the documentation's release at http://blog.benhall.me.uk/2007/08/mbunit-new-website-and-documentation.html.

You can read about the MbUnit project at: http://www.mbunit.com/.

Interactive Brokers C# API

Dinosaur Technology and Trading has ported a Java socket interface to C# for the Interactive Brokers API.

Their DocSite can be found at http://ibhelp.dinosaurtech.com/default.aspx and you can read about the utility at http://www.dinosaurtech.com/utilities/.

Nice work guys.

Style

The API topics for each DocSite were built using Sandcastle's Hana presentation style, which adds a number of improvements over MSDN's style; however, it seems that the final VS 2008 (Orcas) documentation will probably still use the VS 2005 style according to a post at Sandcastle's blog.

You can read about the Hana style here.

August 14, 2007

DocProject Conversion to 2008 and Backlog

In case anyone's interested, this post contains my notes for converting the code base of DocProject 1.7.0 to support Visual Studio 2008 Beta 2. After that I've included DocProject's backlog of tasks, which may contain some ideas that you'd be interested in seeing me implement in a subsequent release.

Notes on the Conversion from 1.7.0 to 2008

It was no easy task, though I don't think my experience is any indication of what must be done for the conversion of general 2.0 Framework applications since DocProject relies heavily on Visual Studio itself at runtime, which is uncommon (at least until future apps start using the new Visual Studio Shell).

While logging this information I was also running into various issues building the source code and running the program. In order to ease maintenance I made whatever changes I could to both the 1.7.0 and 2008 code bases to fix the bugs while keeping them in sync. As a result the code base of DocProject 2008 is 99.99% the same as DocProject 1.7.0, with a few special adjustments such as a different hard-coded installation path under the root installation directory, for example. Many of the updates that I logged were simply changes to references of version numbers but there were also many modifications to the InstallPrep and Installer projects. I don't think any bug fixes are logged here, however.

This is not an exhaustive list of what was required for the conversion. If the time comes again that I must convert DocProject's source I'll certainly follow these notes as a guideline, but I'm sure the process will require my attention.

  1. After installing DocProject on the build machine, copy the Source folder to "My Documents\Visual Studio 2008\Projects\DaveSexton.DocProject08\" and uninstall DocProject.
  2. Update the Installer project to search for "VSORCAS OR VCSORCASEXPRESS OR VBORCASEXPRESS", but rename the search and conditions to use 2008 instead of ORCAS; e.g., VS2008 OR ...
    NOTE: In 1.7.0 RC for VS 2005 the VSORCAS searches were removed to support side-by-side installation. The VS2005 searches should actually be replaced with VS2008 searches instead.
  3. Update the DaveSexton.DocProject.InstallPrep.CustomActions.Contents property to return ONLY the Orcas content.
  4. Update all of the projects, including the templates, to use the 3.5 framework. Note that the DocSite templates require a temporary modification to the web.config file before hand unless the modification is done manually. To automate it, use the properties dialog - Application tab for C# projects and Compile tab > Advanced Compile Options button for VB projects.
  5. Replace the 2.0 framework prereq with the 3.5 framework in the Installer project.
  6. Update the DaveSexton.DocProject VCProjectEngine reference to 9.0.0.0
  7. Update DaveSexton.DocProject.InstallPrep\PostBuildEvents\DaveSexton.DocProject.bat, and the Sandcastle, Sandcastle/Deployment and HtmlEditor projects' post build events:
        "%programfiles%\Microsoft SDKs\Windows\v6.0A\bin\gacutil.exe" ...
  8. Update DaveSexton.DocProject.InstallPrep:
    1. BootStrapper\vs_piaredist should be changed to vs90_piaredist.
    2. Copy in the vs90_piaredist files and update the Readme.txt file.
    3. Update the Readme.txt file in the project's root to reference vs90_piaredist instead of vs_piaredist.
    4. Contents\Installer\DocProject.AddIn file should be updated to version 9.0 in the HostApplication element.
    5. Replace all occurances of <ContentVersion>1.0</ContentVersion> with <ContentVersion>2.0</ContentVersion> in Contents\Installer\DocProject.vscontent
  9. Delete the DaveSexton.DocProject.DocProjectAddIn.GetMSBuildBinPath methods since they are no longer used (see next step)
  10. Delete all the code in the DaveSexton.DocProject.MSBuild.MSBuildAnyProject(string) class constructor above and including the local variable named, "vsVersion" and use the parameterless Engine() constructor instead. (There is a warning about this in the Error List when building the DocProject Add-In assembly).
  11. Delete Microsoft.Build... references from DaveSexton.DocProject and DaveSexton.DocProject.ExternalUI and replace them with references to the new 3.5 versions.
  12. Remove the DaveSexton.DocProject.MSBuild.BuildDocProject.MSBuildBinPath property and all remaining related code, including code that constructs and MSBuild Engine class using a custom bin path. (There will be warnings in the Error List after building the DocProject Add-In assembly).
  13. Update the DaveSexton.DocProject.InstallPrep\DaveSexton.DocProject.targets file by removing the MSBuildBinPath attribute from the BuildDocProject and CleanDocProject tasks.
  14. Update DocProject and DocSite project templates (found in the InstallPrep's Contents\[template name] folders):
    1. Change version to <ProductVersion>9.0.20706</ProductVersion>
    2. Append <TargetFrameworkVersion>v3.5</TargetFrameworkVersion> to the first PropertyGroup
    3. Change targets reference to <Import Project="$(DocProjectPath)\VS2008\bin\DaveSexton.DocProject.targets" />
      (NOTE: DocProjectPath environment variable must be defined - this is indicated below)
  15. Update DocSite templates:
    1. Change type guids to (actually, I don't think this step is necessary):
      C#: <ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{fae04ec0-301f-11d3-bf4b-00c04f79efbc}</ProjectTypeGuids>
      VB: <ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>
    2. Change web app target to <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\WebApplications\Microsoft.WebApplication.targets" />
  16. Global search for "8.0" just in case any more updates are required for "9.0" (NOTE: the binding redirect in the External UI project's app.config file should NOT be updated to 9.0.0.0)
  17. In the InstallPrep project delete the WiRunSQL.vbs file and the UpdateMSI.bat file, and then remove the Installer project's post build event.
  18. In the Installer project:
    1. In the File System Editor, change the Application Folder's DefaultLocation to "[ProgramFilesFolder][Manufacturer]\DocProject"
    2. Create a folder under the Application Folder and name it "VS2008"
    3. Drag each root folder into the VS2008 folder and then drag the files located in the root into VS2008 as well.
    4. Changeg the Version to 1.0.0
    5. Change the ProductCode and the UpgradeCode.
    6. Change the Subject to "DocProject 2008 (Full)"
    7. Change the ProductName to "DocProject 2008"
    8. Open the Registry Editor. Hard-code "DocProject" instead of using [ProductName] used in the key for the InstallPath value (for both the HKCU and HKLM).
    9. In the Registry Editor, change the value of each SafeImports key (DocProject value) to [TARGETDIR]VS2008\bin\DaveSexton.DocProject.targets
    10. In the Registry Editor, change the "8.0" key to "9.0" in each of the paths to the SafeImports values.
    11. For each SafeImports key (DocProject value) also udpate the condition to use 2008 instead of ORCAS; e.g., VS2008, VCS2008EXPRESS and VB2008EXPRESS
    12. Change the VS2008\bin\DaveSexton.DocProject.dll.config file's TargetName property to DaveSexton.DocProject08.dll.config
  19. Change the assembly and file version for each project to "0.0.1.0"
  20. Append "08" to each project's assembly name (Applications tab in the project properties dialog).
  21. Do a global search in the entire solution for "DaveSexton.DocProject (with a single quote prefix). Where appropriate, append "08" to the assembly name of references and upate the version to "0.0.1.0" as well, for all projects - not just DaveSexton.DocProject.dll.
  22. Rename the solution to "DaveSexton.DocProject08". Then locate the .sln file in the Installer project's File System Editor under the Source folder. Delete the existing file and add the new 08 .sln file.
  23. Rename the DocProject.AddIn and DocProject.vscontent files in the InstallPrep project to DocProject08.AddIn and DocProject08.vscontent. They're located in the Contents\Installer folder. Also update DocProject08.vscontent so that it references the renamed .AddIn file.
  24. Append "08" to the name of each template folder and update the Installer\DocProject08.vscontent file to point to the renamed .zip files.
  25. Update the InstallPrep project's Install.CustomActions class:
    1. Repeat the previous step on the static fields that reference the template zip files and the .AddIn file.
    2. Update the InstallEnvironmentVariables method to use "targetEnvironmentVariable, Target"
    3. Update the UninstallEnvironmentVariables to use "targetEnvironmentVariable, null"
    4. Delete the TargetBin property
    5. Change the buildPathEnvironmentVariable field to: private const string targetEnvironmentVariable = "DocProjectPath";
    6. Add a new property (after the Target property) named TargetFlavor that returns "Path.Combine(Target, @"VS2008\");"
    7. Update the CreateInstaller method to use TargetFlavor instead of Target.
  26. Update the DocProjectAddIn class's InstallPath property to return Path.Combine(installPath, @"VS2008\");
  27. Update the BuildOutput.bat file in the root of InstallPrep:
    1. Update the template names (only the second argument must be updated). For example: CALL :ZIP_TEMPLATE "%TemplateDirRoot%\VBDocSiteTemplate" VBDocSiteTemplate08
    2. SET InstallerName=DocProjectInstaller08
  28. Build the InstallPrep project once. Then open the Installer project's File System Editor and navigate to Application Folder\VS2008 in the tree:
    1. Delete the DocProjectInstaller.vsi file and replace it with the new DocProjectInstaller08.vsi file found in the InstallPrep project directory.
    2. Open the Setup\AddIn folder and delete DocProject.AddIn. Add DocProject08.AddIn from the Contents\Installer folder.
    3. Open the Setup\Templates folder and delete all of them. Add them back from the Contents\Templates folder (hidden in Solution Explorer), but only choose the 08 versions (previous versions can be deleted since this is a temporary folder and they are no longer in use).
  29. Run Code Analysis on each project, including InstallPrep but not the templates or the Installer. In some projects there may be certain rules that should be disabled. For example:
    1. Naming: CA1703 (resource spell-checker)
    2. Naming: CA1704 (identifier spell-checker)
    3. Naming: CA1716 (indentifiers that are keywords, such as "step")
  30. Make sure to exclude all detected dependencies in the Installer project before building it.
Build and Test:
Make sure that the solution builds without any errors. Then build the InstallPrep project and make sure that it completes without any errors.
(Check the build output window for DSZip errors). Build the Installer project and install DocProject on the build machine.
NOTES:
  • Install DSZip on the build machine.
  • Make sure to copy the "vs90_piaredist" bootstrapper folder into the "C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bootstrapper\Packages\" folder on the build machine.

Backlog

Here's some stuff that's been on the backburner for quite some time now.

  • Auto-merge changes to partial-build items. Concrete engines are responsible, but the BuildEngine class could copy all partial items to a temporary folder before a build. Then after the build completes, call the derived-implementation of protected abstract void MergePartialBuildItem(DocProjectItem oldItem, DocProjectItem newItem) for each file in the temp folder. Finally, each temp file would be deleted. This method should be added to the IBuildEngine interface as well. If an implementation can't or shouldn't merge the changes then it doesn't have to do anything with the temp file, or even open it.
  • Create an item template for Sandcastle's build component (the New Project Wizard may have to be responsible for referencing the BuildAssemblerLibrary assembly unless the item template supports some sort of wizard that can do the job – but this needs to be researched)
  • Components dialog that served as an interface for the .config files. It would be a list of components found in the Sandcastle assemblies, the project's assembly, and any registered assemblies. The order and parameters could be specified, somehow.
    • Create a wizard step that asks the user to choose from various build components to be included in the project's {root}\Components\ folder. For example, XAMLSyntaxGenerator.cs, which would automatically be registered in the .config file.
    • Allow users to add cusotm build components to a DocProject or DocSite via an item template.
    • List of "registered" assemblies on left (registered assemblies cannot be removed from the list if there are stack components that reference them)
    • Build component stack displayed on right ("New" drop-down button, with "External Component", like a browse button, and "Internal Component", that searches the project's assembly (also with "Add New" as an option, somewhere)
    • Up down arrows that control the order that components are executed in the stack
    • Reset button that will import the configuration file for the project's presentation, replacing all changes (Are you sure? dialog)
  • Have a start debug button ">" in the Sandcastle toolbar that will put Visual Studio into debug mode and attach to an external "msbuild.exe /project" process for debugging custom build componenents and the build process component. For DocProjects, this may be possible without a custom button by setting the startup program on the Debug tag, if it's possible to do that from the templates.
  • ApiFilter Issues:
    • reference method signatures?
    • discover generic type name? (currently, the id is parsed by searching for .name` where name is the value of the name attribute of the corresponding apidata element)
  • API Topic Management Dialog:
    • Add/Save/Load feature: Allow users to add a filter (regex or category) to a list that can be ordered using up/down arrows. Saving the list would create an XML doc. Loading the XML doc would recreate the list in the dialog, which could be applied by clicking the list's Apply button (each individual filter would be processed one at a time, top-down).
  • Attribute Filter Dialog for Syntax Generator ("Syntax Attribute Filter")
  • Automatic token replacement in code comments:
    • Have a dialog that lists token values that can expand to full HTML, which can be edited by clicking an ellipses button that opens an Html Editor dialog, for each individual token.
    • Users can import and export the list to and from an xml file.
    • Make the token replacement start and end strings (both "$" in the examples below) configurable in the tools options page in case the default values conflict with some other device. The values are required and cannot contain ANY white-space. Token replacement will only occur if the following regular expression is matched: \$[^\s\$]+\$
    • When customized, for example, if the start token is % and the end token is ^ then the expression would be: \%[^\s\%]+\^
  • Examples:
    Custom Tokens
    
    $isnull$      "is a <strong>null</strong> reference (<strong>Nothing</strong> in Visual Basic)"
                  (this could be built-in, but editable)
    $my-token$    "Hello <strong>World</strong>!"
    more custom...
    

    Uneditable Tokens (cannot be modified, imported or exported):
    $time$        DateTime.Now.ToLongTimeString(), in the current culture
    $date$        DateTime.Now.ToLongDateString(), in the current culture
    $datetime$    DateTime.Now.ToString(), in the current culture
    $user$        Expands to the current user's name.  e.g., Administrator
    $name$        Expands to the name of the member or type to which the comment applies.  
                  e.g., ToString, IsEnabled, BuildEngineProvider
    $fullname$    Expands to the full $name$.  e.g., MyNamesapce.MyObject.ToString, 
                  MyNamesapce.MyObject.IsEnabled, MyNamesapce.BuildEngineProvider
    $type$        Expands to the containing type of the member, or the type itself if the 
                  comment applies directory to a type.
    $-type$       Exapnds $type$ to a link.  Same as: <see cref="$type$"/>
    $-x|y$        Expands to a hyperlink: <a href="y" mce_href="y">x</a>
                  or <a href="x" mce_href="x">x</a> if the vertical bar is not specified.
    $project$     Expands to the name of the project: {project name}
    $-project$    Expands to the project link:  <a href="R_Project.htm" >{project name}</a>
                  (or the GUID form)
    $home$        Expands to a hyperlink with its url designated by the user.  By default, this token 
                  will be ignored unless the user enters a valid Uri and name.  (custom editor, I guess)
    $art$         Expands to the virtual artwork path; currently, /Art
    

Several of these ideas are already work items that you can vote on. Please let me know if any of them seem interesting to you. I have several other ideas as well that I haven't written down yet but I will when the time comes.

The only reason why I haven't started on the token replacement feature is because I think Microsoft already has an internal implementation that they use, but I'm just itching to create it myself. I'm waiting now to see if they'll make theirs public first before I attempt to reinvent the wheel, if in fact their implementation supports the feature set that I'm after.

August 13, 2007

DocProject 1.7.0 RC and 2008 Beta

The latest version of DocProject has been deployed.  Also, there is a new version called DocProject 2008, currently in beta, that provides support for Visual Studio 2008 Beta 2, Visual C# 2008 Express Beta and Visual Basic 2008 Express Beta.

You can find DocProject at http://www.codeplex.com/DocProject.

July 29, 2007

DocProject for Sandcastle 1.7.0 RC Preview

It's about that time again…

Just FYI, DocProject 1.6.2 RC had more downloads in the last month than any other version to date. Thanks to everyone who tried it and provided valuable feedback!

And BTW, I wrote a new article called How to Diagnose and Resolve Issues that provides guidance for gathering diagnostic information if you're having issues with DocProject and would like to try to solve the problem yourself or if you would like to ask for assistance.  You can find other How To... articles here.

In the next version, 1.7.0, there are a number of really cool features that I think people will like, as well as some bug fixes that should solve the problem with Team Build and provide a better experience using the HTML Editor. In this blog post I'll show a preview of some of the new features and describe my plans for them in the future.

Like usual, please let me know what you think so that with your support I can continue to make DocProject a useful community tool.

New Features

  • XML Documentation Editor
  • API Topic (Shared Content) Designer Improvements
  • Missing Dependencies List
  • External UI (DocProject.exe)

I'm still testing and finishing up some other tasks, features and bugs. I'm running a bit late since I was hoping for a mid-July release, but a few of the features that I will describe below somehow snuck into this release (that seems to happen a lot with me ;). I'm also quite busy with work but I think I might be able to manage a deployment next week, without any showstoppers popping up. So far, things are looking good though.

XML Documentation Editor

The API Topic Management Dialog now supports authoring XML documentation (summary, remarks and example) for all API members, not just the project and namespace summaries. See Figure 1 below.

I've created a new control named, ContentEditor and another control named, ContentListEditor, which is used for the new and improved content dialogs: API Topic Management (Figure 1) and the API Topic Designer (Figure 4). As you can see in both figures, there are now two tabs at the bottom for switching between design-mode and source-mode and a list of items that can be edited. Only one item can be edited at a time by selecting it in the list. After editing an item its name becomes bolded in the list until you save your changes.

Note: The content editor control actually provides another tab named, Preview, which raises an event that controls can handle to provide read-only HTML based on the changes made in the editor. Because this feature has not been implemented yet the Preview tab is currently hidden at runtime.

Figure 1: API Topic Management – Xml Documentation Editor

Xml Documentation

Remember that this is an xml documentation editor, which means that tags such as see, c and code may be used in Source mode even though they won't be formatted in Design mode. Sandcastle uses these tags when it builds your documentation to apply special formatting or to link to other topics.

One feature that I plan on adding in the future is the ability to drag nodes from the tree onto the designer and have it serialize <see cref="{id}">name</see> automatically. I'd also like to show see links as hyperlinks in the designer, but maybe as green or using double-underline to distinguish them from normal hyperlinks. Another possibility is to use the stylesheets from your DocProject's or DocSite's selected presentation style and then format all of the xml documentation tags in the designer as they will appear in the compiled documentation.

Another feature that I plan on adding is API type-specific sections in the item list. In other words, if you select a method node to edit then the editor should automatically list each of the method's parameters next to summary, remarks and example. Tags such as permissions and seealso should be listed as well, although I'd like to open a custom editor interface for these tags.

Currently, you can add custom tags to the list using DocProject's configuration file (DaveSexton.DocProject.dll.config, normally found in C:\Program Files\Dave Sexton\DocProject\bin), but they'll apply to every API element. I'd like to implement a feature that allows you to specify which sections apply to which API elements though (probably using DocProject's configuration file, partially shown in Figure 2 below).

Note: Do not confuse the DaveSexton.DocProject.dll.config file with the DocProject.exe.config file (I'll explain more on that below) located in the same directory.

Figure 2: DaveSexton.DocProject.dll.config xmlDocumentationTags attribute

HTML to XML Conversion

I've made a few improvements to my HTML Editor control and the way HTML is converted to XML (a few bug fixes).

The HTML Editor control that you see in Figure 1, which you may already be familiar with if you've used DocProject 1.6.0, is based on the WebBrowser control. The WebBrowser control is basically a managed wrapper around Internet Explorer. For the HTML Editor it's a wrapper around IE's design mode, which only produces HTML markup (in some cases, very poor HTML markup.) Sandcastle requires valid XML, so DocProject converts your HTML to XML automatically. The results of the conversion can be viewed in Source mode. The markup that you enter in Source mode is also converted automatically, so don't be surprised if when you forget to add a closing p tag, DocProject adds it for you when switching views or saving your changes :)

The HTML Editor is a work-in-progress, so I expect issues with HTML to XML conversions to spring up every now and then. Please let me know if you experience an error. If you can provide me with the exact steps to reproduce the problem and the markup that you used, I'll do my best to fix parser bugs and improve the editor's usefulness.

Persistence

Your comments are saved in individual xml files in your DocProject's or DocSite's Comments folder, at the root of the project. An {assembly name}.xml file is created for each assembly and one project.xml file is created to contain the project and namespace summaries. If you prefer you can edit these files using Visual Studio's XML editor. If you haven't edited any API elements for a particular assembly then its XML file will not be present in the Comments folder, but you can create it yourself. The name of an assembly's xml documentation file must be the display name of the assembly with .xml appended to the end or else DocProject will not be able to merge your documentation with imported documentation.

Merging

XML documentation that appears in your code is not shown in this dialog. By default, comments created in the dialog overwrite code comments, on a section by section basis; e.g., summary, remarks or example. But you can change this setting to give code comments precedence instead of being overwritten by selecting KeepSource for the Merge xml documentation setting found in Tools > Options > DocProject > Active Projects > Content.

Figure 3: Merge xml documentation options for documentation written in the API Topic Management dialog

As you can see in the figure above, the API Topic Management dialog can be accessed from the option immediately below the Merge xml documentation option.

Xml Documentation Clean-up

If you create documentation for a type that is later removed from your project then the documentation will still remain in the XML file for the assembly in which the API element is defined, however it will no longer be displayed in the API Topic Management dialog so you must remove it from the file manually. In the future I may add a feature that will allow you to clean up your xml documentation files at the click of a button.

API Topic (Shared Content) Designer Improvements

The API Topic Designer uses the new content list editor control and provides a list of content items that you can edit, such as the header, footer and various titles.   The items are loaded from the Presentation\Style\Content\shared_content.xml file in your DocProject or DocSite.  Changes to the items using the dialog are written directly to this file as well when they are saved.  Deleting an item in the file will remove it from the list and adding an item to the file will add it to the list the next time the dialog is opened.

Figure 4: API Topic Designer in 1.7.0

Just like in 1.6.0, the API Topic Designer appears as a step in the New Project Wizard with all of the new features.  The header section is the default item selected.

As of the June 2007 CTP release of Sandcastle, presentations have a hard-coded header and footer that will probably be inappropriate for your projects.  I've decided not to handle this in DocProject so you must delete the header and footer manually if you do not want them to appear in your documentation.  Although the New Project Wizard is a good place to do that, you can still update the header and footer at a later time using the API Topic Designer.

FYI, I've already requested in the forums that the Sandcastle team set these values to empty in the default presentations to save effort on behalf of end-users.

Text-only Content Items

Not all of the items are XML-editable using the API Topic Designer. Most are text-only since there will probably not be a need to use HTML formatting for them. Text-only items only show the Source tab and will escape any XML markup that you enter. To configure which items use the HTML editor on a per-presentation basis you must edit DocProject's configuration file (DaveSexton.DocProject.dll.config, normally found in C:\Program Files\Dave Sexton\DocProject\bin):

Figure 5: DaveSexton.DocProject.dll.config sharedContentXmlItems attribute

Missing Dependencies List

A while back, around version 1.3.0, I attempted to build documentation for DocProject using DocProject itself but I quickly found that a bug in Sandcastle's MRefBuilder program (actually, CCI) relating to binding redirection prevented me from doing so. In the last version, 1.6.2, I decided to try again but this time I found that DocProject's ReferenceResolver class wasn't able to resolve the Interop.MSHelpCompiler.dll reference (provides the Help 2.x compiler services) for the Sandcastle project (DaveSexton.DocProject.Sandcastle.dll). The problem is that DocProject's Sandcastle assembly is being resolved in the GAC, but the MSHelpCompiler interoperability assembly is not installed in the GAC, so it cannot be found. I do plan to fix this bug in a future release, but since I've expierenced this issue myself I assume that there may be other scenarios that cannot be handled automatically by the reference resolver.  This features was added so that users can manually provide the path to dependencies that cannot be resolved automatically.

FYI, nobody has actually mentioned any problems with DocProject's automatic discovery of source dependencies so I assume that this feature will be used quite rarely, if at all. But then again, it must be used by DocProject itself just to build its own documentation ;)

I suggest that you do not use this feature unless you get an error when building. The error will be produced by Sandcastle's MRefBuilder utility and will state something about one or more unresolved dependencies, mentioning the full assembly name as well. From the information in the build output window you should be able to locate and add the missing dependency yourself using the new dialog:

Figure 6: Missing Dependencies List

This dialog is similar to the External Sources dialog that was added to DocProject 1.6.2 RC. You can find both dialogs in Tools > Options > DocProject > Active Projects > Build.

To add an assembly (.dll or .exe that cannot be resolved automatically by DocProject) to the list you must enter the full path and name into the text box in the new row (the row with the *) or click the button with the ellipses on the right side to select a file. You can also edit existing rows or delete rows by clicking the row header and pressing the Delete key.

The button on the left side of each row will map an absolute path to a path that is relative from the project's root directory, like the example path in Figure 6. This is useful for team and build-server scenarios.

External UI

DocProject 1.6.0 provided preliminary support for Visual C# Express and Visual Basic Express editions of Visual Studio. Unfortunately, because add-ins are only supported by Visual Studio Standard and higher, Express users had to manually edit the project files of DocProjects and DocSites to configure them after using the New Project Wizard since all of DocProject's options and dialogs are hosted by the add-in.

DocProject 1.7.0 RC has an external UI that uses a property grid, just like DocProject's Active Projects tools options page, so Express users can configure their DocProjects and DocSites using the same tools as Visual Studio Standard+ users: 

Figure 7: DocProject.exe

For Standard+ users, though this feature may be handy at times, I still recommend using Visual Studio in most situations to configure your DocProjects (via the tools options pages) and to build them in Visual Studio since it will automatically build dependency projects for you. Also, the Include Project Output Dialog does not work in the external UI and if you add an image to the XML documentation editor or the API Topic Designer while using the external UI, the image will not be included as a project item in Visual Studio (although it will still be imported into the Art folder - you'll have to Show All Files and include it manually).

Automatically Open the Active DocProject

You can add a link to the program in Visual Studio for quick access by going to Tools > External Tools... and clicking Add (this can be done in Visual Studio Standard and higher as well if desired).

By specifying the arguments in Figure 8 below, when you run the External UI from Visual Studio's Tools menu the program will automatically open the active DocProject or DocSite in Solution Explorer (that is, the project that contains the current file that is opened or the current file/folder that is selected in Solution Explorer).

The DocProjectBuildPath environment variable used in Figure 8 is registered on your system when you install DocProject.  It points to DocProject's bin directory, commonly found at C:\Program Files\Dave Sexton\DocProject\bin.

Figure 8: DocProject.exe - Visual Studio Tools Registration

Building DocProjects and DocSites

While developing this program I realized that it would be quite easy to build a DocProject from this interface using DocProject's BuildController class, so I gave it a shot. It actually worked so well that I now build my test DocProjects and DocSites using the GUI for debugging purposes. Breakpoints in code being hosted by Visual Studio's process tend to time-out after a few seconds of inactivity, due to COM issues, but since DocProject.exe is fully-managed it makes it much easier to attach, debug and continue, without having to restart Visual Studio and load up an entire solution.

Figure 9: DocProject.exe build trace

DocProject.exe will not only build your DocProject's or DocSite's help, but will also build the project's assembly output using MSBuild first (remember, DocProjects and DocSites are MSBuild projects). During a build you'll see descriptive status information describing the current step and the progress bar, just like you see in Visual Studio. You can even cancel builds at any time.

The program will also remember some settings for when you restart, such as the last position, size and window state, with multi-monitor support, the position of the toolbars and the height of the build trace window (Figure 9).

Currently, this program does not support DocProjects and DocSites that have managed C++ projects as sources since they can only be built using a solution file, but I plan to look into implementing that feature in a future version of DocProject.

DocProject.exe Extensibility

The code that is used to configure DocProjects and DocSites in Visual Studio Standard+ is the same code that is executed by the external UI. This means that if you have created a custom build engine provider for your organization it will automatically work with the external UI. Not only does this include the options in your DocProjectOptions implementation, which will appear as new properties in the grid, but also any Visual Studio toolbars that are created by your provider will appear as well. For example, in Figure 7 above you can see the Sandcastle toolbar that appears in Visual Studio Standard and higher in the DocProject UI (the right-most toolbar).

Sandcastle and Sandcastle/Deployment are examples of build engine providers that are registered in DocProject's configuration file (DaveSexton.DocProject.dll.config, commonly found in C:\Program Files\Dave Sexton\DocProject\bin).  Several of the options that appear in the figures above, including the API Topic Designer and Management Dialogs are provided by the Sandcastle build engine provider plug-in.

The next major How To... article on my to-do list is guidance for creating custom providers that can be used across projects in an organization, but if you feel like testing the waters yourself check out the DaveSexton.DocProject.Sandcastle and DaveSexton.DocProject.DeploymentSandcastle projects in the source code and try creating your own plug-in.

Conclusion

With the 1.7.0 RC, Visual Studio Express users will be able to do much more with DocProject than they could before with the new VS-like external UI. As a documentation author, you'll be able to write long examples and remarks sections using an HTML Editor and store all of the comments in your DocProject's Comments folder, which is merged with your xml code comments automatically before each build. You'll also be able to use the HTML editor to design topics' shared content, such as the header and footer – like before – but with the addition of every other item that appears in Sandcastle's shared_content.xml file as well. And now you can configure your DocProjects and DocSites to reference dependencies that could not be resolved automatically, however rare that might be.

I hope you like the new features and I look forward to receiving your feedback.

July 10, 2007

Adding Additional Topics Using DocProject

Update 2/10/08: DocProject 1.10.0 RC provides first-class support for Sandcastle's conceptual build process, which compiles additional content written in the XML-based Microsoft Assistance Markup Language (MAML) to produce MSDN-like help topics, optionally in combination with auto-generated reference topics.  19 conceptual templates are provided by DocProject and the MAML schemas are provided for IntelliSense in Visual Studio's XML editor.  You can find more information in my blog, here and here.  If you want to include raw HTML topics into help builds, read this post for more information.

Recently I've been receiving more feedback than usual about the addition of non-API topics when generating compiled help with DocProject (also referred to as custom, additional or conceptual topics). In this blog post I'm going to describe DocProject's current support for conceptual content and my goals for better integration in the future.

Community Feedback

First, I want to say thanks to the users that have emailed me, added feature requests to the issue tracker or started discussions. It's been very helpful.

Special thanks goes to Ben Hall, a computer science student in the UK and also a Microsoft Student Partner (MSP), for his great blog post on how to extend my custom topics tutorial (discussed below in Creating Custom Topics with Partial Builds) to make it work in a more automated fashion.

Ben has also written another blog post on DocProject in which he discusses his experience with DocProject, and for which I have a few responses (a bit-off topic for this blog post though, sorry):

  • I know that developers can give up easily sometimes, myself included, when an open source tool doesn't work correctly the first time around. But as you've discovered there can be problems with Visual Studio that I just can't account for in DocProject so patience and a bit of effort is sometimes required to get things going. So thanks for trying DocProject again :)
  • You mention that all of the settings are saved in the project.xml file, but actually the settings are saved in the project file; e.g., .csproj or .vbproj. The project.xml file is simply used for the project and namespace summaries (but that will be changing in a subsequent release - I'm going to give each namespace summary its own file.)
  • Technically speaking, DocSites don't have a search – they have an index filter. I've already submitted a feature request in the issue tracker for a DocSite search, which I'm currently working on and have been planning for a while now. The indexer and search engine are being written from the ground-up, but are nowhere near completion yet. I also plan to provide an administration screen for managing the index and viewing statistics. The search will be in addition to the filter and will appear in the top-right corner of the DocSite template, just below the link to download the compiled help file.
  • The TOC icons, as you mentioned, can be changed easily since it's just an ASP.NET TreeView control. To change the icons open the DocSiteContents.ascx file and change the ImageSet attribute's value from WindowsHelp to Msdn, for example. Maybe I'll change the default in the next release :)

Thanks again Ben! And now back to our scheduled programming…

Custom Topics - But other tools are doing it!

DocProject supports the addition of conceptual content through its partial-build functionality, which I'll explain below. I'm aware that this may not be ideal for most authors.

If you've used other community projects that automate Sandcastle you may have noticed one important difference between them and DocProject: Other tools provide extensions to Sandcastle, such as Eric Woodruff's Sandcastle Help File Builder (SHFB), which apparently has support for additional topics, custom build components such as one that formats code examples, and even patches for some of Sandcastle's presentation bugs. I think it's really great that Eric has done all of that work, but my philosophy for DocProject is different (and no, it's not to slack off ;).

First and foremost, DocProject's purpose is the integration of Sandcastle into Visual Studio 2005 and to expose its core functionality through easy-to-use dialogs and the New Project Wizard. DocProject does not provide any custom build components or bug fixes for Sandcastle, and does not add any additional functionality to Sandcastle's standard build process. DocProject exposes Sandcastle's configuration and transformation files to end-users with minimal changes so that they can customize Sandcastle without being restricted to GUIs, the same as on the command-line without using DocProject or any other automation tools.

A feature that allows users to add additional non-API topics may require a custom build component, custom configuration file or custom transformation files to be shipped with DocProject, possibly breaking DocProject's open-Sandcastle-source policy. In addition, or alternatively, DocProject could make changes to Sandcastle's intermediary files during the help-build process to support the addition of conceptual content, but IMO that would be a hack. Sandcastle already supports conceptual content for Microsoft's internal use according to this blog post – it just hasn't been exposed to the community yet.

As soon as Sandcastle provides a standardized and supported way to include conceptual content in the canonical build process, I'll begin work so that DocProject will support it too.

If Sandcastle goes RTW before providing this functionality, which to be perfectly honest is what I've expected for a while now, then I will attempt to extend DocProject so that it supports the addition of conceptual topics without Sandcastle's support. But let's hope Microsoft realizes how important it is that Sandcastle provides this functionality before it is RTW.

I also plan to offer a Custom Topic Management Dialog that will allow users to manage their conceptual content and even create new topics based on an HTML template, specific for the Sandcastle presentation chosen for DocProjects and DocSites.

Once DocProject fully supports the addition and management of conceptual content it will be a practical solution for generating compiled help in Visual Studio 2005, with designer support, and even without the need to build API documentation.

If you haven't already then hopefully now you see why DocProject does not support the addition of custom topics in the same way as the other community tools.

Creating Custom Topics with Partial Builds

When I first started getting some feedback from the community about conceptual content, not too long ago, I decided to write a tutorial that would help users take advantage of the partial-build functionality offered by DocProject. The tutorial is meant for anyone that uses DocProject, but is especially useful for help content authors that are already familiar with Html Help Workshop's file formats and manually editing them to add additional content. However, this tutorial doesn't describe the end of DocProject's support for custom content, but currently it's the easiest way without having to modify DocProject's code-base.

Partial Builds

To see a partial build in action, load up a DocProject or DocSite and build it as you normally would, but make sure that it will output compiled help (configure your project to produce Help 1.x, for example). After it builds successfully, delete the compiled help file (.chm, for example) and try building the project again. Notice that there is only 1 build step this time since DocProject detects that the file dependencies used to generate the compiled help are still in-tact and that none of the source projects have changed. Only the compiled help file is rebuilt, forgoing the entire Sandcastle build process.

This allows authors to modify the Workshop TOC file manually (.hhc) for the addition of conceptual content and DocProject will just rebuild the compiled help, which will include your changes. If you're using a DocSite template then the DocSiteContents.xml file will be synchronized automatically during a partial build, so your changes to the Workshop TOC will appear in the DocSite's TOC as well.

Conclusion

DocProject does support the addition of conceptual content, but not in the way that most end-users desire or expect since it's not entirely automated. The lack of complete automation is because Sandcastle does not provide a standardized means yet for adding conceptual content, but once it does DocProject will integrate the process into Visual Studio 2005, without hiding the implementation details from end-users. The process should still be flexible and customizable to a certain extent. DocProject, as with several other features of Sandcastle, will provide a management dialog for authors to add, edit, delete and organize their conceptual content. As such, DocProject may eventually be a productive environment for creating and managing help topics and building compiled help inside Visual Studio 2005, even without API topics to be documented.

I'd like to know your thoughts on this so please leave a comment here, start a discussion on CodePlex or contact me using my site's contact form. Thanks!

July 06, 2007

DocSites Online

I was browsing the web the other day looking for community feedback on DocProject and I was pleased to come across two public websites that use DocSite templates to provide public API documentation over the web.

XML Library API

The first DocSite that I found is at http://www.xmllab.net/mvpxml. It was created by Microsoft MVPs in XML technologies for their XML Library API on CodePlex. Nice work guys :)

Full Motion Racing

I stumbled across another DocSite at http://lfs.fullmotionracing.com/ for a project called, LFSLib.NET - A Live For Speed InSim/OutSim/OutGauge API. More information about the project can be found at the author's blog: http://www.claassen.net/geek/blog/.  The latest post even mentions DocProject: LFSLib.NET 0.13b released.  Thanks!

Others?

If anyone else has a public DocSite that they'd like to share with me please let me know :)

June 26, 2007

DocProject for Sandcastle 1.6.2 Release Candidate

DocProject 1.6.2 was released and is now available for download on CodePlex.

I want to thank all of DocProject's users that took the time to report bugs and provide diagnostic information to me. You've been helpful.

Recently, Anand blogged about DocProject and some of the other community projects that can automate Sandcastle to generate help for managed APIs. After his post, DocProject's rate of download increased by three times and then kept increasing. And since then I've certainly received more feedback than usual. Thanks Anand!

Updates

The latest release addresses some issues that users were reporting relating to the refresh version of the June 2007 CTP of Sandcastle, which was released a few days after its original release. I would have updated DocProject sooner if I knew :)

There is also support now for adding external sources (assemblies and corresponding xml documentation). This new feature also provides support for Web Site Projects since you must use a Web Deployment Project to generate a managed assembly, which can be added as an external source for your DocProject or DocSite.

In Progress

Currently, I'm looking at providing the ability to create and manage custom topics. I noticed that in Sandcastle's June 2007 CTP refresh there are some transformations and files related to conceptual topics. That sounds interesting to me and may be the feature that people have been asking me to add. I'll definitely research this more.

June 15, 2007

From Setup and Deployment to Self-Extracting Installer

In this blog post I'm going to describe how you can use a Setup and Deployment project in Visual Studio 2005, along with a few other tools, to create a single, self-extracting executable to install your managed software and prerequisites. I'm also going to provide a solution to one of the problems that I encountered when I recently attempted to create a self-extracting installer for one of my open source projects.

Introduction

Using a Setup and Deployment Project in Visual Studio, you can easily build a single Windows Installer file (.msi) that allows you to redistribute your custom software to your end-users, providing an automated and familiar setup experience. If your software depends on other software such as the Microsoft .NET Framework 2.0, as it probably will, then you can install them as prerequisites by having your Setup and Deployment project create a Generic Bootstrapper executable - Setup.exe. End-users should execute the bootstrapper instead of the .msi so that it will discover which dependencies are missing on the target machine, download and install them. The bootstrapper will then automatically run the .msi to install your software as long as the prerequisite installations were successful.

You can also develop custom Generic Bootstrapper packages and have them show up in Visual Studio's Prerequisites dialog, along with predefined packages such as the Microsoft .NET Framework 2.0 and Windows Installer 3.1. Custom packages can be configured to download the prerequisites from a website or they can be deployed along with the bootstrapper executable and your project's .msi file in their own sub-directories, which is extremely useful if the prerequisites are not available for download from some third-party website and if you don't have the means or desire to host them on your own web server.

User Experience

Having a bootstrapper to automatically install your software's dependencies is obviously useful, but your end-users will probably expect a single file to install your software and might be surprised to discover that they must download a Setup.exe file and an .msi file. You may also be tempted to deploy a Readme.txt file along with the installers to instruct users to run the Setup.exe program instead of the .msi. If you have added custom prerequisites to your bootstrapper then you may have to deploy them along with the installers as well if they aren't available for download from a web server. But since your end-users aren't going to be happy having to download several different files and then manually create the appropriate folder structure expected by Setup.exe, you're forced to zip all of the files and sub-directories together. An end-user must unzip the files, browse to the target directory in Windows Explorer and then double-click the Setup.exe file to start the installation.

All-In-One Deployment

So how about deploying a single, self-extracting executable that can dump all of the files into a temporary directory and then run the Setup.exe program automatically? There are several commercial products that can produce self-extracting zip files, but if you're like me you'd prefer a no-cost solution for something as simple, practical, and as seemingly common as having a single executable to install custom software.

Enter IExpress. Pre-installed on Windows systems, this program can create a self-extracting installer by compressing the files of your choice and may be configured to execute your Setup.exe bootstrapper upon extraction. The result can be distributed as a single, stand-alone file that contains the bootstrapper executable, any prerequisites that must be deployed along with your installer, and the .msi installer itself. That's right, pre-installed on Windows!

Start > Run > iexpress

The Directory Structure Problem

However, this amazing tool doesn't come without some pain. IExpress 2.0, the current version that ships with Windows Vista, doesn't preserve the folder structure of the files that it compresses. But the Setup.exe bootstrapper program is automatically built to look for each prerequisite that is not already being downloaded from a web server, in a local folder of the same name as the prerequisite itself. This obviously presents a problem when using IExpress since it will extract each prerequisite installer into the same temporary directory as the Setup.exe bootstrapper. Running a simple test reveals that the Setup.exe bootstrapper complains that it cannot locate one or more of the prerequisite installers.

But have no fear, there is a way. In order to better explain the process of using IExpress to extract and run the Setup.exe bootstrapper program with local file dependencies I'm going to use an example.

DocProject

DocProject is software that I've written to integrate Microsoft's managed API documenter, Sandcastle, into Visual Studio 2005. Up until the 1.6.0 Release Candidate of DocProject its installer shipped as a stand-alone Windows Installer file (.msi). But as of 1.6.0, a few of Microsoft's Primary Interop Assemblies for Visual Studio have become a prerequisite (namely, Microsoft.mshtml and stdole), which means that vs_piaredist.exe must be deployed along with the .msi. In order to ensure that end-users have Microsoft's redistributable assemblies installed on their systems before the DocProject installer is executed, I've create a Generic Bootstrapper package for vs_piaredist.exe that is required by DocProject's Setup and Deployment project.

As described in the introduction, DocProject's Setup and Deployment project builds a Setup.exe bootstrapper that checks for the prerequisites on the target machine and then, if the prerequisites aren't found, executes the vs_piaredist.exe file to install them. After the prerequisites are installed or skipped, the Setup.exe program then runs the .msi to install DocProject on the user's system.

When researching DocProject's prerequisites I discovered that Microsoft does not provide vs_piaredist.exe over the web, which means that I could have either zipped it with DocProject's installer, hosted it on my own web server, or figured out a way to include it into IExpress. I chose the latter since it requires the least amount of effort on mine and the user's part, but wasn't sure how it could work or if it was even possible at all.

Solution to the Directory Structure Problem

The Setup.exe bootstrapper is actually a generic executable that has a few null resources. When you build a Setup and Deployment project that has prerequisites, the MSBuild GenericBootstrapper Task is used to make a copy of the generic Setup.exe program. The copy is then loaded with a resource file that tells the bootstrapper how to install your program's prerequisites. Unfortunately, the MSBuild task doesn't provide the option to have the configuration resource use prerequisite installers found in the target directory, so you must manually update the appropriate resource file to remove the hard-coded path that looks for prerequisites in a sub-directory of the same name.

Sounds scary, but it's actually quite easy to do. Here are the instructions to create DocProject's installer:

  1. Build the Setup and Deployment project.
  2. Open the Setup.exe program in Visual Studio's resource editor:
    1. File > Open > File (or Ctrl+O)
    2. Browse to the bin directory of the Setup and Deployment project and you'll find the Setup.exe file in the folder for the current configuration (e.g., Debug or Release), along with the other installer output.
  3. Double-click the resource named, SETUPCFG in the 41 folder
  4. Search for vs_piaredist\, exactly like that - with the trailing slash. Two (2) occurrences should be found. For each, make sure the entire vs_piaredist\ string is highlighted and press the Delete key to remove it from the resource.
  5. Save the resource file and the Setup.exe executable will be updated automatically.
  6. Start > Run > iexpress
  7. Create a new package by following the IExpress wizard's steps and make sure to include the following files:
    1. The Windows Installer file (.msi).
    2. The Setup.exe bootstrapper file.
    3. The vs_piaredist.exe file.
  8. Note: You should select the Store files using Long File Name inside Package option in one of the last wizard steps.

Step #4 is how to reconfigure the Setup.exe bootstrapper to look in the current directory for vs_piaredist.exe instead of in a subdirectory of the same name. We do this by simply removing the folder from the path in two different locations within the same resource file.

You should repeat step #4 for each of the local prerequisites that your Setup.exe bootstrapper will install, but instead of vs_piaredist\ use the name of the folder as it appears in your Setup and Deployment project's bin directory.

Conclusion

Visual Studio's Setup and Deployment project, the Generic Bootstrapper and IExpress are powerful tools that you can use to create installers for your custom software.  Using IExpress to build a self-extracting installer that automatically runs your software's Setup.exe bootstrapper requires additional steps if the bootstrapper depends on prerequisite installers that are deployed with your software. By using Visual Studio's resource editor you can modify the Setup.exe file to look for prerequisites in the current directory.  This allows you to build a self-extracting installer that contains all of your software's depencencies, providing a nice installation experience for your end-users.

June 01, 2007

DocProject for Sandcastle 1.6.0 RC Preview

In this blog post you'll see and read about some of the new features of DocProject that will be released with the 1.6.0 Release Candidate. But first, I'd just like to say thanks to everyone who downloaded DocProject and provided feedback. I appreciate it. Please keep the comments and feedback coming!

If you're only interested in screen shots – scroll down :)

1.6.0 Changes

This is one of the biggest releases yet, IMO. Much of the code-base has been refactored to support new features and to improve existing ones. Changes were made not only to the source code, but also to the solution and project files, and to almost every aspect of DocProject. There were also several new additions to the code-base in the form of source code and new projects. But don't worry; the end-user experience has not changed other than to include new features and improve existing ones. And the source code, for the most part, is still basically the same. Your existing knowledge of how to use DocProject will still apply, but it will no longer be the whole story.

Please note that existing DocProjects and DocSites may no longer work as expected. You are encouraged (if not required) to create new documentation projects again by removing existing ones and using the New Project Wizard to recreate them. I have not tested older projects with DocProject 1.6.0, but I suspect that they will not be able to complete a help-build.

Solution and Projects

The first change to the solution was to start using solution folders to categorize the projects. The Sandcastle code-base was then pulled out of the Add-In project (which has been renamed) and included in a new project named, DaveSexton.DocProject.Sandcastle, which appears under the Plug-Ins solution folder along with the Sandcastle/Deployment project. There is also a new project named, DaveSexton.Controls.HtmlEditor. I'll leave that one alone for now and let you figure out its purpose for yourself ;) There have also been several changes to the InstallPrep and Installer projects, although existing functionality has not really been removed or altered, for the most part. The preparation process is still the same as it was before, but with some additional files and requirements.

Building the Installer

If you're planning on building the Installer project then the only concern you should have is that the HtmlEditor project requires the Microsoft.mshtml PIA to be installed on the target machine, so the installer requires a bootstrapper to be built (Setup.exe) that includes the redistributable vs_piaredist.exe file, which installs the required dependencies. In order to include MSHtml in the bootstrapper I created a prerequisite bootstrapper package that is included in the InstallPrep project under the Bootstrapper folder, the contents of which should be copied to Visual Studio's bootstrapper directory manually. The Readme.txt file that appears in the same folder explains its use and also provides import legal information that you must agree to before using DocProject's source code. The agreement is also presented in a dialog when DocProject is installed, but only if the required dependencies must be installed on your system. You should read this file before attempting to build and redistribute the installer and the source code.

Although I haven't tested yet whether building the installer will fail without having the MSHtml bootstrapper package present on your system, I assume that it will. I'll test this to make sure and update the appropriate wikis on CodePlex to indicate end-user requirements to build the installer using the source code, but for now just assume that you must follow the instructions in the Readme.txt file.

1.6.0 Features

Here's a preview of some of the new features that you can expect in DocProject 1.6.0 RC. I'm still testing some of them on Vista, XP, and Windows Server 2003, but things are going well. I expect to deploy by Monday June 4, 2007.

Installation

The installer will now detect the following versions of Visual Studio on your system and automatically install the templates and Add-In, where appropriate, for both Just Me and Everyone installations. Having more than one version of Visual Studio on your system at the same time is acceptable since they can all share the same templates or have their own copies if you install for everyone:

  • Visual Studio 2005 Standard+
    • The Add-In and all templates are installed.
  • Visual C# 2005 Express
    • Only the C# Project Documentation template is installed.
  • Visual Basic 2005 Express
    • Only the VB.NET Project Documentation template is installed.

Note: Currently, the DocSite templates will not work in Visual Web Developer 2005 Express.

The DocProject installation directory will always have the same contents regardless of what versions of Visual Studio are on your system when you install DocProject.  The only variance based on the different versions of Visual Studio are updates to the registry and the location where the Add-In and/or templates are copied.

There is only limited support for express editions, which I'll detail in the release notes for the 1.6.0 RC. Basically, express editions get the New Project Wizard and use the MSBuild task, but projects in express editions do not have Add-In-specific features such as the Include Project Output Dialog and the Tools > Options > DocProject pages. Users can still configure DocProject by editing the project file manually. However, Sandcastle is still configurable in the same way as the full version of DocProject; i.e., by editing files in the Presentation folder inside Visual Studio. Projects are also built in the same way, with support for a build process component as well.

Visual C++ Source Projects

DocProject now provides support for Visual C++ projects as sources for DocProjects and DocSites. Reference Visual C++ projects as you would any other project reference and their APIs will be included in the compiled help.

Html Files Names: GUID or Friendly

By default, the Sandcastle Build Assembler utility, which generates the HTML topic files, is now configured to use GUID file names instead of friendly names that corresponded to the names of the API topics, as used previously. This allows projects that have long, nested directory structures to build without throwing a System.IO.PathTooLongException (special thanks to the people who pointed this out to me :). You can change a setting to use the friendly names if you want: Tools > Options > DocProject > Active Projects > Build > Use friendly HTML file names (disabled, by default).

MSBuild Task and Command-Line Support

The template files for DocProjects and DocSites have been refactored to invoke a custom target named, BuildHelp, which references the DaveSexton.DocProject.targets file.  The targets file, located in DocProject's bin folder, starts the assembly-build process and then, afterwards, starts the help-buld process, which is executed by the Add-In in much the same way that it has in previous versions of DocProject.

To support this feature the installer creates an environment variable named, DocProjectBuildPath, which points to DocProject's bin directory. The environment variable is created with user-scope if you select Just Me in the installer and system-scope if you select Everyone.

The MSBuild task simply delegates the help-build process to the DocProject Add-In, which has been refactored so that it can execute outside of the Visual Studio IDE as well as inside. In other words, MSBuild is now used to start the build process and DocProject's Add-In takes over, using the appropriate method depending upon whether it has been initialized as a Visual Studio Add-In or simply invoked out-of-process by the MSBuild engine. This means that the Add-In will build your project files when they are used as Visual Studio projects, as they always have been, but also on build servers that use MSBuild - without running Visual Studio. No changes to the DocProject or DocSite project files are required for them to be buildable in any environment.

You can build DocProjects and DocSites on the command-line by simply running MSBuild and specifying the project file as an argument. For example, open the Visual Studio Command Prompt and enter the following to build a C# DocProject named, "your project":

msbuild.exe "c:\your project.csproj"

Note: Visual C++ projects must be built by the solution file. MSBuild will warn you about this if you attempt to build DocProjects or DocSites that reference a Visual C++ project. In that case, just pass the solution file to msbuild.exe.

Your DocProjects and DocSites can be specified as targets in other MSBuild project files on a build server as long as DocProject has been installed on the server (or at least with the DocProject assemblies and appropriate configuration present, even if DocProject is not actually installed). Make sure that the relative references to other projects in the DocProject or DocSite project file are valid.

New Project Wizard Header

The first change that you'll notice when creating a new DocProject or DocSite is in the New Project Wizard's header, modified slightly just to keep things interesting ;) (See Figure 1 below)

New Project Wizard – Choose a Sandcastle Presentation

The first useful change that you'll notice in the New Project Wizard is that the Choose a Sandcastle Presentation step provides readable names and a short description of the presentations that are available. (See Figure 1 below)

The data is located in DocProject's configuration file (commonly found at, C:\Program Files\Dave Sexton\DocProject\bin\DaveSexton.DocProject.dll.config) in the new presentation configuration section. This section describes the presentations that are installed with Sandcastle, indicating a human-readable name and description, and providing the locations to all of the tools and directories that DocProject expects. There is also a place to add custom Regular Expression and XPath expressions that will be used by DocProject to automatically transform the imported Build Assembler configuration file (sandcastle.config) when creating a new DocProject or DocSite. You can modify, remove or add new transformations to customize the default configuration files as they are imported into new projects.

Note: You can use the new presentation configuration section to add your own custom presentations too!

Figure 1: New Project Wizard - Chooose a Sandcastle Presentation
Figure 1: New Project Wizard - Choose a Sandcastle Presentation

As you can see from the screen-shot, the wizard looks a little bit different now. Other than the slight change to appearance and the new pre-load behavior, it functions identically to earlier versions of DocProject.

New Project Wizard – Create Shared Content

The next feature is the ability to create a header and footer using an Html Editor. The header and footer content are shared by all API topics and can be found in your project's Presentation\Content\shared_content.xml file, which you may edit manually as well. I'm also considering whether to add more sections that would correspond with other items in the shared_content.xml file, for a future version of DocProject.

Figure 2: New Project Wizard - Create Shared Content Figure 2.1: Html Editor Context Menu
Figure 2: New Project Wizard - Create Shared Content
Figure 2.1: Html Editor Context Menu

Notice in Figure 2 that the Header and Footer sections may be toggled between visible and hidden states by clicking their respective title bars. The box next to the title will contain a plus sign (+) when a section is collapsed and a minus sign (-) when a section is expanded.

The header and footer may be edited at a later time using the API Topic Designer, which is located at Tools > Options > DocProject > Active Projects > Content > API Topic Designer. Locate this setting for your project and click the ellipses button to open the editor, which is identical to the editor in the Create Shared Content wizard step, but without the wizard's chrome.

The Html Editor that is used here may be the beginning of a topic editor control as well; although I'd much rather use Visual Studio's HTML editor instead of reinventing the wheel, which would not be easy. One of the reasons that I wanted to use Sandcastle inside Visual Studio in the first place is so I could use its HTML and XML editors, but more on this discussion another time :)

API Topic Management Dialog

The API Topic Management dialog provides some new options for filtering API topics based on categories and regular expressions, such as the ability to find the next match only and to specify whether the dialog should automatically show matched topics in the TOC (i.e., if a matched node is hidden then its parent nodes will be expanded until it becomes visible, and then it will be scrolled into view).

Figure 3: API Topic Management Dialog - Topic Filters
Figure 3: API Topic Management Dialog – Topic Filters

The TOC is loaded on-demand now, by initially loading only the root topics and then loading the children of each individual topic as they are manually expanded for the first time. This saves some initialization time, but may increase the time that it takes for the filters to work. For this reason, the filters now execute asynchronously and the find next match only option is selected by default.

The API Topic Management dialog also provides support for editing project and namespace summaries in text mode (the contents of which must be valid xml just like with code comments) or using an Html Editor, as in Figure 4 below. You can freely switch back and forth between text mode and html mode and your changes will be persisted.

The Summary Text and Summary HTML tabs are only activated when the project node is selected, as in Figure 4, or when one of the namespace nodes is selected. For example, the Contoso namespace that appears in Figure 4 can be selected in order to edit its summary. If the AppConfigElement node is selected, however, the summary editor will be disabled.

Figure 4: API Topic Management Dialog - Summary HTML Editor
Figure 4: API Topic Management Dialog – Summary HTML Editor

You can open this dialog by locating Tools > Options > DocProject > Active Projects > Configuration > API Topic Management and clicking the ellipses button. It's in the same place that it was in 1.5.0 :)

And More…

There are other features that I haven't discussed here, but I think the ones that I've mentioned are probably the most important.

I hope you're excited to use these new features, since I worked hard to create them. I'd love to know what you think about them and your ideas for other features, so please use the contact form here or on CodePlex, or leave a comment on this blog post and let me know what's on your mind.

Thanks!