For the past few weeks I've been experimenting with a number of new and not-so-new technologies that seem to combine very well.  The technologies include:

• ASP.NET MVC:  This is a new framework from Scott Guthrie's team at Microsoft that provides a real model-view-controller implementation for ASP.NET.  The framework includes a robust URL routing engine that is flexible enough to do real RESTful routing, and can generate URLs from the routes (allowing you to change your routing later without breaking your app).  You don't need to use MVC to use the routing engine either.
• IIS 7.0: The new version of IIS is lean and modular, and supports integration with ASP.NET applications.  This allows individual apps to plug into the processing pipeline and completely take over processing from IIS.  For me, this allows custom authentication plus clean RESTful URLs with no file extensions.
• ADO.NET Entity Framework:  This is the new ORM tool from Microsoft.  It can reverse-engineer your database and generate a data model with mapping between the physical database, logical data model and entity objects.  I and a colleague of mine have written a handful of serious ORM frameworks, and although so far I've really only kicked EF's tires, it seems to do everything I need.  A key feature for me is that it supports querying using LINQ to SQL (among other methods).
• DI: Dependency Injection is a technique for decoupling the components of an application to enable future extensibility and better testing.  I haven't started implementing this yet in this app, but I plan to very soon.  A couple of frameworks I'm looking at are Unity and NInject.
• REST: Not so much a technology as a mindset, REST has intrigued me ever since I read RESTful Web Services a few months ago.  Microsoft tools haven't supported it well at all - until now.  I thought this was a good time to explore an alternative (OK, "non-Microsoft") way to design and build web applications.
• Ajax:  I've used Ajax in web apps for several years now, but only for specific, narrowly-defined purposes.  I wanted to build an app that leveraged it to the hilt, and an app based on RESTful design seemed to present the best opportunity to do so.
• Dojo:  This is a client-side Javascript framework that provides a cornucopia of capabilities including Ajax support, widgets, fancy UI effects, event wiring, etc. etc.  In many cases it can eliminate the need for server-side controls.
• CSS:  OK, this is not new, and I've used CSS for years; but I wanted to create an app that leverages CSS for all style information, and avoids cluttering the document with it.  In particular, I wanted to avoid using tables for layout, except for tabular data.

For my guinea pig, I chose to reimplement the web application that supports my audio product for Second Life, SlipStream.  The new version will be completely web-driven, eliminating almost all notecard configuration within the device in-world.  The system must do the following:

• Allow management of customers, plus their devices, playlists, managers, etc. from both a browser-based web UI and scripts in Second Life (SL);
• Allow indirect but real-time searching of the Shoutcast directory;
• Provide configuration and stream information to device scripts in SL;
• Track the current state of devices in SL, based on the calls they make to the application.

The requirement to support both SL script clients and web browsers causes some issues.  First, the scripting language in Second Life, LSL, is pretty limited when it comes to making HTTP calls, despite being compiled to and run on Mono (this feature is currently in an almost-production stage).  You can't, for example, set custom request headers or check the response headers, and there's no facility for parsing JSON or XML data returned by the call.  Plus, the size of data that can be sent and received is severely restricted, to 255 bytes per call last time I checked.  And HTTP requests are throttled to 100 per 100 seconds per object in SL.

I'm dealing with most of this by carefully designing the resources I expose and the representation formats I use when talking to a script client in SL.  But another wrinkle is authentication - SL scripts will need to authenticate on every call by passing the appropriate query string parameters.  Browser clients, however, are asked to log in only when they try to access a restricted resource, and they will get an authentication token they can send back on subsequent requests.  The authentication scheme is pretty much completely custom as a result.  It consists of an HTTPModule that checks for the necessary credentials, and establishes a custom security principal and identity for the current request. 

Authorization is done pretty much the normal way.  The app only has three classes of users:  anonymous users, authenticated normal users, and administrators.  For simple, "static" authorization, the controller action methods use a PrincipalPermission attribute to check membership in a role.  However, this type of checking is rarely enough.  Typically I would provide another layer of mapping for static authorization: instead of Users <-> Roles, I would have Users <-> Groups <-> Permissions.  This can be done using the .NET administrative tools, but most people aren't aware of that and don't use them.  Anyway, for this app, the simple role-based authorization suffices for static authorization.

But many apps also have what I call "dynamic" authorization - that is, authorization that is dependent on current application state, typically the resource that the user is trying to access.  An example of this would be a system where user Fred can only access documents he created, but not other people's documents.  In this case, a simple role check won't cut it - you need to find out who authored the document being requested, and then see if that matches the current user.  Anyway, this application has some scenarios like that, and so I have some helper code built into my controllers to do this type of checking.

Just for grins, I also want to support conditional HTTP GET on the resources and actions where it's appropriate (which as it turns out, aren't that many, but hey).  So, I wrote some code in a base class for my views that tacks on the appropriate response headers based on the timestamp of the current resource from my database, and examines incoming headers to see if it can optimize the response body away. 

The default routes provided in the MVC templates from Microsoft are not (IMHO) completely RESTful - they include the action in the URL:  for example /{controller}/{action}.  I felt that the action should be expressed by the HTTP method used to make the request, so I reworked the routes accordingly:  /{controller}/{id}/{format}, for example. 

The application is completely stateless; that is, it does not use session state at all.  This allows it to keep working through a restart - the user's authentication token will be accepted on the next request after a restart, and the user never knows the difference.  The user cannot fake an authentication token because it's encrypted on the server using the server's private key;

On the client side, the app uses CSS for all styling except tabular data, including form layout.  All the forms submit via Ajax, eliminating postbacks and allowing things like password fields to retain their values when you submit.  I'm using Dojo for the Ajax calls and for a few other things here and there... I'll be looking at expanding to use it for all my form widgets, but whether I stick to that approach will depend on whether the resulting code is XHTML- and CSS-compliant.

So far, I have the basics of the application working:  authentication, authorization, fetching data, supplying forms, handling form submissions, etc.  I need to do some more styling and finish building out the rest of the features, but it's starting to come together.  When I get the app done, then it will be time to build out the device scripts in LSL, package the product and start selling! 

Let me begin by saying that despite ASP.NET's strengths, its state management leaves some things to be desired.  If you've read some of my other blog posts you might not be surprised to hear me say that... I'm known in my office for being an old codger (at 37) who complains about lots of things.

The first issue is that the state management mechanisms in ASP.NET mix the concepts of scope and lifetime.  It's true that there is a loose correlation between the two; for example, Application state is scoped to all users of a given application, and lasts for the duration of that application.  Session state is accessible to the session, and lasts for the duration of the session.  The third option, ViewState, is scoped roughly to the session (actually more narrowly), and lasts across multiple requests for a single form.  You could say it's scoped to the form, but I think it's useful to define it based on lifetime.

Now examine the statements above.  The phrases "application state" and "session state" are pretty intuitive... but "view state"?  That should have been named "form state", except for the inconvenience of wanting it for individual controls as well as the form.  The distinction between lifetime and scope is never made, perhaps because with only these three options, the distinction is somewhat moot.

But once you're aware of the distinction, other possibilities present themselves.  For example, why can't we have state that persists for the duration of a "business process" (across multiple forms), instead of for the lifetime of a session?  What about state scoped to multiple users - all administrators, for example?  Or state with a lifetime spanning multiple sessions?  All of these would be useful for certain scenarios.

For anything but the three supported options, you're on your own.  Web developers have gotten good at using all the various techniques at their disposal - for instance using cookies to preserve state across sessions, using page-level variables to maintain state during the processing of a single page, etc.  But basically, once you need more than what the three main state facilities give you, you're on your own.

So there's a learning curve issue here in that state is not accessed through a consistent mechanism - new developers must learn not only how to use Application, Session, and ViewState, but cookies, Cache, page variables, hidden HTML fields, HTTP context, thread-local storage, etc. etc. 

The problem is worsened by the fact that each of these mechanisms has implications relating to the physical implementation, so the developer has to worry about those too.  For example, using the logical concept of "state scoped to the form with the lifetime of the form" means using ViewState, which means sending data to the client and back on each request.  With the exception of Session, ASP.NET doesn't really give you any options when it comes to the physical implementation.

As if all that weren't enough, in a multi-tiered application, maintaining state should not be be the presentation layer's concern.  When the presentation layer maintains state, one of two things will generally happen:  either the presentation layer will need to pass state to the business layer, or the business layer will need to know more than it should about the presentation layer.  Both are bad.

Finally, none of the state mechanisms in ASP.NET really offer immutability as a way to address multi-threading issues.  This might or might not be considered a fault of the platform, but if it existed and were transparent it would be handy.

In summary, I think there are a number of problems with ASP.NET state management:

• confusion of scope and lifetime;
• limited state options;
• management of state through many independent, inconsistent mechanisms;
• dependence on the physical implementation of each mechanism;
• blurring of lines between the presentation layer and business layer;
• lack of immutability.

What can we do about it?  I propose a state manager that attempts to do the following:

• manage state based on various combinations of scope and lifetime, providing options beyond what ASP.NET and related facilities offer;
• expose state capabilities through a single consistent interface;
• hide the underlying physical implementation of each type of state, and allow use of different implementations where appropriate;
• explicitly support the business layer, and maintain a clean separation between the business layer and presentation layer.

As I envision it, this state manager would be a class in the business layer, with a limited set of interfaces for supporting "state providers" that would manage the actual state data.  The class would enforce serializability of state data; although this is not required for the underlying implementations of all state providers, requiring it up front preserves the ability to switch implementations.  All access to state data would occur through the class, probably through static methods.  The specification of supported state options, and the mapping of providers to those options, would be done through configuration. 

The state manager would (ideally) support notions of scope based on the following:

• Application: accessible by all users of an application;
• Role: specific to all users with a certain role;
• User: specific to an individual user (note that this is NOT the same as session);
• Session*: specific to all requests in a given session.  This has lifetime implications because the session is only valid for a certain length of time;
• Business Process*: specific to requests associated with a given logical business process.  This also may have lifetime implications due to the finite length of the business process, but at any rate it's separate from session scope.  Consider multiple sessions associated with the same business process, for example;
• Form*: specific to the processing of a given form, from initial entry (IsPostBack=false) to the user leaving the form.  Equivalent to ViewState;
• Request*: specific to the processing of a given request.

*Has lifetime implications.

The state manager would support similar options with regard to lifetime:

• Application: lasts the duration of the application;
• Session: lasts for the duration of the current session;
• Business Process: lasts for the duration of a logical business process, which may or may not span multiple sessions;
• Form: lasts for the duration of processing of a given form, from initial entry (IsPostBack=false) to the user leaving the form.  Equivalent to ViewState;
• Request: lasts for the duration of processing of a given request.

It might be useful to put the scope vs. lifetime options above on a grid.  Some combinations might be unworkable, but I think a surprising number of them would be useful.

To limit the knowledge the business tier would have of the presentation tier, an inversion-of-control pattern could be used.  Interfaces could expose state capabilities available in the presentation layer, without requiring a direct reference to the System.Web assembly.  The presentation layer would pass a reference to a class (or classes) in that layer that implement the appropriate interfaces, allowing the state manager to use it to take advantage of things like ViewState.

A key goal here is divorce the logical concepts of state from their physical implementations.  The underlying implementations will always impose constraints - for example, storing session state in the database will make it difficult to know when the session ends.  But when enough providers are implemented options will become available.  In time, for example, I might be able to reconfigure the app to store form state (ViewState) on the server rather than sending it to the browser, without having to touch a line of code.

Sure would be nice :-)

I'm going to throw out a somewhat radical, even heretical, idea: that foreign key constraints, as traditionally implemented, might not be worth it.  What?!? you say.  Foreign keys are essential to your data integrity?  They're worth the minor performance impact, right?  They're also worth the hassle of dealing with them?  OK, fine.  But read on.

Foreign key constraints have other less obvious drawbacks.  Consider this: the foreign keys in your database probably make it difficult or impossible for you to prevent deadlocks.  Let's say tables A and B are related by a foreign key (B -> A).  Now let's assume Alice wants to (within a transaction) insert records in both tables, and Bob wants to delete records from both tables.  Because of the foreign keys, Alice must access the tables in this order: A, B; while Bob must access them in this order:  B, A.  Whenever you have multiple clients accessing more than one resource in a different order, you have a recipe for deadlock.  Alice locks data in table A, Bob locks data in table B, and then they wait on each other.  Forever, in principle.

Now a supposedly smart database like SQL Server will detect this deadlock and shut down one of the transactions ("You have been chosen as the deadlock victim.  Buh-bye.").  But that's not very nice, is it?  Now the caller has to handle that scenario and retry.... ugh.   It's also possible to reduce the likelihood of a deadlock by partitioning data, using finer-grained locking, shortening transaction times through various means, and/or specifying loose transaction isolation.

One very good idea is to access your tables in a consistent order - if you can pull it off, this will completely prevent deadlocks, though you will still get some temporary blocking.  This is almost always worth it for a high-volume application.  The presence of foreign key constraints, however, will prevent it from being bulletproof if you must support both adds and deletes, for the reasons mentioned above; adds go top-down, but deletes go bottom-up, which means you're not accessing things in a consistent order any more.

So what to do?  Besides the ideas above, you could consider removing the foreign key constraints from your database, if you're willing to write (and test) the significant application code required to ensure that your data integrity is bulletproof.  A side benefit of this is slightly better performance.  To date I have not found it necessary to go to this extreme.  But here's what I would like (Microsoft, are you listening?): I want constraints that are not checked until my transaction is committed.  This idea is not new; in fact Oracle has had it for quite a while, and Microsofties have been discussing it for a a while.  Here's one blog entry that talks about pros and cons of the idea.

I had an interesting discussion with a former colleague, Clay Lenhart, where he said he's currently disenchanted with test-driven development because of the need to constantly update the unit tests as the application code changes.  This struck me as a valid point; TDD, viewed in that light, would seem to better suit a traditional waterfall development process than an agile one.  Agile processes supposedly embrace the change that will inevitably occur in requirements, design, etc., and try to mitigate its impact by (among other tactics) reducing the effort involved with refactoring.  Writing the test first seems to assume that you can get your requirements right up front - the hallmark of your father's development process, right?

The more I think about this, the more I think there's an appropriate time to write unit tests, and it can be discovered by looking at code churn.  When the churn in a module falls to a certain value, the code is stable enough to test, and only then do you write the tests and automate them.  This allows you to operate at maximum speed, and helps keep you from avoiding refactoring, early on when the code is being ripped apart and put back together wholesale; it also preserves the benefits of unit testing later on in the project. 

Broadly speaking, I would relate this approach to my general approach to decision-making, which is to defer decisions until I feel that either 1) I have to make a decision to proceed, regardless of the information available, or 2) little or no more useful information is forthcoming.  In an even broader sense, the point is that there's a tradeoff between doing things sooner vs. later, and you have to find the point on the timeline that yields the most benefit.  Deep Thoughts By Mike ;-)

Speaking of unit testing, I'm thinking of developing an API and tool that will generate test data based on policies specified by the developer, and in general facilitate unit test development.  Here are some ideas:

Test Data Generation

• Provide scheme for specifying generated data formats
  • Minimum length (character data)
  • Maximum length (character data)
  • Minimum value (numeric data)
  • Maximum value (numeric data)
  • Hardcoded portions
  • Random portions
    • Within limits
  • Data types (string, numeric, strings of only certain characters, etc
  • Values from another table
    • Generate a dependency in the test data generation, so that if we’re generating data for the remote table, we do that first
  • Manual specification of test data by the developer
• Specify test data generation policies for each desired table and column in the database
  • Number of rows desired
  • Whether to violate data generation policy
  • Provide reasonable defaults so not everything needs to be specified
• Allow scheduled, programmatic or ad-hoc test data generation
• Allow generation of test data without stepping on existing (presumably user-entered) data

Unit Test Support API

• Clear table
• Generate test data in accordance with specified policies
• Assert existence of specified rows
• Assert existence of a specified number of specified rows
• Assert non-existence of specified rows
• Delete specified rows
• Delete test data generated as part of this test
• Support notification upon success and/or failure (does NUnit do this already?)
• Support generation of a report with test results (does NUnit do this already?
• Clean up

Unit Test Generation?

• Template-driven
• Mocking (probably skip this in favor of third-party framework or built-in VS support)

Anyway, the first step is to research what's already out there.

I've been having issues with my old wireless router (a Linksys WRT54G v5) - once or twice a month it just locks up and stops passing traffic through.  When that happens I have to either bounce the router or at least go into the admin interface, change a setting and have it refresh.  This has been highly annoying and given that I run this web site on a server at home, disruptive.  Through some web research plus trial and error over the space of a few months, I determined that it had something to do with the wireless connection itself; so for the past few weeks I've had wireless disabled.  My house is wired for ethernet anyway.

But who wants to pay for something and not get it?  I do occasionally want to work upstairs in the dining room, and my wife's PC is plugged in up there, so wireless (assuming it works) is the way to go.  So I bought another router.  This one's a Linksys WRT54GL, Linux-based, with excellent reviews.  I installed it tonight and put a third-party firmware on it (DD-WRT).  The new firmware has tons more options, and so far, the unit seems to be working like a champ.  Problem solved!  I hope.

I've been hesitant to start my recording project because I have a lot going on at work and because my home studio has some serious drawbacks.  The room is an acoustical pigsty, and it's worsened by the fact that my laptop (which I've been trying to use to record) sounds like a Boeing 747.

However, I've virtualized and turned off the server I had running in there, and I'm making progress on work issues, so as soon as I can build a quieter PC I'm looking forward to doing some recording.  This should happen by Christmas, at which point I plan to have one or more demo tracks available on this site.  Then I'll be a REAL musician ;-)

Vendor v = new Vendor();
v.VendorName = "Oracle";

Product p = new Product();
p.Vendor = v;
p.ProductSKU = "ORDB10G";
p.ProductName = "Oracle 10g";
v.Products.Add(p);

p = new Product();
p.Vendor = v;
p.ProductSKU = "ORSVC10";
p.ProductName = "Oracle Services For Java 10";
v.Products.Add(p);

v.Persist();

Here's an example showing how to load all instances of a given entity, using a static entity method:

List<Customer> custs = Customer.LoadAll();

Two more examples showing ways to load entities using a template (query-by-example):

Product pt = new Product();
pt.ProductSKU = "VS2005PRO";
List<Product> list1 = Product.LoadByTemplate(pt);

Vendor vt = new Vendor();
vt.VendorName = "Microsoft";
TemplateLoader<Vendor> ldr = new TemplateLoader<Vendor>(vt);
List<Vendor> list2 = ldr.Load();

Loading one or many entities by key:

Order o = Order.LoadByKey("123");

List<Order> orders = Order.LoadByKey(new string[] { "123", "456" });

Slipjig can set up a data command to call any arbitrary stored procedure.  It also provides a method to set all the parameter values in one shot.  This is more of an ad-hoc convenience thing than anything else; in particular, SetParamValues() is not type-safe and wouldn't be part of everyday usage.  Here's a sample calling a stored proc that takes an instance of every data type:

using (IDbSql db = (IDbSql)DbFactory.GetDB("Test")) {
    IDbCommand cmd = db.GetStoredProcCommand("usp_test");
    db.SetParamValues(cmd, 1, 1, 1, 1, "a", "a", "a", "a", "a", "a", "a", "2006-02-24", "2006-02-24", 0, new byte[] {}, new byte[] {}, new byte[] {}, new byte[] {}, 0m, 0m, 0, 0, 0, Guid.NewGuid(), 0);

    int recs = db.ExecuteNonQuery(cmd);
}

For several years now I've disliked how ASP.NET provides a page lifecycle that is unrelated to the "business", or let's say, "functional" side of the application domain, and is pretty inflexible.  The ASP.NET page lifecycle is strictly in the technical domain, with the possible exception of the Validate() method.  This lifecycle is pretty low-level.
 
However, typical applications have a higher-level set of actions, many of which are performed by the user:  things like "Save", "Save and Add New", "Authorize", "Create From Template", "Cancel", "Delete", "Select", etc.  The natural thing to do is create an API within the application to support these activities.  If you're lucky, you can make this a cross-cutting API that works generically for each feature on all the applicable pages.
 
The problem is that there's an impedance mismatch between the ASP.NET lifecycle API and the (for lack of a better word) "business domain" API.  It can be hard to determine where in the ASP.NET model to put certain code.  What's worse, it may need to be in a different place in one page than in another; for example, before loading some data in Page_Load(), you may need to know whether you will be handling some event from a control, or you may need to handle the event first.
 
I understand that in .NET 2.0 it's actually possible to override the code that provides the ASP.NET page lifecycle, but this seems like a drastic solution and one that leads to a hard-to-maintain application.  I've also read about some efforts towards aspect-orientation that might provide some relief.
 
One solution I thought of is to have some sort of state machine that would allow specification of the lifecycle in a database or other structure, and then fire things off dynamically.  This would be embedded in Page_Load() or some other standard place, and would give developers full control over how the page runs, with the benefit of being able to use a standard, business-oriented lifecycle API.
 
Food for thought :-)

For the past year or two I've been crystallizing some thoughts about frameworks based on my experiences developing and using ASInet 2, Philidor and Morphy (applications and frameworks I developed for my company). Although these frameworks are useful, they rely too heavily on ADO.NET DataSets and DataTables, expose too many low-level tasks to the developer, have a steep learning curve, and generate a lot of code which leads to long startup times and low productivity.

With these thoughts in mind, in my spare time (!) in the last year I've been working on a new framework, tentatively called "Slipjig", that attempts to address these concerns. Slipjig is based on .NET 2.0 and is much more dynamic than my previous efforts. Here are the major design ideas:

• The entities contain their own data, rather than binding to DataRows. LINQ will enable us to query an entity tree as if it were a database.
• Entities have direct references to other entities, providing for parent, child, peer, and many-to-many relationships. The framework can load and save an entire entity tree automatically.
• The entities use attributes to specify object-relational mapping information, rather than code in the entity or in separate but related data access classes. This reduces the project footprint considerably and yields faster startup times.
• A property on the entity controls how the entity will get persisted. Though normally this property is set automatically based on how the entity is used, the developer has complete control over how each entity is persisted.
• The data access layer eschews DataSets completely. Instead, it uses dynamically-generated mapping classes to move data between entities and ADO.NET DataCommands/DataReaders. The mapping classes are generated only on first data access for each entity, yielding even faster startup times. So if you are unit testing a code change in only one module out of 100, the framework only generates and loads the necessary code.
• The data access layer uses the concept of a "loader" - a class that pulls data from a data source and populates a number of entities from it. There are different loaders for different scenarios, and developers can write their own loaders. The data access layer provides implicit transaction support with no need for the developer to do anything.
• The data access layer automatically batches commands for better performance. It also observes a consistent save order for the tables, to prevent deadlocks in the database. The framework uses a table and stored procedure to assign its own keys, rather than use identity columns. This provides a number of advantages, chief among them the ability to assign keys in advance of hitting the database. The framework grabs a batch of keys from the database when it needs to, and doles them out to entities as needed, rather than hit the database for each key. This avoids contention on the key table. Keys are assigned to the entities automatically - the developer does not need to worry about this.

The current status is that I'm working on the loaders, to get them loading the entire entity tree. I'm making changes as necessary to support that. The "save" side of things is working, though I have a few to-dos there too: getting the table save order and command batching fully implemented, and providing more hooks for the developer to control exactly which parts of an entity tree get persisted.

If you're a developer and want to take a look at the framework, give me a shout and I'll get you SVN access :-)

OK OK, so in this day and age that's not that impressive.  In fact, blog ownage is practically a requirement for citizenship, and they should put it on your driver's license ("BLOG OWNER" in red text next to "ORGAN DONOR").  But for some reason I've resisted having a blog of my own, though I've posted the occasional bit to Slashdot or whatever.  I think it's because I generally feel that talk is cheap - everyone has something to say, and as a result there's way too much opinion out there for anybody to wade through in a reasonable amount of time.  Why add to the cacophony?

Well, without answering that question, I finally gave in and added this blog to my web site.