Tuesday, September 25, 2007

It's 9/25. Do you have your Halo 3?

I knew the Halo 3 launch date had arrived last night when I look over and see my 1 1/2 year old putting on the headset and grabbing the controller.  I had to take a a few quick snap.  (never mind his dirty shirt -- he had just finished eating without a bib!)

2007-09-25 001 035

Monday, September 24, 2007

Connector/Net 5.2 now supports batching

Just finished checking in initial support for batching in 5.2.  Even though MySQL doesn't support batching in the protocol, the connector sends updates and deletes in multi-statement fashion (read: sent at the same time but separated by a semi-colon).  For inserts the connector tries to be a little smarter.  If the insert command is simple and uses a simple VALUES(...) block, then it will construct a multi-value INSERT statement of the form INSERT INTO <table> (col, col, ..) VALUES (...), (...), (...).  This is significantly faster for the server than sending the inserts in multi-statement mode.

For those not familiar with .NET, we have an object called a DataTable.  This object contains columns and rows and is what the user uses to insert, update, and delete data on the client side.  When the user wants to update the data on the server, this table (or a series of these tables) is passed to the MySqlDataAdapter.Update method.  A data adapter object holds references to at most 4 different commands, one each for select, update, insert, and delete.

For each changed row in a data table, the data adapter will choose the appropriate command and ask that command to execute giving it the values in that data row as parameters.  A user can set the data adapter's UpdateBatchSize to tell the system how many commands should be sent at one time in a "batch".  Setting it to 0 tells it to send as many as the server will allow.

Batching saves tons of time when you have many rows of the same operation in a data table.  For one it saves on network round trips.  To get an idea of how much faster batching is I ran a simple test.  I inserted 100,000 rows into a data table and then setup a data adapter to do the insert to a database running on Ubuntu in a virtual pc I was running (just so the network round trip wasn't just to localhost).

Without batching:  1 minute 47 seconds

With batching:  4.5 seconds.

Yea.

Friday, September 21, 2007

NCover is now commercial, but friendly to open source

Unless you've been coding under a blogosphere rock  you've heard that NCover has now gone commercial.  NCover has become a staple in many developer toolboxes.  I can understand this move.  People are so busy today that it's becoming increasingly hard to get people to contribute time to a good open source project.  And even fewer contribute money.  Unless you are functionally wealthy (read: actually wealthy or a nothing-better-to-do college kid), finding the time to significantly push projects forward can be daunting.

So the guys behind NCover and NCoverExplorer have banded together and formed a new organization named Gnoso.  Their first product is NCover 2.0 (now up to 2.0.1) and is available in x32 professional and x64 enterprise editions.

While the prices of these two products are fair, I really appreciate their friendliness to open source.  They are offering free pro licenses to developers who fall into one of three different categories:  prior donators, educational developers, or open source developers.  You can read all about it here.

I've got my license.  Have you?

Connector/Net working with Visual Studio 2008 beta 2? Oh yeah!

Microsoft says that Visual Studio packages are binary compatible from 2005 so any packages that loaded successfully into VS 2005 should work in VS 2008.  Err, no.  At least not with VS 2008 beta 2.  I admit it didn't take much of a change to fix things, but it wasn't seamless.

So, I just committed a patch to our trunk repository that enables Connector/Net in VS 2008 beta 2.  Here's a shot of it running.

image

This trunk repository will eventually be our 5.2 release and you can grab a sneak peek by pulling a snapshot from here

Thursday, September 20, 2007

NUnit testing with dynamic configuration files

My use NUnit for all my unit tests in Connector/Net. Over the past few years I've added a few features to my test suite library to make testing easier. One of the better ones I implemented was test subclasses. I had been looking for a way to run my test fixtures multiple times against the same host/database but with different
configuration options. I didn't want to restart the test process because I wanted to collect coverage stats without having to resort to merging multiple output files. Test subclasses handled that nicely.
My most recent test related task was writing unit tests for the web providers we include in Connector/Net 5.1. I had already written some tests but they really needed to be run manually and they included an app.config file that would have to be changed with every release. Here is the app config file I am talking about.

<configuration>
   <connectionstrings>
     <remove name="LocalMySqlServer">
     <add connectionstring="server=localhost;uid=root;database=test;pooling=false" name="LocalMySqlServer">
   </connectionstrings>
   <system.web>
     <membership defaultprovider="MySQLMembershipProvider">
       <providers>
         <remove name="MySQLMembershipProvider">
         <add applicationname="/" connectionstringname="LocalMySqlServer" enablepasswordreset="true" enablepasswordretrieval="false" maxinvalidpasswordattempts="5" minrequirednonalphanumericcharacters="1" minrequiredpasswordlength="7" name="MySQLMembershipProvider" passwordattemptwindow="10" passwordformat="Hashed" passwordstrengthregularexpression="" requiresquestionandanswer="true" requiresuniqueemail="false" type="MySql.Web.Security.MySQLMembershipProvider, MySql.Web, Version=5.1.3, Culture=neutral, PublicKeyToken=c5687fc88969c44d"/>
       </providers>
     </membership>
   </system.web>
 </configuration>

As you can see, this config file directly references not only the testing host and database in the LocalMySqlServer connection string but it directly references version 5.1.3 of MySql.Web (the assembly that houses our providers). I like this to be very automatic so this will never do.

I already have config files that I use for testing against MySQL 4.1, 5.0, and 5.1. All that is included in these files are the port number, named pipe name, and shared memory name (and I could remove these last two and generate them dynamically now that I think about it). I wanted to reuse these config files for my web testing so I started writing code to read these config values in. However I then realized that I had already done this with the base test class in my core assembly test suite. So rather than rewrite that, I added a virtual method named LoadStaticConfiguration (so web tests could extend it) and made the base test publicly acessible.

Now my web test library includes a class named BaseWebTest that extends BaseTest. At this point, all my web testing fixtures inherit the config reading and dynamic database setup that is in the base testing library. However, there was still a problem. My server testing config files didn't include any connection string and membership provider registration (and I didn't want to add it).
Let me explain why these parts are important. The web providers are assemblies that integrate into an existing provider framework provided by .NET. There are static methods that a user can call such as 'Membership.CreateUser(…)' that will use the information found in the config file to know what providers to instantiate and what connection strings to use. I needed to be able to test this type of behavior so the config files are important. Still, I wasn't about to hand update my config files on every product release.

So I decided what I needed was to dynamically update the config system when the test starts up. There wasn't a lot of information in the blogosphere on how to do this so that's why I'm writing this. Here you see my implementation of LoadStaticConfiguration.

protected override void LoadStaticConfiguration()
    {
        base.LoadStaticConfiguration();
       
        ConnectionStringSettings css = new ConnectionStringSettings();
        css.ConnectionString = String.Format(
            "server={0};uid={1};password={2};database={3};pooling=false",
            BaseTest.host, BaseTest.user, BaseTest.password, BaseTest.database0);
        css.Name = "LocalMySqlServer";
        Configuration config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
        config.ConnectionStrings.ConnectionStrings.Add(css);
      
        MembershipSection ms = (MembershipSection)config.SectionGroups["system.web"].Sections["membership"];
        ms.DefaultProvider = "MySQLMembershipProvider";
        ProviderSettings ps = new ProviderSettings();
        ps.Name = "MySQLMembershipProvider";
        Assembly a = Assembly.GetAssembly(typeof(MySQLMembershipProvider));
        ps.Type = "MySql.Web.Security.MySQLMembershipProvider, " + a.FullName;
        ps.Parameters.Add("connectionStringName", "LocalMySqlServer");
        ps.Parameters.Add("enablePasswordRetrieval", "false");
        ps.Parameters.Add("enablePasswordReset", "true");
        ps.Parameters.Add("requiresQuestionAndAnswer", "true");
        ps.Parameters.Add("applicationName", "/");
        ps.Parameters.Add("requiresUniqueEmail", "false");
        ps.Parameters.Add("passwordFormat", "Hashed");
        ps.Parameters.Add("maxInvalidPasswordAttempts", "5");
        ps.Parameters.Add("minRequiredPasswordLength", "7");
        ps.Parameters.Add("minRequiredNonalphanumericCharacters", "1");
        ps.Parameters.Add("passwordAttemptWindow", "10");
        ps.Parameters.Add("passwordStrengthRegularExpression", "");
        ms.Providers.Add(ps);
      
        config.Save();
        ConfigurationManager.RefreshSection("connectionStrings");
        ConfigurationManager.RefreshSection("system.web/membership");
    }

Lines 5-11 add our connection string to the connection strings section. Note how we are using String.Format along with the base class defined statics host, user, password, and database0. This one was easy.

Lines 13-31 add our membership provider to the system.web section. The only tricky part is that you have to index into the SectionGroups array with the name "system.web" and then index into that groups Sections array with the name "membership" and then cast the return value to a MembershipSection. Once you have this you can just new up a ProviderSettings class, fill in all the values, and add it to the providers array on the membership section object. One area to note is lines 17 and 18. Remember we want this to be completely dynamic so we use Assembly.GetAssembly to get a reference to the assembly that includes our provider (not ExecutingAssembly since we are currently executing inside the testing assembly), grab it's full name, prefix on our provider class name and use that for our provider type.

Line 33 tells the configuration system to save itself. At this point you might think things should work but they won't. You've changed the values on disk but not what's cached in memory. Lines 34 & 35 take care of that. They simply instruct the configuration system that the next time values from the "connectionStrings" or "system.web/membership" sections are accessed they should be reread from disk.

That's it! Now my web testing framework will work even without a config file. All you have to do is point it at a running mysql instance.

MySQL Connector/Net 5.1.3 RC has been released

Hi,
MySQL Connector/Net 5.1.3 a new version of the all-managed .NET driver for MySQL has been released.

Connector/Net 5.1 represents a change in how we package our products. Until now, we've shipped our core provider and the Visual Studio integration bits as separate downloads. This has become a bit of a problem. Often we would fix a bug that involved changing code both in the VS package and in the core provider. This provided a versioning problem where users would need to upgrade both products to see the benefit of the bug fix. To solve this, we've decided to discontinue Tools for Visual Studio as a separate product and have, instead, integrated it into a new Connector/Net installer. We hope this provides a better "out of box" experience for our users.
All previous versions of Tools for Visual Studio should be uninstalled prior to installing this product.

Version 5.1.3 works with all versions of MySQL including MySQL-4.1,  MySQL-5.0, MySQL-5.1 beta or the MySQL-6.0 Falcon "Preview".  Please be aware that this version is beta quality software and should not be used in production environments.

It is now available in source and binary form from [http://dev.mysql.com/downloads/connector/net/5.1.html] and mirror sites (note that not all mirror sites may be up to date at this point of time - if you can't find this version on some mirror, please try again later or choose another download site.)

For a full list of changes, see the full announcement with change log here.

MySQL Conference and Expo 2008 -- Start planning now!

Yes it's September but it's not too early to start planning on attending the biggest and best conference dedicated to providing you with the very latest and greatest information about the database you either are or soon will be using.

No matter if your second language is SQL, C#, Java, PHP, Ruby, or Boo (!) we'll have something for you.  Or rather you'll have something for all of us!  That's right our conference is great mainly because of great presentations by our users and the 2008 call for proposals is now open!

If you have an interesting story, lesson learned, or are simply an expert in some aspect of MySQL, we want to hear from you.  Hurry, the deadline is Oct 30.  By the way, I'm especially interested in hearing about possible .NET related presentations.  I would love to see talks on .NET 3.0, Silverlight integration, Entity SQL integration, .NET related tools that are built around Connector/Net, and anything else CLR-based! 

So?  What are you still reading for?  Go submit a proposal today and hopefully I'll see you there!

Monday, September 17, 2007

iPods on Linux

Miguel seems to be quite perturbed that Apple would change existing iPods or new iPods in such a way as to make them not function with Linux.  As I understand it, they [Apple] are adding some type of hash generation to new and existing iPods (through a firmware upgrade) and to the latest build of the iTunes software.  Client software has to generate a matching hash before it can access the music library on the device.

Miguel calls this anti-competitive.  He's right.  He then calls it an unfair business practice.  He's wrong.  First, I seriously doubt this move by Apple has anything to do with Linux.  Apple simply cannot be worried about such a minority.  Second, let's agree that this has nothing to do with the iTunes license agreement since a user could easily purchase an iPod and plan to use it without ever installing iTunes or agreeing with their license.

Apple is free to make their iPods and their software as they see fit.  You may not like it and it may not be smart business on the part of Apple, but that's not the point.  Your only option is to either legally look for a workaround or use a different product.  It now seems that some clever hackers have discovered the hash algorithm and, once again, unlocked the iPods for use on non-iTunes clients (remember we stopped believing it's just about Linux).

Don't get me wrong.  I fully support use of the iPod on any client software you wish and hope someone out there is able to crack the hash every time it's changed.  I just needed to be the voice of reason here and remind everyone that this is not a legal matter or governmental matter.  There are lots of options out there for music players so this is a truly free market.  If the leader does something you don't like, just use a different product.

Maybe I should call my senator because my XBox media center software doesn't work with Vista (I don't remember agreeing to a license that forbid that)?

Thursday, September 13, 2007

Compiled faster than scripted? Huh. Imagine that!

Just read this article about running PHP applications on Phalanger and found it greatly interesting that PHP apps are often quite a bit faster running on Phalanger than on the Zend engine.  In fact the article states that phpBB serves up pages twice as fast from Phalanger than from Zend.

Also interesting is the existence of a managed MySQL extension for Phalanger.  This appears to be based on libmysql however.

Phalanger works great on .NET but also works on Mono.  This makes it very easy to use C# (or any other CLR language) to write some layers of your app, like data access code, while allowing you to write the UI in PHP.  With Phalanger you use the full range of managed components like Connector/Net, LinQ, generics, custom attributes, events, and others.