Archive for September, 2007
Connector/Net 5.2 now supports batching
Just finished checking in initial support for batching in 5.2. Even though MySQL doesn’t support batching in the protocol, the connector sends updates and deletes in multi-statement fashion (read: sent at the same time but separated by a semi-colon). For inserts the connector tries to be a little smarter. If the insert command is simple and uses a simple VALUES(…) block, then it will construct a multi-value INSERT statement of the form INSERT INTO <table> (col, col, ..) VALUES (…), (…), (…). This is significantly faster for the server than sending the inserts in multi-statement mode.
For those not familiar with .NET, we have an object called a DataTable. This object contains columns and rows and is what the user uses to insert, update, and delete data on the client side. When the user wants to update the data on the server, this table (or a series of these tables) is passed to the MySqlDataAdapter.Update method. A data adapter object holds references to at most 4 different commands, one each for select, update, insert, and delete.
For each changed row in a data table, the data adapter will choose the appropriate command and ask that command to execute giving it the values in that data row as parameters. A user can set the data adapter’s UpdateBatchSize to tell the system how many commands should be sent at one time in a “batch”. Setting it to 0 tells it to send as many as the server will allow.
Batching saves tons of time when you have many rows of the same operation in a data table. For one it saves on network round trips. To get an idea of how much faster batching is I ran a simple test. I inserted 100,000 rows into a data table and then setup a data adapter to do the insert to a database running on Ubuntu in a virtual pc I was running (just so the network round trip wasn’t just to localhost).
Without batching: 1 minute 47 seconds
With batching: 4.5 seconds.
Yea.
NCover is now commercial, but friendly to open source
Unless you’ve been coding under a blogosphere rock you’ve heard that NCover has now gone commercial. NCover has become a staple in many developer toolboxes. I can understand this move. People are so busy today that it’s becoming increasingly hard to get people to contribute time to a good open source project. And even fewer contribute money. Unless you are functionally wealthy (read: actually wealthy or a nothing-better-to-do college kid), finding the time to significantly push projects forward can be daunting.
So the guys behind NCover and NCoverExplorer have banded together and formed a new organization named Gnoso. Their first product is NCover 2.0 (now up to 2.0.1) and is available in x32 professional and x64 enterprise editions.
While the prices of these two products are fair, I really appreciate their friendliness to open source. They are offering free pro licenses to developers who fall into one of three different categories: prior donators, educational developers, or open source developers. You can read all about it here.
I’ve got my license. Have you?
Connector/Net working with Visual Studio 2008 beta 2? Oh yeah!
Microsoft says that Visual Studio packages are binary compatible from 2005 so any packages that loaded successfully into VS 2005 should work in VS 2008. Err, no. At least not with VS 2008 beta 2. I admit it didn’t take much of a change to fix things, but it wasn’t seamless.
So, I just committed a patch to our trunk repository that enables Connector/Net in VS 2008 beta 2. Here’s a shot of it running.
This trunk repository will eventually be our 5.2 release and you can grab a sneak peek by pulling a snapshot from here.
MySQL Connector/Net 5.1.3 RC has been released
Hi,
MySQL Connector/Net 5.1.3 a new version of the all-managed .NET driver for MySQL has been released.
Connector/Net 5.1 represents a change in how we package our products. Until now, we’ve shipped our core provider and the Visual Studio integration bits as separate downloads. This has become a bit of a problem. Often we would fix a bug that involved changing code both in the VS package and in the core provider. This provided a versioning problem where users would need to upgrade both products to see the benefit of the bug fix. To solve this, we’ve decided to discontinue Tools for Visual Studio as a separate product and have, instead, integrated it into a new Connector/Net installer. We hope this provides a better “out of box” experience for our users.
All previous versions of Tools for Visual Studio should be uninstalled prior to installing this product.
Version 5.1.3 works with all versions of MySQL including MySQL-4.1, MySQL-5.0, MySQL-5.1 beta or the MySQL-6.0 Falcon “Preview”. Please be aware that this version is beta quality software and should not be used in production environments.
It is now available in source and binary form from [http://dev.mysql.com/downloads/connector/net/5.1.html] and mirror sites (note that not all mirror sites may be up to date at this point of time – if you can’t find this version on some mirror, please try again later or choose another download site.)
For a full list of changes, see the full announcement with change log here.
NUnit testing with dynamic configuration files
I use NUnit for all my unit tests in Connector/Net. Over the past
few years I've added a few features to my test suite library to make
testing easier. One of the better ones I implemented was test
subclasses. I had been looking for a way to run my test fixtures
multiple times against the same host/database but with different
configuration options. I didn't want to restart the test process
because I wanted to collect coverage stats without having to resort to
merging multiple output files. Test subclasses handled that nicely.
My
most recent test related task was writing unit tests for the web
providers we include in Connector/Net 5.1. I had already written some
tests but they really needed to be run manually and they included an
app.config file that would have to be changed with every release. Here
is the app config file I am talking about.
1: <configuration>
2: <connectionStrings>
3: <remove name="LocalMySqlServer"/>
4: <add name="LocalMySqlServer" connectionString="server=localhost;uid=root;database=test;pooling=false"/>
5: </connectionStrings>
6: <system.web>
7: <membership defaultProvider="MySQLMembershipProvider">
8: <providers>
9: <remove name="MySQLMembershipProvider"/>
10: <add name="MySQLMembershipProvider"
11: type="MySql.Web.Security.MySQLMembershipProvider, MySql.Web, Version=5.1.3, Culture=neutral, PublicKeyToken=c5687fc88969c44d"
12: connectionStringName="LocalMySqlServer" enablePasswordRetrieval="false"
13: enablePasswordReset="true" requiresQuestionAndAnswer="true"
14: applicationName="/" requiresUniqueEmail="false" passwordFormat="Hashed"
15: maxInvalidPasswordAttempts="5" minRequiredPasswordLength="7"
16: minRequiredNonalphanumericCharacters="1" passwordAttemptWindow="10"
17: passwordStrengthRegularExpression=""/>
18: </providers>
19: </membership>
20: </system.web>
21: </configuration>
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
]]>
As you can see, this config file directly references not only the
testing host and database in the LocalMySqlServer connection string but
it directly references version 5.1.3 of MySql.Web (the assembly that
houses our providers). I like this to be very automatic so this will
never do.
I already have config files that I use for testing against MySQL
4.1, 5.0, and 5.1. All that is included in these files are the port
number, named pipe name, and shared memory name (and I could remove
these last two and generate them dynamically now that I think about
it). I wanted to reuse these config files for my web testing so I
started writing code to read these config values in. However I then
realized that I had already done this with the base test class in my
core assembly test suite. So rather than rewrite that, I added a
virtual method named LoadStaticConfiguration (so web tests could extend
it) and made the base test publicly accessible.
Now my web test library includes a class named BaseWebTest that
extends BaseTest. At this point, all my web testing fixtures inherit
the config reading and dynamic database setup that is in the base
testing library. However, there was still a problem. My server
testing config files didn't include any connection string and
membership provider registration (and I didn't want to add it).
Let me explain why these parts are important. The web providers are
assemblies that integrate into an existing provider framework provided
by .NET. There are static methods that a user can call such as
'Membership.CreateUser(…)' that will use the information found in the
config file to know what providers to instantiate and what connection
strings to use. I needed to be able to test this type of behavior so
the config files are important. Still, I wasn't about to hand update
my config files on every product release.
So I decided what I needed was to dynamically update the config
system when the test starts up. There wasn't a lot of information in
the blogosphere on how to do this so that's why I'm writing this. Here
you see my implementation of LoadStaticConfiguration.
1: protected override void LoadStaticConfiguration()
2: {
3: base.LoadStaticConfiguration();
4:
5: ConnectionStringSettings css = new ConnectionStringSettings();
6: css.ConnectionString = String.Format(
7: "server={0};uid={1};password={2};database={3};pooling=false",
8: BaseTest.host, BaseTest.user, BaseTest.password, BaseTest.database0);
9: css.Name = "LocalMySqlServer";
10: Configuration config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
11: config.ConnectionStrings.ConnectionStrings.Add(css);
12:
13: MembershipSection ms = (MembershipSection)config.SectionGroups["system.web"].Sections["membership"];
14: ms.DefaultProvider = "MySQLMembershipProvider";
15: ProviderSettings ps = new ProviderSettings();
16: ps.Name = "MySQLMembershipProvider";
17: Assembly a = Assembly.GetAssembly(typeof(MySQLMembershipProvider));
18: ps.Type = "MySql.Web.Security.MySQLMembershipProvider, " + a.FullName;
19: ps.Parameters.Add("connectionStringName", "LocalMySqlServer");
20: ps.Parameters.Add("enablePasswordRetrieval", "false");
21: ps.Parameters.Add("enablePasswordReset", "true");
22: ps.Parameters.Add("requiresQuestionAndAnswer", "true");
23: ps.Parameters.Add("applicationName", "/");
24: ps.Parameters.Add("requiresUniqueEmail", "false");
25: ps.Parameters.Add("passwordFormat", "Hashed");
26: ps.Parameters.Add("maxInvalidPasswordAttempts", "5");
27: ps.Parameters.Add("minRequiredPasswordLength", "7");
28: ps.Parameters.Add("minRequiredNonalphanumericCharacters", "1");
29: ps.Parameters.Add("passwordAttemptWindow", "10");
30: ps.Parameters.Add("passwordStrengthRegularExpression", "");
31: ms.Providers.Add(ps);
32:
33: config.Save();
34: ConfigurationManager.RefreshSection("connectionStrings");
35: ConfigurationManager.RefreshSection("system.web/membership");
36: }
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
]]>
Lines 5-11 add our connection string to the connection strings
section. Note how we are using String.Format along with the base class
defined statics host, user, password, and database0. This one was
easy.
Lines 13-31 add our membership provider to the system.web section.
The only tricky part is that you have to index into the SectionGroups
array with the name "system.web" and then index into that groups
Sections array with the name "membership" and then cast the return
value to a MembershipSection. Once you have this you can just new up a
ProviderSettings class, fill in all the values, and add it to the
providers array on the membership section object. One area to note is
lines 17 and 18. Remember we want this to be completely dynamic so we
use Assembly.GetAssembly to get a reference to the assembly that
includes our provider (not ExecutingAssembly since we are currently
executing inside the testing assembly), grab it's full name, prefix on
our provider class name and use that for our provider type.
Line 33 tells the configuration system to save itself. At this
point you might think things should work but they won't. You've
changed the values on disk but not what's cached in memory. Lines 34
& 35 take care of that. They simply instruct the configuration
system that the next time values from the "connectionStrings" or
"system.web/membership" sections are accessed they should be reread
from disk.
That's it! Now my web testing framework will work even without a
config file. All you have to do is point it at a running mysql
instance.
MySQL Conference and Expo 2008 — Start planning now!
Yes it’s September but it’s not too early to start planning on attending the biggest and best conference dedicated to providing you with the very latest and greatest information about the database you either are or soon will be using.
No matter if your second language is SQL, C#, Java, PHP, Ruby, or Boo (!) we’ll have something for you. Or rather you’ll have something for all of us! That’s right our conference is great mainly because of great presentations by our users and the 2008 call for proposals is now open!
If you have an interesting story, lesson learned, or are simply an expert in some aspect of MySQL, we want to hear from you. Hurry, the deadline is Oct 30. By the way, I’m especially interested in hearing about possible .NET related presentations. I would love to see talks on .NET 3.0, Silverlight integration, Entity SQL integration, .NET related tools that are built around Connector/Net, and anything else CLR-based!
So? What are you still reading for? Go submit a proposal today and hopefully I’ll see you there!
iPods on Linux
Miguel seems to be quite perturbed that Apple would change existing iPods or new iPods in such a way as to make them not function with Linux. As I understand it, they [Apple] are adding some type of hash generation to new and existing iPods (through a firmware upgrade) and to the latest build of the iTunes software. Client software has to generate a matching hash before it can access the music library on the device.
Miguel calls this anti-competitive. He’s right. He then calls it an unfair business practice. He’s wrong. First, I seriously doubt this move by Apple has anything to do with Linux. Apple simply cannot be worried about such a minority. Second, let’s agree that this has nothing to do with the iTunes license agreement since a user could easily purchase an iPod and plan to use it without ever installing iTunes or agreeing with their license.
Apple is free to make their iPods and their software as they see fit. You may not like it and it may not be smart business on the part of Apple, but that’s not the point. Your only option is to either legally look for a workaround or use a different product. It now seems that some clever hackers have discovered the hash algorithm and, once again, unlocked the iPods for use on non-iTunes clients (remember we stopped believing it’s just about Linux).
Don’t get me wrong. I fully support use of the iPod on any client software you wish and hope someone out there is able to crack the hash every time it’s changed. I just needed to be the voice of reason here and remind everyone that this is not a legal matter or governmental matter. There are lots of options out there for music players so this is a truly free market. If the leader does something you don’t like, just use a different product.
Maybe I should call my senator because my XBox media center software doesn’t work with Vista (I don’t remember agreeing to a license that forbid that)?
Compiled faster than scripted? Huh. Imagine that!
Just read this article about running PHP applications on Phalanger and found it greatly interesting that PHP apps are often quite a bit faster running on Phalanger than on the Zend engine. In fact the article states that phpBB serves up pages twice as fast from Phalanger than from Zend.
Also interesting is the existence of a managed MySQL extension for Phalanger. This appears to be based on libmysql however.
Phalanger works great on .NET but also works on Mono. This makes it very easy to use C# (or any other CLR language) to write some layers of your app, like data access code, while allowing you to write the UI in PHP. With Phalanger you use the full range of managed components like Connector/Net, LinQ, generics, custom attributes, events, and others.