Posted By: Rydal Williams | posted @ Saturday, March 06, 2010 8:40 PM |
Before I start cramming code, I’ll like to know what you guys think. I’ve been working heavily on my key value store and it has been working as expected, however, the main reason for me building this store does not seem to be completely eliminated with the functionalities that I have in place so far. therefore, I’ll like to add an extra piece to the jigsaw but before I do so I’ll like to hear what you have to say.
The storage engine itself is solid and it works with the minimal functionalities that it currently has. As a king of simplicity, I intentionally built it with everything in its simplest form and no worries, I’ll beef them up as need for whatever changes arise. Ok, that being said for you to use this database, you’ll need to use a client and at this point in time, I have implemented two clients; namely memory and network and I’ll like to add a third which is what this post is all about.
Note: Regardless of which client you are using persistent storage is always available.
Memory Client
Allows you to run the store completely in memory and disk if persistent mode is enabled. This could be used more or less like a caching mechanism or an embedded database.
Network Client
With this client you can connect to the store using an IP address and port number across a network or local, you have to be running the network server client somewhere. This client is what you will use to connect to where the network server is running.
Web Application Client (proposed)
Since the reason for building this database was because of a long grown frustration which is (speed and simplicity), I wanted a database that will allow me to put out a web application very fast and efficiently while being reliable and fast. After implementing the above 2 clients, a thought came to me – what if I also add a third client which allows the developer to build a web application right on top of my KVS? By doing so, developers will be able to build applications using (XHTML, JSON & JavaScript) completely client side. it doesn’t get any better than that. Here are some of the requirements:
- should allow for ASP.NET pages including ASMX pages
- should allow because XHTML
- should allow JavaScript
- should contain a canned admin tool
- should allow use of existing websites
- should implement some sort of template engine
- should allow the developer the ability to indicate which framework should be included such as JQuery
- should include its own KVS JavaScript framework, that will allow the developer to communicate with the database easily
- should be very light weight
- should be RESTful
So what do you think, does the Web Application Client raise your eyebrows? Do you see yourself opening up the KVS UI application, click on”Add New Web Application”, config a few things, open up your favorite editor, code a whole web app in XHTML and JavaScript and using JSON as your data type – everything is fast and client side while also supporting your existing ASP.NET pages and code behind code?
Posted By: Rydal Williams | posted @ Saturday, February 13, 2010 6:30 PM |
As I stated in my previous posts here and here, I have set out to write my own NoSQL database simply because I can – well not really but because it will give me the edge, learning experience will be great and there isn’t really too many options when it comes to native .NET NoSQL Databases. I want to be able to build web applications without the RDBMS overhead and schema design time.
I haven’t done much yet with distributed or replicating side of things yet, so far all I wanted was a prototype to show that it works. I don’t have a name yet, therefore, I’m allowed to call it random names. I want to build it in such a way that it will support 4 modes listed below.
- In Memory only - sort of a caching server
- In Memory with Persistent – a caching server that can persist its state and recover
- Network – this mode will allow you to setup the server itself on a different machine and access it over the network
- Network Replicated – data is replicated across other nodes which gives you the ability to load balance, it will support the “Eventually Consistent Model”.
I’ve proven to myself that 1 & 2 is completely feasible as I have a working prototype that works with amazing speeds, however, I’m still working on 3 & 4 as a simple communication protocol is not that easy to develop. What I have so far is way too complex, I get lost in my own code – so I’m re-writing while trying hard so that the perfectionist side of me does eat up the best of my time.
My NoSQL database currently supports the following functions – in progress.
/// <summary>
/// Version Alpha 0.1.3
/// </summary>
interface IClient
{
/// <summary>
/// Truncates the database and remove all old versions
/// from the journal file system structure, if your database has
/// grown outrageously, this is what you want to do.
/// </summary>
void Truncate();
/// <summary>
/// Shuts down the database, why would you want to do that?
/// </summary>
void ShutDown();
/// <summary>
/// Stores an object into the storage unit
/// </summary>
/// <param name="key">the key that uniquely identifies the object</param>
/// <param name="obj">the object to store</param>
void Put(string key, object obj);
/// <summary>
/// stores an object in the database and return the object ID (OID)
/// that was generated for you, if you were so lazy not to provide a key
/// </summary>
/// <param name="value">the value to store</param>
/// <returns>the key (OID)</returns>
string Put(object value);
/// <summary>
/// Gets a stored object by name
/// </summary>
T Get<T>(string key);
/// <summary>
/// removes a record from the database
/// </summary>
/// <param name="Key">the key of the record to remove</param>
/// <returns>true or false indicating the record was removed of not</returns>
void Delete(string Key);
/// <summary>
/// returns a list of all keys
/// </summary>
/// <returns>ilist of keys</returns>
IList<string> Keys();
/// <summary>
/// returns a list of keys that match the specified pattern
/// </summary>
/// <param name="pattern">the pattern to search for</param>
/// <returns>ilist of keys that match</returns>
IList<string> Keys(string pattern);
/// <summary>
/// indicates whether or not the specified key exist in this database
/// </summary>
/// <param name="key">the key to search for</param>
/// <returns>boolean indicate if the key was found or not</returns>
bool Exists(string key);
/// <summary>
/// saves this database to disk
/// will only work if you are using in-memory mode and you decide
/// to save it to disk for some unknown reason
/// </summary>
/// <param name="async">indicates a synchronous or asynchronous write</param>
void Save(bool async);
/// <summary>
/// puts an entry into a list that is part of the database, meaning
/// the value itself is of type IList
/// Works for IList or ArrayList, etc.
/// </summary>
/// <param name="key">the database key</param>
/// <param name="listKey">the key to the object with the IList object</param>
/// <param name="listValue">this object itself is of type IList</param>
void ListPut(string key, string listKey, object value);
/// <summary>
/// puts an entry into the database that its value is a Hashtable
/// </summary>
/// <param name="key">the database entry key</param>
/// <param name="hashKey">the hashtable entry key</param>
/// <param name="value">the hashtable value</param>
void HashPut(string key, string hashKey, object value);
/// <summary>
/// increment the value of this key by 1, therefore, the
/// value of this key as to be a number
/// </summary>
/// <remarks>this could be very useful for tracking ids
/// that are some sort of a unique key such as a primary key in an RDBMS</remarks>
/// <param name="key">the key to increment</param>
/// <returns>returns the new incremented value</returns>
int Increment(string key);
/// <summary>
/// decreases the value of this key by 1, therefore,
/// the value of this key as to be a number
/// </summary>
/// <param name="key">the key which value to decrement</param>
/// <returns>returns the new decremented value</returns>
int Decrement(string key);
/// <summary>
/// renames the key of and entry in this database to a new key
/// </summary>
void Rename(string oldKey, string newKey);
/// <summary>
/// gets multiple keys and return there values
/// </summary>
/// <param name="keys">the list of keys to search for</param>
/// <returns>an IDictionary with a list of keys and objects that were found</returns>
IDictionary<string, object> MultiGet(IList<string> keys);
/// <summary>
/// inserts multiple values into the database
/// </summary>
/// <param name="keys">the list of keys</param>
/// <param name="values">the list of values</param>
void MultiPut(IList<string> keys, IList<object> values);
/// <summary>
/// will traverse through an IDictionary and put all
/// of its entries into the database
/// </summary>
/// <param name="entries">the KeyValuePair entries that need to be inserted</param>
void MultiPut(IDictionary<string, object> entries);
/// <summary>
/// will traverse through a set of key/value pairs and
/// insert each entry into the database
/// </summary>
/// <param name="entries">the key/value pairs that need to be inserted</param>
void MultiPut(KeyValuePair<string, object> entries);
}
Posted By: Rydal Williams | posted @ Saturday, February 06, 2010 12:53 AM |
In Part 1 of this series, I mentioned that I really don’t see any need for RDBMS’s and I’ve yet to find a developer that can prove me wrong, however, there is one thing that I’ve yet to solve – that is reporting, although I don’t necessarily think its a problem. There is always the comment that you can’t report on databases that are not relational, how true is that? Very true! oh wait – not really!
Read David’s article – about how it all started.
However, SQL Server did not start at the latest version with all the bells and whistles, it started really small with probably nothing and it was built upon as demands are made. Therefore, I’m leaning towards popularity, I believe that if the concept of OODBMS or KVS (object oriented database management systems or key value stores) get popular enough, tools will be built that will report on data as you need, it probably won’t be easy but as any other technology evolution – it will evolve!
Reporting is not a show stopper for me and shouldn’t be for anymore, what are reports and what are they used for anyway; It’s more likely just a different way of looking at your existing data – there is no stopping you from writing a program that can create similar reports from a NOSQL database. I know you are probably thinking that I must be nuts because there won’t be any relationships, or inbuilt calculations, joins, functions, aggregations and etc. but think about it for a second, does it really need to be that way? Come 'on, lets say you want to get total orders per day in your so call report for the big guy at the 114th floor, shouldn’t you just have an entry that looks similar to [date, totalorders] in your NOSQL solution and simply increment totalorders when you receive a new order or increment the outstanding balance once a withdrawal is made, the whole idea of calculating totalorders after the fact is archaic, calculating it in real time is as or if not more efficient, accurate and up to date.
Lets take another real time example, COMCAST, yes I said COMCAST, I rented my house out and got a town home in the city (Atlanta) so that I can live the city life, the move went well and everything was needy and greedy but however, few weeks later my business internet stopped working, after calling Comcast and jumping hoops and feeding their hamsters, it came to my attention that my account wasn’t synced, my payments have been going to my home address – ha, problem #1! Secondly, even after the correction, it got disconnected twice again after that, to cut this story short, it was all because the system that feeds their accounting system receives data based on a reporting model and my changes weren’t their, they claim it takes up to a week to apply a balance, a week? Holy smokes – it felt like 1970! Now, if they were using a NOSQL solution and my account was balanced in real time, all other systems and departments will see the data in real time – there will be no need to wait for a scheduled generated report.
I’ve also been working on a secret analytics tool with a bang but guess what, building the system to collect the data was easy, now-storing and reporting on the data is the most painful process. As a matter of fact 70% of my time so far has been spent on storing, aggregating and reporting on the data – hence the reason why Google analytics has a 24hr lag, well and other reasons. In short, using an RDBMS has only made my application complex and over a year to build, with a NoSQL solution – it would have been so much easier.
NOSQL(http://en.wikipedia.org/wiki/NoSQL) – is the key to getting up and going in a matter of minutes.
Don’t get me wrong, I’m not really saying jump up and ditch your relational databases, I’m simply saying pick the right tool for the job. I build several web applications for my clients throughout the year and I can pretty much count with one hand of how many of them ask for some sort of report that I couldn’t imagine how to generate with an NoSQL solution. Therefore, NoSQL may not be the way to go for the next SalesForce but it sure could be the way to go for your new FaceBook competitor. The right tool for the job – heck, you can even start with a NoSQL solution and migrate to a relational database if you need to later, key point here is you will get up and running quickly and don’t have to worry about schemas and relationships.
You probably going - Ok shut up, get the point, so what are you going to do about it? Well, this is what I want to do, I want to build a NoSQL data store and provide it to the .NET community – it might not be all super powerful but it will be functional and that is as much as I can do to get the word out. I’ll work on two types of NoSQL solution for .NET a KVS and OODBMS – starting with the KVS first. Richard has a list here - http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores of key-value stores but as you can see, your .NET options are limited. I’m going to start with a KVS because I don’t want to get distracted with the underlying granularity of serializing custom objects and indexing – although I do have a pretty good idea of how I’ll write it but let’s start small – a prototype!
So how do I plan on going about building this KVS store and use it; it’s pretty simple - quickly, I’ll build a light weight prototype version of it and start using it in the real world. I’ll probably start with a simple website and this blog and my other sites and the rest of the world. Oh and it will probably need some sort of distributed capability. But in essence, this is my vision in pseudo code:
using(Database db = new Database("dotnetclr.db")){
db.Put("FirstName", "Rydal");
db.Close();
}
See how simple that is! Expect a whole lot more post on my progress in building my version of a key-value store in C#. RDBMS have done their time, we should move on!
Posted By: Rydal Williams | posted @ Saturday, February 06, 2010 12:35 AM |
Yep, the title is right, I’ve been working for years and years building a toolset that will help me develop applications very fast without the setup delay. I want to be able to focus on the application itself rather than the entities around it and one of the most irritating bottlenecks is the database. I mean don’t get me wrong, I’ve worked out a way of speeding up that arena a bit such as using active record and other ORM tools. At this point after designing my database I can simply run my tool and it will create Active Record classes for that database and all the needy-greedy things that I need. However, time has to be allocated for architecting the database and implementing it and blah blah blah.
So, I may have eliminated the time it takes to use ADO and etc to get and put data into my database but the time of creating one is still there. Lets look at a very simple development life cycle, you start off with a project and you perform the discovery and research needed and you then head off to development which for the web, entails the following (design, code and data). Lets say you are building an ecommerce site, the database has to have numerous tables such as products, categories, etc and their relationships and store procedures, views, etc.. gracious – it doesn’t get any easier and the more complex the application gets the more complex your database gets.
Little bit off topic here but I’ll make my active record tool available to the public, as soon as I can find the time to publish it and write up a getting started guide of some nature.
Ok, I know you are probably wiggling your head and saying huh? If I don’t have a database, how should I store my data – I’m not saying no database, I say NOSQL, I just don’t believe in RDBMS’s anymore, why do we need them. I’ve been thinking about this for a while and asking myself, do I really need a RDBMS, I do need a place to persist my data but does it have to be an RDBMS, why are all the developers hooked on some sort of RDBMS. So, I set out to discuss with my team and some developer friends and I’ve yet to find a valid reason for an RDBMS, granted I have a few that stated ease of reporting – I’ll get to that point soon.
I’m done, tired of RDBMS and I demand a faster, smarter and easier solution, I want something that just works and scales as needed. We see new technologies everyday, new versions of everything - CSS, HTML, .NET, etc…everything is changing but there is at least one area that has remain constant for a long time, wait – two things. Commercial flight and Data Persistent. Well flight is a topic for another day, so let’s focus on data persistent; The way and means that we persist data hasn’t changed, the mindset of most developers haven’t change on how to store and retrieve their data and I am demanding that we change!!!
There are layers and layers or tools such as ORM that you can use to add just another layer on top of your database but at the end its only simulating simplicity and not really making it simple. You DON’T need an RDBMS, there are other ways of persistent your data such as object databases, key values stores, etc. The only people that tend to use light weight solutions like these days are the Google’s (http://labs.google.com/papers/bigtable.html), FaceBook’s(http://www.facebook.com/note.php?note_id=24413138919) and Amazon’s(http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html) – why don’t we all do the same? Why do we have to wait until we need to handle 5,000,000 page views before we start spinning the wheel. There are also object database such as db4o, object store or PERST – why not use them?
One of the most relevant points that I’m about to make is probably the key to the problem – developers and corporations are not aware that there are other means. Think outside the box people! The enterprises these days will not bulge if a certain system wasn’t built or used by Microsoft, why? Microsoft and the big dogs have our brains crushed on RDBMS but I don’t get free investment shares for using them!
I believe developers should start thinking databases only when they are needed, they should be implemented easily and quickly and in a very light weight fashion. I know so many smart people that have awesome ideas that they will like to build but they spend so much time in data structure that by the time they get it right, the idea is old and its 12 years later. Pun intended!
I’m I going crazy or do I have people that are thinking the same thing?
Posted By: Rydal Williams | posted @ Friday, November 27, 2009 1:22 PM |
I’ve seen lots of questions on how you can add a “class” attribute to any html tag such as the “a” tag while using the HtmlHelper class in ASP.NET MVC. The problem here is “class” is a reserved keyword in .NET so you just can’t use “class” in the htmlAttributes object of the HtmlHelper class. i.e. this will work perfectly.
<%=Html.ActionLink("Log in","Login", null, new { id = "idforthislink" } %>
but this won’t
<%=Html.ActionLink("Log in","Login", null, new { id = "idforthislink", class ="red" } %>
It will error out on class. Here is the proper way of doing it.
<%=Html.ActionLink("Log in","Login", null, new { id = "idforthislink", @class="red" } %>
have fun!
Posted By: Rydal Williams | posted @ Tuesday, September 29, 2009 3:44 PM |
Sometimes you need to return a single column that contains values that are comma separated or pipe separated or what ever floats your boat from a multiple row result set, a perfect example is returning the category id’s for a product that could be in one of more categories but you don’t want to return multiple rows with duplicate information.
There are two ways of doing this or two that I know of – meaning simple enough.
1. The COALESCE method, which is probably the most common. This method works by tricking the coalesce function into doing something else, COALESCE is very similar to ISNULL except it can take in more than 2 arguments.
select @results = COALESCE(cast(ProductCategoryId as varchar)+',','')
from ProductCategory where ProductId = 24
2. Using the FOR XML PATH expression which works very well and its probably my preferred method because it takes order into consideration. If you use the FOR XML PATH without specifying the column name, you get concatenation – cool trick huh?
select cast(ProductCategoryId as varchar) + ',' from Productcategory
where ProductId = 24 for xml path('')
Have fun!!!
Posted By: Rydal Williams | posted @ Tuesday, August 18, 2009 10:02 PM |
In the old days, we have to manually create custom configuration sections if we want to do cool stuff with app/web.config or simply just put the values in the appSettings section. However, there are a few config options that are out there that most developers don’t know about, a perfect example is the mailSettings config section which allows you to setup your mail settings in your configuration file and simply call the SMTPClient to send an email and by doing so, changing settings can be done on the fly and oh’ if you are worried about security, don’t forget you can definitely encrypt your configuration file if you need to.
<system.net>
<mailSettings>
<smtp deliveryMethod="Network" from="name@domain.com">
<network host="smtp.mail.com"
userName="name@domain.com"
password="blog.dotnetclr.com" port="25"/>
</smtp>
</mailSettings>
</system.net>
And here is the sample C# code that will use the above configuration settings
public void SendBy(string to, string subject, string body)
{
var mailMessage = new System.Net.Mail.MailMessage();
mailMessage.To.Add(to);
mailMessage.Subject = subject;
mailMessage.Body = body;
var smtpClient = new SmtpClient();
smtpClient.EnableSsl = true;
smtpClient.Send(mailMessage);
}
Posted By: Rydal Williams | posted @ Friday, August 14, 2009 11:58 AM |
If you are are using Microsoft SQL Server 2008 Management Studio, you would have noticed that it does have intellisense built in and very addictive if I must add. Anyway, problem here is SQL Server caches schema’s, therefore, when you make a schema change such as adding a new column to a table, intellisense doesn’t pick it up immediate. To quickly refresh the cache strike (CTRL+Shift+R) and that should do it. Viola!
Posted By: Rydal Williams | posted @ Tuesday, July 28, 2009 10:13 AM |
Here is a very simple way to resize an image using a bitmap as a canvas and the graphics object as a the resizing tool. Don’t forget to reference System.Drawing.
/// <summary>
/// Resizes an image from a source file to a destination file
/// the destination file will contain the exact image at the size specified
/// </summary>
/// <param name="sourceImagePath">the source image file that needs to be resized</param>
/// <param name="destinationImagePath">the destination path that the source
/// image file needs to be resized to</param>
/// <param name="width">the new width</param>
/// <param name="height">the new height</param>
static void ResizeImage(string sourceImagePath,
string destinationImagePath, int width, int height)
{
// Create an image object of the original image
System.Drawing.Image
originalImage = System.Drawing.Image.FromFile(sourceImagePath, true);
// Create a new bitmap that we will use as a canvas to drawn on the new resized imaged
Bitmap newBitmap = new Bitmap(originalImage, new System.Drawing.Size(width, height));
// Using the graphics object draw the orginal image on the canvas that was created above
using (Graphics g = Graphics.FromImage(newBitmap))
{
g.InterpolationMode =
System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic;
g.DrawImage(originalImage, 0, 0, width, height);
}
// Save the new resize image to a new path
((Image)newBitmap).Save(destinationImagePath);
}
Posted By: Rydal Williams | posted @ Friday, July 24, 2009 1:40 PM |
One of the most efficient and non-intrusive ways you can implement an intercepting procedure is via an HttpModule. An HttpModule simply receives every request going to or coming from IIS and it allows you to manipulate the content as you see fit and pass it along. I wanted to spit out the time it takes from receiving a request to writing the response on one or more web pages and there was no other better way of doing it except writing an HttpModule, it toke me all but 10mins. Here is the code, feel free to blow it up.
class PageLoadTime: System.Web.IHttpModule
{
#region IHttpModule Members
public void Dispose()
{
// nothing to dispose off
}
public void Init(System.Web.HttpApplication context)
{
context.BeginRequest += new EventHandler(context_BeginRequest);
context.EndRequest += new EventHandler(context_EndRequest);
}
bool IsReady(System.Web.HttpContext ctx)
{
return (ctx.IsDebuggingEnabled
&& ctx.Request.IsLocal
&& ctx.Response.ContentType.Equals("text/html"));
}
void context_BeginRequest(object sender, EventArgs e)
{
if (IsReady(HttpContext.Current))
{
Stopwatch sw = new Stopwatch();
HttpContext.Current.Items.Add("PageLoadTimeStopWatch", sw);
sw.Start();
}
}
void context_EndRequest(object sender, EventArgs e)
{
if (IsReady(HttpContext.Current))
{
// stop the watch
Stopwatch sw = (Stopwatch)HttpContext.Current.Items["PageLoadTimeStopWatch"];
sw.Stop();
// render the time elapsed
string elapsedTime = string.Format(
"<b>Page Loaded In <font color=red>{0}</font> secs</b>",
decimal.Round(decimal.Parse(sw.Elapsed.TotalSeconds.ToString()), 2));
HttpContext.Current.Response.Write(string.Format("{0}", elapsedTime));
}
}
#endregion
}
Next entries »