Search Mage

(From http://weblogs.asp.net/danielfe/archive/2005/02/22/378343.aspx)

Responding to Richard Grimes article on .NET

I recently ran across an article written by Richard Grimes in Dr. Dobbs Journal titled Mr. Grimes farewell. I wanted to respond to some of Richard's assertions and points in the article. You should take Richard's article with a grain of salt as he does clearly state that the article is "his opinion". That being said, Richard's article is supposed to be his view of the current state of .NET, but rather then discuss how far we've come and where we're going, he instead argues three points - the .NET Framework is too large blocking adoption, issues with the design of the .NET Framework, and over half the article bashing Visual Basic, and finally that Microsoft is "losing confidence" in the .NET Framework. His quotes, italicized in gray, and my responses are below.

On the size of the .NET Framework blocking adoption

   * RG:The framework redistributable is 25 MB, which is many times larger than the Java redistributable. One of the lessons of the early versions of Visual Basic was that the shareware and freeware market created the popularity of the language. While there are some shareware apps written in .NET, I often hear people complain about the huge redistributable.

My Response: Maybe I'm being too picky, but the size is 23,698K or 23.7MB. While Java's runtime is smaller, it still weighs in at 15MB. Throughout the article, Richard refers to .NET applications when he's really referring to client or public client (meaning not in the firewall) For example, installing the .NET Framework on a server or within an intranet where you can control the environment certainly isn't an issue. Even for public client machines, there's plenty of commercial shareware from games to RSS readers that require the .NET Framework. I've also talked to plenty of shareware developers and they certainly aren't using Java. Many use C/C++, Visual Basic, or Delphi. As far as adoption "and the state of .NET",  which the article is supposed to be about is best summarized by Soma, the Developer Division Vice President in his post on .NET Momentum.

Soma: We have seen over 70 million downloads of the .NET framework from Windows Update and the Microsoft Download center to date.  For a simple guy like me, that translates to about 5.5 million downloads a month.  Another interesting datapoint is that in 2004, we expect to have about 54 million new PCs shipping with the .NET framework installed/preloaded.  We also have over 2.5M developers targeting managed code.

On the Design of the .NET Framework

   * RG: Almost the first thing I posted on the technical preview newsgroups was a simple console application in Cool, and its equivalent in Java with the rhetoric question to spot the difference.
   * RG: There are classes that are mere wrappers around Win32, but there are other classes that appear to be ported from other frameworks. Before it released .NET, Microsoft had its own Java framework library called WFC, and it also had a managed library as part of the Visual Basic (classic) runtime. It would be nice to know how many classes from WFC and VB were ported to .NET.

My Response: These two points contradict each other. In the first he implies that the .NET Framework is a replica of Java, but in the next statement he claims that the .NET Framework is simply ported Win32 classes, Windows Foundation Classes (WFC), and VB runtime classes. Which one is it? If his point is that you can write a simple application and that it will look the same in C# and Java or C++, I don't think that really proves much. Below is an example of a for loop.

for (int i = 0; i < x; i++) {...}

Guess what language it's written in? If you said either C, C++, C#, and Java, then you are correct. I don't see what he's trying to prove here. If he tries to build a more robust application then "Hello World" your going to run into framework or library specific features (ATL is not, MFC is not EJB, etc).

On Interface-based Programming and Remoting

   * RG: Interfaces are elegant but .NET's preference for class-based solutions has marked the death of interfaces. Look at .NET remoting: This has been provided to allow an object to run in the context where it was created, and be accessed from another context. This means that the object state is kept local, and it is the behavior that is remoted. Thus, remoting is an interface-based facility. You can use .NET remoting with interfaces, but reading through the documentation and all of the “how-tos” on the Web, you wouldn't realize this.

Point #1 - Interfaces are dead

   * My Response: Interfaces are used everywhere in the .NET Framework and are especially valuable given single inheritance in languages like VB and C#. Even the simple string class has IComparable, ICloneable, IConvertible, and IEnumerable interfaces. Going forward, one of the key new features for the .NET Framework 2.0, generics, uses interfaces for constraining data types.

Point #2 - Lack of documentation on using interfaces with .NET Remoting

   *  My Response: I am by no means saying that our documentation is flawless, but here's a link to the .NET Framework SDK Samples on Remoting. Notice the fifth sample down is using Interfaces with remoting. I also outline our guidance on using interfaces for remoting below.
   * RG: .NET can use interfaces but the preferred way is to use classes.
   * RG: Instead, Microsoft prefers people to use a class-based approach, which often results in the bizarre situation of people deploying their server assembly to the client so that the client has the metadata of the service objects available, or a soapsuds assembly, which basically was a hack to get around the problems of having class-based remoting.

My Response: We don't "prefer" any mechanism, per se. While we may offer guidance, developers can choose to develop their applications as they see fit. Our Patterns and Practices group does provide guidance and best practices on these and other points and even includes guidelines on how to design a remote interface. I don't see a class-based favoritism over interfaces, in fact, there has been an increasing movement to use messages and service orientation rather then object orientation. I concede his point on deploying server assemblies to the client, but poor design is poor design. People deploying a server assembly to a client just so that the metadata of the service objects is available could have easily used an interface or schema instead. That being said, there are some situations where you do in fact want to have server code in each client, the example being a peer-to-peer chat where each client acts as both a client and a server.



On Microsoft using the .NET Framework for their applications

   * RG: Microsoft treats .NET as a useful library to extend its products, and to date, it has not shown any more conviction to the framework. There have been a few .NET products written entirely in .NET; one such product is Microsoft CRM....They do not want the expense of rewriting their existing code for .NET, and there is no compulsion to provide all new code in .NET; instead, .NET will be hosted as and when it's needed, particularly to allow extensibility through user-supplied code.  

My Response: We should dissect exactly what Richard says here. He says that Microsoft is using .NET to extend existing products and that Microsoft doesn't want the expense of rewriting applications from scratch in .NET. This makes perfect sense to me, why would we re-write perfectly good code? .NET code can interoperate with existing code, and you bet we're going to take advantage of the interoperability layer to add new features that exploit the best managed code has to offer.  As I pointed out previously, Microsoft is using .NET in all sorts of software from operating systems, to developer tools, to Office.

   * RG: Microsoft's current operating systems, XP and Windows 2003, do not depend on .NET; and with XP, .NET is an optional component.

My Response: This is a half-truth at best. While Windows XP Professional does not use the .NET Framework, that's because the .NET Framework was released after Windows XP Professional shipped. Let's look at the operating systems that shipped after the .NET Framework was released:

   * Windows XP Media Center edition requires the .NET Framework for MCE-specific applications.
   * Windows XP Professional Tablet PC Edition requires the .NET Framework for handwriting recognition which is a managed application.
   * Windows Server 2003 requires the .NET Framework to use ASP.NET, UDDI Services, or Sharepoint Team Services.
   * Windows Small Business Server 2003 requires the .NET Framework for ASP.NET as do SBS specific applications like Remote Web Workplace, and the Backup Snap-in.

On Longhorn and the death of browser applications

   * RG: My opinion is that Avalon, or more specifically, XAML, will mark the death of ASP. The reason is that Avalon is a client-side technology, but the browser is an important part of the distribution model. XAML is so rich that a browser-contained XAML application will look no different to a process-based Avalon application, and coupled with Web Services or Indigo (as the mechanism to access remote code), an XAML application will make an ASP.NET application look paltry and antiquated. Why would Microsoft want to kill ASP? Well, with installation of ASP.NET Microsoft sells a single copy of Windows 2003 and perhaps a handful of copies of Visual Studio.NET. The clients don't have to be Windows, so there is no extra sale to Microsoft (whether as a product or licence). This is hardly a great revenue generator, and worse, ASP.NET actually makes it easy to write the application so that it can be used by browsers other than IE.

My Reponse: I respectfully disagree. XAML will allow for rich interfaces, but ASP.NET and HTML are not going away. Our value is that we can take the best of both worlds, and provide an optimized experience to XAML browsers while still maintaining compatibility with old computers. It should also be noted that there is a difference between client applications and server applications. The server market itself is *totally different* then the client/consumer market. While people talk about Microsoft's dominance in client operating systems at around 90%, we are nowhere near that number in the server market. We're competing against products and companies like IBM WebSphere, hundreds of middleware products, Oracle in the database market, etc. If we want to win the server market, we need to have the fastest, most reliable, most secure, most productive and affordable solution for creating Web applications. To say that browser applications are a threat to Microsoft is so....1996. The threat is not the Web. If it was, wouldn't Microsoft have "lost" already given that the Web is already incredibly successful and popular. How much more popular does the Web have to be before this is proven untrue? Richard then goes on to say that Microsoft needs to do this because of client revenue. The client operating system market, as stated above ~90% is pretty saturated. The server market is where the opportunity for revenue growth truly lies (PS the server and tool business grew 18% last quarter! - see slide 9). On the question of revenue, a typical server deal is in the thousands of dollars as you're paying for several parts including:

   * Operating Systems
   * Transaction Engine
   * Middleware
   * Database

Depending on the complexity of the solution, this can range from thousands of dollars to several million dollars. Server products are expensive. If you look at the market for Web content-management solutions, the average price can be around ,000 for a one-proc enterprise license. My point being that there are plenty of revenue opportunities and competitive threats from the likes of IBM and others in the server market. Do you know how large the market for database software is in terms of revenue? Would you agree that it's billions of dollars? Do you know that Oracle is the #2 software company and their primary revenue is from databases? That's just *one* of the market opportunities for server software. Back to my point - We are totally committed to our server products and to making Windows Server and ASP.NET the best platform for creating Web applications. Period.

On Longhorn

   * RG: I take the decision to make Avalon available to other versions of Windows as a lack of confidence in the sales of Longhorn.
   * RG: However, Microsoft's announcement that Avalon will be available for other versions of Windows indicates to me that they are not so confident on the uptake of Longhorn, and developers will not write applications for Avalon if they are not convinced that there will be the clients that will run it.

My Reponse: The decision to make Avalon available to other versions of Windows was driven by one thing, customer demand. Any rudimentary Web search turns up results of customers complaining about not being able to have this functionality on down-level operating systems.

On using the .NET Framework for shipping products

   * RG: So, with the announcements they have made last year, Microsoft has indicated that Longhorn will not be the great .NET innovation that we were lead to believe it was from PDC 2003. This indicates to me that Microsoft is losing confidence in .NET.  

My Reponse: Richard, the key parts of Longhorn you've mentioned in your article, Avalon and Indigo, are written in managed code. How does that indicate we are losing confidence in .NET when we've decided to bet the success of our next operating system on the .NET Framework? You might then make the follow up argument that because the entire operating system isn't managed, that we have "lost confidence". While that's your opinion, I don't think managed code is right for every scenario and Microsoft has never claimed that it is. Microsoft still fully supports C/C++ and we have a very large existing C/C++ code base and C++ customer constiuency. We'll use managed code where it makes sense.

   * RG: The framework has become Visual Basic—it's intended for users to develop applications, but not for Microsoft to create operating systems or the revenue generating products that they base their profits on.

My Response: I'll avoid responding to the VB bashing as someone is already working on that. There are two points here, one is using the .NET Framework for creating operating systems, the other on Microsoft revenue. On creating operating systems using the .NET Framework, I don't think we've *ever* said that you should be creating an operating system from scratch based solely on managed code. The truth is, gasp, the vast majority of our customers are not creating operating systems.  For those customers who are or who need that level of control and performance, we have C/C++ and we absolutely have not abandoned that. On revenue generating profits, I've already listed several revenue generating applications using the .NET Framework. Microsoft is divided into the seven business groups listed below and let's see which ones are using managed code.

   * Client - Check
   * Information Worker - Check
   * Server & Tools - Check
   * Home and Entertainment - Check
   * MSN - Check
   * MBS - Check
   * Mobile and Embedded Devices - Check

Ready, Fire, Aim

by Cliff Atkinson

The next time you aim to write something, the best place to start might not be paper, but your PowerPoint slides.

Silicon Valley legend Guy Kawasaki was one of the people responsible for the success of the Macintosh computer, and today spends his time helping people start new businesses. I asked him recently about the interrelationship between his live presentations, his slides, and his books; and he said:

"I work in the opposite order of most people. First, I make the slides. Then I give the presentation many times. Then I write the book. I like to perfect the slides and presentation first and elaborate on them to write a book. If your speeches work, then your book will too. The opposite is not true."

At first it might sound odd to present an idea before writing it down, but by flipping conventional wisdom on its head, Guy is helping to re-write the process of writing.

During the days of the dot-com boom, the phrase "Ready, Fire, Aim" meant that you should get out a new idea fast and early in its unfinished state, so you can get valuable market feedback before you fine-tune it to a specific purpose and market. The principle holds true long after the dot-com bust, especially for your presentations.

The key to this approach ties back to your fundamental mindset that gives form to your presentations. Is your PowerPoint static, inflexible, and fixed in stone; or is it fluid, flexible and living? Is your presentation a bucket for you to pour data and information into an audience's head, or is it a river for you to nourish a conversation and engage with your changing environment?

There may be times a static PowerPoint might help you get the results you want, but if you choose a fluid version, consider how entrepreneurs approach their presentations every day. These days, it's common for startup CEOs to "write" their business plans in PowerPoint slides first, before they write out the ideas in longer, narrative detail. Then they hit the road and show the presentation to anyone who will listen -- friends, family, co-workers, and potential customers.

During one of these presentations, the CEO will ask what people think, and someone in the audience might say, "I just don't get it", or "There's no way that's going to work," or, "Your basic premise is flawed." Some less-hardy entrepreneurs might view this negative feedback as reason to pack up the PowerPoint and go home. But the hardier breed will look at every comment as an opportunity to improve the ideas and the message. Every disagreement is an opportunity to engage, reflect and either strengthen or adapt. Only after going through these rigors of reality will the CEOs develop out the ideas in more detail, and then approach investors about it.

This approach can actually reduce the risk that an idea will be rejected, because it's been pre-tested in the marketplace of ideas first, and pre-qualified against 3 principles:

1. Your idea grows stronger as it moves from the abstract to the concrete. An idea in your mind doesn't do anyone else any good until you communicate it to other people. Get your idea out early and find out what the world thinks.

2. Whatever doesn't kill your idea makes it stronger. Ask people what they think, and appreciate people who disagree -- they're the ones you can give most credit toward strengthening your idea. Get feedback, find out what works, discard what doesn't, and keep moving forward.

3. Your excitement for your idea is what gets other people excited. As you share your ideas with other people in person, you have the unique opportunity to demonstrate the passion that will help you overcome the forces that will surely work against your idea. Without your passion, no one else will become passionate.

Whether you're writing a book, a business plan, or any other media document to communicate an idea, try a "ready-fire-aim" approach to make sure your idea stays on target.

Tip: The next time you have a new idea, try field-testing it first in a PowerPoint presentation. If it's a brief idea, present the idea on a single slide, and then ask other people what they think about it. Or if you have a much larger idea or a sequence of thoughts, put a single idea on each slide and develop it out in a storyboard (or try the kit). It may be easier for people like Guy who are professional speakers to make a presentation, but that's all the more reason why you should stretch yourself to do something you're not comfortable doing. If you limit yourself to only a single media expression, you're limiting yourself to only your known universe. Challenge yourself to do what's most difficult for you. Anyone who is a writer should stretch herself to speak and design. Anyone who is a speaker should stretch herself to write and design. Anyone who is a designer should stretch himself to write and speak. When you challenge yourself to do the counter-intuitive, like writing your slides first, you open yourself to new possibilities you may not have seen from a previously limited view.

Introduction to Bayes Nets

An Intuitive Explanation of Bayesian Reasoning

Salton, G. (1997). Expert Systems and Information Retrieval. SIGIR Forum (21:3-4), 1987, pp. 1-5. Reprinted in SIGIR Forum, 1997 (31:1), pp. 39-44.

The purpose of this paper is to vote against the expert systems approach in Information Retrieval, and to suggest the older but much simpler index-based approach. The author began with two reasons why expert systems were introduced in retrieval systems in the early 70s: (1) when there's no expert system, users had to learn a variety of access languages and methods in order to use the system, and (2) these languages and methods were hard to learn. So here came the expert systems, essetially based on a knowledge base and a set of rules to derive results.

However, the expert system approach had two drawbacks: (1) the rules were sometimes too simple in contrast with the complicated relationships between concepts, and (2) the hierarchical representation of knowledge catalogues was too confining to formulate queries. So the authors suggested empahsis to be put on the user information needs and document content, instead of the structured systems approach which was static and sometimes not properly configured.

While we still use cataloguing in Information Retrieval (at least, in libraries; or on the Web we still have directories for Web pages), I would argue that the hierarchical approach will eventually vanish when there is more and more unstructured information which is hard to be tagged. Or instead of a standadized catalogue scheme, we will be able to create our own tags, just like in Flickr and Gmail.

Imagine a day when we can tag every Web page we have visited. Personailzation comes with flexibility.

Lawrence, S. and Giles, L. (1999). Accessibility of Information on the Web. Science, 400:107-109.

Google has announced on Wednesday that its index nearly doubled to be more than 8 billion pages. And on Thursday, the new release of MSN Search (beta) has come with an index over 5 billion pages, which makes it the 2nd largest indexed document collection in the world. Followers are Yahoo, with 3 billion, and Teoma, with less than 2 billion.

So how much information is out there on the Web? How are they roughly organized? And how do Web search engines cope with the evolving Web?

This paper intended to answer these questions, by randomly selecting a sample of Web servers to investigate the amount and distribution of Web information, and also to analyze the data with search-centric metrics. The number of Web servers as of February 1999 was estimated to be 2.8 million, and the number of publicly indexable Web pages to be 800 million. The techniques used in this paper generated a bias as it relied heavily on the assumption that the distribution of Web pages was well-proportioned to different sample sizes.

It's also interesting to note that the paper reported a low usage of meta tags, and a lack of standardization to represent metadata. I would suspect this situation has changed a lot because more and more Web authoring tools, which by default inject metadata to the document, are used in recent years, and more metadata initiatives have come out - besides Dublin Core that was mentioned in the paper, we now have INDECS, EAD, OAI, etc.

Another interesting notes found in this paper is "search engines make extensive use of meaures of 'popularity' to rank the relevance of pages" so that this may generate a bias in ranking scheme because link popularity does not necessarily relate to content relevance. Yet now we would be glad to know that Google in their last November's "Florida Update" deployed Hilltop algorithm to fine-tune the biases generated by pure PageRank algorithm.

As this paper was written five years ago, we can read a more up-to-date analysis in [Ntoulas et al. 2004].

Reference:

Ntoulas, Alexandros, Cho, Junghoo, Olsten, Christopher. (2004). What's New on the Web? The Evolution of the Web from a Search Engine Perspective. WWW 2004. New York, New York, USA.

Tversky, A. and Kahneman, D. (1974). Judgement Under Uncertainty: Heuristics and Biases. Science, 185:1124-1131.

(Daniel Kahneman is the 2002 Nobel laureate in economics.)

Although a bit aged, this paper is among the most engrossing and thought-provoking readings I've done in this semester. Often I find myself intrigued by those interesting examples (or more exactly, IQ questions?) and, after reading through the discussions, I regret to learn that although I normally assume myself to be rather "logical" and "rational", I have still produced a whole lot of intuitive biases.

To briefly sum up what this paper has talked about, we can begin with the observation that we make many of our decisions based on uncertainty. Here I would like to further break down "uncertainty" into two sub-categories: (1) uncertainty due to inadequate information, and (2) uncertainty due to inacurate information.

In such circumstances, we'd employ three heuristic principles in an effort to reduce the complexity of the decision task. These heuristics are: (1) representativeness, which relates to categorization problems and relies on the estimation of similarity, just like pattern classification in AI, (2) availability, which relates to frequency prediction, and (3) adjustment based on an anchor, which relates to numerical prediction when relevant - whether it's informative or not is unknown - data (aka. anchor) is available. Potential cognitive biases that come with each of these decision-making heuristics are discussed in great detail, which I consider to be the best part.

So how does this paper relate to Information Retrieval? Frankly, I don't know. But what I do know is that, by studying this paper and learning these potential biases that even "experienced researchers are prone to make", maybe we can achieve better performances in our own personal information seeking activities.

Know Thyself, Know Thy God.

The video of Tim Berners-Lee's keynote at the MIT Emerging Technologies Conference.

http://mitworld.mit.edu/video/236/

ABOUT THE SPEAKER:

A 1976 graduate of Oxford University, Tim Berners-Lee invented the World Wide Web, an internet-based hypermedia initiative for global information sharing. Today he leads the World Wide Web Consortium, an open forum of companies and organizations with the mission to lead the Web to its full potential. Berners-Lee wrote the first web client (browser-editor) and server in 1990 while working at CERN, the European Particle Physics Laboratory in Geneva, Before coming to CERN, Tim worked with Image Computer Systems, of Ferndown, Dorset, England and before that as a principal engineer with Plessey Telecommunications, in Poole, England. Tim Berners-Lee has received numerous honors, including a MacArthur Fellowship, the Charles Babbage award, the Electronic Freedom Foundation's pioneer award and the Japan Prize from the Science and Technology Foundation of Japan. In 2004 Tim was listed in the New Year’s honors list for a knighthood (KBE) for services to the global development of the Internet and was awarded the first Millennium Technology Prize. He was knighted by H.M. the Queen on 16th July, 2004.

ABOUT THE LECTURE:

It’s hard to keep up with the mastermind of the World Wide Web. In his fast-paced address, Tim Berners-Lee not only recaps the origins of the Internet, but sketches its future. The Web began as “a primeval soup of many things that know each other but haven’t been put together,” says Berners-Lee. In the late 1980s, he began experimenting with a way for fellow physicists to exchange information. According to Berners-Lee, the invention of uniform resource identifiers and hyper text markup language was critical to the Internet’s phenomenal success. “HTML would be the warp and weft of the web, and within it, the jewels would be movies, and pictures and databases and …other formats that would evolve.” Here’s what’s coming: the “semantic web,” a way of indexing and linking together different kinds of web content. He envisions computers becoming even more useful, deploying a common, non-proprietary “resource description framework” that enables them to draw connections between disparate sorts of information. “What’s nifty”, says Berners-Lee “is putting links between objects and even concepts. It allows a query on one database to morph into a query into others.” He imagines the semantic web emerging as a “killer application” in the life sciences, where correlating data from different fields has become increasingly critical.

According to Tom Warren -

Back in July, Microsoft purchased a company called Lookout. The company made a popular tool that allowed users of Outlook 2000 or above to search through their email at greater speed and accuracy to the standard Outlook search tool. Since Microsoft acquired Lookout, the MSN team have been steadily working on Desktop Search and web search technologies.

More to read here - MSN Desktop Search Revealed, with a bunch of screenshots.

Eight tips to think like a genius, a brief summary below:

(1) Look at problems from different perspectives - either find a new solution or a new problem!

(2) Visualize - a picture is worth 1000 words (hmmm... I've been using diagrams a lot in my work, and I quite enjoy drawing them!)

(3) Produce - don't just think, act! And don't be afraid of failures.

(4) Make novel combinations - sometimes old ideas can be twisted in a new way!

(5) Link things together - find the subtle relationship between different dissimilar subjects.

(6) Think in opposites - well, "the opposite" is one of the perspectives; so don't forget it when applying (1).

(7) Think metaphorically - a higher level of (5)? In (5) we look for connections; in (7) we visualize it.

(8) Prepare yourself for chance - if the chance doesn't knock, build a door.

Sometimes I'd find maintaining the address book a nightmare - with more than 200 contacts, some of them have changed their cellphone several times, some have adopted new email addresses, and some have transferred to new companies or new graduate programs...

Then I find Plaxo.

Plaxo is a service to keep one's up-to-date contact information somewhere online, so that this information can be shared among one's friends, families, and colleagues. As long as they have kept their information updated through Plaxo, all I need to do is just to click the "update" button to keep my address book sync-ed. Pretty cool, huh?

Now I've started to push it among my high school classmates.