A side note: If you wish to work with VLDB, consider MapReduce. MapReduce cannot solve EVERY problem, but it is an interesting framework nonetheless.

Data Replication: When dealing with a large MySQL database, it is important that we do not query/analyze a live database i.e., database in which live inserts/updates are in process. MySQL locks the entire table when doing an insert/update or select. Thus, querying a large live database will affect the query performance as well as the insert performance. In this scenario, it will be better to replicate the database as follows:

Note: paths are specific to Ubuntu 8.10 and MySQL 5.0.67

mysqldump /path/to/database > <databasename>.out

datadir variable in /etc/mysql/my.cnf contains the value of path to database. The default path is /var/lib/mysql.

Under mysql, create a new database. Then do source <databasename>.out. This will load all the tables and data in the new database.

Indexing: Indexing is important to retrieve rows quickly. When a particular column is indexed, queries can retrieve rows directly, instead of scanning each row and comparing the value. However, figuring out proper indexes is a tricky business. To aide developers, MySQL provides an explain statement. When we give any SELECT statement to explain, we get query optimizer’s best guess about the number of rows that need to be scanned and the indexes that will be used. Explain statement is described in more detail here.

Proper Database Engine: MySQL provides a multitude of storage engines that are appropriate for different scenarions. The default MyISAM engine is appropriate for large number of SELECTs and small number of INSERTs/UPDATEs. This is the most appropriate model for web usage, which is the main reason behind MySQL’s popularity. However, note that in our replicated database, there are absolutely no INSERTs. In addition, MyISAM uses table level locking to preserve data integrity i.e., for each insert/update and select, MyISAM locks the entire table. Table level locks can seriously affect the performance of parallel queries. In our case, we can work without any locks at all, because we are only reading data, which cannot affect the integrity of data. The best we can do with MySQL is to use InnoDB engine instead of MyISAM. InnoDB uses row-level locking instead of table-level locking, which can boost the performance of parallel queries. Few performance numbers for MyISAM and InnDB are provided here.

A MyISAM table can be converted into an InnDB table as follows:

ALTER TABLE <table name> ENGINE=InnoDB;

Note: Engine conversion takes long time. It took almost 1 whole day to convert a 33 million row MyISAM table to InnoDB table.

Python and MySQL – A match made in heaven: Python’s MySQL API makes it really easy to query the database. However, when issuing multiple independent queries it is importat to use a new a connection and a new cursor. Also, make sure to close the cursor once the query is completed.

Limits are bad bad bad: The Limit clause in a select statement allows us to retrieve a certain part of result set. For example, limit 1000 10 will return 10 rows starting from the 1000th row. In my opinion, limit clause is only useful for visual isnpection of results. However, if we issue limit queries during analysis, then it only slows down the process. For example, if the final results contain 33 million rows and we use a limit clause for 1000 rows, then instead of issuing 1 qeury, we will issue 33,000 queries. Very very bad!!!

If you know that the query will return large number of results which would overwhelm your local memory, then use SSCursor instead of default cursor. SSCursor stores the query results on the server, instead of the client. When fetchone or fetchmany calls are issued, the local client fetches the rows from the server instead of the local memory.

Design guidelines are not bible: Database normalization always leads to beautiful database design. However, it also leads to multiple joins. If you are joining multiple tables, which contain millions of rows, for majority of queries, then it might be helpful to combine those tables into one. While it is a bad design choice, it will seriously improve query performance.

If you have other suggestions for improving query performance over a large database, please post them in comments section.

This discussion got me thinking about the role that left-brained and right-brained folks play in this world. It is said that the majority of the people are left-brained. That makes sense because lot of people like to have order and logic in their life. But, is order and logic all there is that matters? Without creative right-brained people, where would this world be?

If we look around, example of Steve Jobs comes to mind as a right-brained individual. One sign of right-brained folks is that they arrive at conclusions without a reason. Steve Jobs has clearly showed signs of this property over time. For example, over the years business people have tried to convince him that Apple must not bundle software and hardware together. In-fact Apple should not be in the business of selling hardware at all. But, that does not matter to Steve. It just MAKES SENSE to him that hardware and software should come from the same vendor. That insistence has led to recent flawless experience for Apple customers. Moreover, his love for art and design has given us beautiful Apple computers and the amazing iPhone.

Steve’s madness, insistence and obsessiveness is not without chaos, which is yet another property of right-brained people. Apple’s Newton, Cube, Lisa and Apple III were all huge commercial disasters. Even Jobs himself was fired from the very company he co-founded.

Now, if we look for a prominent and comparable example of a left-brained person, we find Bill Gates. Jobs has famously claimed that Microsoft has contributed enormously to the computing industry. But, they do not have a taste. That is exactly right. Bill has always followed a logical and analytical process. When Microsoft was starting out, Bill realized early on that IBM was in the business of making hardware and that it will be very hard to compete against IBM. So he collaborated with them and provided software. He never thought along the lines of Jobs’ thinking that homogeneity in software and hardware was important. His goal was to become big businessman in the computing industry and he had a process in place for that. He never went with his gut instincts or did anything that MADE SENSE.

If we compare Gates’ and Jobs’ impact on the computing industry, then, for better or for worse, Gates comes out on top. Gates has shaped the computing industry, positively as well as negatively, for nearly 2 decades. But if we dig a little deeper we find that Gates & co. seldom did it themselves. First Windows version was almost an exact copy of still-in-progress Macintosh. Microsoft had no idea about an Internet browser until Netscape Navigator came along. Then, Microsoft pushed Netscape out of business by bundling Internet Explorer with its then popular Windows. On countless occasions, Microsoft has bought or bullied innovative and creative organizations. In other words, willingly or unwillingly, the left-brained, logical and analytical Gates has always collaborated (a sugar-coated term) with right-brained, innovative and creative folks. Thus, the bottom line is, right-brained and left-brained people complement each other. The weaknesses of one are overcome by the strengths of other. Thus, these two groups should collaborate in real-life to achieve greater good.

I tend to keep my interests relatively broad. That mentality probably comes from my fear of voluntary commitment. As a result, I like anything that is related to networks (except theory – I hate theory).

I am rather inclined to choose hot topics in networks. This is primarily because I love the recognition. This approach has the added advantage of finding the research problems easily.

I have following areas in mind so far:

1. Botnet defense
2. Scalability and Security in Online Social Networks (Facebook, Twitter e.t.c.)
3. Efficient algorithms for decentralized tracker based CDNs
4. Data Center Network design.
5. Cloud Computing.

I am most interested in botnet defense at the command and control (C&C) level. The related work in this area seems to be focused at defending probable victims (e.g. Google, Amazon e.t.c), using network shields. I believe the assumption is that so far C&C mechanisms has been pretty straightforward. Thus, as soon as a botnet is found to be lethal and attacking network shields, it can be disrupted by finding central points of failure. Failure poins exist in current botnets because botmasters assume that in one way or another bots and botmaster must know each other. However, recently, I have conceived a generic a botnet in which bots and botmaster do not need to know each other. I believe, defense mechanisms against my proposal are important. However, I am struggling with this question: Is it worth pursuing defenses against an attack that has not been employed yet. However, I do believe it is important to conceive new attacks and stay one step ahead of attackers.

Social networks are really beautiful applications. However, I believe that the design of these networks is bound to fail. The volume of interactions seen by these networks must be creeping towards becoming unsustainable.  In addition, the security and privacy mechanisms employed by these applications are at really primitive levels. These are important and interesting challenges that must be dealt with. However, I do not have any concrete ideas yet.

I have recently monitored the Azureus DHT using my work on Montra. Monitoring Azureus DHT essentially gives us access to millions of tracker logs (32 million total). This gives us the opportunity to experiment with different tracker algorithms. However, I havent been able to find a compelling research problem as to why one would want to invent new tracker algorithms.

Finally, I am broadly interested in data center networks and cloud computing. However, I haven’t played with these areas that much. I have read a few papers and attended a few presentations.

I would also encourage people to look at Accountable Internet Protocol (AIP)

I think its about time we give network security the importance it deserves. Deploying a new architecture at a global scale, however, is going to be a challenge.

Give Synergy a try. Its lot of fun. Instructions to setup a synergy server on Ubuntu can be found here

http://www.stsc.hill.af.mil/CrossTalk/2008/01/0801DewarSchonberg.html

This is an excellent article about how damaging it is to use Java as a first programming language for CS students. I wish this article was around when I was in college because Java was my first programming language.

Recommendation: Buy Linksys WRT54GL and upgrade it with Tomato

However, I recently learnt from a very good friend that Ph.D. is a lonely road. It is lonely because one has to do it for him/herself. Advisors and other students can provide guidance but at the end of the day, you have to pull yourself ahead. There will be times when no one pays attention to your work and you absolutely hate your own work. But you have to keep yourself motivated that there will be better days ahead. I think that if someone had given me this advise a year ago then it would have been really helpful. I hope incoming Ph.D. students read this and learn from it.