Just a girl in Tech

Hive is a distributed data warehouse that runs on top of Apache Hadoop and enables analyses on huge amount of data.  It provides its own query language HiveQL (similar to SQL) for querying data on a Hadoop cluster. It can manage data in HDFS and run jobs in MapReduce without translating the queries into Java. The mechanism is explained below: " When MapReduce jobs are required, Hive doesn’t generate Java MapReduce programs. Instead, it uses built-in, generic Mapper and Reducer modules that are driven by an XML file representing the “job plan.” In other words, these generic modules function like mini language interpreters and the “language” to drive the computation is encoded in XML. " This text was extracted from Programming Hive . Hive was initially developed by Facebook, with the intention to facilitate running MapReduce jobs on a Hadoop cluster, since sometimes writing Java programs can be challenging for non-Java developers (and for some Java develope...

Recently I've been working with Hive and had some troubles working with Ant . For this reason, I bought the b ook " Ant in Action " and I'll share some thing I've learned with it, and some other experiences I had working with Ant. Ant is a tool written in Java to build Java . By build I mean compile, run, package it and even test it. It is designed for helping software teams develop big projects, automating tasks of compiling code . To run Ant in you project you should have a build.xml file that describes how the projects should be built. There should be one build per project, except if the project is too big. Then you might have subprojects with " sub build " files. These sub builds are coordinated by the main build file. The goals of the build are listed as targets . You can have, for example, a target init that creates initial necessary directories. You can also have a target compile that depends on the previous ini...

Today another curious issue happened regarding MySQL.  I had a scenario where the rows in a table grew greatly, so that the previous PRIMARY KEY type, a SMALLINT (up to 32767 different ids), could not bear the new amount of data. So I had to modify the PRIMARYKEY type. When trying to alter the field, the following error appeared: ERROR 1025 (HY000): Error on rename of '.\temp\#sql-248_4' to '.\temp\country' (errno: 150)input Checking this error number in a shell: $ perror 150 MySQL error code 150: Foreign key constraint is incorrectly formed Basically this error was a consequence of trying to modify a field that was a FOREIGN KEY, with the command ALTER TABLE mytable MODIFY COLUMN id INT NOT NULL AUTO_INCREMENT; The problem is that in MySQL there is no way of updating a field's type "on cascade". So to update a field that is a FOREIGN KEY one should drop the FOREIGN KEY Relation, change the field type, and then bring the relation b...

Recently, I came across with a problem where I had to copy and insert some MySQL data from one database to another. The issue regarded the same table structure in both databases, but with different data content in each. It was vital not to lose any information when importing the dump from one table to the other. One of the most important details was that it was possible to have the same tuple repeated in both tables and I could not afford overwriting any content. So, here follows the command used: mysqldump -u user -p  --no-create-info --insert-ignore mydatabase mytable > file.dump The --no-create-info makes sure no DROP or CREATE info are added, so your table is not deleted and recreated and you don't lose any information. With the --insert-ignore parameter, the data is inserted with the INSERT IGNORE method. Using so, when trying to insert a tuple with an already existent  primary key it is simply ignored. The duplicated key tuple is discarded an...

Hello, I have just posted an implementation of the SlopeOne Recommender System in Java. I designed it to work with Movielens Dataset (a bunch of ratings on movies). I hope that helps everybody that is starting to develop their own recommender system. https://github.com/renataghisloti/SlopeOne-with-Movielens-Dataset

Seeing that Recommender Systems has drawn a lot of attention in this past year, I would like to recommend further reading to those who want to obtain greater knowledge in the subject. I will indicate some articles that have helped me study the matter: G. Adomavicius and A. Tuzhilin Towards the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. 2001 This article written by Adomavicius introduces Recommender Systems very well. It explains the main three types of these systems (Content-Based, Collaborative Filtering and Hybrid Recommendation). I also gives a formal mathematical definition of a Recommender Systems, which for some people can be great. I greatly recommend any other article you may find of Adomavicius. Laurent Candillier , Frank Meyer , Kris Jack, Françoise Fessant.A State-of-the-Art Recommender Systems. This paper also provides great overview of Recommender Systems and a very interesting comparison between...

The Apache Hadoop is a framework for distributed computing applications, inspired by Google's MapReduce  and GFS paper. It is an Open Source software that enables the processing of  massive amounts of data with commodity. First introduced by Doug Cutting, who named the project after his son's toy (a yellow elephant), Hadoop it is now one of the greatest Apache projects. It involves many contributors and users around the world such as Yahoo!, IBM, Facebook and many others.   The framework presents a master/worker shared nothing architecture . The Hadoop cluster is composed of a group of single nodes (computers), being one of these nodes the master server and the other nodes the workers. On the master node, the Namenode deamon and the JobTracker daemon usually run. The Namenode deamon keeps files metadata, and the JobTracker one manages the mapreduce tasks executed on the cluster. The management and monitoring of tasks are made by the Hadoop serv...