Just a girl in Tech

Hive is a distributed data warehouse that runs on top of Apache Hadoop and enables analyses on huge amount of data.  It provides its own query language HiveQL (similar to SQL) for querying data on a Hadoop cluster. It can manage data in HDFS and run jobs in MapReduce without translating the queries into Java. The mechanism is explained below: " When MapReduce jobs are required, Hive doesn’t generate Java MapReduce programs. Instead, it uses built-in, generic Mapper and Reducer modules that are driven by an XML file representing the “job plan.” In other words, these generic modules function like mini language interpreters and the “language” to drive the computation is encoded in XML. " This text was extracted from Programming Hive . Hive was initially developed by Facebook, with the intention to facilitate running MapReduce jobs on a Hadoop cluster, since sometimes writing Java programs can be challenging for non-Java developers (and for some Java develope...

Today another curious issue happened regarding MySQL.  I had a scenario where the rows in a table grew greatly, so that the previous PRIMARY KEY type, a SMALLINT (up to 32767 different ids), could not bear the new amount of data. So I had to modify the PRIMARYKEY type. When trying to alter the field, the following error appeared: ERROR 1025 (HY000): Error on rename of '.\temp\#sql-248_4' to '.\temp\country' (errno: 150)input Checking this error number in a shell: $ perror 150 MySQL error code 150: Foreign key constraint is incorrectly formed Basically this error was a consequence of trying to modify a field that was a FOREIGN KEY, with the command ALTER TABLE mytable MODIFY COLUMN id INT NOT NULL AUTO_INCREMENT; The problem is that in MySQL there is no way of updating a field's type "on cascade". So to update a field that is a FOREIGN KEY one should drop the FOREIGN KEY Relation, change the field type, and then bring the relation b...

Recently, I came across with a problem where I had to copy and insert some MySQL data from one database to another. The issue regarded the same table structure in both databases, but with different data content in each. It was vital not to lose any information when importing the dump from one table to the other. One of the most important details was that it was possible to have the same tuple repeated in both tables and I could not afford overwriting any content. So, here follows the command used: mysqldump -u user -p  --no-create-info --insert-ignore mydatabase mytable > file.dump The --no-create-info makes sure no DROP or CREATE info are added, so your table is not deleted and recreated and you don't lose any information. With the --insert-ignore parameter, the data is inserted with the INSERT IGNORE method. Using so, when trying to insert a tuple with an already existent  primary key it is simply ignored. The duplicated key tuple is discarded an...