Downloading all files from a Amazon S3 bucket

I was trying to download all files from an Amazon S3 bucket, and did not feel like clicking through all of the files. Here is the little Python(2) script I came up with: import urllib2   print("Retrieving file list ...") url = urllib2.urlopen('https://s3.amazonaws.com/tripdata?max-keys=9999999') data = url.read() url.close()   print("Parsing file list ...") import xml.etree.ElementTree e Continue reading →

Backing up (MySQL Database) via LVM Snapshot

I have recently come across a nice solution for backing up ("large" MySQL databases without having to handle a slave explicitly for dumps). That is, if a short outage is fine (stopping the slave for a few seconds). We maintain a hot swap slave (read-only) which will jump in for the master if it fails. Continue reading →

Cloudera, Spark and MySQL

I am using a Cloudera Cluster (CDH-5.4.2-1.cdh5.4.2.p0.2) to run Spark (1.3.0). I wanted to access data from an MySQL database: val photos = sqlContext.load( "jdbc", Map( "driver" -> "com.mysql.jdbc.Driver", "url" -> "jdbc:mysql://testserver:3306/test?user=tester&password=testing", "dbtable" -> "photo")) photos.countval photos = sqlContext.load( "jdbc", Map( "driver" -> "com.mysql.jdbc.Driver", "url" -> "jdbc:mysql://testserver:3306/test?user=tester&password=testing", "dbtable" -> "photo")) photos.count Unfortunately, this does not Continue reading →

Linux: Splitting files in two

Here are two scripts splitting the lines of a file into two files based on a given ratio. #!/bin/bash   # This script writeis the first part of the lines from the given input file into one output file and the rest of the lines into another output file. # The frist output file (with Continue reading →

Spring MVC: properties in the Application Context VS in the Servlet Context

I was deploying a web app based on Spring MVC (3.2.6.RELEASE). In this web app I was trying to use properties in the application context as well as in the servlet context. Now, I determined experimentally (I would have to check the code to be absolutely sure) that in the application context when using missing Continue reading →

MySQL: Install locally (not as root) from binaries

Under Ubuntu, I have tried to setup MySQL from binaries for a local user (not root) with another MySQL instance already running. This works, but is documented rather vaguely (I did not find anything that documented the whole process). So I will sum up the solution I have come up with here: download a MySQL Continue reading →

MySQL 5.5 vs 5.6: ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

I was building an index which was too long using MySQL 5.5. MySQL returned a WARNING 1071 (42000): Specified key was too long; max key length is 767 bytesWARNING 1071 (42000): Specified key was too long; max key length is 767 bytes The index got silently truncated. And, at least on the surface, everything worked. Continue reading →

Tomcat, web.xml and "[xX][mM][lL]" is not allowed

Problem So, I had the following issue when trying to run my web application from Eclipse deploying to a local Tomcat. The processing instruction target matching "[xX][mM][lL]" is not allowed.The processing instruction target matching "[xX][mM][lL]" is not allowed. Solution It turned out that I had an empty line at the beginning of my web.xml!

On first glance this seems to be a nice tutorial for circular replication in MySQL: http://dba.stackexchange.com/questions/17529/setting-circular-replication-in-mysql

MyBatis, MySQL and Zero Dates

I was using MyBatis together with MySQL, now I kept getting something like 13:42:32.681 [main] DEBUG o.s.jdbc.datasource.DataSourceUtils - Returning JDBC Connection to DataSource Exception in thread "main" org.springframework.dao.TransientDataAccessResourceException: ### Error querying database. Cause: java.sql.SQLException: Value ' 6181070861^L39078830@N04^S0000-00-00 00:00:00 13.371369 52.515972^@^@' can not be represented as java.sql.Timestamp ### The error may exist in de/uniwue/dmir/flickr/db/mapper/FlickrMapper.java (best guess) Continue reading →