Dutch software makes supercomputer from laptop

Two astronomers from the University of Groningen (The Netherlands) developed a software library that can effortlessly generate visualisations based on hundreds of millions of data points. Maarten Breddels and Jovan Veljanoski initially developed their software to handle the enormous quantity of data from the Gaia mission.

However, the software can also show patterns in other large data files. The software is open source, and free to use. The researchers explain the ins and outs in an article that has been accepted for publication in the journal Astronomy and Astrophysics.

Breddels and Veljanoski call their software vaex, which stands for visualisation and exploration of big tabular datasets. The software can generate visualisations of billions of data points in only one second. This makes it interactive. It behaves similarly to Google Maps. When panning or zooming, one needs to wait shortly for an updated or more detailed map to appear. However, Google Maps run on fast, powerful servers, while vaex works on a laptop.

The power of vaex lies in the combination of several smart techniques. First, it utilises a smart algorithm that maximises all available computing power. Then, it read only the required data from the hard disk and sends it directly to the main memory of the computer. Finally, it is extremely memory efficient, and the working memory does not store unnecessary copies of the data.

Breddels has showcased vaex live at several conferences. As an example, he used a dataset consisting of 1 billion entries related to the Yellow Cab taxis in New York City. He shows which taxi rides are the most lucrative, and where the taxis should wait in any part of the day to maximise their profit. This example shows how vaex can be interesting and beneficial for general applications outside of astronomy.

The software, including the documentation and tutorials, is available for free via here

"Vaex: Big Data exploration in the era of Gaia." Maarten A. Breddels and Jovan Veljanoski, 2018. To appear in Astronomy and Astrophysics.