HOW TO: Get your UDFs working with Python libraries after Exasol 6 upgrade

We recently took the plunge and upgraded to Exasol V6. There’s a few changes that you can find about, listed in the upgrade notes here. Like all upgrades, I thoroughly recommend you study these notes in detail – as there can always be something to trip you up.

This happened to us, with the changed implementation of UDFs and the way libraries are stored and used. You can read all about how it used to work here, but for now let’s go through how to make your Python libraries work with your UDFs again! I’ll be splitting up the content of this blog into a couple of future ones too, as there’s quite a lot to go through!

Get Curl!

First off, you’re going to need Curl. Get it here. Extract it on your computer, and remember where you put it!

Setup Bucket Services

Next up, let’s check out the changes in ExaOperation. There’s the addition of this new tab of ExaBuckets. Head on in there, and click Add.AddBucketService

By default the HTTP port for the service bucket is 2580, if you create another service bucket after this, you’ll need to change the port number (which I need to here – so I’m using 2585). Add a Description, this will be how you access the bucket. I’ll use “Demo”.

My ExaBuckets now looks like this. It has bfsdefault – which will be the service bucket you have, by default, when you get Version 6.


Click on the ID of the service you just created. This will take you through to where we can create the actual bucket. Click on Add.


I’ve added the name “libraries”, we’ll need this later. I’ve checked Public readable, and provided read (‘xyz’) and write passwords (123) for the demo.


Your bucket will be displayed like this.


Upload library to Bucket using Curl

Ok, now let’s use the bucket we created. I’m going to be using Windows, so open up a cmd window. Change the directory to your Curl one from earlier. Let’s list out what we find…. The libraries bucket.

c:\curl>curl Your_IP_here:2585


To put something in the bucket, first download the library locally that you need, from a reputable source, like Python.

Now we’ll get it in our bucket. We use the curl put command, and use the write password we set earlier, along with the bucket name we specified (libraries). Here, I’m uploading the boto3 library.

c:\curl>curl -X PUT -T C:\boto3-1.4.7.tar.gz http://w:123@YOUR_IP_HERE:2585/libraries/boto3-1.4.7.tar.gz

Check your library is there by listing out the bucket.

c:\curl>curl YOUR_IP_HERE:2585/libraries


Use library in UDF script

So, we’ve uploaded our library to the bucket, but our udfs still don’t work! Your script needs the path specifying before the import statement will work.


Exasol have added a video of how to do some of this here.

Personally, I wish they’d kept the GUI in ExaOperation (or a flavor of it), as it was way quicker to use, and this is such a faff to do. This is another distinction between just writing a Python script with a traditional import, and an Exasol specific thing to remember.

Anyway, it’s all there for you if you read it (from me or Exasol!), and a great reminder to study the upgrade notes and think about their implications.


HOW TO: Restart the licence node

Welcome back for the fourth instalment of the AdminSeries. Catch up on the previous posts here:

For this post we’ll be restarting the licence node.

To do this, I recommend you follow all the steps in the above posts beforehand.

That said, head over to the Nodes tab in ExaOperation. Scroll down past the nodes (if you have a few!).


In the License Nodes section, click Reboot. If all goes well, this will take down the page you are viewing, indeed all of ExaOperation (as it primarily resides on the licence node). You should be left with your browser’s page not found or something similar.

Again, I hope you have your patience ready. For me, this step takes around 5 minutes, in which cold sweats ensue (WTF have I done!). But it’s ok.

Refresh the page periodically until ExaOperation comes back.

This concludes Restarting the licence node.

HOW TO: Shutdown the Data Nodes

Time for the third post in the AdminSeries – how to shutdown the data nodes. This post follows the previous two: How to Shutdown the Database(s) and How to Shutdown the Storage. Before you read much further, go check them out first.

With that out the way, let’s get started.

In ExaOperation, head over to the Nodes tab. It should look something like this:nodeson

In this shot, you can see the State of the nodes is Running and they are Powered On.

Next up, select the checkboxes next to the Nodes that you want to shutdown. From the Actions drop down, select Shutdown, and then the button Execute.shutdownnodes

It may take some time for the nodes to shutdown, as with all things ExaOperation, be patient and wait for it. You can refresh the page and watch for the node state to change. Sometimes the Nodes change state, one at a time, so you may be waiting for some nodes.

Eventually your page should look like this. All of the Nodes will have the state of Installed, and Power Off. Your nodes are now Shutdown.


That brings us to the end of How to Shutdown Data Nodes. Next up, we’ll be looking at How to Restart the licence node.

HOW TO: Shutdown the Storage

The second post in the AdminSeries is going to be about how to Shutdown the Storage. Before you read this post, or are trying to do this, go back and make sure you have Shutdown the Database(s) first.

So, assuming that you have already Shutdown the Database(s), head over to the EXAStorage tab in EXAOperation (a lot of EXA’s going on there guys!).

It should look a little something like this:


I’ve got two databases, with three nodes each and a licence node. This means I have 6 volumes listed up top, and 7 disks from my nodes at the bottom. You may have more or less depending on your setup.

Click Shutdown Storage Service.

The page should then look like this:


Here you can see the 7 nodes – 6 data nodes and 1 licence server.

Seeing the EXAStorage page look like this, means that the Storage is now stopped.

Having Shutdown the Database(s) and the Storage, we’ll move onto how to Shutdown the Nodes.

HOW TO: Shutdown the database(s)

This post is the first in a series of posts about how to administer Exasol. We’re going to be covering how to Shutdown databases, Shutdown storage, shutdown the nodes, restart the licence server, install CentOS patches and restart the whole lot. For all of the HOW TO: posts regarding administration, you will need access to ExaOperation.

Firstly, a quick disclaimer: This set of posts is based around 5.0.17, and so should be used in that context. If you are unsure about what you are doing, seek professional help. Do some of these things wrong, and you could jeopardise your data.

Ok, so without further ado…

In general, you should always shutdown the database first, before other shutdowns such as the storage etc. Obviously in the case of an unplanned system shutdown, you don’t have much choice.

So first log into ExaOperation, then head over to the EXASolution tab.

In my case, I have two databases running. I can tell that they are running because of the Status, and also that they are Online and behaving well – with the green light. Databases can be Running and Stopped separately. However, if you intend to perform further maintenance, like Shutting down all nodes or Stopping the Storage, you should stop them all.Inkeddatabasesrunning_LI

To Shutdown the databases, select the tick box next to the databases and then select Shutdown.

You can then manually refresh this page. It may take up to a minute to stop the databases.

When they are stopped the page looks like this:


We can tell that the databases are stopped, because they have a Status of “Created”, and the light indicating that the database is Online has gone out/grey.

So that’s it, your database has stopped.

The next post is then How to Shutdown the Storage.