Since taking the Aster Data class, I wanted to learn more about this exciting new technology. After downloading the 2 VMware images (Queen and Worker), I needed another Worker node to be able to increase the Replication Factor (RF) from 1 to 2.
The Replication Factor is the number of copies of data that are stored in Aster to provide tolerance against hardware failure. For example, with a RF = 2, when you load data, the system will copy the same data on 2 different physical Worker nodes. So if 1 node goes down, the system can still function. So it’s similar to what Teradata does when you configure Fallback on the AMPs. An added bonus with RP = 2 is that Aster will also replica the Queen’s data, which ensures you can perform faster Queen replacement.
I wanted to test this concept by intentionally bringing an Active Worker node down while executing a query to see if the system could recover and bring back the answer set in real time. With the nCluster User’s Guide at my side, I set aside a full day to change the RF. What possibly could go wrong?
Before we begin, I do want to point out the resources needed to run 3 VMware images on one physical PC. You will probably want a newer PC that has scads of RAM. I'm running this demo on a Quad-CPU with 8 GB of RAM and performance is reasonable. According to Task Manager, I'm consuming almost 5.5 GB of RAM so I would not attempt this on system will less than 6 GB RAM. Having said that, let's roll.
This seemed easy enough. Just make a copy of the existing Vmware image for the Worker node and then fire it up. I figured I would have to change the host name and IP address on this new image. That was quite simple using the GUI tool provided in the image. But when I attempted to add a Worker node from the Aster Management Console (AMC), it would not do so. I then noticed what might be the problem. When adding a Worker, you can specify IP or MAC address.
I was guessing since I had 2 identical MAC addresses for the 2 Worker nodes, this is why the new node was failing. After a Google search on how to change MAC address’s on Linux, I edited the boot.local file and inserted a fictitious MAC address as shown below.
Sure enough, this time I was able to register the Worker node using the AMC. However, the Worker node would never enter Active Status, but rather stayed Passive. With Status = Passive, the Worker node is in standby mode which clearly is not what I wanted. I needed it to be Active so it could be available to process queries. Hmmm, there must some other configurations I need to do. But since it was already dark outside, I figured I had enough for one day.
Now I was at a standstill since the Worker-2 node would not enter Status = Active. At that point, I knew I needed an intervention. With an e-mail to Aster Support Team, I laid out my dilemma. Sure enough, a reply mentioned it might have something to do with Partition Splitting. Another trip back to User’s Guide shed light on this topic.
Partition Splitting is an unfortunate name as far as I’m concerned. ‘Partition’ has a lot of different definitions depending on whom you ask in the Aster world. Basically, Partition Splitting is a feature that adds v-Workers. Let’s back up a minute to get perspective on this.
To scale out your cluster, you add Worker nodes. Of course as you add Workers, you get more CPUs on those nodes. To execute queries, Worker nodes need v-Workers. V-Workers do all the heavy lifting for a Worker node. Aster Data recommends you have about 2 CPU cores per v-Worker. If you have less than this, the CPUs may become underutilized. Since I was unable to change the RF to 2, maybe it had something to do with a lack of v-Workers.
In the User’s Guide, it mentioned a file that held the current partition count. So from the command shell of the Queen, I ran:
$ cat /home/beehive/config/totalPartitionCount
That resulted in a v-Worker count = 1. So at this point, I had potentially solved the mystery. Worker-2 node was Passive since I only had 1 v-Worker available and it was already handed out to Worker-1 node. To confirm this, I went to the AMC under Partition Map and sure enough, it stated I had 1 Primary v-Worker and it was on Worker-1 node.
Time to do some Partition Splitting. From the Queen, I ran the following to increase the v-Workers to 2:
After a Soft Reset (which restarts the software on the Nodes), I did an Activate Cluster ( which brings Nodes online). A Balance Data (which moves v-Workers among Worker nodes) was done automatically as was a Balance Process (which sets v-Workers as Primary or Secondary).
At this point I had 2 Active Worker nodes as shown below.
The Partition Map now displayed each Worker had 1 Primary v-Worker (green box within each Worker node). This meant Aster could work in parallel. However I still have not achieved RP=2.
RP = 2 would have to wait for tomorrow; I still have my day job to do.
To change the RP = 2, go to a command prompt on the Queen and navigate to:
Here you will need to edit the goalReplicationFactor file. I changed mine from 1 to 2, then saved the file. Next, from the AMC, do a Soft Reset followed by an Activate Cluster.
Data Balancing occurred automatically meaning in my case, all the data on Worker-1 was being replicated on Worker-2. You could see this occurring in real-time via the Partition Map. Notice both Worker nodes have a Primary v-Worker and in the green box in the upper right-hand corner a ‘Replication in progress’.
Once Data Balancing was complete, I had RF = 2.
Now if a Worker node went down, the Secondary v-Worker on the other Node would become Primary, and answer the query. At least that’s what the book. Time to prove it.
I submitted a long-running query against the database, confirmed it was running on both Worker’s, then disabled the Network card on 1 of the Workers. In my case, after a delay of 4 minutes, my query came back.
Here’s what the Partition Map looked like when I took down the 1 Worker node. As you can clearly see, I went from 1 Primary v-Worker to 2 Primary v-Workers on Worker-1 node.
So basically the Secondary v-Worker originally named 1S in the previous screen shot became a Primary v-Worker named 1P. At that point, Worker-1 node could answer the query for the downed Worker-2 node since it had a copy all the rows Worker-2 originally had before it went down.
Victory at last. It took me 72-hours to configure fault tolerance in Aster, but it was worth it. I learned a lot along the way that wasn’t in the User’s Manual. All in a day’s work.