Brainspace Memory Allocation

Application Server

Brainspace automatically allocates 70% of the total amount of RAM to itself (displayed in the UI as JVM memory for the Application Service), leaving the remaining 30% for other processes such as normal OS functionality, PostgreSQL, Lucene, and any other services.

• Datasets that are in an Active status will live in Memory (i.e. JVM memory already allocated to Brainspace).

• For active datasets’ main indexes to be retained in filesystem cache, the remaining memory (i.e. the 30%) needs to be equal to the total or greater of the main indexes.

• On dataset start-up, the application will load the entire main-index of each dataset into the filesystem cache for Lucene searching (e.g., Concept search).

If you want to determine if you currently have enough memory on your Application Server, the first step would be to run the dataset-ram.sh script found in /var/lib/brains/scripts/. This will give you an idea if you have enough memory to run the datasets currently in Brainspace.

Another option would be to gather the size of the main-index directory of all datasets (at least all active datasets); this will give you the minimum amount of RAM needed to be left over for your operating system's filesystem cache to support fast searches. To determine this, you would need to use the following command:

du -ch /data/brainspace/builds/*/output/doc-index/lucene/main-index/ | tail -1 | cut -f 1

Note

This command will give you the size of the main-index of all the datasets.

Once you have the output of the above command, you will then want to compare that to the amount of memory being displayed as available from the free -h command.

As you can see in this example, we only have a total of ~1G (adding free and available together), so if we wanted speedy results on searches we would need to add about 2.6G (round to 3 for good measure) to the Application Server.

Analytics/On-Demand Analytics Server

Brainspace will allocate only half a gig of memory for the Java process, this is so it can run the brainspace-analysis service; the remaining memory will be used by batchtools when doing the analytics work.