Restoring Search and Graph Indices from Local Database

If search or graph services go down or you have made changes to them that require reindexing, you can restore them from the aspects stored in the local database.

When a new version of the aspect gets ingested, GMS initiates an MAE event for the aspect which is consumed to update the search and graph indices. As such, we can fetch the latest version of each aspect in the local database and produce MAE events corresponding to the aspects to restore the search and graph indices.

Quickstart

If you're using the quickstart images, you can use the datahub cli to restore indices.

datahub docker quickstart --restore-indices

See this section for more information.

Docker-compose

If you are on a custom docker-compose deployment, run the following command (you need to checkout the source repository) from the root of the repo to send MAE for each aspect in the Local DB.

./docker/datahub-upgrade/datahub-upgrade.sh -u RestoreIndices

If you need to clear the search and graph indices before restoring, add -a clean to the end of the command.

Refer to this doc on how to set environment variables for your environment.

Kubernetes

Run kubectl get cronjobs to see if the restoration job template has been deployed. If you see results like below, you are good to go.

NAME                                          SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
datahub-datahub-cleanup-job-template          * * * * *   True      0        <none>          2d3h
datahub-datahub-restore-indices-job-template  * * * * *   True      0        <none>          2d3h

If not, deploy latest helm charts to use this functionality.

Once restore indices job template has been deployed, run the following command to start a job that restores indices.

kubectl create job --from=cronjob/datahub-datahub-restore-indices-job-template datahub-restore-indices-adhoc

Once the job completes, your indices will have been restored.

Through API

See Restore Indices API.