Child pages
  • Restoring Domains and Buckets
Skip to end of metadata
Go to start of metadata


This section describes how to recover deleted domains or buckets in your storage cluster if the domain or bucket has left behind orphan content that is not accessible without its parent context. Domain or bucket deletes may now be performed with a recursive query argument that indicates any content contained in the deleted context may be asynchronously deleted by the health processor after the time period indicated by the health.recursiveDeleteDelay parameter (default of one week). If you delete a domain or bucket without the recursive query argument and without first deleting all objects contained in the bucket or domain, the cluster will log critical errors indicating there is orphaned content.

Unless the space consumed by the orphaned content is minimal, best practice is to re-create the parent domain or bucket and then either delete the child objects manually or redelete the domain or bucket with the recursive query argument so the health processor can perform the cleanup on your behalf.

Recovering a Deleted Domain

The following procedure shows how to create the domain from the command line using the previous domain's UUID.

If you create a domain with an identical name that is currently in the Netmail Admin Console, the new domain is mapped to a different UUID. Since the buckets in the deleted domain reference the deleted domain UUID (for example, Castor-System-CID), you cannot access the buckets unless you set the new domain's UUID to the previous value.

If the domain was created via the Netmail Admin Console and required authentication, you will also need the Castor-Authorization header corresponding to the domain's protection setting, as shown in the table below:

Protection SettingCastor-Authorization Header
All Users. No authentication required.

Castor-Authorization: domain-name/
_administrators, POST= (since this header does not enforce
authentication, it is optional on a re-creation)

Only users in this domain.

Castor-Authorization: domain-name/
_administrators, POST=domain-name

Only users in domain

Castor-Authorization: domain-name/
_administrators, POST=domain-name
The difference between this protection setting and the preceding is that in
this case, domain-name is the name of a different domain in the cluster.

To recover the domain:

1. Record the critical log message related to the missing domain. The critical log message should look as follows:

Domain 'example.com' (uuid=...) has been deleted with orphan content.
Consider recreating.

The UUID listed in the log message is required to re-create the missing domain.

2. POST the previous domain's UUID using the recreatecid query argument to create the new domain, passing in the Castor-Authorization, Castor-Stream-Type, and lifepoint headers exactly as shown below.

curl -i -X POST
  -H 'Castor-Authorization: protection-setting'
  -H 'Castor-Stream-Type: admin'
  -H 'lifepoint: [] reps=16'
  --databinary ''
  --post301
  --location-trusted
  'http://node-ip?domain=domainname&admin&recreatecid=previous-domain-UUID'
  --anyauth -u 'yourusername:your-password'
  [-D log-file-name]

Caution: Be sure to enter these headers exactly as shown so that they match the headers used when domains are created by the Console. lifepoint: [] reps=16 enables the domain to be replicated as many times as possible. Use Castor-Stream-Type: admin for all root objects that are accessed frequently and all objects that use a Castor-Authorization header.

For example, if the domain name is cluster.example.com with the protection setting Only users in this domain, and the old domain alias was c0d0fa42bccac73cd3f2324bb53e40a5, enter the following command:

curl -i -X POST -H 'Cache-Control: no-cache-context'
  -H 'Castor-Authorization: cluster.example.com/_administrators,
  POST=cluster.example.com'
  -H 'Castor-Stream-Type: admin'
  -H 'lifepoint: [] reps=16'
  --data-binary ''
  --post301
  --location-trusted
  'http://172.16.0.35?domain=cluster.example.com&admin
  &recreatecid=c0d0fa42bccac73cd3f2324bb53e40a5'
  --anyauth -u 'admin:ourpwdofchoicehere'

3. Create the _administrators bucket for the domain by entering the following command:

curl -i -X POST -H 'Cache-Control: no-cache-context'
  -H 'Castor-Authorization: domain-name/_administrators'
  -H 'Castor-Stream-Type: admin'
  -H 'lifepoint: [] reps=16'
  --data-binary ''
  --post301
  --location-trusted
  'http://node-ip/_administrators?domain=domain-name&admin'
  --anyauth -u 'your-username:your-password'
  [-D log-file-name]

If you see the following error, you omitted --post301 from the command:

CAStor Error Content-Length header is required

For example, to create the cluster.example.com/_administrators bucket:

curl -i -X POST
  -H 'Cache-Control: no-cache-context'
  -H 'CastorAuthorization: cluster.example.com/_administrators'
  -H 'CastorStream-Type: admin'
  -H 'lifepoint: [] reps=16'
  --data-binary ''
  -post301
  --location-trusted
  'http://172.16.0.35/_administrators?domain=cluster.example.com&admin'
  --anyauth -u 'admin:ourpwdofchoicehere'

4. To verify the procedure, start the Netmail Store Admin Console.

5. On the Cluster Status page, click Settings.

6. In the Cluster Tenants section, ensure that the domain name and protection setting display correctly.

7. Click Edit next to the name of the domain you just restored.

8. Click Add Domain Manager.

9. Follow the prompts on your screen to create a domain manager.

If you added a domain manager successfully, the procedure completed successfully.

The Netmail Store Admin Console may display an alert for not having an _administrators bucket in the domain.

10. (optional) Return to the Cluster Settings page and click the IP address of any node with a red Alert message that is similar to the following:

Error reading admin bucket 'cluster.example.com/_administrators' ([Errno 2] Bucket not found)

11. Click Clear Errors to confirm.

Recovering a Deleted Bucket

When you delete a bucket, the objects in the bucket are not deleted, but they are inaccessible until you recover the bucket.

In the following procedure, assume that an application developer notifies you that the following objects are not accessible: photo1.jpg, photo2.jpg, photo3.jpg

You do not know the name of the bucket in which the objects were contained.

To recover the bucket:

1. Record the critical log message related to the missing bucket. The critical log message should look as follows:

Bucket 'test_bucket' (uuid=...) in domain 'example.com' has been deleted with orphan content. Consider recreating.

The UUID listed in the log message is required to recreate the missing bucket.

2. After you locate the bucket's UUID, use the following command to recover it:

curl -i -X POST --location-trusted --post301
  --anyauth -u 'admin:password'
  --data-binary ''
  'http://node-ip/bucket-name?domain=name&admin&recreatecid=alias-uuid'

You must provide the domain name or IP address as the Host in the request. This example recovers a bucket mybucket with the alias UUID 75edd708dc250137849bbf590458d401 in the domain named cluster.example.com:

curl -i -X POST --location-trusted --post301
  -anyauth -u 'admin:ourpwdofchoicehere'
  --data-binary ''
  'http://172.16.0.35/mybucket?domain=cluster.example.com&admin
  &recreatecid=75edd708dc250137849bbf590458d401'

  • No labels