Backup to GCS - is that coming soon?

Help
1

I saw there is support to take backup/incremental backup to azure/s3.
is there a plan to add support for backup to GCS?

2

Yes, we are working on support for Google Cloud Storage as well. That will be supported in a future release. It will support Backup as well as pipelines and Select to Outfile as well.

Rick

1 Like
3

I have attempted to backup to GSC without any luck yet.

What credentials should be used? I tried both private_key_id, private_key, client_email and client_id from a JSON key belonging to a Service Account.

Should I instead authenticate through API key or OAuth?

What are the correct authentication method, @rick?

General error: 2298 Could not perform remote backup. SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

4

Did you set the compatibility flag (as documented here: BACKUP DATABASE · SingleStore Documentation)? Also, what version of MemSQL are you using? This is a recently added capability.

Rick

5

The feature was added in 6.8.16.

Rick

6

Yes, I’m setting the compatibility flag, but as a bool instead of a string as mentioned in the docs. Otherwise I receives this error:

General error: 2298 Could not perform remote backup. Invalid JSON in CONFIG: json: cannot unmarshal string into Go struct field s3ConfigComponents.compatibility_mode of type bool

Version: 7.0.9

Can you confirm the expected fields from the JSON key is client_email and private_key?

I’m also forced to remove \n characters from the private key, otherwise this error is thrown:

General error: 2298 Could not perform remote backup. Invalid JSON in CREDENTIALS: invalid character ‘\n’ in string literal

Final SQL:

BACKUP DATABASE ... TO S3 '...' CONFIG '{"compatibility_mode":true,"endpoint_url":"https:\/\/storage.googleapis.com\/"}' CREDENTIALS '{"aws_access_key_id":"...","aws_secret_access_key":"..."}

I’ve also tried the endpoint_url without escapes, however this doesn’t seems to be the issue.

7

Hi!

Could you, please, try to specify your google_access_key_id as your aws_access_key_id and your google_access_key_secret as your aws_secret_access_key. Your google_access_key_id should start with GOOG, and secret key should be base64 encoded string (i believe something like 40 chars long). A little more info here: Fully migrate from Amazon S3 to Cloud Storage  |  Google Cloud

So eventually it should look like:

BACKUP DATABASE ... TO S3 '...' CONFIG '{"compatibility_mode":true, "endpoint_url":"https://storage.googleapis.com/"}' CREDENTIALS '{"aws_access_key_id": "GOOG...", "aws_secret_access_key": "bGoa+V7g/yqDX..."}'
1 Like
8

It works! Thanks @yznovyak-ua

Wow, I’ve been using GCP for 5 years and never heard of GCS keys.

However, the endpoint_url must not include the trailing slash:

General error: 2205 Backup subprocess nonzero exit value. Endpoint: storage.googleapis.com/ does not follow ip address or domain name standards.

Perhaps the docs should be updated with compability_mode using a bool and without the trailing slash in the endpoint_url (GCP does the same). Also, the link you provided me might be helpful to others. :slight_smile:

9

Really glad that we could help!

Sorry for the inaccuracies in the docs (re: trailing slash, and quotes around true, and stuff about credentials) – we will definitely update them.

10

No problem, I’m glad to be able to give some feedback in return for a great feature - and product. :+1:

11

Hi Rick,
what about support (GCS) to Pipelines - is it already found?

Thanks,

12

Hello,

Running MemSQL Pipelines using GCS as a data source is not supported yet, but it will be supported in version 7.1, which is targeted for release this Spring.

Best,
Roxanna