CPB-API-STORAGE v0.1.1

REST API provides the facilities for the products data operations on Google Cloud Storage (GCS)

Table of contents

• homegitlab-runnerbuilds34206e200reactjscpb-storage-apiapiindex.js
  • GET ALL STORE INFORMATION
• ShopifyStore
  • [GET] /filestore list all shopify stores and top-level dirs for the bucket
  • [GET] /filestore/:shopIdOrName SHOPIFY STORE PRODUCTS AND FILES
  • Get Shopify Product Config by id from the filestore

CUSTOM PRODUCT BUILDER BACKEND STORAGE REST API SERVICE

Overview

The service uses standard ExpressJS and Google Cloud NodeJS API Libraries.
It provides the REST API for the Cloud Storage operations for the CPB Backend services.

Requirements

There are no requirements besides the nodejs v12-16.
it should be able to run on any platform.
Currently only nodejs v16 have been tested on OSX.
The nodejs runtime is set to require v16 in the package.json file in the project root.

Documentation

The jsdoc and apidoc (apidoc-markdown) libraries are used for the documentation.
The internal documentation is in doc dir in the project root. It is not checked into git.

The documentation consists of two components:

• The internal modules docs - the doc/cpb-storage-api/${version} dir in the jsdoc format.
• The API docs (README.md in the project root ) - apidoc format rendered as markdown (apidoc-md) due to the
  Gitlabs' restriction for rendering the html files. The Gitlab Pages are required for the html rendering. This doc
  content is also included into the module documentation.

The documentation is generated during the development install process (npm run installDev).
It is also can be regenerated at any time by invoking the npm run jsdoc command.
The file INSTALL.md (this file) is prepended to the apidoc output README.md and included into the resulting jsdocs.

### install DEVELOPMENT dependencies
npm run installDev
### generate API documentation
npm run apidoc
### generate complete documentation (internal and api docs)
npm run jsdoc

Structure

This project uses Node Module structure in order to use the import pragmas and for the code isolation.
The modules source code is in the src directory. Each top-level folder under the src root corresponds to its
own module with the naming convention of cpb-{dirname}. Each module's description and dependencies are stored
within its package.json file.

├── Dockerfile
├── INSTALL.md            # this file
├── README.md
├── datastore             # cloud datastore indexes
├── doc                   # generated documentation
├── index.js
├── node_modules
├── out                   # temp dir
├── package-lock.json
├── package.json
src
├── api                   # package:cpb-api
├── apidoc.json           # apidoc config
├── common                # package:cpb-common
├── storage               # package:cpb-storage

Installation

The installation scripts are defined within main package.json file.
For the server instance configuration the environment variables are used.

Environment Variables

• BUCKET - Google Cloud Storage Bucket for operations. Defaults to custom-product-builder
• PORT - Server listening port. Defaults to 8080

##### Development Mode
npm run installDev

##### Production Mode
npm install

##### Start Service with default settings
npm start

##### Override default settings
PORT=80 BUCKET=my_bucket npm start

##### Docker (Production)
docker build -t cpb-storage-api:dev .
docker run -p 80:8080 -t cpb-storage-api:dev

homegitlab-runnerbuilds34206e200reactjscpb-storage-apiapiindex.js

GET ALL STORE INFORMATION

Back to top

GET /cpb/:shopIdOrShopName

Parameters - Parameter

Query Parameters

ShopifyStore

[GET] /filestore list all shopify stores and top-level dirs for the bucket

Back to top

Contains Shopify Store IDs, dirs for the legacy app installs (*.myshopify.com), and miscellaneous dirs for the bucket defined at %ENV.BUCKET% at the server runtime or via app.set('bucket') in the src/api/index.js

GET /filestore

Query Parameters

Examples

Get all stores with files and misc and legacy directories within the bucket::

curl localhost:8080/filestore | jq .

Get only filestore ids for the bucket

curl http://localhost:8080/store?files=1&fetch=0&legacy=0&misc=0

Success response

Success response - Success 200

Success response example

Success response example - all stores with files and misc and legacy directories within the bucket: Success-Response:

{
  "bucket": "custom-product-builder",
  "ids": [
    10003054628,
    10012622910,
    10025467982
  ],
  "misc": [
    "cpb-assets/",
    "custom-product-builder-stage/",
    "customproductbuilder/",
    "dev-test-shop/"
  ],
  "legacy": [
    "alpha-crystal-jewellery.myshopify.com/",
    "alpha-wraps.myshopify.com/",
    "alumepixalayof.myshopify.com/",
    "amasal.myshopify.com/"
  ],
  "files": [
  {
          "name": "64audio-5540496998550-J9gcVRJDZzGfeiSZc1ZZslq5.jpg",
          "versions": [
            {
            "id": "custom-product-builder/64audio-5540496998550-J9gcVRJDZzGfeiSZc1ZZslq5.jpg/1627414331517243",
            "generation": 1627414331517243,
            "metadata": {
               "md5Hash": "rBkJMvwCwAYY0QtuO4pYRA==",
               "crc32c": "CsVSMQ==",
               "timeCreated": "2021-07-27T19:32:11.547Z",
               "updated": "2021-07-27T19:32:11.547Z"
            },
          "isDeleted": false,
          "isCurrent": true,
          "size": 52167,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/64audio-5540496998550-J9gcVRJDZzGfeiSZc1ZZslq5.jpg?generation=1627414331517243&alt=media"
          }
          ],
            "isDeleted": false,
            "currentFileSize": 52167,
            "generation": 1627414331517243,
            "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/64audio-5540496998550-J9gcVRJDZzGfeiSZc1ZZslq5.jpg?generation=1627414331517243&alt=media",
            "totalSize": 52167,
            "previousFileVersionsCount": 0,
            "previousFileVersionsSize": 0
      }
  ],
  "deletedFiles": [],
  "stats": {
  "ids": 7177,
  "legacy": 66,
  "misc": 42,
  "files": 1005,
  "deletedFiles": 0
}
  "debug":{
     "params": {},
     "query": {}
  }
}

Success response example - Get only filestore ids for the bucket: Success-Response:

{
  "bucket": "custom-product-builder",
  "ids": [
    10003054628,
    10012622910,
    10025467982,
    10039296064
  ],
  "stats": {
    "ids": 7183,
    "legacy": 66,
    "misc": 42,
    "files": 1033,
    "deletedFiles": 0
  },
  "debug": {
    "bucket": "custom-product-builder",
    "fetch": false,
    "versions": false,
    "showFiles": true,
    "misc": false,
    "legacy": false,
    "request": {
      "query": {
        "files": "1",
        "fetch": "0",
        "legacy": "0",
        "misc": "0"
      },
      "params": { }
    },
    "timestamp": "2021-12-19T13:17:46.382Z"
  }
}

[GET] /filestore/:shopIdOrName SHOPIFY STORE PRODUCTS AND FILES

Back to top

Get the filestore data for the given :shop_id with the summarized sizes and ShopifyProductId

GET /filestore/:shopIdOrName

Parameters - Parameter

Query Parameters

Examples

Example usage:

STORE_ID=10003054628 curl localhost:8080/filestore/${STORE_ID} | jq .

Success response

Success response - Success 200

Success response example

Success response example - Success-Response:

{
  "id": 10003054628,
  "bucket": "custom-product-builder",
  "products": [
    7000615387329,
    7000615420097,
    7000615452865,
    7000615813313
  ],
  "deletedProducts": [],
  "deletedFiles": [],
  "files": [
    {
      "name": "7000615387329.json",
      "versions": [
        {
          "id": "custom-product-builder/10003054628/7000615387329.json/1635936486341218",
          "generation": "1635936486341218",
          "metadata": {
            "timeCreated": "2021-11-03T10:48:06.376Z",
            "updated": "2021-11-03T10:48:06.376Z",
            "timeDeleted": "2021-11-03T10:48:06.641Z"
          },
          "isDeleted": true,
          "size": 100904,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615387329.json?generation=1635936486341218&alt=media"
        },
        {
          "id": "custom-product-builder/10003054628/7000615387329.json/1635936486517468",
          "generation": "1635936486517468",
          "metadata": {
            "timeCreated": "2021-11-03T10:48:06.641Z",
            "updated": "2021-11-03T10:48:06.641Z"
          },
          "isCurrent": true,
          "size": 100904,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615387329.json?generation=1635936486517468&alt=media"
        }
      ],
      "currentFileSize": 100904,
      "generation": 1635936486517468,
      "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615387329.json?generation=1635936486517468&alt=media",
      "totalSize": 201808,
      "previousFileVersionsSize": 100904
    },
    {
      "name": "7000615420097.json",
      "versions": [
        {
          "id": "custom-product-builder/10003054628/7000615420097.json/1635936486855324",
          "generation": "1635936486855324",
          "metadata": {
            "timeCreated": "2021-11-03T10:48:06.890Z",
            "updated": "2021-11-03T10:48:06.890Z",
            "timeDeleted": "2021-11-03T10:48:07.148Z"
          },
          "isDeleted": true,
          "size": 39404,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615420097.json?generation=1635936486855324&alt=media"
        },
        {
          "id": "custom-product-builder/10003054628/7000615420097.json/1635936487029707",
          "generation": "1635936487029707",
          "metadata": {
            "timeCreated": "2021-11-03T10:48:07.148Z",
            "updated": "2021-11-03T10:48:07.148Z"
          },
          "isCurrent": true,
          "size": 39404,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615420097.json?generation=1635936487029707&alt=media"
        }
      ],
      "currentFileSize": 39404,
      "generation": 1635936487029707,
      "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615420097.json?generation=1635936487029707&alt=media",
      "totalSize": 78808,
      "previousFileVersionsSize": 39404
    },
    {
      "name": "7000615452865.json",
      "versions": [
        {
          "id": "custom-product-builder/10003054628/7000615452865.json/1635936486472693",
          "generation": "1635936486472693",
          "metadata": {
            "timeCreated": "2021-11-03T10:48:06.508Z",
            "updated": "2021-11-03T10:48:06.508Z",
            "timeDeleted": "2021-11-03T10:48:06.777Z"
          },
          "isDeleted": true,
          "size": 90487,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615452865.json?generation=1635936486472693&alt=media"
        },
        {
          "id": "custom-product-builder/10003054628/7000615452865.json/1635936486654280",
          "generation": "1635936486654280",
          "metadata": {
            "timeCreated": "2021-11-03T10:48:06.777Z",
            "updated": "2021-11-03T10:48:06.777Z"
          },
          "isCurrent": true,
          "size": 90487,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615452865.json?generation=1635936486654280&alt=media"
        }
      ],
      "currentFileSize": 90487,
      "generation": 1635936486654280,
      "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615452865.json?generation=1635936486654280&alt=media",
      "totalSize": 180974,
      "previousFileVersionsSize": 90487
    },
    {
      "name": "7000615813313.json",
      "versions": [
        {
          "id": "custom-product-builder/10003054628/7000615813313.json/1635936606084578",
          "generation": "1635936606084578",
          "metadata": {
            "timeCreated": "2021-11-03T10:50:06.120Z",
            "updated": "2021-11-03T10:50:06.120Z"
          },
          "isCurrent": true,
          "size": 652,
          "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615813313.json?generation=1635936606084578&alt=media"
        }
      ],
      "currentFileSize": 652,
      "generation": 1635936606084578,
      "url":
  "https://storage.googleapis.com/download/storage/v1/b/custom-product-builder/o/10003054628%2F7000615813313.json?generation=1635936606084578&alt=media",
      "totalSize": 652,
      "previousFileVersionsSize": 0
    }
],
  "stats": {
    "files": 4,
    "previousFileVersions": 7,
    "products": 4,
    "size": 462242,
    "previousFileVersionsSize": 230795,
    "currentFileSize": 231447
  },
  "debug": {
    "params": {
      "id": "10003054628"
    },
    "query": {}
  }
}

Get Shopify Product Config by id from the filestore

Back to top

Get Shopify Product Config by id from the filestore

1. Restriction of the public file downloads only to the registered client app to prevent unauthorized data copy and reverse engineering.
2. Removing of unneeded attributes from the file and sending sanitized version to client while resaving the processed file.
3. Adding additional attributes to the config.
4. Storage migrations.
5. Advanced requests logging for the analytical or debugging purposes
6. Restriction of the data access for the expired trial app.

GET /filestore/:shop_id/product/:product_id

Error response

Error response - Error 4xx