Meter Ingestion - Event schema

The default ingest event schema supported by Amberflo is as follows:

{
    "customerId": string,
    "meterApiName": string,
    "meterValue": number,
    "meterTimeInMillis": number,
    "uniqueId": string
    "dimensions": map<string, string>
}

Amberflo supports the following input formats

JSON
NDJSON
CSV

Amberflo also supports defining a custom schema, which transforms the input into Amberflo ingest events. This guide explains how to use custom schema for ingesting meter events.

JSON format

Sample Input in JSON format.

[{
     "customerId": "customer-123",
     "meterApiName": "ComputeHours",
     "meterValue": 5,
     "meterTimeInMillis": 1619445706909,
     "dimensions": {
        "region": "us-west-2",
        "az": "az1"
     }
}, {
    ...
}]

NDJSON format

Amberflo also supports the NDJSON format, where each record is separated by a newline.

Sample Input in NDJSON format.

{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 5, "meterTimeInMillis": 1619445706909 }
{ "customerId": "customer-321", "meterApiName": "ComputeHours", "meterValue": 4, "meterTimeInMillis": 1619445712341 }
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 1, "meterTimeInMillis": 1619445783456 }

CSV format

Sample Input in CSV format.

meterApiName,meterValue,meterTimeInMillis,customerId,region
ComputeHours,5,1619445706909,customer-123,us-west-2
ComputeHours,4,1619445712341,customer-321,us-east-1
ComputeHours,1,1619445783456,customer-123,us-west-2

Difference in behavior for CSV and JSON inputs

If custom schema is not used, then for CSV input the columns not part of Amberflo default schema will be converted to dimensions, whereas for JSON input keys which are not part of Amberflo default schema are excluded.

Lets take a look at the below table to understand this.

Input

Transformed Input

CSV

meterApiName	meterValue	meterTimeInMillis	customerId	region
m1	1	1714070079000	c1	us-west-2

{
  "meterApiName": "m1", 
  "meterValue": 1, 
  "meterTimeInMillis": 1714073679000, 
  "customerId": "c1",
  "dimensions": {
    region": "us-west-2"
  }
}

JSON

{
  "meterApiName": "m1", 
  "meterValue": 1, 
  "meterTimeInMillis": 1714073679000, 
  "customerId": "c1",
  "region": "us-west-2"
}

{
  "meterApiName": "m1", 
  "meterValue": 1, 
  "meterTimeInMillis": 1714073679000, 
  "customerId": "c1" 
}

As we can see that the region column/key in CSV and JSON inputs does not belong to default Amberflo schema. So we can see in the transformed input that the region column has been transformed into dimensions for CSV input whereas it is excluded for the JSON input.

If a custom schema is used then only the columns/key mapped in the schema sub components will be considered rest others will be excluded for both CSV and JSON.

Defining custom ingest schema

The Amberflo Custom Ingest Schema API lets you define your own custom schema based on your events.
This custom schema can be used with both CSV and JSON inputs. The schema has sub components which define the mapping of a particular source column-name/json-key to the amberflo ingest event parameter.

Note

Each schema should contain at least sub components for meter api name, meter value, customer id and event time.
Each sub component should have a different source name, multiple sub components cannot have same source name.

For instance consider the following custom schema:

{
    "id": "custom-schema-id",
    "schema": {
        "type": "explode",
        "subComponents": [
            {
                "type": "meter_api_name",
                "sourceName": "met"
            },
            {
                "type": "meter_value",
                "sourceName": "val"
            },
            {
                "type": "event_time",
                "timeFormat": "EPOCH_SECONDS",
                "sourceName": "time"
            },
            {
                "type": "customer_id",
                "sourceName": "c_id"
            }
        ]
    }
}

In this custom schema you have 4 sub components which map 4 different columns/keys to ingest event parameters.

CSV example

Sample CSV Input

met	val	time	c_id
m1	1	1714070079	c1
m1	1	1714070079	c2

Using the custom schema the above CSV input will be transformed to the following ingest payload

[
    { 
        "meterApiName": "m1",
        "customerId": "c1",
        "meterValue": 1,
        "meterTimeInMillis": 1714070079000
    },
    { 
        "meterApiName": "m1",
        "customerId": "c2",
        "meterValue": 1,
        "meterTimeInMillis": 1714070079000
    }
]

JSON example

Sample JSON input

[
  {"met": "m2", "c_id": "c3", "val": 10, "time": 1714073679},
  {"met": "m2", "c_id": "c4", "val": 10, "time": 1714073679}
]

Using the custom schema the above json input will be transformed to the following ingest payload

[
    { 
        "meterApiName": "m2",
        "customerId": "c3",
        "meterValue": 10,
        "meterTimeInMillis": 1714073679000
    },
    { 
        "meterApiName": "m2",
        "customerId": "c3",
        "meterValue": 10,
        "meterTimeInMillis": 1714073679000
    }
]

Using custom ingest schema

Once you have created the custom ingest schema, you can use this schema to parse your meter events.

Ingesting through API

To use the custom schema for parsing ingest payload

Get the id of the schema that you want to use for parsing.
Provide the schema id as a query string parameter named ingestSchemaId.

Example

https://app.amberflo.io/ingest?ingestSchemaId=<schema-id>

Sample curl request to ingest events using the custom schema defined in previous section

curl --request POST \
     --url 'https://app.amberflo.io/ingest?ingestSchemaId=custom-schema-id' \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --header 'x-api-key: <replace-with-api-key>' \
     --data '
{
  "c_id": "c3",
  "met": "m2",
  "val": 1,
  "time": 1714070079
}
'

Ingesting through AWS S3

To use custom schema when ingesting events using AWS S3 the ingestSchemaId should be provided in the s3 file key.

S3 key format for using custom schema

/ingest/amberdata/06-07-2022/ingestSchemaId=<schema-id>/0000a2e4-e6ad-11ec-8293-6a8da1c4f9f0