doc/AGDB/6_AGDB_enhanced_query_logic.md

Certainly, we can flatten the JSON structure to simplify the node storage and make it CSV-friendly while maintaining the predefined schema for efficient querying. This approach should make it easier to store time-series data as rows and facilitate faster lookup and relationship traversal, especially for time-based queries. Let’s structure it to match your requirements and then break down the logic for setting up predefined checkpoints.

Revised JSON Structure for Time-Series Data in AGDB

This flattened structure will store metadata, schema definitions, and node data in a way that optimizes for simplicity and quick access. Each entry in the nodes section will follow a CSV-like format but retains enough structure to be directly loaded and queried.

{
    "metadata": {
        "title": "BTC-USD Time Series Data",
        "source": "AGT Platform",
        "description": "Time-series AGDB for BTC-USD trading data with predefined checkpoints",
        "created_at": "2024-11-04",
        "timezone": "UTC"
    },
    "schema": {
        "entity": "BTC_USD_Data",
        "type": "TimeSeriesNode",
        "domain": "TradingData",
        "attributes": ["Time", "Node_ID", "Open", "High", "Low", "Close", "Volume"]
    },
    "data": [
        // Flattened time-series data entries in CSV-like format
        ["2024-10-14 07:30:00", "node_0001", 50, 52, 48, 51, 5000],
        ["2024-10-14 07:31:00", "node_0002", 51, 55, 43, 55, 3000],
        // Additional entries go here
    ],
    "relationships": [
        // Predefined relationships for cardinal (checkpoints) and standard nodes
        {
            "type": "temporal_sequence",
            "from": "node_0001",
            "to": "node_0002",
            "relationship": "next"
        }
    ],
    "policies": {
        "AGN": {
            "trading_inference": {
                "rules": {
                    "time_series_trend": {
                        "relationship": "temporal_sequence",
                        "weight_threshold": 0.5
                    },
                    "volatility_correlation": {
                        "attributes": ["High", "Low"],
                        "relationship": "correlates_with",
                        "weight_threshold": 0.3
                    }
                }
            }
        }
    }
}

Explanation of Each Section

1. Metadata:

   • Provides information about the dataset, source, description, and creation timestamp. This is particularly useful for keeping track of multiple AGDBs.

2. Schema:

   • Defines the structure of each data entry (or node) in the data section.
   • The attributes field specifies the order of fields in the data rows, similar to a CSV header row, making it easier to map attributes to node properties.

3. Data:

   • Flattened time-series data where each entry is a row of values matching the schema's attributes.
   • Each entry begins with a timestamp (formatted in YYYY-MM-DD HH:MM:SS), followed by Node_ID, and then the financial data values: Open, High, Low, Close, and Volume.
   • This structure simplifies parsing, storage, and querying.

4. Relationships:

   • Stores predefined relationships between nodes, including temporal sequences (e.g., next, previous), which allow traversal through the time series.
   • Cardinal (checkpoint) nodes can be defined here, such as daily or hourly intervals, to act as reference points for efficient time-based queries.

5. Policies:

   • Specifies inference rules for AGNs that apply to this dataset. For example, relationships like temporal_sequence or correlates_with can guide AGN in deriving insights across nodes.

Enhanced Query Logic Using Cardinal Nodes (Checkpoints)

To optimize queries for large datasets, we can introduce cardinal nodes that act as checkpoints within the time series. Here’s how these checkpoints can be structured and utilized:

1. Define Checkpoints:

   • Create a cardinal node for each hour (or other intervals, like days) that can link to the closest time-based nodes within that period.
   • Example: If the dataset starts at 8:00 AM, create an hourly checkpoint at 08:00, 09:00, and so on, which links to the first node of that hour.

2. Node-Checkpoint Relationships:

   • Each checkpoint node will connect to the nodes within its respective hour.
   • For instance, 2024-10-14 08:00:00 checkpoint links to all nodes within 08:00 - 08:59, helping you skip directly to relevant entries.

3. Example Relationships for Checkpoints:

   {
       "relationships": [
           {
               "type": "temporal_checkpoint",
               "from": "2024-10-14 08:00:00",
               "to": "node_0800",
               "relationship": "hourly_start"
           },
           {
               "type": "temporal_sequence",
               "from": "node_0800",
               "to": "node_0801",
               "relationship": "next"
           }
       ]
   }
   

4. Querying with Checkpoints:

   • When querying for a specific time, first find the nearest checkpoint. From there, navigate within the hour to locate the exact timestamp.
   • Example query: If searching for 2024-10-14 10:45, start at 10:00 checkpoint and navigate forward until reaching 10:45.

API Queries and Command Logic

Using the proposed flattened structure, we can create a simplified command set for interacting with the data. Here’s how each command might be structured and used:

1. create-graph:

   • Initializes a graph structure based on the schema and metadata defined in JSON. If the schema is time series, it creates relationships accordingly.

2. create-node:

   • Adds a new row of data to data, following the structure in schema.
   • Can specify relationships, such as linking a new node to the previous node in time.

3. get-node:

   • Retrieves the data for a specific node, either by node ID or timestamp.
   • Supports attribute filtering, e.g., get-node.attribute -name "2024-10-14 08:30:00" -attributes "Open, Close".

4. set-attribute:

   • Allows updating node attributes, for example, to modify the Close value of a specific timestamped node.

5. create-relationship:

   • Defines relationships between nodes, such as next, previous, or custom relationships like volatility correlation between attributes.

6. get-relationship:

   • Retrieves relationships based on filters, such as get-relationship -node_id node_0800 -type temporal_sequence.

Example JSON Query Logic

To make queries more efficient, here’s how we might structure and execute a typical query:

1. Query Example: Retrieve data for a specific time range, 2024-10-14 08:00 to 2024-10-14 08:30.

   • Step 1: Start at 08:00 checkpoint.
   • Step 2: Traverse forward, retrieving each node until reaching 08:30.
   • API Call Example:

     {
         "command": "get-node",
         "start": "2024-10-14 08:00:00",
         "end": "2024-10-14 08:30:00"
     }
     

2. Relationship-based Query Example: Find volatility correlation nodes linked by correlates_with.

   • Command:

     {
         "command": "get-relationship",
         "type": "correlates_with",
         "attributes": ["High", "Low"]
     }
     

   • This command retrieves relationships based on the attributes and relationship type defined in the policies.

Final Thoughts

This flattened structure, combined with the cardinal nodes, simplifies the JSON file while retaining its flexibility for both time-series data and other structured data. By using this approach:

• Efficient Querying: With cardinal nodes, time-based queries can jump directly to relevant checkpoints, enhancing retrieval efficiency.
• Flexible Schema: You can still add new attributes or relationships, making the AGDB flexible for diverse datasets.
• Scalable Relationships: With structured data stored in a CSV format, you maintain scalability while ensuring that AGNs/AGDBs can handle complex relationships.

Let’s proceed with this approach, refining the query logic and API commands to ensure it covers your use case fully.