Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Last updated: 2024-09-23 12:08:18.202967 File source: link on GitLab
This document is aimed to outline, describe and specify (and work iteratively on) the general NuNet model of computing. It expands on the NuNet initial white-paper and and conceptual ideas outlined there taking into account implementation work done so far as well as additional research done [or in progress] at NuNet as well as broader SingularityNET ecosystem so far.
The immediate goal is to design [as much as possible] semi-formal specification of the computing model that NuNet is implementing for guiding the technical work done by the core development team (see open development board) as well as broader developer community (join our discord server to participate).
Besides, above and around the immediate goal of design specification, we attempt to describe and specify as much as possible the conceptual basis for specifications in the light of the overall vision of NuNet as a global economy of decentralized computing.
Finally, this document / yellow-paper attempts to show relations between all somewhat loosely technically connected protocol specifications, design patterns, workflow constructions, API specifications, languages, research areas, etc. More often than not, these areas can be related only via reference to the conceptual vision of the computational ecosystem that NuNet aims to help bootstrapping.
This yellow-paper is intended to be part of living public technical documentation of NuNet. While we do not expect main aspects of the content of this document to change or be updated too frequently, it is still subject to change and update along the path of development. The paper is broken in seven parts, some of them more developed than others. These are (in the order of importance):
A. Ecosystem introduces a basis of the general business model, as much as relevant for the technical architecture;
B. Computing model explains conceptual architecture of the platform with relations to current implementations and technical decisions;
C. Observability Framework enables to monitor and analyze computational events;
D. Tokenomics is the interoperability layer between computational engine and crypto-micro-payments layer, including the micro-payments itself;
E. Platform Development Model explains use-case based platform development model and provides details related to the general business model;
F. Necessary architectural aspects are technical aspects that are needed for the overall NuNet's technical vision to be achieved;
G. Advanced architectural aspects may not be necessary for achieving the technical vision but are important for larger context within SingularityNET ecosystem.
maintained by @kabir.kbr; please assign all merge requests for edit of this page accordingly;
Welcome to NuNet's Platform Yellowpaper. This document outlines the evolving design and specifications of NuNet’s decentralized computing platform. Building upon the initial concepts from our whitepaper, it integrates lessons learned from ongoing implementation efforts and research within both NuNet and the broader SingularityNET ecosystem.
Intended as part of NuNet’s living documentation, this Yellowpaper will be periodically updated to reflect the latest developments and insights. It is organized into seven sections:
Ecosystem
Computing model
Observability Framework
Tokenomics
Platform Development Model
Necessary architectural aspects
Advanced architectural aspects
Last updated: 2024-09-23 12:08:18.628069 File source: link on GitLab
The concept of global economy of decentralized computing rests on surprisingly large body of research done since the dawn of modern computing era (e.g. @RobustComposition). The idea, models and formalisms of decentralized computing and ecosystems of autonomous communicating computing processes almost seem as an alternative path of the computing industry / universe which has never ceased to develop, but was somewhat overshadowed by the mainstream 'centralized' computing paradigms, models and philosophies. During at least half-decade of research and industry development it seems that most of the pieces of the puzzle are in place for this path to resurface.
NuNet is primarily based on the Actor model of computation [@hewitt_actor_model_2010]. The major premise and background of Actor model of computation (also shared by the object-capability model), is addressing the complexity of modern system designs not by seeking universal solutions to ever increasing coordination problems, but by providing maximum autonomy for smallest components of the system and providing abstraction mechanisms for crafting diverse solutions adapted to the needs of many applications. In the context of economy of interacting computer processes (i.e. system components) -- the context in which NuNet is operating -- this premise is a necessary condition.
Second major component of NuNet's conceptual design is the Graph Computing, which allows for interacting processes to access the topology of the network (into which they are embedded) via local neighbors connections. Graph Computing obviously has roots in graph theory, but extends it to practical contexts. We extend and modify it further for our purposes.
The unification of Actor model and Graph computing frameworks produce the concept of of 'open-ended decentralized computing' which is the conceptual inspiration for NuNet's computing model. As important as this concept is for our NuNet's computing model, detailed description / discussion of it is outside the scope of this document. Please refer to prior research on this topic for more details. Actor model and Graph computing frameworks contain fairly large bodies of computer-scientific knowledge as well as adaptations for NuNet. They are detailed in separate sections:
B.1. Computing model / Actor model of computation
B.2. Computing model / Graph computing
Following the expression of the global network of computing, we can [approximately] map each identified aspect with its implementation within the NuNet computing model and its main software component - device-management-service.
All computational components are implemented both as actors and as graph vertices which allows to leverage both flexibility and asynchrony of the actor model of computation and indirect communication via evolving shared data structure enabled by graph computing. In NuNet platform ontology, Actors are Nodes and Allocations, which implement both Actor interface and Graph interface for implementing the logic of the computing model.
Proposed interfaces (currently discussed and updated for integration into the next release of the device-management-service):
Node, implementing Actor;
Allocation, implementing Actor
Graph -- a placeholder for the graph traversal logic;
Executables or programs are well defined processes (AI agents, programs, container images...), which, when executed, take specific $data$ as input and produce some $data$ as output (which should be at least well formed). Note, that programs will be executed on a concrete hardware (machines) provided to the NuNet network by independent compute providers, which is the main focus of NuNet decentralized deployment and orchestration mechanism.
Major related device-management-service packages:
executor, abstracting executable interface from the containerization frameworks and executors themselves; current implementation contains two sub-packages:
docker for running docker containers;
firecracker for running firecracker virtual machines;
orchestrator, implementing search and match compute resources with posted job requirements and sending the job to fitting hardware resource provider in the network;
jobs takes care of the job management on each Node (machine);
Data as input and output data structures as defined above; note, that $data$ can actually be a program, which can be executed and then it becomes an $executable$, therefore the boundary is not definable a-priori, but is resolved dynamically -- which is important to do for our model;
Current proposals related device-management-service packages / functionalities:
storage package;
note by @kabir.kbr 2025-05-15: updated with references to current interface proposals until here
State is the compute state at every moment of execution; in some sense this is also can be considered $data$, when, say, a computation is stopped, a snapshot is saved and then restarted at an other place. However, the way we will be treating this two kinds of data is different, therefore it makes sense to distinguish them in the model;
When reasoning about state of decentralized computing platform / engine, it is useful to separate three distinct aspects:
Private state of an Actor which is essentially a state of the algorithmic implementation of business logic / program that an actor is running and is closest to the conventional understanding of program / execution state. While actor model allows for Actors to maintain state (obviously), it however constraints other parts of the actor system to access that state -- therefore it is private for the running Actor itself and cannot be shared outside its boundaries. In order to enforce this principle, NuNet is keeping clear Actor implementation boundaries in form of interfaces and their implementations. NuNet, however, considers the notion of 'persistent execution state' in its design and implementation, which in this context is a mechanism that allows to preserve snapshots of the private Actor space and move them across different compute environments - which does not contradict the constraints of the model.
Global state refers to the notion of a execution state of the whole platform. We do not have this notion at NuNet and do not use it in design or implementation considerations. The notion of global state of any reasonably complex system (also computing engine and NuNet platform as a whole) is quite problematic on at least epistemological grounds, since one needs to posit a global omniscient observer in order to reason in terms of this concept. For architecture design of NuNet platform it is suffice to mention that positing global state and global observer contradicts Actor model and is therefore not used.
Job orchestration state (for the lack of a better word in the context of NuNet platform architecture) refers to the state of a subset of compute processes within the platform, which directly interact with each other and may need to coordinate their actions in order to achieve the goal of a complex job and have some knowledge of each other's state. In order to comply to the Actor model, this shall be done only via message passing mechanism and strictly respecting the privacy of Actor states.
Proposed interfaces related to the notions (although no currently proposed interfaces / implementation) is directly related to state management at NuNet:
Resources are simply hardware resources (memory, processing power, GPU/CPU devices, sensors, actuators...) that are needed for executables to process the data; note, that a special case of 'processing data' is a a sensor which actually collects data from environment and executes effects on the larger environment. In NuNet design, we also use the notion of Capability, which also encompass hardware resources, in order to consider other properties of the execution environment (like libraries, containerization system, etc.).
Related [currently proposed] interfaces and types:
Connections are a special resources within the computing system, which is may be defined as resources that is needed for data to travel from one executable to another; these resources have their own special properties (speed, bandwidth, reliability, price...) and, while in centralized computing they are usually taken for granted, in for reasoning about decentralized computing, they necessarily have to be elevated to the first class citizens of computational ecosystem.
In order to realize the notion of computational reflection, which is required to implement 'compute over data' aspect of a decentralized computing system, each Actor should be able to access data and reason about the properties of links and connections between itself and other hardware and software actors of the system.
Related proposed interfaces / types;
Communication are the actual processes that are using $connections$. In our model of computation, all communications are defined as a set of well defined messages, therefore to some extent $communications$ can be considered equivalent to $messaging$
Related proposed interfaces:
Payments is the necessary component of an economics system, or, in our case, a tokenomics system; the design and logic of the tokenomics system itself is discussed in detail in a separate section (see D. Tokenomics), which is as important as the model of computation; for defining the economy of computation at the present level of abstraction it is enough to say that the system should be able to provide a layer that will enable every actor in the system to engage into economic/tokenomic transactions;
Providers are the soco-economic entities (legal or natural persons) that are owners of the components of the computational universe. Since we are designing the economy of computing which is grounded in the social and economic fabric of society as a whole, we necessarily have to include this component into the model; Besides that, the system has to provide interfaces that are necessary for providers to manage their resources.
Users are are also socio-economic entities which are using the computational universe to achieve their goals in the larger socio-economic environment. The main category of users are application developers which will run their application backends on NuNet;
Related resources:
NuNet Framework Stakeholder Mapping - high level stakeholder mapping concepts published as a research blog post
Low lever stakeholder mapping maintained as gitlab repo and shall be updated constantly;
This section summarizes the concept of the NuNet computing model. It is best to read it after referring to subsections on B.1 Computing model/Actor model and B.2 Computing model/Graph computing.
The architecture for open-ended decentralized computing unites the actor model and graph computing into one framework:
The actor model enables massively scalable implementation of a population of independent and heterogeneous computational processes, and their custom behaviours. Actors can have wildly different computational complexities and resource requirements. Computational processes of actors communicate with each other by passing mes- sages which encapsulate commands and data structures. The totality of all traces of messages constitute a dynamic message-passing graph – a shared data structure – where every actor is also a vertex.
Graph computing enables actors to access and change this graph by executing vertex-centric graph traversals based on their unique perspectives and custom behaviours. The message-passing graph, while best reasoned about in terms of a logical structure, is actually scattered across local memories of actors and does not constitute a single object. Therefore, it is not accessible directly, but only via the fact that an actor which accesses it is itself a part of the graph and can issue traversals from its location and via its links with immediate neighbours. Most importantly, the notion of message and graph traversal are different aspects of the same communication event. This fact alone unites the actor model and graph computing into one framework by allowing actors to specify legal paths of messages. In this way messages (i.e. information and control flow) propagate not randomly or directly, but depending on internal logic and the local graph structures, which develop and change because of other messages.
Actor model allows the operation of the framework to be radically decentralized and asynchronous, while graph computing allows for local computations. The framework enables the bottom-up emergence of computation graphs to be realized as persistent assemblages (sub-graphs) of computational processes and, in general, stigmergic computing and progressive determination. This is a high-level description of the architecture of open-ended decentralized computing in a nutshell. Figure below depicts the architecture by showing the actor system and graph computing engine as integrated aspects.
Last updated: 2024-09-23 12:08:19.769605 File source:
It may appear that platform development model does not belong in the technical-yellow-paper context, but the technology and especially current status if its development considering the overall vision is related to the process. It will be explained here.
TBD
See .
See .
Last updated: 2024-09-23 12:08:19.534040 File source:
In order to support a system in which actors would autonomously yet collectively search for assemblages leading to "better" configurations, as required by different tasks and problems, we need the notion and implementation of *computational economy. Such an economy would define the costs of physical computing resources and data transfer that actors would need to take into account when forming assemblages – or larger computational units. Moreover, in order to describe such an economy, notions of actors’ location, mobility and execution context should be defined in a way that could be accessed and reasoned by the actors themselves. This requires an extension of the otherwise location-transparent actor model with a global yet dynamic topological structure that can be accessed and modified by actors.
We implement the basis / fundamentals of tokenomic model by designing and implemented the abstract interoperability layer that allows computing model (and deployment layer) to interact with any micro-payments infrastructure that is required for business needs / use-cases of different user classes participating in the network.
The basic abstract interface and its implementation is developed within package;
TBD
TBD
Last updated: 2024-09-23 12:08:19.123243 File source: link on GitLab
Graph computing is a quickly developing new paradigm of computing which leverages network science and graph theory to represent data, programs and their interactions in distributed computing environments. It is a large set of technologies and a way of modelling the world in terms of entities (nodes) and their relations (semantic or analogous associations, causality, information, control flows, mapping between h hardware and software, etc.). The graph computing field is very diverse and evolves quickly by featuring new and augmented ways to computing on theoretical and practical levels.
Graph computing concepts allow to express dynamic topology of the computation network in terms of links between actors. It also allows for implementation of location awareness of each actor in terms of access to global graph topology vial local traversals.
Actors can know the mail addresses of other actors, which makes them acquaintances, or neighbours and is a basis of the local network topology. Mail addresses can be contained in messages, which allows for a dynamic actor interconnection topology, where actors can connect directly to each other based on indirectly passed and received messages.
The actor system therefore is a dynamic network structure, where nodes are actors with links to their mutual acquaintances. Using these links, every actor can in principle reach every other actor and learn local or global network structure via distributed message-passing algorithms. Note, that first, such an actor system can be represented a graph form and using graph formalism. Second, in order for any actor to have partial (or, in the edge case, full) representation of the global state of the system, it has (1) to know all other actors and (2) to have a local representation of each actor’s state in its own memory.
Graph traversal is a systematic method of exploring vertices and edges in a graph. Traversals are represented using special purpose graph traversal languages which formalize an abstract description of a legal path through a graph. Graph traversing is then a process of visiting (checking, updating or modifying) vertices and links of a graph, based on the imperatively defined constraints in this language by a user.
The concept is important for us because by expressing NuNet network / platform as a well formed graph structure and implementing interfaces exposing graph traversal logic, we allow each Actor in the NuNet platform (so each Node and Allocation) to execute Turing-complete programs on top of decentralized graph. This is essential for decentralized computing ecosystems in general and for implementing NuNet computing model in particular.
Consider the Web as an example. When a person, program or a mobile application "queries" the web for information, one does not have access to all information residing in all other nodes – servers, web pages, programs or mobile applications – at once. One does not even know, or need to know, how many other nodes exist in the network. Instead, one accesses the network via its local node and follows from link to node to another link selectively and iteratively until reaching the desired node or content. This is a primitive example of vertex-centric graph traversal where each traversal has to start from an existing node in the graph and traverse through its neighbouring links. This operation can be written formally in imperative graph traversal language.
In the full-fledged Graph traversal context, traversers can mutate a graph when walking it – i.e. change properties on links and vertexes, create new vertices (e.g. Nodes or Allocations) or connect them via new links.
Using graph computing concept in the decentralized computing context promises to altogether decouple content, data and executable code from any physical location, server or client implementation. Such a dynamic network of interacting processes is best conceived as a decentralized computing infrastructure in terms of interactive computation, actor model and an ecosystem of emergent computation graphs -- a fluid topology that changes and adapts to the computations performed in the network.
At NuNet such fluid topology is provided by integrating the message-passing framework of the actor model to graph computing. Based on our use-case based platform development model (see E. Platform Development Model).
Prior research at Synthetic Cognitive Development/Architecture for Open-Ended decentralized Computing/Graph Computing, on which this section is largely based.
Last updated: 2024-09-23 12:08:18.920556 File source: link on GitLab
Actor model provides NuNet with ability to express and implement a potentially infinite space of concurrent, non-locking, scalable and fault-tolerant computations; This component is crucial for enabling non-trivial computing in a decentralized (without single point of control, coordination or ownership) network of heterogenous devices and processes communicating via unsafe and unstable network connections (i.e. Internet).
The actor model is a mathematical model of concurrent computation, proposed by Hewitt, Bishop, and Steiger (1973) and further developed by Clinger (1981), Greif (1975), Agha (1986) and others. It is conceptually based on one kind of object – an autonomous communicating actor (processor, agent, etc.) which does not presuppose any representation of primitive data or control structures. Data structures can be programmed or hard-wired and encapsulated or dynamically evolved to each actor separately or to an ensemble.
Actors communicate via immutable messages and in this sense constitute a message-passing framework. Patterns of message passing between actors is the only control structure which defines the functionality and behaviour of a particular actor system as a whole. There are no constraints on message-passing patterns or even requirements for their existence by the lowest level of the actor model. System engineers enforce these constraints via high level design or let them emerge during computation.
An actor itself is a self-contained, interacting, independent component having a well defined behaviour and a mail address. An actor’s behaviour is expressed by a computational process having inputs, outputs and computational resource requirements, including storage. Actors communicate with each other via asynchronous message passing using their mail addresses.
In summary, an Actor is the universal (and the only) primitive of concurrent computation, that can:
receive a message from other actors;
make local decisions / perform computations [possibly in response to messages];
send messages to other actors;
create other actors;
maintain local state (and therefore be able to respond to messages differently);
All computations in the actor model are built by these primitives only, which are necessarily executed on behalf of an actor in an actor system. No primitive can be executed without an actor that executes it.
The main requirements of the model are (rigorously followed in all levels of implementation of NuNet platform):
The fundamental advance of the actor model was the decoupling of a sender from the communications it sends – once a message has been sent, it loses any relation to the sender and is delivered on a best effort basis. This is a sharp difference from other models of concurrent computation, where message sending is tightly coupled to the sender, which synchronously transfers the message to some persistent structure – e.g. buffer, queue, mailbox, channel, broker or server – for temporary storage and retrieval by a reader.
This requirement results is the absence of the global state in the actor model and allows it to embrace inconsistencies inherent in large decentralized systems. Actually, consistency and synchronous execution are special cases within the actor model and have to be explicitly enforced if considered necessary by designers.
In order to prove mathematical properties of actor model, message sending is even not considered an event in the formal representation of the model.
Each actor implement a conventional sequential or parallel computing. Actually, actor model does not constrain what and how actors do computation inside their boundaries. Actors can modify their own private state (or decide to expose to other actors via message communication), but can affect each other only indirectly through messages asking for certain behaviors via remote-procedure-call-type requests;
Actors do not pre-assume anything outside their boundaries, and therefore there is no a-priori trust between actors or, in this sense, in system as a whole; any trust between actors has to be negotiated and established peer-wise. This principle does not prevent from defining external trust providers, which can be used by actors in the network in case they explicitly
There is no global knowledge or global controller that has access to the whole system and therefore any controlling mechanism should be implemented via an actor which needs to establish trust relations with other actors in the system;
Object-capability model [@RobustComposition] conceptually is very similar to actor model of computation but provides additional concepts for reasoning about security and permissions model in the decentralized system with by default 'zero-trust' and and 'end-point security', which are the basic principles of NuNet platform security architecture.
Capability-based security is a concept in the design of secure computing systems (not only decentralized!) where capability is a communicable, unforgeable token of authority, which can be attached to each message / request between peers. This token necessarily includes:
unique key of the issuing actor (e.g. public key);
audience of the message (paths or addresses of the target actors(s));
capability declarations -- declaration of what resources can be accessed;
Importantly for a decentralized computing, object-capability model allows capabilities to be delegated -- i.e. an actor can propagate its capabilities to other actors. A token of authority where capability is delegated includes the proof of delegation chain -- without centralized Access Control List or other centralized authority. Capability-based security is often contrasted with traditional UNIX permissions and Access Control Lists.
Implementation of actor model achieves the following higher level properties of the decentralized computing ecosystem. Some of these properties were listed as computational and functional requirements in NuNet's initial whitepaper 2.0.
A key issue in developing multi-agent systems that the actor model allows us to address at a fundamental level is mobility. The implementation of the model requires computing resources for enabling processing, storage and communication for processes which actors encapsulate, which is precisely the rationale behind NuNet framework.
The physical implementations of such computing resources are by convention called nodes – i.e. servers, processors, computing centres, etc. – that together make a distributed computing infrastructure of cloud infrastructures, sub-networks and eventually a global network. Mobility allows an agent to migrate freely from one such node in a distributed computing environment to another to seek "better" execution environments.
This flexibility can be approached from at least two aspects – a computer-scientific and economic – both stretching the limits of computational systems as usually understood, which we are attempting to do at NuNet.
Customizable execution contexts and computational reflection enable individual actors to have a continuous interaction with their environment to determine available resources, relate to their own state, search for and request additional resources and by that provide evolving resource consumption strategies. These are needed for an agent to specify requirements for negotiation with a hosting node, besides the bare estimation of computational resource requirements – e.g. programming languages, library dependencies, hardware components, containerization engines, etc.
The power of the actor model comes from the exploitation of parallelism, distribution and mobility in ensembles of agents. It promises orders of magnitude greater computational abilities than that of individual actors, which are nothing more than conventional sequential programs. As each problem-solving agent possesses only an incomplete local view of the system and limited computational power, it must coordinate with other agents in order to achieve coherent and globally viable solutions. Since in the actor model there is no global coordinating actor, it has to happen from the bottom-up via synchronization, coherence and functional coordination. In order to provide a minimal basis for coordination in a computational medium such as an actor system, interaction protocols and policies have to be in place. Apart from functional coordination, interaction policies enable the system to deal with failures, which is the norm rather than the exception in distributed and decentralized systems.
Last updated: 2024-09-23 12:08:18.428785 File source: link on GitLab
We distinguish the conceptual/dynamic and technical aspects of the ecosystem into:
Framework
Platform
As a framework, NuNet will enable to position AI processes, interfaces and data optimally within the global network and establish machine-to-machine payment and data streaming channels between diverse constituents. We do not know what will be the precise constituents of the ecosystem in order to keep it open and encourage its evolution, but we can fairly well position them into the constituent classes. NuNet's stakeholder mapping is maintained by:
NuNet Framework Stakeholder Mapping research blog;
Stakeholder mapping sub-project, aimed at constantly updated categories and their description.
Main conceptual components of NuNet platform are Actor model and its relation to network topology via Graph computing concepts. Taken together and related via lower level implementation packages, they constitute NuNet computing model, largely inspired by prior research of open-ended computing. Another two major components of the platform are Observability framework (a.k.a. Telemetry) and Tokenomics. The latter is most directly related to the implementation of decentralized computing economy model, inspired and framed by offer networks research.
Interaction of these four major aspects constitute the technological basis for global economy of decentralized computations.
Technically, NuNet platform is a set of open APIs, communication protocols – and open source software components which expose those APIs and protocols – that enables the global economy of the decentralized computing: the framework that connects compute providers, data providers, AI developers, application developers into a single cloud (see picture above).
For the sake of engineering the economy of interacting computational processes, we define the global network of computations (without much ambition for theoretical generality) as follows:
As mentioned above, the mapping between NuNet framework's stakeholders and platform's components is maintained in nunet/architecture/mapping repository and includes:
Further detalization of these categories and their relations to specific platform packages is provided in the Computing model section.
Last updated: 2024-09-23 12:08:20.159645 File source: link on GitLab
These are the aspects of NuNet which are part of the technology vision within broader SNET ecosystem. They are in a different phase of technology research, requiring much more fundamental research / alignment with protocols and platforms/solutions developed across the SNET ecosystem.
Last updated: 2024-09-23 12:08:19.955960 File source: link on GitLab
We have quite a number of important architectural aspects to consider in NuNet which are not directly related to computing model, but should be approached considering that model and properly aligned.
TBD
Last updated: 2024-09-23 12:08:19.325681 File source: link on GitLab
The nature of open-ended decentralized computing dictates that computing processes cannot be directly observed or controlled from a single point in software architecture. The absence of the requirement of persistent global state and synchronous execution, while allowing for richer and more scalable models of computation, is also difficult to reason about and creates a great deal of confusion as it contrasts with the usual practices of software development, which has possibly contributed to complications in the adoption of the actor model in practical computing and software development. It would be probably fair to say that this confusion is created by the very principle of open-ended decentralized computing, which posits that there is no global observer understanding the 'true and only nature' of the system. However that does not prevent to posit local observers which can reason about some properties of the system and the nature of selected aspects of it. For that purpose, we are building an observability framework.
Therefore, in order to provide observability over computational processes, the NuNet computing model considers two fairly separate subsystems – implementation of the B.1 Computing model and Observability framework. This section considers Observability framework in the context of the overall model.
Computing model enables implementation of all aspects of open-ended de-centralized computing: (1) heterogeneous agent’s logic, (2) actor framework, (3) graph computing engine and persistent network topology (distributed or otherwise);
Observability framework collects information of events happening in the network and performs real-time or post-mortem analysis or visualizations as required by software design requirements and use-cases.
Observability framework is currently implemented by:
device-management-service as a telemetry package, which defines and implements interfaces that are then used by each Node in the network to observe internal events and send them to registered data Collectors (which can be external or internal to the network);
External to the platform, but part of NuNet infrastructure Elasticsearch cluster, which will be by default configured to serve as the main collector of events in the whole network. Initially, in the development environment, the NuNet-run Elasticsearch cluster will be used for collecting traces, metrics and logs for the debugging purposes. Eventually in the production environment, it will be used for network telemetry dashboard as well as data source for reputation system's infrastructure.
Observability framework implemented in NuNet is loosely based on adapting the following prior research Offer networks: a model of decentralized exchange/Software architecture/Monitoring and analysis engine done by SingularityNET as well as learnings from further NuNet's efforts.
Observability framework is related to the notion of decentralized indexing, which is the most abstract conceptual framework to think about that. Indexing is an an essential technical aspect of graph computing and the operation of decentralized networks in general.
Indexing therefore can be seen as a way to introduce specialized supernodes which effectively reduce the diameter of a graph and make information propagation between nodes more efficient. Another aspect of indices in general and search engines in particular is that they provide a mapping between node content and address for routing purposes. Actually, modern search engines introduce a kind of content addressing into the web – often one does not even notice the actual URL of a server or webpage one follows from a search engine’s page.
Qualitatively speaking, search engines and reputation systems and their indexes are also nodes in the web graph, alongside other servers, programs and devices. Yet they have two notable quantitative differences: (1) a much larger degree of connectivity to other nodes (which is the whole point of search engines/reputation systems – i.e. providing a gateway to other nodes that are needed for some business use-case / purpose) and (2) more computing resources (storage space and processing power). These are the supernodes. In network science and graph theory, a supernode is a vertex of a graph that has a disproportionally large number of connections to other vertices.
Graph computing allows any Actor in the network (so in NuNet case Node or Allocation) to construct any use-case specific index it needs for its goals and by that become a 'supernode' in the network, if network participants consider that index important for their own goals. Observability framework shall provide low level technical implementation allowing for any user/stakeholder/actor to create such a supernode in the NuNet platform that would serve purposes of the network efficiency of business needs of one or more class of its users. This is related to the NuNet reputation system infrastructure architecture.
