neo4j a grafove databazy :: Remember, remember, the velvet November


node:	neo4j a grafove databazy
template:	3
parent:	Software
owner:	ulkas
viewed by:
created:	25.07.2016 - 10:20:03

cwbe coordinatez:
101
63540
63542
8188908

ABSOLUT
KYBERIA

permissions
you:	r,
system:	public
net:	yes

nihil	0
toxygen	0
week	0
nudzo	0
ventYl	0
ulkas	0
Angus MacGyver	1
sunrise	1
TomX	1
drakh	1
mattoni	1
s7	2
jinx	3
vygidor	3
oolerin	4
Jay	4
ddd	4
ianus	4

https://neo4j.com/

najpouzivanejsia grafova databaza sucasnosti.
zakladnym principom grafovych databaz je rozbitie dat na zakladne jednotky. oproti SQL databaze to znamena vylucenie tabuliek ako takych, a len drzanie samostatnych riadkov a ich buniek ako datovych jednotiek v jednej velkej akokeby tabulke.

pri dotazoch sa potom nedohladava konkretna tabulka, ale vzdy len startovaci bod, od ktoreho chcem dostat relevantne data. v zmysle ryzhlosti to znamena, ze ked pri SQL sa musi dana tabulka (a viac tabuliek) nacitat a drzat kompletne v pamati (aj ked vacsina dat z tabulky bude ignorovana), u neo4j sa najde ten jeden konkretny bod, a potom uz len data najblizsie k nemu podla dotazu.

v praxi to znamena, ze je to vhodne ako rozumny storage pre socialne siete, pre kyberku, pre vztahy medzi slovami v ramci nejakeho jazyka, pre data mining ak chceme hladat vztahy medzi entitami, pre vsetko, co akokolvek javi nejaku vztahovu siet.
nie je to vhodne pre fulltext data, ako napriklad indexovanie clankov atd. ako priklad u kyberky by to znamenalo, ze vsetky nody a uzivatelia sa drzia v neo4j, ale konkretny obsah nod by bol nacitavany z ineho uloziska.

0000010100063540000635420818890809301107

https://ladybugdb.com/

0000010100063540000635420818890808379966

https://fielddb.github.io/

An offline/online field database which adapts to its user's terminology and I-Language.

It is fundamentally an app written in 100% Javascript which runs entirely client side, backed by a NoSQL database (we are currently using CouchDB and its offline browser wrapper PouchDB alpha). It has a number of webservices which it connects to in order to allow users to perform tasks which require the internet/cloud (ie, syncing data between devices and users, sharing data publicly, running CPU intensive processes to analyze/extract/search audio/video/text). While the app was designed for "field linguists" it can be used by anyone collecting text/audio/video data or collecting highly structured data where the fields on each data point require encryption or customization from user to user, and where the schema of the data is expected to evolve over the course of data collection while in the "field."
https://www.lingsync.org/

0000010100063540000635420818890808341396

In its continued efforts to make Azure a platform that appeals to the widest range of developers possible, Microsoft announced a range of new features at Build, its annual developer conference. Many of the features shown today had a data theme to them. The most novel feature was the release of Cosmos DB, a replacement for, or upgrade to, Microsoft's Document DB NoSQL database. Cosmos DB is designed for "planet-scale" applications, giving developers fine control over the replication policies and reliability. Replicated, distributed systems offer trade-offs between latency and consistency; systems with strong consistency wait until data is fully replicated before a write is deemed to be complete, which offers consistency at the expense of latency. Systems with eventual consistency mark operations as complete before data is fully replicated, promising only that the full replication will occur eventually. This improves latency but risks delivering stale data to applications. Document DB offered four different options for the replication behavior; Cosmos DB ups that to five. The database scales to span multiple regions, with Microsoft offering service level agreements (SLAs) for uptime, performance, latency, and consistency. There are financial penalties if Microsoft misses the SLA requirements. Many applications still call for traditional relational databases. For those, Microsoft is adding both a MySQL and a PostgreSQL service; these provide the familiar open source databases in a platform-as-a-service style, removing the administrative overhead that comes of using them and making it easier to move workloads using them into Azure. The company is also offering a preview of a database-migration service that takes data from on-premises SQL Server and Oracle databases and migrates it to Azure SQL Database. Azure SQL Database has a new feature in preview called "Managed Instances" that offers greater compatibility between on-premises SQL Server and the cloud variant, again to make workload migration easier.
https://arstechnica.com/information-technology/2017/05/azure-adds-mysql-postgresql-and-a-way-to-do-cloud-computing-outside-the-cloud/

0000010100063540000635420818890808317649

Pred par rokmi som pracoval na projekte bitemporalnej grafovej/objektovej databazy. Vkuse sme sa stretavali (a vlastne sa stale stretavam) s projektami, ktore su postavene na relacnych databazach a tento model si beru aj do navrhu aplikacie a je to uplne nahovno.

Neslo to ani cestou objektovych databaz, kde drvivu vacsinu prace spravi engine, ale za cenu toho, ze je treba objekt ulozit v znamom formate, ani cestou noSQL, kde vsetku robotu urobi aplikacia.

Bol to svojim sposobom experiment, Dali sme dokopy noSQL BLOB ACID-compliant storage rozsireny o podporu temporality (bitemporalny storage a solvovanie temporalnych constraintov nad tymto storage-om), pridali podporu vytvarania "ofarbenych" prepojeni a nad tym sme postavili mechanizmus grafovych temporalnych constraintov pre kontrolu ofarbeni (a ak map pamat neklame aj hierarchickych typovych tagov). Komunikacne rozhranie bol SQL-podobny jazyk modifikovany pre potreby grafovo orientovanych dotazov. Zhruba to vedelo definovat typove tagy (classy) vratane dedicnosti, definovat ofarbenia, constrainty pre kombinaciu ofarbenia a typov v case, CRUD dotazy a vyhladavanie. Kedze sme nechceli, aby databaza vedela cokolvek o vnutornostiach ulozenych dat (pretoze to znizuje rychlost a limituje flexibilitu) tak bolo vyhladavanie postavene grafovo a musela na nom participovat aplikacia. Vedeli sme vytiahnut nejaky podgraf na zaklade filtra vlastnosti, ziskat kliku, rozdiely grafov a take prkotiny. Zda sa mi, ze sme nejako mali vymyslene aj lexikalne hladanie, ale uz sa nepamatam, ako. Neviem ci na to nebol do aplikacie callback.

Bohuzial v stadiu tesne pred sparovanim backendu a frontendu zomrelo financovanie a cely vyvojovy tim bol rozpusteny a vypusteny, takze sa nikdy nedozviem, ci by to bolo zivotaschopne. Sem-tam uvazujem nad tym, ze by som vyzobral zdrojaky a nejako to dotiahol ako open-source.

Shitty life is like radiation. You can sustain it for long time if daily doses are small.

000001010006354000063542081889080831764908321954

heh, dnes som narazil na jeden google projekt:
https://opensource.google.com/projects/badwolf
https://github.com/google/badwolf

00000101000635400006354208188908083176490832195408322371

na ten si spominam, ale uz ti nepoviem, preco som sa po jeho studiu rozhodol, ze je aj tak dobry napad napisat si vlastnu databazu :)

Shitty life is like radiation. You can sustain it for long time if daily doses are small.

000001010006354000063542081889080831764908317718

https://arxiv.org/abs/1604.08568
https://github.com/SocioPatterns/neo4j-dynagraph/wiki/Representing-time-dependent-graphs-in-Neo4j

btw, o kolko dat sa jednalo?

00000101000635400006354208188908083176490831771808317728

ten paper este v tej dobe neexistoval (2011 / 2012) :) dokonca tusim, ze v tej dobe temporalitu do tej miery nemala implementovana ziadna relacna databaza (bitemporalne tabulky s temporalnymi constraintami tusim nevie ani orekl mozno ani dnes). existovali akurat specifikacie pre jazyk TSQL92.

Netrufnem si odhadnut aka by bola skalovatelnost, keby sa frontend sparoval s backendom. Performance schedulera bola velka neznama. Datovy storage bol Berkeley DB a pokial viem, najvacsim performance bottleneckom tam bolo solvovanie temporalnych constraintov (pretoze to je NP uplna mrska).

Ak by si velmi chcel, tak najdem diplomovku a skusim sa pozriet, ci v nej mam nejake performance testy solvera. Nasim cielom bolo na to nasadit system na spravu a monitoring inteligentnej budovy vratane storage-u prevadzkovych logov, takze zaznamov by boli radovo statisice.

Shitty life is like radiation. You can sustain it for long time if daily doses are small.

0000010100063540000635420818890808317649083177180831772808317743

ak pod temporalnymi constraitmi myslis kliku a tak, tak kopec tych veci sa pre ohranicene velkosti sa to uz vie robit celkom rychlo, jeden smer sa vola sa to fixed-parameter tractability, aj ked teda zrovna klika tam nespada :)

000001010006354000063542081889080831764908317718083177280831774308317791

Velmi zjednodusene: temporalne constrainty su constrainty pre casove vztahy (medzi zaznamami). ucebnicovy priklad temporalneho constraintu je, ze ty sa nemozes narodit skor ako sa narodil tvoj otec a neskor ako by zomrel (ak zanedbame 9 mesiacov tehotenstva). Vypoved v praci nemozes dostat skor, nez si do nej nastupil, atd.

Temporalna databaza je potom taka databaza, ktora ma k zaznamu priradeny aj interval v ktorom je zaznam platny. Z principu potom ide o databazy verzovane, pretoze pre rozne intervaly mozes mat rozne verzie zaznamu.

Bi-temporalna databaza je potom taka databaza, ktora ma timestampy 2. Nie len interval pre platnost entity, napr. ze ty ako osoba si existoval od svojho narodenia a existujes do svojej smrti. Ale aj interval pre platnost zaznamu (t.j. ze zaznam bol vytvoreny v den, ked ti dali vystavit rodny list, ... a bol platny kym sa v nom nespravila zmena).

Tu sa veci zacinaju celkom zaujimavo komplikovat, pretoze skala vztahov je celkom siroka. Napr. mozes chciet vynutit, aby medzi intervalmi platnosti dvoch po sebe iducich verzii zaznamu pre tu istu entitu bola nulova medzera, ale nie je to vseobecne nutne pravidlo. Naopak nie je ziaduce, aby v jednom case boli platne 2 verzie zaznamu. Ale moze sa stat, ze existuju prekryvajuce sa intervaly existencie dvoch roznych entit, atd.

Fundamenty mas potom na seba navrstvene nasledovne:

Na spodku mas storage engine, u nas to bolo BDB, takze ACID compliant s perspektivou clusteringu.

Nad tym je vrstva pridavajuca podporu pre bi-temporalne zaznamy, constraint recorder (ten je v podstate interne tiez len privatna tabulka v ramci storage enginu) a constraint solver. Volanie constraint solvera je automaticke pomocou callbackov pri CRUD operaciach nad datami.

Nad tym je postavena vrstva implementujuca grafovo/lean-objektovo orientovany pristup k datam. Tu su definovane typove tagy a ich hierarchia (opat ulozene ako privatna tabulka v ramci enginu, potencialne temporalna - takze je mozne verzovat aj typovy system grafovej databazy!), ofarbenia hran (tiez potencialne temporalne) a constrainty pre konstrukciu grafu (opat s podporou temporality, navyse tu aj na urovni definicie constraintu, takze bolo mozne zachytit vztah casu platnosti / existencie zaznamov vo vztahu k hrane).

Uplne na vrchu bol parser SQL-like jazyka, scheduler a sietova vrstva. Jazyk mal "DDL" na typovy system, system ofarbenia hran a temporalne constrainty. "DML" pracoval s timestampovanymi BLOBmi (takze DB nevidela dovnutra zaznamu, ale vedela rozsah platnosti) a mal podporu pre robenie rezov.

Takze vo vysledku bolo mozne nadefinovat typovy system pre data v databaze, system ofarbenia hran a definovat pravidla, ktore umoznovali zachytit temporalitu vztahov.

Co taka temporalna pacmaga dokaze out of box a je celkom zaujimave: Ak sa databazou zacnu robit rezy. Rez je v podstate to, ak sa nebudes pytat na vztahy medzi zaznamami poslednej znamej verzie pre aktualny timestamp, ale bud budes cestovat v case po verziach zaznamov (co si databaza o tomto svete myslela pred rokom?) alebo po realnom case (ako vyzeral svet pred rokom?). Bi-temporalna databaza vie robit rezy v oboch dimenziach naraz. Toto bolo implementovane na vrstve pridavajucej temporalitu, takze z pohladu grafoveho layera to bolo transparentne (e.g. nemal by byt vykonnostny rozdiel medzi dotazom na poslednu znamu verziu 'obrazu' skutocneho sveta a dotaz na nejaky snapshot z minulosti).

Bi-temporalne rezy v relacnych databazach bez podpory temporalnych constraintov casto dost skripu, pretoze nie je velky problem najst "deravy" rez, ale pri grafovych databazach to IMHO funguje lepsie.

V ~~prvej~~nultej verzii sme myslim nemali aktivne zapnutu podporu verzovania typovych systemov ani constraintov, pretoze sme chceli vediet, ci to vobec poriadne funguje, ale perspektiva na to tam bola (a z hladiska implementacie slo len nastavit storage pre constrainty ako temporalny a pridat podporu do parsera.

Shitty life is like radiation. You can sustain it for long time if daily doses are small.

00000101000635400006354208188908083176490831771808317728083177430831779108317906

haluz, ja osobne by som to vrstvenie robil uplne opacne, db engine by obsahoval vsetky zname data a logika temporality by bola v aplikacnej casti na urovni kodu.
tj, vzdy by sa robila vseobecna query na zobratie subgrafu (aj so vsetkymi verziami entity) a az priamo v kode aplikacie by sa filtrovalo na temporalitu ci bi-temporalitu.

0000010100063540000635420818890808317649083177180831772808317743083177910831790608318044

tym ti ale typicky v case rastie objem dat, ktore bude aplikacia na rovnaku kveru dostavat proporcionalne s tym, aky bude traffic na databaze.

plus poziadavka na vytiahnutie vsetkych multiverzii rovnakeho zaznamu v domene transakcneho casu nie je uplne typicky workflow. v domene realneho casu je to workflow pomerne zriedkavy a grafove operacie sa nad tym dost dobre robit nedaju, pretoze integrita skrz verzie zaznamu nie je garantovana.

ale ano, existuju riesenia pre podporu temporaliy (zvacsa v relacnych databazach), ktore funguju podobnym sposobom.

Shitty life is like radiation. You can sustain it for long time if daily doses are small.

0000010100063540000635420818890808300954

https://github.com/BloodHoundAD/BloodHound/wiki

0000010100063540000635420818890808300738

AllegroGraph is a closed source triplestore which is designed to store RDF triples, a standard format for Linked Data.[2] AllegroGraph is currently in use in Open source projects,[3] commercial projects[4][5][6][7] and Department of Defense projects.[8] It is also the storage component for the TwitLogic project[9] that is bringing the Semantic Web to Twitter data.[10]

AllegroGraph® is a modern, high-performance, persistent graph database. AllegroGraph uses efficient memory utilization in combination with disk-based storage, enabling it to scale to billions of quads while maintaining superior performance. AllegroGraph supports SPARQL, RDFS++, and Prolog reasoning from numerous client applications.
http://franz.com/agraph/allegrograph/
http://allegrograph.com/

0000010100063540000635420818890808300732

http://orientdb.com/orientdb/

OrientDB - The World's First Distributed Multi-Model NoSQL Database with a Graph Database Engine

000001010006354000063542081889080830073208300735

OrientDB is the first Multi-Model DBMS with Document & Graph engine. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing, Reactive Queries and has a small memory footprint. OrientDB is licensed with Apache 2 license and the development is driven by OrientDB LTD and a worldwide Open Source community. http://orientdb.com
https://github.com/orientechnologies/orientdb

0000010100063540000635420818890808300729

https://www.arangodb.com/
Production ready highly available Multi-Model NoSQL database

000001010006354000063542081889080830072908300730

ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.
https://github.com/arangodb/arangodb

0000010100063540000635420818890808300727

ulkas 15.02.2017 - 09:36:55 , level: 1, UP NEW

JGraphT

a platform for applying graph algorithms without a persistence layer.
http://jgrapht.org/



directed and undirected graphs.

graphs with weighted / unweighted / labeled or any user-defined edges.

various edge multiplicity options, including: simple-graphs, multigraphs, pseudographs.

unmodifiable graphs - allow modules to provide "read-only" access to internal graphs.

listenable graphs - allow external listeners to track modification events.

subgraphs graphs that are auto-updating subgraph views on other graphs.

all compositions of above graphs.

https://github.com/jgrapht/jgrapht

0000010100063540000635420818890808300721

ulkas 15.02.2017 - 09:31:47 , level: 1, UP NEW

JANUSGRAPH - Distributed Graph Database

JanusGraph is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster. JanusGraph is a transactional database that can support thousands of concurrent users executing complex graph traversals in real time.

In addition, JanusGraph provides the following features:



Elastic and linear scalability for a growing data and user base.

Data distribution and replication for performance and fault tolerance.

Multi-datacenter high availability and hot backups.

Support for ACID and eventual consistency.

Support for various storage backends:

Apache Cassandra

Apache HBase

Oracle BerkeleyDB

Support for global graph data analytics, reporting, and ETL through integration with big data platforms:

Apache Spark

Apache Giraph

Apache Hadoop

Support for geo, numeric range, and full-text search via:

ElasticSearch

Solr

Lucene

Native integration with the TinkerPop graph stack:

Gremlin graph query language

Gremlin graph server

Gremlin applications

Open source under the Apache 2 license.

https://github.com/JanusGraph/janusgraph/
http://janusgraph.org/

0000010100063540000635420818890808300718

Graph Engine
= RAM Store + Computation Engine + Graph Model
Graph Engine (GE) is a distributed in-memory data processing engine, underpinned by a strongly-typed RAM store and a general distributed computation engine.

The distributed RAM store provides a globally addressable high-performance key-value store over a cluster of machines. Through the RAM store, GE enables the fast random data access power over a large distributed data set.

The capability of fast data exploration and distributed parallel computing makes GE a natural large graph processing platform. GE supports both low-latency online query processing and high-throughput offline analytics on billion-node large graphs.
https://www.graphengine.io/

https://github.com/Microsoft/GraphEngine

0000010100063540000635420818890808300716

Trinity is a general purpose distributed graph system over a memory cloud. Memory cloud is a globally addressable, in-memory key-value store over a cluster of machines. Through the distributed in-memory storage, Trinity provides fast random data access power over a large data set. With the power of fast graph exploration and distributed parallel computing, Trinity supports both low-latency online query processing and high-throughput offline analytics on billion-node scale large graphs.
https://www.microsoft.com/en-us/research/project/trinity/
https://www.microsoft.com/en-us/research/publication/trinity-a-distributed-graph-engine-on-a-memory-cloud/

0000010100063540000635420818890808300712

Sometimes the relationships between the data you've gathered are more important than the data itself. That's when a graph processing system comes in handy. It's an important but often poorly understood method for exploring how items in a data set are interrelated. Microsoft's been exploring this area since at least 2013, when it published a paper describing the Trinity project, a cloud-based, in-memory graph engine. The fruits of the effort, known as the Microsoft Graph Engine, are now available as an MIT-licensed open source project as an alternative to the likes of Neo4j or the Linux Foundation's recently announced JanusGraph. Microsoft calls Graph Engine (GE) as "both a RAM store and a computation engine." Data can be inserted into GE and retrieved at high speed since it's kept in-memory and only written back to disk as needed. It can work as a simple key-value store like Memcached, but Redis may be the better comparison, since GE stores data in strongly typed schemas (string, integer, and so on). How does all this shape up against the leading open source graph database, Neo4j? For one, Neo4j has been in the market longer and has an existing user base. It's also available in both an open source community edition and a commercial product, whereas GE is only an open source project right now

0000010100063540000635420818890808243393

http://neo4j.com/graphgist/8173017/

In this Graph Gist, I’m using k-NN with cosine similarity ^[7]^ as the similarity metric to calculate movie recommendations. I wanted a fun data set, so I asked people on Twitter and bothered a few people via email to fill out this form. Using their movie ratings, I’ll calculate the cosine similarity between each person. I’ll then calculate movie recommendations for a person’s unrated movies based on an average rating from that person’s k-nearest neighbors. My methodology will be explained in detail throughout.

0000010100063540000635420818890808218662

GraphQL is a query language for APIs. Its purpose is to provide a complete description of the data in your API, which can then be used by clients to ask for specific data from the API. A GraphQL query is a string that is sent to a server to be interpreted and fulfilled, which then returns JSON back to the client. The developers differentiate the way GraphQL works as handling data not in terms of resource URLs, secondary keys, or join tables; rather as a graph of objects and the models ultimately used in apps like NSObjects or JSON.
http://www.i-programmer.info/news/90-tools/10095-graphql-leaves-tech-preview.html

0000010100063540000635420818890808190019

dnes akcia free book:
https://www.packtpub.com/packt/offers/free-learning

Use R's powerful graphing capabilities to design and create professional-level graphics
Learn how to use Base R to analyze your data and generate statistical graphs
Create attractive graphics using advanced functions such as qplot and ggplot for research and analysis
A step-by-step guide, packed with examples using real-world data sets that can prove helpful to R programmers

stiahnut sa to da tu:
https://kyberia.sk/id/8190026

0000010100063540000635420818890808188896

https://neo4j.com/blog/neo4j-world-healthcare-part-2

0000010100063540000635420818890805696454

(nejen) Moje poznamky o praci s grafama v PgSQL.
Nechci si je syslit pro kyberia.cz, treba nekoho budou inspirovat...
A naopak pokud je mezi vami PostgreSQL odbornik, nebojte se inspirovat me :-)

Kyberka uz davno prerostla detskou ohradku MySQL a je potreba se porozhlidnout po necem novym (to na kyberia.cz taky delame). Ja osobne sem vyzkousel spoustu NoSQL databazi (docela me zaujal Riak a myslim si, ze ma do velmi vzdaleneho budoucna potencial), probihali dlouhe diskuze a hadky u piva, zbourali sme nekolik hospod a jeden kostel bylo po nasi diskuzi nutne znovu vysvetit, ale nakonec jsme zjistili, ze vsechny tyhle DB rychlokvasky zatim funguji i pres neutuchajici a temer psychedelicke nadseni svych vyvojaru spise teoreticky nez v praxi a tak jsme zkoncili u stareho dobreho osvedceneho PostgreSQL.

PostgreS forum: https://kyberia.sk/id/4999650/

axone SQL
axone recommender systems / ML frameworks
axone forumz