Merge branch 'master' of github.com:matrix-org/synapse

This commit is contained in:
Kegan Dougal 2014-08-13 15:53:44 +01:00
commit 38c7e92331
13 changed files with 160 additions and 146 deletions

35
MAP.rst Normal file
View file

@ -0,0 +1,35 @@
Directory Structure
===================
Warning: this may be a bit stale...
::
.
├── cmdclient Basic CLI python Matrix client
├── demo Scripts for running standalone Matrix demos
├── docs All doc, including the draft Matrix API spec
│   ├── client-server The client-server Matrix API spec
│   ├── model Domain-specific elements of the Matrix API spec
│   ├── server-server The server-server model of the Matrix API spec
│   └── sphinx The internal API doc of the Synapse homeserver
├── experiments Early experiments of using Synapse's internal APIs
├── graph Visualisation of Matrix's distributed message store
├── synapse The reference Matrix homeserver implementation
│   ├── api Common building blocks for the APIs
│   │   ├── events Definition of state representation Events
│   │   └── streams Definition of streamable Event objects
│   ├── app The __main__ entry point for the homeserver
│   ├── crypto The PKI client/server used for secure federation
│   │   └── resource PKI helper objects (e.g. keys)
│   ├── federation Server-server state replication logic
│   ├── handlers The main business logic of the homeserver
│   ├── http Wrappers around Twisted's HTTP server & client
│   ├── rest Servlet-style RESTful API
│   ├── storage Persistence subsystem (currently only sqlite3)
│   │   └── schema sqlite persistence schema
│   └── util Synapse-specific utilities
├── tests Unit tests for the Synapse homeserver
└── webclient Basic AngularJS Matrix web client

View file

@ -1,16 +1,52 @@
About
=====
Quick Start
===========
Matrix is an ambitious new ecosystem for open federated Instant Messaging and
VoIP[1].
VoIP[1]. The basics you need to know to get up and running are:
Matrix specifies a set of pragmatic RESTful HTTP JSON APIs as an open standard,
providing:
- Chatrooms are distributed and do not exist on any single server. Rooms
can be found using names like ``#matrix:matrix.org`` or
``#test:localhost:8080`` or they can be ephemeral.
- Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
you will normally refer to yourself and others using a 3PID: email
address, phone number, etc rather than manipulating matrix user IDs)
The overall architecture is::
client <----> homeserver <=================> homeserver <-----> client
e.g. matrix.org:8080 e.g. mydomain.net:8080
To get up and running:
- To simply play with an **existing** homeserver you can
just go straight to http://matrix.org/alpha.
- To run your own **private** homeserver on localhost:8080, install synapse
with ``python setup.py develop --user`` and then run one with
``python synapse/app/homeserver.py``
- To run your own webclient:
``cd webclient; python -m SimpleHTTPServer`` and hit http://localhost:8000
in your web browser (a recent Chrome, Safari or Firefox for now,
please...)
- To make the homeserver **public** and let it exchange messages with
other homeservers and participate in the overall Matrix federation, open
up port 8080 and run ``python synapse/app/homeserver.py --host
machine.my.domain.name``. Then come join ``#matrix:matrix.org`` and
say hi! :)
About Matrix
============
Matrix specifies a set of pragmatic RESTful HTTP JSON APIs for VoIP and IM as an
open standard, providing:
- Creating and managing fully distributed chat rooms with no
single points of control or failure
- Eventually-consistent cryptographically secure[2] synchronisation of room
state across a global open network of federated servers and services
state across a global open network of federated servers and services
- Sending and receiving extensible messages in a room with (optional)
end-to-end encryption[3]
- Inviting, joining, leaving, kicking, banning room members
@ -57,74 +93,6 @@ Thanks for trying Matrix!
[3] End-to-end encryption is currently in development
Quick Start
===========
The basics you need to know about Matrix are:
- Chatrooms look like ``#matrix:matrix.org`` or ``#test:localhost:8080``
- Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
you will normally refer to yourself and others using a 3PID: email
address, phone number, etc rather than manipulating matrix user IDs)
- To simply play with an **existing** homeserver (e.g. matrix.org), you can
just go straight to http://matrix.org/alpha, specify a homeserver
(defaults to matrix.org) and sign up and use it. (Sign-up security is
currently work-in-progress)
- To run your own **private** homeserver on localhost:8080, install synapse
with ``python setup.py develop --user`` and then run one with
``python synapse/app/homeserver.py``
- To run your own webclient:
``cd webclient; python -m SimpleHTTPServer`` and hit http://localhost:8000
in your web browser (a recent Chrome, Safari or Firefox for now,
please...)
- For now, register some accounts like ``@testing:localhost:8080`` from
different browsers, join a room like ``#test:localhost:8080`` and have a
play.
- To quickly run a **public** homeserver that can exchange messages with
other homeservers and participate in the overall Matrix federation, open
up port 8080 and run ``python synapse/app/homeserver.py --host
machine.my.domain.name``. Then come join ``#matrix:matrix.org`` and
say hi! :)
Directory Structure
===================
::
.
├── cmdclient Basic CLI python Matrix client
├── demo Scripts for running standalone Matrix demos
├── docs All doc, including the draft Matrix API spec
│   ├── client-server The client-server Matrix API spec
│   ├── model Domain-specific elements of the Matrix API spec
│   ├── server-server The server-server model of the Matrix API spec
│   └── sphinx The internal API doc of the Synapse homeserver
├── experiments Early experiments of using Synapse's internal APIs
├── graph Visualisation of Matrix's distributed message store
├── synapse The reference Matrix homeserver implementation
│   ├── api Common building blocks for the APIs
│   │   ├── events Definition of state representation Events
│   │   └── streams Definition of streamable Event objects
│   ├── app The __main__ entry point for the homeserver
│   ├── crypto The PKI client/server used for secure federation
│   │   └── resource PKI helper objects (e.g. keys)
│   ├── federation Server-server state replication logic
│   ├── handlers The main business logic of the homeserver
│   ├── http Wrappers around Twisted's HTTP server & client
│   ├── rest Servlet-style RESTful API
│   ├── storage Persistence subsystem (currently only sqlite3)
│   │   └── schema sqlite persistence schema
│   └── util Synapse-specific utilities
├── tests Unit tests for the Synapse homeserver
└── webclient Basic AngularJS Matrix web client
Homeserver Installation
=======================
@ -152,6 +120,12 @@ may need to also run:
$ sudo apt-get install python-pip
$ sudo pip install --upgrade setuptools
If you get errors about ``sodium.h`` being missing, you may also need to
manually install a newer PyNaCl via pip as setuptools installs an old one. Or
you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
installing it. Installing PyNaCl using pip may also work (remember to remove any
other versions installed by setuputils in, for example, ~/.local/lib).
This will run a process of downloading and installing into your
user's .local/lib directory all of the required dependencies that are
missing.
@ -168,8 +142,8 @@ This should end with a 'PASSED' result::
PASSED (successes=143)
Running The Synapse Homeserver
==============================
Setting up Federation
=====================
In order for other homeservers to send messages to your server, it will need to
be publicly visible on the internet, and they will need to know its host name.
@ -215,7 +189,7 @@ Running a Demo Federation of Homeservers
If you want to get up and running quickly with a trio of homeservers in a
private federation (``localhost:8080``, ``localhost:8081`` and
``localhost:8082``) which you can then point a demo webclient at, simply run::
``localhost:8082``) which you can then access through the webclient running at http://localhost:8080. Simply run::
$ demo/start.sh
@ -279,8 +253,8 @@ as the primary means of identity and E2E encryption is not complete. As such,
we're not yet running an identity server in public.
How does it all work?!
======================
Where's the spec?!
==================
For now, please go spelunking in the ``docs/`` directory to find out.
@ -297,3 +271,4 @@ sphinxcontrib-napoleon::
Building internal API documentation::
$ python setup.py build_sphinx

View file

@ -116,9 +116,13 @@ federation.]]
Protocol URLs
=============
All these URLs are namespaced within a prefix of
/matrix/federation/v1/...
For active pushing of messages representing live activity "as it happens":
PUT /send/:transaction_id/
PUT .../send/:transaction_id/
Body: JSON encoding of a single Transaction
Response: [[TODO(paul): I don't actually understand what
@ -132,7 +136,7 @@ For active pushing of messages representing live activity "as it happens":
To fetch a particular PDU:
GET /pdu/:origin/:pdu_id/
GET .../pdu/:origin/:pdu_id/
Response: JSON encoding of a single Transaction containing one PDU
@ -142,7 +146,7 @@ To fetch a particular PDU:
To fetch all the state of a given context:
GET /state/:context/
GET .../state/:context/
Response: JSON encoding of a single Transaction containing multiple PDUs
@ -153,7 +157,7 @@ To fetch all the state of a given context:
To paginate events on a given context:
GET /paginate/:context/
GET .../paginate/:context/
Query args: v, limit
Response: JSON encoding of a single Transaction containing multiple PDUs
@ -167,7 +171,7 @@ To paginate events on a given context:
To stream events all the events:
GET /pull/
GET .../pull/
Query args: origin, v
Response: JSON encoding of a single Transaction consisting of multiple PDUs

View file

@ -37,7 +37,7 @@ class Feedback(object):
class PresenceState(object):
"""Represents the presence state of a user."""
OFFLINE = 0
BUSY = 1
ONLINE = 2
FREE_FOR_CHAT = 3
OFFLINE = u"offline"
UNAVAILABLE = u"unavailable"
ONLINE = u"online"
FREE_FOR_CHAT = u"free_for_chat"

View file

@ -88,9 +88,8 @@ def setup_logging(verbosity=0, filename=None, config_path=None):
'%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(message)s'
)
if not verbosity or verbosity == 0:
level = logging.INFO
elif verbosity == 1:
level = logging.INFO
if verbosity:
level = logging.DEBUG
# FIXME: we need a logging.WARN for a -q quiet option

View file

@ -33,6 +33,9 @@ import re
logger = logging.getLogger(__name__)
PREFIX = "/matrix/federation/v1"
class TransportLayer(object):
"""This is a basic implementation of the transport layer that translates
transactions and other requests to/from HTTP.
@ -84,9 +87,9 @@ class TransportLayer(object):
logger.debug("get_context_state dest=%s, context=%s",
destination, context)
path = "/state/%s/" % context
subpath = "/state/%s/" % context
return self._do_request_for_transaction(destination, path)
return self._do_request_for_transaction(destination, subpath)
@log_function
def get_pdu(self, destination, pdu_origin, pdu_id):
@ -104,9 +107,9 @@ class TransportLayer(object):
logger.debug("get_pdu dest=%s, pdu_origin=%s, pdu_id=%s",
destination, pdu_origin, pdu_id)
path = "/pdu/%s/%s/" % (pdu_origin, pdu_id)
subpath = "/pdu/%s/%s/" % (pdu_origin, pdu_id)
return self._do_request_for_transaction(destination, path)
return self._do_request_for_transaction(destination, subpath)
@log_function
def paginate(self, dest, context, pdu_tuples, limit):
@ -130,14 +133,14 @@ class TransportLayer(object):
if not pdu_tuples:
return
path = "/paginate/%s/" % context
subpath = "/paginate/%s/" % context
args = {"v": ["%s,%s" % (i, o) for i, o in pdu_tuples]}
args["limit"] = limit
return self._do_request_for_transaction(
dest,
path,
subpath,
args=args,
)
@ -166,7 +169,7 @@ class TransportLayer(object):
code, response = yield self.client.put_json(
transaction.destination,
path="/send/%s/" % transaction.transaction_id,
path=PREFIX + "/send/%s/" % transaction.transaction_id,
data=data
)
@ -189,7 +192,7 @@ class TransportLayer(object):
# This is when someone is trying to send us a bunch of data.
self.server.register_path(
"PUT",
re.compile("^/send/([^/]*)/$"),
re.compile("^" + PREFIX + "/send/([^/]*)/$"),
self._on_send_request
)
@ -207,7 +210,7 @@ class TransportLayer(object):
# This is for when someone asks us for everything since version X
self.server.register_path(
"GET",
re.compile("^/pull/$"),
re.compile("^" + PREFIX + "/pull/$"),
lambda request: handler.on_pull_request(
request.args["origin"][0],
request.args["v"]
@ -218,7 +221,7 @@ class TransportLayer(object):
# data_id pair.
self.server.register_path(
"GET",
re.compile("^/pdu/([^/]*)/([^/]*)/$"),
re.compile("^" + PREFIX + "/pdu/([^/]*)/([^/]*)/$"),
lambda request, pdu_origin, pdu_id: handler.on_pdu_request(
pdu_origin, pdu_id
)
@ -227,7 +230,7 @@ class TransportLayer(object):
# This is when someone asks for all data for a given context.
self.server.register_path(
"GET",
re.compile("^/state/([^/]*)/$"),
re.compile("^" + PREFIX + "/state/([^/]*)/$"),
lambda request, context: handler.on_context_state_request(
context
)
@ -235,7 +238,7 @@ class TransportLayer(object):
self.server.register_path(
"GET",
re.compile("^/paginate/([^/]*)/$"),
re.compile("^" + PREFIX + "/paginate/([^/]*)/$"),
lambda request, context: self._on_paginate_request(
context, request.args["v"],
request.args["limit"]
@ -244,7 +247,7 @@ class TransportLayer(object):
self.server.register_path(
"GET",
re.compile("^/context/([^/]*)/$"),
re.compile("^" + PREFIX + "/context/([^/]*)/$"),
lambda request, context: handler.on_context_pdus_request(context)
)
@ -300,7 +303,7 @@ class TransportLayer(object):
@defer.inlineCallbacks
@log_function
def _do_request_for_transaction(self, destination, path, args={}):
def _do_request_for_transaction(self, destination, subpath, args={}):
"""
Args:
destination (str)
@ -313,7 +316,7 @@ class TransportLayer(object):
data = yield self.client.get_json(
destination,
path=path,
path=PREFIX + subpath,
args=args,
)

View file

@ -96,7 +96,7 @@ class FederationTestCase(unittest.TestCase):
# Empty context initially
(code, response) = yield self.mock_http_server.trigger("GET",
"/state/my-context/", None)
"/matrix/federation/v1/state/my-context/", None)
self.assertEquals(200, code)
self.assertFalse(response["pdus"])
@ -121,7 +121,7 @@ class FederationTestCase(unittest.TestCase):
)
(code, response) = yield self.mock_http_server.trigger("GET",
"/state/my-context/", None)
"/matrix/federation/v1/state/my-context/", None)
self.assertEquals(200, code)
self.assertEquals(1, len(response["pdus"]))
@ -132,7 +132,7 @@ class FederationTestCase(unittest.TestCase):
)
(code, response) = yield self.mock_http_server.trigger("GET",
"/pdu/red/abc123def456/", None)
"/matrix/federation/v1/pdu/red/abc123def456/", None)
self.assertEquals(404, code)
# Now insert such a PDU
@ -151,7 +151,7 @@ class FederationTestCase(unittest.TestCase):
)
(code, response) = yield self.mock_http_server.trigger("GET",
"/pdu/red/abc123def456/", None)
"/matrix/federation/v1/pdu/red/abc123def456/", None)
self.assertEquals(200, code)
self.assertEquals(1, len(response["pdus"]))
self.assertEquals("m.text", response["pdus"][0]["pdu_type"])
@ -177,7 +177,7 @@ class FederationTestCase(unittest.TestCase):
self.mock_http_client.put_json.assert_called_with(
"remote",
path="/send/1000000/",
path="/matrix/federation/v1/send/1000000/",
data={
"ts": 1000000,
"origin": "test",
@ -212,7 +212,7 @@ class FederationTestCase(unittest.TestCase):
# MockClock ensures we can guess these timestamps
self.mock_http_client.put_json.assert_called_with(
"remote",
path="/send/1000000/",
path="/matrix/federation/v1/send/1000000/",
data={
"origin": "test",
"ts": 1000000,
@ -234,7 +234,8 @@ class FederationTestCase(unittest.TestCase):
self.federation.register_edu_handler("m.test", recv_observer)
yield self.mock_http_server.trigger("PUT", "/send/1001000/",
yield self.mock_http_server.trigger("PUT",
"/matrix/federation/v1/send/1001000/",
"""{
"origin": "remote",
"ts": 1001000,

View file

@ -27,7 +27,7 @@ from synapse.handlers.presence import PresenceHandler, UserPresenceCache
OFFLINE = PresenceState.OFFLINE
BUSY = PresenceState.BUSY
UNAVAILABLE = PresenceState.UNAVAILABLE
ONLINE = PresenceState.ONLINE
@ -149,12 +149,12 @@ class PresenceStateTestCase(unittest.TestCase):
yield self.handler.set_state(
target_user=self.u_apple, auth_user=self.u_apple,
state={"state": BUSY, "status_msg": "Away"})
state={"state": UNAVAILABLE, "status_msg": "Away"})
mocked_set.assert_called_with("apple",
{"state": 1, "status_msg": "Away"})
{"state": UNAVAILABLE, "status_msg": "Away"})
self.mock_start.assert_called_with(self.u_apple,
state={"state": 1, "status_msg": "Away"})
state={"state": UNAVAILABLE, "status_msg": "Away"})
yield self.handler.set_state(
target_user=self.u_apple, auth_user=self.u_apple,
@ -555,7 +555,7 @@ class PresencePushTestCase(unittest.TestCase):
content={
"push": [
{"user_id": "@apple:test",
"state": 2},
"state": "online"},
],
}),
call(
@ -564,7 +564,7 @@ class PresencePushTestCase(unittest.TestCase):
content={
"push": [
{"user_id": "@apple:test",
"state": 2},
"state": "online"},
],
})
], any_order=True)
@ -582,7 +582,7 @@ class PresencePushTestCase(unittest.TestCase):
"remote", "m.presence", {
"push": [
{"user_id": "@potato:remote",
"state": 2},
"state": "online"},
],
}
)
@ -646,7 +646,7 @@ class PresencePushTestCase(unittest.TestCase):
content={
"push": [
{"user_id": "@apple:test",
"state": 2},
"state": "online"},
],
}),
call(
@ -655,7 +655,7 @@ class PresencePushTestCase(unittest.TestCase):
content={
"push": [
{"user_id": "@banana:test",
"state": 0},
"state": "offline"},
],
}),
], any_order=True)
@ -666,7 +666,7 @@ class PresencePushTestCase(unittest.TestCase):
self.handler._user_cachemap[self.u_clementine] = UserPresenceCache()
self.handler._user_cachemap[self.u_clementine].update(
{"state": PresenceState.ONLINE}, self.u_clementine)
{"state": ONLINE}, self.u_clementine)
self.room_members.append(self.u_potato)
yield self.distributor.fire("user_joined_room", self.u_clementine,
@ -680,7 +680,7 @@ class PresencePushTestCase(unittest.TestCase):
content={
"push": [
{"user_id": "@clementine:test",
"state": 2},
"state": "online"},
],
}),
)
@ -882,7 +882,7 @@ class PresencePollingTestCase(unittest.TestCase):
content={
"push": [
{"user_id": "@banana:test",
"state": 0,
"state": "offline",
"status_msg": None},
],
},

View file

@ -29,7 +29,7 @@ from synapse.handlers.profile import ProfileHandler
OFFLINE = PresenceState.OFFLINE
BUSY = PresenceState.BUSY
UNAVAILABLE = PresenceState.UNAVAILABLE
ONLINE = PresenceState.ONLINE
@ -125,12 +125,12 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
yield self.handlers.presence_handler.set_state(
target_user=self.u_apple, auth_user=self.u_apple,
state={"state": BUSY, "status_msg": "Away"})
state={"state": UNAVAILABLE, "status_msg": "Away"})
mocked_set.assert_called_with("apple",
{"state": 1, "status_msg": "Away"})
{"state": UNAVAILABLE, "status_msg": "Away"})
self.mock_start.assert_called_with(self.u_apple,
state={"state": 1, "status_msg": "Away",
state={"state": UNAVAILABLE, "status_msg": "Away",
"displayname": "Frank",
"avatar_url": "http://foo"})
@ -220,7 +220,7 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
content={
"push": [
{"user_id": "@apple:test",
"state": 2,
"state": "online",
"displayname": "Frank",
"avatar_url": "http://foo"},
],
@ -238,7 +238,7 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
"remote", "m.presence", {
"push": [
{"user_id": "@potato:remote",
"state": 2,
"state": "online",
"displayname": "Frank",
"avatar_url": "http://foo"},
],

View file

@ -31,7 +31,7 @@ logging.getLogger().addHandler(logging.NullHandler())
OFFLINE = PresenceState.OFFLINE
BUSY = PresenceState.BUSY
UNAVAILABLE = PresenceState.UNAVAILABLE
ONLINE = PresenceState.ONLINE
@ -69,7 +69,7 @@ class PresenceStateTestCase(unittest.TestCase):
def test_get_my_status(self):
mocked_get = self.mock_handler.get_state
mocked_get.return_value = defer.succeed(
{"state": 2, "status_msg": "Available"})
{"state": ONLINE, "status_msg": "Available"})
(code, response) = yield self.mock_server.trigger("GET",
"/presence/%s/status" % (myid), None)
@ -87,12 +87,12 @@ class PresenceStateTestCase(unittest.TestCase):
(code, response) = yield self.mock_server.trigger("PUT",
"/presence/%s/status" % (myid),
'{"state": 1, "status_msg": "Away"}')
'{"state": "unavailable", "status_msg": "Away"}')
self.assertEquals(200, code)
mocked_set.assert_called_with(target_user=self.u_apple,
auth_user=self.u_apple,
state={"state": 1, "status_msg": "Away"})
state={"state": UNAVAILABLE, "status_msg": "Away"})
class PresenceListTestCase(unittest.TestCase):
@ -234,7 +234,7 @@ class PresenceEventStreamTestCase(unittest.TestCase):
# I'll already get my own presence state change
self.assertEquals({"start": "0", "end": "1", "chunk": [
{"type": "m.presence",
"content": {"user_id": "@apple:test", "state": 2}},
"content": {"user_id": "@apple:test", "state": ONLINE}},
]}, response)
self.mock_datastore.set_presence_state.return_value = defer.succeed(
@ -251,5 +251,5 @@ class PresenceEventStreamTestCase(unittest.TestCase):
self.assertEquals(200, code)
self.assertEquals({"start": "1", "end": "2", "chunk": [
{"type": "m.presence",
"content": {"user_id": "@banana:test", "state": 2}},
"content": {"user_id": "@banana:test", "state": ONLINE}},
]}, response)

View file

@ -106,7 +106,7 @@ h1 {
background-color: #38AF00;
}
.away {
.unavailable {
background-color: #FFCC00;
}

View file

@ -123,17 +123,14 @@ angular.module('RoomController', [])
var member = $scope.members[chunk.content.user_id];
if ("state" in chunk.content) {
var ONLINE = 2;
var AWAY = 1;
var OFFLINE = 0;
if (chunk.content.state === ONLINE) {
if (chunk.content.state === "online") {
member.presenceState = "online";
}
else if (chunk.content.state === OFFLINE) {
else if (chunk.content.state === "offline") {
member.presenceState = "offline";
}
else if (chunk.content.state === AWAY) {
member.presenceState = "away";
else if (chunk.content.state === "unavailable") {
member.presenceState = "unavailable";
}
}

View file

@ -14,7 +14,7 @@
<img class="userAvatarGradient" src="img/gradient.png" width="80" height="24"/>
<div class="userName">{{ info.displayname || name }}</div>
</td>
<td class="userPresence" ng-class="info.presenceState === 'online' ? 'online' : (info.presenceState === 'away' ? 'away' : '')" />
<td class="userPresence" ng-class="info.presenceState === 'online' ? 'online' : (info.presenceState === 'unavailable' ? 'unavailable' : '')" />
</table>
</div>