cylc protocols & APIs

10 Members
HTTPS, WebSocket, ZeroMQ; REST, GraphQL1 Servers

Load older messages

Timestamp Message
24 Mar 2019
00:06:25@dwsutherland:matrix.orgDavid Sutherland

That article is old.. CPP is the default implementation

(protographene) sutherlander@cortex-vbox:learn_protobuf$ pip freeze
(protographene) sutherlander@cortex-vbox:learn_protobuf$ python -c "from google.protobuf.internal import api_implementation; print(api_implementation._default_implementation_type)"

So we only need pip install protobuf on any platform... see this issue:

C++ is the default implementation for python

00:07:49@dwsutherland:matrix.orgDavid SutherlandThere's probably a way of of making it pure python (which is slower).. but would we need that?
00:12:57@dwsutherland:matrix.orgDavid SutherlandThe protobuf project is really active, I'm sure more improvements and fixes are made all the time, last release 2019-03-01: https://github.com/protocolbuffers/protobuf/releases
01:25:13@revilo666:matrix.orgHilary Oliver

So we only need pip install protobuf on any platform...

David Sutherland - what you're claiming is not possible in general, unless binary runtimes are supplied for "any platform". Still, it's probably good news for the platforms that we support at the moment, since all HPC has gone Linux theses days.

01:28:20@revilo666:matrix.orgHilary OliverA secondary question is: if using the C++ protobuf runtime, does our own message schema (or whatever the protobuf terminology is for that) have to be compiled into C++ or just "compiled" into Python? (i.e. does it create Python classes as an interface to the C++ runtime). If the latter (Python) then I guess we are good to go so long as we only support platforms that the C++ runtime is available for.
01:29:44@revilo666:matrix.orgHilary Oliver *platforms that the pre-compiled C++ runtime is available for.
01:51:16@dwsutherland:matrix.orgDavid Sutherlandhmm .... they are supplied, and maybe it reverts to python if C++ not present?
01:51:33@revilo666:matrix.orgHilary OliverAfter a bit more looking ... "the C++ implementation for Python messages is built as a Python extension ... to use it you need to Install the C++ protobuf runtime library,"
01:51:57@dwsutherland:matrix.orgDavid Sutherlandmaybe not included in the pip install
01:52:01@dwsutherland:matrix.orgDavid Sutherlandof course
01:52:24@revilo666:matrix.orgHilary OliverQuestion is, how to you "install the C++ protobuf runtime library" - is it just another binary dependency that Cylc will acquire, requiring separate installation via the sysetm package manager?
01:53:18@revilo666:matrix.orgHilary Oliver(If so, that's fine by me!)
01:58:02@dwsutherland:matrix.orgDavid SutherlandI guess it's important that is works without it also
01:58:37@dwsutherland:matrix.orgDavid Sutherlandbut being able to install it easily if possible should be our default I suppose
03:26:20@dwsutherland:matrix.orgDavid Sutherland

There is also a C++ implementation for Python messages via a Python extension for better performance. Implementation type is controlled by an environment variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION (valid values: "cpp" and "python"). The default value is currently "python" but will be changed to "cpp" in future release.

Note that the environment variable needs to be set before installing the protobuf library, in order to build and install the python extension. The C++ implementation also requires CPython platforms. See python/INSTALL.txt for detailed install instructions.

At the bottom of

05:52:46@dwsutherland:matrix.orgDavid Sutherland

Hmm.. is it not available (o,O)

05:52:59@dwsutherland:matrix.orgDavid Sutherland
(protographene) sutherlander@cortex-vbox:protobuf$ ls pyext/
cpp_message.py  __init__.py  _message.cpython-37m-x86_64-linux-gnu.so  __pycache__  python_pb2.py
(protographene) sutherlander@cortex-vbox:protobuf$ python -c "from google.protobuf.internal import api_implementation; print(api_implementation._default_implementation_type)"
(protographene) sutherlander@cortex-vbox:protobuf$ python
Python 3.7.1 (default, Oct 22 2018, 11:21:55) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.protobuf.pyext import _message
__main__:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
>>> help(_message)

>>> exit()
(protographene) sutherlander@cortex-vbox:protobuf$ python
Python 3.7.1 (default, Oct 22 2018, 11:21:55) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.protobuf.empty_pb2 import Empty; Empty.FromString(b'')

>>> exit()
06:26:05@dwsutherland:matrix.orgDavid Sutherland Other wise the readme here or downloading prebuilt according to the system info? ...
06:27:12@dwsutherland:matrix.orgDavid Sutherlandhttps://github.com/protocolbuffers/protobuf/releases
06:34:36@dwsutherland:matrix.orgDavid SutherlandIdk .. I'll have a play tomorrow
20:53:36@kinow:matrix.orgBruno P. Kinoshita Their setup.py is quite interesting! First time reading it. There is some code for searching which protoc is available, and also for deciding to build with c++ extensions: https://github.com/protocolbuffers/protobuf/blob/3a3956e8a258784461270961c6577341356bce52/python/setup.py#L167
25 Mar 2019
00:44:42@kinow:matrix.orgBruno P. Kinoshita

Had a look at the issue linked by David Sutherland (really useful, thanks!), and also at their setup.py and release.sh.

I believe they now provide the extension module (C/C++?) for protobuf, but for doing that, they provide the module for mac/linux, and a different package for windows. pip will choose which version to install (and possibly conda too) based on the user platform. It is possible to find the packages available here: https://pypi.org/project/protobuf/#files

If you compare with isodatetime, for now we ship only a simple py3 wheel (binary, which is actually a zip, just like java's jar/ear/war/etc).

So I believe Linux/Mac users should be using the fast Python + C extension when installing the dependencies via setup.py. i.e. once we have Cylc's setup.py working, running pip install cylc --user should install cylc and dependencies, including protobuf with its fast module 🤞

00:45:41@kinow:matrix.orgBruno P. Kinoshita Learned a lot just reading their setup.py. But definitely don't want to start with such a complex setup.py. All good if we evolve slowly to a more complex build pipeline, but for now just solving the CYLC_DIR issue and getting it running in virtualenv/Conda, and normal Python env will be great ahaah
00:47:09@dwsutherland:matrix.orgDavid Sutherland👍
00:47:14@dwsutherland:matrix.orgDavid SutherlandNice
00:47:24@revilo666:matrix.orgHilary OliverInteresting, sounds good. Agree with the complexity thing - start simple!
26 Mar 2019
02:20:22@dwsutherland:matrix.orgDavid SutherlandApologies - I hope to get some action on this in the next few days
02:21:33@kinow:matrix.orgBruno P. Kinoshita All good David Sutherland , I'm busy with setup.py and a few issues blocking its use. But hoping to be able to go back to UI server in a few days. And thanks for the review on that xrandom pull request.
02:21:46@kinow:matrix.orgBruno P. KinoshitaBut I thought that would be a much much simpler pull request
02:24:21@dwsutherland:matrix.orgDavid SutherlandAll good, first one!

There are no newer messages yet.

Back to Room List