gRPC, proto file

PeterMM · December 11, 2019, 12:06pm

I’m looking into ZeeBe to find out if I can access ZeeBe from our own applications written mostly in Delphi. Looking at the docs that should be possible, using gRPC.

So far I’ve succeeded installing running and connecting to ZeBee. I’ve been able to import the gateway.proto file and build the corresponding delphi data-structures. I was able to call “rpc Topology” and “rpc CreateWorkflowInstance” , so basically it seems OK.

But now I’m running into some troubles: rpc ListWorkflows fails with error “12” (not found) so does rpc CreateJob

And what about CreateWorker? (not in the proto file)….

Any suggestions?

jwulf · December 11, 2019, 1:18pm

Read the source code of the Node client: https://github.com/creditsenseau/zeebe-client-node-js

Your proto file is out of date. Use 0.21 or later of the broker, and this file: https://github.com/creditsenseau/zeebe-client-node-js/blob/master/proto/zeebe.proto

There is no listWorkflows. It was removed.

CreateWorker is not a thing. You need to write a class that sends an ActivateJobs gRPC command. See the ZBWorker class in the Node library.

If you make your client an open-source lib on GitHub, I’m happy to help with PRs. I wrote and maintain the Node lib, and I’m a Delphi dev from back in the day.

PeterMM · December 11, 2019, 2:03pm

Txs jwulf for pointing me to the right direction. That’s what if expected: the proto file is outdated. So I’ll restart with the current one.
At the moment, it is all out of curiosity and to see what it would take to interface to zeebe. Maybe I’ll come back to your offer…

PeterMM · December 12, 2019, 3:19pm

I’ve updated to the current zeebe.proto. Looks way better now (ListWorkflows, CreateJob gone).
Before I start with the Worker Class, I’ll try something simpler (hope so): CreateWorkflowInstance. According to the proto-file the request has 4 properties (Delphi Syntax):

   TCreateWorkflowInstanceRequest = record
     workflowKey: UInt64;
     bpmnProcessId: string;
     version: Integer;
     variables: string;
   end;

But If I call CreateWorkflowInstance like that, I get an error 13 (Unexpected error occurred during the request processing). I found that it succeeds if I define the request like that:

   TCreateWorkflowInstanceRequest = record
     workflowKey: UInt64;
     version: Integer;
     variables: string;
   end;

(that is: without the bpmnProcessId string). Strange to me: It is against the definition in proto, I need to now the key (looked it up using the ZeeBe Simple Monitor – it works ) and the variables seems to be ignored.

PeterMM · December 12, 2019, 3:34pm

Additional info: If I define the request like this:

   TCreateWorkflowInstanceRequest = record
     bpmnProcessId: string;
     version: Integer;
     variables: string;
  end;

I get: Error in CreateWorkflowInstance: Server error “5” with message: Command rejected with code ‘CREATE’: Expected to find workflow definition with key ‘0’, but none found

jwulf · December 13, 2019, 2:22am

See this note in the Node client README: https://github.com/creditsenseau/zeebe-client-node-js#type-difference-from-other-zeebe-clients

You may have to dig in to the serialization of the gRPC library you are using. It took a bunch of time to track that down in the Node client.

jwulf · December 13, 2019, 6:18am

See here: https://github.com/creditsenseau/zeebe-client-node-js/blob/master/src/zb/ZBClient.ts#L292

You should set either the bpmnProcessId or the workflowKey, but not both. I use bpmnProcessId only.

I would inspect the request you are sending. In that second example, what is the actual runtime value of that string?

PeterMM · December 13, 2019, 8:41am

IC - there are two variants for that request - Can’t be seen from the definition in the proto file.
The runtime value of ProcessId in my example is a string with value ‘order-process’.
Inspecting the request is sort of “moving target” as long as you didn’t know what the master is expecting….

PeterMM · December 13, 2019, 8:53am

Digging into 3-party libraries is not what I like to do at the moment - wanted to build a “proof of concept” for solving a real world problem… Maybe I can do so by using the zbcli only.
But if I get time for this task: Do you know of an gRPC Monitor that can show me, what the master really receives?

jwulf · December 14, 2019, 12:10am

This is the type definition no?

Somewhere in your code you must be creating an instance of the record. I would log out that record.

Which gRPC library are you using? I see a couple of different ones for Delphi.

PeterMM · December 14, 2019, 8:24am

Which gRPC library are you using?
If found only this one:
GitHub - ultraware/DelphiGrpc: DelphiGrpc is a Delphi implementation of the realtime and streaming gRPC protocol (http://grpc.io).
up to now.

Somewhere in your code you must be creating an instance of the record.
The request record Definition is (like yours):

   TCreateWorkflowInstanceRequest_1 = record
     bpmnProcessId: String;
     version: Integer;
     variables: string;
   end;

and initiated like that:

var
  aWFIRequest: TCreateWorkflowInstanceRequest_1;
  aWFIResponce: TCreateWorkflowInstanceResponse;

begin
   :
  aWFIRequest.bpmnProcessId := 'order-process';
  aWFIRequest.version := -1;
  aWFIRequest.variables := '{\"orderId\": \"1234\", \"orderValue\": \"99\"}';
  :

(in Delhi, records are static - so no instanciation).
I don’t know much About gRPC, But it would be nice to have an monitoring tool, that shows, which data the Server side receives. I can build a stub-server for that using DelphiGrpc, but that would hide problems in the LIB Code if that is symmetric. A possibilty at ZeeBe server side that reflects the received request would be best…

PeterMM · December 14, 2019, 2:53pm

I found some time to debug into the send request. The serializer is producing two arrays:

aRecord:
('order-process', -1, '{\"orderId\": \"1234\", \"orderValue\": \"99\"}')
Result:
(10, 13, 111, 114, 100, 101, 114, 45, 112, 114, 111, 99, 101, 115, 115, 16, 1, 26, 47, 123, 92, 34, 111, 114, 100, 101, 114, 73, 100, 92, 34, 58, 32, 92, 34, 49, 50, 51, 52, 92, 34, 44, 32, 92, 34, 111, 114, 100, 101, 114, 86, 97, 108, 117, 101, 92, 34, 58, 32, 92, 34, 57, 57, 92, 34, 125)

and the header:

aHeader:
(0, 0, 0, 0, 66)

Altogether the result / request is combined like that:

(0, 0, 0, 0, 66, 10, 13, 111, 114, 100, 101, 114, 45, 112, 114, 111, 99, 101, 115, 115, 16, 1, 26, 47, 123, 92, 34, 111, 114, 100, 101, 114, 73, 100, 92, 34, 58, 32, 92, 34, 49, 50, 51, 52, 92, 34, 44, 32, 92, 34, 111, 114, 100, 101, 114, 86, 97, 108, 117, 101, 92, 34, 58, 32, 92, 34, 57, 57, 92, 34, 125)

I wonder about the CR/LF (10,13) sequence in front of the data block. Is that OK?

Amendment:
It is not CR/LF: A closer look at the serialization showed me that $0A ist a tag-field, while $0D is the length of the string ‘order-process’.

PeterMM · December 18, 2019, 12:21pm

Someone around who can point me to the right direction?
How should the “protocol-buffer” represention of a createWorkflowInstanceRequest should look like? (protocol-buffer depends on the correct sequence - right?)

The proto file says:

  int64 workflowKey = 1;
  string bpmnProcessId = 2;
  int32 version = 3;
  string variables = 4;

The JS-Client source says (ZBClient.ts #299ff):

  this.executeOperation('createWorkflowInstance', () =>
     this.gRPCClient.createWorkflowInstanceSync(
  	  stringifyVariables(createWorkflowInstanceRequest)
)

where createWorkflowInstanceRequest looks like that:

  {
   bpmnProcessId: string
   variables: Variables
   version: number
  }

OK that is jason-serialization only which has fieldnames. But I don’t know about the conversion to protocol buffers.

My own delphi testing showed that only this structure ist accepted:

    TCreateWorkflowInstanceRequest = record
      [Serialize(1)] variables: string;
      [Serialize(2)] bpmnProcessId: string;
      [Serialize(3)] version: Integer;
      [Serialize(4)] workflowKey: UInt64;
    end;

(Remark: the [Serialize(n)] anotations are used by the serializer that fills the “protocol buffer” byte-array. See example in my previous post).
That works, but the variables seems to be ignored.

Zelldon · December 19, 2019, 10:08am

Hey @PeterMM ,

sorry that you currently have some problems with using Zeebe.

Maybe I can help a bit here. As @jwulf already wrote bpmnProcessId and workflowKey should be used exclusively. The Broker expects either one or the other value. If you use workflowKey you don’t need to specify the version, because it is the unique identifier of that workflow. If you use bpmnProcessId you need to specify the version as well, because you could deploy different version with the same process id. You can also see this in the Java API that you can either specify the bpmnProcessId or the workflowKey https://github.com/zeebe-io/zeebe/blob/develop/clients/java/src/main/java/io/zeebe/client/api/command/CreateWorkflowInstanceCommandStep1.java

Hope that helps.

P.S.: We probably should add a hint in the gateway protocol comments as well.

Greets
Chris

PeterMM · December 19, 2019, 1:05pm

Txs Chris,
what I try to do is build a client in Delphi and to do so I need some more basic information, normally given in the gRPC proto file:

As you said there are two alternative implementations for a “create a workflow instance” RPC. One that takes bpmProcessID, version, variables as input and another one that takes workflowkey, version, variables – right? (the proto file shows only one, taking all 4 Parameters - no optionals, no defaults)

Are both versions reachable, using gRPC, with the same RPC call? According to the proto file there is only one call named “rpc CreateWorkflowInstance”.

gRPC and the underlaying “protocol buffers” are depending on the right field sequence. So what is right sequence for the parameters? Again, the proto file only shows a sequence for “rpc CreateWorkflowInstance” that takes all 4 Parameters (workflowkey ,bpmProcessID, version, variables in that sequence).

And sorry, my knowledge in JAVA / JS is very limited. So I wasn’t able to find the information I need in your source code.

Zelldon · December 19, 2019, 1:13pm

Hey @PeterMM ,

ah OK. I might misunderstood your problem before. So you need to write this protobuf thing by yourself because there is no official delphi support right?

Did you tried https://github.com/ultraware/DelphiGrpc ?

I think you need to send all field but with null values. The order is defined in the protocol.

Greets
Chris

PeterMM · December 19, 2019, 1:44pm

Yes Ultraware DelphtíGrpc is the library I use. But for to do so, I need the Infos I’ve asked for. That is all about gPRC, not Delphi.

Zelldon · December 20, 2019, 5:56am

Normally if you use a grpc lib it generates the needed code based on the given protocol for you. So normally you don’t need this information.

Here you can see the generated code for go https://github.com/zeebe-io/zeebe/blob/develop/clients/go/pkg/pb/gateway.pb.go#L516

As I already wrote "
I think you need to send all field but with null values. The order is defined in the protocol."

Greets
Chris

PeterMM · December 20, 2019, 8:08am

Yes, but the base info for all that ist a reliable “.proto” file. Thats basicly what I’m asking for. Please reread my previous posts about the problems I found with existing one.

Using some „try and error“ I finally found, that the “null” value for workflowKey is -1 (for bpmnProcessId I guess it is an empty string – don’t tested).
Second problem was, that the Delphi gRPC has problems serializing negative integer values. As a workaround I had to redefine the integer fields to unsigned integer. So the Delphi Definition now looks like that:

   TCreateWorkflowInstanceRequest_N = record
      [Serialize(1)] workflowKey: UInt64; {null: UInt64(-1) }
      [Serialize(2)] bpmnProcessId: string;
      [Serialize(3)] version: UInt32; {latest: UInt(-1)}
      [Serialize(4)] variables: string;
   end;

A great help to find out was gRPC-Tools (https://github.com/bradleyjkemp/grpc-tools). Especially grpc-dump shows what’s going on at the protocol level.

jwulf · December 21, 2019, 8:01am

Legendary.