operations_service.rst 10.2 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
The Operations Service
======================

The peer and the orderer host an HTTP server that offers a RESTful "operations"
API. This API is unrelated to the Fabric network services and is intended to be
used by operators, not administrators or "users" of the network.

The API exposes the following capabilities:

- Log level management
- Health checks
- Prometheus target for operational metrics (when configured)

Configuring the Operations Service
----------------------------------

The operations service requires two basic pieces of configuration:

19
20
21
22
23
- The **address** and **port** to listen on.
- The **TLS certificates** and **keys** to use for authentication and encryption.
  Note, **these certificates should be generated by a separate and dedicated CA**.
  Do not use a CA that has generated certificates for any organizations
  in any channels.
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

Peer
~~~~

For each peer, the operations server can be configured in the ``operations``
section of ``core.yaml``:

.. code:: yaml

  operations:
    # host and port for the operations server
    listenAddress: 127.0.0.1:9443

    # TLS configuration for the operations endpoint
    tls:
      # TLS enabled
      enabled: true

      # path to PEM encoded server certificate for the operations server
      cert:
        file: tls/server.crt

      # path to PEM encoded server key for the operations server
      key:
        file: tls/server.key

50
51
52
      # most operations service endpoints require client authentication when TLS
      # is enabled. clientAuthRequired requires client certificate authentication
      # at the TLS layer to access all resources.
53
54
55
56
57
58
59
60
61
62
63
64
65
      clientAuthRequired: false

      # paths to PEM encoded ca certificates to trust for client authentication
      clientRootCAs:
        files: []

The ``listenAddress`` key defines the host and port that the operation server
will listen on. If the server should listen on all addresses, the host portion
can be omitted.

The ``tls`` section is used to indicate whether or not TLS is enabled for the
operations service, the location of the service's certificate and private key,
and the locations of certificate authority root certificates that should be
66
67
68
69
70
trusted for client authentication. When ``enabled`` is true, most of the operations
service endpoints require client authentication, therefore
``clientRootCAs.files`` must be set. When ``clientAuthRequired`` is ``true``,
the TLS layer will require clients to provide a certificate for authentication
on every request. See Operations Security section below for more details.
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

Orderer
~~~~~~~

For each orderer, the operations server can be configured in the `Operations`
section of ``orderer.yaml``:

.. code:: yaml

  Operations:
    # host and port for the operations server
    ListenAddress: 127.0.0.1:8443

    # TLS configuration for the operations endpoint
    TLS:
      # TLS enabled
      Enabled: true

      # PrivateKey: PEM-encoded tls key for the operations endpoint
      PrivateKey: tls/server.key

      # Certificate governs the file location of the server TLS certificate.
      Certificate: tls/server.crt

      # Paths to PEM encoded ca certificates to trust for client authentication
96
      ClientRootCAs: []
97

98
99
100
      # Most operations service endpoints require client authentication when TLS
      # is enabled. ClientAuthRequired requires client certificate authentication
      # at the TLS layer to access all resources.
101
102
103
104
105
106
107
108
109
      ClientAuthRequired: false

The ``ListenAddress`` key defines the host and port that the operations server
will listen on. If the server should listen on all addresses, the host portion
can be omitted.

The ``TLS`` section is used to indicate whether or not TLS is enabled for the
operations service, the location of the service's certificate and private key,
and the locations of certificate authority root certificates that should be
110
111
112
113
114
trusted for client authentication.   When ``Enabled`` is true, most of the operations
service endpoints require client authentication, therefore
``RootCAs`` must be set. When ``ClientAuthRequired`` is ``true``,
the TLS layer will require clients to provide a certificate for authentication
on every request. See Operations Security section below for more details.
115
116
117
118
119
120
121
122
123
124
125
126

Operations Security
~~~~~~~~~~~~~~~~~~~

As the operations service is focused on operations and intentionally unrelated
to the Fabric network, it does not use the Membership Services Provider for
access control. Instead, the operations service relies entirely on mutual TLS with
client certificate authentication.

When TLS is disabled, authorization is bypassed and any client that can
connect to the operations endpoint will be able to use the API.

127
128
129
130
131
132
When TLS is enabled, a valid client certificate must be provided in order to
access all resources unless explicitly noted otherwise below.

When clientAuthRequired is also enabled, the TLS layer will require
a valid client certificate regardless of the resource being accessed.

133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
Log Level Management
~~~~~~~~~~~~~~~~~~~~

The operations service provides a ``/logspec`` resource that operators can use to
manage the active logging spec for a peer or orderer. The resource is a
conventional REST resource and supports ``GET`` and ``PUT`` requests.

When a ``GET /logspec`` request is received by the operations service, it will
respond with a JSON payload that contains the current logging specification:

.. code:: json

  {"spec":"info"}

When a ``PUT /logspec`` request is received by the operations service, it will
read the body as a JSON payload. The payload must consist of a single attribute
named ``spec``.

.. code:: json

  {"spec":"chaincode=debug:info"}

If the spec is activated successfully, the service will respond with a ``204 "No Content"``
response. If an error occurs, the service will respond with a ``400 "Bad Request"``
and an error payload:

.. code:: json

  {"error":"error message"}

Health Checks
-------------

The operations service provides a ``/healthz`` resource that operators can use to
help determine the liveness and health of peers and orderers. The resource is
a conventional REST resource that supports GET requests. The implementation is
intended to be compatible with the liveness probe model used by Kubernetes but
can be used in other contexts.

When a ``GET /healthz`` request is received, the operations service will call all
registered health checkers for the process. When all of the health checkers
return successfully, the operations service will respond with a ``200 "OK"`` and a
JSON body:

.. code:: json

  {
    "status": "OK",
    "time": "2009-11-10T23:00:00Z"
  }

If one or more of the health checkers returns an error, the operations service
will respond with a ``503 "Service Unavailable"`` and a JSON body that includes
information about which health checker failed:

.. code:: json

  {
    "status": "Service Unavailable",
    "time": "2009-11-10T23:00:00Z",
    "failed_checks": [
      {
        "component": "docker",
        "reason": "failed to connect to Docker daemon: invalid endpoint"
      }
    ]
  }

In the current version, the only health check that is registered is for Docker.
Future versions will be enhanced to add additional health checks.

When TLS is enabled, a valid client certificate is not required to use this
205
service unless ``clientAuthRequired`` is set to ``true``.
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307

Metrics
-------

Some components of the Fabric peer and orderer expose metrics that can help
provide insight into the behavior of the system. Operators and administrators
can use this information to better understand how the system is performing
over time.

Configuring Metrics
~~~~~~~~~~~~~~~~~~~

Fabric provides two ways to expose metrics: a **pull** model based on Prometheus
and a **push** model based on StatsD.

Prometheus
~~~~~~~~~~

A typical Prometheus deployment scrapes metrics by requesting them from an HTTP
endpoint exposed by instrumented targets. As Prometheus is responsible for
requesting the metrics, it is considered a pull system.

When configured, a Fabric peer or orderer will present a ``/metrics`` resource
on the operations service.

Peer
^^^^

A peer can be configured to expose a ``/metrics`` endpoint for Prometheus to
scrape by setting the metrics provider to ``prometheus`` in the ``metrics`` section
of ``core.yaml``.

.. code:: yaml

  metrics:
    provider: prometheus

Orderer
^^^^^^^

An orderer can be configured to expose a ``/metrics`` endpoint for Prometheus to
scrape by setting the metrics provider to ``prometheus`` in the ``Metrics``
section of ``orderer.yaml``.

.. code:: yaml

  Metrics:
    Provider: prometheus

StatsD
~~~~~~

StatsD is a simple statistics aggregation daemon. Metrics are sent to a
``statsd`` daemon where they are collected, aggregated, and pushed to a backend
for visualization and alerting. As this model requires instrumented processes
to send metrics data to StatsD, this is considered a push system.

Peer
^^^^

A peer can be configured to send metrics to StatsD by setting the metrics
provider to ``statsd`` in the ``metrics`` section of ``core.yaml``. The ``statsd``
subsection must also be configured with the address of the StatsD daemon, the
network type to use (``tcp`` or ``udp``), and how often to send the metrics. An
optional ``prefix`` may be specified to help differentiate the source of the
metrics --- for example, differentiating metrics coming from separate peers ---
that would be prepended to all generated metrics.

.. code:: yaml

  metrics:
    provider: statsd
    statsd:
      network: udp
      address: 127.0.0.1:8125
      writeInterval: 10s
      prefix: peer-0

Orderer
^^^^^^^

An orderer can be configured to send metrics to StatsD by setting the metrics
provider to ``statsd`` in the ``Metrics`` section of ``orderer.yaml``. The ``Statsd``
subsection must also be configured with the address of the StatsD daemon, the
network type to use (``tcp`` or ``udp``), and how often to send the metrics. An
optional ``prefix`` may be specified to help differentiate the source of the
metrics.

.. code:: yaml

  Metrics:
      Provider: statsd
      Statsd:
        Network: udp
        Address: 127.0.0.1:8125
        WriteInterval: 30s
        Prefix: org-orderer

For a look at the different metrics that are generated, check out
:doc:`metrics_reference`.

.. Licensed under Creative Commons Attribution 4.0 International License
308
   https://creativecommons.org/licenses/by/4.0/